Requests Reset - NAT

Hellos guys,

I have a vyos firewall. We are experiencing random accessibility problems (works 2 out of 10 times) to our web server. A nat rule is made from the ETH0 interface on our public ip to the eth1 interface to the internal ip of the server port 443.
The server is a virtual machine on ESXi with VMXNET3 drivers. The version of vyos is: 1.2.1.
Below you will find the connection logs. At the top a failed request and at the bottom an OK request.

If you have any ideas, I’m interested.

Thanks in advance,

“Not OK” packets show responses from the server with the RST flag:
RST (reset): Signify the connection is down or maybe the service is not accepting the requests

Requests must first come to the server for this to work. Thus, it looks like VyOS is passing packets to the server, but the server refuses to establish a connection for some reason.

Hello and Thanks Nikolaï,

I tried with a different web server and a different port. Random connections are the same.

Can you provide traffic capture from the server?

Requests KO : I have no log that appears

Requests OK :

host side (customer):

Requests KO :

host side (customer) :
Requests OK :

Is there some triangular routing? On first KO log, I only see traffic from the server to client, not the other way around
On 1st OK log, 2-way traffic is OK

Hello 16again,

Nop, we have only one routing. It’s the Vyos.


Is eth0 WAN interface?
There must be something in front.
1st log shows responses, whereas original requests never shows up.

What is MAC address of source in 1st KO example? Maybe some other host on WAN has same IP, and you see its reply

Requests from the client to the server web are displayed on the firewall in R. (reset)
Yes Eth0 is WAN interface
I tried with another ip wan and it’s samething.

But you can’t have responses without seeing the original request. So why doesn’t tcpdump see SYN packet that came before it?

This is a good question…
I juste have requests [R.] on firewall Vyos from host client.

I have seen situations where WISP CPE equipment randomly(?) injected RST packets on TCP sessions for no apparent reason. Try performing a packet capture writing to a file (eg. tcpdump -i eth0 -w /tmp/eth0-data-capture.cap) and then copy it to your computer and examine it in Wireshark. (Personally, I find data readability much easier in Wireshark than just raw TCPdump output. :grin: )

In the case I was troubleshooting, we were able to determine the responsible equipment thanks to the TTL of the injected RST packet (with a spoofed source IP) not matching the TTL of the other packets in the TCP session. For more information on such troubleshooting, see this Medium post by Shriram Sharma.

Hello and thanks Marvin,

I followed that you told me but found nothing. I have a doubt about the TTL. Here is a screenshot on the client side.
SYN → 128 TTL

What do you think ?

Request KO :

Request OK :

Here’s some assumptions:

  1. OK session: the SYN/ACK is likely starting out with an origin TTL of 128, so it’s making 15 hops to get to the client.
  2. Reset session: the RST packet is likely starting out with an origin TTL of 64, so it’s making 12 hops to get to the client.

Also, since the OK traffic seems to be using a TTL of 128 at each end, that would seem to be a dead giveaway of injection, though pinpointing the exact origin beyond this could be difficult, unless you have access to the routers the sessions are traversing. If you do have access, you can do more captures along the path and look at Source MAC addresses to 100% confirm the source of the unwanted RST packets.

I’m curious: if you run a packet capture on your VyOS WAN interface while replicating the problem, are you getting a RST packet too? And if so, what does TTL does that have in comparison to the OK traffic?

FWIW, you can right-click about any packet parameter in the Wireshark details pane and select Apply as a Column for easier visibility and comparison.

So thanks,

This is screen on the server vyos side.

Request KO (TTL 64) :

Request OK (TTL 115 / 127) :

Does that mean “interface on VyOS that connects to server”, or “interface on server that connects to VyOS”?

In any case, you’ve got a packet capture with a TTL of 64 on it: so if that’s from your webserver, it’s the one resetting the connection; I have no idea why.

Client try to connect on the server web through way vyos.

Hello guys,

I have a new information to trying help me :slight_smile:
I added a new public ip on the eth0 interface. Changed the destination ip on my NAT rule. And I do not encounter any worries.
IP requests KO: X.X.124.197 / 27
IP requests OK: X.X.124.205 / 27

I also tried the problematic IP to a new web server. And the problem is present.