NAT issues? Can ping from other hosts, but TCP connections seem to only work from certain hosts

Definitely bizarre! One last thing, can you provide the full output of:

sudo nft list ruleset

Yep, here you go! I’m hoping someone can figure out what’s happening here haha, I haven’t been able to in the last few weeks of poking at this (since I started poking at the issue about a week before the first post)

table ip nat {
	chain VYOS_PRE_SNAT_HOOK {
		type nat hook postrouting priority srcnat - 1; policy accept;
		return
	}
}
table inet mangle {
	chain FORWARD {
		type filter hook forward priority mangle; policy accept;
	}
}
table ip raw {
	chain VYOS_TCP_MSS {
		type filter hook forward priority raw; policy accept;
	}

	chain vyos_global_rpfilter {
		return
	}

	chain vyos_rpfilter {
		type filter hook prerouting priority raw; policy accept;
		counter packets 629 bytes 88904 jump vyos_global_rpfilter
	}

	chain VYOS_PREROUTING_HOOK {
		type filter hook prerouting priority raw; policy accept;
	}
}
table ip6 raw {
	chain VYOS_TCP_MSS {
		type filter hook forward priority raw; policy accept;
	}

	chain vyos_global_rpfilter {
		return
	}

	chain vyos_rpfilter {
		type filter hook prerouting priority raw; policy accept;
		counter packets 381 bytes 72674 jump vyos_global_rpfilter
	}

	chain VYOS_PREROUTING_HOOK {
		type filter hook prerouting priority raw; policy accept;
	}
}
table inet vrf_zones {
	map ct_iface_map {
		typeof iifname : ct zone
	}

	chain vrf_zones_ct_in {
		type filter hook prerouting priority raw; policy accept;
	}

	chain vrf_zones_ct_out {
		type filter hook output priority raw; policy accept;
	}
}
table ip vyos_nat {
	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
		counter packets 17 bytes 2900 jump VYOS_PRE_DNAT_HOOK
		iifname "eth0" ip saddr 172.26.0.230 ip daddr 172.26.0.57 counter packets 1 bytes 84 log prefix "[DST-NAT-5]" dnat to 10.255.255.2 comment "DST-NAT-5"
		iifname "eth0" ip daddr 172.26.0.57 counter packets 0 bytes 0 log prefix "[DST-NAT-10]" dnat to 10.255.255.2 comment "DST-NAT-10"
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		counter packets 1 bytes 84 jump VYOS_PRE_SNAT_HOOK
		oifname "eth1" ip saddr 0.0.0.0/0 counter packets 1 bytes 84 masquerade comment "SRC-NAT-1"
	}

	chain VYOS_PRE_DNAT_HOOK {
		return
	}

	chain VYOS_PRE_SNAT_HOOK {
		return
	}
}
table ip vyos_static_nat {
	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
	}
}
table ip vyos_conntrack {
	chain VYOS_CT_IGNORE {
		return
	}

	chain VYOS_CT_TIMEOUT {
		return
	}

	chain PREROUTING {
		type filter hook prerouting priority raw; policy accept;
		counter packets 38 bytes 5692 jump VYOS_CT_IGNORE
		counter packets 38 bytes 5692 jump VYOS_CT_TIMEOUT
		counter packets 38 bytes 5692 jump FW_CONNTRACK
		counter packets 38 bytes 5692 jump NAT_CONNTRACK
		counter packets 0 bytes 0 jump WLB_CONNTRACK
		notrack
	}

	chain PREROUTING_HELPER {
		type filter hook prerouting priority filter - 5; policy accept;
		counter packets 38 bytes 5692 jump VYOS_CT_HELPER
	}

	chain OUTPUT {
		type filter hook output priority raw; policy accept;
		counter packets 0 bytes 0 jump VYOS_CT_IGNORE
		counter packets 0 bytes 0 jump VYOS_CT_TIMEOUT
		counter packets 0 bytes 0 jump FW_CONNTRACK
		counter packets 0 bytes 0 jump NAT_CONNTRACK
		notrack
	}

	chain OUTPUT_HELPER {
		type filter hook output priority filter - 5; policy accept;
		counter packets 0 bytes 0 jump VYOS_CT_HELPER
	}

	chain VYOS_CT_HELPER {
		return
	}

	chain FW_CONNTRACK {
		return
	}

	chain NAT_CONNTRACK {
		accept
	}

	chain WLB_CONNTRACK {
		return
	}
}
table ip6 vyos_conntrack {
	chain VYOS_CT_IGNORE {
		return
	}

	chain VYOS_CT_TIMEOUT {
		return
	}

	chain PREROUTING {
		type filter hook prerouting priority raw; policy accept;
		counter packets 29 bytes 5684 jump VYOS_CT_IGNORE
		counter packets 29 bytes 5684 jump VYOS_CT_TIMEOUT
		counter packets 29 bytes 5684 jump FW_CONNTRACK
		counter packets 29 bytes 5684 jump NAT_CONNTRACK
		notrack
	}

	chain OUTPUT {
		type filter hook output priority raw; policy accept;
		counter packets 0 bytes 0 jump VYOS_CT_IGNORE
		counter packets 0 bytes 0 jump VYOS_CT_TIMEOUT
		counter packets 0 bytes 0 jump FW_CONNTRACK
		counter packets 0 bytes 0 jump NAT_CONNTRACK
		notrack
	}

	chain VYOS_CT_HELPER {
		return
	}

	chain FW_CONNTRACK {
		return
	}

	chain NAT_CONNTRACK {
		return
	}
}

That looks fine too.

A couple more things to try:

  1. Clear the conntrack table on the off chance there’s a stale entry and test again. reset conntrack
  2. Modify DNAT rule 5 with these to see if it hits when it’s very explicit. Run show nat destination statistics afterwards for hitcounts
set nat destination rule 5 destination port '80'
set nat destination rule 5 protocol 'tcp'
  1. Lastly, since you’re on an older rolling release, try to update to a newer rolling. Don’t go to the newest ones, since there’s a bug that breaks a lot of op commands in it. Go to maybe 2025.06.03-1541-rolling

One more thing I should have mentioned, make sure you delete the offloads off of eth0 and eth1 if they’re configured.

Tried the first two things you mentioned without any luck, waiting on add system image to finish downloading the release you mentioned. I’ll check that before I reboot though!

Editing to add this, since I’ve apparently hit the number of replies I can make for the day? >.>

So… not sure why, but even trying to scp the image over to the router is having the transfers stall, but I can SSH in fine, with my SSH connection not having issues while the transfer is going.

I did double check though and I don’t have any offloads configured on eth0/1 (wanted to check as I didn’t know if any were enabled by default).

… One thing I noticed after I started typing this, while checking to see if I can just scp the iso from 172.26.0.230, is that… even with NAT rules all disabled, I can’t SSH into the router itself while I can from 10.255.0.2, nor can I SSH from the router to 172.26.0.230 (or even into a brand new virtual machine, with nothing done on it except installing an SSH server).

I’m going to poke at this more tomorrow (or maybe later tonight, with my fucked sleep schedule), I’m getting exhausted from this for now haha. If you have any ideas I’ll try them next time I’m on my computer, though.

This tracks with the behavior you’re seeing. The issue really isn’t NAT, it’s that it seems VyOS is doing early dropping of the SYN before it has the chance to be NATted. My suspicion is there may be something in the TCP header that VyOS is viewing as malformed and drops the packet early. It could be useful to run this and look for errors:

sudo cat /proc/net/dev

You can also run these for more granular errors if there are any:

sudo ethtool -S eth0
sudo ethtool -S eth1

So, I ran sudo cat /proc/net/dev across the VyOS system, the GNS3 host, and the Proxmox host, and on all three they reported zero errors. Took a while to reply because of the posting limit, but… yeah, nothing there.

But, I think I just found the issue. It might be entirely within GNS3, and I’m going to try spinning up a test network on the proxmox host machine instead of within GNS3 to test. I just found out that GNS3… doesn’t do anything sane like using bridges to connect networks between nodes together - it uses a userspace program called ubridge, which stuff like qemu connects to over UDP for networking, which I’m thinking might be where my issues are coming from.

… Okay, editing again!


… Well, damn. It looks like GNS3 was the issue. I spun up a VyOS VM outside GNS3 and got it in my existing prod network, and I’m able to SSH into that from 172.26.0.230.
I guess my next step would be to report a bug to ubridge? But I’m unsure what information to even include or how to describe the issue, especially as I only a few hours ago found out ubridge was even a component of GNS3!

1 Like

(you should be free to post now)

1 Like

I’ve built probably thousands of labs in GNS3 at this point, and it doesn’t appear to be a ubridge issue. You can see the packet is arriving on VyOS, it isn’t getting dropped by the GNS3 backplane. That’s not to say it isn’t an issue from within GNS3, though it could be as simple as the driver you’re using for the NICs in there. I use e1000 for VyOS images in GNS3 and that seems to be the most consistent.

I use virtio
Zero issues :wink:

I was previously using virtio-net-pci. I just switched to e1000, but still having the same issue - in this case I’m testing by trying to SSH out from 172.26.0.57 to another machine (172.26.0.230) in the network, as it seems to have the same issue as trying to connect in to 172.26.0.57

I ended up making a post over on the GNS3 forums about it here, if anyone wants to see where it ends up going there.