Pointers diagnosing dropped NAT traffic

General questions
,
1

I found my VyOS firewall buried in my VM library yesterday and gave it another shot. The one last thing — two with policy routing — I had to figure out was how to NAT.

Before, I had discovered that whenever it was natting, it started having traffic drops, but the oddest thing is that it is to specific domains only. I’ve ask about it various times but other issues were more important at the time and I was able to work around it anyway.

My infrastructure is largely the same as before, it kind of has to; I’m on an Active Directory domain with zero Windows clients. It’s very mixed, and it must be stable. Infrastructure services are each their own thing or a tiny cluster. DNS, DHCP, RADIUS, NAC, NTP all that. This allows me to quickly eliminate them as causes. I also tried changing NICs: VMXNET3 backed by Intel, Mellanox, Broadcom. Then tried them all out plus some from TP-Link as PCI pass through but it still be droppin’ (I may have just regretted that). Then tried again as SRIOV virtual functions…no improvement.

I discovered, which I mentioned as well the last time I was around here, that using another firewall to handle the PPPoE session and NAT, leaving VyOS as a firewall only — no NAT — eliminates the issue.

So that leaves either NAT of the PPPoE session. I don’t know how to separate the two since I had to create some sort of transparent firewall/bridge and I have no idea how would I get the dynamically assigned address onto VyOS if PPPoE is what gets me to layer 3. So instead, I gathered all the errors I could find to come ask for help.
There weren’t a lot of them, first is a packet capture. I had already had one of these but I didn’t [and still don’t] know what to make of it.

dropped-vyos-traffic-compressed
dropped-vyos-traffic-compressed1918×1385 166 KB

I took a fresh capture, straight from VyOS using tcpdump without selecting any interface, I just set a capture filter from the machine I was on, towards TCP ports 80 and 443 i.e. ‘src host 10.9.0.32 and (dst port 80 or dst port 443)’, then cleaned up the normal traffic.

Again, I’m not sure what to look for here, but the only thing that got my attention (besides the timeouts and resets, of course) was the outdated protocol in Client Hello packets.

Attached is the cleaned up capture; it is the .*\\.pcapng file ending with a txt extension stacked on it. It wouldn’t let me attach it otherwise. The other file is VyOS’ CLI output of show conntrack statistics which was the only other place where it showed errors. show interfaces detail shows no errors.

Though it just shows some column with unlabeled rows. Hopefully somebody can tell me what do they mean. Neither the PPPoE interface nor the VLAN interface it is on have errors or drops.

droppedtraffic.pcapng.txt (8.6 KB)
run-show-conntrack-statistics.txt (13.4 KB)

Thanks for your help.

— Almost forgot —
It’s version:

Version:          VyOS 1.5-rolling-202407100021
Release train:    current
Release flavor:   generic

Built by:         [email protected]
Built on:         Wed 10 Jul 2024 02:43 UTC
Build UUID:       931ee68f-2f60-439e-97ab-ae9d2e91b433
Build commit ID:  16753c9d3a6138

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware7,1
Hardware S/N:     VMware-56 4d 8a 02 d4 2f 34 d9-b4 73 c2 78 e5 58 03 da
Hardware UUID:    028a4d56-2fd4-d934-b473-c278e55803da

I know it’s kind of outdated, but it’s been upgraded and rebuilt several times, I don’t think the version at this point will be too much of an issue. I’ll upgrade it if I’m told to, though. Thanks again.

2

Should be the first thing to try - grab the latest version and try and see if the problem remains.

1 Like
3

Sir, yes sir! :laughing:

...

Seriously, thanks for the link though. Should be back in a few, I’ll take a snapshot just in case.

Never mind. It was so fast that the buffer from the radio station playing kept it going. It didn’t play out the same way for natting though.

Unfortunately, it didn’t work.

...

I tried again the sites with the same results. I tried also another site I discovered just now, WordReference.com, that also fails.

Could it be server-side? Do these sites have something in common?

DuckDuckGo has some relationship with Bing in the backend, which being Microsoft’s must be a big back end. Yahoo! is big too. Together they have over 100ASNs, but that’s like saying my little Schnauzer bark a to me today. She did, but it’s hardly conclusive of anything. And it doesn’t explain WordReference.com.

Sorry, I meant to reply hours ago but I forgot to click on the big orange button and left it sitting there (now hidden).

I found a few more websites that don’t respond though. It’s probably not useful other than to test. Any ideas what could be causing this?

Or could you explain what are the rows in show conntrack statistics in ops mode? or alternatively recommend other places to look into?

If it’s of any use, I’m attaching my whole config this time, there’s some lingering config from previous attempts but it’s inactive. I’m using a bridge on the intranet, a VLAN on the Internet, but for what its worth, I have an old Ubiquiti UniFi router that’s configured similarly which can cope with the same configuration just fine. UniFi is like caveperson technology compared to VyOS, I don’t get it.

vyos9-config-json.txt (40.0 KB)

4

It’s VyOS 1.5-rolling-202410081209 now, BTW—I knew I was forgetting something.

5

A common thing when a few services works while other dont is to check MTU’s.

Or rather make sure that you allow the needed ICMP type/code to make Path MTU Discovery work for the client all the way to the server.

Sometimes the server end lacks this so they will try to send up to 1500 bytes packets (due to MTU is set by default to 1500) but your clients behind PPPoE can only received 1472 bytes (or whatever it is).

Other things that might be helpful is to enable “clamp-mss” or as its called in VyOS “adjust-mss” (Ethernet — VyOS 1.5.x (circinus) documentation) so TCP-traffic will have its MSS altered to the lower accetable value (doesnt work on UDP).

https://www.speedguide.net/analyzer.php is a great site to test and verify MTU (MSS) and PMTU.

6

I have another update, this on looks promising. :smiley:

I mentioned earlier I don’t know how to separate the PPPoE session from the NAT, but I was thinking just a few minutes ago, I don’t have to. Not to test at least.

So I grabbed that old USG, made it establish the PPPoE session, and set up a masquerade NAT rule targeting just the subnet VyOS is on (everything’s on trunk ports, so to avoid false positives from routes leaking or whatever.)

A added a new VLAN interface on VyOS and added to it another masquerade rule, in other words, double NAT but this is only testing so not a big deal I reasoned.

I went to those sites not loading before and you should know by now where I’m going with this… yes.

The issue isn’t NAT at all, it’s something related to PPPoE.

I took some notes, disconnected the UniFi device and reconnected the VyOS “device”, this is my first draft of observations:

# UniFi
pppoe1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN qlen 100
    link/ppp
    inet xxx.xxx.19.176 peer xxx.xxx.193.226/32 scope global pppoe1
       valid_lft forever preferred_lft forever

    RX:  bytes    packets     errors    dropped    overrun      mcast
       2605839      20666          0          0          0          0
    TX:  bytes    packets     errors    dropped    carrier collisions
       2220165      26236          0          0          0          0

# VyOS
pppoe1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    link/ppp
    inet xxx.xxx.103.210 peer xxx.xxx.193.226/32 scope global pppoe1
       valid_lft forever preferred_lft forever
    inet6 fe80::f069:c1ff:fe6b:7444/64 scope link
       valid_lft forever preferred_lft forever

    RX:  bytes  packets  errors  dropped  overrun       mcast
            90        4       0        0        0           0
    TX:  bytes  packets  errors  dropped  carrier  collisions
           486        9       0        0        0           0

I noticed different, pfifo_fast vs fq_codel. Since tuned pfSense and OPNsense before for bufferbloat, I remember about these, they’re schechulers, right? som’n like that. Notably, VyOS guessed(??) the right queue length for the interface, I had measure myself some time ago and took notes. According to my notes (on paper for some reason) the queue length target is the rounded up worst RTT at the interface’s MTU, ie. ping at 1500 (-8, -8, -20 for PPPoE, ICMP and IPv4 respectively) the immediate gateway, my ISP’s. The worst average is 2.3ms, so qlen 3 is perfect. And yet it’s breaking traffic.

The UniFi device — not unlike VyOS BTW — routes at line speed, except it doesn’t drop pseudo random traffic, “randomly specific”?? Doesn’t matter, I think I might be because of that fairy-tale-sounding pfifo thing.

This is only the first assessment, as I mentioned, but I don’t think there’s much diffent otherwise. If anything, while VyOS’ PPPoE config is a child directly on the interfaces config tree, on UniFi it is buried deep on an ethernet-type interface or subinterface, but then appear from nowhere at the root level as pppoe# once it makes the first connection, same as in VyOS.

The good thing is that I know enough about routing not to have double NAT, but delegating PPPoE to another device kinda goes against the spirit of this weird experiment I’m doing. I think. To be honest, I sort of forgot what was the goal of it. :laughing: It’ll come back.

Any comment, suggestion is welcome. If not, I hope at least some dev catches this.

7

Thanks, that’s exactly1 what I was just sending an update about, but I was offline while I switched back the routing around so I’m only now seeing your answer.

I’ll check it out in after I’m done with something else that came up. I have noticed more MTU-related things lately though I’m also tighter within the margins when setting tunnels and the like. In the PPPoE side though, it has always worked itself out, and usually it would be best not to mess with it. MSS clamping I’ve set though, I don’t believe I have in VyOS though. I’ll keep updating.

Thanks !

1

If by “exactly” we mean “more like ballpark”

8

I got it fixed. It was the MTU.

TLDR: find and enable clamp-mss-to-pmtu at on the interface.

I first found it on system though, where I was also surprised by option for multipath TCP. If I wasn’t motivated to fix this that, did it. I remember the goal I had too: fixing it. Just because. No special reason.

However, it was in the interface config where there were more MTU-related options, among those MSS clamping. But before using it [and without reading any docu] I explored around on my own: I set the correct MTU and enabled clamp-mss-to-pmtu. The search engine with the aggressive bird finally loaded, and so Yahoo! and all those other test sites I normally would never visit (hence they’re not cached making them great for testing).

I didn’t set the specific value MSS clamping option, or read the documentation yet. I’m still stuck in another thing but took a break to put it in writing because it took me more than two years getting here. I don’t wish that on anybody.

Here’s the modified parts of the config:

 dummy dum1 {
     address xxx.xxx.200.0/32
 }
 ethernet eth1 {
     hw-id xx:xx:xx:xx:xx:01
     offload {
         gro
         gso
         sg
         tso
     }
     vif 191 {
         address xxx.xxx.0.2/24
     }
     vif 881 {
     }
 }
 loopback lo {
 }
 pppoe pppoe1 {
     ip {
         adjust-mss clamp-mss-to-pmtu
     }
     no-peer-dns
     source-interface eth1.881
 }
 wireguard wg1 {
     ip {
         adjust-mss clamp-mss-to-pmtu
     }
     mtu 1432
 }
 wireguard wg2 {
     ip {
         adjust-mss clamp-mss-to-pmtu
     }
     mtu 1432
 }

In the case so PPPoE, since it already seemed to know the MTU, I took a chance and left it undeclared and just enabled clamp-mss-to-pmtu and it worked. For the Wireguard tunnels I took measurements, though…and math. I also don’t wish that on anybody. :upside_down_face: How is there not a drooling imbecile emoji, it would fit me perfectly.

Anyway, there’s a probing feature that’s supposed to detect oversized MTU and probe to adjust accordingly, but I couldn’t make it work nor tried too hard either but it seems like a really cool feature if it indeed works.

Thanks again!

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.