MSS Clamping policy route not working as expected

General questions
#1

I’m setting up a new VyOS install and am running into a bit of an issue that has me stumped.

I connect to my ISP (DSL) using PPPoE, a niche aspect of this config is that I have to set MSS clamping or I get some really odd behavior. The following 3 links seem to be just about all there is to say about this with VyOS:

https://docs.vyos.io/en/latest/configuration/policy/examples.html#clamp-mss-for-a-specific-ip
https://support.vyos.io/en/kb/articles/pppoe-sub-interfaces-2#comments

I’ve ended up with the following configuration:

policy {
    route pppoe-mss-clamp {
        description "MSS clamp for PPPoE"
        interface eth1.1020
        interface eth1.1200
        interface eth1
        interface pppoe0
        rule 5 {
            protocol tcp
            set {
                tcp-mss 1412
            }
            tcp {
                flags {
                    syn
                }
            }
        }
    }
}

(for reference the eth1* interfaces are my LAN side interfaces. I know attaching it to the PPPoE interface shouldn’t do anything, mostly added it out of desperation just to see if anything happened.)

As far as I can tell this should work but it isn’t. I ran a PCAP on the PPPoE interface while attempting some problematic traffic from an internal server and the capture shows the MSS value of TCP SYN packets leaving my network as the default 1460. I can’t seem to figure out a command in VyOS to show me if/where the policy route is being applied to traffic, the ‘show policy route statistics’ command in the manual doesn’t seem to be valid. For reference I’m running 1.4-rolling-202302270317 but also observed this behavior on 1.4-rolling-202302150317.

Any advice on how to further troubleshoot the policy route (or another way to implement this?)

#2

https://docs.vyos.io/en/latest/configuration/interfaces/pppoe.html?highlight=tcp-mss#cfgcmd-set-interfaces-pppoe-interface-ip-adjust-mss-mss-clamp-mss-to-pmtu

#3

Thanks, I must’ve not re-checked that after I realized I had been looking at the 1.3 documentation. Unfortunately after setting that I’m still seeing the same behavior, running a capture on my PPPoE interface shows TCP SYN packets with default MSS 1460

#4

Is it incoming our Outcoming SYN?
From which device do you send SYN from the router itself or from the LAN network device behind VyOS?

#5

Outgoing SYN, from devices on the LAN side of the router

#6

I’ve tried both commands in my config:

set interfaces pppoe pppoe0 ip adjust-mss 'clamp-mss-to-pmtu'

set interfaces pppoe pppoe0 ip adjust-mss '1412'

and running captures on my PPPoE interface is showing the MSS value not being modified at all with either command set.

As a sanity check I plugged my old EdgeRouter back in (I know it’s not exactly the same thing as VyOS but it has enough in common to be comparable) which has the following setting configured:

set firewall options mss-clamp mss 1412

Running a capture on the EdgeRouter’s PPPoE interface I can see the MSS value being set appropriately.

Any thoughts on if I’m somehow implementing that command wrong on VyOS, or is this a bug?

#7

How do you tested it ?

I use pppoe and mss clamping on 1.3 and it works.

admin@gw-multi# show firewall options 
 interface pppoe0 {
     adjust-mss 1452
 }

admin@gw-multi# sudo tcpdump -i pppoe0 -n -v "tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
07:22:36.850109 IP (tos 0x0, ttl 64, id 24320, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.56126 > 10.200.254.12.22: Flags [S], cksum 0x119d (correct), seq 1795564415, win 65340, options [mss 1452,sackOK,TS val 4144044395 ecr 0,nop,wscale 7], length 0

I tested it on 1.4-rolling-202302280651 and it works too.

admin@gw-multi# show interfaces pppoe 
 pppoe pppoe0 {
     authentication {
         password ********
         username ********
     }
     ip {
         adjust-mss 1452
     }
     mtu 1492
     source-interface eth0.9
 }

admin@gw-multi:~$ sudo tcpdump -i pppoe0 -n -v "tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
07:56:42.872190 IP (tos 0x10, ttl 64, id 11614, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.57798 > 10.200.254.12.22: Flags [S], cksum 0xe0de (correct), seq 1347098711, win 65340, options [mss 1452,sackOK,TS val 3045305090 ecr 0,nop,wscale 7], length 0
#8

I am using essentially the same TCPDump command, after your comment I tried 0228 and 0301 with no change in the result. I also tried deleting my PPPoE interface completely and setting it up from scratch, which didn’t change anything either. My interface config is:

 pppoe pppoe0 {
        authentication {
            password ****************
            username ****************
        }
        description CenturyLink
        ip {
            adjust-mss 1412
        }
        mtu 1492
        source-interface eth0
    }

and my TCPDump looks like this:

vyos@prairie-rtr-01:~$ tcpdump -i pppoe0 -n -v "src 63.153.8.186 and tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
13:53:03.455719 IP (tos 0x0, ttl 62, id 2229, offset 0, flags [DF], proto TCP (6), length 60)
    63.153.8.186.60002 > 164.90.255.4.443: Flags [S], cksum 0x6b5a (correct), seq 4171150216, win 64240, options [mss 1460,sackOK,TS val 2655108987 ecr 0,nop,wscale 7], length 0
13:53:06.352063 IP (tos 0x0, ttl 62, id 36914, offset 0, flags [DF], proto TCP (6), length 60)
    63.153.8.186.60008 > 164.90.255.4.443: Flags [S], cksum 0x8f7a (correct), seq 2032141201, win 64240, options [mss 1460,sackOK,TS val 2655111883 ecr 0,nop,wscale 7], length 0
13:53:08.385546 IP (tos 0x0, ttl 62, id 37122, offset 0, flags [DF], proto TCP (6), length 60)
    63.153.8.186.60022 > 164.90.255.4.443: Flags [S], cksum 0xd49c (correct), seq 2544801760, win 64240, options [mss 1460,sackOK,TS val 2655113917 ecr 0,nop,wscale 7], length 0
13:53:09.213372 IP (tos 0x0, ttl 63, id 44770, offset 0, flags [DF], proto TCP (6), length 60)
    63.153.8.186.51166 > 13.83.98.203.443: Flags [S], cksum 0xf053 (correct), seq 359858733, win 64240, options [mss 1460,sackOK,TS val 1345286881 ecr 0,nop,wscale 7], length 0
13:53:09.218717 IP (tos 0x0, ttl 63, id 51206, offset 0, flags [DF], proto TCP (6), length 60)
    63.153.8.186.45172 > 13.83.98.103.443: Flags [S], cksum 0x4c4e (correct), seq 2896041305, win 64240, options [mss 1460,sackOK,TS val 3471050152 ecr 0,nop,wscale 7], length 0

(again, essentially unchanged between 0227/0228/0301)

#9

Strange, I remove adjust-mss 1452 and tcpdump show the same result:

tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
14:38:27.328353 IP (tos 0x10, ttl 64, id 3646, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.49252 > 10.200.254.12.22: Flags [S], cksum 0x1d33 (correct), seq 2224239761, win 65340, options [mss 1452,sackOK,TS val 3672316462 ecr 0,nop,wscale 7], length 0

The same as I set any value:

admin@gw-multi# set interfaces pppoe pppoe0 ip adjust-mss 536

admin@gw-multi:~$ sudo tcpdump -i pppoe0 -n -v "tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
14:47:15.917304 IP (tos 0x10, ttl 64, id 41226, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.60826 > 10.200.254.12.22: Flags [S], cksum 0x117c (correct), seq 803821287, win 65340, options [mss 1452,sackOK,TS val 3672845051 ecr 0,nop,wscale 7], length 0

It’s look like some auto mss clamping based on negotiated MTU.

admin@gw-multi:~$ ip l sh dev pppoe0
7: pppoe0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 3
    link/ppp

In your case you have probably MTU 1500 on pppoe0 interface.

#10

As shown in my attached config snippet my PPPoE interface MTU is set to 1492 and I do see that reflected when I run show interface pppoe pppoe0.

Previously I did attempt setting a low MTU value on my PPPoE interface and it did not make any difference to the MSS value - are you able to replicate that behavior on your instance?

In your first reply to this thread, was your PPPoE interface already configured or had you set up a new interface to test this?

#11

I tested. With config:

admin@gw-multi# show interfaces pppoe pppoe0 
 authentication {
     password ********
     username ********
 }
 mtu 1400
 source-interface eth0.9

tcpdump show:

admin@gw-multi:~$ sudo tcpdump -i pppoe0 -n -v "tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
15:10:14.990715 IP (tos 0x10, ttl 64, id 22496, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.49450 > 10.200.254.12.22: Flags [S], cksum 0xe4da (correct), seq 1791126177, win 65280, options [mss 1360,sackOK,TS val 3674224124 ecr 0,nop,wscale 7], length 0

If I add:

admin@gw-multi# set interfaces pppoe pppoe0 ip adjust-mss 1300

nothing changed.

Already configured, but just migrated from 1.3.

#12

Perfect, thanks. I’ll do a little more testing tonight and see if I can replicate that somehow.

#13

The same results on 1.3. But all my tests was made from router itself.
So I made tests from system behind this router and all worked as expected.

Now I make tests on 1.4.

#14

On 1.4-rolling-202302280651, tests made from system behind router worked as expected.

admin@gw-multi# show interfaces pppoe pppoe0 
 authentication {
     password ********
     username ********
 }
 ip {
     adjust-mss 1400
 }
 mtu 1492
 source-interface eth0.9

admin@gw-multi:~$ sudo tcpdump -i pppoe0 -n -v "src 172.21.7.253 and tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
17:24:02.609353 IP (tos 0x0, ttl 63, id 8256, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.52604 > 217.74.71.145.443: Flags [S], cksum 0xf390 (correct), seq 1947883772, win 64240, options [mss 1400,sackOK,TS val 1868462367 ecr 0,nop,wscale 7], length 0
17:24:03.117555 IP (tos 0x0, ttl 63, id 57959, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.33626 > 217.74.71.140.443: Flags [S], cksum 0xa45f (correct), seq 3271302734, win 64240, options [mss 1400,sackOK,TS val 2632765621 ecr 0,nop,wscale 7], length 0
17:24:03.125430 IP (tos 0x0, ttl 63, id 57465, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.48610 > 217.74.71.132.443: Flags [S], cksum 0x21ab (correct), seq 90669396, win 64240, options [mss 1400,sackOK,TS val 3623237743 ecr 0,nop,wscale 7], length 0

admin@gw-multi# set interfaces pppoe pppoe0 ip adjust-mss clamp-mss-to-pmtu

admin@gw-multi:~$ sudo tcpdump -i pppoe0 -n -v "src 172.21.7.253 and tcp[tcpflags] & (tcp-syn) != 0"
tcpdump: listening on pppoe0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
17:25:42.169351 IP (tos 0x0, ttl 63, id 21087, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.59550 > 217.74.71.145.443: Flags [S], cksum 0x45b9 (correct), seq 2153994827, win 64240, options [mss 1452,sackOK,TS val 1868561927 ecr 0,nop,wscale 7], length 0
17:25:43.105541 IP (tos 0x0, ttl 63, id 37207, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.59560 > 217.74.71.145.443: Flags [S], cksum 0x2360 (correct), seq 319284814, win 64240, options [mss 1452,sackOK,TS val 1868562863 ecr 0,nop,wscale 7], length 0
17:25:44.479186 IP (tos 0x0, ttl 63, id 64197, offset 0, flags [DF], proto TCP (6), length 60)
    172.21.7.253.46806 > 212.77.99.29.443: Flags [S], cksum 0x87cd (correct), seq 997462536, win 64240, options [mss 1452,sackOK,TS val 1799907446 ecr 0,nop,wscale 7], length 0
#15

Ok, I think I’ve finally narrowed down the issue (@pepe thank you very much for your extensive testing, it was a helpful sanity check.)

I’m running VyOS on bare metal, with a standard 1GB NIC for my WAN/PPPoE connection and a Mellanox ConnectX-3 card at 10GB for my LAN side. Seemingly no matter what I do, as long as traffic comes into the router through the Mellanox card the MSS value is not set correctly. However, I’ve swapped out a couple different 1GB NICs for the LAN side and when using any of those the router adjusts the MSS value going out the PPPoE interface as expected (At no point has the 1GB card I’m using for my PPPoE connection been changed.)

I’m fairly new to 10GB networking, any idea if this is normal or is it an issue with my specific Mellanox card? Unfortunately I don’t have any other 10GB cards on hand to test with at the moment, open to any suggestions on models that are fairly compatible with VyOS.

#16

Try disabling all offload on Mellanox card.

delete interfaces ethernet ethX offload
#17

Sorry for the delay, had something come up and couldn’t work on this for a day or two. I tried running the command you suggested, however it failed as I didn’t have any offload options set on the interface.

Out of curiosity I tried enabling GSO & GRO offloading on the interface and now MSS is being adjusted properly per the config! I’m not entirely sure what combination of offload options I can/should be using, however considering this seems to have fixed my issue I’m inclined to leave it alone (unless you have other recommendations.)

Thanks for your help!