Strange SNAT behaviour on TLS handshakes for some endpoints with a basic configuration

1

Hey folks!

I’m using VyOS since a while now and started using Github Copilot, when I noticed that the SSL handshake gets stuck from any clients behind SNAT.
When requesting the endpoint directly from the router, it just works; same as with an OpenWRT based router.

I’m wondering if this is a VyOS bug or I have something strange in my configuration.
It’s fairy basic, a PPPoE for my local ISP upstream, SNAT for IPv4 and a prefix delegation for IPv6 (attached at the bottom).

Here’s a traffic capture I have from running curl -v -XPOST https://copilot-proxy.githubusercontent.com/v1/engines/copilot-code on one of the client hosts:

xvzf@apu6b4:~$ tcpdump -i any -vv host 20.250.85.194 | grep -v eth3
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
    192.168.0.104.50369 > 20.250.85.194.https: Flags [S], cksum 0xfd03 (correct), seq 1561756951, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2652161900 ecr 0,sackOK,eol], length 0
15:51:05.153387 br0   In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.0.104.50369 > 20.250.85.194.https: Flags [S], cksum 0xfd03 (correct), seq 1561756951, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2652161900 ecr 0,sackOK,eol], length 0
15:51:05.153636 pppoe0 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    p579dc6b4.dip0.t-ipconnect.de.50369 > 20.250.85.194.https: Flags [S], cksum 0x9fc2 (correct), seq 1561756951, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2652161900 ecr 0,sackOK,eol], length 0
15:51:05.169419 pppoe0 In  IP (tos 0x0, ttl 49, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    20.250.85.194.https > p579dc6b4.dip0.t-ipconnect.de.50369: Flags [S.], cksum 0x45ba (correct), seq 3797559195, ack 1561756952, win 65160, options [mss 1440,sackOK,TS val 2181095567 ecr 2652161900,nop,wscale 7], length 0
15:51:05.169616 br0   Out IP (tos 0x0, ttl 48, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    20.250.85.194.https > 192.168.0.104.50369: Flags [S.], cksum 0xa2fb (correct), seq 3797559195, ack 1561756952, win 65160, options [mss 1440,sackOK,TS val 2181095567 ecr 2652161900,nop,wscale 7], length 0
    20.250.85.194.https > 192.168.0.104.50369: Flags [S.], cksum 0xa2fb (correct), seq 3797559195, ack 1561756952, win 65160, options [mss 1440,sackOK,TS val 2181095567 ecr 2652161900,nop,wscale 7], length 0
    192.168.0.104.50369 > 20.250.85.194.https: Flags [.], cksum 0xc820 (correct), seq 1, ack 1, win 2052, options [nop,nop,TS val 2652161924 ecr 2181095567], length 0
15:51:05.174529 br0   In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.104.50369 > 20.250.85.194.https: Flags [.], cksum 0xc820 (correct), seq 1, ack 1, win 2052, options [nop,nop,TS val 2652161924 ecr 2181095567], length 0
15:51:05.174738 pppoe0 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    p579dc6b4.dip0.t-ipconnect.de.50369 > 20.250.85.194.https: Flags [.], cksum 0x6adf (correct), seq 1, ack 1, win 2052, options [nop,nop,TS val 2652161924 ecr 2181095567], length 0
    192.168.0.104.50369 > 20.250.85.194.https: Flags [P.], cksum 0xc18a (correct), seq 1:346, ack 1, win 2052, options [nop,nop,TS val 2652161929 ecr 2181095567], length 345
15:51:05.180398 br0   In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 397)
    192.168.0.104.50369 > 20.250.85.194.https: Flags [P.], cksum 0xc18a (correct), seq 1:346, ack 1, win 2052, options [nop,nop,TS val 2652161929 ecr 2181095567], length 345
15:51:05.180604 pppoe0 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 397)
    p579dc6b4.dip0.t-ipconnect.de.50369 > 20.250.85.194.https: Flags [P.], cksum 0x6449 (correct), seq 1:346, ack 1, win 2052, options [nop,nop,TS val 2652161929 ecr 2181095567], length 345
15:51:05.195859 pppoe0 In  IP (tos 0x0, ttl 49, id 39376, offset 0, flags [DF], proto TCP (6), length 52)
    20.250.85.194.https > p579dc6b4.dip0.t-ipconnect.de.50369: Flags [.], cksum 0x6f70 (correct), seq 1, ack 346, win 507, options [nop,nop,TS val 2181095593 ecr 2652161929], length 0
15:51:05.196052 br0   Out IP (tos 0x0, ttl 48, id 39376, offset 0, flags [DF], proto TCP (6), length 52)
    20.250.85.194.https > 192.168.0.104.50369: Flags [.], cksum 0xccb1 (correct), seq 1, ack 346, win 507, options [nop,nop,TS val 2181095593 ecr 2652161929], length 0
    20.250.85.194.https > 192.168.0.104.50369: Flags [.], cksum 0xccb1 (correct), seq 1, ack 346, win 507, options [nop,nop,TS val 2181095593 ecr 2652161929], length 0
15:51:05.198145 pppoe0 In  IP (tos 0x0, ttl 49, id 39379, offset 0, flags [DF], proto TCP (6), length 729)
    20.250.85.194.https > p579dc6b4.dip0.t-ipconnect.de.50369: Flags [P.], cksum 0x058b (correct), seq 2897:3574, ack 346, win 507, options [nop,nop,TS val 2181095596 ecr 2652161929], length 677
15:51:05.198349 br0   Out IP (tos 0x0, ttl 48, id 39379, offset 0, flags [DF], proto TCP (6), length 729)
    20.250.85.194.https > 192.168.0.104.50369: Flags [P.], cksum 0x62cc (correct), seq 2897:3574, ack 346, win 507, options [nop,nop,TS val 2181095596 ecr 2652161929], length 677
    20.250.85.194.https > 192.168.0.104.50369: Flags [P.], cksum 0x62cc (correct), seq 2897:3574, ack 346, win 507, options [nop,nop,TS val 2181095596 ecr 2652161929], length 677
    192.168.0.104.50369 > 20.250.85.194.https: Flags [.], cksum 0x6349 (correct), seq 346, ack 1, win 2052, options [nop,nop,TS val 2652161950 ecr 2181095593,nop,nop,sack 1 {2897:3574}], length 0
15:51:05.200615 br0   In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.0.104.50369 > 20.250.85.194.https: Flags [.], cksum 0x6349 (correct), seq 346, ack 1, win 2052, options [nop,nop,TS val 2652161950 ecr 2181095593,nop,nop,sack 1 {2897:3574}], length 0
15:51:05.200818 pppoe0 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    p579dc6b4.dip0.t-ipconnect.de.50369 > 20.250.85.194.https: Flags [.], cksum 0x0608 (correct), seq 346, ack 1, win 2052, options [nop,nop,TS val 2652161950 ecr 2181095593,nop,nop,sack 1 {2897:3574}], length 0
15:51:06.190967 pppoe0 In  IP (tos 0x0, ttl 49, id 39383, offset 0, flags [DF], proto TCP (6), length 52)
    20.250.85.194.https > p579dc6b4.dip0.t-ipconnect.de.50369: Flags [F.], cksum 0x5d82 (correct), seq 3574, ack 346, win 507, options [nop,nop,TS val 2181096588 ecr 2652161950], length 0
15:51:06.191162 br0   Out IP (tos 0x0, ttl 48, id 39383, offset 0, flags [DF], proto TCP (6), length 52)
    20.250.85.194.https > 192.168.0.104.50369: Flags [F.], cksum 0xbac3 (correct), seq 3574, ack 346, win 507, options [nop,nop,TS val 2181096588 ecr 2652161950], length 0
    20.250.85.194.https > 192.168.0.104.50369: Flags [F.], cksum 0xbac3 (correct), seq 3574, ack 346, win 507, options [nop,nop,TS val 2181096588 ecr 2652161950], length 0
    192.168.0.104.50369 > 20.250.85.194.https: Flags [.], cksum 0x5efe (correct), seq 346, ack 1, win 2052, options [nop,nop,TS val 2652163048 ecr 2181095593,nop,nop,sack 1 {2897:3575}], length 0
15:51:06.299722 br0   In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    192.168.0.104.50369 > 20.250.85.194.https: Flags [.], cksum 0x5efe (correct), seq 346, ack 1, win 2052, options [nop,nop,TS val 2652163048 ecr 2181095593,nop,nop,sack 1 {2897:3575}], length 0
15:51:06.299927 pppoe0 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 64)

     interface pppoe0 {
     }
 }
 interfaces {
     bridge br0 {
         address 192.168.0.1/24
         address fd00:b4d:c0fe::1/64
         description lan
         disable-link-detect
         member {
             interface eth2 {
             }
             interface eth3.100 {
             }
         }
     }
     ethernet eth0 {
         description WAN
         hw-id 00:0d:b9:60:37:20
         vif 7 {
             description WAN_dtag.vlan7
         }
     }
     ethernet eth1 {
         hw-id 00:0d:b9:60:37:21
     }
     ethernet eth2 {
         hw-id 00:0d:b9:60:37:22
     }
     ethernet eth3 {
         hw-id 00:0d:b9:60:37:23
         vif 100 {
             description lan
         }
     }
     loopback lo {
     }
     pppoe pppoe0 {
         authentication {
             password <redacted>
             user <redacted>
         }
         description dtag.pppoe
         dhcpv6-options {
             pd 0 {
                 interface br0 {
                     address 1
                     sla-id 0
                 }
                 length 56
             }
         }
         ipv6 {
             address {
                 autoconf
             }
         }
         mtu 1492
         no-peer-dns
         source-interface eth0.7
     }
 }
 nat {
     source {
         rule 100 {
             outbound-interface pppoe0
             source {
                 address 192.168.0.0/16
             }
             translation {
                 address masquerade
             }
         }
     }
 }
 service {
     dhcp-server {
         listen-address 192.168.0.1
         shared-network-name LAN {
             authoritative
             subnet 192.168.0.0/24 {
                 default-router 192.168.0.1
                 domain-name oasis.local
                 lease 86400
                 name-server 192.168.0.1
                 range 0 {
                     start 192.168.0.100
                     stop 192.168.0.254
                 }
             }
         }
     }
     dns {
         forwarding {
             allow-from 0.0.0.0/0
             allow-from ::/0
             cache-size 0
             listen-address 192.168.0.1
             listen-address fd00:b4d:c0fe::1
             name-server 2606:4700:4700::1111
             name-server 2606:4700:4700::1001
             name-server 1.0.0.1
         }
     }
     lldp {
         interface all {
         }
     }
     ntp {
         allow-client {
             address 0.0.0.0/0
             address ::/0
         }
         server time1.vyos.net {
         }
         server time2.vyos.net {
         }
         server time3.vyos.net {
         }
     }
     router-advert {
         interface br0 {
             link-mtu 1492
             name-server fd00:b4d:c0fe::1
             prefix ::/64 {
                 preferred-lifetime 3600
                 valid-lifetime 172800
             }
         }
     }
     ssh {
         disable-password-authentication
         port 22
     }
 }```
2

(I know I’m leaking my IP here, it’s not an issue :wink: )

3

Okay, there was an issue on the MTU configuration.

set interfaces pppoe pppoe0 ip adjust-mss 1452 fixes it :slight_smile:

1 Like
4

Another strange issue it is duplicated packets
Check 1 and 2 packets ( SYN sends 2 times) and so on

5

I think the “duplicate” is because tcpdump is showing packets traversing both the bridge and pppoe interfaces, so that both in/out sides are seen from router’s point of view. Notice how the ttl value decreases.

I could be mistaken, though :slight_smile:

1 Like