Bridge with SNAT: does not work

Bugs
1

I’m coming from EdgeRouter and exploring VyOS. In my ER-X SFP I have a switch0 which I use to spread vlans over multiple ports: eth0, eth1 and eth2 are in the switch and all have vlan 100.

Next I have port forwarding (destination nat) and hairpin nat enabled on the ER. This is also working fine.

Now I’m using VyOS to get the same working principle. My (Qotom) box doesn’t have a switch so I use a bridge to get the vlan 100 to eth0, 1 and 2: br100 has members eth0.100, eth1.100, eth2.100. This is also working fine.

Now I made destination NAT rules which work great from the outside to my WAN ip. Adding source NAT rules to make the hairpin nat available (in combination with the destination NAT) this does not work on the bridge.

If I use “tcpdump -ni br100” to see what kind of traffic is flowing through the bridge the SNAT immediately starts working - until I stop tcpdump again.

Using this same configuration not on a bridge but on a single port (eth0.100), by transferring the IP address from the bridge to the eth0.100 port the exact same NAT rules work as expected.

How can this be? Seems like a bug / unexpected behaviour with bridging to me.

2

Hey, a tcpdump set an interface in promiscuous mode.
please add a -e when you tcpdump the traffic and look if the MACs and IPs belongs together.

If this shows nothing weird can you post your NAT config?

3

Example here:
MacBook = 10.0.100.150 (8c:85:90:24:a0:4d)
Router = 10.0.100.1 (40:62:31:13:eb:8f)
Internal server = 10.0.100.6 (32:19:63:dd:f2:5f)

WAN is connected via pppoe0

Bridge config:

 address 10.0.100.1/24
 member {
     interface eth0.100 {
     }
     interface eth1.100 {
     }
     interface eth2.100 {
     }
 }

NAT config:

 destination {
     rule 1 {
         destination {
             port 8443
         }
         inbound-interface pppoe0
         protocol tcp
         translation {
             address 10.0.100.6
         }
     }
     rule 100 {
         destination {
             address <MY-WAN-IP-HERE>
             port 8443
         }
         inbound-interface br100
         protocol tcp
         translation {
             address 10.0.100.6
         }
     }
 }
 source {
     rule 100 {
         destination {
             address 10.0.100.0/24
         }
         outbound-interface br100
         protocol tcp
         source {
             address 10.0.100.0/24
         }
         translation {
             address masquerade
         }
     }
     rule 9000 {
         description "Internet out"
         outbound-interface pppoe0
         protocol all
         translation {
             address masquerade
         }
     }
 }

Without br100 in promiscuous mode

Connecting from MacBook (10.0.100.150) to server (10.0.100.6) port tcp/8443: succes.
Connecting from MacBook (10.0.100.150) to WAN IP port tcp/8443: nothing.

On my mac: tcpdump -nei en0

09:50:44.369899 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56225 > <MY-WAN-IP>.8443: Flags [S], seq 53325316, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210952979 ecr 0,sackOK,eol], length 0
09:50:45.373179 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56225 > <MY-WAN-IP>.8443: Flags [S], seq 53325316, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210953979 ecr 0,sackOK,eol], length 0
09:50:46.373287 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56225 > <MY-WAN-IP>.8443: Flags [S], seq 53325316, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210954979 ecr 0,sackOK,eol], length 0
09:50:47.374485 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56225 > <MY-WAN-IP>.8443: Flags [S], seq 53325316, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210955979 ecr 0,sackOK,eol], length 0
09:50:48.379480 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56225 > <MY-WAN-IP>.8443: Flags [S], seq 53325316, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210956979 ecr 0,sackOK,eol], length 0

With br100 in promiscuous mode (with tcpdump)

Starting up tcpdump -nei br100 on router.
Connecting from MacBook (10.0.100.150) to WAN IP port tcp/8443: succes.

On my mac: tcpdump -nei en0 port 8443

09:49:10.565962 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56216 > <MY-WAN-IP>.8443: Flags [S], seq 3025485842, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210859482 ecr 0,sackOK,eol], length 0
09:49:10.568875 40:62:31:13:eb:8f > 8c:85:90:24:a0:4d, ethertype IPv4 (0x0800), length 74: <MY-WAN-IP>.8443 > 10.0.100.150.56216: Flags [S.], seq 1432450948, ack 3025485843, win 65160, options [mss 1460,sackOK,TS val 4165071171 ecr 1210859482,nop,wscale 7], length 0
09:49:10.568980 8c:85:90:24:a0:4d > 40:62:31:13:eb:8f, ethertype IPv4 (0x0800), length 66: 10.0.100.150.56216 > <MY-WAN-IP>.8443: Flags [.], ack 1, win 2058, options [nop,nop,TS val 1210859485 ecr 4165071171], length 0

On router: tcpdump -nei br100 port 8443

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br100, link-type EN10MB (Ethernet), capture size 262144 bytes
09:39:23.361789 8c:85:90:24:a0:4d > 32:19:63:dd:f2:5f, ethertype IPv4 (0x0800), length 78: 10.0.100.150.56188 > 10.0.100.6.8443: Flags [S], seq 657712525, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210266758 ecr 0,sackOK,eol], length 0
09:39:23.361831 40:62:31:13:eb:8f > 32:19:63:dd:f2:5f, ethertype IPv4 (0x0800), length 78: 10.0.100.1.56188 > 10.0.100.6.8443: Flags [S], seq 657712525, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1210266758 ecr 0,sackOK,eol], length 0
09:39:23.362110 32:19:63:dd:f2:5f > 8c:85:90:24:a0:4d, ethertype IPv4 (0x0800), length 74: 10.0.100.6.8443 > 10.0.100.150.56188: Flags [S.], seq 1987005752, ack 657712526, win 65160, options [mss 1460,sackOK,TS val 4164476466 ecr 1210266758,nop,wscale 7], length 0
09:39:23.362133 40:62:31:13:eb:8f > 8c:85:90:24:a0:4d, ethertype IPv4 (0x0800), length 74: <MY-WAN-IP>.8443 > 10.0.100.150.56188: Flags [S.], seq 1987005752, ack 657712526, win 65160, options [mss 1460,sackOK,TS val 4164476466 ecr 1210266758,nop,wscale 7], length 0
09:39:23.364711 8c:85:90:24:a0:4d > 32:19:63:dd:f2:5f, ethertype IPv4 (0x0800), length 66: 10.0.100.150.56188 > 10.0.100.6.8443: Flags [.], ack 1, win 2058, options [nop,nop,TS val 1210266761 ecr 4164476466], length 0
09:39:23.364750 40:62:31:13:eb:8f > 32:19:63:dd:f2:5f, ethertype IPv4 (0x0800), length 66: 10.0.100.1.56188 > 10.0.100.6.8443: Flags [.], ack 1987005753, win 2058, options [nop,nop,TS val 1210266761 ecr 4164476466], length 0

As mentioned: using this exact same NAT config, but removing the bridge and putting the IP etc. on eth0.100 only this setup works just fine.

4

Hey i test in my lab and i got no issue with your configuration.

Bildschirmfoto 2020-10-28 um 21.55.32
Bildschirmfoto 2020-10-28 um 21.55.321410×722 69.6 KB

Config of VyOS:

set interfaces bridge br0 address '10.0.0.1/24'
set interfaces bridge br0 member interface eth0
set interfaces bridge br0 member interface eth1
set interfaces ethernet eth0 hw-id '50:01:00:01:00:00'
set interfaces ethernet eth1 hw-id '50:01:00:01:00:01'
set interfaces ethernet eth2 address '1.1.1.1/24'
set interfaces ethernet eth2 hw-id '50:01:00:01:00:02'
set nat destination rule 10 destination port '8443'
set nat destination rule 10 inbound-interface 'eth2'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 translation address '10.0.0.200'
set nat destination rule 100 destination address '1.1.1.1'
set nat destination rule 100 destination port '8443'
set nat destination rule 100 inbound-interface 'br0'
set nat destination rule 100 protocol 'tcp'
set nat destination rule 100 translation address '10.0.0.200'
set nat source rule 10 destination address '10.0.0.0/24'
set nat source rule 10 destination port '8443'
set nat source rule 10 outbound-interface 'br0'
set nat source rule 10 protocol 'tcp'
set nat source rule 10 source address '10.0.0.0/24'
set nat source rule 10 translation address 'masquerade'

all my host are vyos boxes so i open a netcat listener on the ‘server’

sudo nc -l 8443

i got a connection from the client and from extern with

sudo telnet 1.1.1.1 8443

also from client with raw ip of the server

sudo telnet 10.0.0.200 8443

with version of vyos you use?

show version

5

Very strange indeed. I tried using VyOS native on the box, and also via KVM (proxmox) with PCI passthrough. Both not working.

Version: VyOS 1.3-rolling-202010250804

6

I update to latest rolling “VyOS 1.3-rolling-202009230118” worked also fine.

This lab is in eve-ng, with qemu virtual hosts.
At last i also added vlans to be excluded this as an issue. but nothing break my nat rules.