Cant make untagged VLAN to work with bridge (br0) and vrf

Below is using VyOS 1.5-rolling-202310240118.

Im currently trying to reconstruct my VyOS template so it will be using VLANs on bridges instead of directly on physical interfaces (the later have already been confirmed to be working).

That is instead of configure “interface ethernet ethX” with a vif (or setting address without a vif), I will create a vif on a bridge and then make physical interfaces members to this bridge (and define which VLANs will be exposed to each member along which should be 802.1Q tagged and untagged (only one can be untagged per physical interface)).

The advantage is that with a bridge you can put the same VLAN on several physical interfaces making the configuration more like how you setup a L3-switch with switchports from other vendors.

And then you can select which VLANs will be pure layer2 VLANs and which will have a VLAN-interface configured to be used as gateway for the hosts who belongs to this VLAN. Along with which will be 802.1Q tagged and which will be untagged.

Using L3-switches from other vendors you would do something like:

To create the VLANs:

vlan 99, 100, 200, 300

And then if you want to set IP-address for such VLAN you would do:

interface Vlan99
   description MGMT
   ip address 192.168.56.2/24
   vrf MGMT
!

And finally attach this VLAN to a physical interface and select if this should be 802.1Q tagged or untagged in both directions:

interface Ethernet0
   description MGMT
   switchport access vlan 99
   switchport trunk native vlan 99
   switchport trunk allowed vlan 99, 100, 200, 300
   switchport mode trunk
   vrf MGMT
!

In above example if you will only do untagged traffic then just the “switchport access” would be defined while if you would mix and want vlan 99 to be the untagged vlan (in both directions) you would use the “switchport trunk” lines and finally if vlan 99 would only exist as a tagged vlan you would remove that “switchport trunk native vlan” line. Also dont forget to change the operation with “switchport mode”.

Now… when I attempt to do the above in VyOS I fail… that is use a template where I in the end define that vlan 99 would be untagged on a particular member interface.

My MGMT-interface cannot ping any other host and other hosts cannot ping the MGMT-interface of VyOS, no arp shows up.

Digging through the manual I found this:

https://docs.vyos.io/en/latest/configuration/interfaces/bridge.html#enable-vlan-aware-bridge

It is not valid to use the vif 1 option for VLAN aware bridges because VLAN aware bridges assume that all unlabeled packets belong to the default VLAN 1 member and that the VLAN ID of the bridge’s parent interface is always 1

Which is also confirmed if looking in /usr/lib/python3/dist-packages/vyos/ifconfig/bridge.py who at around line 312 says:

# VLAN of bridge parent interface is always 1. VLAN 1 is the default
# VLAN for all unlabeled packets.

and then the command:

cmd = f'bridge vlan add dev {self.ifname} vid 1 pvid untagged self'

I assume my problem could be related to above because what I try to do when it comes to the MGMT-interface is to set it up as with any other VLAN-interface.

Even if this isnt necessary (I could just ignore using a bridge for that and configure it directly on the physical interface instead) I would use the same method no matter what kind of interface I am configuring (MGMT, WAN, LAN etc) and also have the advantage of having the MGMT-interface as a vif in a bridge which gives I can easily enable 2 (or more) physical interfaces to be part of the MGMT or for that matter select if it should be tagged or untagged at which member interface.

Or am I doing something wrong with my current attempt in VyOS?

The current config looks like this:

set interfaces bridge br0 description 'VRF_MGMT'
set interfaces bridge br0 enable-vlan
set interfaces bridge br0 member interface eth0 allowed-vlan '99' 
set interfaces bridge br0 member interface eth0 native-vlan '99'
set interfaces bridge br0 vif 99 address '192.168.56.2/24'
set interfaces bridge br0 vif 99 description 'MGMT'
set interfaces bridge br0 vif 99 vrf 'VRF_MGMT'
set interfaces bridge br0 vrf 'VRF_MGMT'

The current idea is that br0 is used by VRF_MGMT and br1 is used by VRF_INTERNET (as in one bridge per VRF).

I mean sure I could just ignore defining vif 99 and configure IP-address directly on the br0 but this would break the idea of using “same template no matter what kind of purpose” (as in first define vif in each bridge, then define which vif will be used at which member (physical interface) and finally if that vif would be 802.1Q tagged or untagged on that member).

Doing a bridge vlan show it looks ok?

vyos@vyos:~$ bridge vlan show
port		vlan-id
eth0		99 PVID Egress Untagged
eth1		100 PVID Egress Untagged
eth2		200 PVID Egress Untagged
eth3		300 PVID Egress Untagged
br0			1 PVID Egress Untagged
			99
br1			1 PVID Egress Untagged
			100
			200
			300

I think the error is when VyOS receives the packets - it will then internally put untagged ethernet frames as VLAN 1 which of course wont work.

I want whatever untagged is received on member eth0 to be put into VLAN 99.

So Im guessing I might be missing something here?

According to Bridge — VyOS 1.5.x (circinus) documentation native-vlan should take care of untagged traffic in both directions:

Set the native VLAN ID flag of the interface. When a data packet without a VLAN tag enters the port, the data packet will be forced to add a tag of a specific vlan id. When the vlan id flag flows out, the tag of the vlan id will be stripped

And there is an example who claims that native-vlan should be enough (as in the same vlan does not have to be defined as allowed-vlan aswell):

https://docs.vyos.io/en/latest/configuration/interfaces/bridge.html#using-vlan-aware-bridge

However here are some observations…

TEST1:

Using this (that is define the vrf at bridge-level, hoping that members will inherit vrf from parent):

set interfaces bridge br100 description 'VRF_MGMT'
set interfaces bridge br100 enable-vlan
set interfaces bridge br100 ip arp-cache-timeout '240'
set interfaces bridge br100 member interface eth0 native-vlan '99'
set interfaces bridge br100 vif 99 address '192.168.56.2/24'
set interfaces bridge br100 vif 99 description 'MGMT'
set interfaces bridge br100 vrf 'VRF_MGMT'

set interfaces bridge br101 description 'VRF_TEST'
set interfaces bridge br101 enable-vlan
set interfaces bridge br101 ip arp-cache-timeout '240'
set interfaces bridge br101 member interface eth1 native-vlan '100'
set interfaces bridge br101 member interface eth2 native-vlan '200'
set interfaces bridge br101 member interface eth3 native-vlan '300'
set interfaces bridge br101 vif 100 address '192.168.1.2/24'
set interfaces bridge br101 vif 100 description 'WAN'
set interfaces bridge br101 vif 200 address '192.168.2.2/24'
set interfaces bridge br101 vif 200 description 'DMZ'
set interfaces bridge br101 vif 300 address '192.168.3.2/24'
set interfaces bridge br101 vif 300 description 'LAN'
set interfaces bridge br101 vrf 'VRF_TEST'

Pinging using “ping 192.168.56.1 vrf MGMT” I get:

/bin/ping: connect: Network is unreachable

and no arp entries in VyOS.

But if I ping from external host that host will get arp replies but no ping replies.

Once that is done then the “arp -an” on the VyOS will display the arp of the external host but still no pings going through in either direction.

Also “show ip route vrf MGMT” and “show ip route vrf TEST” are empty.

TEST2:

Adding vrf for each vif along with vrf at the bridge itself like so:

set interfaces bridge br100 description 'VRF_MGMT'
set interfaces bridge br100 enable-vlan
set interfaces bridge br100 ip arp-cache-timeout '240'
set interfaces bridge br100 member interface eth0 native-vlan '99'
set interfaces bridge br100 vif 99 address '192.168.56.2/24'
set interfaces bridge br100 vif 99 description 'MGMT'
set interfaces bridge br100 vif 99 vrf 'VRF_MGMT'
set interfaces bridge br100 vrf 'VRF_MGMT'

set interfaces bridge br101 description 'VRF_TEST'
set interfaces bridge br101 enable-vlan
set interfaces bridge br101 ip arp-cache-timeout '240'
set interfaces bridge br101 member interface eth1 native-vlan '100'
set interfaces bridge br101 member interface eth2 native-vlan '200'
set interfaces bridge br101 member interface eth3 native-vlan '300'
set interfaces bridge br101 vif 100 address '192.168.1.2/24'
set interfaces bridge br101 vif 100 description 'WAN'
set interfaces bridge br101 vif 100 vrf 'VRF_TEST'
set interfaces bridge br101 vif 200 address '192.168.2.2/24'
set interfaces bridge br101 vif 200 description 'DMZ'
set interfaces bridge br101 vif 200 vrf 'VRF_TEST'
set interfaces bridge br101 vif 300 address '192.168.3.2/24'
set interfaces bridge br101 vif 300 description 'LAN'
set interfaces bridge br101 vif 300 vrf 'VRF_TEST'
set interfaces bridge br101 vrf 'VRF_TEST'

Pinging using “ping 192.168.56.1 vrf MGMT” I get:

PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data.
From 192.168.56.2 icmp_seq=1 Destination Host Unreachable
From 192.168.56.2 icmp_seq=2 Destination Host Unreachable

Output of “arp -an” now brings <incomplete> for the 2 IP-addresses Im trying to ping.

And if I ping from external host that host will get arp replies and once that was done the VyOS will have the arp of the external hosts but still no pings passing by.

“show ip route vrf MGMT” shows:

C>* 192.168.56.0/24 is directly connected, br10.99, 00:00:45

and “show ip route vrf TEST” shows:

C>* 192.168.1.0/24 is directly connected, br101.100, 00:00:46
C>* 192.168.2.0/24 is directly connected, br101.200, 00:00:46
C>* 192.168.3.0/24 is directly connected, br101.300, 00:00:46

TEST3:

Removing the vrf definition on the bridge but keep it on the vif’s gave the same result as with TEST1 above.

TEST4:

Completely remove all vrf definitions in the bridge configuration like so:

set interfaces bridge br100 description 'VRF_MGMT'
set interfaces bridge br100 enable-vlan
set interfaces bridge br100 ip arp-cache-timeout '240'
set interfaces bridge br100 member interface eth0 native-vlan '99'
set interfaces bridge br100 vif 99 address '192.168.56.2/24'
set interfaces bridge br100 vif 99 description 'MGMT'

set interfaces bridge br101 description 'VRF_TEST'
set interfaces bridge br101 enable-vlan
set interfaces bridge br101 ip arp-cache-timeout '240'
set interfaces bridge br101 member interface eth1 native-vlan '100'
set interfaces bridge br101 member interface eth2 native-vlan '200'
set interfaces bridge br101 member interface eth3 native-vlan '300'
set interfaces bridge br101 vif 100 address '192.168.1.2/24'
set interfaces bridge br101 vif 100 description 'WAN'
set interfaces bridge br101 vif 200 address '192.168.2.2/24'
set interfaces bridge br101 vif 200 description 'DMZ'
set interfaces bridge br101 vif 300 address '192.168.3.2/24'
set interfaces bridge br101 vif 300 description 'LAN'

Pinging using “ping 192.168.56.1” I get same result as in TEST2 (“Destination Host Unreachable”).

“arp -an” brings incomplete until external host have pinged the VyOS but still no pings in either direction.

“show ip route” shows all networks as expected directly connected:

C>* 192.168.1.0/24 is directly connected, br101.100, 00:02:15
C>* 192.168.2.0/24 is directly connected, br101.200, 00:02:15
C>* 192.168.3.0/24 is directly connected, br101.300, 00:02:15
C>* 192.168.56.0/24 is directly connected, br10.99, 00:02:15

Conclusion:

So… what am I doing wrong here?

Have I somehow stumbled upon a bug involving vlan-aware bridge in combo with native-vlan and vrf?

I have also rebooted the VyOS box between each test due to “did you turn it off and on again?” :wink:

The more I look at this the more Im getting convinced that something is broken with the bridge.py, but it would be handy if there is somebody else in this forum that could verify this?

It seems like its this code who will actually create the bridge in the Linux kernel:

Examining this file the following commands shows up (searching for “cmd”):

add_port:
ip link set dev {value} master {ifname}

del_port:
ip link set dev {value} nomaster

add_vlan:
bridge vlan add dev {self.ifname} vid {vlan} self

del_vlan:
bridge vlan del dev {self.ifname} vid {vlan} self

add_vlan_1:
bridge vlan add dev {self.ifname} vid 1 pvid untagged self

add_vlan?:
bridge vlan add dev {interface} vid {vlan} master

add_native-vlan?:
bridge vlan add dev {interface} vid {native_vlan_id} pvid untagged master

del_vlan?:
bridge vlan del dev {interface} vid {vlan} master

And thats it?

Not a word about vrf?

Why is the add_vlan_1 stuff going on in the “def set_vlan_filter(self, state):” function?

Looking at posts such as:

Linux: Why is ARP not answered via VLAN Bridge - Server Fault and
(Vlan-aware) Bridges on Linux – SDN Clinic
linux - What are "self" and "master" options for in "bridge vlan add"? - Unix & Linux Stack Exchange
bridge(8) — iproute2 — Debian bookworm — Debian Manpages

among others the syntax is slightly different.

As it seems that this would be the proper(?) way to do it?

# Create bridge with "enable_vlan"
ip link add br0 type bridge vlan_filtering 1
ip link set br0 up

# Add vlan
bridge vlan add dev br0 vid 100 self
bridge vlan add dev br0 vid 110 self
bridge vlan add dev br0 vid 120 self
bridge vlan add dev br0 vid 200 self
bridge vlan add dev br0 vid 210 self
bridge vlan add dev br0 vid 220 self

# Add member interface
ip link set eth1 master br0
ip link set eth2 master br0

# Delete default vlan from member interface
bridge vlan del dev eth1 vid 1
bridge vlan del dev eth2 vid 1

# Add vlan to member interface
bridge vlan add dev eth1 vid 100 pvid untagged master
bridge vlan add dev eth1 vid 110 master
bridge vlan add dev eth1 vid 120 master
bridge vlan add dev eth2 vid 200 pvid untagged master
bridge vlan add dev eth2 vid 210 master
bridge vlan add dev eth2 vid 220 master

# Show result
bridge vlan show
3 Likes

Hey @Apachez I saw your message today.

I am not familiar with vrf yet and I am not sure why this is not working in your setup

So after some troubleshooting I finally nailed it down…

My VyOS is running in Virtualbox with following interfaces:

eth0 (MGMT), host-only adapter on host vboxnet0
eth1 (WAN), bridged adapter on host eth0
eth2 (DMZ), bridged adapter on host eth0
eth3 (LAN), bridged adapter on host eth0

It turns out that the default setting of “Deny” for “Promiscious mode” (under advanced for each adapter in Virtualbox) will filter dstmac towards the VM guest (VyOS) - but not srcmac from the VM guest (VyOS) !?

Since configuring bridges will make VyOS (well Linux) by default create new mac-addresses for each bridge and vif (vlan within bridge) the returning traffic back to the VM guest (VyOS) never actually reached VyOS itself.

The strange thing was that Virtualbox would happily let the srcmac through in outbound direction as in I could see arp requests and icmp echo requests (when configured static arp) leaving VyOS and be seen on the physical wire of the host. And then see the remote host reply and but VyOS never got the reply.

So the fix is to change “Promiscious mode” (under advanced for each adapter in Virtualbox) to “Allow”.

Another workaround if you cant/dont want to change this setting in your VM host is to statically configure mac for each vif (within the bridge) to be the same as the “hw-id” of the physical interface the vlan will be using (from the VyOS point of view).

This workaround will only work if each VLAN of the bridge only use a single physical interface (from the VyOS point of view).

If you want br100.99 to have both eth0 and eth1 as members to the bridge you will most likely need to alter that “Promiscious mode” of the VM host.

Here is the final config with “Promiscious mode” set to “Allow” at the VM host (Virtualbox):

set interfaces bridge br100 description 'VRF_MGMT'
set interfaces bridge br100 enable-vlan
set interfaces bridge br100 ip arp-cache-timeout '240'
set interfaces bridge br100 member interface eth0 native-vlan '99'
set interfaces bridge br100 vif 99 address '192.168.56.2/24'
set interfaces bridge br100 vif 99 description 'MGMT'
set interfaces bridge br100 vif 99 vrf 'VRF_MGMT'
set interfaces bridge br100 vrf 'VRF_MGMT'
set interfaces bridge br101 description 'VRF_TEST'
set interfaces bridge br101 enable-vlan
set interfaces bridge br101 ip arp-cache-timeout '240'
set interfaces bridge br101 member interface eth1 native-vlan '100'
set interfaces bridge br101 member interface eth2 native-vlan '200'
set interfaces bridge br101 member interface eth3 native-vlan '300'
set interfaces bridge br101 vif 100 address '192.168.1.2/24'
set interfaces bridge br101 vif 100 description 'WAN'
set interfaces bridge br101 vif 100 vrf 'VRF_TEST'
set interfaces bridge br101 vif 200 address '192.168.2.2/24'
set interfaces bridge br101 vif 200 description 'DMZ'
set interfaces bridge br101 vif 200 vrf 'VRF_TEST'
set interfaces bridge br101 vif 300 address '192.168.3.2/24'
set interfaces bridge br101 vif 300 description 'LAN'
set interfaces bridge br101 vif 300 vrf 'VRF_TEST'
set interfaces bridge br101 vrf 'VRF_TEST'

And here is the workaround if you dont want to change that “Promiscious mode” to “Allow” at the VM host (Virtualbox):

Note: The mac used below matches the “hw-id” of the interface the VyOS will use.

set interfaces bridge br100 description 'VRF_MGMT'
set interfaces bridge br100 enable-vlan
set interfaces bridge br100 ip arp-cache-timeout '240'
set interfaces bridge br100 member interface eth0 native-vlan '99'
set interfaces bridge br100 vif 99 address '192.168.56.2/24'
set interfaces bridge br100 vif 99 description 'MGMT'
set interfaces bridge br101 vif 99 mac '08:00:27:7d:40:c5'
set interfaces bridge br100 vif 99 vrf 'VRF_MGMT'
set interfaces bridge br100 vrf 'VRF_MGMT'
set interfaces bridge br101 description 'VRF_TEST'
set interfaces bridge br101 enable-vlan
set interfaces bridge br101 ip arp-cache-timeout '240'
set interfaces bridge br101 member interface eth1 native-vlan '100'
set interfaces bridge br101 member interface eth2 native-vlan '200'
set interfaces bridge br101 member interface eth3 native-vlan '300'
set interfaces bridge br101 vif 100 address '192.168.1.2/24'
set interfaces bridge br101 vif 100 description 'WAN'
set interfaces bridge br101 vif 100 mac '08:00:27:cc:6b:0a'
set interfaces bridge br101 vif 100 vrf 'VRF_TEST'
set interfaces bridge br101 vif 200 address '192.168.2.2/24'
set interfaces bridge br101 vif 200 description 'DMZ'
set interfaces bridge br101 vif 200 mac '08:00:27:a7:73:6e'
set interfaces bridge br101 vif 200 vrf 'VRF_TEST'
set interfaces bridge br101 vif 300 address '192.168.3.2/24'
set interfaces bridge br101 vif 300 description 'LAN'
set interfaces bridge br101 vif 300 mac '08:00:27:d7:4b:4c'
set interfaces bridge br101 vif 300 vrf 'VRF_TEST'
set interfaces bridge br101 vrf 'VRF_TEST'
3 Likes

A drawback is that it seems like the performance using the bridge approach is worser than having the IP-address configured directly on the interface :frowning: