VyOS-Cisco 802.1Q Trunk for multiple VLANs over L2TPv3

Hello all,

This is on VyOS 1.2.0-rc9

My requirement is to extend a whole bunch of VLANs in a trunk over L3 between two sites. Switching equipment is Cisco, 3750X on both sides. I’ve successfully set up L2TPv3 pseudowire, so the two switches see each other as CDP neighbours. The switch ports on Cisco that connected to the L2 VyOS interfaces eth1/br0 are dot1q trunks. I am successfully able to communicate on VLAN1 VIF but seemingly not other VLANs… How could I make sure all the VLANs pass through so the clients on the other side acquire their IPs on their normal VLANs and can communicate through as normal. We are talking about a max of ~200 VLANs, but realistically about 20-80 will be at the remote site. I don’t require a VPN at this time, so it is a simple L2TPv3 over IP case. Kind of stuck. At HQ VLANs are defined and managed by a distinct VTP master, so all the IPs and subnets are defined there for all the VLANs that are desired to be trunked. Hence no IPs on bridge and VIF interfaces.

Any idea? I am sure there is just a bit I am missing somewhere.

HQ R1

Bridge, I need STP:

set interfaces bridge br0 aging '300'
set interfaces bridge br0 hello-time '2'
set interfaces bridge br0 max-age '20'
set interfaces bridge br0 priority '0'
set interfaces bridge br0 stp 'true'

L3 endpoint:

set interfaces ethernet eth0 address '<R1IP1>/24'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 smp-affinity 'auto'
set interfaces ethernet eth0 speed 'auto'

L2 interface:

set interfaces ethernet eth1 bridge-group bridge 'br0'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 smp-affinity 'auto'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth1 vif 1 description 'VLAN 1'
set interfaces ethernet eth1 vif 105 description 'VLAN 105'
...
set interfaces ethernet eth1 vif 113 description 'VLAN 113'

(VLAN 1 works; others don’t)

L2TPv3:

set interfaces l2tpv3 l2tpeth0 bridge-group bridge 'br0'
set interfaces l2tpv3 l2tpeth0 destination-port '10000'
set interfaces l2tpv3 l2tpeth0 encapsulation 'ip'
set interfaces l2tpv3 l2tpeth0 local-ip '<R1IP>'
set interfaces l2tpv3 l2tpeth0 mtu '1300'
set interfaces l2tpv3 l2tpeth0 peer-session-id '1'
set interfaces l2tpv3 l2tpeth0 peer-tunnel-id '1'
set interfaces l2tpv3 l2tpeth0 remote-ip '<R2IP>'
set interfaces l2tpv3 l2tpeth0 session-id '1'
set interfaces l2tpv3 l2tpeth0 source-port '10000'
set interfaces l2tpv3 l2tpeth0 tunnel-id '1'

REMOTE R2

Is configured identically, except the endpoint IPs are reversed. That is the configs on both ends are symmetrical. The packets do flow through, hosts on VLANs 100+ can’t access stuff in HQ nor acquire their IPs via DHCP (my dchp-relay is enabled, the DHCP server IPs are reachable from VyOS).

A related post from … 2014:

@Gunnymeden did you ever get this working?

For anyone looking for an answer to this question, the answer is vlan tags ARE in fact carried through using a l2tpv3 bridge. At least they are when testing with the 1.3-rolling version of VyOS paired with an EdgeRouter Lite.

I think where your config went sideways is when you added the vifs to the eth1 interface. Ideally this interface is blank, that way the kernel doesn’t strip any VLAN tags. It then takes any packets received and bridges them to the bridge interface, and then the l2tpv3 interface. Adding the vif may have told the kernel to strip the VLAN tags before sending them to the bridge.

Here’s my working config -

Remote Side:

vyos@vyos# show interfaces 
 bridge br123 {
     aging 300
     hello-time 2
     max-age 20
     member {
         interface eth1 {
         }
         interface l2tpeth123 {
         }
     }
     priority 32768
 }
 ethernet eth0 {
     address dhcp
     hw-id 52:54:00:2a:50:61
 }
 ethernet eth1 {
     hw-id 52:54:00:3d:45:07
 }
 l2tpv3 l2tpeth123 {
     destination-port 5000
     encapsulation ip
     local-ip LocalRouterIP
     mtu 1500
     peer-session-id 110
     peer-tunnel-id 10
     remote-ip RemoteRouterIP
     session-id 110
     source-port 5000
     tunnel-id 10
 }

`

Local Side:

ubnt@ubnt# show interfaces 
 bridge br123 {
     aging 300
     bridged-conntrack disable
     hello-time 2
     max-age 20
     priority 32768
     promiscuous disable
     stp false
 }
 ethernet eth0 {
     duplex auto
     speed auto
 }
 ethernet eth1 {
     bridge-group {
         bridge br123
     }
     duplex auto
     speed auto
 }
 ethernet eth2 {
     address dhcp
     duplex auto
     speed auto
 }
 l2tpv3 l2tpeth123 {
     bridge-group {
         bridge br123
     }
     destination-port 5000
     encapsulation ip
     local-ip LocalRouterIP
     mtu 1500
     peer-session-id 110
     peer-tunnel-id 10
     remote-ip RemoteRouterIP
     session-id 110
     source-port 5000
     tunnel-id 10
 }

tcpdump can confirm vlans are being carried across the l2tpv3 interface -

Local:

root@ubnt:/home/ubnt# tcpdump -i br123 -e vlan and icmp -vvvn
19:30:45.052846 98:f2:b3:17:de:fe > 98:f2:b3:17:e3:f8, ethertype 802.1Q (0x8100), length 102: vlan 210, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 11790, offset 0, flags [none], proto ICMP (1), length 84)
192.168.210.253 > 192.168.210.254: ICMP echo reply, id 4, seq 959, length 64

Remote:

root@vyos:/home/vyos# tcpdump -i br123 -e vlan and icmp -vvvn
00:30:44.412726 98:f2:b3:17:de:fe > 98:f2:b3:17:e3:f8, ethertype 802.1Q (0x8100), length 102: vlan 210, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 11790, offset 0, flags [none], proto ICMP (1), length 84)
192.168.210.253 > 192.168.210.254: ICMP echo reply, id 4, seq 959, length 64

Only thing i’m not sure about is if this would’ve caused an MTU issue. Over WAN it’s possible the MTU is clamped and the addition of a VLAN tag may push it over the limit. However, testing locally i didn’t run into that issue, i can send super large icmp (8000 bytes) without issue.

Sorry for bumping an old thread, just wanted anyone looking for this in the future to know that it is possible to trunk vlans across l2tpv3 with VyOS.