VyOS VXLAN UDP packets dropped

[rgomez]

Hi,

Running simple VyOS setup for VXLAN with two VyOS VMs in VMware. Version = VyOS 1.3-rolling-202209301229.

The configuration on router A is:

set interfaces bridge br300 member interface eth1.300
set interfaces bridge br300 member interface vxlan300
set interfaces ethernet eth0 address ‘192.168.1.1/27’
set interfaces ethernet eth1 vif 300
set interfaces vxlan vxlan300 group ‘239.0.0.241’
set interfaces vxlan vxlan300 policy
set interfaces vxlan vxlan300 port ‘4789’
set interfaces vxlan vxlan300 source-interface ‘eth0’
set interfaces vxlan vxlan300 vni ‘10300’

Eth0 is the underlay and Eth1 is just a trunk.

On Router B we have the following, note the eth0 is via L3. Not sure what’s configured on that L3 since we don’t control that.

set interfaces bridge br300 member interface eth1.300
set interfaces bridge br300 member interface vxlan300
set interfaces ethernet eth0 address ‘192.168.2.1/27’
set interfaces ethernet eth3 vif 300
set interfaces vxlan vxlan300 group ‘239.0.0.241’
set interfaces vxlan vxlan300 policy
set interfaces vxlan vxlan300 port ‘4789’
set interfaces vxlan vxlan300 source-interface ‘eth0’
set interfaces vxlan vxlan300 vni ‘10300’

In the example below 1.1.1.1 is sourced from Router A and 2.2.2.2 is connected to eth1.300 at Router B.

What we are seeing is a pcap on eth1.300 on Router B shows:

14:36:26.158359 IP 1.1.1.1.1645 > 2.2.2.2.1812: RADIUS, Access-Request (1), id: 0x8f length: 330
14:36:26.429935 IP 2.2.2.2.1812 > 1.1.1.1.1645: RADIUS, Access-Accept (2), id: 0x8f length: 1272
14:36:26.430099 IP 2.2.2.2 > 1.1.1.1: ip-proto-17
14:36:26.495283 IP 1.1.1.1.1645 > 2.2.2.2.1812: RADIUS, Access-Request (1), id: 0x90 length: 369
14:36:26.598975 IP 2.2.2.2.1812 > 1.1.1.1.1645: RADIUS, Access-Accept (2), id: 0x90 length: 328

Pcap on br300 on Router B shows:

14:36:26.158330 IP 1.1.1.1.1645 > 2.2.2.2.1812: RADIUS, Access-Request (1), id: 0x8f length: 330
14:36:26.430099 IP 2.2.2.2.1812 > 1.1.1.1.1645: RADIUS, Access-Accept (2), id: 0x8f length: 1506
14:36:26.495258 IP 1.1.1.1.1645 > 2.2.2.2.1812: RADIUS, Access-Request (1), id: 0x90 length: 369
14:36:26.598975 IP 2.2.2.2.1812 > 1.1.1.1.1645: RADIUS, Access-Accept (2), id: 0x90 length: 328

And pcap on vxlan300 on router B shows:

14:36:26.158330 IP 1.1.1.1.1645 > 2.2.2.2.1812: RADIUS, Access-Request (1), id: 0x8f length: 330
14:36:26.430140 IP 2.2.2.2 > 1.1.1.1: ip-proto-17
14:36:26.495258 IP 1.1.1.1.1645 > 2.2.2.2.1812: RADIUS, Access-Request (1), id: 0x90 length: 369

I’m a bit lost here. Any tips what’s going on?

Thanks,

[Viacheslav]

Try to use static members without multicast group

[fernando]

also , check the MTU on this path what is the maximum without fragmentation on eth0 , this configuration has additional overhead and could give those drops.

[rgomez]

It’s dropping inside Router B so I’m not sure MTU can cause that.

[Apachez]

Assuming you dont have jumbo frames available on the underlay (uplink) means its limited to regular MTU 1500. This means that the overlay (downlink) packets that can pass can not be larger than 1450 bytes since VXLAN will encapsulate the packets with 50 bytes of information before sending them out on the underlay (uplink).

So the overlay aka downlink interface must have MTU set to 1450 bytes and preferly enable “mss-adjust” along with configure clients to not send packets larger than 1450 bytes (mss-adjust normally only affect TCP traffic so in order to fix UDP traffic the client must set proper MTU).

Also on Router B you have a mismatch between the br300 definition (eth1.300) and where its actually being used (eth3 vif300). Router A configuration seems to be matching (eth1.300 and eth1 vif 300).