VXLAN over Internet not working - Simple use case

1

Hello VyOS community,

I’m trying to set up a simple VXLAN tunnel between two VyOS 1.4 routers over the Internet, but I’m encountering issues. I’ve configured what seems to be a straightforward unicast VXLAN setup, but the tunnel is not establishing. Here are the configurations on both routers:

Router 1:

set interfaces vxlan vxlan0 address '192.168.10.2/24'
set interfaces vxlan vxlan0 mtu '1450'
set interfaces vxlan vxlan0 port '4789'
set interfaces vxlan vxlan0 remote '1.1.1.1'
set interfaces vxlan vxlan0 source-interface 'eth0'
set interfaces vxlan vxlan0 vni '1010'

Router 2:

set interfaces vxlan vxlan0 address '192.168.10.1/24'
set interfaces vxlan vxlan0 mtu '1450'
set interfaces vxlan vxlan0 port '4789'
set interfaces vxlan vxlan0 remote '2.2.2.2'
set interfaces vxlan vxlan0 source-interface 'eth0'
set interfaces vxlan vxlan0 vni '1010'

please note I repleaced the public ips with 1.1.1.1 and 2.2.2.2

I’ve verified that:

  1. The public IPs can ping each other.
  2. UDP port 4789 is open on both firewalls.
  3. There are no conflicting NAT rules.
  4. The VNI matches on both sides.

Despite this setup, I’m unable to establish the VXLAN tunnel. When I try to ping across the tunnel (192.168.10.1 to 192.168.10.2 or vice versa), I get no response.

Montiro Traffic sees no VXLAN traffic receiving however when I use nc -zvu 1.1.1.1 4789 I can see at least traffic

monitor traffic interface eth0 filter "host 169.150.222.98"

I’m puzzled as to why this simple use case isn’t working. Has anyone successfully set up a VXLAN tunnel over the Internet using VyOS 1.4? Are there any additional configurations or troubleshooting steps I should try?

Any help or insights would be greatly appreciated. Thank you in advance!

2

Delete firewall at all

3

I’m getting the exact same issues with VyOS 1.5.

4

3 possible things come to mind.

  • Firewall is blocking either input or output traffic on one or both ends
    • You can disable the firewall temporarily to attempt to rule that out like Viacheslav said
  • The provider is blocking UDP traffic, or even specifically UDP/4789
    • Not much you can do here, but you can try to change the port to something that may be allowed (like UDP/53)
  • TTL is being exceeded on the packets (default vxlan interface will have a TTL of 16)
    • You can set the TTL with something like:
      set interfaces vxlan vxlan0 parameters ip ttl 64
5

Also verify that you actually do have 1500 L3-MTU between the routers (since VXLAN on its own adds 50 bytes and I see you already defined mtu 1450 for the VXLAN tunnel).

Best is to use ping (with DF set) and try with “-s 1472” because ICMP header of 28 bytes will be added so the result will be 1500 bytes (also verify with tcpdump/tshark/wireshark).

You could also try changing port in case one of the ISP’s are doing something fishy.

Generally speaking I would never do VXLAN without some kind of encryption specially not over the Internet. Preferly using wireguard to protect the content (or IPsec as second option).

When doing VXLAN internally I use jumboframes so they will fit both the VXLAN headers and whatever encryption is used so the end-to-end for the customers can be 1500 L3-MTU.