Using vxlan with multiple unicast-remotes

Hi everybody,

with vxlan it is possible to use multiple remote addresses to connect 3 or more gateways to one cluster.
Vyos currently allows only one remote peer (or a multicast group or L3VPN).

Using bridges (and spanning tree) to bridge multiple vxlan interfaces aren’t a good workaround because load-balancing (something like this Help in Load-Balance Over VXLAN Tunnel) is not working.

With debian 11 an iproute2 I am using something like this in /etc/network/interfaces:

iface vxlan100
        vxlan-id 42
        vxlan_remoteip 10.1.2.1
        vxlan_remoteip 10.2.2.1
        mtu 1450

Proxmox also uses that for their SDN feature.

The “ip” command unfortunately don`t support adding interfaces with multiple remotes.
So you had to add the Interface without remote and then add them using “bridge” commands like this:

bridge fdb append to 00:00:00:00:00:00 dst 10.1.2.1 dev vxlan100
bridge fdb append to 00:00:00:00:00:00 dst 10.2.2.1 dev vxlan100

(its also possible to define the port for every remote)

Sources:
https://wiki.kleen.ch/vxlan
https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/
https://tecden.co.uk/linux/vxlan-and-namespaces-basic-concepts-creating-a-simple-isolated-router/

I wrote a little patch for vyos but I am not experienced enough in python and I am sure that could be done better.

First I had to add “multi:” to /opt/vyatta/share/vyatta-cfg/templates/interfaces/vxlan/node.tag/remote/node.def

Then I patched /usr/lib/python3/dist-packages/vyos/ifconfig/vxlan.py

vyos@testrouter# diff /tmp/vxlan.py  "/usr/lib/python3/dist-packages/vyos/ifconfig/vxlan.py"
65c65
<             'remote'                     : 'remote',
---
> #            'remote'                     : 'remote',
84a85,89
>         remote = dict_search('remote', self.config)
>         for rem in remote:
>             self.config["rem"] = rem
>             cmd2 = 'bridge fdb append to 00:00:00:00:00:00 dst {rem} dev {ifname}'
>             self._cmd(cmd2.format(**self.config))

Now this is possible:

set interfaces vxlan vxlan100 remote '10.1.2.1'
set interfaces vxlan vxlan100 remote '10.2.2.1'
set interfaces vxlan vxlan100 vni '10'

It’s working for some weeks. Also with “1.4-rolling-202112230317”.
Is it possible to get this feature into an official release?
Maybe also backported to Vyos 1.3?

It would be possible to implement this, but why not take it one step further and just start using l2vpn/evpn which is supported in VyOS 1.4?

You are right. L2VPN/EVPN is a really great feature and I already tested and love it :smiley:
For small setups it is a little bit too much overhead running BGP for just connecting 3-4 networks together.

Also if there a some Debian only instances (without FRR running) involved.
Plain vxlan is supported nearly everywhere.
Even on old VyOS 1.1.8 routers (i received a dozen 32-bit x86 PC-Engine Alix-Boards as a gift and gave them to my whole family and friends :wink:).

I gave it a try on Phabricator (⚓ T4120 [VXLAN] add ability to set multiple unicast-remotes) and also posted a Pullrequest (https://github.com/vyos/vyos-1x/pull/1127)
I hope I can satisfy the maintainers that this feature would be very useful :wink:

If I’ve done something wrong please don’t punch me (I am still learning) and feel free to correct me.