VXLAN works on VirtualBox but not on KVM


#1

I had never played with VXLAN before until a couple of weeks ago.

I tried it out and couldn’t get it working on VirtualBox until I figured out that the LAN facing NICs needed to be set to promiscuous mode in the VirtualBox settings.

After that it worked great.

See Vyos Forum question: (SOLVED) How to get VXLAN on VyOS 1.1.8 or VyOS 1.2.0-rc8 working? for the configuration I’m using.

However, my production routers run on KVM. I copied my working VirtualBox proof of concept VMs (all four routers, r1 - r4) test setup to a server running KVM and now it is back to not working again on my KVM hosted VMs.

I suspect the same thing as on VirtualBox, the NIC promiscuous mode setting.

Now on KVM, I can ping from each router r1 <-> r2, r2 <-> r3, r3 <-> r4, but I can’t ping from r1 to r4 which are on the same VXLAN (which is my goal).

I would expect this to “Just Work ™” as it does on VirtualBox (but with a special trick of setting promiscous mode) on VirtualBox NICs.

So now the question is: How do I set promiscous mode on the either physical or virtual NICs on KVM to get VXLAN working?

Please help! It is probably something simple but I’m tearing my hair out trying to figure it out.

What normally would have been a 30 minute job is now up to two weeks trying different things and still not working.

Thanks in advance for any help.


#2

sigh

I am not totally sure why I’ve spent 2 weeks trying to get it to work on KVM and it wasn’t working and then today it finally started working???

The only thing I know I did was update CentOS 7.5 to CentOS 7.6. I don’t see how that would “fix” it.

I have been having some problems getting the VXLAN settings to “stick” on VyOS.

I create the VXLAN, save the configuration and reboot and the VXLAN shows in the configuration but the actual VXLAN interface would not be created.

Then VyOS was confused and would say that I couldn’t delete the configuration.

I would have to edit the /config/config.boot file, take out the VXLAN configuration there and reboot and then re-configured the VXLAN.

This happened many, may times over the course of two weeks. I don’t know why.

I have been switching back and forth between 1.1.8 and 1.2.0-rc10 several times today.

I also was showing some configuration error that I don’t know what it was on the 1.1.8 boot, maybe that was causing problems.

It seems stable now, I’ve rebooted multiple times on r1 and r3 and the VXLAN continues to come up correctly and work.

So, … I still don’t know why it wouldn’t work 2 weeks ago all the way up to today.

Maybe some combination of bugs?

My current working configuration is:

CentOS 7.6 (KVM) host
VyOS 1.2.0-rc10 (Guest for routers r1, r2, r3, r4)

This is my test setup.

Next step is to try putting it in production.


#3

Still not working on production network.

Production KVM network is now at exact same versions as test network:

CentOS 7.6 (KVM HOST)
VyOS 1.2.0-rc10 (KVM Guest routers for r2 and r3)

The only difference I can think of at the moment is that on my test KVM network I’ve got virtual networks set up between the routers and not physical networks.

Test: r1 <— Virtual Network —> r2 <— OpenVPN over Virtual Network —> r3 <— Virtual Network —> r4

Production: r1 <— physical VLAN —> r2 (KVM Guest) <— OpenVPN over physical (internet) network —> r3 (KVM Guest) <— physical VLAN —> r4

I actually am seeing broadcasts from r1 to r4 but not pings. I wonder what would hold off pings from working?

Please help.

Thanks in advance.


Should BGP work between VyOS versions?