(SOLVED) How to get VXLAN on VyOS 1.1.8 or VyOS 1.2.0-rc8 working?


#1

I’m trying to get VXLAN working for the first time, I’ve never used it before.

I spent all today trying to get VXLAN to work.

I only found two example configurations when searching google, one on VyOS documentation WIKI and another tutorial at https://www.reddit.com/r/CBRjack/comments/5ya0tc/vxlan_implementation_with_vyos_routers/ URL.

I’m testing this out in VirtualBox first before trying to implement it on my production network.

I’ve got 4 VyOS routers r1 <-> r2 <-> r3 <-> r4, my goal is to have r1 and r4 on the same LAN 10.255.15.0/24 with r1 = 10.255.15.101/24 and r4 as 10.255.15.102/24.

r2 and r3 are the “link” routers that communicate via VXLAN setup between each other. r2 = 1.1.1.1/24, r3 = 1.1.1.2/24.

Here is my configuration on r2:
interfaces {
bridge br15 {
address 10.255.15.1/24
aging 300
hello-time 2
max-age 20
priority 0
stp false
}
ethernet eth0 {
bridge-group {
bridge br15
}
duplex auto
hw-id 08:00:27:90:8e:d6
smp_affinity auto
speed auto
}
ethernet eth1 {
address 1.1.1.1/24
duplex auto
hw-id 08:00:27:8d:9e:c3
smp_affinity auto
speed auto
}
ethernet eth2 {
address dhcp
duplex auto
hw-id 08:00:27:bc:98:1b
smp_affinity auto
speed auto
}
loopback lo {
}
vxlan vxlan15 {
bridge-group {
bridge br15
}
group 239.0.0.15
link eth1
vni 15
}
}
system {
config-management {
commit-revisions 20
}
console {
device ttyS0 {
speed 9600
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password $1$6ndBU0vK$0fI7beXbfjN.UW1Gq4yqN/
plaintext-password “”
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
package {
auto-sync 1
repository community {
components main
distribution helium
password “”
url http://packages.vyos.net/vyos
username “”
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone UTC
}

/* Warning: Do not remove the following line. /
/
=== vyatta-config-version: “cluster@1:config-management@1:conntrack-sync@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@4:nat@4:qos@1:quagga@2:system@6:vrrp@1:wanloadbalance@3:webgui@1:webproxy@1:zone-policy@1” === /
/
Release version: VyOS 1.1.8 */

Here is the configuration on r3:
interfaces {
bridge br15 {
address 10.255.15.2/24
aging 300
hello-time 2
max-age 20
priority 0
stp false
}
ethernet eth0 {
address 1.1.1.2/24
duplex auto
hw-id 08:00:27:1c:bd:a4
smp_affinity auto
speed auto
}
ethernet eth1 {
bridge-group {
bridge br15
}
duplex auto
hw-id 08:00:27:df:b8:0d
smp_affinity auto
speed auto
}
ethernet eth2 {
address dhcp
duplex auto
hw-id 08:00:27:a4:85:14
smp_affinity auto
speed auto
}
loopback lo {
}
vxlan vxlan15 {
bridge-group {
bridge br15
}
group 239.0.0.15
link eth0
vni 15
}
}
system {
config-management {
commit-revisions 20
}
console {
device ttyS0 {
speed 9600
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password $1$9139mcWM$kA3SIzLWGlzYmrA2M.2SF0
plaintext-password “”
}
level admin
}
}
ntp {
server 0.pool.ntp.org {
}
server 1.pool.ntp.org {
}
server 2.pool.ntp.org {
}
}
package {
auto-sync 1
repository community {
components main
distribution helium
password “”
url http://packages.vyos.net/vyos
username “”
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone UTC
}

/* Warning: Do not remove the following line. /
/
=== vyatta-config-version: “cluster@1:config-management@1:conntrack-sync@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@4:nat@4:qos@1:quagga@2:system@6:vrrp@1:wanloadbalance@3:webgui@1:webproxy@1:zone-policy@1” === /
/
Release version: VyOS 1.1.8 */

I can ping from r1 to r2 and r4 to r3.

Here is what is not working:

I’m trying to ping from r1 to r4.

When I do tcpdump on r4 I see arp requests and replies.

The strange thing is that when I do tcpdump on r3 eth1 I see arp requests going to r4 but not arp replies coming back from r4.

How could I NOT be seeing arp replies from r4 to r3? There is no firewall and they are on the same LAN, I can ping from r3 to r4.

I think that is the key but I don’t understand why I’m not seeing the arp replies on r3 from r4 that should be sent back to r1.

I first tried this on VyOS 1.2.0-rc8 but when it didn’t work I switched back to trying VyOS 1.1.8 and it didn’t work either.

It is probably something simple that I’m missing, either a mistake in the configuration or a mistaken in my understanding of how this works.

Does anyone have any ideas, thoughts, suggestions on how to get this to work?

It seems it should be “simple” but I’ve spent all day trying things and it is not working yet.

Thanks in advance for any help.


VXLAN works on VirtualBox but not on KVM
#2

Not sure on the VXLAN stuff though it looks really interesting I shall need to take a look into that
so if I’m way off here just tell me, but have you checked that the arp requests you’re seeing from r3 to r4 are requesting an address that r4 has? If not then that’d make sense why r4 isn’t sending back a reply?


#4

No, the whole purpose of VXLAN is to extend the LAN to another router across the network so when r1 tries to ping r4 VXLAN is supposed to transport both the request to the remote end (r4 in this case) and then transport the reply from r4 back to r1.

So you are correct. r3 does not have the MAC address that is being replied to (hope I got that right, it is confusing), but I’m surprised that r4 is not even seeing the ARP replies that r3 is sending out.

The ARP requests are getting from r1 to r4, and the replies are being sent from r4 but then are not even getting back to r3 to be able to send to r1 via r2 for some reason.

I’m probably missing some detail of how VXLAN is supposed to work.

If someone has a good detailed drawing and setup of a working VXLAN then it would be very helpful to share.

The two examples I’ve seen so far leave out lots of details.

They skip steps and information and have no details of how to troubleshoot if it doesn’t work.

They just say: here is my (partial) config and, hey, it works! You can ping!

When you are first learning something then even if there are 2 unknown variables it is very difficult to solve the equations.


#5

OK, I finally found the solution:

On VirtualBox, the r2 and r3 LAN facing NICs need to be set to allow promiscuous ethernet mode.

I set that and rebooted and IMMEDIATELY (if not sooner :slight_smile: I can now ping between r1 and r4.

I knew I was missing SOMETHING but I didn’t know what until I found that video.

Great thanks to the author of that video.


#6

BTW, I’m planning on using this to connect two PlayStation 3 boxes together over VPN so we can play WarkHawk after the Sony decommissions the Sony PlayStation servers for WarHawk on 2019-01-31 :frowning:


#7

Glad you got it working. Sorry I didn’t get back to you, I labbed it up myself and got the same issues you were having. Didn’t get around to finding the solution. Nice find


#8

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.