I’m having an issue with a vxlan lab that I’ve been working on and was hoping someone could shed some light on the issue that I’m having. I have two sites connected via fortigate ipsec. vxlan traffic sits on top ospf.
Site A subnet (172.17.0.0/24)
site B subnet (172.17.128.0/24)
vxlan subnet (172.17.90.0/24)
The issue is that when I attempt to communicate between site A client to the vxlan subnet on site B then traffic gets encapsulated and shipped without issue. However, when performing the same test on the reverse order (B->A) the traffic never returns. I can see traffic being encapsulated as its being sent but the return traffic isnt bein encapsulated and thus never completes.
If you wait until the evpn mac table clears and initiate traffic in the opposite direction (client on site B → vxlan on site A) everything works as expected but now site A behaves as described above. In short, traffic will only work as expected in one direction at a time.
vyos configuration is as follows
Site A
set interfaces bridge br90 address ‘xxx.xxx.90.254/24’
set interfaces bridge br90 mac ‘xx:xx:xx:xx:xx:01’
set interfaces bridge br90 member interface eth1
set interfaces bridge br90 member interface vxlan90
set interfaces ethernet eth0 address ‘xxx.xxx.0.254/24’
set interfaces ethernet eth0 hw-id ‘xx:xx:xx:xx:xx:00’
set interfaces ethernet eth1 description ‘VIRTUAL-SUBNET’
set interfaces ethernet eth1 hw-id ‘xx:xx:xx:xx:xx:01’
set interfaces ethernet eth2 address ‘xxx.xxx.0.1/24’
set interfaces ethernet eth2 description ‘A_TO_INSIDE’
set interfaces ethernet eth2 hw-id ‘xx:xx:xx:xx:xx:02’
set interfaces loopback lo address ‘xxx.xxx.2.1/32’
set interfaces vxlan vxlan90 port ‘4789’
set interfaces vxlan vxlan90 source-address ‘xxx.xxx.2.1’
set interfaces vxlan vxlan90 vni ‘90’
set protocols bgp address-family ipv4-unicast maximum-paths ibgp ‘10’
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp neighbor xxx.xxx.2.2 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor xxx.xxx.2.2 address-family l2vpn-evpn nexthop-self
set protocols bgp neighbor xxx.xxx.2.2 capability dynamic
set protocols bgp neighbor xxx.xxx.2.2 remote-as ‘100’
set protocols bgp neighbor xxx.xxx.2.2 update-source ‘lo’
set protocols bgp system-as ‘100’
set protocols ospf area 0 network ‘xxx.xxx.0.0/24’
set protocols ospf area 0 network ‘xxx.xxx.90.0/24’
set protocols ospf area 0 network ‘xxx.xxx.2.1/32’
set protocols ospf auto-cost reference-bandwidth ‘100000’
set protocols ospf interface eth0 passive disable
set protocols ospf parameters router-id ‘xxx.xxx.2.1’
set protocols ospf redistribute static
set protocols static route xxx.xxx.0.0/22 next-hop xxx.xxx.0.1
Site B
set interfaces bridge br90 address ‘xxx.xxx.90.254/24’
set interfaces bridge br90 mac ‘xx:xx:xx:xx:xx:01’
set interfaces bridge br90 member interface eth1
set interfaces bridge br90 member interface vxlan90
set interfaces ethernet eth0 hw-id ‘xx:xx:xx:xx:xx:00’
set interfaces ethernet eth1 description ‘VIRTUAL_SUBNET’
set interfaces ethernet eth1 hw-id ‘xx:xx:xx:xx:xx:01’
set interfaces ethernet eth2 address ‘xxx.xxx.128.1/24’
set interfaces ethernet eth2 description ‘B_TO_INSIDE’
set interfaces ethernet eth2 hw-id ‘xx:xx:xx:xx:xx:02’
set interfaces loopback lo address ‘xxx.xxx.2.2/32’
set interfaces vxlan vxlan90 port ‘4789’
set interfaces vxlan vxlan90 source-address ‘xxx.xxx.2.2’
set interfaces vxlan vxlan90 vni ‘90’
set protocols bgp address-family ipv4-unicast maximum-paths ibgp ‘10’
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp neighbor xxx.xxx.2.1 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor xxx.xxx.2.1 address-family l2vpn-evpn nexthop-self
set protocols bgp neighbor xxx.xxx.2.1 capability dynamic
set protocols bgp neighbor xxx.xxx.2.1 remote-as ‘100’
set protocols bgp neighbor xxx.xxx.2.1 update-source ‘lo’
set protocols bgp system-as ‘100’
set protocols ospf area 0 network ‘xxx.xxx.128.0/24’
set protocols ospf area 0 network ‘xxx.xxx.90.0/24’
set protocols ospf area 0 network ‘xxx.xxx.2.2/32’
set protocols ospf auto-cost reference-bandwidth ‘100000’
set protocols ospf parameters router-id ‘xxx.xxx.2.2’
Below is a monitor output from client on site A → VXLAN Client on site B showing the lack of encapsulation on return
01:31:38.756227 IP 10.1.2.1.44918 > 10.1.2.2.4789: VXLAN, flags [I] (0x08), vni 90
IP 172.17.0.10 > 172.17.90.2: ICMP echo request, id 31652, seq 1, length 64
01:31:38.757025 IP 172.17.90.2 > 172.17.0.10: ICMP echo reply, id 31652, seq 1, length 64
01:31:40.755122 IP 10.1.2.1.44918 > 10.1.2.2.4789: VXLAN, flags [I] (0x08), vni 90
IP 172.17.0.10 > 172.17.90.2: ICMP echo request, id 32164, seq 2, length 64
01:31:40.756095 IP 172.17.90.2 > 172.17.0.10: ICMP echo reply, id 32164, seq 2, length 64
Any help or advice would be greatly appreciated.