Please note that all device in here are VM in Openstack cloud with vNIC, CE is just simple routing device do the static route, each single link between devices has about 15Gbps capacity, and I cannot configure the underlay/hardware network
Everything works great with tenant-a (VM1 and VM2), execept following issues:
Iperf throughput between VM-1 and VM-2 is very low, about 400Mbps, even though each link can handle 15Gbps. later on I find out because of VXLAN encapsulation. If I remove VXLAN, only use BGP, the throughput can achieve 3.5Gbps
As I prepared the 2nd tenant (tenant-b for VM3 and VM4), I created 2nd vrf with table ID, as soon as I commit the vni for 2nd vrf in PE-1, the bgp routing table for tenant-a is messed up, and VM1 cannot connect to VM2. More specific, PE1 did not advertise the static route for VM1 anymore.
So my question as below:
How can I configure the second VRF without messing the BGP routing table of the first VRF?
Why the performance with VXLAN is too low and how can I fix it? I tried all offload features, change MTU but not helping
Even if I not use VXLAN, the performance is 3.5Gbps is way less than maximum capacity of the link, how can I improve it? the VyOS has 1 core and 1G ram but the CPU/RAM usage is very low during iperf, is that normal?
show version:
Version: VyOS 1.4-rolling-202110240217
Release train: sagitta
Built on: Sun 24 Oct 2021 02:17 UTC
Build UUID: 08131379-f272-430c-9969-ee0ffcad495a
Build commit ID: f6c1a927ca63da
Architecture: x86_64
Boot via: installed image
System type: KVM guest
Hardware vendor: OpenStack Foundation
Hardware model: OpenStack Nova
Hardware S/N: 30e74c18-1e6c-4ec5-b2bb-600e5872601e
Hardware UUID: 30e74c18-1e6c-4ec5-b2bb-600e5872601e
Copyright: VyOS maintainers and contributors
show configuration commands for PE-1:
set firewall all-ping 'enable'
set firewall broadcast-ping 'enable'
set firewall config-trap 'disable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall state-policy established action 'accept'
set firewall state-policy invalid action 'accept'
set firewall state-policy related action 'accept'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces bridge br1001 address '192.168.12.10/24'
set interfaces bridge br1001 member interface eth3
set interfaces bridge br1001 member interface vxlan1001
set interfaces bridge br1001 vrf 'tenant-a'
set interfaces ethernet eth1 address '192.168.10.241/24'
set interfaces ethernet eth1 mtu '1442'
set interfaces ethernet eth2 address '10.0.0.11/24'
set interfaces ethernet eth2 mtu '1442'
set interfaces ethernet eth3 mtu '1300'
set interfaces loopback lo
set interfaces vxlan vxlan1001 mtu '1500'
set interfaces vxlan vxlan1001 parameters nolearning
set interfaces vxlan vxlan1001 port '4789'
set interfaces vxlan vxlan1001 source-address '10.0.0.11'
set interfaces vxlan vxlan1001 vni '1001'
set protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set protocols bgp address-family l2vpn-evpn advertise-all-vni
set protocols bgp local-as '65000'
set protocols bgp neighbor 10.0.0.12 address-family l2vpn-evpn
set protocols bgp neighbor 10.0.0.12 remote-as '65000'
set protocols bgp neighbor 10.0.0.12 update-source 'eth2'
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'pe-1'
set system login user vyos authentication encrypted-password '$6$t1hKqDrcJ1uMPlMX$W6DJCsriXUFPU2OVKiacGk5Tv9cbige6kx0kRrMrU273R6vAtwfh8VV56srPj61.i/tXRIrDuWj9fZ8EBfXDQ1'
set system login user vyos authentication plaintext-password ''
set system ntp server time1.vyos.net
set system ntp server time2.vyos.net
set system ntp server time3.vyos.net
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set vrf name tenant-a protocols bgp address-family ipv4-unicast redistribute connected
set vrf name tenant-a protocols bgp address-family ipv4-unicast redistribute static
set vrf name tenant-a protocols bgp address-family l2vpn-evpn advertise ipv4 unicast
set vrf name tenant-a protocols bgp local-as '65000'
set vrf name tenant-a protocols static route 192.168.11.0/24 next-hop 192.168.12.1
set vrf name tenant-a protocols static route 192.168.21.0/24
set vrf name tenant-a table '1001'
set vrf name tenant-a vni '1001'
Hi @hoai.nam.1512 , as VyOS uses software process packets in bridges, performance might be slightly different and depends on many factors (CPU performance, packet sizes, etc.). In this thread we see that using 2 cores on an e5-2670 provided up to 8-9 Gbps of speed. Maybe you could try to check different configurations for the VyOS hosts and compare performance.
For the VRF issue, I think you hit a known problem that is described here. As a workaround you can manually add missed VRF in VTYSH configuration, restart BGP on both sides and check if that helps.
