VXLAN throughput issue?

I am new to VyOS and I’m hoping we can make use of this as our primary routers instead of Mikrotiks. However, I am facing a problem with VXLAN wherein throughput is limited. Here’s a high level view of the topology

The underlay is capable of 10G throughput. I’ve tested it using iPerf3 by trunking a VLAN from between 2 iPerf3 hosts. Even if both hosts are on separate broadcast domain, iPerf3 test between 2 hosts through underlay routing, I am getting close to 10G. However, through VXLAN, I am seeing throughput limitation close to 1Gbps. Below are the test results.
There are 4CPUs on each devices, they are beefy one so its I don’t believe we’re CPU is peaking. I also checked the server CPU while iPerf is running and it was below 50%.

First comment on this post shows the iperf results.

Any ideas?

VyOS1 config
set interfaces vxlan vxlan2005 vni 2005
set interfaces vxlan vxlan2005 mtu 2000
set interfaces vxlan vxlan2005 source-address 10.221.3.26
set interfaces vxlan vxlan2005 remote 10.221.2.6
set interfaces bridge br2005 member interface ‘eth0.2005’
set interfaces bridge br2005 member interface ‘vxlan2005’
set interfaces ethernet eth0 description ‘VXLAN host V2005’
set interfaces ethernet eth0 vif 2005 mtu 1700

VM1 and VM2 host MTU is 1600

VyOS2 config
set interfaces vxlan vxlan2005 vni 2005
set interfaces vxlan vxlan2005 mtu 2000
set interfaces vxlan vxlan2005 source-address 10.221.2.6
set interfaces vxlan vxlan2005 remote 10.221.3.26
set interfaces bridge br2005 member interface ‘eth2’
set interfaces bridge br2005 member interface ‘vxlan2005’
set interfaces ethernet eth2 description ‘VXLAN host V2005’
set interfaces ethernet eth2 mtu 1700
set interfaces bridge br2005 address 10.4.192.30/24

^ VyOS facing VM2 is untagged on VMNIC hence there’s no VIF 2005 on eth2

here’s the iperf test results

What is the MTU of the VxLAN?
You may be experiencing a fragmentation problem

VXLAN MTU is set to 2000.
Underlay MTU is 9000.
Interface facing host and bridge interface is default 1500.

Please also share your config

Added to the post itself.

Did you enable ethernet offloading options? set interfaces ethernet eth0 offload?

Looking at the config, seems like it is already there on both routers, all interfaces.

set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces ethernet eth2 offload gro
set interfaces ethernet eth2 offload gso
set interfaces ethernet eth2 offload sg
set interfaces ethernet eth2 offload tso

What does this do?

Tells the Linux driver to perform vertain actions in hardware if possible

OK since there’s no improvement. Is there any things I can try? I’d like to eliminate these basic issues before we subscribe to the paid support

Did you try:

set system option performance throughput

or

set system option performance latency

I was having a throughput issue on a fairly fast computer (quad Xeon 16GB RAM) and either of these fixed it, I stuck with the latency option since this was more important to me.

I tried this and the one thing it did I didn’t like was to use up all the CPU of the VMWare. CPU is pegged at 100%

Hi. Which exactly NIC defined for Router VM, VMXNET3 or E1000e?

The NIC is using VMXNET3

Could you try to run iperf3 with -P key?
iperf3 -c x.x.x.x -P 2
The potential bottleneck can be single VXLAN flow, that hypervisor can not balance properly using IRQs