Virtualized VyOS iperf3 slower than other VMs

General questions
1

Hi,

I’ve just started out with VyOS so I’m very new at this. The short version is: when testing with iperf3 in a virtualized setup, I’m getting about 5.5-6 Gbits/s VyOS (client) → Debian VM but other tests yielded much higher speeds:

  • Debian VM → VyOS VM ~ 20 Gbits/s
  • Deb VM → Deb VM ~ 30 Gbits/s
  • Deb VM → VyOS VM → Deb VM ~ 5.5 - 6 Gbits/s
  • Deb VM → Deb VM (w/ FRR) → Deb VM ~ 20 Gbits/s

Tried VyOS vyos-1.3-rolling-202008260118 and vyos-1.3-rolling-202008250118

The Guest VM setup is basic, I’m using fresh Debian 10 and Ubuntu 20.04.1 test VMs. VyOS is also a fresh install, I’ve went through the quick setup guide and only configured interfaces and a dhcp server. I’ve also tried to replicate the setup from this guide.

I’ve tested these scenarios on 2 different Proxmox 6.2-9 hosts. Xeon E3-1230 and Xeon E5-2660.
Guests have 2-4 Cores, 2-4 GB RAM, virtio NICs

Tried all sorts of things: NUMA, HyperThreading, disabled checksumming in VyOS. The guest CPU doesn’t even get to high load during tests (max 50%) and host CPU is very low (<25% and I have some other VMs running).

In my final test, I’ve tried to setup a Debian 10 VM with FRR and route between interfaces and it worked at 20 Gbits/s. Although, TBH I’m not sure it’s using FRR (I’m noob at this too).

2

I’ve installed a VyOS 1.2.5 image and this one routes at 20 Gbits/s. So, I’ve reinstalled VyOS 1.3 but it still works at 6 Gbits/s .

3

I’ve checked the differences between the vyos 1.2.5 and 1.3. The latter has some options off by default, such as tcp-segmentation-offload, generic-segmentation-offload, generic-receive-offload. I’ve enabled all of them with:

ethtool -K ethX sg on
ethtool -K ethX gso on
ethtool -K ethX gro on

Iperf3 tests now show 20 Gbits/s between routed VMs.

2 Likes
4

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.