Windows performance behing VyOS VM

[millap]

Hi All,

I have a curious problem with Windows devices behind a VyOS VM which sits on a Klas TRX R6. The VyOS unit is running multiple VLAN’s, doing NAT, and firewalling between VLAN’s, along with running some containers to do healthchecks, flow monitoring, and log export.

MacOS, iOS, Android devices which connect to a wireless network fronted by Cisco WLC AP’s have great throughput (we get around 70+Mbps on the backhaul we’re using which is kind of expected). Windows devices, no so much (0.3Mbps). If we put the Windows device on a switch port in the same VLAN as the wireless, the throughput is the same. If we move the Windows machine to be directly connected to the TRX R6 bypassing VyOS, we get 70+Mbps, so we’re fairly confident the issue is something to do with VyOS, but what we’re not sure.

The VM on the TRX is configured simply with -

!
vm vyos
 hdd create vyos.img disk1: 200 type virtio
 cdrom disk1: vyos.iso
 cpu 4
 ram 16384
 nic 1 vSwitch 9 type virtio
 nic 2 vSwitch 10 type virtio
 start
!

Within VyOS itself, we configure the interfaces like so -

set interfaces dummy dum0 description '***MIRROR***'
set interfaces ethernet eth0 address {{ wan_ip }}
set interfaces ethernet eth0 description 'UNTRUST'
set interfaces ethernet eth1 mirror egress 'dum0'
set interfaces ethernet eth1 mirror ingress 'dum0'
set interfaces ethernet eth1 address {{ mgmt_ip }}
set interfaces ethernet eth1 description 'MGMT'
set interfaces ethernet eth1 vif 121 address {{ net1_ip }}
set interfaces ethernet eth1 vif 121 description 'NET1'
set interfaces ethernet eth1 vif 122 address {{ net2_ip }
set interfaces ethernet eth1 vif 122 description 'NET2'
set interfaces ethernet eth1 vif 123 address {{ net3_ip }}
set interfaces ethernet eth1 vif 123 description 'NET3'
set interfaces ethernet eth1 vif 124 address {{ net4_ip }}
set interfaces ethernet eth1 vif 124 description 'NET4'
set interfaces ethernet eth1 vif 125 address {{ net5_ip }}
set interfaces ethernet eth1 vif 125 description 'NET5'
set interfaces ethernet eth1 vif 126 address {{ net6_ip }}
set interfaces ethernet eth1 vif 126 description 'NET6'
set interfaces ethernet eth1 vif 127 address {{ net7_ip }}
set interfaces ethernet eth1 vif 127 description 'NET7'
set interfaces ethernet eth1 vif 128 address {{ net8_ip }}
set interfaces ethernet eth1 vif 128 description 'NET8'
set interfaces ethernet eth1 vif 140 address {{ net9_ip }}
set interfaces ethernet eth1 vif 140 description 'NET9'
set interfaces ethernet eth1 vif 142 address {{ net10_ip }}
set interfaces ethernet eth1 vif 142 description 'NET10'
set interfaces loopback lo

We’ve tried both vanilla Windows, and corporate image Windows. Any ideas on what we might do to identify what the issue is?

Cheers
Andy

[tjh]

Smells like an MTU issue to me.
If you tcpdump some traffic do you see a lot of fragmented packets?

You could also try turning hardware offloads on/off to see if that helps.

[millap]

Hi @tjh,

We do on the Windows client capture. Running a speedtest cli, 13% of all packets in the capture are retransmissions. This is on a corporate build with zScaler enabled (but not capturing, so traffic is direct). The capture from start to end is 12MB in size

On an Android client device, the capture size is 302MB and only 1.2% is retransmissions

We do have the following on both eth0/eth1 interfaces -

offload gro
offload gso
offload sg
offload tso

Removing this in a lab test makes no difference, but I will see if it does on the TRX R6 environment.

I’ve also tinkered with the MSS values on the eth1/vif interfaces, but this has made no difference.

I still come back to the ‘why only Windows’ question.

Cheers
Andy

[millap]

OK, so removing this from the VyOS VM interfaces resovled the issue -

offload gro
offload gso
offload sg
offload tso

The TRX R6 vSwitch has this already.

[tjh]

@millap

OK, so removing this from the VyOS VM interfaces resovled the issue

Thanks for coming back and confirming the fix - it’s always good when people do this so that future people might find a solution! Cheers.

[Apachez]

It seems (perhaps not for all offloading options but most of them) that they do more harm than good when being set in a VM-guest - unless the NIC is being passthrough to the VM-guest.

While on the VM-host they can still set various offloading options with great success.

So basically:

VM-guest: Disable all offloading options.
VM-guest with NIC passthrough: Enable all (or most) offloading options.
Baremetal: Enable all (or most) offloading options.

Proper way is to test one offloading at a time to verify CPU, interrupts, throughput, latency and then enable them in combo to verify that they still have an positive effect.