1.3 RC1 Performance regression, high CPU usage

Sure!

Go here: Create a New Account

Register a new account.

Then under the VYOS 1.3 Equuleus heading on the left, click the “Submit Bug Report” button and fill out the form that is provided.

Please provide as much detail as possible so that it is possible for someone to reproduce it. You can link to this forum topic as well, but please do include good information in the bug report, not just a link to the forum.

Thanks!

Hi @hiteshhapani ,

Could you please provide more details regarding the machine specs where you ran those tests and saw a performance degradation from 1.2.5 LTS to 1.3 RC?
Also, was it a VM or bare metal install?

Would you be willing to test again with RC4 so I can add this information on the bug report?

Thanks.

@Ralm We tested it on VM(2G RAM/4 CPU). Same result on RC4 as well

IPERF3 between two interfaces on my VYOS VM (Proxmox, J1900 4-core Celron). VM is identical in config and specs, literally just swapping system image between tests.

v1.2.7:
2.90 Gbits/sec
ksoftirqd/0 using ~1% CPU
Likely limited by my host

v1.3.0-rc4:
306 Mbits/sec
ksoftirqd/0 using ~98% CPU

This is a massive performance degradation. About to test on bare metal.

Edit 1:
Same issue with baremetal. This is on an old PC with two NICs. Obviously limited by the 1 gig NICs in this test. 1.3.4 came close to line speed, but ksoftirqd was still using almost 100% CPU.

v1.2.7:
938 Mbits/sec
ksoftirqd/0 using ~3% CPU

v1.3.0-rc4:
905 Mbits/sec
ksoftirqd/0 using ~95% CPU

Edit 2:
Also can confirm the same regression in 1.3 rolling and 1.4 rolling as of today.

@Ralm / @hiteshhapani did you guys ever submit a phabricator ticket? I couldn’t find one so went ahead and created this:

https://phabricator.vyos.net/T3619

@srnoth Can you enable some offloads?

sudo ethtool -K eth1 gro on
sudo ethtool -K eth1 sg on
sudo ethtool -K eth1 tso on
sudo ethtool -K eth1 lro on

sudo ethtool -K eth2 gro on
sudo ethtool -K eth2 sg on
sudo ethtool -K eth2 tso on
sudo ethtool -K eth2 lro on

Replied on the Phabricator ticket as well. Those settings got me almost back to the performance of 1.2.7.

1 Like

This is great news @snorth - Thanks so much for sticking this out and helping the team debug it.

1 Like

Setting GRO alone got my performance back to what I had in Vyos 1.2.
My NIC also supported SG and TSO, that further decreased CPU usage from interrupts significantly (around 40% less CPU usage), which is great.

These reset on reboot, how can I apply these permanently?

@srnoth I was going to, but since I didn’t have an phabricator account yet I delayed it and eventually forgot :frowning:
Thank you for opening the ticket.

This is what I use, though it’s on Vyos 1.2.7 but I expect the same works on 1.3

tim@ferrari# show interfaces ethernet eth0
 duplex auto
 mtu 9000
 offload-options {
     generic-receive on
     generic-segmentation on
     scatter-gather on
     tcp-segmentation on
 }

Thanks.

Just tried the equivalent on 1.3 and worked.
Leaving here the options for others:

vyos@vyos#  set interfaces ethernet eth1 offload
Possible completions:
   gro          Enable Generic Receive Offload
   gso          Enable Generic Segmentation Offload
   rps          Enable Receive Packet Steering
   sg           Enable Scatter-Gather
   tso          Enable TCP Segmentation Offloading
   ufo          Enable UDP Fragmentation Offloading
vyos@vyos# show interfaces ethernet eth1
 address dhcp
 duplex auto
 offload {
     gro
     gso
     sg
     tso
 }
 speed auto

Also, leaving a screenshot of my CPU usage now, doing a Speedtest and saturating my 1Gbit internet connection:

I know this is not the most scientific way to test, but you can see the massive difference, even from 1.2.
With 1.2 (screenshots on the first post), I had 255% CPU usage, now with 1.3 and the offloads enabled I have around 120%, which is freaking incredible. Much better then the 380% on 1.3 without offloads :slight_smile:

Looks like a success. Previously I had those options in my vyos-postconfig-bootup.script file.

Would you apply that to all ethernet interfaces (WAN and LAN) or only to the WAN interface?

As it’s for forwarding, I think it must be configured for both interfaces.

@Viacheslav Given the performance impact it seems to have, should these settings not be default on, with the option to disable them?

@tjh In theory in some cases, some of the offloads or their combinations can cause a kernel panic. So we need more tests on different platforms/nics.

To be honest, I would probably focus on the generic offloads more as they get the normal performance we were used to on 1.2.

The others I agree can be very nic and driver dependent in terms of stability.

1 Like

@Viacheslav Options suggested here solves ipsec performance issue but issue still remains for macsec.

Try to enable rps

set interfaces ethernet eth0 offload rps

@Viacheslav Not helping. Tried all the available offload options via vyos cli.

I can suggest that the CPU doesn’t have some accelerated crypto flags/instructions.