Help troubleshooting routing performance issue 1 way only

Tried setting Ring-Buffers to max (4096) and no change in transfer performance :frowning:

Also tried adding RPS back and also no change on the poor performance with RPS offload.

I’m still suffering from this issue.

Sharing more info regarding my config.
Please note that eth1 is not included as its being used for WAN only, even though its part of the same NIC.

Interrupts statistics
vyos@vyos:~$ cat /proc/interrupts | grep eth
 119:          1          0          0          0   PCI-MSI 1572864-edge      eth1
 120:          0    4756172          0          0   PCI-MSI 1572865-edge      eth1-rx-0
 121:          0          0   11153264          0   PCI-MSI 1572866-edge      eth1-rx-1
 122:          0          0          0    4765661   PCI-MSI 1572867-edge      eth1-rx-2
 123:   13479626          0          0          0   PCI-MSI 1572868-edge      eth1-rx-3
 124:          0    2466110          0          0   PCI-MSI 1572869-edge      eth1-tx-0
 125:          0          0    4103189          0   PCI-MSI 1572870-edge      eth1-tx-1
 126:          0          0          0    2346949   PCI-MSI 1572871-edge      eth1-tx-2
 127:    4619146          0          0          0   PCI-MSI 1572872-edge      eth1-tx-3
 128:          0          0          0          0   PCI-MSI 2621440-edge      eth0
 131:          0          0          0          2   PCI-MSI 1574912-edge      eth2
 132:          8          0     556778          0   PCI-MSI 1574913-edge      eth2-rx-0
 133:          0          5          0     245218   PCI-MSI 1574914-edge      eth2-rx-1
 134:     212749          0          5          0   PCI-MSI 1574915-edge      eth2-rx-2
 135:          0     246818          0          5   PCI-MSI 1574916-edge      eth2-rx-3
 136:          9          0   11362234          0   PCI-MSI 1574917-edge      eth2-tx-0
 137:          0          9          0   10907001   PCI-MSI 1574918-edge      eth2-tx-1
 138:   12660048          0          5          0   PCI-MSI 1574919-edge      eth2-tx-2
 139:          0   10844228          0          5   PCI-MSI 1574920-edge      eth2-tx-3
 140:          0          2          0          0   PCI-MSI 2097152-edge      eth3
 141:          0          0     223597          0   PCI-MSI 2097153-edge      eth3-rx-0
 142:          0          0          0     199833   PCI-MSI 2097154-edge      eth3-rx-1
 143:     185573          0          0          0   PCI-MSI 2097155-edge      eth3-rx-2
 144:          0     184136          0          0   PCI-MSI 2097156-edge      eth3-rx-3
 145:          0          0     161055          0   PCI-MSI 2097157-edge      eth3-tx-0
 146:          0          0          0     160949   PCI-MSI 2097158-edge      eth3-tx-1
 147:     184269          0          0          0   PCI-MSI 2097159-edge      eth3-tx-2
 148:          0     162101          0          0   PCI-MSI 2097160-edge      eth3-tx-3
 149:          0          0          2          0   PCI-MSI 2099200-edge      eth4
 150:          0          0    2557205         12   PCI-MSI 2099201-edge      eth4-rx-0
 151:          6          0          0    5424251   PCI-MSI 2099202-edge      eth4-rx-1
 152:    3456736          6          0          0   PCI-MSI 2099203-edge      eth4-rx-2
 153:          0    4529822          6          0   PCI-MSI 2099204-edge      eth4-rx-3
 154:          0          0     443412          9   PCI-MSI 2099205-edge      eth4-tx-0
 155:          6          0          0     325663   PCI-MSI 2099206-edge      eth4-tx-1
 156:     441623         11          0          0   PCI-MSI 2099207-edge      eth4-tx-2
 157:          0     472967          6          0   PCI-MSI 2099208-edge      eth4-tx-3
ethtool -l
vyos@vyos:~$ sudo ethtool -l eth2
Channel parameters for eth2:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       4

vyos@vyos:~$ sudo ethtool -l eth3
Channel parameters for eth3:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       4

vyos@vyos:~$ sudo ethtool -l eth4
Channel parameters for eth4:
Pre-set maximums:
RX:             0
TX:             0
Other:          1
Combined:       8
Current hardware settings:
RX:             0
TX:             0
Other:          1
Combined:       4
interfaces ethernet ethX physical
vyos@vyos:~$ show interfaces ethernet eth2 physical
Settings for eth2:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
Ring parameters for eth2:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512

driver: igb
version: 5.6.0-k
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
vyos@vyos:~$ show interfaces ethernet eth3 physical
Settings for eth3:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off (auto)
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
Ring parameters for eth3:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512

driver: igb
version: 5.6.0-k
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
vyos@vyos:~$ show interfaces ethernet eth4 physical
Settings for eth4:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: on (auto)
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
Ring parameters for eth4:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512

driver: igb
version: 5.6.0-k
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Any idea would be greatly appreciated.

Hi @Ralm did you try to run iperf between each device with multiple threads?
And one more question, do you have configured firewall?
Also, please check scaling_governor policies

find / -name scaling_governor

Hi @Dmitry,
I can try run with multiple threads, however since I max out a 1Gbit line with the desktop machine acting as a iperf3 as a server, didn’t consider to a client or server limitation.
Would iPerf running in multiple threads explain why Vyos is using only 1 thread (with CPU maxed) when a client on Vlan 50 does a iPerf to a server on Vlan 1 ?
The opposite doesn’t happen, there are multiple threads used.

Regarding the firewall, yes its enabled, but on the LAGG it set on Local and I only have a 2 rules regarding access to Vyos itself:

I would expect the firewall to affect both directions of traffic and not only 1.

Lastly, about the scaling_governor policies, they are all set to “powersave”, I don’t know what that means :frowning:

scaling_governor policies screenshots

image

@Dmitry adding iPerf3 tests 1 stream vs 4 streams.

1 Stream

4 Parallel streams


Vyos Top during iPerf tests:

1 Stream

4 Streams

Very interesting that for some reason it will split the interrupts over 4 cores with 4 streams with Client on Vlan 50 and Server on Vlan 1.
But then, even with 1 Stream doing the opposite, it will split automatically.

Still, the CPU usage with the server on Vlan 1, is really really high.

For reference, here is Vyos top when doing server on Vlan 50 and client on Vlan 1 with 1 Stream:

Hope it helps understanding what might be going on here.
I’m pretty sure I didn’t have this issue with 1.2, but atm I’m not able to do the test of starting the 1.2 image to test it unfortunately.

Hi @Ralm , try to set performance to scaling_governor

sudo su -l
echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy2/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy3/scaling_governor

Also I see that RPS did not applied properly, try to set it manually

echo "f" > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-3/rps_cpus

echo "f" > /sys/class/net/eth2/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-3/rps_cpus

echo "f" > /sys/class/net/eth3/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-3/rps_cpus

echo "f" > /sys/class/net/eth4/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-3/rps_cpus

Thank you for the suggestion @Dmitry,

When trying to set RPS manually I’m getting Access Denied.
image

I also double checked that when I enable RPS in the Vyos config, these stay at 0.

Lastly, after I set these, do I need to restart Vyos?
I managed to set it with nano (even though I was getting errors everywhere, I think it overwritten the file), but after restarting, it reverted to 0.

Why you did not do this under sudo su -l?

sudo su -l
echo "f" > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-3/rps_cpus

echo "f" > /sys/class/net/eth2/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-3/rps_cpus

echo "f" > /sys/class/net/eth3/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-3/rps_cpus

echo "f" > /sys/class/net/eth4/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-3/rps_cpus

To apply these changes after restart you have to use post-boot scripts Command Scripting — VyOS 1.3.x (equuleus) documentation

Adding some more info.
Tested using UDP stream, the performance issue is the same.

Vyos:


Ok, running in sudo context worked, I only tried with running the command in sudo.

The values updated, but I don’t see any difference.

Something strange that I’ve been noticing, even before activating RPS, is that the performance seems to vary greatly when doing tests back to back.

Looks like you did not apply RPS, provide please sudo cat /sys/class/net/eth*/queues/rx-*/rps_cpus
and also cat /proc/interrupts

The values did, however not sure if they are taking any effect.

Please note I didn’t apply on eth1 as its being used for WAN exclusively.

sudo cat /sys/class/net/eth*/queues/rx-*/rps_cpus

cat /proc/interrupts

@Ralm try to increase and combine queues

sudo ethtool -L eth2 combined 4
sudo ethtool -L eth3 combined 4
sudo ethtool -L eth4 combined 4

And show cat /proc/interrupts again

It was already set to 4 combined.

What about this? The previous output tell that no

I didn’t add it because I didn’t change anything.

But here you have once more:

cat /proc/interrupts

Look, you have 8 splitted interrupts for RX and TX
e.g. eth4-rx-0…eth4-tx-0… Usually, when it is configured combined, it should be eth4-TxRx-0. But maybe it depends on NIC HW and driver.

Interesting fact, I see this a bit strange situation too

114:    1106604          0          0          0  IR-PCI-MSI 2627585-edge      eth8-rx-0
115:          0    1106604          0          0  IR-PCI-MSI 2627586-edge      eth8-rx-1
116:          0          0    1106604          0  IR-PCI-MSI 2627587-edge      eth8-rx-2
117:          0          0          0    1106604  IR-PCI-MSI 2627588-edge      eth8-rx-3
118:    1106604          0          0          0  IR-PCI-MSI 2627589-edge      eth8-tx-0
119:          0    1106604          0          0  IR-PCI-MSI 2627590-edge      eth8-tx-1
120:          0          0    1106604          0  IR-PCI-MSI 2627591-edge      eth8-tx-2
121:          0          0          0    1106604  IR-PCI-MSI 2627592-edge      eth8-tx-3
122:  379998572          0          0          0  IR-PCI-MSI 4196352-edge      eth10-TxRx-0
123:          0  431702291          0          0  IR-PCI-MSI 4196353-edge      eth10-TxRx-1
124:          0          0  406990121          0  IR-PCI-MSI 4196354-edge      eth10-TxRx-2
125:          0          0          0  390113523  IR-PCI-MSI 4196355-edge      eth10-TxRx-3

eth8 - igb
eth10 - ixgbe

ethtool -l eth8
Channel parameters for eth8:
Pre-set maximums:
RX:		0
TX:		0
Other:		1
Combined:	8
Current hardware settings:
RX:		0
TX:		0
Other:		1
Combined:	4

ethtool -l eth10
Channel parameters for eth10:
Pre-set maximums:
RX:		0
TX:		0
Other:		1
Combined:	63
Current hardware settings:
RX:		0
TX:		0
Other:		1
Combined:	4

Is there any command of process for me to be able to validate if traffic is being offloaded or not?
I would like to test and see if traffic is being offloaded in specific traffic direction vs the other and if not why.

Thanks

Usually, you can see this information with top and press 1
I guess this issue can be potentially related to the LACP. Could you try to configure RR (round-robin) mode for bonding interfaces and test it.
I’m not sure that for LACP hash-policy can help you.