Tried setting Ring-Buffers to max (4096) and no change in transfer performance 
Also tried adding RPS back and also no change on the poor performance with RPS offload.
I’m still suffering from this issue.
Sharing more info regarding my config.
Please note that eth1 is not included as its being used for WAN only, even though its part of the same NIC.
Interrupts statistics
vyos@vyos:~$ cat /proc/interrupts | grep eth
119: 1 0 0 0 PCI-MSI 1572864-edge eth1
120: 0 4756172 0 0 PCI-MSI 1572865-edge eth1-rx-0
121: 0 0 11153264 0 PCI-MSI 1572866-edge eth1-rx-1
122: 0 0 0 4765661 PCI-MSI 1572867-edge eth1-rx-2
123: 13479626 0 0 0 PCI-MSI 1572868-edge eth1-rx-3
124: 0 2466110 0 0 PCI-MSI 1572869-edge eth1-tx-0
125: 0 0 4103189 0 PCI-MSI 1572870-edge eth1-tx-1
126: 0 0 0 2346949 PCI-MSI 1572871-edge eth1-tx-2
127: 4619146 0 0 0 PCI-MSI 1572872-edge eth1-tx-3
128: 0 0 0 0 PCI-MSI 2621440-edge eth0
131: 0 0 0 2 PCI-MSI 1574912-edge eth2
132: 8 0 556778 0 PCI-MSI 1574913-edge eth2-rx-0
133: 0 5 0 245218 PCI-MSI 1574914-edge eth2-rx-1
134: 212749 0 5 0 PCI-MSI 1574915-edge eth2-rx-2
135: 0 246818 0 5 PCI-MSI 1574916-edge eth2-rx-3
136: 9 0 11362234 0 PCI-MSI 1574917-edge eth2-tx-0
137: 0 9 0 10907001 PCI-MSI 1574918-edge eth2-tx-1
138: 12660048 0 5 0 PCI-MSI 1574919-edge eth2-tx-2
139: 0 10844228 0 5 PCI-MSI 1574920-edge eth2-tx-3
140: 0 2 0 0 PCI-MSI 2097152-edge eth3
141: 0 0 223597 0 PCI-MSI 2097153-edge eth3-rx-0
142: 0 0 0 199833 PCI-MSI 2097154-edge eth3-rx-1
143: 185573 0 0 0 PCI-MSI 2097155-edge eth3-rx-2
144: 0 184136 0 0 PCI-MSI 2097156-edge eth3-rx-3
145: 0 0 161055 0 PCI-MSI 2097157-edge eth3-tx-0
146: 0 0 0 160949 PCI-MSI 2097158-edge eth3-tx-1
147: 184269 0 0 0 PCI-MSI 2097159-edge eth3-tx-2
148: 0 162101 0 0 PCI-MSI 2097160-edge eth3-tx-3
149: 0 0 2 0 PCI-MSI 2099200-edge eth4
150: 0 0 2557205 12 PCI-MSI 2099201-edge eth4-rx-0
151: 6 0 0 5424251 PCI-MSI 2099202-edge eth4-rx-1
152: 3456736 6 0 0 PCI-MSI 2099203-edge eth4-rx-2
153: 0 4529822 6 0 PCI-MSI 2099204-edge eth4-rx-3
154: 0 0 443412 9 PCI-MSI 2099205-edge eth4-tx-0
155: 6 0 0 325663 PCI-MSI 2099206-edge eth4-tx-1
156: 441623 11 0 0 PCI-MSI 2099207-edge eth4-tx-2
157: 0 472967 6 0 PCI-MSI 2099208-edge eth4-tx-3
ethtool -l
vyos@vyos:~$ sudo ethtool -l eth2
Channel parameters for eth2:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
vyos@vyos:~$ sudo ethtool -l eth3
Channel parameters for eth3:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
vyos@vyos:~$ sudo ethtool -l eth4
Channel parameters for eth4:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
interfaces ethernet ethX physical
vyos@vyos:~$ show interfaces ethernet eth2 physical
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Ring parameters for eth2:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512
driver: igb
version: 5.6.0-k
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
vyos@vyos:~$ show interfaces ethernet eth3 physical
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Ring parameters for eth3:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512
driver: igb
version: 5.6.0-k
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
vyos@vyos:~$ show interfaces ethernet eth4 physical
Settings for eth4:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Ring parameters for eth4:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512
driver: igb
version: 5.6.0-k
firmware-version: 1.2.1
expansion-rom-version:
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Any idea would be greatly appreciated.
Hi @Ralm did you try to run iperf between each device with multiple threads?
And one more question, do you have configured firewall?
Also, please check scaling_governor policies
find / -name scaling_governor
Hi @Dmitry,
I can try run with multiple threads, however since I max out a 1Gbit line with the desktop machine acting as a iperf3 as a server, didn’t consider to a client or server limitation.
Would iPerf running in multiple threads explain why Vyos is using only 1 thread (with CPU maxed) when a client on Vlan 50 does a iPerf to a server on Vlan 1 ?
The opposite doesn’t happen, there are multiple threads used.
Regarding the firewall, yes its enabled, but on the LAGG it set on Local and I only have a 2 rules regarding access to Vyos itself:
I would expect the firewall to affect both directions of traffic and not only 1.
Lastly, about the scaling_governor policies, they are all set to “powersave”, I don’t know what that means
@Dmitry adding iPerf3 tests 1 stream vs 4 streams.
Vyos Top during iPerf tests:
Very interesting that for some reason it will split the interrupts over 4 cores with 4 streams with Client on Vlan 50 and Server on Vlan 1.
But then, even with 1 Stream doing the opposite, it will split automatically.
Still, the CPU usage with the server on Vlan 1, is really really high.
For reference, here is Vyos top when doing server on Vlan 50 and client on Vlan 1 with 1 Stream:
Hope it helps understanding what might be going on here.
I’m pretty sure I didn’t have this issue with 1.2, but atm I’m not able to do the test of starting the 1.2 image to test it unfortunately.
Hi @Ralm , try to set performance to scaling_governor
sudo su -l
echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy2/scaling_governor
echo performance > /sys/devices/system/cpu/cpufreq/policy3/scaling_governor
Also I see that RPS did not applied properly, try to set it manually
echo "f" > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-3/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-3/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-3/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-3/rps_cpus
Thank you for the suggestion @Dmitry,
When trying to set RPS manually I’m getting Access Denied.
I also double checked that when I enable RPS in the Vyos config, these stay at 0.
Lastly, after I set these, do I need to restart Vyos?
I managed to set it with nano (even though I was getting errors everywhere, I think it overwritten the file), but after restarting, it reverted to 0.
Why you did not do this under sudo su -l?
sudo su -l
echo "f" > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-3/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-3/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth3/queues/rx-3/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-1/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-2/rps_cpus
echo "f" > /sys/class/net/eth4/queues/rx-3/rps_cpus
To apply these changes after restart you have to use post-boot scripts Command Scripting — VyOS 1.3.x (equuleus) documentation
1 Like
Ok, running in sudo context worked, I only tried with running the command in sudo.
The values updated, but I don’t see any difference.
Something strange that I’ve been noticing, even before activating RPS, is that the performance seems to vary greatly when doing tests back to back.
Looks like you did not apply RPS, provide please sudo cat /sys/class/net/eth*/queues/rx-*/rps_cpus
and also cat /proc/interrupts
The values did, however not sure if they are taking any effect.
Please note I didn’t apply on eth1 as its being used for WAN exclusively.
@Ralm try to increase and combine queues
sudo ethtool -L eth2 combined 4
sudo ethtool -L eth3 combined 4
sudo ethtool -L eth4 combined 4
And show cat /proc/interrupts again
What about this? The previous output tell that no
Look, you have 8 splitted interrupts for RX and TX
e.g. eth4-rx-0…eth4-tx-0… Usually, when it is configured combined, it should be eth4-TxRx-0. But maybe it depends on NIC HW and driver.
Interesting fact, I see this a bit strange situation too
114: 1106604 0 0 0 IR-PCI-MSI 2627585-edge eth8-rx-0
115: 0 1106604 0 0 IR-PCI-MSI 2627586-edge eth8-rx-1
116: 0 0 1106604 0 IR-PCI-MSI 2627587-edge eth8-rx-2
117: 0 0 0 1106604 IR-PCI-MSI 2627588-edge eth8-rx-3
118: 1106604 0 0 0 IR-PCI-MSI 2627589-edge eth8-tx-0
119: 0 1106604 0 0 IR-PCI-MSI 2627590-edge eth8-tx-1
120: 0 0 1106604 0 IR-PCI-MSI 2627591-edge eth8-tx-2
121: 0 0 0 1106604 IR-PCI-MSI 2627592-edge eth8-tx-3
122: 379998572 0 0 0 IR-PCI-MSI 4196352-edge eth10-TxRx-0
123: 0 431702291 0 0 IR-PCI-MSI 4196353-edge eth10-TxRx-1
124: 0 0 406990121 0 IR-PCI-MSI 4196354-edge eth10-TxRx-2
125: 0 0 0 390113523 IR-PCI-MSI 4196355-edge eth10-TxRx-3
eth8 - igb
eth10 - ixgbe
ethtool -l eth8
Channel parameters for eth8:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
ethtool -l eth10
Channel parameters for eth10:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 63
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
Is there any command of process for me to be able to validate if traffic is being offloaded or not?
I would like to test and see if traffic is being offloaded in specific traffic direction vs the other and if not why.
Thanks
Usually, you can see this information with top and press 1
I guess this issue can be potentially related to the LACP. Could you try to configure RR (round-robin) mode for bonding interfaces and test it.
I’m not sure that for LACP hash-policy can help you.