Vyos dont balance queues in 10GB interface

Hi,

I have an vyos in production with a card of 10Gb Intel DA520.
I see the System interrupt is very high in one processor, then I look the queues of interface:

sudo ethtool -S eth2 |grep rx_queue_[0-7]_packet
rx_queue_0_packets: 33242347639
rx_queue_1_packets: 0
rx_queue_2_packets: 0
rx_queue_3_packets: 46
rx_queue_4_packets: 0
rx_queue_5_packets: 31378
rx_queue_6_packets: 12541
rx_queue_7_packets: 0

I have only one queue that receive all packets in…
Can I change this on vyos?

Thanks,

Sidnei

hi you can take a look at this
https://wiki.vyos.net/wiki/Performance_Tuning

Hi, thank you by you return,

the cpus are distribute:

ls /sys/class/net/eth2/device/msi_irqs/
45 46 47 48 49 50 51 52 53

~ sudo cat /proc/irq/45/smp_affinity_list 0 ~ sudo cat /proc/irq/46/smp_affinity_list
2
~ sudo cat /proc/irq/47/smp_affinity_list 4 ~ sudo cat /proc/irq/48/smp_affinity_list
6
~ sudo cat /proc/irq/49/smp_affinity_list 1 ~ sudo cat /proc/irq/50/smp_affinity_list
3
~ sudo cat /proc/irq/51/smp_affinity_list 5 ~ sudo cat /proc/irq/52/smp_affinity_list
7
~$ sudo cat /proc/irq/53/smp_affinity_list
0

but the rx-0 - rx-7 don’t distribute.

sudo ethtool -S eth3|grep rx_queue_[0-7]_packet
rx_queue_0_packets: 108408595351
rx_queue_1_packets: 0
rx_queue_2_packets: 0
rx_queue_3_packets: 0
rx_queue_4_packets: 0
rx_queue_5_packets: 6268
rx_queue_6_packets: 0
rx_queue_7_packets: 0

The rx_queue_0 of de nic receive all packets, and my softirq is very high on the processor of rx_queue_0.

Sidnei

A “better config” would use RSS on the NIC , and distribute the IRQs over multiple cores, instead of relying on single core per NIC.

Never used it myself though

Hi @sidnei, what traffic type do you use L2 or L3? maybe pppoe or QinQ?

Hi,

I’m using L3 for bgp routing with QinQ.

Sidnei

ixgbe driver doesn’t work correctly with RSS+QinQ. Exist some ixgbe patches, but I think exist more easy way to solve this problem. Can you provide information about softirq?
sudo cat /proc/softirqs

Also, you can try some network tuning, but be careful. As root sudo su -l

echo "32768" > /proc/sys/net/core/rps_sock_flow_entries

echo "2" > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo "4" > /sys/class/net/eth0/queues/rx-1/rps_cpus
echo "8" > /sys/class/net/eth0/queues/rx-2/rps_cpus
echo "1" > /sys/class/net/eth0/queues/rx-3/rps_cpus

echo "2048" > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
echo "2048" > /sys/class/net/eth0/queues/rx-1/rps_flow_cnt
echo "2048" > /sys/class/net/eth0/queues/rx-2/rps_flow_cnt
echo "2048" > /sys/class/net/eth0/queues/rx-3/rps_flow_cnt

Thanks by return, my softirq:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
HI: 2 1 0 0 0 0 0 0
TIMER: 8592438 6365449 3060905 2839901 38447 566886 36826 951880
NET_TX: 25144 13 3127 24121 296174 242595 193622 140963
NET_RX: 8055998 24630 104072 11422 2813179380 2832554169 378273356 990486154
BLOCK: 71070 26 9 14 1 5 0 1
BLOCK_IOPOLL: 0 0 0 0 0 0 0 0
TASKLET: 101 10 13 11 0 0 0 1
SCHED: 8116868 5881578 2812912 2645547 35700 495376 36394 430474
HRTIMER: 31 24 19 21 17 42 15 63
RCU: 4704361 3348961 1651020 1393003 66591 2465440 65187 1135857

but at this time I’m using the setting:

smp_affinity 40,80 to interface eth3 and
smp_affinity 10,20 to interface eth2

to balance the interrupts…

but I was already suspicious of QinQ. I’ll try remove from eth3 to testing…

Sidnei

I think smp_affinity used only for RSS. For traffic type as QinQ or pppoe must use RPS,RFS,XPS.

Hello,

I was remove the QinQ from my eth3, and the queues began to receive packets.
See my rx queues now:
sudo ethtool -S eth3|grep rx_queue_[0-7]_packet
rx_queue_0_packets: 858159375
rx_queue_1_packets: 121687989
rx_queue_2_packets: 117217568
rx_queue_3_packets: 115071130
rx_queue_4_packets: 116948573
rx_queue_5_packets: 115479120
rx_queue_6_packets: 611730132
rx_queue_7_packets: 131488653

With QinQ active the queues from this interface were like this:
sudo ethtool -S eth3|grep rx_queue_[0-7]_packet
rx_queue_0_packets: 108408595351
rx_queue_1_packets: 0
rx_queue_2_packets: 0
rx_queue_3_packets: 0
rx_queue_4_packets: 0
rx_queue_5_packets: 6268
rx_queue_6_packets: 0
rx_queue_7_packets: 0

Now, I need know if have any way to distribute the rx queues in QinQ mode.

Sidnei

Hello, I wrote few messages earlier. You can trying manually enable RPS and RFS, XPS. If it will work correct, I think need find way to implement this feature to cli, like smp-affinity
Follow this example, but change correct interface name, I think you have eth3 which works with QinQ

echo "32768" > /proc/sys/net/core/rps_sock_flow_entries

echo "2" > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo "4" > /sys/class/net/eth0/queues/rx-1/rps_cpus
echo "8" > /sys/class/net/eth0/queues/rx-2/rps_cpus
echo "1" > /sys/class/net/eth0/queues/rx-3/rps_cpus

echo "2" > /sys/class/net/eth0/queues/tx-0/xps_cpus
echo "4" > /sys/class/net/eth0/queues/tx-1/xps_cpus
echo "8" > /sys/class/net/eth0/queues/tx-2/xps_cpus
echo "1" > /sys/class/net/eth0/queues/tx-3/xps_cpus

echo "2048" > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
echo "2048" > /sys/class/net/eth0/queues/rx-1/rps_flow_cnt
echo "2048" > /sys/class/net/eth0/queues/rx-2/rps_flow_cnt
echo "2048" > /sys/class/net/eth0/queues/rx-3/rps_flow_cnt

hi @Dmitry ,

I have the same problem, can you provide recommendations for manual configuration of RPS, RFS, and XPS with 18 cores 36 threads of cpu?

Hello @maimun.najib , try set proper bitmask for 0 queue
For 18 cores:
echo "3ffff" > /sys/class/net/eth0/queues/rx-0/rps_cpus
For 36 cores:
echo "fffffffff" > /sys/class/net/eth0/queues/rx-0/rps_cpus
Also if you have enabled HT, you need to disable it.

hi @Dmitry,

is it not recommended to activate HT? and whether for rx-1, rx-2 and so on need to be configured as well?

HT cores technically use the same real cores for IRQs.
Usually required to define bitmask only for rx-0 queue, to confirm this try to run sudo top and press 1