VyOS 1.3.0 NIC Interrupts

General questions
1

Hello,

I have installed VyOS 1.3.0 (self build) on my home router. The router is a Protectli FW1 with 4 Intel NIC’s and 4 core CPU. VyOS 1.2.X had SMP Affinity which would balance all 4 NIC interrupts evenly across all 4 CPU cores. With VyOS 1.3.0 this option is removed and the behavior does not seem consistent and now I end up with NIC interrupts sharing a CPU. Is anyone else experiencing this issue? Is there a new VyOS CLI method to balance the NIC interrupts and make them predictable/consistent? As you can see below my eth0 and eth1 are now sharing CPU2 with CPU1 having no NIC interrupt.

vyos@vyos:~$ cat /proc/interrupts | grep eth
  91:          0          0       3993          0   PCI-MSI 524288-edge      eth0
  94:          0          0       5012          0   PCI-MSI 1048576-edge      eth1
  95:          0          0          0        246   PCI-MSI 1572864-edge      eth2
  96:        246          0          0          0   PCI-MSI 2097152-edge      eth3
2

Sounds like it might be the same problem mentioned in this thread?
I don’t think any final solution/fix has been found, but read through and see if there’s anything there that helps you troubleshoot/fix it.

3

Hi @p252 could you please share interface info for eth0 and eth1:

run show interfaces ethernet eth0 physical 
run show interfaces ethernet eth1 physical 
sudo ethtool -l eth0
sudo ethtool -l eth1
1 Like
4

Hello,

Thank you for the reply. I have uploaded the interface info as text files as the output would have been quite long in the post. The NIC does not support multi queue so ethtool command just gave an error.

eth0.txt (1.6 KB)
eth1.txt (1.6 KB)

1 Like
5

Tried to manually set the nic interrupts via /config/scripts/vyos-postconfig-bootup.script but the IRQ numbers do not seem consistent across reboots. Very annoying. Is there a reason smp_affinity config commands were removed from VyOS 1.3?

6

As a workaround I have added the following to my vyos-postconfig-bootup.script. I am not great at scripting so not sure how safe or efficient it is but it seems to work for now.

ETH0_IRQ=$(grep eth0 /proc/interrupts | cut -c1-4 | tr -dc '0-9')
ETH1_IRQ=$(grep eth1 /proc/interrupts | cut -c1-4 | tr -dc '0-9')
ETH2_IRQ=$(grep eth2 /proc/interrupts | cut -c1-4 | tr -dc '0-9')
ETH3_IRQ=$(grep eth3 /proc/interrupts | cut -c1-4 | tr -dc '0-9')
echo 4 > /proc/irq/$ETH0_IRQ/smp_affinity
echo 2 > /proc/irq/$ETH1_IRQ/smp_affinity
echo 8 > /proc/irq/$ETH2_IRQ/smp_affinity
echo 1 > /proc/irq/$ETH3_IRQ/smp_affinity
7

Hi @p252 , script looks good. Also increase ring buffer to reach more performance.

configure
set interfaces ethernet eth0 ring-buffer rx 4096
set interfaces ethernet eth0 ring-buffer tx 4096 
set interfaces ethernet eth1 ring-buffer rx 4096
set interfaces ethernet eth1 ring-buffer tx 4096
...
commit
1 Like
8

Thanks for the input.