hello team,
I was really happy that Vyos currently supports dpdk over vpp so I gave it a try. but i have problem when i add 10G interface to VPP. here i attach my vyos logs vyos vpp log.txt (3.5 KB). Previously, I tried it on the 1G interface, the results could be committed, but when I tried it on the 10G interface, it couldn’t be committed. and when it fails to commit the interface is not immediately detected on the system.
vyos@vyos:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
bond0 - u/u
bond0.1183 112.xx.xx.xx/30 u/u
eth0 - A/D
eth1 - A/D
eth2 - A/D
eth3 - A/D
eth8 - u/u
eth9 - u/u
eth10 - u/u
eth11 - u/u
lo 127.0.0.1/8 u/u
::1/128
as additional information I use Intel I350 Network card for 1G NIC and Intel X722 and XL710 (4x10G mode) for 10G NIC
any solution?
Possible for you to paste the vpp configuration along with interface configuration?
VPP in VyOS is selective so you must enable VPP per interface if you want to fully utilize the VPP features (performance).
this is my vyos vpp config with 1G network interface
current vyos vpp config 1G.txt (3.6 KB)
but when I add the following config vyos can’t be committed
set vpp interface eth4 num-rx-desc '256'
set vpp interface eth4 num-rx-queues '512'
set vpp interface eth4 num-tx-desc '256'
set vpp interface eth4 num-tx-queues '512'
set vpp interface eth4 pci '0000:b7:00.0'
set vpp interface eth4 rx-mode 'polling'
set vpp interface eth5 num-rx-desc '256'
set vpp interface eth5 num-rx-queues '512'
set vpp interface eth5 num-tx-desc '256'
set vpp interface eth5 num-tx-queues '512'
set vpp interface eth5 pci '0000:b7:00.1'
set vpp interface eth5 rx-mode 'polling'
set vpp interface eth6 num-rx-desc '256'
set vpp interface eth6 num-rx-queues '512'
set vpp interface eth6 num-tx-desc '256'
set vpp interface eth6 num-tx-queues '512'
set vpp interface eth6 pci '0000:b7:00.2'
set vpp interface eth6 rx-mode 'polling'
set vpp interface eth7 num-rx-desc '256'
set vpp interface eth7 num-rx-queues '512'
set vpp interface eth7 num-tx-desc '256'
set vpp interface eth7 num-tx-queues '512'
set vpp interface eth7 pci '0000:b7:00.3'
set vpp interface eth7 rx-mode 'polling'
vyos@vyos:~$ lspci | grep Ethernet
17:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) (eth8)
17:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) (eth9)
17:00.2 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) (eth10)
17:00.3 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) (eth11)
66:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) (eth0)
66:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) (eth1)
66:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) (eth2)
66:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) (eth3)
b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04) (eth4)
b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04) (eth5)
b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04) (eth6)
b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04) (eth7)
vyos@vyos:~$ show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
bond0 - u/u
bond0.1183 112.xx.xx.xx/30 u/u
eth0 100.64.69.1/24 u/u (already add vpp)
100.64.0.1/16
eth1 - u/D (already add vpp)
eth2 - u/D (already add vpp)
eth3 - u/D (already add vpp)
eth4 - u/D (failed bind vpp interface)
eth5 - u/D (failed bind vpp interface)
eth6 - u/D (failed bind vpp interface)
eth7 - u/D (failed bind vpp interface)
eth8 - u/u (bonding member)
eth9 - u/u (bonding member)
eth10 - u/u (bonding member)
eth11 - u/u (bonding member)
lo 127.0.0.1/8 u/u
::1/128
As debugging verify that you dont have any offloading or ring-buffers configured for the interfaces (I got errors when I attempted VPP in a VM environment the other day)?
Does it matter if you remove interfaces from the bond before attempting to add them to vpp?
Verify that the pci numbers are correct for each interface?
Also your config of:
set vpp cpu corelist-workers '2'
set vpp cpu main-core '1'
looks a bit odd to me. Im not sure if the first core should be named ‘0’ or '1".
That is a 4-core system would do something like:
set vpp cpu corelist-workers '1-3'
set vpp cpu main-core '0'
but again I can be wrong on this.
I don’t think it’s due to the number of “worker cores” or “main core”. and it has nothing to do with the interface I’m bonding to because I purposely didn’t bind it to the vpp. I have also tried changing the number of “worker cores” and “main cores” but it’s the same
For the PCI ID itself, I have confirmed that it is correct on each interface.
root@vyos:~# dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:66:00.0 'I350 Gigabit Network Connection 1521' drv=uio_pci_generic unused=igb
0000:66:00.1 'I350 Gigabit Network Connection 1521' drv=uio_pci_generic unused=igb
0000:66:00.2 'I350 Gigabit Network Connection 1521' drv=uio_pci_generic unused=igb
0000:66:00.3 'I350 Gigabit Network Connection 1521' drv=uio_pci_generic unused=igb
Network devices using kernel driver
===================================
0000:17:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' if=eth8 drv=i40e unused=uio_pci_generic
0000:17:00.1 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' if=eth9 drv=i40e unused=uio_pci_generic
0000:17:00.2 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' if=eth10 drv=i40e unused=uio_pci_generic
0000:17:00.3 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' if=eth11 drv=i40e unused=uio_pci_generic
0000:b7:00.0 'Ethernet Connection X722 for 10GBASE-T 37d2' if=eth4 drv=i40e unused=uio_pci_generic
0000:b7:00.1 'Ethernet Connection X722 for 10GBASE-T 37d2' if=eth5 drv=i40e unused=uio_pci_generic
0000:b7:00.2 'Ethernet Connection X722 for 10GbE SFP+ 37d0' if=eth6 drv=i40e unused=uio_pci_generic
0000:b7:00.3 'Ethernet Connection X722 for 10GbE SFP+ 37d0' if=eth7 drv=i40e unused=uio_pci_generic
Question is if this isnt some driver compatability issue?
Over at 27. I40E Poll Mode Driver — Data Plane Development Kit 23.07.0-rc4 documentation they talk about “i40e PMD (librte_net_i40e)” as the driver to be used with DPDK along with specific firmwares for the card itself?
Another thing is if this can be workedaround using ethtool or modprobe to set various flags or not?
Like the content of /etc/modprobe.d/