VPP Using Main Core over Workers

General questions
1

I currently have VyOS setup on a box and have tried to configure VPP following the commands specified in Performance notice | VyOS VPP Addon Documentation.

However, when the box goes under some load (in this case I generated ~140K packets per second), it seems to use the main core when, from my understanding, this shouldn’t be using that much CPU?

Could someone help me to see if my understanding is incorrect or if there’s something wrong with my config etc.

Configuration

set interfaces ethernet eth0 address ‘xxx.xxx.81.51/29’
set interfaces ethernet eth0 hw-id ‘xx:xx:xx:xx:xx:48’
set interfaces ethernet eth0 ring-buffer rx ‘8192’
set interfaces ethernet eth0 ring-buffer tx ‘8192’
set interfaces ethernet eth1 hw-id ‘xx:xx:xx:xx:xx:49’
set interfaces ethernet eth1 ring-buffer rx ‘8192’
set interfaces ethernet eth1 ring-buffer tx ‘8192’
set interfaces loopback lo
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.81.49
set service ssh
set system config-management commit-revisions ‘100’
set system console device ttyS0 speed ‘115200’
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system name-server ‘xxx.xxx.1.1’
set system name-server ‘xxx.xxx.9.9’
set system option kernel disable-mitigations
set system option kernel memory hugepage-size 2M hugepage-count ‘3072’
set system option reboot-on-upgrade-failure ‘5’
set system sysctl parameter net.core.rmem_default value ‘134217728’
set system sysctl parameter net.core.rmem_max value ‘536870912’
set system sysctl parameter net.core.wmem_default value ‘134217728’
set system sysctl parameter net.core.wmem_max value ‘536870912’
set system syslog local facility all level ‘info’
set system syslog local facility local7 level ‘debug’
set vpp settings cpu corelist-workers ‘0-12’
set vpp settings cpu main-core ‘14’
set vpp settings host-resources shmmax ‘6442450944’
set vpp settings interface eth0 dpdk-options num-rx-queues ‘1’
set vpp settings interface eth0 dpdk-options num-tx-queues ‘1’
set vpp settings interface eth0 driver ‘dpdk’
set vpp settings interface eth1 dpdk-options num-rx-queues ‘1’
set vpp settings interface eth1 dpdk-options num-tx-queues ‘1’
set vpp settings interface eth1 driver ‘dpdk’
set vpp settings lcp netlink rx-buffer-size ‘536870912’
set vpp settings memory main-heap-page-size ‘default-hugepage’
set vpp settings memory main-heap-size ‘4G’
set vpp settings statseg page-size ‘default-hugepage’
set vpp settings statseg size ‘256M’

Before generating the packet load (C1-13 are workers, C15 is main)

BeforeGeneratingALoad
BeforeGeneratingALoad492×266 9.15 KB

While generating the packet load (C1-13 are workers, C15 is main)

498×269 9.82 KB

2

  1. Do there show up some logs or dropped packets or other counters etc when you increase the load? Im thinking since all the “metadata” will still be handled by the main core.

  2. When you increase the load is it more packets for existing sessions or do you also increase amount of sessions? Im thinking since new sessions needs to be offloaded into a worker so the new session metadata will still hit the main core.

Other than that VPP is a kernel feature to offload things while DPDK is the stuff you really want for true hw offloading (by using dedicated cores).

Do you get the same result if you enable DPDK instead of VPP?

3

The only thing I really see increasing from vppctl runtime on the main core is:

1920×925 244 KB

Whereas one of the worker cores looks like this:
1629×763 190 KB

Vppctl’s logging is full of warnings “rte_eth_xstats_get vector size mismatch”

Increasing the load is from one of my internal boxes and sending from one source IP, random UDP source ports to 1 UDP destination port. Shouldn’t be much new session stuff.

DPDK should be enabled, using a supported mlx5_core NIC as well.

Any commands you need running/want output from let me know.

4

1209×275 81.1 KB

VPPCTL Node Counters, nothing concerning there

5

Your config is set to use the DPDK driver for VPP, the other reply seems to have some confusion between VPP and its drivers. The worker threads will always be at 100% by default unless you use an adaptive or interrupt polling mode that may or may not be supported by your driver (I don’t believe it is for DPDK). The most common cause i have seen for the main thread having abnormally high load is traffic being punted from the VPP dataplane to the linux kernel dataplane. Without knowing more about your testing setup and topology i can only speculate but my guess would be your testing to or from software running on the router itself which would cause traffic to go to the linux kernel dataplane from VPP or the route/destination for the traffic does not exist in the VPP dataplane and it is being punted to the kernel which i believe is the default behavior if you have a default route in the kernel table.

6

A session is defined by the 5-tuple meaning the combo of:

  • Protocol
  • Source IP-address
  • Destination IP-address
  • Source port
  • Destination port

Since you seems to be using random UDP source port this means the packet will be detected as a new session/flow meaning stuff will hit the main cores regarding metadata and to get offloaded into an available worker core. Which is probably why you see a large increase of main core usage during your tests.

I would redo the tests and make sure that the 5-tuple is NOT changed between packets and then observe the result regarding how VPP then will behave regarding main cores vs worker cores.