Jittery on bare metal with full table BGP

[jcre]

Hi,

We’re running VyOS on Dell VEP4600 routers (Xeon-D 16 x 2Ghz Cores, 64Gb RAM). There’s a mix of X722 and X710 NICs (we’ve tried other vendors with the same issues).

We ran this for a while with no issues, the issues seem to have started when we connected the devices to peering exchanges with large amounts of connected devices. The issue is that on our latency graphs we see jitter spikes of about +50ms sometimes 1 packet every few minutes, sometimes fine for a few hours. Different hosts behind the VyOS boxes are affected at different times, and other hosts aren’t affected at all. It all seems to be luck of the draw.

We’ve got everything connected to X710 NICs for now to minimise what we have to focus on. All our interfaces are configured as below

 description "Core: Peering LAN XYZ"
 hw-id e4:43:4b:FF:FF:FF
 offload {
     gro
     gso
     hw-tc-offload
     rfs
     rps
     sg
     tso
 }
 ring-buffer {
     rx 4096
     tx 4096
 }
 vif 100 {
     address 1.2.3.4/22
     address fa80::1/64
 }

We’ve tried with the box in performance and in latency mode. SSH/etc are listening on none public IPs so there should be no bot traffic.

We’ve used the Intel irq affinity script (with the -x flag) to put the interfaces on cores 8-15 whilst using taskset to put bgpd and zebra on their own cores. Doing all of this reduced the peak latency but not really the frequency. We’ve tried giving the interfaces either dedicated cores, or multiple shared cores

If I do a file transfer through the box to increase traffic levels, I see interupts appearing 3 cores typically, the in and out interfaces, and a third random core also gets an interupt I believe (this is based on the “si” column in top). I’m thinking that it’s these interupts that seem to be sat on any core that are causing the jitter, perhaps if they hit at the same time as a bgp update is being processed it causes the packets to get delayed?

I’ve been trying to fix this off and on for a few months now, and welcome any insight. Load average is 0.00, we aren’t taxing the hardware. The links aren’t saturated either (under 10% utilisation).

Thanks!

[rgrant]

I had a similar “jitter spike” problem on different hardware - AMD Rome on Mellanox ConnectX-6 - and finally think we’ve resolved it by turning off AMD SMT (or Intel Hyperthreading).

Most tuning examples recommend turning it off but I’ve never thought it made that much difference - but when it does, it really does.

If you haven’t done it already, that is…

[tjh]

I agree with @rgrant. Additional things to try if that doesn’t help - try disabling those offloads.
And also then try forcing the nmi_watchdog to be disabled on the kernel boot. You’ll have to edit

/etc/grub/grub.cfg.d/vyos-versions/<your version>.cfg and where the line:

set boot_opts="boot=live rootdelay=5 noautologin net.ifnames=0 biosdevname=0 vyos-union=/boot/1.4.0 mitigations=off"

is, add

nmi_watchdog=0

and reboot - confirm it worked by looking at the output of

sudo dmesg | grep "Kernel command"

and ensuring that nmi_watchdog=0 is part of the Kernel command line.

Note that won’t stick across new versions of Vyos, you’ll have to manually add it each time.

[jcre]

Thanks for the replies, hyperthreading is indeed already off.

I’ll disable the offloads and see what happens, and schedule a time to disable nmi_watchdog. It’s not something I’ve come across in my research into what the problem might be so I’m hopeful we’ll see a difference

[jcre]

Thanks for your help, I assume you meant /boot/grub/grub.cfg.d/vyos-versions/*.cfg? As I don’t have those files in /etc

When I add nmi_watchdog=0 it doesn’t seem to be there after reboot, nor is it present when I grep dmesg.

I upgraded from a 1.4 rc to 1.5 rolling this morning so we’ll see if this has any effect. I also disabled mitigations / disabled power saving as part of that.

[jcre]

I’ve managed to prove that if I drop down to bare minimum routes (default route from transit + filter out peer routes to only the ones I need) the jitter goes away.

So my theory is it’s somehow related to the routes being installed into the kernel, as opposed to an issue with the network cards

[talmakion]

Do you have a few bilateral peering sessions (beyond the IX route servers) and/or are any of the prefixes coming out of BGP in ECMP?

If some problem prefixes are multi-pathed, you could try introducing some local-prefs to break the ECMP and stabilise routing, see if that makes a difference. There could be configuration/capacity differences on the other side that you cannot see or control and that would narrow it down a little.

There may also be other routing problems in play, like a direct peer intermittently (re-)announcing bad prefixes or other routing protocols leaking across the IX fabric if they’re not properly filtered. Traffic may not necessarily end up dropped but it may loop off in strange directions with that kind of misconfiguration.

[jcre]

Thanks for replying, things seemed to get better when I started to disable our full table IPv6 feeds and peers. We have route server and bilateral sessions on the IXs.

We have jitter between us and bi-lateral peers, us and route-server peers, us and transit providers, us and us (routes through VyOS), even pinging a loopback address

When I stopped accepting full tables and went to defaults, and trimmed down what we accepted from the route servers but kept the bilateral peers up all the jitter vanished.We run transits at 100, peers (bilateral or route server) all at 150, we don’t have multipath enabled

It’s definitely an odd problem, and I am seeing similar behaviour on other people I know run VyOS - but I think it’s people not noticing, we monitor everything at 1 second intervals.