Tuning for better performance? Especially with Mellanox

I’m doing some performance testing with Minisforum MS-A2 and MS-01 mini-PCs, and I’m hitting kind of odd throughput problems that feel like something that should be fixable with different tuning options.

For context, I have a ConnectX-5 in the VyOS system and I’m running VyOS Stream 1.5-2025-q2. I’m using TRex to pump traffic through it with a variety of packet sizes from 64 through 1500 bytes.

In general, with the MS-A2 I’m hitting a wall around 16 Mpps and ~85 Gbps. Those are decent numbers, but it’s pretty clear that the CPU isn’t the limit here – I’ve seen test runs just a few percent slower than the max running with not much over 50% CPU utilization, and then increasing the traffic just slightly slams the system to 100% load and starts dropping packets. The system should have enough bus bandwidth for more traffic (x8 PCIe 4.0, so ~110 Gbps should be possible), and I doubt I’m running out of memory bandwidth. It feels like I’m saturating something and the kernel or driver is stalling waiting for buffer space/locks/etc and burning a bunch of CPU in the process.

Right now I’m running with a mostly-default VyOS config, with just a couple performance settings tweaked:

set firewall flowtable default offload software  # and associated firewall rules to enable
set system option kernel disable-mitigations
set system performance network-throughput

I tried copying some of the rmem/wmem sysctl settings from the VyOS performance doc, but they cost me 5-10% performance over the default. And I’ve tried monkeying with BIOS power settings; most of them boost the clock speed and cut sustained performance. Just about everything else has been a wash, performance-wise.

I’m planning on testing with nosmt and playing with a few of the MLX5 PCI settings (cqe?), but that rapidly turns into a swamp with a million options, very few of which are obviously useful.

Are there any particular kernel/driver/ethtool settings that I should test here that have a reasonable chance of improving performance under load? I’m not looking for answers, just for reasonable things to add to my experiment.

1 Like

I would read this from start to finish. Some of these settings can be set in VyOS itself, some you might have to manually ethtool yourself.

Be keen to hear if any of these help.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.