I’m doing some performance testing with Minisforum MS-A2 and MS-01 mini-PCs, and I’m hitting kind of odd throughput problems that feel like something that should be fixable with different tuning options.
For context, I have a ConnectX-5 in the VyOS system and I’m running VyOS Stream 1.5-2025-q2. I’m using TRex to pump traffic through it with a variety of packet sizes from 64 through 1500 bytes.
In general, with the MS-A2 I’m hitting a wall around 16 Mpps and ~85 Gbps. Those are decent numbers, but it’s pretty clear that the CPU isn’t the limit here – I’ve seen test runs just a few percent slower than the max running with not much over 50% CPU utilization, and then increasing the traffic just slightly slams the system to 100% load and starts dropping packets. The system should have enough bus bandwidth for more traffic (x8 PCIe 4.0, so ~110 Gbps should be possible), and I doubt I’m running out of memory bandwidth. It feels like I’m saturating something and the kernel or driver is stalling waiting for buffer space/locks/etc and burning a bunch of CPU in the process.
Right now I’m running with a mostly-default VyOS config, with just a couple performance settings tweaked:
set firewall flowtable default offload software # and associated firewall rules to enable
set system option kernel disable-mitigations
set system performance network-throughput
I tried copying some of the rmem/wmem sysctl settings from the VyOS performance doc, but they cost me 5-10% performance over the default. And I’ve tried monkeying with BIOS power settings; most of them boost the clock speed and cut sustained performance. Just about everything else has been a wash, performance-wise.
I’m planning on testing with nosmt and playing with a few of the MLX5 PCI settings (cqe?), but that rapidly turns into a swamp with a million options, very few of which are obviously useful.
Are there any particular kernel/driver/ethtool settings that I should test here that have a reasonable chance of improving performance under load? I’m not looking for answers, just for reasonable things to add to my experiment.