Hey, folks. Which hardware specifications would you indicate to be capable of routing traffic between 8x 100G interfaces with linerate speed for small 64 byte packets? (Roughly 1.2 Bpps with VyOS + VPP) ?
I would go for:
- AMD EPYC rather than Intel for both performance but also fewer security vulnerabilities. For each fix in Intel world their performance compared to launchdate decreases. Phoronix have identified up to ~20% performance loss compared to launchdate for some Intel CPU-models and tasks.
- Fewer but faster cores rather than more but slower for a given TDP - also check baseclock vs turboclock. For example 9575F have 64C/128T 3.3/5.0GHz 256MB 400W while 9475F use 48C/96T 3.65/4.8GHz 256MB 400W. So the 9475F might be a better choice out of these two. On the other hand 9175F seems to have the highest baseclock with 16C/32T 4.2/5.0 GHz 512MB 320W. Reason for fewer but faster cores is mainly because you are often locked to the performance of a single core to push packets through a single NIC. With VPP/DPDK each flow is limited to the performance of a single core (easiest way to avoid packets out-of-order when forwarded).
- Newer generations of CPUs will of course improve IPC and whatelse so per clockcycle they will be able to do more than previous generations.
- Consider to disable SMP/HT in the BIOS. This way more CPU cache will be available per thread. And more cache hits will yield in more effective performance.
- Verify how the PCIe lanes are setup for the motherboard you use. As in which slots might be shared with other things.
- If possible try to maximize available memorychannels. I think AMD EPYC normally can max out at 12x memory. Here there will of course be a thing of 6400MT vs 5200MT or even 4800MT memory.
- When it comes to NICs there is a short whitelist of “tested and verified” NICs for VPP in VyOS. There is a command to overrule that. I dunno about VPP but my current gut feeling is that broadcom NICs are currently the “least worst” on the market. Both in terms of hardware but also the software quality regarding drivers.
- While at it dont forget to update and use latest firmware/BIOS for motherboard but also the NICs.
- And of course make sure you have enough cooling so you dont end up with thermal throttling.
Hey @Apachez, thank you for your input here. And did you had experience with a similar hardware setup ? I mean, by using basic BGP routing with one full routing table only, were you able to forward in linerate at these very high small packet rates ?
Forgot to mention.
Since you look at 100G then make sure you use single nic cards.
Since 100Gbps is about 11.65GB/s (per direction) so you need PCIe 4.0 x8 or PCIe 5.0 x4 or higher:
Or if you do get dualnics make sure their lane capability along with which slots you put them in your motherboard matches the needed bandwidth.