I use an vyos in an Dell R210 single processor, 8Gb, a run fine.
I need transfer to an better server, then I used an R710, dual processor, 32Gb, same pci board SFP+ dual, but when I transfer the traffic for it, I don’t need throughput, and the ping fo to 800ms or more.
I yet tried use the version 1.2, 1.3 and nothing. And I yet tried with an IBM M3.
If I return to R210 then it works normal.
Try cleaning your fiber cables, switch SFP+'s etc, it’s something hardware related for sure, not software. Also, seeing as the problem persists no matter the PCI card, seriously look at fiber & optics, they are highly sensitive.