As I recall it UniFi devices will accept just about any transceiver you plug into them while Intel NIC’s dont.
That is the transceiver you plug into an Intel NIC must be “Intel” coded.
That AOC cable - how was it ordered regarding coding and did you plug the correct end of the cable into that Intel NIC?
Thinking about Unsupported SFP+ transceivers on Intel NIC where i40e driver doesnt seem to support “allow_unsupported_sfp”.
Other things to test is to use a regular transceiver instead of that AOC and see how that link up.
You can also reach out to the FS.com support on how to troubleshoot the hardware itself (again my best bet is the coding of that end of AOC you plugged into that Intel NIC).
Your NIC got 2 transceiver slots so in theory you could plug that AOC cable into both and in VyOS setup VRF where NIC1 goes for VRF:RED and NIC2 goes for VRF:BLUE and then ping between them (in VyOS you switch vrf in op-mode with force vrf <vrfname>.
Other than that I would try to install 1.4-rolling to see if that also brings you the same error (it uses the current Linux LTS kernel while the 1.3.3 is using a much older one). Or for that matter boot on a Ubuntu Live cd/dvd/usb which is using the latest stable kernel (newer than the current Linux LTS kernel) to see if that changes anything (just to rule out driver version issues).
Regarding statically setting speed/duplex you can try to do this directly through “sudo ethtool” and see if that helps. Technically a transceiver normally only supports a single speed/duplex but that is often dealt with by the driver so you can continue to use auto/auto from the OS point of view even if the transceiver itself only supports lets say 25G/Full Duplex.