Intel XXV710 on Vyos 1.3.3, transceiver detected and understood, link not detected

General questions
,
1

Home lab. I’m running Vyos 1.3.3 on a Supermicro with an AMD EPYC and 64GB of ECC DDR 2667 (lotta compute power, but nothing extra fancy):

$ show version

Version:          VyOS 1.3.3
Release train:    equuleus

Built by:         Sentrium S.L.
Built on:         Mon 29 May 2023 12:55 UTC
Build UUID:       a302f99b-4d44-4a40-82ba-1a4275902d5e
Build commit ID:  bc64a3a72244b9

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   Super Server
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-ac1f6bea8be6

Copyright:        VyOS maintainers and contributors

I installed an Intel XXV710-AM2-based dual 25G SFP28 network card from FS. It’s recognized and it’s using the i40e driver:

$ show interfaces ethernet eth2 physical 
Settings for eth2:
        Supported ports: [ ]
        Supported link modes:   25000baseCR/Full 
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: None BaseR RS
        Advertised link modes:  25000baseCR/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: None BaseR RS
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no
Ring parameters for eth2:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        0
RX Jumbo:       0
TX:             512

driver: i40e
version: 2.8.20-k
firmware-version: 6.02 0x80003888 1.1853.0
expansion-rom-version: 
bus-info: 0000:41:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I then connected a 2m 25G SFP28 Active Optical Cable from FS to the card and to a UniFi Enterprise Aggregation Switch. It is also recognized and nothing looks out of place (though I am no expert on some of these values):

$ show interfaces ethernet eth2 transceiver 
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x23 (No separable connector)
        Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x08 0x00 0x00 0x01
        Transceiver type                          : Active Cable
        Transceiver type                          : Extended: 100G AOC or 25GAUI C2M AOC with worst BER of 5x10^(-5)
        Encoding                                  : 0x00 (unspecified)
        BR, Nominal                               : 25500MBd
        Rate identifier                           : 0x00 (unspecified)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 0m
        Length (62.5um)                           : 0m
        Length (Copper)                           : 2m
        Length (OM3)                              : 0m
        Active Cu cmplnce.                        : 0x00 (unspecified) [SFF-8472 rev10.4 only]
        Vendor name                               : FS
        Vendor OUI                                : 64:9d:99
        Vendor PN                                 : S28-AO02
        Vendor rev                                : 
        Option values                             : 0x08 0x00
        Option                                    : Retimer or CDR implemented
        BR margin, max                            : 103%
        BR margin, min                            : 0%
        Vendor SN                                 : C2301200935-1
        Date code                                 : 230209
        Optical diagnostics support               : Yes
        Laser bias current                        : 6.466 mA
        Laser output power                        : 1.3205 mW / 1.21 dBm
        Receiver signal average optical power     : 1.1033 mW / 0.43 dBm
        Module temperature                        : 64.00 degrees C / 147.20 degrees F
        Module voltage                            : 3.3410 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : Off
        Laser rx power high warning               : Off
        Laser rx power low warning                : Off
        Laser bias current high alarm threshold   : 15.000 mA
        Laser bias current low alarm threshold    : 1.000 mA
        Laser bias current high warning threshold : 12.000 mA
        Laser bias current low warning threshold  : 2.000 mA
        Laser output power high alarm threshold   : 2.2387 mW / 3.50 dBm
        Laser output power low alarm threshold    : 0.1258 mW / -9.00 dBm
        Laser output power high warning threshold : 1.9999 mW / 3.01 dBm
        Laser output power low warning threshold  : 0.2511 mW / -6.00 dBm
        Module temperature high alarm threshold   : 80.00 degrees C / 176.00 degrees F
        Module temperature low alarm threshold    : -10.00 degrees C / 14.00 degrees F
        Module temperature high warning threshold : 70.00 degrees C / 158.00 degrees F
        Module temperature low warning threshold  : 0.00 degrees C / 32.00 degrees F
        Module voltage high alarm threshold       : 3.6300 V
        Module voltage low alarm threshold        : 2.9700 V
        Module voltage high warning threshold     : 3.4650 V
        Module voltage low warning threshold      : 3.1350 V
        Laser rx power high alarm threshold       : 2.2387 mW / 3.50 dBm
        Laser rx power low alarm threshold        : 0.0501 mW / -13.00 dBm
        Laser rx power high warning threshold     : 1.9999 mW / 3.01 dBm
        Laser rx power low warning threshold      : 0.1000 mW / -10.00 dBm

Here’s the configuration:

# show interfaces ethernet eth2
 address 192.168.192.3/24
 description "Intel SFP28 Dual NIC: Top"
 duplex auto
 hw-id 64:9d:99:b1:ef:1b
 speed auto

Both the Unifi switch and Vyos are showing the interface as “down,” and I cannot get it to come up. You may have noticed above:

        Link detected: no

I imagine that ^^ should say “yes.” Also, this shows it down:

$ show interfaces ethernet eth2
eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 64:9d:99:b1:ef:1b brd ff:ff:ff:ff:ff:ff
    inet 192.168.192.3/24 brd 192.168.192.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::669d:99ff:feb1:ef1b/64 scope link tentative 
       valid_lft forever preferred_lft forever
    Description: Intel SFP28 Dual NIC: Top

    RX:  bytes  packets  errors  dropped  overrun       mcast
             0        0       0        0        0           0
    TX:  bytes  packets  errors  dropped  carrier  collisions
             0        0       0        0        0           0

I have tried setting the speed and duplex, but I get this error, which I don’t think I should be getting, given that the physical details above shows that it does support this mode:

# set interfaces ethernet eth2 speed 25000
[edit]
# set interfaces ethernet eth2 duplex full
[edit]
# commit

Adapter does not support changing speed and duplex settings to:
25000/full!

[[interfaces ethernet eth2]] failed
Commit failed
[edit]

Still, with these set to “auto” and the UniFi switch set to “auto,” this should work, and yet … it’s like I don’t have a cable plugged in.

Suggestions on what to try next?

2

As I recall it UniFi devices will accept just about any transceiver you plug into them while Intel NIC’s dont.

That is the transceiver you plug into an Intel NIC must be “Intel” coded.

That AOC cable - how was it ordered regarding coding and did you plug the correct end of the cable into that Intel NIC?

Thinking about Unsupported SFP+ transceivers on Intel NIC where i40e driver doesnt seem to support “allow_unsupported_sfp”.

Other things to test is to use a regular transceiver instead of that AOC and see how that link up.

You can also reach out to the FS.com support on how to troubleshoot the hardware itself (again my best bet is the coding of that end of AOC you plugged into that Intel NIC).

Your NIC got 2 transceiver slots so in theory you could plug that AOC cable into both and in VyOS setup VRF where NIC1 goes for VRF:RED and NIC2 goes for VRF:BLUE and then ping between them (in VyOS you switch vrf in op-mode with force vrf <vrfname>.

Other than that I would try to install 1.4-rolling to see if that also brings you the same error (it uses the current Linux LTS kernel while the 1.3.3 is using a much older one). Or for that matter boot on a Ubuntu Live cd/dvd/usb which is using the latest stable kernel (newer than the current Linux LTS kernel) to see if that changes anything (just to rule out driver version issues).

Regarding statically setting speed/duplex you can try to do this directly through “sudo ethtool” and see if that helps. Technically a transceiver normally only supports a single speed/duplex but that is often dealt with by the driver so you can continue to use auto/auto from the OS point of view even if the transceiver itself only supports lets say 25G/Full Duplex.

3

We have had alot of issues with intel nics pretending some transceivers dont exist because they aren’t intel coded too.

4

Hmm. I thought about the coding being a problem (I got the cable coded Ubiquiti), BUT … as I showed above, it’s recognizing the transceiver. I’m not getting the EEPROM error that others have reported when the transceivers are not correctly coded.

That said … I could buy-and-try an Intel-coded cable, or an Ubuntu Live boot, as suggested.

5

If you got more than a few transceivers/AOC/DAC-cables from FS.com you can invest in one of their FS BOX which makes it possible for you to reprogram the transceiver/AOC/DAC on your own. You need a modern webbrowser and an internet connection to do so:

1 Like
6

Wow! That’s an incredibly cool tool that I did not know about. I … might go ahead and try that …

After discussing with their support, they recommended I buy a custom cable coded to Intel on one end and Ubiquiti on the other, but that’s a 6-week lead time at least, and I’d rather not wait that long. So I bought stock cables coded Intel on both ends, to arrive Wednesday. If those don’t work, I’ll have to decide between trying out a box (might try the “I want one for free” program) and waiting on a custom cable.

7

Please keep the community updated on how this progress :slight_smile:

In the meantime you could try to boot your box on a Ubuntu live cd (22.04) just to rule out any driver/kernel issues.

The ISO: Download Ubuntu Desktop | Download | Ubuntu

How to make that into a USB drive: GitHub - balena-io/etcher: Flash OS images to SD cards & USB drives, safely and easily.

1 Like
8

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.