10G SFP+ trouble with linking (Intel X553)

,

I did look at the already existing topic, but that didin’t help me. Any ideas as to what’s the issue here?

My module on the VyOS side is Intel coded from FS.com and Juniper module on the other end. I did verify that it does link up if both modules are in the juniper.

        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 10G Ethernet: 10G Base-SR
        Encoding                                  : 0x06 (64B/66B)
        BR, Nominal                               : 10300MBd
        Rate identifier                           : 0x02 (8/4/2G Rx Rate_Select only)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 80m
        Length (62.5um)                           : 30m
        Length (Copper)                           : 0m
        Length (OM3)                              : 300m
        Laser wavelength                          : 850nm
        Vendor name                               : FS
        Vendor OUI                                : 00:1b:21
        Vendor PN                                 : SFP-10GSR-85
        Vendor rev                                : A
        Option values                             : 0x00 0x3a

I seem to be getting very frequently an warning regarding flow control settings? I’m running 1.4-rolling-202308040306.

What exactly is the error you are getting?

I’m not seeing an error so that’s why I’m confused.

The only error I have is WARNING: could not change "eth0" flow control setting! when committing.

Add this to your config then try committing:
set interfaces ethernet eth0 disable-flow-control

That seems to only get rid of the error, but the link is still not up, despite it working on JunOS with both modules and same cable.

Are there any clues in either of these commands?:
show interfaces ethernet eth0 physical
show interfaces ethernet eth0 transceiver

I don’t believe so?

It also seems that when rebooting, it does light up on both ends, but after VyOS starts it’s not anymore.

show interfaces ethernet eth0 physical
Settings for eth0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4096
Current hardware settings:
RX:             512
RX Mini:        n/a
RX Jumbo:       n/a
TX:             512
RX Buf Len:             n/a
CQE Size:               n/a
TX Push:        off
TCP data split: n/a
driver: ixgbe
version: 6.1.42-amd64-vyos
firmware-version: 0x80000bd7
expansion-rom-version: 
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
vyos@gateway#  show interfaces ethernet eth0 transceiver 
        Identifier                                : 0x03 (SFP)
        Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 10G Ethernet: 10G Base-SR
        Encoding                                  : 0x06 (64B/66B)
        BR, Nominal                               : 10300MBd
        Rate identifier                           : 0x02 (8/4/2G Rx Rate_Select only)
        Length (SMF,km)                           : 0km
        Length (SMF)                              : 0m
        Length (50um)                             : 80m
        Length (62.5um)                           : 30m
        Length (Copper)                           : 0m
        Length (OM3)                              : 300m
        Laser wavelength                          : 850nm
        Vendor name                               : FS
        Vendor OUI                                : 00:1b:21
        Vendor PN                                 : SFP-10GSR-85
        Vendor rev                                : A
        Option values                             : 0x00 0x3a
        Option                                    : RX_LOS implemented
        Option                                    : TX_FAULT implemented
        Option                                    : TX_DISABLE implemented
        Option                                    : RATE_SELECT implemented
        BR margin, max                            : 0%
        BR margin, min                            : 0%
        Vendor SN                                 : G2304040205
        Date code                                 : 230411
        Optical diagnostics support               : Yes
        Laser bias current                        : 6.706 mA
        Laser output power                        : 0.5514 mW / -2.59 dBm
        Receiver signal average optical power     : 0.6284 mW / -2.02 dBm
        Module temperature                        : 55.27 degrees C / 131.49 degrees F
        Module voltage                            : 3.2907 V
        Alarm/warning flags implemented           : Yes
        Laser bias current high alarm             : Off
        Laser bias current low alarm              : Off
        Laser bias current high warning           : Off
        Laser bias current low warning            : Off
        Laser output power high alarm             : Off
        Laser output power low alarm              : Off
        Laser output power high warning           : Off
        Laser output power low warning            : Off
        Module temperature high alarm             : Off
        Module temperature low alarm              : Off
        Module temperature high warning           : Off
        Module temperature low warning            : Off
        Module voltage high alarm                 : Off
        Module voltage low alarm                  : Off
        Module voltage high warning               : Off
        Module voltage low warning                : Off
        Laser rx power high alarm                 : Off
        Laser rx power low alarm                  : Off
        Laser rx power high warning               : Off
        Laser rx power low warning                : Off
        Laser bias current high alarm threshold   : 13.000 mA
        Laser bias current low alarm threshold    : 4.000 mA
        Laser bias current high warning threshold : 12.500 mA
        Laser bias current low warning threshold  : 5.000 mA
        Laser output power high alarm threshold   : 1.0000 mW / 0.00 dBm
        Laser output power low alarm threshold    : 0.2512 mW / -6.00 dBm
        Laser output power high warning threshold : 0.7943 mW / -1.00 dBm
        Laser output power low warning threshold  : 0.3162 mW / -5.00 dBm
        Module temperature high alarm threshold   : 78.00 degrees C / 172.40 degrees F
        Module temperature low alarm threshold    : -13.00 degrees C / 8.60 degrees F
        Module temperature high warning threshold : 73.00 degrees C / 163.40 degrees F
        Module temperature low warning threshold  : -8.00 degrees C / 17.60 degrees F
        Module voltage high alarm threshold       : 3.7000 V
        Module voltage low alarm threshold        : 2.9000 V
        Module voltage high warning threshold     : 3.6000 V
        Module voltage low warning threshold      : 3.0000 V
        Laser rx power high alarm threshold       : 1.0000 mW / 0.00 dBm
        Laser rx power low alarm threshold        : 0.0100 mW / -20.00 dBm
        Laser rx power high warning threshold     : 0.7943 mW / -1.00 dBm
        Laser rx power low warning threshold      : 0.0158 mW / -18.01 dBm

I’m not familiar with Juniper switches, but I assume there are some show commands that will show transceiver / port information. Anything showing up there? Any leftover configuration on the Juniper port that may be conflicting?

If nothing out of the ordinary shows up on that end, I’m out of ideas and hopefully somebody else will have insight.

I have tested with Intel and juniper modules in juniper and it links up together. Same with two Intel modules in VyOS connected together.

So it seems quite wierd as to why it won’t link up with Intel and Juniper coded ones together when they are in the right devices.

Juniper does have command for the same diagnostics but that doesn’t seem like anything abnormal to me.

Have you tried forcing the speed/duplex on both sides? Your output definitely does the transceiver receiving/sending -2.0 dBm of light, so they see each other (I would confirm on the Juniper side just to be sure.

You could also make a hard loop (plug both ends of the single fiber into the optic) then plug into the Intel coded optic to see if it will even link to itself. If it does, then you can have some certainty VyOS does recognize it/would work with it.

I haven’t as VyOS says it’s not supported. I’m using ixgbe driver.

Unfortunately not possible as not a single cable but rather the one with both already attached.

For reference, both of them see each other, but no actual link despite that.

VyOS:

 Laser bias current                        : 6.634 mA
 Laser output power                        : 0.5536 mW / -2.57 dBm
 Receiver signal average optical power     : 0.6262 mW / -2.03 dBm

JunOS:

Laser bias current                        :  6.074 mA
Laser output power                        :  0.5870 mW / -2.31 dBm
Receiver signal average optical power     :  0.6170 mW / -2.10 dBm

Do you have another SFP on hand you can try on the Vyos side?

I have tested that both intel coded modules do work on the VyOS box as link will come up when connected together aswell as Intel module in Juniper and Juniper coded module in Juniper will also work when both are in juniper.

I do have generic gig SFP modules, but not sure if it will like them.
EDIT: Generic gig SFP modules linked up immediately on both sides but that doesn’t really help as I want 10G :frowning:

What exact vendor and model are these intel and juniper modules?

Got a link for both?

And are you sure that both are multimode modules aka 850nm (since the one in your first post is)?

Also can you produce such output as in your first post for all of the modules to verify that the coding is correct (so you didnt by accident get the wrong modules or so)?

If 2 of the Intel coded modules works in the same VyOS box and 2 of the Juniper coded modules works in the same Juniper box and you are using the same cable (multimode since its multimode modules) then it shouldnt be anything wrong with the modules themselves or how the NIC identifies them. As long as all 4 modules are of the same type (10GBASE-SR or so).

Otherwise you can have stuff like the cable itself isnt straight but crossed over so the modules cant see each other and stuff like that. Also make sure that the cable is at least 2 meters of length so the optic doesnt blind itself (you could lightly roll up the cable aswell to introduce some more attentuation in it).

Some vendors/models/nics only accept vendor coded modules, as in your Juniper coded module probably wont work in the Intel NIC in your VyOS box and the other way around.

Something else to also verify is when you test 2 of the models in the VyOS box is to test all SFP+ slots and rotate the modules (as in if module1 was in slot1 and module2 was in slot2 then rotate them so module2 ends up in slot1 and module1 in slot2).

Sometimes boxes have more than one internal nic so it could be that for example slot1-2 go to NIC A while slot 3-6 go to NIC B. That is perhaps NIC A is very picky about the coding while NIC B isnt.

The issue is either driver or just VyOS/Debian in general. It works in OPNsense live env, but doesn’t even in Debian 12.

Juniper module is: https://www.fs.com/de-en/products/31287.html
Intel module is: https://www.fs.com/de-en/products/71385.html
Generic 1gig modules were: https://www.fs.com/de-en/products/75332.html

I would like to point out that the following config did link up in juniper:

  • 1 X INTEL MODULE with 1 X JUNIPER MODULE does link up together
  • 1 X generic one gig SFP with generic one gig in vyos does work

I would like to also point out that the following worked in vyos:

  • 1 x INTEL MODULE with 1 x JUNIPER module linked up together
  • 1 x generic gig SFP with another of same kind linked up no issue

Do you get the same result if you try the latest 1.5-rolling (2023-10-03 as of writing)?

OPNsense (23.7) is based on FreeBSD 13.2 while VyOS (1.5) is based on Debian 12.1 meaning Linux LTS Kernel 6.1 (6.1.55 as of writing).

Im not sure which file contains the version info about the driver itself but here is the current on from 6.1.55 (used by VyOS 1.5-rolling):

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/net/ethernet/intel/ixgbe?h=v6.1.55

Here is from 6.6-rc4 (latest Linux Kernel as of writing):

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe?h=v6.6-rc4

And here is the one from FreeBSD 13.2:

The source for the Intel NIC drivers seems to be:

1 Like

I haven’t tried 1.5, but I’m currently using 1.4-rolling-202310020306 in which it doesn’t appear to work.

If the kernel is 6.1.55 in 1.4-rolling aswell then you dont have to test 1.5-rolling for this case (unless you really want to :slight_smile:

“uname -a” should tell you wish kernel you got there.

That’s indeed true, it’s 6.1.55-amd64-vyos in my 1.4 rolling.

It may be necessary to install the Intel network adapter driver correctly on Debian. If an incorrect driver is installed, the Intel network adapter may indeed experience the phenomenon you mentioned in some drivers. We also encounter this situation during operation and maintenance. The Vyos 1.4 version of the Intel network adapter driver requires reinstalling the official driver on Debian. This is how we solve this problem