10G Sfp+ Pci Card

huseyintr27 · January 18, 2024, 10:30am

I have 3 dual port sfp pci cards of the same brand and model. When I wear them, only one of them can be seen. Does Vyos impose any restrictions on this issue?

tomcat667 · January 18, 2024, 11:56am

look with this command are you seen all cards

lspci -vv

huseyintr27 · January 18, 2024, 12:41pm

08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 36
        NUMA node: 0
        IOMMU group: 15
        Region 0: Memory at fd800000 (64-bit, non-prefetchable) [size=1M]
        Region 2: I/O ports at 5000 [size=32]
        Region 4: Memory at fdb00000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fda00000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 71
        NUMA node: 0
        IOMMU group: 16
        Region 0: Memory at fd900000 (64-bit, non-prefetchable) [size=1M]
        Region 2: I/O ports at 5020 [size=32]
        Region 4: Memory at fdb04000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fda80000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

0b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin B routed to IRQ 89
        NUMA node: 0
        IOMMU group: 18
        Region 2: I/O ports at 6000 [size=32]
        Capabilities: <access denied>
        Kernel modules: ixgbe

0b:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 89
        NUMA node: 0
        IOMMU group: 19
        Region 2: I/O ports at 6020 [size=32]
        Capabilities: <access denied>
        Kernel modules: ixgbe

Viacheslav · January 18, 2024, 4:40pm

Check the file:

cat /etc/modprobe.d/ixgbe-options.conf

And try to change to:

options ixgbe allow_unsupported_sfp=1,1,1

then reboot

huseyintr27 · January 18, 2024, 10:18pm

I tried what you said but it didn’t work.

04:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        DeviceName: NIC Port 4
        Subsystem: Hewlett-Packard Company Ethernet 1Gb 4-port 331FLR Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 53, NUMA node 0, IOMMU group 17
        Memory at fc360000 (64-bit, prefetchable) [size=64K]
        Memory at fc350000 (64-bit, prefetchable) [size=64K]
        Memory at fc340000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at fc060000 [virtual] [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: tg3
        Kernel modules: tg3

08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 36, NUMA node 0, IOMMU group 15
        Memory at fd800000 (64-bit, non-prefetchable) [size=1M]
        I/O ports at 5000 [size=32]
        Memory at fdb00000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fda00000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 71, NUMA node 0, IOMMU group 16
        Memory at fd900000 (64-bit, non-prefetchable) [size=1M]
        I/O ports at 5020 [size=32]
        Memory at fdb04000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fda80000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

0b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 3
        Flags: fast devsel, IRQ 89, NUMA node 0, IOMMU group 18
        I/O ports at 6000 [size=32]
        Capabilities: <access denied>
        Kernel modules: ixgbe

0b:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 3
        Flags: fast devsel, IRQ 89, NUMA node 0, IOMMU group 19
        I/O ports at 6020 [size=32]
        Capabilities: <access denied>
        Kernel modules: ixgbe

patient0 · January 25, 2024, 9:37am

In your output of lspci -vv you can see that only the first has the line Kernel driver in use: so that is the only one the system uses.

Searching the dmesg output may give you an error message to work with:

sudo dmesg | egrep '(08:00|0b:00)'

huseyintr27 · January 25, 2024, 1:39pm

It still doesn’t see the installed ones.

[    2.775218] ixgbe 0000:08:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[    2.775587] ixgbe 0000:08:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    2.776004] ixgbe 0000:08:00.0: MAC: 2, PHY: 19, SFP+: 6, PBA No: G43260-009
[    2.776080] ixgbe 0000:08:00.0: 38:ea:a7:15:92:c0
[    2.779565] ixgbe 0000:08:00.0: Intel(R) 10 Gigabit Network Connection
[    3.333580] ixgbe 0000:08:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
[    3.333979] ixgbe 0000:08:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    3.334394] ixgbe 0000:08:00.1: MAC: 2, PHY: 19, SFP+: 5, PBA No: G43260-009
[    3.334474] ixgbe 0000:08:00.1: 38:ea:a7:15:92:c1
[    3.338043] ixgbe 0000:08:00.1: Intel(R) 10 Gigabit Network Connection
[    3.346528] ixgbe: probe of 0000:0b:00.0 failed with error -5
[    3.347768] ixgbe: probe of 0000:0b:00.1 failed with error -5
[    3.354297] ixgbe 0000:08:00.0 e5: renamed from eth3
[    3.363303] ixgbe 0000:08:00.1 e7: renamed from eth0
[   33.994506] ixgbe 0000:08:00.0 eth4: renamed from e5
[   34.020743] ixgbe 0000:08:00.1 eth5: renamed from e7
[  144.984201] ixgbe 0000:08:00.0: registered PHC device on eth4
[  145.487992] ixgbe 0000:08:00.0 eth4: detected SFP+: 6
[  145.628033] ixgbe 0000:08:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[  149.583257] ixgbe 0000:08:00.1: registered PHC device on eth5
[  149.849006] ixgbe 0000:08:00.1 eth5: detected SFP+: 5
[  149.988019] ixgbe 0000:08:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX

patient0 · January 25, 2024, 1:51pm

There’s something to search the internet for.

And maybe my grep was to strict, try something like sudo dmesg | egrep ‘ixgbe’ and look for more errors.

Additions: One of the suggestions on the internet is that it is maybe not a supported SFP+ module … but that message would show up in the log.

huseyintr27 · January 25, 2024, 8:21pm

[    3.338043] ixgbe 0000:08:00.1: Intel(R) 10 Gigabit Network Connection
[    3.346528] ixgbe: probe of 0000:0b:00.0 failed with error -5
[    3.347768] ixgbe: probe of 0000:0b:00.1 failed with error -5
[    3.354297] ixgbe 0000:08:00.0 e5: renamed from eth3
[    3.363303] ixgbe 0000:08:00.1 e7: renamed from eth0
[   33.994506] ixgbe 0000:08:00.0 eth4: renamed from e5
[   34.020743] ixgbe 0000:08:00.1 eth5: renamed from e7
[  144.984201] ixgbe 0000:08:00.0: registered PHC device on eth4
[  145.487992] ixgbe 0000:08:00.0 eth4: detected SFP+: 6
[  145.628033] ixgbe 0000:08:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[  149.583257] ixgbe 0000:08:00.1: registered PHC device on eth5
[  149.849006] ixgbe 0000:08:00.1 eth5: detected SFP+: 5
[  149.988019] ixgbe 0000:08:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX

patient0 · January 25, 2024, 8:42pm

That’s the same again as before, no new info to help you.

Please provide:

Is VyOS running on hardware?
If yes on what?
If yes, did the hardware with the 10Gbit NICs in exact the same configuration work (= cards in the same PCI slot, etc) with another OS? (e.g. try booting from an USB stick or CD ROM into an recent Ubuntu 22.04+)
If above doesn’t work, remove one card and see what happens. Could be a hardware problem or a PCI init issue
or swap cards around and see which cards work in which slot, or does it not matter
If no, what is the host OS and how to you pass the cards to VyOS
Do you use the same SFP+ module manufacturer for all the ports?

huseyintr27 · January 25, 2024, 11:29pm

Hi @patient0 ,

1- Yes. HP DL380p Gen8 2x Riser Card
2- We used vyos 1.5.x rolling iso while installing. It works smoothly on other hardware.
3- Removing the card will be a problem, the network may be cut off.
4- It doesn’t matter which home they are in.
5- Vyos 1.5.x rolling .iso Link
6- Yes. Huawei SFP+ Module

patient0 · January 26, 2024, 5:49pm

I don’t completely understand what you wrote:

Does it work with 1.5-rolling, or on what version are you right now? And is the machine already productive that you can’t remove a network card? Booting Ubuntu shows the same issue? And what does work on other hardware, vyos 1.5-rolling or the 10Gbit cards?

It seems to me it’s either the kernel, the motherboard/slots or the card(s). Without removing some of the cards and see what’s happening it’s hard (for me at least) to know what causes the issue.

The cards are quite old and should be very well supported by way older Linux kernels but it may be an issue with the kernel and options used by VyOS. Booting a recent Ubuntu or Debian would help narrow it down.

Apachez · January 27, 2024, 6:43am

This!

When in doubt liveboot some other linuxdistro (or for that matter bsd-based distro) to see if you get the same behaviour.

Would narrow down if this is a driver/settings related issue or a hardware/incompatibility/bios issue.

When it comes to intel there are both the kernel drivers who are lagging when it comes to support of features and intels own “out of tree” drivers where the later ones are prefered.

There are at least one task over at https://vyos.dev regarding this but I dont think it have been resolved yet (the task is that VyOS should use the out-of-tree drivers for intel nics rather than the kernel drivers).

huseyintr27 · January 27, 2024, 9:17am

What they see now is the 1.5 rolling version, but they do not see the ones on the other card. I thought of downgrading VYOs. If we switch to versions like 1.2 or 1.3, the problem seems to be solved. I cannot remove the card while it is working. One of the modules working on the device connected to the data center uplink port is the uplink port.

huseyintr27 · January 27, 2024, 11:36am

I upgraded to vyos version 1.2, unfortunately it still doesn’t show up.

08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 36
        NUMA node: 0
        IOMMU group: 15
        Region 0: Memory at fd800000 (64-bit, non-prefetchable) [size=1M]
        Region 2: I/O ports at 5000 [size=32]
        Region 4: Memory at fdb00000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fda00000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 71
        NUMA node: 0
        IOMMU group: 16
        Region 0: Memory at fd900000 (64-bit, non-prefetchable) [size=1M]
        Region 2: I/O ports at 5020 [size=32]
        Region 4: Memory at fdb04000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at fda80000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

0b:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin B routed to IRQ 89
        NUMA node: 0
        IOMMU group: 18
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Region 2: I/O ports at 6000 [size=32]
        Region 3: Memory at <ignored> (32-bit, non-prefetchable)
        Capabilities: <access denied>
        Kernel modules: ixgbe

0b:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560SFP+ Adapter
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 89
        NUMA node: 0
        IOMMU group: 19
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Region 2: I/O ports at 6020 [size=32]
        Region 3: Memory at <ignored> (32-bit, non-prefetchable)
        Capabilities: <access denied>
        Kernel modules: ixgbe

patient0 · January 27, 2024, 1:01pm

Please boot from an USB stick with Ubuntu 22.04 LTS or later.

If it’s a VyOS issue then it’s a possibility that the kernel config they use is creating an issue for you on different version.
That’s why a test with different distro is really necessary.

huseyintr27 · January 27, 2024, 1:13pm

saw all the ports in ubuntu 22.04.

huseyintr27 · January 27, 2024, 1:14pm

saw all the ports in ubuntu 22.04.

patient0 · January 27, 2024, 1:51pm

Ok, good to have it narrowed it down. I can’t help you with that but I hope someone else has an idea how to compile the kernel with a different config or an out-of-tree ixgbe driver. Naturally there would be no official support I assume.

Apachez · January 27, 2024, 1:53pm

Or rather get a Debian 12.x ISO image and boot from that since VyOS is based on Debian and not Ubuntu.

On the other hand VyOS is also using a custom vanilla kernel with patches.