10GbE SFP+ link not coming up

Hi there,

I’ve been using VyOS for a few years now and this week decided to move from a Intel ET2 Quad port NIC to a dual SFP+ 10GbE ports NIC (Chelsio n320e, though of giving it a try since I have it just laying around).

To connect VyOS (1.3.0-epa3) to the switch, I’m using a direct attached copper cable from Cisco (SFP-H10GB-CU3M), which connects to a Netgear S3300-28X.

I know that the switch is ok with this transceivers, as I’m using them to connect a server to the switch, working fine.
I’ve also used this NIC with this same Cisco cable in the past, however in a server box, not Vyos, and it also worked.

In VyOS, I’ve manually set the following (mostly for testing, just want the bloody thing to come up before I start actually configuring things for good):

ethernet eth5 {
     address 10.22.33.1/24
     disable-flow-control
     duplex full
     hw-id 00:07:43:0c:4f:e8
     mtu 1500
     speed 10000
}

When I get out of config mode and do show interfaces ethernet eth5 I get:

eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:07:43:0c:4f:e8 brd ff:ff:ff:ff:ff:ff
    inet 10.22.33.1/24 brd 10.22.33.255 scope global eth5
       valid_lft forever preferred_lft forever
    inet6 fe80::207:43ff:fe0c:4fe8/64 scope link
       valid_lft forever preferred_lft forever

    RX:  bytes  packets  errors  dropped  overrun       mcast
             0        0       0        0        0           0
    TX:  bytes  packets  errors  dropped  carrier  collisions
          3870       31       0        0        0           0

Command: show interfaces ethernet eth5 physical

Settings for eth5:
        Supported ports: [ AUI FIBRE ]
        Supported link modes:   10000baseT/Full
                                40000baseCR4/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: no
Ring parameters for eth5:
Pre-set maximums:
RX:             16384
RX Mini:        0
RX Jumbo:       16384
TX:             16384
Current hardware settings:
RX:             1024
RX Mini:        1024
RX Jumbo:       512
TX:             1024

driver: cxgb3
version: 1.1.5-ko
firmware-version: T 7.12.0 TP 1.1.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

On the switch side, a matching configuration is set in terms of speed (10GB), Full duplex and 1500mtu.

Any guidance and tips on why its not working would be greatly appreciated.

Thank you.

Hi @Ralm,

can you please do the following:

Log into VyOS and run:
sudo systemctl stop vyos-configd
touch /tmp/vyos.ifconfig.debug
configure

Now change speed/duplex to auto and see if it works. Them change it back to fixed.

During commit you will see a lot of messages on your terminal, please attach them here.

Hi there,

Changing to Auto both duplex and speed and then deleted the actual configuration:

 ethernet eth5 {
     address 10.22.33.1/24
     disable-flow-control
     hw-id 00:07:43:0c:4f:e8
     mtu 1500
 }

After commit link still down:

[ interfaces ethernet eth5 ]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG write '0' > '/sys/class/net/eth5/queues/rx-0/rps_cpus'
DEBUG/IFCONFIG NIC driver does not support changing speed/duplex settings!
{'address': ['10.22.33.1/24'],
 'disable_flow_control': {},
 'duplex': 'auto',
 'hw_id': '00:07:43:0c:4f:e8',
 'ifname': 'eth5',
 'ip': {'arp_cache_timeout': '30'},
 'mtu': '1500',
 'speed': 'auto'}
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth5/link_filter'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip link set dev eth5 nomaster'
DEBUG/IFCONFIG read '30000' < '/proc/sys/net/ipv4/neigh/eth5/base_reachable_time_ms'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth5/arp_filter'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/arp_accept'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/arp_announce'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/arp_ignore'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/proxy_arp'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/proxy_arp_pvlan'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth5/forwarding'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/rp_filter'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth5/forwarding'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth5/accept_ra'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv6/conf/eth5/autoconf'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth5/dad_transmits'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'tc qdisc del dev eth5 handle ffff: ingress 2> /dev/null;tc qdisc del dev eth5 handle 1: root prio 2> /dev/null;set $?=0'
DEBUG/IFCONFIG cmd 'ip link set dev eth5 up'

All I see is that message of “DEBUG/IFCONFIG NIC driver does not support changing speed/duplex settings!”.

Changing Speed back to 10000 and Duplex to full:

[ interfaces ethernet eth5 ]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG write '0' > '/sys/class/net/eth5/queues/rx-0/rps_cpus'
DEBUG/IFCONFIG cmd 'ethtool --change eth5 speed 10000 duplex full autoneg off'
{'address': ['10.22.33.1/24'],
 'disable_flow_control': {},
 'duplex': 'full',
 'hw_id': '00:07:43:0c:4f:e8',
 'ifname': 'eth5',
 'ip': {'arp_cache_timeout': '30'},
 'mtu': '1500',
 'speed': '10000'}
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth5/link_filter'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip link set dev eth5 nomaster'
DEBUG/IFCONFIG read '30000' < '/proc/sys/net/ipv4/neigh/eth5/base_reachable_time_ms'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth5/arp_filter'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/arp_accept'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/arp_announce'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/arp_ignore'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/proxy_arp'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/proxy_arp_pvlan'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth5/forwarding'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth5/rp_filter'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth5/forwarding'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth5/accept_ra'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv6/conf/eth5/autoconf'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth5/dad_transmits'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth5'
DEBUG/IFCONFIG returned (out):
[{"ifindex":3,"ifname":"eth5","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"00:07:43:0c:4f:e8","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":81,"max_mtu":65535,"inet6_addr_gen_mode":"none","num_tx_queues":8,"num_rx_queues":8,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'tc qdisc del dev eth5 handle ffff: ingress 2> /dev/null;tc qdisc del dev eth5 handle 1: root prio 2> /dev/null;set $?=0'
DEBUG/IFCONFIG cmd 'ip link set dev eth5 up'

Link still not coming up :frowning:

Please let me know if you need anything other information or tests.
Thank you

Update:

To double check, I’ve connected the Chelsio card directly to one of my servers to see if it would come up and it did automatically.
Meaning this might be some issue or incompatibility between Chelsio and Netgear.

Will be doing some more research.

Update 2:

Replaced the Chelsio N320e card for a Intel X520-DA2 and everything worked out of the box.
Its a shame as I wanted to use the DA2 on a server.
It seems that the Chelsio card will once more go to the drawer due to the headaches it causes.

Thanks for the update.

Sorry to hear about those issues.

Yeah I was going to say, I use Intel X520 and X710 without issue. The ixgbe driver that come with X520 is the most trusted dispite its age. The newer i40 driver that comes with X710 had bugs in the past but Intel seems to fix most of it. As for me I plan on using X710 to be future proof but I would always keep an X520 as an insurance plan.

1 Like