X550-T2 Bug With New VyOS Version

Hydra166 · August 29, 2021, 12:54am

I’m having an issue with a new version of VyOS and 2500Base/T. I’m using a X550-T2 network card. By default, the card advertises negotiation at 100/1000/10000, but supports 2500BaseT/5000BaseT if you set the link speed/negotiation advertisement. I’m linking to a cable modem with a 2500BaseT support, and I was unable to set the speed as a fixed speed since I believe the modem or the card’s software requires you to autonegotiate. I was able to set the advertised link modes as 2500Base/T by running 'ethtool -s eth0 advertise 0x800000000000’. I put this in the postconfig boot up script.

I’m linking with an S33 cable modem.

On the June 08, 2021 version, my ethtool looks the following:
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 2500baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 2500Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

On the August 21st/August 28th versions (both tested but probably more in between and before). I get this. Running the ethtool command directly makes no difference. Even changing advertised link modes does nothing at this point. Maybe there was some driver change? My eth1 connected to my server also only links at Gigabit instead of 10G

Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 2500baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: no

When I switch back to the June 08 version, it all works again.
Any ideas or diagnostics I could do here? Maybe there was a driver change?

Dmitry · August 31, 2021, 8:01am

Hello @Hydra166 ,

Did you try to set speed and duplex manually on the version begin 21st Aug?
I see that speed ad duplex are Unknown

Hydra166 · September 1, 2021, 12:31am

I tried that with 2.5gbps before and I did not have luck. But things may have changed.Auto negotiation seems to play better with the drivers for Intel. Don’t see a reason why autoeng shouldn’t work, and I think they’re unknown because ethtool thinks there’s no link (says "Link detected: no) considering it’s the same setup, but I’ll still give it a shot regardless in the next ~12-24 hours. I will try to see if there’s any logs when i try again as well.

Hydra166 · September 4, 2021, 2:18am

Sorry for the delay, work has been super busy. Will do this tonight.

Hydra166 · September 5, 2021, 5:19pm

Unfortunately, that did not work. Speed and duplex are still unknown and a link is not detected. As soon as I switch back to the old version, my old way of setting autoneg advertisements at 2500Base/T work fine. Setting speed and duplex never really worked on either version for me that well for 2500Base/T. Any ideas?

Dmitry · September 5, 2021, 7:06pm

Hi @Hydra166 , provide please show interfaces output.

Hydra166 · September 6, 2021, 5:36am

For some reason on the very latest VyOS rolling version I had an issue getting my config to post but it worked fine on the 0807 version. I will investigate further.

Will update you tomorrow.

SCCFIT · September 14, 2021, 1:39am

@Hydra166 anything to report? It sounds like it could be related to this thread. I was able to upgrade as far as 20210831 before I started getting errors relating to autonegotiate with an X520.

Hydra166 · September 14, 2021, 1:53am

Nothing much yet, but I think it is the same bug. Autoneg was working fine before and suddenly I try updating and it all breaks. I believe 202108210117 was the first one that caused issues for me. @Dmitry - the interfaces look fine and the same as normal, it is just tough to get a screenshot when you can only see a little in console (I have a lot of VLANs)

Hydra166 · September 14, 2021, 1:55am

Do you have Slack, Discord or Matrix? Maybe we could chat on this.

SCCFIT · September 14, 2021, 2:18am

Just sent you a private message

Hydra166 · October 5, 2021, 6:11am

I’m still experiencing those issues with my X550-T2.

(No Link)
(Config Success)
(dmesg conflicting info)

Originally, I was seeing a no failure on the configuration loading and then I was seeing “Unknown” on both interfaces of my X550-T2 network card (first update in May worked fine then August-now showed that behavior.). Then in the beginning of September, the configuration started to fail as well as that. Then there was some sort of bug found with autonegotiation that was fixed (⚓ T3874 D-Link Ethernet Interface not working.). After the fix, my VyOS is back to its state of the configuration not failing and both interfaces of the X550-T2 refusing to link or detect a link… I suspect some sort of driver issue or interaction issue between VyOS and Debian because the link information in dmesg conflicts with what actually happens with the card in ethtool. In dmesg, eth0 (notable that eth0 only supports 100/1000/2500) seems to link at some point but is not reflected at all in ethtool.

Do you have any ideas? I’m still running on the May version because of these issues and it’s killing me not to update.

Before-June: Worked fine

July-August Versions: No Configuration Fail, No Link Detected

Late August-September Versions: Configuration Fail, No Link Detected on X550-T2

September 31st-October 5rd: No Link Detected on X550-T2

eth0 should link at 2500Base/T and its advertisements are set as so via a startup script, and ethtool verifies this… eth1 should link at 10000Base/T and advertisements are set as defaults.

Hydra166 · October 5, 2021, 6:15am

You can’t see the top but essentially ‘show interfaces’ shows eth0 with – because there’s a missing WAN IP which I acquire through DHCP and all the other interfaces and outputs look normal. The only difference is the lack of link in ethtool. I’m going to file a report on Phabricator when my account gets approved because I’m pretty confident there’s some sort of bug her.

jahanson · July 14, 2023, 12:29am

Did you ever find a resolution to this? I have an X550-AT2 and my link speed will only negotiate to 1000

Apachez · July 14, 2023, 7:03am

Googling on “ethtool advertise X550-T2” shows that this is a current issue with Intel X550 series and their linux drivers.

According to page 9 at https://cdrdv2-public.intel.com/333717/X550_SpecUpdate_Rev2.9.pdf

NBASE-T Speed Advertisement

NBASE-T speed advertisement (2.5 GbE/5 GbE) is disabled by default in the X550 NVM. An NBASE-T enabled driver should be used to link at NBASE-T speeds.
Linux driver version 4.3.9 is currently the only supported software solution for NBASE-T.
For future planned enhancements, contact your Intel representative.

Also according to IEEE using autoneg is mandatory for anything at 1Gbps and above (when using TP-cabling, autoneg also brings auto mdi/mdix which with autoneg disabled also gets disabled).

Personally I would try to alter the advertise list so if you want 2.5G it would only advertise 2.5G and nothing else like so:

sudo ethtool -s eth0 speed 2500 duplex full autoneg on

Verify with ethtool eth0 that the link with “Advertised link modes” only shows a single speed/duplex combo like so:

Advertised link modes: 2500baseT/Full

The reason for this I think, and I have experienced this myself in the wild, is compatiblity issues.

Its no fun if you got a NBASE-T interface on your end advertising 100M, 1G, 2.5G, 5G, 10G and your end ends up with lets say 5G and the other end ends up with 1G - the result is that no traffic will pass (which is a shitty situation for a remote site with some traverling distance to visit).

I think the fix in VyOS would be to allow for setting 2500M (2.5G) along with full/half duplex along with autoneg enabled/disabled (autoneg should be default according to IEEE).

This way you could do this through config mode instead of having to hack some init-scripts (which seems to be the current workaround - that is alter /config/scripts/vyos-preconfig-bootup.script or /config/scripts/vyos-postconfig-bootup.script).

Ref: https://bbs.archlinux.org/viewtopic.php?id=262075

jahanson · July 14, 2023, 11:52am

You hit the nail on the head with this one, setting to 2500 autoneg with ethtool fixed this right up.Thank you for such a detailed answer!

Hydra166 · July 14, 2023, 1:20pm

Yeah! It wasn’t working for a few VyOS versions but it does work now on the latest versions. You can set the ethtool command as a startup script and it will automatically run.

Apachez · July 19, 2023, 7:11pm

It seems like setting a speed/duplex statically in 1.4 rolling release the autoneg remains enabled.

So something like this should be sufficient to overcome the NBASE-T issues which might exist in the wild:

set interface ethernet eth0 speed 2500
set interface ethernet eth0 duplex full

However it can be handy to be able to select multiple speed/duplex while using autoneg or for that matter select only one and having autoneg disabled.

I have filed a feature request regarding this: ⚓ T5375 It should be possible to set multiple advertised speed/duplex with autoneg on through config