New Supermicro Mini Server with built in 10GB

General questions
1

We’ve been using vyatta/vyos for 6 or 7 years now. Until today we’ve been very happy with it. Many gigabit units and the last 3 or 4 years all vmware vmxnet 10GB virtual servers. Things have worked great.

Until today. First we had to disable the XHCI and USB 3.0 to get it to install whatsover from USB.

Then none of the 10GB interfaces are coming up.

Tried the latest 1.1.7 and the build of today with the 1.2.999.

Supermicro server here: http://shop.ironnetworks.com/supermicro-superserver-sys-5018d-fn8t 2 built in intell D 1500 (x552/577) (Lab Confirmation: Intel Xeon D 10G LAN is X552 / X557-AT) 10GB Nics.

We also added a PCI-e card for 2 more. Broadcom B5700 based. https://www.amazon.com/gp/product/B01M0F51QO/ref=oh_aui_detailpage_o05_s00?ie=UTF8&psc=1

Vyos seems to not been picking up on them online at all. Just ONCE we saw the internal ones ‘up’ with the 1.2 beta code. It appears the broadcom ones have never even been detected.

Is Vyos dying out? The underlying OS seems to be very outdated and there is no effort to keep it up.

Any ideas? We had high hopes of these in datacenters. Not sure what to do now.

Interface IP Address S/L Description

eth0 10.3.0.1/16 u/u
10.1.3.1/16
eth1 - u/D
eth2 - u/D
eth3 - A/D
eth4 - u/D
eth5 - u/D
eth6 10.1.3.16/16 A/D
eth7 10.1.3.17/16 u/D
eth8 10.1.3.18/16 u/D
10.3.3.18/16
eth9 10.1.3.19/1 u/D
10.3.3.19/16
lo 127.0.0.1/8 u/u

8 and 9 are the internal intel nics…

We have cables plugged in and transmissions lights on 8 and what would be 10 but it’s not showing up.

All ideas welcome.

Thanks

2

Hello,
we don’ t have supermicro servers nor spare time for them,
so until someone with device start fixing issues, it will be like that.
For us VyOS is not dying :slight_smile: We just focused on things we need

3

Interesting approach and timelines. The devices in them are not supermicro, they are intel and broadcom NICs The two top NICs used by enterprise which are not only moving to 10G but also 40G and 100G. 1G Nics are not used anywhere in data centers now. For VyOS to progress beyond the “Home Lab” these are the type of things that need fixed. As Enterprise admins we are willing to try opensource things, but if there is no support from the native community it is time to move on. Very sad as we have been exclusive vyatta, vyos for 8 to 10 years.

Tim

4

To add to this: We thought this might be the perfect VyOS setup:

https://www.supermicro.com/products/system/1u/5018/SYS-5018D-FN8T.cfm

Small quite form factor.
Xeon CPU
6 onboard 1GB Nics,
2 onboard 10GB Nics

Added a second 10GB PCIe card for 2 more 10GB.
NVMe on motherboard.

This should be a workhorse. 2 of these for bare metal with failover is the plan. Both for under $2000 total.

Really not sure why you would not want to support this.

Known issues:

  1. Cannot support USB install without modifying bios. (Turn off XHCI and USB 3.0 Compatibility as a work around) This issue has been open for a couple years.
  2. Broadcom 5700 NICs not being detected.
  3. Intel X552/X557 NICs not “up” when connected.

We have already figured our next step. Just posting this for the next guys.

Tim

5

here is the issue on the onboard controllers being detected but not working. It appears that one of the required hardware files is not on the system.

vyos@vyos:/$ lspci -nn | grep “10 Gigabit”
06:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57810S 10 Gigabit Ethernet [14e4:168e] (rev 10)
06:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57810S 10 Gigabit Ethernet [14e4:168e] (rev 10)

vyos@vyos:/$ dmesg |grep -i eth6
[ 12.745229] bnx2x: [bnx2x_init_firmware:12502(eth6)]Can’t load firmware file bnx2x/bnx2x-e2-7.8.17.0.fw
[ 12.857689] bnx2x: [bnx2x_func_hw_init:5724(eth6)]Error loading firmware
[ 12.937899] bnx2x: [bnx2x_nic_load:2621(eth6)]HW init failed, aborting

vyos@vyos:/$ ls -al /lib/firmware/bnx2x/bnx2x-e2*
-rw-r–r-- 1 root root 247008 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-6.0.34.0.fw
-rw-r–r-- 1 root root 247232 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-6.2.5.0.fw
-rw-r–r-- 1 root root 247504 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-6.2.9.0.fw
-rw-r–r-- 1 root root 290952 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-7.0.20.0.fw
-rw-r–r-- 1 root root 289448 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-7.0.23.0.fw
-rw-r–r-- 1 root root 289848 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-7.0.29.0.fw
-rw-r–r-- 1 root root 295072 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-7.2.16.0.fw
-rw-r–r-- 1 root root 295264 Feb 10 2015 /lib/firmware/bnx2x/bnx2x-e2-7.2.51.0.fw

For VyOS 1.1.7 you can put this:

driver in /lib/firmware/bnx2x/ and it appears to bring the interface up.

for VyOS 1.2.xxx copy this file to /lib/firmware/bnx2x/:

to bring the interface up.

Appears to be working. Testing further. Starting on the Intel card now.

1 Like
6

Well,
i not told that we don’t want support that,
just that don’t have possibility spend time for that.
Still everybody have access to sources and build script and welcome to submit pull requests.

7

Hi
What did you figure out? I was thinking about purchasing a couple of these myself.
But if the internal nics are a problem, I’ll rather find something that allows me to use a pci-e dual nic.

8

I know this is an old thread but the SFP ports not coming up have nothing to do with VyOS. I am running the same server on CentOS 7 and it doesn’t come up because the linux drivers don’t support auto negotiation.

If you want to use the SFP+ ports you have to configure the speed/duplex settings because the controller cannot auto negotiate. In CentOS 7 if you try “ethtool -s eno8 speed 1000 duplex full” the link will come up.
I had a 1Gbit SFP on the switch connected to the server thats why the speed was set to 1000. I am not familiar with VyOS yet but I am planning to run it on these servers.

You have to edit the interface config to make the speed/duplex config permanent.

See the text below for the workaround from the driver release notes here: https://downloadmirror.intel.com/26556/eng/README.txt

NOTE: For the Intel(R) Ethernet Connection X552 10 GbE SFP+ you must specify
the desired speed.

1 Like