VyOS 1.2.0 is not working with Supermicro motherboards?

vyos-1-2

#1

My hardware:
Supermicro MBD-A1SRI-2758F-O
8GB ECC RAM
128 SSD
Intel X520 with dual 10GbE

I have been fighting with the vyos-config. I have tried all of the rolling releases from 20190319 to 20190405. The main problem is that I have with the rolling releases between 20190319 and 20190402 is that the interfaces loads randomly. For example, one release would load eth0, eth1 and eth4, but not the rest of the interfaces. Another release, would load eth0 to eth4. eth0 will not be functional, but the eth2 and eth3 will be the Intel X520.

Another issue which applies to rolling release and the VyOS Crux ISO builder is the command “show interface ethernet eth# identify” does not work with onboard NIC, but it works with Intel X520.

My last issue, I could not get the vyos-config to load no matter what release I used (rolling release and ISO builder). This is also true for version 1.1.8. The 1.1.8 is the worse one for me because it doesn’t save the config no matter what after reboot, but it is the version the loads all the interfaces properly.
This is the error I was getting at the login:

[37.166869] vyos-router [946]: Starting VyOS router: migrate rl-system firewall configure failed!
[ 37.190384] vyos-config[1015]: Configuration error

The only time that I would get a success is the first reboot after the upgrade. If I reboot again, the error above is what I always get.

I loaded the ISO I built to my Proxmox server and I don’t get any problem that I encountered with my hardware.


#2

Can you share your config? Looks like there is a deprecated option in there which prevents the full load, vyos the falls back to a very minimal default.


#3

I simply cannot commit to full config because of the effort it would take then the system would simply disregard it. I was just testing a small config as shown below:

set interface ethernet eth2 address 10.0.3.100/24
set protocol static route 0.0.0.0/0 next 10.0.3.1
set service ssh listen-address 0.0.0.0

If I hardcoded the config in /config/config.boot, the system seems to ignore this as well.
Also, the ssh listen-address would not take a host address. SSH listen-address only works with 0.0.0.0. If I have just host addresses something like 10.0.11.21 and 10.0.3.1, I would get a “connection refused”.

EDIT:
It seems like I misunderstood the ssh listen-address command. I thought this is a source address of the client. However, even if I specified a specific interface address for listen-address, I still could not ssh to my vyos box. There is no acl and zone-based firewall rules.
I am on 1.1.8 at the moment since this is the only one that sees all the Interfaces of my Supermicro. I nmap’d my 1.1.8 and it says ssh is closed.


#4

Try:
‘set service ssh port 22’
‘commit’

Then try to login via ssh again.


#5

I rebooted then somehow I got ssh again with specific listen-address. I am running 1.1.8 at this point. I still have not fully configure the box yet. I have a lot of config converting to do from Junos to vyos.

Any ideas about my 1.2.0 vyos-config situation?
What is the deprecated option that you mentioned earlier?


#6

Hello, @netengnir!

Are you trying to upgrade existing 1.1.8 instance, move config file from 1.1.8 to 1.2.0, or trying to configure 1.2.0 via CLI?
Could you try to make fresh, clean install 1.2.0, reboot system and run show configuration command? Check if all Ethernet interfaces have hw-id option there.


#7

Are you trying to upgrade existing 1.1.8 instance, move config file from 1.1.8 to 1.2.0, or trying to configure 1.2.0 via CLI?

This is a fresh/clean install, but could not pass the following:
1.2.0 build and rolling release

  • after the fresh installation and the required reboot, the system could not load the config.boot or the vyos-config https://i.imgur.com/j8MbcV3.jpg?1
  • the nic does not load properly
  • tried to re-arrange the nic by MAC address, but it never works for onboard NIC, but worked with PCIe Intel x520.

Could you try to make fresh, clean install 1.2.0, reboot system and run show configuration command? Check if all Ethernet interfaces have hw-id option there.

Yes, I ran the show interfaces under the configuration mode and there are missing interfaces. Depending on the version of the rolling release, sometimes I don’t get eth5 and sometimes I don’t get eth3. I meant the config literally missing the eth# block. When I tried to add the missing eth#, I get an error stating that the interface does not exist.


#8

This is from the latest 1.2.0 rolling release. As you can see, it does not match the config.boot

vyos@bgdvyos:~$ show version 
Version:          VyOS 1.2.0-rolling+201904060337
Built by:         autobuild@vyos.net
Built on:         Sat 06 Apr 2019 03:37 UTC
Build ID:         53f4c329-24fe-43d6-b319-94ea38930547

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Supermicro
Hardware model:   A1SAi
Hardware S/N:     Unknown
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors
vyos@bgdvyos:~$

Here is the interface block from the config.boot

interfaces {
    bridge br3 {
        aging "300"
        description "man gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br5 {
        address "10.0.5.1/24"
        aging "300"
        description "security gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br6 {
        address "10.0.6.1/24"
        aging "300"
        description "unifi AP gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br7 {
        address "10.0.7.1/24"
        aging "300"
        description "server gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br9 {
        address "10.0.9.1/24"
        aging "300"
        description "vpn gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br10 {
        address "10.0.10.1/24"
        aging "300"
        description "wired gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br11 {
        aging "300"
        description "wireless gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br12 {
        address "10.0.12.1/24"
        aging "300"
        description "guest gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br15 {
        address "10.0.15.1/24"
        aging "300"
        description "iot gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    bridge br20 {
        address "10.0.20.1/24"
        aging "300"
        description "dmz gateway"
        hello-time "2"
        max-age "20"
        priority "0"
        stp "false"
    }
    dummy dum0 {
        address "10.0.0.1/32"
        description "in-band management"
    }
    ethernet eth0 {
        address "dhcp"
        description "untrust interface - the Internet"
        duplex "auto"
        hw-id "ac:1f:6b:b1:d3:a8"
        smp-affinity "auto"
        speed "auto"
    }
    ethernet eth1 {
        address "10.0.3.100/24"
        duplex "auto"
        hw-id "ac:1f:6b:b1:d3:a9"
        smp-affinity "auto"
        speed "auto"
    }
    ethernet eth2 {
        duplex "auto"
        hw-id "ac:1f:6b:b1:d3:aa"
        smp-affinity "auto"
        speed "auto"
    }
    ethernet eth3 {
        duplex "auto"
        hw-id "ac:1f:6b:b1:d3:ab"
        smp-affinity "auto"
        speed "auto"
        hw-id 00:1b:21:b9:f4:a1
    }
    ethernet eth4 {
        duplex "auto"
        hw-id "00:1b:21:b9:f4:a0"
        smp-affinity "auto"
        speed "auto"
    }
    ethernet eth5 {
        duplex "auto"
        hw-id "00:1b:21:b9:f4:a1"
        smp-affinity "auto"
        speed "auto"
    }
    loopback     lo
}

Here is the show interface ethernet

vyos@bgdvyos:~$ show interfaces ethernet 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             -                                 u/D  untrust interface - the Internet 
eth1             10.0.3.100/24                     u/u  
eth2             -                                 u/D  
eth3             -                                 u/D  

This is the show configuration command

set interfaces bridge br3 aging '300'
set interfaces bridge br3 description 'man gateway'
set interfaces bridge br3 hello-time '2'
set interfaces bridge br3 max-age '20'
set interfaces bridge br3 priority '0'
set interfaces bridge br3 stp 'false'
set interfaces bridge br5 address '10.0.5.1/24'
set interfaces bridge br5 aging '300'
set interfaces bridge br5 description 'security gateway'
set interfaces bridge br5 hello-time '2'
set interfaces bridge br5 max-age '20'
set interfaces bridge br5 priority '0'
set interfaces bridge br5 stp 'false'
set interfaces bridge br6 address '10.0.6.1/24'
set interfaces bridge br6 aging '300'
set interfaces bridge br6 description 'unifi AP gateway'
set interfaces bridge br6 hello-time '2'
set interfaces bridge br6 max-age '20'
set interfaces bridge br6 priority '0'
set interfaces bridge br6 stp 'false'
set interfaces bridge br7 address '10.0.7.1/24'
set interfaces bridge br7 aging '300'
set interfaces bridge br7 description 'server gateway'
set interfaces bridge br7 hello-time '2'
set interfaces bridge br7 max-age '20'
set interfaces bridge br7 priority '0'
set interfaces bridge br7 stp 'false'
set interfaces bridge br9 address '10.0.9.1/24'
set interfaces bridge br9 aging '300'
set interfaces bridge br9 description 'vpn gateway'
set interfaces bridge br9 hello-time '2'
set interfaces bridge br9 max-age '20'
set interfaces bridge br9 priority '0'
set interfaces bridge br9 stp 'false'
set interfaces bridge br10 address '10.0.10.1/24'
set interfaces bridge br10 aging '300'
set interfaces bridge br10 description 'wired gateway'
set interfaces bridge br10 hello-time '2'
set interfaces bridge br10 max-age '20'
set interfaces bridge br10 priority '0'
set interfaces bridge br10 stp 'false'
set interfaces bridge br11 aging '300'
set interfaces bridge br11 description 'wireless gateway'
set interfaces bridge br11 hello-time '2'
set interfaces bridge br11 max-age '20'
set interfaces bridge br11 priority '0'
set interfaces bridge br11 stp 'false'
set interfaces bridge br12 address '10.0.12.1/24'
set interfaces bridge br12 aging '300'
set interfaces bridge br12 description 'guest gateway'
set interfaces bridge br12 hello-time '2'
set interfaces bridge br12 max-age '20'
set interfaces bridge br12 priority '0'
set interfaces bridge br12 stp 'false'
set interfaces bridge br15 address '10.0.15.1/24'
set interfaces bridge br15 aging '300'
set interfaces bridge br15 description 'iot gateway'
set interfaces bridge br15 hello-time '2'
set interfaces bridge br15 max-age '20'
set interfaces bridge br15 priority '0'
set interfaces bridge br15 stp 'false'
set interfaces bridge br20 address '10.0.20.1/24'
set interfaces bridge br20 aging '300'
set interfaces bridge br20 description 'dmz gateway'
set interfaces bridge br20 hello-time '2'
set interfaces bridge br20 max-age '20'
set interfaces bridge br20 priority '0'
set interfaces bridge br20 stp 'false'
set interfaces dummy dum0 address '10.0.0.1/32'
set interfaces dummy dum0 description 'in-band management'
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'untrust interface - the Internet'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 hw-id 'ac:1f:6b:b1:d3:a8'
set interfaces ethernet eth0 smp-affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address '10.0.3.100/24'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 hw-id 'ac:1f:6b:b1:d3:a9'
set interfaces ethernet eth1 smp-affinity 'auto'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 hw-id 'ac:1f:6b:b1:d3:aa'
set interfaces ethernet eth2 smp-affinity 'auto'
set interfaces ethernet eth2 speed 'auto'
set interfaces ethernet eth3 duplex 'auto'
set interfaces ethernet eth3 hw-id '00:1b:21:b9:f4:a1'
set interfaces ethernet eth3 smp-affinity 'auto'
set interfaces ethernet eth3 speed 'auto'
set interfaces loopback lo

#9

I also want to add this is the lspci output

vyos@bgdvyos:~$ lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Atom processor C2000 SoC Transaction Router [8086:1f08] (rev 03)
00:01.0 PCI bridge [0604]: Intel Corporation Atom processor C2000 PCIe Root Port 1 [8086:1f10] (rev 03)
00:02.0 PCI bridge [0604]: Intel Corporation Atom processor C2000 PCIe Root Port 2 [8086:1f11] (rev 03)
00:03.0 PCI bridge [0604]: Intel Corporation Atom processor C2000 PCIe Root Port 3 [8086:1f12] (rev 03)
00:0b.0 Co-processor [0b40]: Intel Corporation Atom processor C2000 nCPM [8086:1f18] (rev 03)
00:0e.0 Host bridge [0600]: Intel Corporation Atom processor C2000 RAS [8086:1f14] (rev 03)
00:0f.0 IOMMU [0806]: Intel Corporation Atom processor C2000 RCEC [8086:1f16] (rev 03)
00:13.0 System peripheral [0880]: Intel Corporation Atom processor C2000 SMBus 2.0 [8086:1f15] (rev 03)
00:14.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I354 [8086:1f41] (rev 03)
00:14.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection I354 [8086:1f41] (rev 03)
00:14.2 Ethernet controller [0200]: Intel Corporation Ethernet Connection I354 [8086:1f41] (rev 03)
00:14.3 Ethernet controller [0200]: Intel Corporation Ethernet Connection I354 [8086:1f41] (rev 03)
00:16.0 USB controller [0c03]: Intel Corporation Atom processor C2000 USB Enhanced Host Controller [8086:1f2c] (rev 03)
00:17.0 SATA controller [0106]: Intel Corporation Atom processor C2000 AHCI SATA2 Controller [8086:1f22] (rev 03)
00:18.0 SATA controller [0106]: Intel Corporation Atom processor C2000 AHCI SATA3 Controller [8086:1f32] (rev 03)
00:1f.0 ISA bridge [0601]: Intel Corporation Atom processor C2000 PCU [8086:1f38] (rev 03)
00:1f.3 SMBus [0c05]: Intel Corporation Atom processor C2000 PCU SMBus [8086:1f3c] (rev 03)
01:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
02:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
03:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
04:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

#10

@netengnir, OK, I see… Send, please, the logs and interface list.

  1. Reboot router.
  2. Run:
sudo journalctl
sudo ip l

This will help to understand why your NICs are not configuring properly.


#11

It seems like this is what I am getting with rolling release 20190410:
eth0 identified correctly
eth1 identified correctly
eth2 identified correctly
eth3 identified as 10GbE port B (Intel x520)
eth4 identified as 10GbE port A (Intel x520)
eth5 identified as ‘rename7’ interface. eth5 is the 4th onboard NIC.

The interface rename7 shows in sudo ifconfig -a

rename7   Link encap:Ethernet  HWaddr ac:1f:6b:b1:d3:ab  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

#12

Hello, @netengnir!

delete interfaces ethernet eth5 hw-id
commit
save

Then reboot and:

sudo journalctl
sudo ip l

Without a log we are blind in this situation.


#13

I tried to upload but it says new users are not allowed to upload attachment.