I have a working VyOS configuration up and running on 1.2.5. I decided I’d try it out on a 1.3 rolling build (most current build in the directory as of today) and get errors. From the time that “Mounting VyOS Config…done” appears on my console, it takes 3 minuites 18 seconds ± for “Starting VyOS router: migrate rl-system firewall configure failed” and "Configuration error " to appear. I then get a prompt to log in as usual.
Once logged in, I noticed that all my interfaces are misnumbered:
I noticed ordering issues with 1.2.5 due to the NIC installed in my router. I have a Netgate SG-5100 (Lanner 1510 if I recall right?) and so the interfaces for my working config as I look at the router from left to right are: eth4, eth5, eth0, eth1, eth2, eth3! I think this is because eth0-eth3 are NICS directly connected to the CPU. eth4 and eth5 are a separate NIC card.
I’ve attached two photos of the console screen for the error and interface madness:
Hello @ngoehring, can you add vyos-config-debug to grub before boot system like described in Development — VyOS 1.4.x (sagitta) documentation and provide /tmp/boot-config-trace
Also will be interesting to look into sudo cat /run/udev/log/vyatta-net-name.coldplug and sudo dmesg -T | grep eth
Output for the two other commands you asked me to run is as follows;
vyos@VyosOverseas:~$ sudo cat /run/udev/log/vyatta-net-name.coldplug
Mon May 25 20:57:35 2020: lookup eth5 00:90:0b:a2:9f:16
Mon May 25 20:57:35 2020: use hw-id 00:90:0b:a2:9f:16 in config mapped to ‘eth7’
Mon May 25 20:57:35 2020: lookup eth1 00:90:0b:a2:9f:12
Mon May 25 20:57:35 2020: use hw-id 00:90:0b:a2:9f:12 in config mapped to ‘eth9’
Mon May 25 20:57:35 2020: lookup eth0 00:90:0b:a2:9f:11
Mon May 25 20:57:36 2020: use hw-id 00:90:0b:a2:9f:11 in config mapped to ‘eth8’
Mon May 25 20:57:36 2020: lookup eth2 00:90:0b:a2:9f:13
Mon May 25 20:57:36 2020: use hw-id 00:90:0b:a2:9f:13 in config mapped to ‘eth11’
Mon May 25 20:57:36 2020: lookup eth3 00:90:0b:a2:9f:14
Mon May 25 20:57:36 2020: use hw-id 00:90:0b:a2:9f:14 in config mapped to ‘eth10’
Mon May 25 20:57:36 2020: lookup eth4 00:90:0b:a2:9f:15
Mon May 25 20:57:36 2020: use hw-id 00:90:0b:a2:9f:15 in config mapped to ‘eth6’
and
vyos@VyosOverseas:~$ sudo dmesg -T | grep eth
[Mon May 25 20:57:17 2020] igb 0000:03:00.0 eth0: mixed HW and IP checksum settings.
[Mon May 25 20:57:17 2020] igb 0000:03:00.0: added PHC on eth0
[Mon May 25 20:57:17 2020] igb 0000:03:00.0: eth0: (PCIe:2.5GT/s:Width x1)
[Mon May 25 20:57:17 2020] igb 0000:03:00.0 eth0: MAC: 00:90:0b:a2:9f:11
[Mon May 25 20:57:17 2020] igb 0000:03:00.0: eth0: PBA No: 000300-000
[Mon May 25 20:57:17 2020] igb 0000:04:00.0 eth1: mixed HW and IP checksum settings.
[Mon May 25 20:57:17 2020] igb 0000:04:00.0: added PHC on eth1
[Mon May 25 20:57:17 2020] igb 0000:04:00.0: eth1: (PCIe:2.5GT/s:Width x1)
[Mon May 25 20:57:17 2020] igb 0000:04:00.0 eth1: MAC: 00:90:0b:a2:9f:12
[Mon May 25 20:57:17 2020] igb 0000:04:00.0: eth1: PBA No: 000300-000
[Mon May 25 20:57:17 2020] igb 0000:03:00.0 eth0: mixed HW and IP checksum settings.
[Mon May 25 20:57:17 2020] igb 0000:04:00.0 eth1: mixed HW and IP checksum settings.
[Mon May 25 20:57:18 2020] ixgbe 0000:06:00.0 eth2: MAC: 6, PHY: 27, PBA No: 000600-000
[Mon May 25 20:57:18 2020] ixgbe 0000:06:00.0 eth2: Enabled Features: RxQ: 4 TxQ: 4 FdirHash vxlan_rx
[Mon May 25 20:57:18 2020] ixgbe 0000:06:00.0 eth2: Intel® 10 Gigabit Network Connection
[Mon May 25 20:57:18 2020] ixgbe 0000:06:00.1 eth3: MAC: 6, PHY: 27, PBA No: 000600-000
[Mon May 25 20:57:18 2020] ixgbe 0000:06:00.1 eth3: Enabled Features: RxQ: 4 TxQ: 4 FdirHash vxlan_rx
[Mon May 25 20:57:18 2020] ixgbe 0000:06:00.1 eth3: Intel® 10 Gigabit Network Connection
[Mon May 25 20:57:18 2020] ixgbe 0000:08:00.0 eth4: MAC: 6, PHY: 27, PBA No: 000600-000
[Mon May 25 20:57:18 2020] ixgbe 0000:08:00.0 eth4: Enabled Features: RxQ: 4 TxQ: 4 FdirHash vxlan_rx
[Mon May 25 20:57:18 2020] ixgbe 0000:08:00.0 eth4: Intel® 10 Gigabit Network Connection
[Mon May 25 20:57:18 2020] ixgbe 0000:08:00.1 eth5: MAC: 6, PHY: 27, PBA No: 000600-000
[Mon May 25 20:57:18 2020] ixgbe 0000:08:00.1 eth5: Enabled Features: RxQ: 4 TxQ: 4 FdirHash vxlan_rx
[Mon May 25 20:57:18 2020] ixgbe 0000:08:00.1 eth5: Intel® 10 Gigabit Network Connection
[Mon May 25 20:57:32 2020] igb 0000:03:00.0 eth0: mixed HW and IP checksum settings.
[Mon May 25 20:57:32 2020] igb 0000:04:00.0 eth1: mixed HW and IP checksum settings.
[Mon May 25 20:57:35 2020] ixgbe 0000:08:00.1 eth7: renamed from eth5
[Mon May 25 20:57:35 2020] igb 0000:04:00.0 eth1: mixed HW and IP checksum settings.
[Mon May 25 20:57:35 2020] igb 0000:04:00.0 eth9: renamed from eth1
[Mon May 25 20:57:35 2020] igb 0000:03:00.0 eth0: mixed HW and IP checksum settings.
[Mon May 25 20:57:35 2020] igb 0000:03:00.0 eth8: renamed from eth0
[Mon May 25 20:57:36 2020] ixgbe 0000:06:00.0 eth11: renamed from eth2
[Mon May 25 20:57:36 2020] ixgbe 0000:06:00.1 eth10: renamed from eth3
[Mon May 25 20:57:36 2020] ixgbe 0000:08:00.0 eth6: renamed from eth4
[Mon May 25 20:57:39 2020] igb 0000:03:00.0 eth8: mixed HW and IP checksum settings.
[Mon May 25 20:57:39 2020] igb 0000:04:00.0 eth9: mixed HW and IP checksum settings.
[Mon May 25 20:58:53 2020] ixgbe 0000:06:00.1: registered PHC device on eth10
[Mon May 25 20:58:53 2020] IPv6: ADDRCONF(NETDEV_UP): eth10: link is not ready
[Mon May 25 20:58:55 2020] ixgbe 0000:06:00.0: registered PHC device on eth11
[Mon May 25 20:58:56 2020] IPv6: ADDRCONF(NETDEV_UP): eth11: link is not ready
[Mon May 25 20:59:35 2020] igb 0000:04:00.0 eth9: mixed HW and IP checksum settings.
[Mon May 25 20:59:35 2020] igb 0000:04:00.0 eth9: mixed HW and IP checksum settings.
[Mon May 25 20:59:35 2020] igb 0000:04:00.0 eth9: mixed HW and IP checksum settings.
[Mon May 25 20:59:35 2020] igb 0000:04:00.0 eth9: mixed HW and IP checksum settings.
[Mon May 25 20:59:35 2020] igb 0000:04:00.0 eth9: mixed HW and IP checksum settings.
[Mon May 25 20:59:37 2020] IPv6: ADDRCONF(NETDEV_UP): eth9: link is not ready
[Mon May 25 20:59:39 2020] igb 0000:03:00.0 eth8: mixed HW and IP checksum settings.
[Mon May 25 20:59:39 2020] igb 0000:03:00.0 eth8: mixed HW and IP checksum settings.
[Mon May 25 20:59:39 2020] igb 0000:03:00.0 eth8: mixed HW and IP checksum settings.
[Mon May 25 20:59:39 2020] igb 0000:03:00.0 eth8: mixed HW and IP checksum settings.
[Mon May 25 20:59:39 2020] igb 0000:03:00.0 eth8: mixed HW and IP checksum settings.
[Mon May 25 20:59:41 2020] IPv6: ADDRCONF(NETDEV_UP): eth8: link is not ready
[Mon May 25 20:59:43 2020] ixgbe 0000:08:00.1: registered PHC device on eth7
[Mon May 25 20:59:44 2020] IPv6: ADDRCONF(NETDEV_UP): eth7: link is not ready
[Mon May 25 20:59:44 2020] igb 0000:03:00.0 eth8: igb: eth8 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[Mon May 25 20:59:44 2020] IPv6: ADDRCONF(NETDEV_CHANGE): eth8: link becomes ready
[Mon May 25 21:00:12 2020] ixgbe 0000:08:00.0: registered PHC device on eth6
[Mon May 25 21:00:12 2020] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
I believe I found the way to add the command, please correct me if I am wrong.
I am waiting for the router to boot and when presented with the option to run VyOS in KVM, serial, or USB mode, I press “e”. The problem I am having, however, is when I try to move the cursor (using the arrow key) to add the vyos-config-debug to the grub arguments, I can only move the cursor once. If I press it again, a [C is printed on the screen and then it goes back to the grub option screen. I notice that if I press the arrow very slowly, like once every 10 seconds, the cursor advances. This will, of course, take me forever to do and is not a viable option. if I press CTL-C I get a grub command line. Is there a way to call that argument from there? Or is there something I need to do with my console (putty) settings?
On GRUB menu press e key and add vyos-config-debug like in screenshots below
and press Ctrl-x
After booting router, you can get trace file from /tmp/boot-config-trace
OK great! That’s what I thought I needed to do. Please see my earlier message about the difficulties with the cursor movement that I am having. I’m not sure how to get around this as the current situation makes it almost impossible to do anything.
I think you can boot the router and add this to the GRUB via shell, ssh or whatever
edit /boot/grub/grub.cfg and add this param to default booting entry
For me, this is Serial console. After reboot, you can check cat /proc/cmdline | grep debug
I just realized this isn’t a VyOS issue I don’t think. I used the config from my other router which had the HW IDs for the NICs in that device. I’ll clear those out and reboot. Working on the debug info right now.
At this point, after deleting the incorrect HW IDs, the only errors showing up /tmp/boot-config-trace are for missing vpn certs and stuff that I didn’t inport yet. The interfaces are now in order and everything is mapped the way it should be. I will get this router connected to the main network and see how everything goes. As best as I can tell, all should be well.
I do apologize for the mental oversight on my part with regard to the HW IDs. As soon as you mentioned it, I realized I screwed this one up. Sorry for taking time from you for such a dumb mistake. But, of course, thank you for your time and help!
All part of the learning process, I had a similar issue as I misunderstood what the hw-id was and incorrectly used it to try and specify a MAC address for my WAN interface.
It does at least allow you to “physically” re-order your NICs so they are easier to work with.
How is the Netgate hardware with VyOS? I am running a board from pcengines, PC Engines apu2e4 product file, which for my home lab/internet is more than adequate. Cheap as well.
@phillipmcmahon very true in the ordering front. I’ve just gotten used to the order as detected by VyOS but I could certainly change the assignments so that eth0 - eth5 follow a more tradi order.
Netgate hardware works very well. It’s a passively cooled desktop router with plenty of horsepower for my OpenVPN needs and other things. I still use pfse in a couple other applications so I’m happy to help support that project as well. I’ve noticed your name on the edgemax forums as well. I use ubiquiti APs and switches and check in on the state of edgemax development as it’s so similar to VyOS.
I’ve also had a mix of edgerouter and edgeswitch hardware over the years with my fair share of “please help” posts on the edgemax forums.
My config works pretty fine using either EdgeOS or VyOS. I do see Ubiquiti slowing the pace on their edgemax devices over the last 18 months and thought I would give VyOS a go. Initially it was a no-go, my ISP requires a specific DHCP option to be sent which VyOS didn’t support. A feature request for that, saw it implemented and my migration journey began. In all my time on the edgemax foums, I have yet to see a feature request directly implemented…
VyOS doesn’t play too well with dynamic WAN addresses as it hasn’t implemented some features present in EdgeOS. I was about to give up, but realised a cost-neutral move to a static IP would solve this issue too.
While my use cases are served perfectly well on either platform, I have more interest now in where VyOS is going. Development seems highly active and I have high hopes of that continuing.
Which dynamic wan features have you had issues with?
I’ve been intrigued with the CLI type router configuration options that edgemax offers but was disappointed by the performance of the MIPS hardware for OpenVPN and the number of things that turn off hardware offloading if you use them. Them I realized I could have the best of both worlds by using VyOS on x86 hardware.
100% agree with you on the development pace of VyOS versus edgemax. Additionally, Ubiquiti has really done a poor job with QA with a good number of the recent releases, not to mention some of the outstanding issues like the packet reordering one that have existed for years now and haven’t been fixed due to the chip vendor not releasing the needed code/patch.
