Latest rolling is broken - 1.4-rolling-202105020417

Will do that and let you know.

What did you see to deduce that (i.e. what obvious thing staring me in the face did I miss? :smile:

Sadly that did not work. No traffic in or out and only accessible via the ESXi console.

Config had no hw-ids mapped on reboot, see below.

ethernet eth0 {
    address 172.31.255.6/30
    description wan
    duplex full
    ring-buffer {
        rx 4096
        tx 4096
    }
    speed 1000
}
ethernet eth1 {
    address 192.168.68.1/24
    description lan
    duplex auto
    policy {
        route vpn-routing
    }
    ring-buffer {
        rx 4096
        tx 4096
    }
    speed auto
}
ethernet eth2 {
    address 192.168.100.1/24
    description mullvad-us
    duplex auto
    policy {
        route vpn-routing
    }
    ring-buffer {
        rx 4096
        tx 4096
    }
    speed auto
}
ethernet eth3 {
    address 192.168.110.1/24
    description mullvad-gb
    duplex auto
    policy {
        route vpn-routing
    }
    ring-buffer {
        rx 4096
        tx 4096
    }
    speed auto
}

@phillipmcmahon I see in your output that interfaces have wrong binding

eth0
link/ether 00:0c:29:7b:eb:2e brd ff:ff:ff:ff:ff:ff permaddr 00:0c:29:7b:eb:4c
eth1
link/ether 00:0c:29:7b:eb:38 brd ff:ff:ff:ff:ff:ff permaddr 00:0c:29:7b:eb:2e
eth2
link/ether 00:0c:29:7b:eb:42 brd ff:ff:ff:ff:ff:ff permaddr 00:0c:29:7b:eb:38
eth3
link/ether 00:0c:29:7b:eb:4c brd ff:ff:ff:ff:ff:ff permaddr 00:0c:29:7b:eb:42

@jestabro do you have any idea why it can happen?
note: We got the same result on DUT in our LAB.

I can reproduce the issue as well on my home router. Last working was 20210430 as noted. Sometime during boot/around the same time udev runs, one interface is not being renamed.

On the working image, I can see during boot (around system-udevd operations, before config load) that it gets renamed:

May 08 22:53:14 vyos systemd-udevd[705]: Using default interface naming scheme 'v240'.
May 08 22:53:14 vyos kernel: igb 0000:00:11.0 eth4: renamed from eth1
May 08 22:53:14 vyos systemd-udevd[710]: Using default interface naming scheme 'v240'.
May 08 22:53:15 vyos systemd-udevd[719]: Using default interface naming scheme 'v240'.

This isn’t happening on the “broken” images.

Yep, I see it now! Although the config files are correct.

Hopefully, now it can be reproduced a fix shouldn’t be far off.

@Dmitry cf. T3542, just resolved: udev rules were not being added to the image, starting with that very rolling release — note we still have some long outstanding issues, that should be addressed soon, but this bug was certainly not helping :wink:

1 Like

Note that this was just committed, so missed current rolling by a few minutes, but will be available in the next.

2 Likes

Just built it myself and it works!

Confirmed as well via a self-build that all is working. Thanks!

The current rolling version is broken on my config again.

Showing same symptoms as before, not sure where udev rules or routing has got borked again. I had to restore back to 1.3rc4 as need a working connection so didn’t have chance get to get any data.

@Dmitry

@jestabro

Rolling is still broken, routing table looks good, udev rules appear to be in place, yet nothing is working. The device is dead on the network, have to log in via the vmware esxi console. Happy to help diagnose, let me know what you need in advance as it means downtime on my network and faffing about in esxi console.

No issues here on the most recent VyOS 1.4-rolling-202107010537. There was a lot of changes recently it looks with the move from buster to bullseye so you might try again.

What do you mean by “device is dead on the network.” Can’t ping it from lan/ssh I assume? Are your interfaces matching config still? (sudo ip -d link show and configuration hw-id line up)

Can’t ping it on the network, similar to when the udev rules were not applied and the interfaces were incorrectly mapped.

I will check once again that the interfaces are indeed matching the config, and report back.

Interfaces align across the config file and what’s in actual effect. Routing tables looks the same with one small exception. On the rolling release my default route and blackhole routes now all have a weight of 1, 1.3-rc4 doesn’t show any weights on those entries.

1.3-rc4

phillipmcmahon@myrouter:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

S>* 0.0.0.0/0 [1/0] via 172.31.255.5, eth0, 1d15h46m
S>* 10.0.0.0/8 [1/0] unreachable (blackhole), 1d15h46m
C>* 10.0.10.0/24 is directly connected, wg8, 1d15h46m
C>* 10.67.134.85/32 is directly connected, wg0, 1d15h46m
C>* 10.67.136.97/32 is directly connected, wg1, 1d15h46m
S>* 172.16.0.0/12 [1/0] unreachable (blackhole), 1d15h46m
C>* 172.31.255.4/30 is directly connected, eth0, 1d15h46m
S>* 192.168.0.0/16 [1/0] unreachable (blackhole), 1d15h46m
C>* 192.168.32.0/24 is directly connected, wg7, 1d15h46m
C>* 192.168.68.0/24 is directly connected, eth1, 1d15h46m
C>* 192.168.100.0/24 is directly connected, eth2, 1d15h46m
C>* 192.168.110.0/24 is directly connected, eth3, 1d15h46m

1.4-rolling

Too be honest, no idea if that’s significant or not, but it’s the only difference I see.

@jestabro

@Dmitry

Any help here please with diagnosing why 1.4-rolling has suddenly stopped to work with my config?

config-commands.txt (30.4 KB)

Hello @phillipmcmahon ,

Could you provide sudo dmesg | grep eth and some info from the files

 sudo cat /run/udev/log/vyatta-net-name.coldplug

vyatta-net-name.coldplug.log (548 Bytes)
dmesg.txt (876 Bytes)

@phillipmcmahon is it a different VyOS router that you use before?
I see that in your output different mac addresses

Yep, I spun up a new VM to move back to a 1.3-rcx baseline for my config.

Provide please an output sudo ip -d link show from this router