Interface renaming in 1.4 rolling

liljenstolpe · July 4, 2021, 7:13am

I have two pcengine APU4 nodes (4 igb interfaces as external ports and one r8169 interface - internal header). On both, the r8169 comes up after the first igb but before the others. This means that the four front interfaces are labeled eth0, eth1, eth2, eth3, but on the host they are eth0, eth2, eth3, eth4, with eth1 being the internal header.

I really want the system names to match the physical labels. On the first system I was able to get the interfaces renamed by changing (I think - it was a few months ago) the mac addresses in the interface statements. Now, when the system boots, there are system log messages about renaming interfaces (using the ‘renameN’ device names).

I’ve tried the same trick on the second machine, but it’s not working. To the point that vyos (show int detail) and the system (ifconfig -a) don’t agree as to which macs are which interfaces.

How do I fix this?

Thx

c-po · July 4, 2021, 10:29am

Hi @liljenstolpe,

I do not see an r8169 NIC

vyos@vyos:~$ lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
02:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

You can manually edit the /config/config.boot file and move the ethernet port sequence. It gets effective after a reboot.

liljenstolpe · July 5, 2021, 3:02am

Here are the intel nics - notice the skip between eth0 and eth2 - where could eth1 have gone to?

[    5.039491] igb: Intel(R) Gigabit Ethernet Network Driver
[    5.045271] igb: Copyright (c) 2007-2014 Intel Corporation.
[    5.095523] igb 0000:01:00.0: added PHC on eth0
[    5.108411] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    5.115789] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:11:e0
[    5.123280] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    5.128743] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    5.189857] igb 0000:02:00.0: added PHC on eth2
[    5.205793] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[    5.205801] igb 0000:02:00.0: eth2: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:11:e1
[    5.205807] igb 0000:02:00.0: eth2: PBA No: FFFFFF-0FF
[    5.205812] igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    5.308885] igb 0000:03:00.0: added PHC on eth3
[    5.321632] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[    5.331888] igb 0000:03:00.0: eth3: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:11:e2
[    5.331898] igb 0000:03:00.0: eth3: PBA No: FFFFFF-0FF
[    5.345384] igb 0000:03:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    5.431612] igb 0000:04:00.0: added PHC on eth4
[    5.431616] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    5.431623] igb 0000:04:00.0: eth4: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:11:e3
[    5.431628] igb 0000:04:00.0: eth4: PBA No: FFFFFF-0FF

I found it… (sorry, typo on the last one r8169).

[    5.146547] r8169 0000:05:00.0 eth1: RTL8168evl/8111evl, 00:13:3b:21:d6:60, XID 2c9, IRQ 39
[    5.146554] r8169 0000:05:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[  119.519036] r8169 0000:05:00.0 eth1: Link is Down
[  121.792762] r8169 0000:05:00.0 eth1: Link is Up - 1Gbps/Full - flow control rx/tx
[  121.792807] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

And here is my REALLY stripped down config…

interfaces {
    ethernet eth0 {
        hw-id 00:0d:b9:50:11:e0
    }
    ethernet eth1 {
        address 192.0.2.254/24
        hw-id 00:0d:b9:50:11:e1
    }
    ethernet eth2 {
        hw-id 00:0d:b9:50:11:e2
    }
    ethernet eth3 {
        hw-id 00:0d:b9:50:11:e3
    }
    ethernet eth4 {
        hw-id 00:13:3b:21:d6:60
    }
}

Here’s what I see on my other vyos box (VyOS 1.4-rolling-202104260417) wrt to ‘renaming’ of the interfaces. This box does what is expected - notice the “rename” sections - not sure what is driving that.

The box that isn’t working is version VyOS 1.4-rolling-202107010537.

[    2.788690] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.788699] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.838146] igb 0000:01:00.0: added PHC on eth0
[    2.838152] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.838160] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:12:f4
[    2.838166] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    2.838172] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    2.881478] igb 0000:02:00.0: added PHC on eth2
[    2.881484] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.881492] igb 0000:02:00.0: eth2: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:12:f5
[    2.881497] igb 0000:02:00.0: eth2: PBA No: FFFFFF-0FF
[    2.881504] igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    2.918630] igb 0000:03:00.0: added PHC on eth3
[    2.918636] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.918671] igb 0000:03:00.0: eth3: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:12:f6
[    2.918677] igb 0000:03:00.0: eth3: PBA No: FFFFFF-0FF
[    2.918683] igb 0000:03:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    2.950059] igb 0000:04:00.0: added PHC on eth4
[    2.950065] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    2.950072] igb 0000:04:00.0: eth4: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:50:12:f7
[    2.950077] igb 0000:04:00.0: eth4: PBA No: FFFFFF-0FF
[    2.950083] igb 0000:04:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[   78.208786] igb 0000:02:00.0 rename4: renamed from eth2
[   78.263594] igb 0000:04:00.0 rename6: renamed from eth4
[   78.331001] igb 0000:03:00.0 eth2: renamed from eth3
[   78.384650] igb 0000:04:00.0 eth3: renamed from rename6
[   78.640695] igb 0000:02:00.0 eth1: renamed from rename4
[  131.712682] igb 0000:01:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[  136.045794] igb 0000:03:00.0 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[  137.296699] igb 0000:02:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
cdl@heimdal-b:~$ sudo dmesg | grep r8169
[    2.839240] libphy: r8169: probed
[    2.840473] r8169 0000:05:00.0 eth1: RTL8168evl/8111evl, 00:13:3b:21:4b:e5, XID 2c9, IRQ 39
[    2.840483] r8169 0000:05:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[   78.608650] r8169 0000:05:00.0 eth4: renamed from eth1
[  130.119211] RTL8211E Gigabit Ethernet r8169-500:00: attached PHY driver [RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=r8169-500:00, irq=IGNORE)

sipvoip · July 5, 2021, 3:57pm

I am running into the same issue when I upgraded from 1.3 to 1.4 rolling yesterday, my config has hw-id for all interfaces, but eth1 and eth2 look switched:

interfaces {
    bonding bond0 {
        address 204.89.189.4/24
        address 204.117.64.4/24
        description "Public Internet"
        hash-policy layer2
        member {
            interface eth0
            interface eth1
        }
        mode 802.3ad
        mtu 9000
        policy {
            route PBR
        }
        vif 2 {
            address 10.0.0.4/23
            description "Vocinity Private"
            mtu 9000
        }
        vif 3 {
            address 10.0.2.4/24
            description "Vocinity Management"
        }
    }
    ethernet eth0 {
        hw-id 00:02:c9:0d:02:28
    }
    ethernet eth1 {
        hw-id 00:02:c9:0d:02:98
    }
    ethernet eth2 {
        address 70.33.172.138/30
        description "Atlantic Metro 10Gig"
        disable-link-detect
        hw-id 00:02:c9:0d:02:29
    }
    ethernet eth3 {
        hw-id 00:02:c9:0d:02:99
    }

Check out how there is a permaddr that is incorrect for eth1 and eth2:

root@router3:/etc/udev/rules.d# show interfaces detail 
bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:02:c9:0d:02:28 brd ff:ff:ff:ff:ff:ff
    inet 204.89.189.4/24 brd 204.89.189.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet 204.117.64.4/24 brd 204.117.64.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::202:c9ff:fe0d:228/64 scope link 
       valid_lft forever preferred_lft forever
    Description: Public Internet

    RX:      bytes  packets  errors  dropped  overrun       mcast
         109759667  1156015       0     1129        0      916960
    TX:      bytes  packets  errors  dropped  carrier  collisions
          20997791   212103       0        7        0           0
bond0.2@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:02:c9:0d:02:28 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.4/23 brd 10.0.1.255 scope global bond0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::202:c9ff:fe0d:228/64 scope link 
       valid_lft forever preferred_lft forever
    Description: Vocinity Private

    RX:    bytes  packets  errors  dropped  overrun       mcast
         5713245    85281       0        0        0       25825
    TX:    bytes  packets  errors  dropped  carrier  collisions
         7012596    57699       0        2        0           0
bond0.3@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:02:c9:0d:02:28 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.4/24 brd 10.0.2.255 scope global bond0.3
       valid_lft forever preferred_lft forever
    inet6 fe80::202:c9ff:fe0d:228/64 scope link 
       valid_lft forever preferred_lft forever
    Description: Vocinity Management

    RX:    bytes  packets  errors  dropped  overrun       mcast
         2443149    31966       0        0        0       25865
    TX:    bytes  packets  errors  dropped  carrier  collisions
            4080       78       0        3        0           0
eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:02:c9:0d:02:28 brd ff:ff:ff:ff:ff:ff

    RX:      bytes  packets  errors  dropped  overrun       mcast
         109546775  1154347       0        5        0      916407
    TX:      bytes  packets  errors  dropped  carrier  collisions
          20928861   211544       0        1        0           0
eth1: <BROADCAST,MULTICAST,SLAVE> mtu 9000 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 00:02:c9:0d:02:28 brd ff:ff:ff:ff:ff:ff permaddr 00:02:c9:0d:02:29

    RX:   bytes  packets  errors  dropped  overrun       mcast
         212952     1669       0      553        0          94
    TX:   bytes  packets  errors  dropped  carrier  collisions
          68930      559       0        0        0           0
eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:02:c9:0d:02:29 brd ff:ff:ff:ff:ff:ff permaddr 00:02:c9:0d:02:98
    inet 70.33.172.138/30 brd 70.33.172.139 scope global eth2

sipvoip · July 5, 2021, 4:02pm

Is there any way to flip eth1 and eth2? config.boot has the MACs they should be, the same when I was running 1.3.

    ethernet eth1 {
        hw-id 00:02:c9:0d:02:98
    }
    ethernet eth2 {
        address 70.33.172.138/30
        description "Atlantic Metro 10Gig"
        disable-link-detect
        hw-id 00:02:c9:0d:02:29
    }

liljenstolpe · July 8, 2021, 5:10am

The config.boot file exactly replicates the config that I posted earlier, and has the desired ordering.

However, no rename is happening during boot (unlike what happens in the 1.4 rolling release running on the working machine

Version: VyOS 1.4-rolling-202104260417

So, it looks like somewhere, we lost the code that does the interface renaming during bringup (using the renameN placeholder name).

Basically, ifconfig or ip int shows the links as discovered, vyos reports the new mac addresses.

The end result is no-one talks.

~~Christopher~~

sipvoip · July 8, 2021, 8:47pm

Yep, I had to go back to 1.3.

liljenstolpe · July 12, 2021, 5:29am

I’ve downgraded to the last 1.3 rolling, and the problem has resolved as well.

liljenstolpe · July 12, 2021, 5:31am

Here are the rename statements that I see in the earlier (April 2021 1.4 rolling) and the latest 1.3 rolling, but NOT the July 2021 1.4 rolling.

[   30.058505] igb 0000:04:00.0 rename6: renamed from eth4
[   30.074228] r8169 0000:05:00.0 eth4: renamed from eth1
[   30.098836] igb 0000:02:00.0 eth1: renamed from eth2
[   30.114813] igb 0000:03:00.0 eth2: renamed from eth3
[   30.162190] igb 0000:04:00.0 eth3: renamed from rename6

sipvoip · July 19, 2021, 10:32pm

Any update on this? Would love to try 1.4 again.

anderson.reloaded · February 3, 2022, 7:16pm

Hello Folks,
I have been facing interface renumbering issue with Vyos 1.4.
Would like to know if this has been fixed ? Else we need to roll back to 1.3 as suggested by @sipvoip

Thanks.

Nikolay · February 5, 2022, 2:05am

Bug fixed
Details here:
https://phabricator.vyos.net/T3871

Daniel_Niasoff · August 29, 2022, 12:05am

Just tested now and it’s still an issue

Seeing this in the logs

kern.log:Aug 28 23:47:19 localhost kernel: [    0.454894] virtio_net virtio1 e2: renamed from eth0
kern.log:Aug 28 23:47:19 localhost kernel: [   14.043917] virtio_net virtio1 eth0: renamed from e2
grep: live: Is a directory
messages:Aug 28 23:47:19 localhost kernel: [    0.454894] virtio_net virtio1 e2: renamed from eth0
messages:Aug 28 23:47:19 localhost kernel: [   14.043917] virtio_net virtio1 eth0: renamed from e2

And cloud-init fails due to the interface being called e2

Created this horrible hack which waits for eth0 to come up before running cloud-init and it seems to be working for now until a permanent fix is made.

What I don’t understand is that you have net.ifnames=0 biosdevname=0 in grub config so predictive interface naming should be disabled?