Issue with multiple interfaces on Proxmox

General questions
#1

Hi all, I am having a strange issue when adding multiple interfaces to the rolling release when deploying in Proxmox.

Setup has two nics via PCIe passthrough and three virtual nics. Looking at the PCI list in Vyos I see all 5 devices but a “show interfaces” only lists four. Weird thing is if I add an additional virtual nic, a 5th nic does show up. I have not confirmed whether it is the new nic I just added or the original missing nic. It also doesn’t matter if I set the virtual nic types to virtio or E1000.

Another anomaly is that if Vyos is up and running and I add the 5th nic, it shows up as eth4 in the “show interfaces” command, but after a reboot, it is gone…

The release is 1.4-rolling-20226270217

I will get some additional info up if you need it, but I only have access to the console at the moment as I have not yet gotten an IP on any nics.

Here is what I know about the interfaces in Vyos:

eth0 = first pcie passthrough nic
eth1 = second pcie passthrough nic
eth2 = first (net0) virtual nic
eth3 = second (net1) virtual nic
N/A = third (net2) virtual nic : shows up in pci list but not in interfaces
eth4 = fourth (net3) virtual nic : gets added as third vnic, even though it is the fourth

#2

Additional info:

show hardware pci | grep -i ethernet

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5716 Gigabit Ethernet (rev 20)

02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

06:12.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)

06:13.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)

06:14.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)

show interfaces 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             -                                 u/D  Comcast 2.5Gb Interface
eth1             -                                 u/D  Comcast 1Gb Interface
eth2             192.168.1.238/24                  u/u  
eth3             -                                 A/D  
lo               127.0.0.1/8                       u/u  
                 ::1/128

dmesg | grep -i eth
[    1.701060] bnx2 0000:01:00.0 eth0: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 17, node addr 00:0e:b6:56:9f:81
[    1.910318] bnx2 0000:01:00.0 e2: renamed from eth0
[    2.124274] e1000 0000:06:12.0 eth0: (PCI:33MHz:32-bit) 1a:3f:74:b6:5e:ae
[    2.125197] e1000 0000:06:12.0 eth0: Intel(R) PRO/1000 Network Connection
[    2.462420] e1000 0000:06:13.0 eth1: (PCI:33MHz:32-bit) 8e:bd:a7:5b:80:3c
[    2.463333] e1000 0000:06:13.0 eth1: Intel(R) PRO/1000 Network Connection
[    2.791518] e1000 0000:06:14.0 eth2: (PCI:33MHz:32-bit) 4a:48:15:9f:a5:dd
[    2.792469] e1000 0000:06:14.0 eth2: Intel(R) PRO/1000 Network Connection
[    2.796940] e1000 0000:06:13.0 e4: renamed from eth1
[    2.800360] e1000 0000:06:12.0 e3: renamed from eth0
[    2.804699] e1000 0000:06:14.0 e5: renamed from eth2
[    2.900936] ixgbe 0000:02:00.0 e6: renamed from eth0
[   17.215764] bnx2 0000:01:00.0 eth0: renamed from e2
[   17.220553] ixgbe 0000:02:00.0 eth1: renamed from e6
[   18.292258] e1000 0000:06:12.0 eth2: renamed from e3
[   18.308698] e1000 0000:06:14.0 eth3: renamed from e5
[   24.851396] ixgbe 0000:02:00.0: registered PHC device on eth1
[   25.247992] bnx2 0000:01:00.0 eth0: using MSIX
[   25.554933] e1000: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   25.555472] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
#3

More testing and playing has yielded an interesting find.

Having the 2 PCIe pass through and three vnic’s, yields only 4 interfaces in Vyos. If I remove a vnic from the config, which I have not tested before, I see 3 interfaces in Vyos. So the pattern appears to be that that one, last(?), nic is always ignored…

So dropping to one vnic, I still have my 3 interfaces. Adding a second vnic back in, I still only have 3 interfaces showing. Adding a 3rd vnic, I have 4 interfaces. So after the fist vnic, it appears to be that interfaces are 1 less then total network adapters in the VM config.

#4

Try redeploy the instance
Or delete all interface related information from /config/config.boot

I guess there are some old interface data like hw-id.

#5

Only interfaces in /config/config.boot are eth0, eth1, and eth2.

ifconfig under root only shows 3 interfaces. Could this be an underlying linux issue?

root@vyos:~# ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet6 fe80::20e:b6ff:fe56:9f81  prefixlen 64  scopeid 0x20<link>
        ether 00:0e:b6:56:9f:81  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet6 fe80::ec4:7aff:febb:d0f0  prefixlen 64  scopeid 0x20<link>
        ether 0c:c4:7a:bb:d0:f0  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.238  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::183f:74ff:feb6:5eae  prefixlen 64  scopeid 0x20<link>
        ether 1a:3f:74:b6:5e:ae  txqueuelen 1000  (Ethernet)
        RX packets 189550  bytes 25991131 (24.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 52  bytes 8239 (8.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 fe80::200:ff:fe00:0  prefixlen 64  scopeid 0x20<link>
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 14454  bytes 1069596 (1.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14454  bytes 1069596 (1.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0