Hi all, I am having a strange issue when adding multiple interfaces to the rolling release when deploying in Proxmox.
Setup has two nics via PCIe passthrough and three virtual nics. Looking at the PCI list in Vyos I see all 5 devices but a “show interfaces” only lists four. Weird thing is if I add an additional virtual nic, a 5th nic does show up. I have not confirmed whether it is the new nic I just added or the original missing nic. It also doesn’t matter if I set the virtual nic types to virtio or E1000.
Another anomaly is that if Vyos is up and running and I add the 5th nic, it shows up as eth4 in the “show interfaces” command, but after a reboot, it is gone…
The release is 1.4-rolling-20226270217
I will get some additional info up if you need it, but I only have access to the console at the moment as I have not yet gotten an IP on any nics.
Here is what I know about the interfaces in Vyos:
eth0 = first pcie passthrough nic
eth1 = second pcie passthrough nic
eth2 = first (net0) virtual nic
eth3 = second (net1) virtual nic
N/A = third (net2) virtual nic : shows up in pci list but not in interfaces
eth4 = fourth (net3) virtual nic : gets added as third vnic, even though it is the fourth
Additional info:
show hardware pci | grep -i ethernet
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:12.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
06:13.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
06:14.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 - u/D Comcast 2.5Gb Interface
eth1 - u/D Comcast 1Gb Interface
eth2 192.168.1.238/24 u/u
eth3 - A/D
lo 127.0.0.1/8 u/u
::1/128
dmesg | grep -i eth
[ 1.701060] bnx2 0000:01:00.0 eth0: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 17, node addr 00:0e:b6:56:9f:81
[ 1.910318] bnx2 0000:01:00.0 e2: renamed from eth0
[ 2.124274] e1000 0000:06:12.0 eth0: (PCI:33MHz:32-bit) 1a:3f:74:b6:5e:ae
[ 2.125197] e1000 0000:06:12.0 eth0: Intel(R) PRO/1000 Network Connection
[ 2.462420] e1000 0000:06:13.0 eth1: (PCI:33MHz:32-bit) 8e:bd:a7:5b:80:3c
[ 2.463333] e1000 0000:06:13.0 eth1: Intel(R) PRO/1000 Network Connection
[ 2.791518] e1000 0000:06:14.0 eth2: (PCI:33MHz:32-bit) 4a:48:15:9f:a5:dd
[ 2.792469] e1000 0000:06:14.0 eth2: Intel(R) PRO/1000 Network Connection
[ 2.796940] e1000 0000:06:13.0 e4: renamed from eth1
[ 2.800360] e1000 0000:06:12.0 e3: renamed from eth0
[ 2.804699] e1000 0000:06:14.0 e5: renamed from eth2
[ 2.900936] ixgbe 0000:02:00.0 e6: renamed from eth0
[ 17.215764] bnx2 0000:01:00.0 eth0: renamed from e2
[ 17.220553] ixgbe 0000:02:00.0 eth1: renamed from e6
[ 18.292258] e1000 0000:06:12.0 eth2: renamed from e3
[ 18.308698] e1000 0000:06:14.0 eth3: renamed from e5
[ 24.851396] ixgbe 0000:02:00.0: registered PHC device on eth1
[ 25.247992] bnx2 0000:01:00.0 eth0: using MSIX
[ 25.554933] e1000: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 25.555472] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
More testing and playing has yielded an interesting find.
Having the 2 PCIe pass through and three vnic’s, yields only 4 interfaces in Vyos. If I remove a vnic from the config, which I have not tested before, I see 3 interfaces in Vyos. So the pattern appears to be that that one, last(?), nic is always ignored…
So dropping to one vnic, I still have my 3 interfaces. Adding a second vnic back in, I still only have 3 interfaces showing. Adding a 3rd vnic, I have 4 interfaces. So after the fist vnic, it appears to be that interfaces are 1 less then total network adapters in the VM config.
Try redeploy the instance
Or delete all interface related information from /config/config.boot
I guess there are some old interface data like hw-id.
Only interfaces in /config/config.boot are eth0, eth1, and eth2.
ifconfig under root only shows 3 interfaces. Could this be an underlying linux issue?
root@vyos:~# ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet6 fe80::20e:b6ff:fe56:9f81 prefixlen 64 scopeid 0x20<link>
ether 00:0e:b6:56:9f:81 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet6 fe80::ec4:7aff:febb:d0f0 prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:bb:d0:f0 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.238 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::183f:74ff:feb6:5eae prefixlen 64 scopeid 0x20<link>
ether 1a:3f:74:b6:5e:ae txqueuelen 1000 (Ethernet)
RX packets 189550 bytes 25991131 (24.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52 bytes 8239 (8.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 fe80::200:ff:fe00:0 prefixlen 64 scopeid 0x20<link>
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 14454 bytes 1069596 (1.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 14454 bytes 1069596 (1.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0