1.5 night builds: No ethernet adapters avalable in Citrix Hypervisor 8.2 Environment

Hi there,

Start topic is there https://forum.vyos.io/t/cant-move-beyond-1-5-rolling-202409160007/15458/6

The problem is - any night build created after 16 september 2024 could not activate ethernet adapter.

Debian 12 image installed without problem.
Any Vyos 1.5 prior 16 september 2024 works fine.

Is there any idea how to fix issue?

Thanks

Is it possible to do a “sudo lsmod” in a working version, and then a “sudo lsmod” in a non-working version?

It should be possible to pinpoint if there’s a module missing that needs to be re-added to to the build.

Hi,
Here is output for working system (1.5-rolling-202406250020)

Module                  Size  Used by
nft_masq               12288  2
nf_nat_tftp            12288  0
nf_conntrack_tftp      20480  3 nf_nat_tftp
nf_nat_sip             16384  0
nf_conntrack_sip       45056  5 nf_nat_sip
nf_nat_pptp            16384  0
nf_conntrack_pptp      24576  2 nf_nat_pptp
nf_nat_h323            24576  0
nf_conntrack_h323      81920  5 nf_nat_h323
nf_nat_ftp             16384  0
nf_conntrack_ftp       20480  3 nf_nat_ftp
af_packet              65536  2
nft_ct                 24576  23
nft_chain_nat          12288  5
nf_nat                 57344  7 nf_nat_ftp,nf_nat_tftp,nf_nat_pptp,nft_masq,nf_nat_h323,nft_chain_nat,nf_nat_sip
nf_tables             356352  189 nft_ct,nft_masq,nft_chain_nat
nfnetlink_cthelper     16384  6
nf_conntrack          192512  14 nf_nat,nf_conntrack_tftp,nfnetlink_cthelper,nft_ct,nf_nat_ftp,nf_conntrack_pptp,nf_nat_tftp,nf_conntrack_sip,nf_conntrack_h323,nf_nat_pptp,nf_conntrack_ftp,nft_masq,nf_nat_h323,nf_nat_sip
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         12288  1 nf_conntrack
nfnetlink              20480  2 nfnetlink_cthelper,nf_tables
xenfs                  12288  1
xen_privcmd            28672  1 xenfs
intel_rapl_common      36864  0
crct10dif_pclmul       12288  1
crc32_pclmul           12288  0
ghash_clmulni_intel    16384  0
sha512_ssse3           45056  0
sha512_generic         16384  1 sha512_ssse3
sha256_ssse3           32768  0
sha1_ssse3             32768  0
aesni_intel           356352  0
crypto_simd            16384  1 aesni_intel
binfmt_misc            28672  1
cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
pcspkr                 12288  0
sg                     36864  0
evdev                  28672  6
button                 24576  0
tcp_bbr                16384  18
sch_fq_codel           20480  7
mpls_iptunnel          16384  0
mpls_router            49152  1 mpls_iptunnel
ip_tunnel              36864  1 mpls_router
br_netfilter           36864  0
bridge                368640  1 br_netfilter
stp                    12288  1 bridge
llc                    16384  2 bridge,stp
fuse                  180224  1
efi_pstore             12288  0
configfs               61440  1
ip_tables              36864  0
x_tables               61440  1 ip_tables
autofs4                57344  2
usb_storage            81920  0
ohci_hcd               65536  0
sd_mod                 65536  0
squashfs               81920  1
lz4_decompress         24576  1 squashfs
loop                   32768  2
overlay               196608  2
ext4                 1056768  1
crc16                  12288  1 ext4
mbcache                16384  1 ext4
jbd2                  192512  1 ext4
nls_cp437              16384  0
vfat                   20480  0
fat                    90112  1 vfat
efivarfs               24576  0
nls_ascii              12288  0
hid_generic            12288  0
usbhid                 69632  0
hid                   159744  2 usbhid,hid_generic
sr_mod                 28672  0
cdrom                  77824  1 sr_mod
xen_netfront           45056  3
xen_blkfront           49152  2
ata_piix               45056  0
crc32c_intel           16384  2
libata                438272  1 ata_piix
uhci_hcd               57344  0
i2c_piix4              28672  0
scsi_mod              303104  5 sd_mod,usb_storage,libata,sg,sr_mod
scsi_common            16384  6 scsi_mod,sd_mod,usb_storage,libata,sg,sr_mod
ehci_hcd              106496  0

For system with mentioned issue I don’t able to make the text copy from hypervisor console only printscreens as below


Знімок екрана_2-11-2024_151619_10.1.0.56


Hope this would help
Thanks

A lspci -v for both the working and the non-working installation would be handy aswell.

Here they are

Old, working

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
        Subsystem: Red Hat, Inc. Qemu virtual machine
        Flags: bus master, fast devsel, latency 0

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
        Subsystem: Red Hat, Inc. Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0

00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] (prog-if 80 [ISA Compatibility mode-only controller, supports bus mastering])
        Subsystem: XenSource, Inc. 82371SB PIIX3 IDE [Natoma/Triton II]
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0
        I/O ports at 01f0 [size=8]
        I/O ports at 03f4
        I/O ports at 0170 [size=8]
        I/O ports at 0374
        I/O ports at c100 [size=16]
        Kernel driver in use: ata_piix
        Kernel modules: ata_piix

00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) (prog-if 00 [UHCI])
        Subsystem: XenSource, Inc. 82371SB PIIX3 USB [Natoma/Triton II]
        Physical Slot: 1
        Flags: bus master, fast devsel, latency 0, IRQ 23
        I/O ports at c1c0 [size=32]
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci_hcd

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
        Subsystem: Red Hat, Inc. Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0, IRQ 9
        Kernel modules: i2c_piix4

00:02.0 VGA compatible controller: Cirrus Logic GD 5446 (prog-if 00 [VGA controller])
        Subsystem: XenSource, Inc. GD 5446
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0
        Memory at f0000000 (32-bit, prefetchable) [size=32M]
        Memory at f3060000 (32-bit, non-prefetchable) [size=4K]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]

00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
        Subsystem: XenSource, Inc. Xen Platform Device
        Physical Slot: 3
        Flags: bus master, fast devsel, latency 0, IRQ 28
        I/O ports at c000 [size=256]
        Memory at f2000000 (32-bit, prefetchable) [size=16M]
        Kernel driver in use: xen-platform-pci

New, no ethernet



xen_netfront appears in both of them, I’m fairly sure that’s the ethernet driver you’ll be using.

On the working system can you do
ls /sys/class/net
and
sudo dmesg | grep net

And show us the output. Can you then do ethtool <the nic> i.e. ethtool eth0 if eth0 is the Interface you’re having issues with and then ethtool -i <the nic>

Then on the non-working machine can you do the same
ls /sys/class/net and sudo dmesg | grep net
And if the interface exists do the same ethtool <the nic> and ethtool -i <the nic>

Are you sure that the “old, working” is a correct paste since I dont see any network adapter being used there?

Sure, it looks very strange but it did.

$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address         MAC                VRF        MTU  S/L    Description
-----------  -----------------  -----------------  -------  -----  -----  -----------------------
eth0         10.1.0.15/24       da:8c:3e:4b:c2:b9  default   1500  u/u    Local network Interface
eth1         xx.xx.xx.xx/25     da:10:44:d6:57:a9  default   1500  u/u    ISP if
eth2         10.0.0.115/24      0e:d6:aa:22:53:3a  default   1500  u/u    DMZ
lo           127.0.0.1/8        00:00:00:00:00:00  default  65536  u/u
             ::1/128

Interfaces correctly recieved IP adresses and there were no issue.

Working system…

$ show system image
Name                      Default boot    Running
------------------------  --------------  ---------
1.5-rolling-202410280007
1.5-rolling-202406250020  Yes             Yes

$ ls /sys/class/net
eth0  eth1  eth2  lo  pim6reg

$ sudo dmesg | grep net
[    0.000000] Command line: BOOT_IMAGE=/boot/1.5-rolling-202406250020/vmlinuz boot=live rootdelay=5 noautologin net.ifnames=0 biosdevname=0 vyos-union=/boot/1.5-rolling-202406250020 console=tty0
[    0.036130] Kernel command line: BOOT_IMAGE=/boot/1.5-rolling-202406250020/vmlinuz boot=live rootdelay=5 noautologin net.ifnames=0 biosdevname=0 vyos-union=/boot/1.5-rolling-202406250020 console=tty0
[    0.706664] audit: initializing netlink subsys (disabled)
[    1.414550] drop_monitor: Initializing network drop monitor service
[    2.091693] xen_netfront: Initialising Xen virtual ethernet driver
[   17.711901] systemd[1]: Reached target network-online.target - Network is Online.
[   17.731418] systemd[1]: Reached target stunnel.target - TLS tunnels for network services - per-config-file target.
[   18.200665] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   21.445423] process '/usr/sbin/netplugd' started with executable stack

$ ethtool -i eth0
driver: vif
version: 6.6.35-amd64-vyos
firmware-version:
expansion-rom-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Same VM with issue - rolling update after 20240916




According to linux/drivers/net/xen-netfront.c at master · torvalds/linux · GitHub the module is named:

MODULE_DESCRIPTION("Xen virtual network device frontend");
MODULE_LICENSE("GPL");
MODULE_ALIAS("xen:vif");
MODULE_ALIAS("xennet");

Looking through vyos-build/scripts/package-build/linux-kernel/arch/x86/configs/vyos_defconfig at current · vyos/vyos-build · GitHub we can see:

CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_VMXNET3=m
# CONFIG_FUJITSU_ES is not set
CONFIG_USB4_NET=m
CONFIG_HYPERV_NET=m
# CONFIG_NETDEVSIM is not set
CONFIG_NET_FAILOVER=m
# CONFIG_ISDN is not set

So I would expect it to exist in the latest nightly. Question is if there is some kind of broken dependency?

If you compare something like this between the two boxes?

sudo find / -iname "*netfront*"

Nothing obvious dependencywise that I can detect over at:

https://cateee.net/lkddb/web-lkddb/XEN_NETDEV_FRONTEND.html

Just to clarify, 1.5-rolling-202411030007 has the same problem

Working system looks almost the same

~$ sudo find / -iname "*netfront*"
find: File system loop detected; ‘/sys/kernel/debug/pinctrl’ is part of the same file system loop as ‘/sys/kernel/debug’.
/sys/module/xen_netfront
/usr/lib/live/mount/rootfs/1.5-rolling-202406250020.squashfs/usr/lib/modules/6.6.35-amd64-vyos/kernel/drivers/net/xen-netfront.ko
/usr/lib/modules/6.6.35-amd64-vyos/kernel/drivers/net/xen-netfront.ko

Seems like xen_netfront exists in both and is also running according to previous posts.

You could try on both using “sudo journalctl -b” or “dmesg” to find out how the lines looks like right after xen_netfront: Initialising Xen virtual ethernet driver if there is some warning or error there?

Other than that it feels like its 2 different VM-guests where you in the one who doesnt work forgot to enable this virtual adapter in the config of the VM-config within the VM-host.

You can also try (as troubleshoot) to manually edit /config/config.boot (take a backup first both locally and through scp or such) and in there on the one who doesnt work remove the “hw-id” lines to see if VyOS will autodetect your eth0 adapter?

I assume if the virtual adapter at the VM-host got a new MAC-address the hw-id in VyOS will try to map eth0 to a non existing MAC-address which obviously will fail and you will end up with no eth0 (I think).

Hi, thanks.
I would try to investigate more accordingly to your thoughts.

You can also try (as troubleshoot) to manually edit /config/config.boot (take a backup first both locally and through scp or such) and in there on the one who doesnt work remove the “hw-id” lines to see if VyOS will autodetect your eth0 adapter?

But definitely it doesn’t help.
Clean installation of any rolling update after 20240915 doesn’t accept simple configuration like

$configure
$set interface ethernet eth0 address xx.xx.xx.xx/24
$commit

I mean that system doesn’t see the existing ethx despite they are all available for configure.

Other than that it feels like its 2 different VM-guests where you in the one who doesnt work forgot to enable this virtual adapter in the config of the VM-config within the VM-host.

I do that on the same VM with all the same parameters.
Simple “sudo reboot” to different images gives absolutely different result.
Old system working, new one couldn’t configure existing ethx.

I have tried different types of VM, make the clean install, check is there any incomaptibility with Debian 12 VM… Nothing helps.
All VM’s work absolutely similar.

Yeah if you reboot using the same VM guest then we can rule out that something is missing at the VM-host.

Even if VyOS is (currently) based on Debian 12 (bookworm) VyOS still uses its own custom kernel based on (if we speak about the nightly builds) latest longterm release (6.6.59 as of writing) according to https://kernel.org/ - and its often fairly quickly built when a new release pops up at kernel.org.

Current defconfig is available at:

Perhaps a crossref should be done vs the official Debian 12 debconfig in case Debian 12 when you boot the same VM-guest on a livecd works?

This seems to be the current Debian Linux Kernel config:

However it seems only to contain diffs from the kernel.org defaults (compared to the config used by VyOS who seems to properly define all available settings if they are y/n/m.

Edit:

And here is the default for 6.6.59:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/x86/configs/x86_64_defconfig?h=v6.6.59

Something I noted is that in the VyOS kernel config this one is unset:

# CONFIG_XEN_VIRTIO is not set

There are a few other virtio options not set aswell which I assume could be related to this case?

Hi, thanks

I’ve checked both dmesg and journalctl output and didn’t anything bad on system based on latest kernel. Everything looks healthy.

But clean install of VYOS and attempt to configure network adapter gives me below error stack, not sure how it would help

Unfortunately, playing with kernel build is far from my level of competence and sometimes i symply don’t understand what i sould do.

at the moment i understand that custom kernel build used in rollings after 20240915 based on latest kernel version 6.6.56 and become incompatible with XEN due to some build options, right?

rolling 20240915 uses 6.6.49 and works fine

Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202409150007
   └ ──┘  current

 * Support portal: https://support.vyos.io
 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Sat Nov  2 14:04:09 2024 from 
[email protected]:~$ uname -a
Linux .local 6.6.49-amd64-vyos #1 SMP PREEMPT_DYNAMIC Mon Sep  9 10:58:53 UTC 2024 x86_64 GNU/Linux

It could be this bug T6764

Hi,

Nice to know i’m not alone…

Edit of pyton file didn’t help

I created a PR with proper fix

2 Likes