Hi there,
Start topic is there https://forum.vyos.io/t/cant-move-beyond-1-5-rolling-202409160007/15458/6
The problem is - any night build created after 16 september 2024 could not activate ethernet adapter.
Debian 12 image installed without problem.
Any Vyos 1.5 prior 16 september 2024 works fine.
Is there any idea how to fix issue?
Thanks
Is it possible to do a “sudo lsmod” in a working version, and then a “sudo lsmod” in a non-working version?
It should be possible to pinpoint if there’s a module missing that needs to be re-added to to the build.
Hi,
Here is output for working system (1.5-rolling-202406250020)
Module Size Used by
nft_masq 12288 2
nf_nat_tftp 12288 0
nf_conntrack_tftp 20480 3 nf_nat_tftp
nf_nat_sip 16384 0
nf_conntrack_sip 45056 5 nf_nat_sip
nf_nat_pptp 16384 0
nf_conntrack_pptp 24576 2 nf_nat_pptp
nf_nat_h323 24576 0
nf_conntrack_h323 81920 5 nf_nat_h323
nf_nat_ftp 16384 0
nf_conntrack_ftp 20480 3 nf_nat_ftp
af_packet 65536 2
nft_ct 24576 23
nft_chain_nat 12288 5
nf_nat 57344 7 nf_nat_ftp,nf_nat_tftp,nf_nat_pptp,nft_masq,nf_nat_h323,nft_chain_nat,nf_nat_sip
nf_tables 356352 189 nft_ct,nft_masq,nft_chain_nat
nfnetlink_cthelper 16384 6
nf_conntrack 192512 14 nf_nat,nf_conntrack_tftp,nfnetlink_cthelper,nft_ct,nf_nat_ftp,nf_conntrack_pptp,nf_nat_tftp,nf_conntrack_sip,nf_conntrack_h323,nf_nat_pptp,nf_conntrack_ftp,nft_masq,nf_nat_h323,nf_nat_sip
nf_defrag_ipv6 24576 1 nf_conntrack
nf_defrag_ipv4 12288 1 nf_conntrack
nfnetlink 20480 2 nfnetlink_cthelper,nf_tables
xenfs 12288 1
xen_privcmd 28672 1 xenfs
intel_rapl_common 36864 0
crct10dif_pclmul 12288 1
crc32_pclmul 12288 0
ghash_clmulni_intel 16384 0
sha512_ssse3 45056 0
sha512_generic 16384 1 sha512_ssse3
sha256_ssse3 32768 0
sha1_ssse3 32768 0
aesni_intel 356352 0
crypto_simd 16384 1 aesni_intel
binfmt_misc 28672 1
cryptd 28672 2 crypto_simd,ghash_clmulni_intel
pcspkr 12288 0
sg 36864 0
evdev 28672 6
button 24576 0
tcp_bbr 16384 18
sch_fq_codel 20480 7
mpls_iptunnel 16384 0
mpls_router 49152 1 mpls_iptunnel
ip_tunnel 36864 1 mpls_router
br_netfilter 36864 0
bridge 368640 1 br_netfilter
stp 12288 1 bridge
llc 16384 2 bridge,stp
fuse 180224 1
efi_pstore 12288 0
configfs 61440 1
ip_tables 36864 0
x_tables 61440 1 ip_tables
autofs4 57344 2
usb_storage 81920 0
ohci_hcd 65536 0
sd_mod 65536 0
squashfs 81920 1
lz4_decompress 24576 1 squashfs
loop 32768 2
overlay 196608 2
ext4 1056768 1
crc16 12288 1 ext4
mbcache 16384 1 ext4
jbd2 192512 1 ext4
nls_cp437 16384 0
vfat 20480 0
fat 90112 1 vfat
efivarfs 24576 0
nls_ascii 12288 0
hid_generic 12288 0
usbhid 69632 0
hid 159744 2 usbhid,hid_generic
sr_mod 28672 0
cdrom 77824 1 sr_mod
xen_netfront 45056 3
xen_blkfront 49152 2
ata_piix 45056 0
crc32c_intel 16384 2
libata 438272 1 ata_piix
uhci_hcd 57344 0
i2c_piix4 28672 0
scsi_mod 303104 5 sd_mod,usb_storage,libata,sg,sr_mod
scsi_common 16384 6 scsi_mod,sd_mod,usb_storage,libata,sg,sr_mod
ehci_hcd 106496 0
For system with mentioned issue I don’t able to make the text copy from hypervisor console only printscreens as below
Hope this would help
Thanks
A lspci -v for both the working and the non-working installation would be handy aswell.
Here they are
Old, working
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
Subsystem: Red Hat, Inc. Qemu virtual machine
Flags: bus master, fast devsel, latency 0
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
Subsystem: Red Hat, Inc. Qemu virtual machine
Physical Slot: 1
Flags: bus master, medium devsel, latency 0
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] (prog-if 80 [ISA Compatibility mode-only controller, supports bus mastering])
Subsystem: XenSource, Inc. 82371SB PIIX3 IDE [Natoma/Triton II]
Physical Slot: 1
Flags: bus master, medium devsel, latency 0
I/O ports at 01f0 [size=8]
I/O ports at 03f4
I/O ports at 0170 [size=8]
I/O ports at 0374
I/O ports at c100 [size=16]
Kernel driver in use: ata_piix
Kernel modules: ata_piix
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) (prog-if 00 [UHCI])
Subsystem: XenSource, Inc. 82371SB PIIX3 USB [Natoma/Triton II]
Physical Slot: 1
Flags: bus master, fast devsel, latency 0, IRQ 23
I/O ports at c1c0 [size=32]
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
Subsystem: Red Hat, Inc. Qemu virtual machine
Physical Slot: 1
Flags: bus master, medium devsel, latency 0, IRQ 9
Kernel modules: i2c_piix4
00:02.0 VGA compatible controller: Cirrus Logic GD 5446 (prog-if 00 [VGA controller])
Subsystem: XenSource, Inc. GD 5446
Physical Slot: 2
Flags: bus master, fast devsel, latency 0
Memory at f0000000 (32-bit, prefetchable) [size=32M]
Memory at f3060000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
Subsystem: XenSource, Inc. Xen Platform Device
Physical Slot: 3
Flags: bus master, fast devsel, latency 0, IRQ 28
I/O ports at c000 [size=256]
Memory at f2000000 (32-bit, prefetchable) [size=16M]
Kernel driver in use: xen-platform-pci
New, no ethernet
xen_netfront appears in both of them, I’m fairly sure that’s the ethernet driver you’ll be using.
On the working system can you do
ls /sys/class/net
and
sudo dmesg | grep net
And show us the output. Can you then do ethtool <the nic> i.e. ethtool eth0 if eth0 is the Interface you’re having issues with and then ethtool -i <the nic>
Then on the non-working machine can you do the same
ls /sys/class/net and sudo dmesg | grep net
And if the interface exists do the same ethtool <the nic> and ethtool -i <the nic>
Are you sure that the “old, working” is a correct paste since I dont see any network adapter being used there?
Sure, it looks very strange but it did.
$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address MAC VRF MTU S/L Description
----------- ----------------- ----------------- ------- ----- ----- -----------------------
eth0 10.1.0.15/24 da:8c:3e:4b:c2:b9 default 1500 u/u Local network Interface
eth1 xx.xx.xx.xx/25 da:10:44:d6:57:a9 default 1500 u/u ISP if
eth2 10.0.0.115/24 0e:d6:aa:22:53:3a default 1500 u/u DMZ
lo 127.0.0.1/8 00:00:00:00:00:00 default 65536 u/u
::1/128
Interfaces correctly recieved IP adresses and there were no issue.
Working system…
$ show system image
Name Default boot Running
------------------------ -------------- ---------
1.5-rolling-202410280007
1.5-rolling-202406250020 Yes Yes
$ ls /sys/class/net
eth0 eth1 eth2 lo pim6reg
$ sudo dmesg | grep net
[ 0.000000] Command line: BOOT_IMAGE=/boot/1.5-rolling-202406250020/vmlinuz boot=live rootdelay=5 noautologin net.ifnames=0 biosdevname=0 vyos-union=/boot/1.5-rolling-202406250020 console=tty0
[ 0.036130] Kernel command line: BOOT_IMAGE=/boot/1.5-rolling-202406250020/vmlinuz boot=live rootdelay=5 noautologin net.ifnames=0 biosdevname=0 vyos-union=/boot/1.5-rolling-202406250020 console=tty0
[ 0.706664] audit: initializing netlink subsys (disabled)
[ 1.414550] drop_monitor: Initializing network drop monitor service
[ 2.091693] xen_netfront: Initialising Xen virtual ethernet driver
[ 17.711901] systemd[1]: Reached target network-online.target - Network is Online.
[ 17.731418] systemd[1]: Reached target stunnel.target - TLS tunnels for network services - per-config-file target.
[ 18.200665] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 21.445423] process '/usr/sbin/netplugd' started with executable stack
$ ethtool -i eth0
driver: vif
version: 6.6.35-amd64-vyos
firmware-version:
expansion-rom-version:
bus-info: vif-0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Same VM with issue - rolling update after 20240916
According to linux/drivers/net/xen-netfront.c at master · torvalds/linux · GitHub the module is named:
MODULE_DESCRIPTION("Xen virtual network device frontend");
MODULE_LICENSE("GPL");
MODULE_ALIAS("xen:vif");
MODULE_ALIAS("xennet");
Looking through vyos-build/scripts/package-build/linux-kernel/arch/x86/configs/vyos_defconfig at current · vyos/vyos-build · GitHub we can see:
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_VMXNET3=m
# CONFIG_FUJITSU_ES is not set
CONFIG_USB4_NET=m
CONFIG_HYPERV_NET=m
# CONFIG_NETDEVSIM is not set
CONFIG_NET_FAILOVER=m
# CONFIG_ISDN is not set
So I would expect it to exist in the latest nightly. Question is if there is some kind of broken dependency?
If you compare something like this between the two boxes?
sudo find / -iname "*netfront*"
Nothing obvious dependencywise that I can detect over at:
Just to clarify, 1.5-rolling-202411030007 has the same problem
Working system looks almost the same
~$ sudo find / -iname "*netfront*"
find: File system loop detected; ‘/sys/kernel/debug/pinctrl’ is part of the same file system loop as ‘/sys/kernel/debug’.
/sys/module/xen_netfront
/usr/lib/live/mount/rootfs/1.5-rolling-202406250020.squashfs/usr/lib/modules/6.6.35-amd64-vyos/kernel/drivers/net/xen-netfront.ko
/usr/lib/modules/6.6.35-amd64-vyos/kernel/drivers/net/xen-netfront.ko
Seems like xen_netfront exists in both and is also running according to previous posts.
You could try on both using “sudo journalctl -b” or “dmesg” to find out how the lines looks like right after xen_netfront: Initialising Xen virtual ethernet driver if there is some warning or error there?
Other than that it feels like its 2 different VM-guests where you in the one who doesnt work forgot to enable this virtual adapter in the config of the VM-config within the VM-host.
You can also try (as troubleshoot) to manually edit /config/config.boot (take a backup first both locally and through scp or such) and in there on the one who doesnt work remove the “hw-id” lines to see if VyOS will autodetect your eth0 adapter?
I assume if the virtual adapter at the VM-host got a new MAC-address the hw-id in VyOS will try to map eth0 to a non existing MAC-address which obviously will fail and you will end up with no eth0 (I think).
Hi, thanks.
I would try to investigate more accordingly to your thoughts.
You can also try (as troubleshoot) to manually edit /config/config.boot (take a backup first both locally and through scp or such) and in there on the one who doesnt work remove the “hw-id” lines to see if VyOS will autodetect your eth0 adapter?
But definitely it doesn’t help.
Clean installation of any rolling update after 20240915 doesn’t accept simple configuration like
$configure
$set interface ethernet eth0 address xx.xx.xx.xx/24
$commit
I mean that system doesn’t see the existing ethx despite they are all available for configure.
Other than that it feels like its 2 different VM-guests where you in the one who doesnt work forgot to enable this virtual adapter in the config of the VM-config within the VM-host.
I do that on the same VM with all the same parameters.
Simple “sudo reboot” to different images gives absolutely different result.
Old system working, new one couldn’t configure existing ethx.
I have tried different types of VM, make the clean install, check is there any incomaptibility with Debian 12 VM… Nothing helps.
All VM’s work absolutely similar.
Yeah if you reboot using the same VM guest then we can rule out that something is missing at the VM-host.
Even if VyOS is (currently) based on Debian 12 (bookworm) VyOS still uses its own custom kernel based on (if we speak about the nightly builds) latest longterm release (6.6.59 as of writing) according to https://kernel.org/ - and its often fairly quickly built when a new release pops up at kernel.org.
Current defconfig is available at:
Perhaps a crossref should be done vs the official Debian 12 debconfig in case Debian 12 when you boot the same VM-guest on a livecd works?
This seems to be the current Debian Linux Kernel config:
However it seems only to contain diffs from the kernel.org defaults (compared to the config used by VyOS who seems to properly define all available settings if they are y/n/m.
Edit:
And here is the default for 6.6.59:
Something I noted is that in the VyOS kernel config this one is unset:
# CONFIG_XEN_VIRTIO is not set
There are a few other virtio options not set aswell which I assume could be related to this case?
Hi, thanks
I’ve checked both dmesg and journalctl output and didn’t anything bad on system based on latest kernel. Everything looks healthy.
But clean install of VYOS and attempt to configure network adapter gives me below error stack, not sure how it would help
Unfortunately, playing with kernel build is far from my level of competence and sometimes i symply don’t understand what i sould do.
at the moment i understand that custom kernel build used in rollings after 20240915 based on latest kernel version 6.6.56 and become incompatible with XEN due to some build options, right?
rolling 20240915 uses 6.6.49 and works fine
Welcome to VyOS!
┌── ┐
. VyOS 1.5-rolling-202409150007
└ ──┘ current
* Support portal: https://support.vyos.io
* Documentation: https://docs.vyos.io/en/latest
* Project news: https://blog.vyos.io
* Bug reports: https://vyos.dev
You can change this banner using "set system login banner post-login" command.
VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Sat Nov 2 14:04:09 2024 from
vyos@.local:~$ uname -a
Linux .local 6.6.49-amd64-vyos #1 SMP PREEMPT_DYNAMIC Mon Sep 9 10:58:53 UTC 2024 x86_64 GNU/Linux
Hi,
Nice to know i’m not alone…
I created a PR with proper fix
