Hey,
Tried booting latest rolling release .iso in Proxmox 6.x with pve-qemu-kvm 5.2.0-6 and it’s working like a charm. Tried the same in Proxmox 7.x with pve-qemu-kvm 6.0.0-4 and it fails to boot. Boot starts, but then it freezes in the middle. Tried with multiple different HW setting options, but to no avail. Anyone else had similar issues?
Cheers, Jan Zorz
Interesting… If I create a VM on a “native” proxmox server - then it boots.
root 2233271 18.1 0.3 5687708 834748 ? SLl 12:31 0:58 /usr/bin/kvm -id 1000 -name vyos-test -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/1000.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/1000.pid -daemonize -smbios type=1,uuid=0e23e340-a122-438c-a868-c22faf60e901 -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/1000.vnc,password=on -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device vmgenid,guid=4ebac7b7-a001-443d-9394-4c87bd510a99 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:e743a1945422 -drive file=/mnt/pve/cephfs/template/iso/vyos-1.4-rolling-202110310317-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=io_uring -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=rbd:ssd/vm-1000-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/ssd.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap1000i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=F2:26:50:54:42:76,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102 -machine type=pc+pve0
pve-qemu-kvm 6.0.0-4
When I tried to do the same in nested proxmox7 server (I’m creating labs in nested proxmox servers to contain the possible effects away from main proxmox cluster) - it did not want to boot at all. I’ll re-install prox7 as nested environment and try again - and report back.
Cheers, Jan
Ok, installed proxmox7 as nested environment and created a KVM in completely the same way as I did it in the “native” proxmox cluster - and latest rolling release doesn’t want to boot.
root 3867 179 0.8 4888376 281416 ? SLl 13:22 5:50 /usr/bin/kvm -id 100 -name vyos-test -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=d484a7fc-cecc-46da-954d-92b9476e0e32 -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/100.vnc,password=on -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device vmgenid,guid=54ebfb45-020c-468c-a423-4a8b314a9639 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:a691c48b93e -drive file=/var/lib/vz/template/iso/vyos-1.4-rolling-202110310317-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=io_uring -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/pve/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=C6:08:CF:6D:6B:7B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102 -machine type=pc+pve0
I did not create a serial device in neither of cases, but in first case (native proxmox) it booted.
Worth mentioning that it boots nicely in KVM on nested proxmox 6, though and this is how it’s running there:
root 1297 10.7 1.2 5031132 801764 ? Sl 13:15 1:15 /usr/bin/kvm -id 100 -name vyos-router -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=b0361b63-6c2d-49e5-9d1e-15538be8a98a -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/100.vnc,password -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device vmgenid,guid=89d4d3fc-9170-4179-88b3-95a689352bdd -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:f0fc4ebf46ef -drive if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/pve/vm-100-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=4E:81:52:79:3B:8E,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102 -netdev type=tap,id=net1,ifname=tap100i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=E6:E9:73:23:BA:76,netdev=net1,bus=pci.0,addr=0x13,id=net1 -netdev type=tap,id=net2,ifname=tap100i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=76:33:36:D9:F3:2D,netdev=net2,bus=pci.0,addr=0x14,id=net2 -netdev type=tap,id=net3,ifname=tap100i3,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=02:F9:39:AE:F9:31,netdev=net3,bus=pci.0,addr=0x15,id=net3 -machine type=pc+pve0
Version of KVM in nested proxmox6 is pve-qemu-kvm 5.2.0-6
Version of KVM in nested proxmox7 is pve-qemu-kvm 6.0.0-4
Any idea?
Cheers, Jan
And this is what I see on serial port (I added serial0) on nested proxmox7 VM:
starting serial terminal on interface serial0
[ 0.000000] Linux version 5.10.76-amd64-vyos (vyos_bld@dfde2d25f355) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Thu Oct 28 18:21:31 UTC 2021
[ 0.000000] Command line: BOOT_IMAGE=/live/vmlinuz boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 initrd=/live/initrd.img
[ 0.000000] x86/fpu: x87 FPU will use FXSAVE
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] kvm-clock: cpu 0, msr b4755001, primary cpu clock
[ 0.000001] kvm-clock: using sched offset of 23679112880 cycles
[ 0.000004] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000007] tsc: Detected 2499.998 MHz processor
[ 0.565963] AGP: No AGP bridge found
[ 0.570801] last_pfn = 0x140000 max_arch_pfn = 0x400000000
[ 0.571446] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.571543] last_pfn = 0xbffda max_arch_pfn = 0x400000000
[ 0.574894] found SMP MP-table at [mem 0x000f5a80-0x000f5a8f]
[ 0.575885] RAMDISK: [mem 0x7e6ac000-0x7fffffff]
[ 0.575925] ACPI: Early table checksum verification disabled
[ 0.575930] ACPI: RSDP 0x00000000000F5880 000014 (v00 BOCHS )
[ 0.575975] ACPI: RSDT 0x00000000BFFE1E8A 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.575983] ACPI: FACP 0x00000000BFFE1C6C 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.575991] ACPI: DSDT 0x00000000BFFDF040 002C2C (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.575996] ACPI: FACS 0x00000000BFFDF000 000040
[ 0.576000] ACPI: APIC 0x00000000BFFE1CE0 000080 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.576005] ACPI: SSDT 0x00000000BFFE1D60 0000CA (v01 BOCHS VMGENID 00000001 BXPC 00000001)
[ 0.576010] ACPI: HPET 0x00000000BFFE1E2A 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.576014] ACPI: WAET 0x00000000BFFE1E62 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.576019] ACPI: Reserving FACP table memory at [mem 0xbffe1c6c-0xbffe1cdf]
[ 0.576021] ACPI: Reserving DSDT table memory at [mem 0xbffdf040-0xbffe1c6b]
[ 0.576023] ACPI: Reserving FACS table memory at [mem 0xbffdf000-0xbffdf03f]
[ 0.576025] ACPI: Reserving APIC table memory at [mem 0xbffe1ce0-0xbffe1d5f]
[ 0.576026] ACPI: Reserving SSDT table memory at [mem 0xbffe1d60-0xbffe1e29]
[ 0.576028] ACPI: Reserving HPET table memory at [mem 0xbffe1e2a-0xbffe1e61]
[ 0.576030] ACPI: Reserving WAET table memory at [mem 0xbffe1e62-0xbffe1e89]
[ 0.578390] No NUMA configuration found
[ 0.578392] Faking a node at [mem 0x0000000000000000-0x000000013fffffff]
[ 0.578398] NODE_DATA(0) allocated [mem 0x13fff9000-0x13fffcfff]
[ 0.578425] Zone ranges:
[ 0.578427] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.578429] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.578432] Normal [mem 0x0000000100000000-0x000000013fffffff]
[ 0.578434] Movable zone start for each node
[ 0.578435] Early memory node ranges
[ 0.578437] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.578439] node 0: [mem 0x0000000000100000-0x00000000bffd9fff]
[ 0.578441] node 0: [mem 0x0000000100000000-0x000000013fffffff]
[ 0.578443] Initmem setup node 0 [mem 0x0000000000001000-0x000000013fffffff]
[ 0.579065] On node 0, zone DMA: 1 pages in unavailable ranges
[ 0.579091] On node 0, zone DMA: 97 pages in unavailable ranges
[ 0.608480] On node 0, zone Normal: 38 pages in unavailable ranges
[ 0.616645] ACPI: PM-Timer IO Port: 0x608
[ 0.616680] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.616841] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[ 0.616844] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.616846] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.616848] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.616850] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.616851] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.616857] Using ACPI (MADT) for SMP configuration information
[ 0.616860] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.616881] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[ 0.616965] [mem 0xc0000000-0xfeffbfff] available for PCI devices
[ 0.616966] Booting paravirtualized kernel on KVM
[ 0.616969] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[ 0.620912] setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:2 nr_node_ids:1
[ 0.621759] percpu: Embedded 45 pages/cpu s146328 r8192 d29800 u1048576
[ 0.621857] kvm-guest: stealtime: cpu 0, msr 13bc184c0
[ 0.621888] kvm-guest: PV spinlocks enabled
[ 0.621892] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear)
[ 0.621914] Built 1 zonelists, mobility grouping on. Total pages: 1032035
[ 0.621916] Policy zone: Normal
[ 0.621919] Kernel command line: BOOT_IMAGE=/live/vmlinuz boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 initrd=/live/initrd.img
[ 0.623623] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.624383] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.624479] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.655409] AGP: Checking aperture…
[ 1.182753] AGP: No AGP bridge found
[ 1.206108] Memory: 4006848K/4193760K available (10242K kernel code, 1156K rwdata, 2860K rodata, 1192K init, 812K bss, 186656K reserved, 0K cma-reserved)
[ 1.206118] random: get_random_u64 called from __kmem_cache_create+0x25/0x4c0 with crng_init=0
[ 1.206254] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 1.207348] rcu: Hierarchical RCU implementation.
[ 1.207351] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[ 1.207353] Tracing variant of Tasks RCU enabled.
[ 1.207356] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[ 1.207358] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 1.207420] NR_IRQS: 16640, nr_irqs: 440, preallocated irqs: 16
[ 1.326917] Console: colour VGA+ 80x25
[ 1.525554] printk: console [tty0] enabled
[ 2.305488] printk: console [ttyS0] enabled
[ 2.313998] ACPI: Core revision 20200925
[ 2.320335] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[ 2.335057] APIC: Switch to symmetric I/O mode setup
[ 2.344473] x2apic enabled
[ 2.350779] Switched APIC routing to physical x2apic.
[ 2.375843] …TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 2.384823] tsc: Marking TSC unstable due to TSCs unsynchronized
[ 2.393564] Calibrating delay loop (skipped) preset value… 4999.99 BogoMIPS (lpj=2499998)
[ 2.394561] pid_max: default: 32768 minimum: 301
[ 2.395646] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 2.396591] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 2.400225] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127
[ 2.400560] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0
[ 2.401567] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ 2.402561] Spectre V2 : Mitigation: Full AMD retpoline
[ 2.403560] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[ 2.405664] Freeing SMP alternatives memory: 28K
[ 2.510368] smpboot: CPU0: AMD Common KVM processor (family: 0xf, model: 0x6, stepping: 0x1)
[ 2.510787] Performance Events: AMD PMU driver.
[ 2.511578] … version: 0
[ 2.512561] … bit width: 48
[ 2.513560] … generic registers: 4
[ 2.514567] … value mask: 0000ffffffffffff
[ 2.515559] … max period: 00007fffffffffff
[ 2.516558] … fixed-purpose events: 0
[ 2.517561] … event mask: 000000000000000f
[ 2.518810] rcu: Hierarchical SRCU implementation.
[ 2.519813] smp: Bringing up secondary CPUs …
[ 2.521806] x86: Booting SMP configuration:
[ 2.522561] … node #0, CPUs: #1
[ 1.180591] kvm-clock: cpu 1, msr b4755041, secondary cpu clock
[ 2.533557] kvm-guest: stealtime: cpu 1, msr 13bd184c0
[ 2.555769] smp: Brought up 1 node, 2 CPUs
[ 2.556565] smpboot: Max logical packages: 1
[ 2.557560] smpboot: Total of 2 processors activated (9999.99 BogoMIPS)
[ 2.560877] devtmpfs: initialized
[ 2.567628] x86/mm: Memory block size: 128MB
[ 2.578668] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[ 2.599572] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[ 2.614596] pinctrl core: initialized pinctrl subsystem
[ 2.625230] NET: Registered protocol family 16
[ 2.635763] audit: initializing netlink subsys (disabled)
[ 2.646696] thermal_sys: Registered thermal governor ‘fair_share’
[ 2.646696] thermal_sys: Registered thermal governor ‘bang_bang’
[ 2.648582] audit: type=2000 audit(1636029104.382:1): state=initialized audit_enabled=0 res=1
[ 2.648563] thermal_sys: Registered thermal governor ‘step_wise’
Hello @JanZ , I installed Proxmox Virtual Environment 7.0-11, and VyOS rolling image boot properly.
Could you please capture the screencast, maybe I’m doing this in a bit different way?
Hey,
It boots fine in native proxmox7 environment, but not if proxmox7 is “nested” inside a VM.
If proxmox6 is nested inside a VM - that works, but with nested proxmox7 it fails to load.
Cheers, Jan