Cannot Install VyOS After Booting Into Live With PXE

lightthemeide · February 13, 2024, 6:51pm

Hello There!

I am currently trying to figure out the series of steps I need to take in order to automate the install of VyOS onto my bare metal host. I have managed to get the host to successfully PXE boot into VyOS live but I am running into issues with installing VyOS to disk. Has anyone every done this before and have some insight as to what I could be doing wrong here?

Appreciate Any Help!
Thanks!

Goal

Automate the install of VyOS onto a bare metal host using PXE.

Current Error

I am able to get my host to PXE boot into live and am able to SSH into VyOS just fine but when I try to run the install image command, I get the following error.

vyos@vyos:~$ install image
The system is already installed. Please use "add system image" instead.

Troubleshooting Steps

vyos@vyos:~$ add system image http://10.10.10.10/isos/vyos/15/vyos-1.5-rolling-202402120819-amd64.iso
The file is 444.000 MiB.
[#################################################################################################################################################################################] 100%
Validating signature
Signature is valid
Validating image checksums
What would you like to name this image? (Default: 1.5-rolling-202402120819)
Would you like to set the new image as the default one for boot? [Y/n] Y
An active configuration was found. Would you like to copy it to the new image? [Y/n] Y
Copying configuration directory
Would you like to copy SSH host keys? [Y/n] Y
Copying SSH host keys
Copying system image files
Cleaning up
Unmounting target filesystems
Removing temporary files
[Errno 2] No such file or directory: '//boot/grub/grub.cfg'
vyos@vyos:~$ sudo wget http://10.10.10.10/netboot/boot/amd64/vyos/15/boot/grub/grub.cfg -O /boot/grub/grub.cfg
--2024-02-13 18:25:25--  http://10.10.10.10/netboot/boot/amd64/vyos/15/boot/grub/grub.cfg
Connecting to 10.10.10.10:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1606 (1.6K) [application/octet-stream]
Saving to: ‘/boot/grub/grub.cfg’

/boot/grub/grub.cfg                             100%[=======================================================================================================>]   1.57K  --.-KB/s    in 0s

2024-02-13 18:25:25 (17.3 MB/s) - ‘/boot/grub/grub.cfg’ saved [1606/1606]

vyos@vyos:~$ add system image http://10.10.10.10/isos/vyos/15/vyos-1.5-rolling-202402120819-amd64.iso
The file is 444.000 MiB.
[#################################################################################################################################################################################] 100%
Validating signature
Signature is valid
Validating image checksums
What would you like to name this image? (Default: 1.5-rolling-202402120819)
Would you like to set the new image as the default one for boot? [Y/n] Y
An active configuration was found. Would you like to copy it to the new image? [Y/n] Y
Copying configuration directory
Cleaning up
Unmounting target filesystems
Removing temporary files
Error: [Errno 17] File exists: '/boot/1.5-rolling-202402120819/rw/opt/vyatta/etc/config'
vyos@vyos:~$ sudo rm -rf /boot/1.5-rolling-202402120819
vyos@vyos:~$ add system image http://10.10.10.10/isos/vyos/15/vyos-1.5-rolling-202402120819-amd64.iso
The file is 444.000 MiB.
[#################################################################################################################################################################################] 100%
Validating signature
Signature is valid
Validating image checksums
What would you like to name this image? (Default: 1.5-rolling-202402120819)
Would you like to set the new image as the default one for boot? [Y/n] Y
An active configuration was found. Would you like to copy it to the new image? [Y/n] Y
Copying configuration directory
Would you like to copy SSH host keys? [Y/n] Y
Copying SSH host keys
Copying system image files
Cleaning up
Unmounting target filesystems
Removing temporary files
vyos@vyos:~$ install image
The system is already installed. Please use "add system image" instead.
vyos@vyos:~$ show system image
Name    Default boot    Running
------  --------------  ---------

System Information

vyos@vyos:~$ show system image
Name    Default boot    Running
------  --------------  ---------
vyos@vyos:~$ show version
Version:          VyOS 1.5-rolling-202402120819
Release train:    current

Built by:         [email protected]
Built on:         Mon 12 Feb 2024 10:20 UTC
Build UUID:       b1a917a9-cda4-49ad-a327-58321f1067dc
Build commit ID:  8c5299e1cf7c7b

Architecture:     x86_64
Boot via:         livecd
System type:      bare metal

Hardware vendor:  Dell Inc.
Hardware model:   OptiPlex 7050
Hardware S/N:     1234567
Hardware UUID:    4c4c4544-0046-4d10-8052-b3c04f384e32

Copyright:        VyOS maintainers and contributors

PXE Configuration

dnsmasq

root@bootstrapper:/mnt/data# cat /etc/dnsmasq.conf
port=0
interface=eth0
dhcp-range=10.10.10.100,10.10.10.200,12h
dhcp-option=option:router,10.10.10.1
dhcp-option=option:dns-server,1.1.1.1
dhcp-boot=pxelinux.0,bootstrapper,10.10.10.10
# DHCP Reservations For Vyos and Hypervisor Mgmt Interfaces
dhcp-host=d8:9e:f3:7b:f4:a9,10.10.10.1
tftp-root=/mnt/data/netboot
log-facility=/var/log/dnsmasq.log
enable-tftp
log-queries

dnsmasq.log

Feb 13 13:21:04 dnsmasq-dhcp[14725]: DHCPDISCOVER(eth0) d8:9e:f3:7b:f4:a9
Feb 13 13:21:04 dnsmasq-dhcp[14725]: DHCPOFFER(eth0) 10.10.10.1 d8:9e:f3:7b:f4:a9
Feb 13 13:21:08 dnsmasq-dhcp[14725]: DHCPREQUEST(eth0) 10.10.10.1 d8:9e:f3:7b:f4:a9
Feb 13 13:21:08 dnsmasq-dhcp[14725]: DHCPACK(eth0) 10.10.10.1 d8:9e:f3:7b:f4:a9
Feb 13 13:21:08 dnsmasq-tftp[14725]: error 0 TFTP Aborted received from 10.10.10.1
Feb 13 13:21:08 dnsmasq-tftp[14725]: sent /mnt/data/netboot/pxelinux.0 to 10.10.10.1
Feb 13 13:21:08 dnsmasq-tftp[14725]: sent /mnt/data/netboot/pxelinux.0 to 10.10.10.1
Feb 13 13:21:08 dnsmasq-tftp[14725]: sent /mnt/data/netboot/ldlinux.c32 to 10.10.10.1
Feb 13 13:21:08 dnsmasq-tftp[14725]: file /mnt/data/netboot/pxelinux.cfg/44454c4c-4600-104d-8052-b3c04f384e32 not found for 10.10.10.1
Feb 13 13:21:08 dnsmasq-tftp[14725]: sent /mnt/data/netboot/pxelinux.cfg/01-d8-9e-f3-7b-f4-a9 to 10.10.10.1
Feb 13 13:21:09 dnsmasq-tftp[14725]: sent /mnt/data/netboot/boot/amd64/vyos/15/live/vmlinuz to 10.10.10.1
Feb 13 13:21:13 dnsmasq-tftp[14725]: sent /mnt/data/netboot/boot/amd64/vyos/15/live/initrd.img-6.6.16-amd64-vyos to 10.10.10.1

pxelinux.cfg

root@bootstrapper:/mnt/data/netboot/pxelinux.cfg# cat 01-d8-9e-f3-7b-f4-a9
DEFAULT VyOS14

LABEL VyOS14
 KERNEL ::boot/amd64/vyos/15/live/vmlinuz
 APPEND initrd=::boot/amd64/vyos/15/live/initrd.img-6.6.16-amd64-vyos boot=live nopersistence noautologin fetch=http://10.10.10.10/netboot/boot/amd64/vyos/15/live/filesystem.squashfs

VyOS ISO Download Process

root@bootstrapper:/mnt/data/isos/vyos/15# for item in $(curl -s https://api.github.com/repos/vyos/vyos-rolling-nightly-builds/releases/latest | jq -r '.assets[] | select(.name | contains ("iso")) | .browser_download_url'); do wget $item; done
root@bootstrapper:/mnt/data/isos/vyos/15# ls
vyos-1.5-rolling-202402120819-amd64.iso  vyos-1.5-rolling-202402120819-amd64.iso.minisig
root@bootstrapper:/mnt/data/isos/vyos/15# mount -o loop -t iso9660 /mnt/data/isos/vyos/15/vyos-1.5-rolling-202402120819-amd64.iso /media
root@bootstrapper:/mnt/data/isos/vyos/15# rsync -av /media/ /mnt/data/netboot/boot/amd64/vyos/15
root@bootstrapper:/mnt/data/isos/vyos/15# umount /media

zero1three013 · February 13, 2024, 9:48pm

I think this is specifically an issue with 1.5, I am able to PXE boot 1.3.5 LTS no problem per the documentation. Didn’t try 1.4, but If I do 1.5-rolling-202402120819 in the same way it errors out

I can however log in and if I do install image I get the same error that it’s already installed

I use iPXE instead of pxelinux (my config below)

#!ipxe

set base-url http://files.example.net

kernel ${base-url}/vyos/vyos15-rolling-202402120819/vmlinuz initrd=initrd.img boot=live nopersistence noautologin nonetworking fetch=${base-url}/vyos/vyos15-rolling-202402120819/filesystem.squashfs
initrd ${base-url}/vyos/vyos15-rolling-202402120819/initrd.img

boot

lightthemeide · February 13, 2024, 10:32pm

Thank you for taking the time to confirm that you are having the same issues. I was really worried that I was doing something wrong with the preparation of the kernel/initrd/filesystem.

I cannot remember if I was getting this exact same issue with 1.4 or not but that is something I can test. I already have a build environment setup for building 1.4 from source. I was hoping to use 1.4 anyways since it is going to be the next LTS. I will give it a test run and post my results soon.

jestabro · February 14, 2024, 4:11pm

@lightthemeide please open a task in https://vyos.dev/ with your results, when available; thanks !

lightthemeide · February 14, 2024, 7:42pm

Currently swamped with work stuff but during my lunch I built the ISO for 1.3, 1.4, and 1.5. For 1.3 I had to do the following to get the build to work.

sandwich@DESKTOP-O61SI18:~/vyos/1.3/vyos-build$ git diff
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 0a1a547..99fb410 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -648,6 +648,7 @@ RUN apt-get update && apt-get install -y \
       rubygems \
       build-essential
 RUN gem install public_suffix -v 4.0.7
+RUN gem install dotenv -v 2.8.1
 RUN gem install --no-document fpm

 # debmake: a native Debian tool for preparing sources for packaging

Build went smoothly without issue for all three otherwise. Once the ISO-es were built I moved them over to my PXE server and did the same process of mount, rsync, and umount as before. The pxelinux.cfg for 1.3 had to have its initrd be initrd.img-5.4.268-amd64-vyos instead of initrd.img-6.6.16-amd64-vyos but other than that it remained the same.

Results

1.3

I was able to PXE boot the host and get the install image command to work as expected

vyos@vyos:~$ show version

Version:          VyOS 1.3-rolling-202402141840
Release train:    equuleus

Built by:         lightthemeide
Built on:         Wed 14 Feb 2024 18:40 UTC
Build UUID:       a4795237-9981-4d8a-8bbe-ce4f968a628e
Build commit ID:  1488e906e26d6b-dirty

Architecture:     x86_64
Boot via:         livecd
System type:      bare metal

Hardware vendor:  Dell Inc.
Hardware model:   OptiPlex 7050
Hardware S/N:     1234567
Hardware UUID:    4c4c4544-0046-4d10-8052-b3c04f384e32

Copyright:        VyOS maintainers and contributors

1.4

I was able to PXE boot the host but the install image command also failed. At boot time I also saw the missing grub error that zer1three013 posted. The install image command failed in the same way that I described with 1.5 originally.

vyos@vyos:~$ show version
Version:          VyOS 1.4-rolling-202402141840
Release train:    sagitta

Built by:         lightthemeide
Built on:         Wed 14 Feb 2024 18:40 UTC
Build UUID:       31b6b185-0d94-4a88-976b-1522017fa827
Build commit ID:  b874f97f0ebf1a

Architecture:     x86_64
Boot via:         livecd
System type:      bare metal

Hardware vendor:  Dell Inc.
Hardware model:   OptiPlex 7050
Hardware S/N:     3FMR8N2
Hardware UUID:    4c4c4544-0046-4d10-8052-b3c04f384e32

Copyright:        VyOS maintainers and contributors

1.5

Failed in the same way as I posted originally

jestabro · February 15, 2024, 1:08am

Thanks for the details; task opened with the observation that the recent revision of image-tools has a simple, and unnecessary, restriction in the logic; a fix will be tested and the task updated.
https://vyos.dev/T6041

lightthemeide · February 15, 2024, 2:42pm

Awesome thank you for getting this tracked. If you need me to test any changes to either versions feel free to ping me with a branch to build a new ISO with.

jestabro · February 16, 2024, 8:18pm

Thanks, @lightthemeide, and thanks again for the report. The simple fix is in the branch mentioned in the task above, if you have a chance to try it. Installing from PXE boot is restored in my tests and a PR will follow after a few other checks.

lightthemeide · February 17, 2024, 5:54pm

Happy Saturday @jestabro! I can go ahead and test that. How do I tell vyos-build to use that branch?

jestabro · February 17, 2024, 6:15pm

If you are building the iso within a Docker container, you can checkout and build (a branch of) the package vyos-1x under vyos-build/packages. A mention of this is in the docs here:
https://docs.vyos.io/en/latest/contributing/build-vyos.html#packages
So irrespective of your build workflow, the point is to add the vyos-1x_*.deb package in that location.

I will PR and merge the fix shortly, so alternatively, you can just try it from the main branch when complete, and let us know of any issues. Thanks !

jestabro · February 17, 2024, 8:15pm

Merged into current and sagitta.

lightthemeide · February 18, 2024, 3:08pm

I tried to build and it looks like we are getting a new error below. This is from the 1.5 build

vyos_bld@8cff9fbb85fe:/vyos$ sudo ./build-vyos-image iso --architecture amd64 --build-by "lightthemeide"
.
.
.
.
.
All runlevel operations denied by policy
Errors were encountered while processing:
 vyos-intel-ixgbe
E: Sub-process /usr/bin/dpkg returned an error code (1)
E: An unexpected failure occurred, exiting...
P: Begin unmounting filesystems...
P: Saving caches...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Del vyos-1x 1.5dev0-1027-g6f490b48e [1871 kB]
Del vyos-1x-smoketest 1.5dev0-1027-g6f490b48e [389 kB]
Del vyos-1x-vmware 1.5dev0-1027-g6f490b48e [4388 B]
Del podman 4.9.2+ds1-2 [12.2 MB]
vyos_bld@1ab583f6cb1f:/vyos/build$ ls -la build/
total 32
drwxr-xr-x  8 root     root     4096 Feb 18 15:02 .
drwxr-xr-x 12 vyos_bld vyos_bld 4096 Feb 14 18:41 ..
drwxr-xr-x  2 root     root     4096 Feb 18 15:00 auto
drwxr-xr-x  2 root     root     4096 Feb 18 15:02 .build
drwxr-xr-x  6 root     root     4096 Feb 18 15:00 cache
drwxr-xr-x 17 root     root     4096 Feb 18 15:00 chroot
drwxr-xr-x 21 root     root     4096 Feb 18 15:00 config
drwxr-xr-x  3 root     root     4096 Feb 18 15:00 local

Update: It would seem that something was messed up with the docker image. Cleaning it all out and starting fresh appears to have made it work. I will go ahead and try the PXE boot now

sandwich@DESKTOP-O61SI18:~/vyos$ docker images
REPOSITORY        TAG       IMAGE ID       CREATED       SIZE
vyos/vyos-build   current   bd69beeab47e   6 weeks ago   2.86GB
sandwich@DESKTOP-O61SI18:~/vyos$ docker rmi bd69beeab47e
Untagged: vyos/vyos-build:current
Untagged: vyos/vyos-build@sha256:717b2f0cd9ce89e060a35fbe8986dbb6ca4a91837a87949963267346dd145119
Deleted: sha256:bd69beeab47e28b89cdaae6a1accca29a6b1456a9b4c44c6eaa26780113dab1f
Deleted: sha256:860a462ef7306ed0ce65e31615dcff25ca33511c2498ccd63addf5b47a037ecf
.
.
.
.
sandwich@DESKTOP-O61SI18:~/vyos$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
sandwich@DESKTOP-O61SI18:~/vyos$ docker images --all
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
sandwich@DESKTOP-O61SI18:~/vyos$ mkdir 1.5.new
sandwich@DESKTOP-O61SI18:~/vyos$ cd 1.5.new/
sandwich@DESKTOP-O61SI18:~/vyos/1.5.new$ docker pull vyos/vyos-build:current
current: Pulling from vyos/vyos-build
bc0734b949dc: Pull complete
f755ca674ca4: Pull complete
.
.
.
.
Digest: sha256:717b2f0cd9ce89e060a35fbe8986dbb6ca4a91837a87949963267346dd145119
Status: Downloaded newer image for vyos/vyos-build:current
docker.io/vyos/vyos-build:current
sandwich@DESKTOP-O61SI18:~/vyos/1.5.new$ git clone -b current --single-branch https://github.com/vyos/vyos-build
Cloning into 'vyos-build'...
remote: Enumerating objects: 11159, done.
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 11159 (delta 0), reused 1 (delta 0), pack-reused 11157
Receiving objects: 100% (11159/11159), 7.44 MiB | 52.16 MiB/s, done.
Resolving deltas: 100% (6629/6629), done.
sandwich@DESKTOP-O61SI18:~/vyos/1.5.new$ cd vyos-build
sandwich@DESKTOP-O61SI18:~/vyos/1.5.new/vyos-build$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash
Current UID/GID: 1000/1000
vyos_bld@8cff9fbb85fe:/vyos$ sudo make clean
[2024-02-18 15:10:44] lb clean
P: Cleaning chroot
vyos_bld@8cff9fbb85fe:/vyos$ sudo ./build-vyos-image iso --architecture amd64 --build-by "lightthemeide"
.
.
.
.
.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
[2024-02-18 15:18:14] lb source
P: Source stage disabled, skipping
P: Build completed successfully
vyos_bld@8cff9fbb85fe:/vyos$

lightthemeide · February 18, 2024, 3:30pm

PXE is working!!!

vyos@vyos:~$ install image
Welcome to VyOS installation!
This command will install VyOS to your permanent storage.
Would you like to continue? [y/N] y
What would you like to name this image? (Default: 1.5-rolling-202402181511)
Please enter a password for the "vyos" user (Default: vyos)
What console should be used by default? (K: KVM, S: Serial, U: USB-Serial)? (Default: K)
Probing disks
1 disk(s) found
The following disks were found:
Drive: /dev/sda (119.2 GB)
Which one should be used for installation? (Default: /dev/sda)
Installation will delete all data on the drive. Continue? [y/N] y
Searching for data from previous installations
No previous installation found
Would you like to use all the free space on the drive? [Y/n] Y
Creating partition table...
Creating temporary directories
Mounting new partitions
Creating a configuration file
Copying system image files
Installing GRUB configuration files
Installing GRUB to the drive
Cleaning up
Unmounting target filesystems
Removing temporary files
The image installed successfully; please reboot now.
vyos@vyos:~$ show version
Version:          VyOS 1.5-rolling-202402181511
Release train:    current

Built by:         lightthemeide
Built on:         Sun 18 Feb 2024 15:11 UTC
Build UUID:       89ad96df-a0f6-47c3-9b46-7a8a89d9cb8d
Build commit ID:  63369b1ac4b537

Architecture:     x86_64
Boot via:         livecd
System type:      bare metal

Hardware vendor:  Dell Inc.
Hardware model:   OptiPlex 7050
Hardware S/N:     1234567
Hardware UUID:    4c4c4544-0046-4d10-8052-b3c04f384e32

Copyright:        VyOS maintainers and contributors

It is worth noting that in order to make this work I had to do legacy (bios) boot and remove nonetworking from the pxelinux.cfg example in your docs in order to get it to be able to do the HTTP fetch of the filesystem.

Thank you for the help!

system · March 19, 2024, 3:30pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.