I’m exploring potentially using VyOS as a guest VM on Linode instead of as a Docker container instance as I have been hitherto planning to do. I’m curious: how is persistence of settings and docker images and containers handled through VyOS binary image upgrades?
I’d like to understand the mechanics a bit and how the filesystem structure works as I have not dug into this yet and feel like it would be nice to hear from the community first before fiddling in this space.
I’ve got a couple of VyOSes in Linode and it works OK. Linode backups don’t, though, they’re file-based rather than snapshots. I use a remote commit-archive so haven’t tried to resolve it, someone else may’ve come up with a proper fix.
As far as I can tell, and I’m not fully across it all myself, VyOS is using a read-only squashfs root image and a persistence volume with overlayfs. Every time you install a new system image, a directory matching the image name is created to become the persistent overlay root. All of that is handled in initrd (I believe) before final pivot.
After full boot-up, you can see the structure of the persistent volume under /usr/lib/live/mount/persistence. The image-specific persistent R/W root is at /usr/lib/live/mount/persistence/boot/<image name>/rw/, and directly under the <image name> dir lives all the normal grub boot machinery and squashfs blob.
When you upgrade, the relevant pieces of config are copied forward, so the old config is retained booting into the next but migrations or changes aren’t shared between images. I believe there’s still efforts ongoing to consolidate as much as possible in the VyOS configuration, to simplify that process, but there are still some pieces (like generated SSH server keys) that get copied along with /config & co. There used to be a lot more, like certs and OpenVPN secrets, which are rolled into config.boot for 1.4 & 1.5.
If you’re not familiar with overlayfs, have a read of Overlay Filesystem — The Linux Kernel documentation, it’ll be handy to know if you’re poking around in it.
Edit: Forgot to mention, the container store is separate to the boot-time stuff, under /usr/lib/live/mount/persistence/container/. It isn’t treated like image-specific areas and just persists in place.
Linode requires a ext4 filesystem with no partitions to use backups+images, which VyOS will create a partition when installed.
You can go through Install a custom distribution docs to modify an installation to meet the requirements. I’ve done this and it works fine. For upgrades/new images, you can do add system image, reboot into it, then delete the old image, rm -rf /etc/ssh/ssh_host*, power off, and image it. You can then just spin up a new Linode from that image, upgrade it, image it again, etc.
@talmakion does a good job describing how the persistence configuration is carried forward, this works fine with the above process.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.