Just wondering:
If I add a new image and boot it, the only thing that is preserved (persistent) is /config. But our security officer is not happy with the fact /var/log is lost, so forensics is no longer possible.
How are you dealing with this? What is the best way to keep these when upgrading?
I like to setup a Graylog server and then use remote syslog to stream logs over to it.
You should always log to a remote host to keep the audit/log trail
Hmm, yeah, you do have a point.
Graylog server
I’ll look into that.
Due to our specific business, costs are always an issue with proposing something new. I hope the security officer will see the value of centralised logging and we can introduce this.
I looked a Graylog and then cried the amount of dependancies it requires and how big it is. Then I discovered VictoriaMetrics and VictoriaLogs specifically. A single binary that does all the hard work, and it’s free and Open Source. Why more people don’t sing it’s praises I don’t know. I use both now and I’m very impressed.
Not suggesting Graylog isn’t great, it really is. But it seemed huge for a small home user like me, and every other click it was “You need Enterprise to do that!” which started to grate after a while.
Or at least as a single container without dependencies to other applications or containers.
CheckMK is great at this, Zabbix - not so much.
Also OP can tell their security officer that local logs should never be trusted.
This is why you should send logs to a syslog server but also monitor this syslogserver in case logs stops to show up.
Also logs are usually the stuff that occupy most space over time so it would be really shitty if they would be copied on every upgrade since VyOS uses persistent directories 
Functionally, both seem over the top for what we would need, which is basically only centralized syslog + some sort of UI to search it.
We use Zabbix for monitoring and alerting, which works fine. But does indeed not replace log monitoring.
We are aware we need to centralize is, but our todo list is a mile long, but there isn’t a lot of money to do all of it, and a lot is done by volunteers without the correct skills.
I am also aware you don’t want logs in persistent per-image storage, but there are other solutions, I could for example create a seperate log partition and mount that on /var/log, so the logs are independent of the image version running.
So I was basically looking for a short term “fix”, to bridge the gap between now and having a permanent solution, because current complexity has created a reluctance to upgrade and stay current (like with recent tcp CVE’s) which isn’t good either.
CheckMK can accept incoming syslog aswell.
I will however give VictoriaLogs a try
It’s really good. Like a lite version of Graylog.
I’m hardly an expert on the ins & outs of VyOS’ mechanism for multiple boot images and its use of overlayfs, but when I look on one of my test systems on which I performed an upgrade, all of the stuff from previous versions appears to be still stored in the file system, because one needs to be able to bring it all back should it be decided to revert to a previous image. As long as you don’t delete the previous images, I believe you can find your previous versions’ /var/log files in
/usr/lib/live/mount/persistence/boot/image name/rw/var/log
Or, at least, it looks like they’re there to me. Am I wrong?
Yeah thats the beauty of persistent storage.
Its still there but not when doing “show log” because then only logs from current persistent storage shows up.
So if its critical for OP to reach old logs (again, mad move - local logs shouldnt be trusted which is why you should have them sent with syslog to logserver) they are still there until you delete that system image (which you probably normally do after an upgrade or at least keep at most the last image remaining on the box).
One possible improvement would be if logs wouldnt be part of the persitent storage.
I filed this as a feature request over at âš“ T9612 Make logs and container storage NOT being part of the persistent storage also including the container volumemounts (who normally ends up at /config/container) who have the same issue as logs.
Also included if the rearrange would cause a reinstall anyway perhaps also ZFS should be evaluated instead of overlayfs (since ZFS brings us compression, checksums, mirroring, encryption (if wanted) but also snapshots).
Similar to how OPNsense does things.
Drawback with ZFS is higher wear levelling due ot being a CoW filesystem. Which somewhat can be mitigated by using logbias=throughput.
Ref:
I have had very bad experiences with ZFS on Linux, so I’d rather stay away from that if you don’t mind… 
I have only had great experience, I would prefer ZFS over ext4 or the others anyday for production.
Try it in a more complex setup than just a stand-alone disk or 2. Our VM farm was designed to run of a ZFS cluster backend with NFS, on SAS disks. Very instable, constant crashes, resilvering hanging forever, etc. Switched to a drbd + mdadm based design, since then 100% uptime.
But this is going off-topic, my point really was that this is a router, ZFS as a file system is very over the top for a router, unless you’re treating it as a DC-in-a-box with lots of bolted on functionality that need ZFS type storage.
Sounds like you were having some other uses.
ZFS is battleproven through TrueNAS, Proxmox among others also scaling for petabyte storage.
Also used by CERN etc.
ZFS on FreeBSD, like TrueNAS, brillant, stable as a rock. Used it for many years. ZFS on Linux on the other hand, different product.
YMMV, but I wouldn’t want it on any piece of critical infrastructure, like a core router or firewall.
Thats just FUD.
OpenZFS is a unified product for both *bsd and Linux since years.
OpenZFS itself have existed since 2013 where the original ZFS dates back to 2001 (originally by SUN Microsystems aquired by Oracle and then opensourced):
TrueNAS Scale (the linux edition of TrueNAS) have existed since 2021.
In 2025 they moved on and leaving TrueNAS Core (the freebsd edition of TrueNAS) behind:
2025: TrueNAS SCALE reaches full feature/performance parity with CORE and is renamed simply “TrueNAS” — TrueNAS Enterprise branded for commercial appliances, with TrueNAS Community Edition available for free download. TrueNAS SCALE and TrueNAS CORE designations have been discontinued for future products.
If you read my response properly, you’d noticed that my experiences weren’t recent.
Fact remains: ZFS (or btrfs) is not a filesystem that is needed or required on devices like routers or firewalls. Neither functionality requires the storage features such filesystem brings.
And as I also wrote, YMMV if you use VyOS as a DC-in-a-box, and load it with additional functionality that would requires such features, so if implemented, it should be optional imho.