Container Crashing

jbhardman · October 31, 2024, 4:25am

My network basically went down today. Really, my DNS server went down. I have some devices that don’t use the backup DNS server (2nd assigned in DHCP), so they stopped working.

I jumped in to see the container that I run in VyOS was down. When I started it, it failed to start.

The appropriate log was this:

Oct 30 17:40:35 ruth. systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Oct 30 17:40:35 ruth. podman[12166]: time="2024-10-30T17:40:35-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Oct 30 17:40:35 ruth. podman[12166]: time="2024-10-30T17:40:35-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/6812f576785386d14782e51e970941320e4bd39a11751756676b642dca8ce7d5/merged\": directory not empty"
Oct 30 17:40:35 ruth. podman[12166]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/6812f576785386d14782e51e970941320e4bd39a11751756676b642dca8ce7d5/merged": directory not empty
Oct 30 17:40:35 ruth. podman[12166]: 2024-10-30 17:40:35.771203802 -0600 MDT m=+0.076966472 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Oct 30 17:40:35 ruth. systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Oct 30 17:40:35 ruth. systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Oct 30 17:40:35 ruth. systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Oct 30 17:40:36 ruth. systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 5.
Oct 30 17:40:36 ruth. systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Oct 30 17:40:36 ruth. systemd[1]: vyos-container-technitium.service: Start request repeated too quickly.
Oct 30 17:40:36 ruth. systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Oct 30 17:40:36 ruth. systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.

I’m looking for input on why this happened. I rebooted the whole machine, no help.

Then, I manually went in and deleted all contents inside /usr/lib/live/mount/persistence/container/storage/overlay/6812f576785386d14782e51e970941320e4bd39a11751756676b642dca8ce7d5/merged, since this was the directory it was complaining that wasn’t empty.

As soon as I did “rm -rf *” in that directory, boom, the container came back online.

What does this mean and is there anyway to avoid it in the future?

talmakion · November 1, 2024, 5:00am

I can’t find anything specific to what you’ve seen, but there have been a number of race-condition bugs in rootless podman where containers are running multiple processes that are similar to the problem you’ve seen.

The recommendation from podman is “don’t do that” or at least have a top-level process that manages its own children properly during shutdowns and crashes. There are several patches going back a while fixing ways this problem can occur. If you’re using a recent VyOS, it should be running a recent podman release (4.9.x) so those fixes will be included already.

There are other problems that can cause unmounting overlays to fail - usually these would be actual device problems, like unresponsive block devices or NFS.

What appeared to be in the overlay merged dir?
Was there anything else in the logs which might indicate an unexpected failure/restart of the container, like an OOM-kill or process crash?
Anything in the logs indicating problems with container stop and fs cleanup?
Any other problems with storage?

jbhardman · November 4, 2024, 6:38pm

Happened again this weekend. Now it’s a real problem. I have been running containers on VyOS for 2 years and this specific container for several months. It just started crashing the first time I reported this.

I got the whole entire log here:

Nov 02 18:27:03 ruth conmon[1212719]: conmon 735db8f904500829d888 <nwarn>: Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596.scope/container/memory.events
Nov 02 18:27:03 ruth conmon[1212719]: conmon 735db8f904500829d888 <nwarn>: stdio_input read failed Input/output error
Nov 02 18:27:03 ruth podman[1728250]: 2024-11-02 18:27:03.655192278 -0600 MDT m=+0.067828906 container died 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 (image=docker.io/technitium/dns-server:latest, name=technitium, product=Technitium DNS Server, project_url=https://technitium.com/dns/, vendor=Technitium, PODMAN_SYSTEMD_UNIT=vy>
Nov 02 18:27:03 ruth podman[1728250]: 2024-11-02 18:27:03.66534354 -0600 MDT m=+0.077980193 container restart 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 (image=docker.io/technitium/dns-server:latest, name=technitium, project_url=https://technitium.com/dns/, vendor=Technitium, PODMAN_SYSTEMD_UNIT=vyos-container-technitium.servi>
Nov 02 18:27:03 ruth conmon[1728279]: conmon 735db8f904500829d888 <nwarn>: Failed to get console terminal settings
Nov 02 18:27:03 ruth conmon[1728279]: conmon 735db8f904500829d888 <nwarn>: runtime stderr: executable file `/usr/bin/dotnet` not found in $PATH: No such file or directory
Nov 02 18:27:03 ruth conmon[1728279]: conmon 735db8f904500829d888 <error>: Failed to create container: exit status 1
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 18:27:04 ruth podman[1728294]: 2024-11-02 18:27:04.118847236 -0600 MDT m=+0.086475192 container remove 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 (image=docker.io/technitium/dns-server:latest, name=technitium, PODMAN_SYSTEMD_UNIT=vyos-container-technitium.service, email=support@technitium.com, github_url=https://github.>
Nov 02 18:27:04 ruth podman[1728294]: Error: cleaning up storage: removing container 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 root filesystem: removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Consumed 52.943s CPU time.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 1.
Nov 02 18:27:04 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Consumed 52.943s CPU time.
Nov 02 18:27:04 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:04 ruth podman[1728307]: time="2024-11-02T18:27:04-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:04 ruth podman[1728307]: time="2024-11-02T18:27:04-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:04 ruth podman[1728307]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:04 ruth podman[1728307]: 2024-11-02 18:27:04.502166879 -0600 MDT m=+0.069832787 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:04 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 2.
Nov 02 18:27:04 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:04 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:04 ruth podman[1728332]: time="2024-11-02T18:27:04-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:05 ruth podman[1728332]: time="2024-11-02T18:27:05-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:05 ruth podman[1728332]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:05 ruth podman[1728332]: 2024-11-02 18:27:05.005559916 -0600 MDT m=+0.069534281 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:05 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 3.
Nov 02 18:27:05 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:05 ruth podman[1728356]: time="2024-11-02T18:27:05-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:05 ruth podman[1728356]: time="2024-11-02T18:27:05-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:05 ruth podman[1728356]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:05 ruth podman[1728356]: 2024-11-02 18:27:05.510678703 -0600 MDT m=+0.068903732 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:05 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 4.
Nov 02 18:27:05 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:05 ruth podman[1728378]: time="2024-11-02T18:27:05-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:06 ruth podman[1728378]: time="2024-11-02T18:27:06-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:06 ruth podman[1728378]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:06 ruth podman[1728378]: 2024-11-02 18:27:06.0057577 -0600 MDT m=+0.070361712 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:06 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 5.
Nov 02 18:27:06 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:06 ruth podman[1728400]: time="2024-11-02T18:27:06-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:06 ruth podman[1728400]: time="2024-11-02T18:27:06-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:06 ruth podman[1728400]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:06 ruth podman[1728400]: 2024-11-02 18:27:06.508172184 -0600 MDT m=+0.069816737 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:06 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 6.
Nov 02 18:27:06 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Start request repeated too quickly.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:06 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.

The problem all seems to start with:

Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596.scope/container/memory.events

Here are the contents of the “merged” directory that I empty out to fix it:

root@ruth:/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged# ls -lah
total 32K
drwx------ 8 root root 4.0K Nov  2 18:27 .
drwx------ 5 root root 4.0K Oct 31 08:26 ..
drwxr-xr-t 2 root root 4.0K Nov  2 18:27 dev
drwxr-xr-t 3 root root 4.0K Nov  2 18:27 etc
drwxr-xr-x 3 root root 4.0K Nov  2 18:27 opt
drwxr-xr-x 2 root root 4.0K Nov  2 18:27 proc
drwxr-xr-x 2 root root 4.0K Nov  2 18:27 run
drwxr-xr-x 2 root root 4.0K Nov  2 18:27 sys

Here is my version info:

$ show version
Version:          VyOS 1.5-rolling-202405221805
Release train:    current
Release flavor:   iso

Built by:         <myemail>
Built on:         Wed 22 May 2024 18:05 UTC
Build UUID:       f3017193-ca4d-40a2-a464-814989a9bcd1
Build commit ID:  fad55b3e9790f6

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Protectli
Hardware model:   FW4B
Hardware S/N:     123456789
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors

I build the ISO myself. I am currently building a new ISO that I will upgrade to and see if it fixes anything since this build was back in May.
Does anyone know of anything else I can do? What causes this?

gianry · November 5, 2024, 7:32pm

I’m curious to know which release you have on your router box as I thought to use the container for Unify server . Actually I’m not able to build myself. I’m using the rolling release. I 've to use the router for home usage .
Thanks

jbhardman · November 5, 2024, 9:02pm

I’m not sure what you mean. The last post I put the version that is running on my router. This is a hardware install, not vm or containerized VyOS, but the VyOS system is running a container with it’s built in Podman controls. That’s what is crashing.

gianry · November 6, 2024, 6:07pm

you wrote you build your Vyos iso or do I 'm wrong ? I 'd like to know if it is safe to use the last rolling release or build as stable release

jbhardman · November 8, 2024, 3:11pm

I installed a completely new build. Fully current as of a couple of days ago. Crashed again this morning. Ran for almost 24 hours. Very similar log, but I don’t see the “Failed to open cgroups file” error. But yes with the “stdio_input read failed”.

I am going to have to move my DNS container off to another machine in my network. I hate to do it as I want DNS host on the router host. I have all my rules setup that way and it’s going to be a pain to move.

Once again, here’s the entire log. I put several space lines in between where it successfully started up yesterday, and then the first log entry when it began to crash this morning:

vyos@ruth:~$ journalctl -b -u vyos-container-technitium.service
Nov 07 09:00:52 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 07 09:00:52 ruth podman[4848]: time="2024-11-07T09:00:52-07:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 07 09:00:52 ruth podman[4848]: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Nov 07 09:00:53 ruth podman[4848]: 2ec87908fd773c20fb4d07147ea76861bada2f3eb5cc325bfa87809d52941ccb
Nov 07 09:00:53 ruth systemd[1]: Started vyos-container-technitium.service - VyOS Container technitium.


Nov 08 08:01:11 ruth conmon[4861]: conmon 2ec87908fd773c20fb4d <nwarn>: stdio_input read failed Input/output error
Nov 08 08:01:11 ruth conmon[1144933]: conmon 2ec87908fd773c20fb4d <nwarn>: Failed to get console terminal settings
Nov 08 08:01:11 ruth conmon[1144933]: conmon 2ec87908fd773c20fb4d <nwarn>: runtime stderr: executable file `/usr/bin/dotnet` not found in $PATH: No such file or directory
Nov 08 08:01:11 ruth conmon[1144933]: conmon 2ec87908fd773c20fb4d <error>: Failed to create container: exit status 1
Nov 08 08:01:11 ruth systemd[1]: vyos-container-technitium.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 08:01:11 ruth podman[1144948]: Error: cleaning up storage: removing container 2ec87908fd773c20fb4d07147ea76861bada2f3eb5cc325bfa87809d52941ccb root filesystem: removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/b>
Nov 08 08:01:11 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 08 08:01:11 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:11 ruth systemd[1]: vyos-container-technitium.service: Consumed 20.347s CPU time.
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 1.
Nov 08 08:01:12 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Consumed 20.347s CPU time.
Nov 08 08:01:12 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 08 08:01:12 ruth podman[1144961]: time="2024-11-08T08:01:12-07:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 08 08:01:12 ruth podman[1144961]: time="2024-11-08T08:01:12-07:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/b309b>
Nov 08 08:01:12 ruth podman[1144961]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/b309bc173dee27efafb8ff8531f555a2fdf43a9b6d49286ddab3111497e47341/merged": dire>
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:12 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 2.
Nov 08 08:01:12 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:12 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 08 08:01:12 ruth podman[1144985]: time="2024-11-08T08:01:12-07:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 08 08:01:12 ruth podman[1144985]: time="2024-11-08T08:01:12-07:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/b309b>
Nov 08 08:01:12 ruth podman[1144985]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/b309bc173dee27efafb8ff8531f555a2fdf43a9b6d49286ddab3111497e47341/merged": dire>
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 08 08:01:12 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:12 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:13 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 3.
Nov 08 08:01:13 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:13 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 08 08:01:13 ruth podman[1145009]: time="2024-11-08T08:01:13-07:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 08 08:01:13 ruth podman[1145009]: time="2024-11-08T08:01:13-07:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/b309b>
Nov 08 08:01:13 ruth podman[1145009]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/b309bc173dee27efafb8ff8531f555a2fdf43a9b6d49286ddab3111497e47341/merged": dire>
Nov 08 08:01:13 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 08 08:01:13 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:13 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:13 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 4.
Nov 08 08:01:13 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:13 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 08 08:01:13 ruth podman[1145035]: time="2024-11-08T08:01:13-07:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 08 08:01:13 ruth podman[1145035]: time="2024-11-08T08:01:13-07:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/b309b>
Nov 08 08:01:13 ruth podman[1145035]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/b309bc173dee27efafb8ff8531f555a2fdf43a9b6d49286ddab3111497e47341/merged": dire>
Nov 08 08:01:13 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 08 08:01:13 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:13 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:14 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 5.
Nov 08 08:01:14 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:14 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 08 08:01:14 ruth podman[1145059]: time="2024-11-08T08:01:14-07:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 08 08:01:14 ruth podman[1145059]: time="2024-11-08T08:01:14-07:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/b309b>
Nov 08 08:01:14 ruth podman[1145059]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/b309bc173dee27efafb8ff8531f555a2fdf43a9b6d49286ddab3111497e47341/merged": dire>
Nov 08 08:01:14 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 08 08:01:14 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:14 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:14 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 6.
Nov 08 08:01:14 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 08 08:01:14 ruth systemd[1]: vyos-container-technitium.service: Start request repeated too quickly.
Nov 08 08:01:14 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 08 08:01:14 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
vyos@ruth:~$

There’s no indicator as to why. It just seems to fail out of nowhere.

Here’s the new build info in case that helps:

vyos@ruth:~$ show version
Version:          VyOS 1.5-rolling-202411041834
Release train:    current
Release flavor:   generic

Built by:         myemail
Built on:         Mon 04 Nov 2024 18:34 UTC
Build UUID:       d4c4c2a6-d385-4391-93de-a3dba23168c3
Build commit ID:  a900353a5b2b85

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal
Secure Boot:      n/a (BIOS)

Hardware vendor:  Protectli
Hardware model:   FW4B
Hardware S/N:     123456789
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors

vyos@ruth:~$ sudo podman --version
podman version 4.9.5

Any help at all would be greatly appreciated.

artooro · November 9, 2024, 4:20am

I wonder if you have some kind of hardware issue. Most of the VyOS routers I’ve installed run at least one container, several of them are running the UniFi Network software as well which is fairly resource intensive and I’ve not had any issues like this. Sorry this probably doesn’t help much.

tjh · November 9, 2024, 5:12am

Are you sure it’s not the DNS process crashing? That would cause the container to stop, right?

A “container” is basically an isolated process, so you need to debug the process dying IMHO.