Container Crashing (cont)

[jbhardman]

This was closed due to time:

Container Crashing

My network basically went down today. Really, my DNS server went down. I have some devices that don’t use the backup DNS server (2nd assigned in DHCP), so they stopped working. I jumped in to see the container that I run in VyOS was down. When I started it, it failed to start. The appropriate log was this: Oct 30 17:40:35 ruth. systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium... Oct 30 17:40:35 ruth. podman[12166]: time="2024-10-30T17:40:35-06:00" level=warn…

Just wanted to follow-up… I stood up another host in my network and ran the Technitium DNS container there. I modified my DHCP server (running in VyOS) to hand out that new server as the DNS server.
I left the container running on VyOS as well, because it could take a while for all clients to get the updated DNS server via DHCP.

Since the majority of the traffic has moved to the new server… the VyOS hosted container has not crashed! Seems to have been affected somewhat by traffic volumes. It’s now been running for like 3 weeks without a crash.

Unfortunately, I have no idea exactly why.

[tjh]

Surely a container crashing though is just the process IN the container crashing, yea?
A container can’t “crash” as such, only the process in it.

Do you have any logs from VyOS as to why Technitium was stopping?

I don’t doubt it’s related to being run within a container environment, but without logs as to why it kept crashing/stopping it’s hard to diagnose any further.

[jbhardman]

Under most circumstances, I’d agree. The process running is the issue. Unfortunately, there is no indication that the process running is the problem. Absolutely nothing in the Technitium DNS Log file at all. But, the only indication of a problem is from the Podman service. I put the logs on the original post, but it was closed due to inactivity. Here it is again:

Nov 02 18:27:03 ruth conmon[1212719]: conmon 735db8f904500829d888 <nwarn>: Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596.scope/container/memory.events
Nov 02 18:27:03 ruth conmon[1212719]: conmon 735db8f904500829d888 <nwarn>: stdio_input read failed Input/output error
Nov 02 18:27:03 ruth podman[1728250]: 2024-11-02 18:27:03.655192278 -0600 MDT m=+0.067828906 container died 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 (image=docker.io/technitium/dns-server:latest, name=technitium, product=Technitium DNS Server, project_url=https://technitium.com/dns/, vendor=Technitium, PODMAN_SYSTEMD_UNIT=vy>
Nov 02 18:27:03 ruth podman[1728250]: 2024-11-02 18:27:03.66534354 -0600 MDT m=+0.077980193 container restart 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 (image=docker.io/technitium/dns-server:latest, name=technitium, project_url=https://technitium.com/dns/, vendor=Technitium, PODMAN_SYSTEMD_UNIT=vyos-container-technitium.servi>
Nov 02 18:27:03 ruth conmon[1728279]: conmon 735db8f904500829d888 <nwarn>: Failed to get console terminal settings
Nov 02 18:27:03 ruth conmon[1728279]: conmon 735db8f904500829d888 <nwarn>: runtime stderr: executable file `/usr/bin/dotnet` not found in $PATH: No such file or directory
Nov 02 18:27:03 ruth conmon[1728279]: conmon 735db8f904500829d888 <error>: Failed to create container: exit status 1
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 18:27:04 ruth podman[1728294]: 2024-11-02 18:27:04.118847236 -0600 MDT m=+0.086475192 container remove 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 (image=docker.io/technitium/dns-server:latest, name=technitium, PODMAN_SYSTEMD_UNIT=vyos-container-technitium.service, [email protected], github_url=https://github.>
Nov 02 18:27:04 ruth podman[1728294]: Error: cleaning up storage: removing container 735db8f904500829d888a933e6b68ab890be00fa073a0b63950b3563cb1ac596 root filesystem: removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Consumed 52.943s CPU time.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 1.
Nov 02 18:27:04 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Consumed 52.943s CPU time.
Nov 02 18:27:04 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:04 ruth podman[1728307]: time="2024-11-02T18:27:04-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:04 ruth podman[1728307]: time="2024-11-02T18:27:04-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:04 ruth podman[1728307]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:04 ruth podman[1728307]: 2024-11-02 18:27:04.502166879 -0600 MDT m=+0.069832787 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:04 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:04 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 2.
Nov 02 18:27:04 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:04 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:04 ruth podman[1728332]: time="2024-11-02T18:27:04-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:05 ruth podman[1728332]: time="2024-11-02T18:27:05-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:05 ruth podman[1728332]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:05 ruth podman[1728332]: 2024-11-02 18:27:05.005559916 -0600 MDT m=+0.069534281 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:05 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 3.
Nov 02 18:27:05 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:05 ruth podman[1728356]: time="2024-11-02T18:27:05-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:05 ruth podman[1728356]: time="2024-11-02T18:27:05-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:05 ruth podman[1728356]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:05 ruth podman[1728356]: 2024-11-02 18:27:05.510678703 -0600 MDT m=+0.068903732 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:05 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 4.
Nov 02 18:27:05 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:05 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:05 ruth podman[1728378]: time="2024-11-02T18:27:05-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:06 ruth podman[1728378]: time="2024-11-02T18:27:06-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:06 ruth podman[1728378]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:06 ruth podman[1728378]: 2024-11-02 18:27:06.0057577 -0600 MDT m=+0.070361712 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:06 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 5.
Nov 02 18:27:06 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: Starting vyos-container-technitium.service - VyOS Container technitium...
Nov 02 18:27:06 ruth podman[1728400]: time="2024-11-02T18:27:06-06:00" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Nov 02 18:27:06 ruth podman[1728400]: time="2024-11-02T18:27:06-06:00" level=warning msg="Unmounting container \"technitium\" while attempting to delete storage: removing mount point \"/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged\": directory not empty"
Nov 02 18:27:06 ruth podman[1728400]: Error: removing storage for container "technitium": removing mount point "/usr/lib/live/mount/persistence/container/storage/overlay/5ac887c64d2b26929f8bad25888fae07beea461e08aa499462a1b16010d9120d/merged": directory not empty
Nov 02 18:27:06 ruth podman[1728400]: 2024-11-02 18:27:06.508172184 -0600 MDT m=+0.069816737 image pull e7a2f99365d24100d22bc9f8bad374c7cd1cf7008f36a1aa0fc9a9c56967578a docker.io/technitium/dns-server:latest
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Control process exited, code=exited, status=125/n/a
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:06 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Scheduled restart job, restart counter is at 6.
Nov 02 18:27:06 ruth systemd[1]: Stopped vyos-container-technitium.service - VyOS Container technitium.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Start request repeated too quickly.
Nov 02 18:27:06 ruth systemd[1]: vyos-container-technitium.service: Failed with result 'exit-code'.
Nov 02 18:27:06 ruth systemd[1]: Failed to start vyos-container-technitium.service - VyOS Container technitium.

This same thing happened about every day or every other day. Now that I stopped sending it traffic to the DNS process running… it’s stable. I’m usually good at diagnosing issues… but I’m stumped.

[Viacheslav]

Didn’t saw the config
Allow more memory for the container
Good to start with this