SOLVED No space left on device - /tmp mount full after less than 24 hours

[DePingus]

My /tmp mount gets full after less than 24 hours running. This prevents me from running many commands. Restarting the system clears everything out and gets everything working. But I can’t be restarting daily.

I removed all logging from all my firewall rule sets. But that didn’t help. I’m not sure where to look from here. I’m on vyos-1.4-rolling-202210150526-amd64. Here is some hopefully helpful output:

vyos@vyos:~$ monitor log
-vbash: cannot create temp file for here-document: No space left on device

  Incomplete command: monitor log

vyos@vyos:~$ show version
-vbash: cannot create temp file for here-document: No space left on device

  Incomplete command: show version

vyos@vyos:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.7G     0  1.7G   0% /dev
tmpfs           339M  1.7M  337M   1% /run
/dev/sda3       117G  759M  110G   1% /usr/lib/live/mount/persistence
/dev/loop0      401M  401M     0 100% /usr/lib/live/mount/rootfs/1.4-rolling-202210150526.squashfs
tmpfs           1.7G     0  1.7G   0% /usr/lib/live/mount/overlay
overlay         117G  759M  110G   1% /
tmpfs           1.7G     0  1.7G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.7G  1.7G     0 100% /tmp
tmpfs           1.7G  216K  1.7G   1% /var/tmp
none            1.7G     0  1.7G   0% /etc/cni/net.d
none            1.7G 1020K  1.7G   1% /opt/vyatta/config
unionfs-fuse    1.7G 1020K  1.7G   1% /opt/vyatta/config/tmp/new_config_7980
tmpfs           339M     0  339M   0% /run/user/1002

vyos@vyos:/tmp$ sudo du -cksh *
0	#102 (deleted)
20K	config.boot.2934
20K	config.boot.8206
8.0K	dhcpd.conf
0	systemd-private-f2df832485c54876938f5280928f317b-haveged.service-HpwFTi
0	systemd-private-f2df832485c54876938f5280928f317b-ntp.service-zTJVVg
0	systemd-private-f2df832485c54876938f5280928f317b-pdns-recursor.service-oX4vyf
0	systemd-private-f2df832485c54876938f5280928f317b-systemd-logind.service-C6Wcki
4.0K	vyos-configd-script-stdout
4.0K	vyos-config-status
56K	total

vyos@vyos:~$ sudo df --inodes -h
Filesystem     Inodes IUsed IFree IUse% Mounted on
udev             419K   328  419K    1% /dev
tmpfs            423K   601  423K    1% /run
/dev/sda3        7.5M  1.5K  7.5M    1% /usr/lib/live/mount/persistence
/dev/loop0        68K   68K     0  100% /usr/lib/live/mount/rootfs/1.4-rolling-202210150526.squashfs
tmpfs            423K     1  423K    1% /usr/lib/live/mount/overlay
overlay          7.5M  1.5K  7.5M    1% /
tmpfs            423K     1  423K    1% /dev/shm
tmpfs            423K     2  423K    1% /run/lock
tmpfs            423K    21  423K    1% /tmp
tmpfs            423K    51  423K    1% /var/tmp
none             423K     1  423K    1% /etc/cni/net.d
none                0     0     0     - /opt/vyatta/config
unionfs-fuse        0     0     0     - /opt/vyatta/config/tmp/new_config_7980
tmpfs             85K    17   85K    1% /run/user/1002

vyos@vyos:~$ free -mh
               total        used        free      shared  buff/cache   available
Mem:           3.3Gi       319Mi       951Mi       1.7Gi       2.1Gi       1.1Gi
Swap:             0B          0B          0B

[DePingus]

INTERESTING UPDATE:. I’ve been watching my disk space with df -h since my last reboot 6 hours ago. The /tmp folder has been at 36K usage, without moving, for about 5 of those hours. I just checked now and its full! Its been been less than 90 minutes since I last checked.

So it seems that the /tmp mount is not filling up slowly over time. Its filling up suddenly! Unfortunately, I can’t “show log” to see what’s happening.

MORE UPDATES: I’ve noticed that 1 of the 4 CPUs is pegged at 100%. Upon further investigation, python3 is stuck doing a “vyos.remote import update”. It looks like it’s trying to upload /tmp/config.boot.7807 to my sftp commit archive server.

I’ve killed that python3 process and /tmp jumped back down to 36K usage. The system is now responding normally (without a restart). We have found the culprit! I’m going to delete the commit archive server and see if that prevents that job from running.

[DePingus]

So it seems I had the wrong password on the commit archive server location. And that was causing that python job to freeze.