/var/tmp is too small for upgrade image

somatica · February 14, 2022, 10:03am

Hi all,

I’m running the VMware image of VyOS.

I have previously updated with an ‘add system image xxx’ and had no issues but this time, I received an out of space message:

vyos@vyos:/$ add system image xxx
Trying to fetch ISO file from xxx…
Downloading…
The file is 467.000 MiB.
Failed to download https://s3.amazonaws.com/s3-us.vyos.io/rolling/current/vyos-1.4-rolling-202201140317-amd64.iso.
OSError: Not enough disk space available in “/var/tmp/install-image.16297”.
Failed to download the ISO file.
Exiting…
vyos@vyos:/$

I deleted old images etc. but the problem remained.
Looking further, the target folder is clearly too small for this 467M download:

vyos@vyos:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 221M 0 221M 0% /dev
tmpfs 48M 2.3M 45M 5% /run
/dev/sda1 9.8G 1.5G 7.8G 16% /usr/lib/live/mount/persistence
/dev/loop0 392M 392M 0 100% /usr/lib/live/mount/rootfs/1.4-rolling-202112041824-.squashfs
tmpfs 236M 0 236M 0% /usr/lib/live/mount/overlay
overlay 9.8G 1.5G 7.8G 16% /
tmpfs 236M 84K 236M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 236M 28K 236M 1% /tmp
tmpfs 236M 200K 235M 1% /var/tmp
none 236M 0 236M 0% /etc/cni/net.d
none 236M 836K 235M 1% /opt/vyatta/config
tmpfs 48M 0 48M 0% /run/user/1002
vyos@vyos:/$

Is there an easy way to extend this filesystem or have I overlooked something here?

Thanks,
James

n.fort · February 14, 2022, 11:16am

Hi @somatica .
You can try this commands:

sudo mount -o remount,size=1G /var/tmp

a.srividya · February 14, 2022, 5:10pm

How much RAM is allocated to your device ?
Alternatively, you can use the following command to download the file and then run add system image command.

$ sudo curl -O https://s3.amazonaws.com/s3-us.vyos.io/rolling/current/vyos-1.4-rolling-202202140317-amd64.iso
$ add system image vyos-1.4-rolling-202202140317-amd64.iso

c-po · February 14, 2022, 5:58pm

Hi all,

there were some 2-4 rolling images where the /var/tmp partition was accidently shrinked which triggered this bug. My fault - sorry!

@somatica any of the above mentioned methods will help you upgrade.

skynw · December 27, 2022, 4:07pm

Aloha,
thnaks to this thread I could manage to uppdate my router today.
Thank you guys.
But

After update I recognized that /var/tmp is still only 242M big.
Is it possible to increase it permanently without new installation?

df -h
Filesystem Size Used Avail Use% Mounted on
udev 225M 0 225M 0% /dev
tmpfs 49M 1.1M 48M 3% /run
/dev/vda1 20G 3.7G 15G 20% /usr/lib/live/mount/persistence
/dev/loop0 377M 377M 0 100% /usr/lib/live/mount/rootfs/1.4-rolling-202212270317.squashfs
tmpfs 242M 0 242M 0% /usr/lib/live/mount/overlay
overlay 20G 3.7G 15G 20% /
tmpfs 242M 84K 242M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 242M 8.0K 242M 1% /tmp
tmpfs 242M 216K 241M 1% /var/tmp
none 242M 0 242M 0% /etc/cni/net.d
none 242M 588K 241M 1% /opt/vyatta/config
tmpfs 49M 0 49M 0% /run/user/1002

df -h /var/tmp/
Filesystem Size Used Avail Use% Mounted on
tmpfs 242M 216K 241M 1% /var/tmp
mreuter@hub02:~$ sh vers
Version: VyOS 1.4-rolling-202212270317
Release train: current

Thanks
Marcel

skynw · June 5, 2023, 10:12am

Hello,
yesterday I upgraded again from version 1.4 20221227 to
latest release
again, my /var/tmp is too small.
Unfortunately on this box I have only 500MB RAM, so I could not used
“sudo mount -o remount,size=1G /var/tmp”
I was able to upgrade instead with curl to another directory.
Question is, is it able to make the /var/tmp partition permament bigger or should I install new from scratch?

~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 788M 1.7M 786M 1% /run
/dev/sda3 117G 14G 98G 12% /usr/lib/live/mount/persistence
/dev/loop0 374M 374M 0 100% /usr/lib/live/mount/rootfs/1.4-rolling-202212170319.squashfs
tmpfs 3.9G 0 3.9G 0% /usr/lib/live/mount/overlay
overlay 117G 14G 98G 12% /
tmpfs 3.9G 84K 3.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 24K 3.9G 1% /tmp
tmpfs 3.9G 216K 3.9G 1% /var/tmp
none 3.9G 0 3.9G 0% /etc/cni/net.d
none 3.9G 904K 3.9G 1% /opt/vyatta/config
tmpfs 788M 0 788M 0% /run/user/1002
:~$ sh vers
Version: VyOS 1.4-rolling-202212170319
Release train: current

Built by: autobuild@vyos.net
Built on: Sat 17 Dec 2022 03:19 UTC
Build UUID: b7edb7cd-de8c-437c-afbe-a81569bd43f1
Build commit ID: f56f9f9d7bc169

Architecture: x86_64
Boot via: installed image
System type: bare metal

Hardware vendor: Protectli
Hardware model: FW6D
Hardware S/N: Default string
Hardware UUID: 03000200-0400-0500-0006-000700080009

Copyright: VyOS maintainers and contributors

thanks
Marcel

raz0rc4 · February 7, 2024, 5:34pm

Hi,

with vyos 1.3.4 to be upgraded to 1.3.5 and a vm with only a 512GB of RAM, this is still an issue.

xxx@xxx:~$ add system image https://xxx/vyos-1.3.5-amd64.iso
Trying to fetch ISO file from https://xxx/vyos-1.3.5-amd64.iso...
Downloading...
The file is 356.000 MiB.
Failed to download https://xxx/vyos-1.3.5-amd64.iso.
OSError: Not enough disk space available in "/var/tmp/install-image.3974".
Failed to download the ISO file.
Exiting...

...

xxx@xxx:~$ df -h | grep /var/tmp
tmpfs           237M  200K  237M   1% /var/tmp
xxx@xxx:~$ show hardware mem
MemTotal:         483764 kB

and even this command worked, the machine hanged for a while and killed the upgrade process.

xxx@xxx:~$ sudo mount -o remount,size=1G /var/tmp
xxx@xxx:~$ df -h | grep /var/tmp
tmpfs           1.0G  200K  1.0G   1% /var/tmp

xxx@xxx:~$ add system image https://xxx/vyos-1.3.5-amd64.iso
Trying to fetch ISO file from https://xxx/vyos-1.3.5-amd64.iso...
Downloading...
The file is 356.000 MiB.
Download complete.########################################################################################______________________________________________________________________________________________________________________________________________________]  41%
Done.
Checking for digital signature file...
Downloading...
The file is 0.308 KiB.
[###############################################################################################################################################################################################################################################################] 100%
Download complete.
Checking digital signature...
Signature check FAILED, trying BACKUP key...
Signature key id in /var/tmp/install-image.4430/vyos-1.3.5-amd64.iso.minisig is 9EA8ECDCBDDCD6D1
but the key id in the public key is 69C20BE1367AEBB0
Digital signature is valid.
free(): double free detected in tcache 2
"/var/tmp/install-image.4430/vyos-1.3.5-amd64.iso" is not a VyOS ISO image file.
Exiting...

and the logs showed this.

Feb 07 17:30:38 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Feb 07 17:30:38 kernel: Call Trace:
Feb 07 17:30:38 kernel:  dump_stack+0x66/0x81
Feb 07 17:30:38 kernel:  dump_header+0x45/0x1d3
Feb 07 17:30:38 kernel:  oom_kill_process.cold.33+0xb/0x10
Feb 07 17:30:38 kernel:  out_of_memory+0x199/0x4c0
Feb 07 17:30:38 kernel:  __alloc_pages_slowpath+0x97e/0xbd0
Feb 07 17:30:38 kernel:  __alloc_pages_nodemask+0x251/0x280
Feb 07 17:30:38 kernel:  pagecache_get_page+0xbb/0x220
Feb 07 17:30:38 kernel:  filemap_fault+0x467/0x8f0
Feb 07 17:30:38 kernel:  ? alloc_set_pte+0xfa/0x560
Feb 07 17:30:38 kernel:  ? xas_load+0x8/0x80
Feb 07 17:30:38 kernel:  ? xas_find+0x173/0x1b0
Feb 07 17:30:38 kernel:  ? filemap_map_pages+0x23f/0x370
Feb 07 17:30:38 kernel:  __do_fault+0x35/0x140
Feb 07 17:30:38 kernel:  __handle_mm_fault+0xed1/0x1540
Feb 07 17:30:38 kernel:  ? __set_current_blocked+0x31/0x50
Feb 07 17:30:38 kernel:  handle_mm_fault+0xd6/0x1b0
Feb 07 17:30:38 kernel:  __do_page_fault+0x23f/0x490
Feb 07 17:30:38 kernel:  page_fault+0x34/0x40
Feb 07 17:30:38 kernel: Mem-Info:
Feb 07 17:30:38 kernel: active_anon:37028 inactive_anon:37235 isolated_anon:0
                         active_file:39 inactive_file:66 isolated_file:0
                         unevictable:2570 dirty:0 writeback:0 unstable:0
                         slab_reclaimable:7135 slab_unreclaimable:16064
                         mapped:1120 shmem:37932 pagetables:643 bounce:0
                         free:2234 free_pcp:78 free_cma:0
Feb 07 17:30:38 kernel: Node 0 active_anon:148112kB inactive_anon:148940kB active_file:156kB inactive_file:264kB unevictable:10280kB isolated(anon):0kB isolated(file):0kB mapped:4480kB dirty:0kB writeback:0kB shmem:151728kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Feb 07 17:30:38 kernel: Node 0 DMA32 free:8936kB min:2688kB low:3360kB high:4032kB active_anon:148112kB inactive_anon:148940kB active_file:156kB inactive_file:276kB unevictable:10280kB writepending:0kB present:511480kB managed:483764kB mlocked:10280kB kernel_stack:3984kB pagetables:2572kB bounce:0kB free_pcp:312kB local_pcp:236kB free_cma:0kB
Feb 07 17:30:38 kernel: lowmem_reserve[]: 0 0 0
Feb 07 17:30:38 kernel: Node 0 DMA32: 149*4kB (UME) 19*8kB (UME) 494*16kB (UME) 5*32kB (E) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8812kB
Feb 07 17:30:38 kernel: 39110 total pagecache pages
Feb 07 17:30:38 kernel: 127870 pages RAM
Feb 07 17:30:38 kernel: 0 pages HighMem/MovableOnly
Feb 07 17:30:38 kernel: 6929 pages reserved
Feb 07 17:30:38 kernel: 0 pages hwpoisoned
Feb 07 17:30:38 kernel: Tasks state (memory values in pages):
Feb 07 17:30:38 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 07 17:30:38 kernel: [    729]     0   729     9613      678    94208        0             0 systemd-journal
Feb 07 17:30:38 kernel: [    735]     0   735    76314    11132   221184        0             0 python3
Feb 07 17:30:38 kernel: [    736]     0   736    30060     4026   118784        0             0 python3
Feb 07 17:30:38 kernel: [    742]     0   742     5540      999    73728        0         -1000 systemd-udevd
Feb 07 17:30:38 kernel: [    759]     0   759    11552      734    81920        0             0 VGAuthService
Feb 07 17:30:38 kernel: [    760]     0   760    49024      882   102400        0             0 vmtoolsd
Feb 07 17:30:38 kernel: [    761]     0   761     2020     1187    53248        0             0 haveged
Feb 07 17:30:38 kernel: [    852]     0   852      580      141    45056        0             0 acpid
Feb 07 17:30:38 kernel: [    858]     0   858      570       18    40960        0             0 atopacctd
Feb 07 17:30:38 kernel: [    860]     0   860     2125      465    53248        0             0 cron
Feb 07 17:30:38 kernel: [    863]   106   863     2171      570    57344        0          -900 dbus-daemon
Feb 07 17:30:38 kernel: [    865]     0   865     4845      749    81920        0             0 systemd-logind
Feb 07 17:30:38 kernel: [    892]     0   892     1371      409    53248        0             0 atd
Feb 07 17:30:38 kernel: [    970]     0   970      572      370    53248        0             0 netplugd
Feb 07 17:30:38 kernel: [   1055]     0  1055     2560      343    65536        0             0 watchfrr
Feb 07 17:30:38 kernel: [   1114]   112  1114   267068     1952   221184        0             0 zebra
Feb 07 17:30:38 kernel: [   1130]   112  1130    48606     1874   135168        0             0 bgpd
Feb 07 17:30:38 kernel: [   1142]   112  1142     7677      858   102400        0             0 ripd
Feb 07 17:30:38 kernel: [   1145]   112  1145     2829      607    61440        0             0 ripngd
Feb 07 17:30:38 kernel: [   1150]   112  1150     8031     1027    98304        0             0 ospfd
Feb 07 17:30:38 kernel: [   1153]   112  1153     7823      938    94208        0             0 ospf6d
Feb 07 17:30:38 kernel: [   1156]   112  1156     3149      722    57344        0             0 isisd
Feb 07 17:30:38 kernel: [   1159]   112  1159     2677      734    53248        0             0 ldpd
Feb 07 17:30:38 kernel: [   1160]   112  1160     2642      746    61440        0             0 ldpd
Feb 07 17:30:38 kernel: [   1161]   112  1161     3030      566    61440        0             0 ldpd
Feb 07 17:30:38 kernel: [   1165]   112  1165     2814      503    61440        0             0 staticd
Feb 07 17:30:38 kernel: [   1168]   112  1168     2808      614    57344        0             0 bfdd
Feb 07 17:30:38 kernel: [   2203]     0  2203     1906      527    53248        0             0 pppd
Feb 07 17:30:38 kernel: [   2311]     0  2311    55432      336    77824        0             0 rsyslogd
Feb 07 17:30:38 kernel: [   2448]   111  2448    19119      506    65536        0             0 ntpd
Feb 07 17:30:38 kernel: [   2479]     0  2479     1313      410    40960        0             0 starter
Feb 07 17:30:38 kernel: [   2480]     0  2480   301417     1151   229376        0             0 charon
Feb 07 17:30:38 kernel: [   2837]     0  2837     3454      545    69632        0         -1000 sshd
Feb 07 17:30:38 kernel: [   2898]     0  2898     1653      437    49152        0             0 login
Feb 07 17:30:38 kernel: [   3012]     0  3012     2722     2572    61440        0             0 atop
Feb 07 17:30:38 kernel: [   3332]     0  3332     3663      755    69632        0             0 sshd
Feb 07 17:30:38 kernel: [   3335]  1004  3335     5255      665    77824        0             0 systemd
Feb 07 17:30:38 kernel: [   3336]  1004  3336    42868      642   106496        0             0 (sd-pam)
Feb 07 17:30:38 kernel: [   3412]  1004  3412     3663      333    69632        0             0 sshd
Feb 07 17:30:38 kernel: [   3413]  1004  3413     2096      987    49152        0             0 vbash
Feb 07 17:30:38 kernel: [   4429]     0  4429     2495      515    53248        0             0 sudo
Feb 07 17:30:38 kernel: [   4430]     0  4430     1759      584    57344        0             0 install-image
Feb 07 17:30:38 kernel: [   4436]     0  4436     9541     5461   106496        0             0 python3
Feb 07 17:30:38 kernel: [   4438]     0  4438     3234      135    57344        0             0 sshd
Feb 07 17:30:38 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/vyos-configd.service,task=python3,pid=735,uid=0
Feb 07 17:30:38 kernel: Out of memory: Killed process 735 (python3) total-vm:305256kB, anon-rss:41876kB, file-rss:2652kB, shmem-rss:0kB, UID:0 pgtables:216kB oom_score_adj:0
Feb 07 17:30:10 ripngd[1145]: [EC 100663313] SLOW COMMAND: command took 6371ms (cpu time 874ms): echo PING
Feb 07 17:30:38 watchfrr[1055]: [EC 100663313] SLOW THREAD: task wakeup_send_echo (559704d6c070) ran for 10480ms (cpu time 684ms)
Feb 07 17:30:38 ripngd[1145]: [EC 100663313] SLOW THREAD: task vtysh_read (7fbe17dac9c0) ran for 16414ms (cpu time 1744ms)
Feb 07 17:30:38 isisd[1156]: [EC 100663313] SLOW COMMAND: command took 15672ms (cpu time 1159ms): echo PING
Feb 07 17:30:38 isisd[1156]: [EC 100663313] SLOW THREAD: task vtysh_read (7f912cebe9c0) ran for 16105ms (cpu time 1166ms)
Feb 07 17:30:38 ldpd[1161]: [EC 100663313] SLOW COMMAND: command took 12390ms (cpu time 977ms): echo PING
Feb 07 17:30:38 ldpd[1161]: [EC 100663313] SLOW THREAD: task vtysh_read (7fdfc1d909c0) ran for 13114ms (cpu time 1005ms)
Feb 07 17:30:38 zebra[1114]: [EC 100663313] SLOW THREAD: task kernel_read (55f7d11accd0) ran for 24514ms (cpu time 2014ms)
Feb 07 17:30:39 systemd[1]: vyos-configd.service: Main process exited, code=killed, status=9/KILL
Feb 07 17:30:39 systemd[1]: vyos-configd.service: Failed with result 'signal'.
Feb 07 17:30:39 systemd[1]: vyos-configd.service: Service RestartSec=100ms expired, scheduling restart.
Feb 07 17:30:39 systemd[1]: vyos-configd.service: Scheduled restart job, restart counter is at 1.
Feb 07 17:30:39 systemd[1]: Stopped VyOS configuration daemon.
Feb 07 17:30:39 systemd[1]: Started VyOS configuration daemon.
Feb 07 17:31:00 kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Feb 07 17:31:00 kernel: ISO 9660 Extensions: RRIP_1991A
Feb 07 17:31:00 vyos-config[4518]: Configuration success
Feb 07 17:31:01 kernel: attempt to access beyond end of device
                        loop1: rw=524288, want=663980, limit=298287
Feb 07 17:31:01 kernel: attempt to access beyond end of device
                        loop1: rw=0, want=661940, limit=298287
Feb 07 17:31:01 systemd[3335]: mnt-cdrom.mount: Succeeded.
Feb 07 17:31:01 systemd[1]: mnt-cdrom.mount: Succeeded.
Feb 07 17:31:01 sudo[4429]: pam_unix(sudo:session): session closed for user root

Apachez · February 7, 2024, 6:03pm

Yeah thats way too little RAM.

Recommended for VyOS is at least 1GB of RAM:

The /var/tmp will automagically be half of your RAM so with 1GB of RAM your /var/tmp will be able to store max 512MB which is enough for the image which currently is at about 420MB.

n.fort · February 8, 2024, 10:28am

Try using this command before upgrade command:

sudo mount -o remount,size=1G /var/tmp

raz0rc4 · February 8, 2024, 7:49pm

I did, see my post. it filled all the 512GB of ram and it killed the process.
it is to be expected, all I am saying is that it should be some check in place for the amount of space and amount of ram available and give the user an option to download the image some place else on the system.

ginko · February 15, 2024, 9:15pm

@raz0rc4

Do you have enough disk space elsewhere you could use? Thinking maybe you could try the add image command this way instead:

$ cd /config
/config$ wget https://xxx/vyos-1.3.5-amd64.iso
/config$ add system image /config/vyos-1.3.5-amd64.iso

Then run your install image command to upgrade

raz0rc4 · February 15, 2024, 11:03pm

yes I did just that to overcome the issue and it worked, thanks for the reply!

ginko · February 16, 2024, 12:22am

Great happy to hear it worked for you @raz0rc4. Grateful if you could mark my suggestion as the solution so that others could benefit from the recommendation in the future if face with the same issue

raz0rc4 · February 16, 2024, 10:53am

this is not my post and it seems i do not have that option.

skynw · February 17, 2024, 11:35am

For me also that didnt work and had to install vyos from scratch…