Disk full


#1

Hello,

We have a 4 GB virtual machine running VyOS. Recently when I committed some changes I figured it was running out of disk space. I can’t find out which occupies the space and how to clear and make some free space. Please help. I have pasted the df info below.

df -h

Filesystem Size Used Avail Use% Mounted on
overlayfs 3.9G 3.9G 0 100% /
tmpfs 249M 0 249M 0% /lib/init/rw
udev 241M 152K 241M 1% /dev
tmpfs 249M 4.0K 249M 1% /dev/shm
/dev/sda1 3.9G 3.9G 0 100% /live/image
/dev/sda1 3.9G 3.9G 0 100% /live/cow
tmpfs 249M 0 249M 0% /live
tmpfs 249M 112K 249M 1% /tmp
/dev/sda1 3.9G 3.9G 0 100% /opt/vyatta/etc/config
tmpfs 249M 116K 249M 1% /var/run
none 249M 3.5M 245M 2% /opt/vyatta/config
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_26102
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_21230
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_7424
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_5219
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_13732
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_21000
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_16518
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_6121
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_7085
df: /opt/vyatta/config/tmp/new_config_31804': No such file or directory df:/opt/vyatta/config/tmp/new_config_354’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_15053': No such file or directory df:/opt/vyatta/config/tmp/new_config_31585’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_12584': No such file or directory df:/opt/vyatta/config/tmp/new_config_24176’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_32617': No such file or directory df:/opt/vyatta/config/tmp/new_config_8489’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_10484': No such file or directory df:/opt/vyatta/config/tmp/new_config_18981’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_1693': No such file or directory df:/opt/vyatta/config/tmp/new_config_8570’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_30691': No such file or directory df:/opt/vyatta/config/tmp/new_config_1156’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_9381': No such file or directory df:/opt/vyatta/config/tmp/new_config_31851’: No such file or directory
df: `/opt/vyatta/config/tmp/new_config_6450’: No such file or directory


#2

this looks suspicious: /dev/sda1 3.9G 3.9G 0 100% /opt/vyatta/etc/config

maybe try looking around to see what’s taking up space:

cd /opt/vyatta/etc/config && du -sh *

cd /opt/vyatta/config && du -sh *


#3

Consider also looking at number of and size of images installed:

i.e. for me:

# du -shc /live/image/boot/*
338M	/live/image/boot/1.1.3
449M	/live/image/boot/VyOS-999.lithium.03010000
241M	/live/image/boot/VyOS-999.lithium.03030000
1.5G	/live/image/boot/VyOS-999.lithium.03040000
2.0M	/live/image/boot/grub
2.5G	total

My larger 1.5G item is because I’m doing devel/compilation but in general they should be closer to 250-400M each image.

Regards,

Chris


#4

Thank you for your support. Here are my results.

$ sudo du -shc /live/image/boot/*
1.1G    /live/image/boot/1.0.4
2.9G    /live/image/boot/1.1.1
2.0M    /live/image/boot/grub
3.9G    total
$ cd /opt/vyatta/etc/config && sudo du -sh *
260K    archive
4.0K    auth
52K     config.boot
20K     config.boot.15561
20K     config.boot.2013-11-21-1159.pre-migration
48K     config.boot.2014-10-08-1401.pre-migration
8.0K    ips
8.0K    scripts
4.0K    support
16K     url-filtering
4.0K    user-data
$ cd /opt/vyatta/config && sudo du -sh *
3.4M    active
0       tmp

#5

Try something like this to narrow down what is consuming the space:

vyos@vygw01:~$ sudo du -a /live/image/boot | sort -rn | head -40
2627516	/live/image/boot
1573320	/live/image/boot/VyOS-999.lithium.03040000
1330128	/live/image/boot/VyOS-999.lithium.03040000/live-rw
459664	/live/image/boot/VyOS-999.lithium.03010000
456696	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr
352256	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root
345892	/live/image/boot/1.1.3
326612	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr/src
326608	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr/src/linux-image
326604	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr/src/linux-image/debian
326600	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr/src/linux-image/debian/build
326596	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr/src/linux-image/debian/build/build-amd64-none-amd64-vyos
262660	/live/image/boot/VyOS-999.lithium.03040000/live-rw/var
246628	/live/image/boot/VyOS-999.lithium.03030000
222868	/live/image/boot/VyOS-999.lithium.03040000/VyOS-999.lithium.03040000.squashfs
222864	/live/image/boot/VyOS-999.lithium.03030000/VyOS-999.lithium.03030000.squashfs
222864	/live/image/boot/VyOS-999.lithium.03010000/VyOS-999.lithium.03010000.squashfs
216476	/live/image/boot/VyOS-999.lithium.03010000/live-rw
216208	/live/image/boot/1.1.3/1.1.3.squashfs
200708	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/software
186984	/live/image/boot/VyOS-999.lithium.03010000/live-rw/var
168552	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/software/snort-2.9.7.0
160376	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/software/snort-2.9.7.0/src
149732	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/etc
127496	/live/image/boot/VyOS-999.lithium.03040000/live-rw/opt
127372	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/etc/so_rules
126020	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/etc/so_rules/precompiled
125532	/live/image/boot/VyOS-999.lithium.03040000/live-rw/opt/p
120768	/live/image/boot/VyOS-999.lithium.03040000/live-rw/home
120764	/live/image/boot/VyOS-999.lithium.03040000/live-rw/home/vyos
110944	/live/image/boot/VyOS-999.lithium.03040000/live-rw/usr/src/linux-image/debian/build/build-amd64-none-amd64-vyos/arch
109920	/live/image/boot/VyOS-999.lithium.03010000/live-rw/var/spool
109916	/live/image/boot/VyOS-999.lithium.03010000/live-rw/var/spool/squid3
109404	/live/image/boot/1.1.3/live-rw
108908	/live/image/boot/1.1.3/live-rw/var
102852	/live/image/boot/1.1.3/live-rw/var/spool
102848	/live/image/boot/1.1.3/live-rw/var/spool/squid3
99236	/live/image/boot/VyOS-999.lithium.03040000/live-rw/root/software/snort-2.9.7.0/src/dynamic-preprocessors
97724	/live/image/boot/VyOS-999.lithium.03040000/live-rw/var/spool
97716	/live/image/boot/VyOS-999.lithium.03040000/live-rw/var/spool/squid3

#6

Ok here we go:

$ sudo du -a /live/image/boot | sort -rn | head -40
4054228 /live/image/boot
3001756 /live/image/boot/1.1.1
2765276 /live/image/boot/1.1.1/live-rw
2764528 /live/image/boot/1.1.1/live-rw/var
2759952 /live/image/boot/1.1.1/live-rw/var/log
2757960 /live/image/boot/1.1.1/live-rw/var/log/auth.log
1050464 /live/image/boot/1.0.4
822136  /live/image/boot/1.0.4/live-rw
821220  /live/image/boot/1.0.4/live-rw/var
816676  /live/image/boot/1.0.4/live-rw/var/log
814536  /live/image/boot/1.0.4/live-rw/var/log/auth.log
216200  /live/image/boot/1.1.1/1.1.1.squashfs
208328  /live/image/boot/1.0.4/1.0.4.squashfs
14944   /live/image/boot/1.0.4/initrd.img-3.3.8-1-amd64-vyatta
14520   /live/image/boot/1.1.1/initrd.img-3.13.11-1-amd64-vyos
4088    /live/image/boot/1.1.1/live-rw/var/cache
4064    /live/image/boot/1.0.4/live-rw/var/cache
3216    /live/image/boot/1.1.1/vmlinuz-3.13.11-1-amd64-vyos
2864    /live/image/boot/1.1.1/live-rw/var/cache/debconf
2844    /live/image/boot/1.0.4/live-rw/var/cache/debconf
2840    /live/image/boot/1.0.4/vmlinuz-3.3.8-1-amd64-vyatta
2756    /live/image/boot/1.1.1/live-rw/var/cache/debconf/templates.dat
2756    /live/image/boot/1.0.4/live-rw/var/cache/debconf/templates.dat
2432    /live/image/boot/1.1.1/System.map-3.13.11-1-amd64-vyos
2112    /live/image/boot/1.0.4/System.map-3.3.8-1-amd64-vyatta
2004    /live/image/boot/grub
884     /live/image/boot/1.1.1/live-rw/var/cache/man
884     /live/image/boot/1.0.4/live-rw/var/cache/man
448     /live/image/boot/1.0.4/live-rw/config
428     /live/image/boot/1.1.1/live-rw/var/backups
420     /live/image/boot/1.0.4/live-rw/var/backups
400     /live/image/boot/grub/locale
396     /live/image/boot/1.1.1/live-rw/config
360     /live/image/boot/1.1.1/live-rw/var/backups/dpkg.status.0
352     /live/image/boot/1.0.4/live-rw/var/backups/dpkg.status.0
336     /live/image/boot/1.1.1/live-rw/var/cache/apt
332     /live/image/boot/1.0.4/live-rw/var/cache/apt
312     /live/image/boot/1.1.1/live-rw/var/cache/apt/pkgcache.bin
308     /live/image/boot/1.0.4/live-rw/var/cache/apt/pkgcache.bin
288     /live/image/boot/1.0.4/live-rw/etc

#7
2757960 /live/image/boot/1.1.1/live-rw/var/log/auth.log
814536  /live/image/boot/1.0.4/live-rw/var/log/auth.log

There is your problem. auth.log records a variety of information (subject to syslog policy). Most notably SSH and su/sudo attempts.

For your logs to be that big, I would guess your device has a public IP address and is receiving many brute force attempts. If that is the case, it is recommended that you firewall remote management (SSH) to reject brute force attempts.

Delete the older log first, and then review the newer log to see what is being logged. Clear out the log once you have determined the cause of such a log size.

Regards,

Chris


#8

Hi Chris,

This was really useful. I have cleared up the auth.log and everything is back to normal. Is there any way we can mitigate brute force attempts on SSH? as this seems to be the issue.

Thanks,
Abilash


#9

Excellent

Re: brute force, it is best to firewall remote management on publically facing interfaces (and even internal ones if it can be done) to ensure only only authorised IPs are allowed for inbound TCP port 22.

In linux terms, /var/log/auth.log should be logrotated to avoid excessive log files but that doesn’t appear to be configured. You could file a bug report but the problem would almost enitirely be mitigated by a appropriately configured firewall.

Regards,

Chris.


#10

Ok, but we can’t lock SSH to certain IP as we may have to access it from different (dynamic) IPs. Does VyOS come with any in-built brute force detector? I was wondering why VyOS did not have a logrotate!?


#11

Have a look at this page:
http://www.rackspace.com/knowledge_center/article/configuring-interface-based-firewall-on-the-vyatta-network-appliance

set rule 300 action 'drop' set rule 300 destination port '22' set rule 300 protocol 'tcp' set rule 300 recent count '3' set rule 300 recent time '30' set rule 300 state new 'enable'

set rule 310 action 'accept' set rule 310 destination port '22' set rule 310 protocol 'tcp'


#12

pirateghost’s suggestion is useful effectively rate-limit new attempts which will reduce the effectiveness of brute force attacks and event logs.

You could consider reconfiguring SSH on a different port:

set service ssh port <x>

This has no value in improving security of course but it will reduce the likelihood of random brute force attacks hitting the SSH daemon.

Another effective measure which doesn’t exist in VyOS (to my knowledge) is port-knocking where a sequence of packets must first hit the firewall to unlock the firewall ruleset to accept new SSH connections on TCP 22. This would be a nice to have in VyOS in the future :slight_smile:


#13

cgb and pirateghost, brilliant solutions! Thanks!! Yes, port knocking is a must have in the firewall. Is there a feature request we can vote for?

I believe someone will get rid of the spam post.