We have a 4 GB virtual machine running VyOS. Recently when I committed some changes I figured it was running out of disk space. I can’t find out which occupies the space and how to clear and make some free space. Please help. I have pasted the df info below.
df -h
Filesystem Size Used Avail Use% Mounted on
overlayfs 3.9G 3.9G 0 100% /
tmpfs 249M 0 249M 0% /lib/init/rw
udev 241M 152K 241M 1% /dev
tmpfs 249M 4.0K 249M 1% /dev/shm
/dev/sda1 3.9G 3.9G 0 100% /live/image
/dev/sda1 3.9G 3.9G 0 100% /live/cow
tmpfs 249M 0 249M 0% /live
tmpfs 249M 112K 249M 1% /tmp
/dev/sda1 3.9G 3.9G 0 100% /opt/vyatta/etc/config
tmpfs 249M 116K 249M 1% /var/run
none 249M 3.5M 245M 2% /opt/vyatta/config
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_26102
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_21230
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_7424
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_5219
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_13732
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_21000
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_16518
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_6121
unionfs-fuse 249M 3.5M 245M 2% /opt/vyatta/config/tmp/new_config_7085
df: /opt/vyatta/config/tmp/new_config_31804': No such file or directory df:/opt/vyatta/config/tmp/new_config_354’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_15053': No such file or directory df:/opt/vyatta/config/tmp/new_config_31585’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_12584': No such file or directory df:/opt/vyatta/config/tmp/new_config_24176’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_32617': No such file or directory df:/opt/vyatta/config/tmp/new_config_8489’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_10484': No such file or directory df:/opt/vyatta/config/tmp/new_config_18981’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_1693': No such file or directory df:/opt/vyatta/config/tmp/new_config_8570’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_30691': No such file or directory df:/opt/vyatta/config/tmp/new_config_1156’: No such file or directory
df: /opt/vyatta/config/tmp/new_config_9381': No such file or directory df:/opt/vyatta/config/tmp/new_config_31851’: No such file or directory
df: `/opt/vyatta/config/tmp/new_config_6450’: No such file or directory
There is your problem. auth.log records a variety of information (subject to syslog policy). Most notably SSH and su/sudo attempts.
For your logs to be that big, I would guess your device has a public IP address and is receiving many brute force attempts. If that is the case, it is recommended that you firewall remote management (SSH) to reject brute force attempts.
Delete the older log first, and then review the newer log to see what is being logged. Clear out the log once you have determined the cause of such a log size.
This was really useful. I have cleared up the auth.log and everything is back to normal. Is there any way we can mitigate brute force attempts on SSH? as this seems to be the issue.
Re: brute force, it is best to firewall remote management on publically facing interfaces (and even internal ones if it can be done) to ensure only only authorised IPs are allowed for inbound TCP port 22.
In linux terms, /var/log/auth.log should be logrotated to avoid excessive log files but that doesn’t appear to be configured. You could file a bug report but the problem would almost enitirely be mitigated by a appropriately configured firewall.
Ok, but we can’t lock SSH to certain IP as we may have to access it from different (dynamic) IPs. Does VyOS come with any in-built brute force detector? I was wondering why VyOS did not have a logrotate!?
set rule 300 action 'drop'
set rule 300 destination port '22'
set rule 300 protocol 'tcp'
set rule 300 recent count '3'
set rule 300 recent time '30'
set rule 300 state new 'enable'
set rule 310 action 'accept'
set rule 310 destination port '22'
set rule 310 protocol 'tcp'
pirateghost’s suggestion is useful effectively rate-limit new attempts which will reduce the effectiveness of brute force attacks and event logs.
You could consider reconfiguring SSH on a different port:
set service ssh port <x>
This has no value in improving security of course but it will reduce the likelihood of random brute force attacks hitting the SSH daemon.
Another effective measure which doesn’t exist in VyOS (to my knowledge) is port-knocking where a sequence of packets must first hit the firewall to unlock the firewall ruleset to accept new SSH connections on TCP 22. This would be a nice to have in VyOS in the future