SNAT error no space left on device

General questions
1

Hello!

Checking the Vyatta firewall log (/var/log/messages) I can see a lot of error messages like this one:

vyatta-dataplane.service dataplane[2652]: FIREWALL: SNAT: map get 1 (10.10.100.14:13639) failed: No space left on device, used 3467/65535

Checking source NAT transactions, seems to be ok, there is less established transactions(3467) than totals (65535)

admin@fw01:~$ show nat source statistics
rule    pkts            bytes                    interface       used/total
----    ----            -----                    ---------       ----------
25      375514601       280697502022             dp0bond1        3467/65535

I have made a “clear nat” but the error still occurring.

The source NAT rule is this one:

admin@fw01:~$ show nat source rules
------------------------
NAT Rulesets Information
------------------------
--------------------------------------------------------------------------------
SOURCE
rule    intf            match                                   translation
----    ----            -----                                   -----------
25     dp0bond1        from 10.10.100.14                       dynamic any -> masquerade

And this is the NAT rule configuration command:

set service nat source rule 25 outbound-interface 'dp0bond1'
set service nat source rule 25 source address '10.10.100.14'
set service nat source rule 25 translation address 'masquerade'

Why the error say that there is no space left on device if we are not reaching total NAT transactions?

I assume this is not related with filesystem state, as there is a lot of free space:

admin@fw01:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G   14M  1.6G   1% /run
/dev/sda2       917G   15G  855G   2% /lib/live/mount/persistence/sda2
/dev/loop0      319M  319M     0 100% /lib/live/mount/rootfs/1801q.09052048.squashfs
tmpfs           7.9G     0  7.9G   0% /lib/live/mount/overlay
overlay         917G   15G  855G   2% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup

Thanks

2

Hello @sanet
I think that you have a lot of very small files (inodes). And it’s indicate a some limits of free inodes.
Can you provide command?
df --inodes

3

Hi @Viacheslav

There is a lot of inodes free except in /dev/loop0, but that filesystem is full and I think that’s normal, right?

admin@fw01:~$ df --inodes
Filesystem       Inodes IUsed    IFree IUse% Mounted on
udev            2047214   392  2046822    1% /dev
tmpfs           2057640   767  2056873    1% /run
/dev/sda2      61014016  6382 61007634    1% /lib/live/mount/persistence/sda2
/dev/loop0        58630 58630        0  100% /lib/live/mount/rootfs/1801q.09052048.squashfs
tmpfs           2057640     1  2057639    1% /lib/live/mount/overlay
overlay        61014016  6382 61007634    1% /
tmpfs           2057640     1  2057639    1% /dev/shm
tmpfs           2057640     4  2057636    1% /run/lock
tmpfs           2057640     5  2057635    1% /sys/fs/cgroup

Thanks

4

For /dev/loop device it’s normal.
I don’t see /tmp dir
Can you check it?
df -i /tmp
free -m
I think there are some system limits, just need to find which ones.

5

Seems ok too:

admin@fw01:/var/log$ df -i /tmp
Filesystem       Inodes IUsed    IFree IUse% Mounted on
overlay        61014016  6382 61007634    1% /


admin@fw01:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:          16075        4938        5469          13        5666       10794
Swap:             0           0           0
6

Hello, @sanet!

According to your logs, you are using Vyatta and this error is related to its dataplane or hardware. As VyOS don’t use hardware dataplanes, it would be better for you to contact about this message support of your Vyatta appliance.