GeoIP - optimise address ranges

I am making use of the geoip feature in 1.4-rolling, meaning I can get rid of some old scripts that I used on 1.3.x. Thanks a lot for that.

In my old scripts I used iprange to reduce the IP dataset that I loaded in using ipset.

Checking the nft ruleset I see a huge list of IP ranges and wondered whether there’s a chance to run something like iprange or aggregate to optimise and reduce this list in the vyos geoip command workflow, or does it not really matter in terms of performance (either loading rules or processing at run time)?

There is an optimization option in the nft binary (that -o flag when you load the ruleset).

I dont know if VyOS makes use of it but you could try it yourself with something like this and see if it makes any changes to the ruleset and along with that any changes to load/restore times but also performance of the box for packets passing by:

sudo nft -s list ruleset > /config/ruleset1.txt
sudo nft flush ruleset
sudo nft -o -f /config/ruleset1.txt
sudo nft -s list ruleset > /config/ruleset2.txt

and then make a diff between ruleset1.txt and ruleset2.txt (please report back if it did any changes in your case).

There is also possible to just make a dry run like so:

sudo nft -s list ruleset > /config/ruleset1.txt
sudo nft -t -o -f /config/ruleset1.txt

Ref: [PATCH nftables,v2 0/7] ruleset optimization infrastructure

Tried it but fails on the loading stage

oifname { eth0, wg0, wg1 } counter masquerade
BUG: invalid input descriptor type 1953722224
nft: erec.c:161: erec_print: Assertion `0' failed.
Aborted

when trying the dummy run it also runs into an error

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

Googling around (dangerous I know as I have no real knowledge when it comes to nft) the error does pop up when overlapping CIDR ranges are present.

The bug below covers an issue moving from ipset to nft.

https://bugzilla.redhat.com/show_bug.cgi?id=1836571

What about if you try to dump current ruleset, flush tables and then reread it without optimization enabled?

Something like this:

sudo nft -s list ruleset > /config/ruleset1.txt
sudo nft flush ruleset
sudo nft -f /config/ruleset1.txt
sudo nft -s list ruleset > /config/ruleset2.txt

If the above works would it be possible for you to paste parts of your firewall config (replace sensitive IP-addresses with 1.1.1.1, 2.2.2.2 and such or pipe it through strip-private like so “show config commands | strip-private”) so this error can be reproduced by others?

Without the optimisation, it reads back in without any errors. The 2 ruleset files are identical as well.

Stripped version attached.

ruleset-stripped.txt (3.0 MB)

Thanks!

It seems like its “GEOIP_CC_wan-lan_120” who has this gigantic list of elements.

Would it be possible for you to do these two tests (make sure that you have a proper backup of config.boot before you begin)?

  1. At around line 117 you have “set GEOIP_CC_wan-lan_120 {” followed by the huge list of elements at line 120-56977. Would it be possible for you to shrink this elements list to just a few rows or so and see how the “sudo nft -t -o -f /config/ruleset3.txt” would perform?

Something like this:

	set GEOIP_CC_wan-lan_120 {
		type ipv4_addr
		flags interval
		elements = { xxx.xxx.128.0/18, xxx.xxx.200.0-xxx.xxx.204.128,
			     xxx.xxx.207.0/24, xxx.xxx.216.118-xxx.xxx.216.255,
			     xxx.xxx.219.0-xxx.xxx.222.255, xxx.xxx.226.0/23,
			     xxx.xxx.101.0-xxx.xxx.103.255, xxx.xxx.209.0-xxx.xxx.211.255,
			     xxx.xxx.220.0/22, xxx.xxx.254.0/24 }
	}
  1. If above still fails it would be interresting to see the result if you completely remove “set GEOIP_CC_wan-lan_120” along with its elements that is delete row 117-56978 AND also remove this line at row 57203:
ip saddr != @GEOIP_CC_wan-lan_120 counter drop comment "wan-lan-120"

and then see how “sudo nft -t -o -f /config/ruleset4.txt” would perform?

This way we could boil it down to if its the elements list which is just too large or if there is something else in your config that the optimization engine dislikes.

Can do that.

Qq - I could edit the ruleset file directly or I could drop some countries from the geoip whitelist in my firewall rule 120.

What do you suggest.

On 1.3.x when using ipset to load up a firewall address group this number of entries was never an issue.

I would prefer if you first dumped the ruleset to a file and then took a backup of that and worked on the copy.

Like so:

sudo nft -s list ruleset > /config/ruleset.txt
sudo cp /config/ruleset.txt /config/ruleset3.txt
sudo cp /config/ruleset.txt /config/ruleset4.txt

Then modify that ruleset3.txt according to 1st testcase and modify ruleset4.txt according to 2nd testcase and then try to make a dry run and see if you get the same error:

sudo nft -c -t -o -f /config/ruleset3.txt
sudo nft -c -t -o -f /config/ruleset4.txt

The 1st testcase will tell us if we run out of file descriptors or such (the error message gives a hint about that: “BUG: invalid input descriptor type 1953722224”) and the 2nd testcase will tell us if its the elements themselves who are the one to blame for the error (or if there is some other error in your config not releated to the geoip stuff at all).

So test case 1 failed. Edited the file so rule 120 was as below

	set GEOIP_CC_wan-lan_120 {
		type ipv4_addr
		flags interval
		elements = { 1.32.128.0/18, 1.32.200.0-1.32.204.128,
			     1.32.207.0/24, 1.32.216.118-1.32.216.255,
			     1.32.219.0-1.32.222.255, 1.32.226.0/23,
			     1.32.231.0/24, 1.32.233.0/24,
			     1.32.238.0/23, 1.32.240.0/24,
			     223.223.220.0/22, 223.255.254.0/24 }
	}

running sudo nft -t -o -f ./ruleset3.txt gave the following error (removed all output until the error point)

BUG: invalid input descriptor type 544367988
nft: erec.c:161: erec_print: Assertion `0' failed.
Aborted

Test case 2 also fails, removed GEOIP_CC_wan-lan_120 and also the line referencing it. Running sudo -t -o -f ./ruleset4.txt gave the following error (removed all output until the error point)

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

Running any of these commands without the -o flag is successful.

Optimisation looks not to be my friend, although I really have quite a basic setup…

Note that I forgot the -c in the syntax when you just want to do a dry run.

What if you change “GEOIP_CC_wan-lan_120” even further like so?

	set GEOIP_CC_wan-lan_120 {
		type ipv4_addr
		flags interval
		elements = { 1.32.128.0/18, 1.32.207.0/24, 223.255.254.0/24 }
	}

Same error, reduced element set to exactly what you had in the post above. Ran the following

sudo nft -t -o -c -f ./ruleset3.txt

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

running without -o, no problems.

Since removing “GEOIP_CC_wan-lan_120” all together including the references to “GEOIP_CC_wan-lan_120” still gives you:

internal:0:0-0: Error: Could not process rule: File exists

I interpret that as the error is elsewhere…

Would be interesting if you got the time to remove block by block to nail it down where in your current ruleset there is something which the optimizer strongly dislikes?

Edit: Related?

http://git.netfilter.org/nftables/commit/?id=5e39a34b196d68b803911aa13066fef2f83dc98c

A task for a rainy afternoon I think. Have Monday off work so will give it a go.

Have you tried optimising your own ruleset just to see whether or not it processes without error?

Yes but my ruleset is pretty much non-existent (I currently use the VyOS more as a router than a firewall in my lab). More or less just the stuff that VyOS itself adds to it:

vyos@vyos:~$ sudo nft -s list ruleset > /config/ruleset.txt
vyos@vyos:~$ sudo nft -c -t -o -f /config/ruleset.txt
Merging:
/config/ruleset.txt:134:3-37: 		ct state established counter accept
/config/ruleset.txt:135:3-31: 		ct state invalid counter drop
/config/ruleset.txt:136:3-33: 		ct state related counter accept
into:
	ct state vmap { established : accept, invalid : drop, related : accept }
Merging:
/config/ruleset.txt:169:3-37: 		ct state established counter accept
/config/ruleset.txt:170:3-31: 		ct state invalid counter drop
/config/ruleset.txt:171:3-33: 		ct state related counter accept
into:
	ct state vmap { established : accept, invalid : drop, related : accept }

Except for that git-link previously I also found this:

It suggests that a possible workaround might be the way that elements list is written that is it should be one CIDR per line instead of multiple in a long line.

There also seems to exist some fixes which I assume already exists in the 1.4-rolling.

I managed to reproduce the error by adding:

        set N_private-nets {                        
                type ipv4_addr           
                flags interval
                elements = { 1.1.0.0/8, 2.2.0.0/12,
                             3.3.0.0/16 }
        }

to my own ruleset within the “table ip vyos_filter” block.

vyos@vyos:~$ sudo nft -c -t -o -f /config/ruleset.txt
Merging:
/config/ruleset.txt:134:3-37: 		ct state established counter accept
/config/ruleset.txt:135:3-31: 		ct state invalid counter drop
/config/ruleset.txt:136:3-33: 		ct state related counter accept
into:
	ct state vmap { established : accept, invalid : drop, related : accept }
Merging:
/config/ruleset.txt:176:3-37: 		ct state established counter accept
/config/ruleset.txt:177:3-31: 		ct state invalid counter drop
/config/ruleset.txt:178:3-33: 		ct state related counter accept
into:
	ct state vmap { established : accept, invalid : drop, related : accept }
BUG: invalid input descriptor type 151665524
nft: erec.c:161: erec_print: Assertion `0' failed.
Aborted

So far havent figured out correct syntax of that section which doesnt make “-o” to puke…

Even with my rule 120 removed it fell over.

I need to check what other lists of IP addresses might be in there causing the issue.

I think I found a possible workaround/solution.

First I dumped the current ruleset using “sudo nft -s list ruleset > /config/ruleset.txt”.

Then added this at begining of “table ip vyos_filter {” inside the above ruleset.txt:

	set P_wg-ports {
		type inet_service
		flags interval
		auto-merge
		elements = { 51820, 51822 }
	}
	
	set N_private-nets {
		type ipv4_addr
		flags interval
		auto-merge
		elements = { 1.1.0.0/8, 2.2.0.0/12,
			     3.3.0.0/16 }
	}

Then I got this as error (when running “sudo nft -c -t -o -f /config/ruleset.txt”):

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

Verified that its my additions who causes these errors by commenting out (using #) the newly added lines in /config/ruleset.txt (after verification I removed the # chars so I get the error again).

Then I added this to the top of the ruleset.txt file:

flush ruleset

If you do this as script you can run:

sudo sed -i '1s/^/flush ruleset\n\n/' /config/ruleset.txt

Now the optimize succeeds without errors:

vyos@vyos:~$ sudo nft -c -t -o -f /config/ruleset.txt
Merging:
/config/ruleset.txt:151:3-37: 		ct state established counter accept
/config/ruleset.txt:152:3-31: 		ct state invalid counter drop
/config/ruleset.txt:153:3-33: 		ct state related counter accept
into:
	ct state vmap { established : accept, invalid : drop, related : accept }
Merging:
/config/ruleset.txt:186:3-37: 		ct state established counter accept
/config/ruleset.txt:187:3-31: 		ct state invalid counter drop
/config/ruleset.txt:188:3-33: 		ct state related counter accept
into:
	ct state vmap { established : accept, invalid : drop, related : accept }

Verified that the optimize is actually being runned by then running:

sudo nft -o -f /config/ruleset.txt
sudo nft -s list ruleset > /config/ruleset2.txt

and verified that ruleset2.txt contains my added lines of “P_wg-ports” and “set N_private-nets”.

So could you test again doing something like:

sudo nft -s list ruleset > /config/ruleset.txt
sudo sed -i '1s/^/flush ruleset\n\n/' /config/ruleset.txt
sudo nft -o -f /config/ruleset.txt
sudo nft -s list ruleset > /config/ruleset2.txt

Would be interresting to know if the filesizes changes anything.

Instead of “sudo nft -o -f /config/ruleset.txt” you could do a dry run with just “sudo nft -c -t -o -f /config/ruleset.txt” to get a dump of which changes the optimizer has detected it can perform.

Sadly no dice

Merging:
ruleset.txt:57913:3-35: 		oifname "eth0" counter masquerade comment "SRC-NAT-9001"
ruleset.txt:57914:3-34: 		oifname "wg0" counter masquerade comment "SRC-NAT-9011"
ruleset.txt:57915:3-34: 		oifname "wg1" counter masquerade comment "SRC-NAT-9021"
into:
	oifname { eth0, wg0, wg1 } counter masquerade
internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists

internal:0:0-0: Error: Could not process rule: File exists


So even with “flush ruleset” at top of your ruleset.txt file you get the errors?

Did you also get the “ruleset.txt:57913:3-35:” information previously (Im thinking if we are standing still or at all moving towards right direction)?

Would it be possible for you to upload the result (after reboot) of “nft -s list ruleset > /config/ruleset.txt” without striping its content of sensitive information?

That is just replace any personal IP-address with “1.2.3.4” or such?

correct.

yes I do.

Attached is my ruleset after a reboot. Nothing has been changed. There’s nothing sensitive in there.

ruleset.txt (2.8 MB)