GeoIP - optimise address ranges

The error you are seeing:

internal:0:0-0: Error: Could not process rule: File exists

seems to be related to “table ip vyos_filter” and these 10 chains:

  • chain VZONE_lan
  • chain VZONE_local_IN
  • chain VZONE_local_OUT
  • chain VZONE_mullvadgb
  • chain VZONE_mullvadus
  • chain VZONE_wan
  • chain VZONE_wg0
  • chain VZONE_wg1
  • chain VZONE_wg7
  • chain VZONE_wg8

Because when removing them (as a test) along with modifying these 3 chains (who reference the above) the error goes away:

  • chain VYOS_ZONE_FORWARD
  • chain VYOS_ZONE_LOCAL
  • chain VYOS_ZONE_OUTPUT

That’s kind of the core of my firewall setup though… :slight_smile:

The challenge being I have those rules and those zones for a reason. It’s interesting that the nft output is valid, but the optimisation process chokes on it.

I pulled all those country IPs and ran iprange against them and the list actually grew which suggests the IPs listed are already somewhat optimised. The long list of IPs doesn’t appear to be the cause as removing that rule, and trying to optimise the resulting nft still throws an error.

It’s not causing any issues, but it would be interesting to understand why this valid nft can’t be optimised.

It seems like the return logic is broken somehow.

This gives error:

        chain VZONE_wg8 {                                                             
                iifname "wg8" counter return                       
                iifname "eth1" counter jump NAME_lan-wg8            
                iifname "eth1" counter return                           
                iifname "eth3" counter jump NAME_mullvadgb-wg8                        
                iifname "eth3" counter return                                             
                iifname "eth2" counter jump NAME_mullvadus-wg8                        
                iifname "eth2" counter return                                         
                iifname "eth0" counter jump NAME_wan-wg8           
                iifname "eth0" counter return                      
                iifname "wg0" counter jump NAME_wg0-wg8            
                iifname "wg0" counter return                                
                iifname "wg1" counter jump NAME_wg1-wg8                               
                iifname "wg1" counter return                                          
                iifname "wg7" counter jump NAME_wg7-wg8            
                iifname "wg7" counter return                                  
                counter drop comment "zone_wg8 default-action drop"                         
        }

while this doesnt:

        chain VZONE_wg8 {                                                             
                iifname "wg8" counter return                       
                iifname "eth1" counter jump NAME_lan-wg8            
#                iifname "eth1" counter return                           
                iifname "eth3" counter jump NAME_mullvadgb-wg8                        
#                iifname "eth3" counter return                                             
                iifname "eth2" counter jump NAME_mullvadus-wg8                        
#                iifname "eth2" counter return                                         
                iifname "eth0" counter jump NAME_wan-wg8           
#                iifname "eth0" counter return                      
                iifname "wg0" counter jump NAME_wg0-wg8            
#                iifname "wg0" counter return                                
                iifname "wg1" counter jump NAME_wg1-wg8                               
#                iifname "wg1" counter return                                          
                iifname "wg7" counter jump NAME_wg7-wg8            
#                iifname "wg7" counter return                                  
                counter drop comment "zone_wg8 default-action drop"                         
        }

What interesting is if you look at the chain you jump to:

	chain NAME_lan-wg8 {
		ct state { established, related } counter return comment "lan-wg8-100"
		ct state invalid counter drop comment "lan-wg8-110"
		counter drop comment "lan-wg8 default-action drop"
	}

has a “return” for established/related, to me this should read “accept” rather than return because if its established/related you want to stop processing and not continue (because both iptables and nft is first-match aka top-down).

Some more digging…

Bad (gives error):

	chain VZONE_wg8 {
		iifname "wg8" counter return
		iifname "eth1" counter jump NAME_lan-wg8
		iifname "eth1" counter return
		iifname "eth3" counter jump NAME_mullvadgb-wg8
		iifname "eth3" counter return
		iifname "eth2" counter jump NAME_mullvadus-wg8
		iifname "eth2" counter return
		iifname "eth0" counter jump NAME_wan-wg8
		iifname "eth0" counter return
		iifname "wg0" counter jump NAME_wg0-wg8
		iifname "wg0" counter return
		iifname "wg1" counter jump NAME_wg1-wg8
		iifname "wg1" counter return
		iifname "wg7" counter jump NAME_wg7-wg8
		iifname "wg7" counter return
		counter drop comment "zone_wg8 default-action drop"
	}

Good (no error):

	chain VZONE_wg8 {
		iifname "wg8" counter return
		iifname "eth1" counter jump NAME_lan-wg8
		iifname "eth1" return
		iifname "eth3" counter jump NAME_mullvadgb-wg8
		iifname "eth3" return
		iifname "eth2" counter jump NAME_mullvadus-wg8
		iifname "eth2" return
		iifname "eth0" counter jump NAME_wan-wg8
		iifname "eth0" return
		iifname "wg0" counter jump NAME_wg0-wg8
		iifname "wg0" return
		iifname "wg1" counter jump NAME_wg1-wg8
		iifname "wg1" return
		iifname "wg7" counter jump NAME_wg7-wg8
		iifname "wg7" return
		counter drop comment "zone_wg8 default-action drop"
	}

That is fix seems to be to remove “counter” for those returns.

Thanks a lot btw for continuing to work on this issue. Very much appreciated.

My lan-wg8 in config.boot is as follows

    name lan-wg8 {
        default-action drop
        rule 100 {
            action accept
            state {
                established enable
                related enable
            }
        }
        rule 110 {
            action drop
            state {
                invalid enable
            }
        }
    }

Is this correctly represented in the nft config?

My wg8 section looks as follows

    name wg8-lan {
        default-action accept
    }
    name wg8-local {
        default-action accept
    }
    name wg8-mullvadgb {
        default-action drop
    }
    name wg8-mullvadus {
        default-action drop
    }
    name wg8-wan {
        default-action accept
    }
    name wg8-wg0 {
        default-action drop
    }
    name wg8-wg1 {
        default-action drop
    }
    name wg8-wg7 {
        default-action drop
    }

How would I remove the “counter” for those returns via config commands, and would sharing my sanitised config.boot help at all?

I have uploaded “ruleset_fixed.txt” which contains the following fixes:

  1. Added “flush ruleset” as the first line.

  2. Modified following 10 chains in “table ip vyos_filter”:

  • chain VZONE_lan
  • chain VZONE_local_IN
  • chain VZONE_local_OUT
  • chain VZONE_mullvadgb
  • chain VZONE_mullvadus
  • chain VZONE_wan
  • chain VZONE_wg0
  • chain VZONE_wg1
  • chain VZONE_wg7
  • chain VZONE_wg8

The modification is to keep only the first iifname "xxx" counter return but the ones following the jumps have their “counter” removed like so:

	chain VZONE_wg8 {
		iifname "wg8" counter return
		iifname "eth1" counter jump NAME_lan-wg8
		iifname "eth1" return
		iifname "eth3" counter jump NAME_mullvadgb-wg8
		iifname "eth3" return
		iifname "eth2" counter jump NAME_mullvadus-wg8
		iifname "eth2" return
		iifname "eth0" counter jump NAME_wan-wg8
		iifname "eth0" return
		iifname "wg0" counter jump NAME_wg0-wg8
		iifname "wg0" return
		iifname "wg1" counter jump NAME_wg1-wg8
		iifname "wg1" return
		iifname "wg7" counter jump NAME_wg7-wg8
		iifname "wg7" return
		counter drop comment "zone_wg8 default-action drop"
	}

Dry run using sudo nft -c -o -f /config/ruleset_fixed.txt completes without errors.

ruleset_fixed.txt (2.8 MB)

Living legend, thanks a lot!

I suspect these back-end manipulations can’t be reflected in my config.boot so any changes would need to be manually updated via the nft dump and reloaded. Is that correct?

Is this caused by the way Vyos generates the config or is this an “issue” with nft that should/could be solved upstream?

You could try this to see if your VyOS can generate a proper nftables output on its own.

Note! Only do this on a box you have physical access to (in case something goes south and you need to restore the file and reboot the box again).

The below has been tested on VyOS 1.4-rolling-202307250317 but the file in question doesnt seem to have changed since 11 jan 2021 so I guess we should be safe to test this :slight_smile:

  1. Take backup of this file: /usr/share/vyos/templates/firewall/nftables-zone.j2 (better safe than sorry :wink: )

  2. Then modify it so you change this:

iifname { {{ zone[from_zone].interface | join(",") }} } counter return

into this:

iifname { {{ zone[from_zone].interface | join(",") }} } return

that is remove “counter” from those lines and then save the file.

The above should exist 3 times in total at around line 45, 51 and 70.

  1. Try to reboot the box.

  2. After reboot try to do the optimizer tests again:

sudo nft -s list ruleset > /config/v2ruleset.txt
sudo sed -i '1s/^/flush ruleset\n\n/' /config/v2ruleset.txt
sudo nft -o -f /config/v2ruleset.txt
sudo nft -s list ruleset > /config/v2ruleset2.txt 

If everything went ok you should be able to load the “v2ruleset.txt” without errors.

And when editing “v2ruleset.txt” and scroll down to the VZONE definitions you should see that they look something like:

	chain VZONE_wg8 {
		iifname "wg8" counter return
		iifname "eth1" counter jump NAME_lan-wg8
		iifname "eth1" return
		iifname "eth3" counter jump NAME_mullvadgb-wg8
		iifname "eth3" return
		iifname "eth2" counter jump NAME_mullvadus-wg8
		iifname "eth2" return
		iifname "eth0" counter jump NAME_wan-wg8
		iifname "eth0" return
		iifname "wg0" counter jump NAME_wg0-wg8
		iifname "wg0" return
		iifname "wg1" counter jump NAME_wg1-wg8
		iifname "wg1" return
		iifname "wg7" counter jump NAME_wg7-wg8
		iifname "wg7" return
		counter drop comment "zone_wg8 default-action drop"
	}

That is only the first iifname have a “counter return” while the others just have “return”.

Im not sure if this is an error in VyOS or nftables - Im leaning towards nftables because it shouldnt output a config it doesnt like to load.

Most likely the error is in the optimizer itself and doing the above manipulation of how VyOS generates the zone-rules should be seen as a workaround until whatever is broken in the optimzier is fixed upstream.

Note that I didnt dig into if removing the counters would have any negative impact. What you miss is statistics regarding which packets ended up in the jump and never returned (accepted or dropped by the jump) vs. which one returned and was returned out of the VZONE chain.

Note according to docs of nftables doing a final return equals to accept so I think it should be further investigated if the current nftables config that VyOS produces is actually being correct or not.

1 Like

I have filed this as a bug towards netfilter: Bug 1697 – Errors when running "nft -o" optimizer due to "counter return"

@ phillipmcmahon how did it go?

Apologies - I thought I’d replied. Your changes worked and allowed the optimisation to run without errors. Thanks a lot for your time spent on this.

The ruleset size before and after optimisation is somewhat negligible, I’ll send over the size difference tomorrow.

Thanks for the followup!

Something I noticed is that your config lacks “auto-merge” for the GEOIP_CC_wan-lan_120 object:

set GEOIP_CC_wan-lan_120 {
	type ipv4_addr
	flags interval
	elements = {

That is I would expect it to look something like:

set GEOIP_CC_wan-lan_120 {
	type ipv4_addr
	flags interval
	auto-merge
	elements = {

but I dunno if thats configurable through VyOS config-mode or not?

But could be worth trying in your case if that does anything for the optimizer or not?

1 Like

I’ll try that tomorrow and see what happens.

I don’t think any of this is available via the Vyos configuration commands.

It does appear the IP list doesn’t shrink at all with optimisation. Maybe your latest suggestion will help with that.

Since you are acting on country codes I wouldnt expect it to shrink at all because a single IP-address/range usually only belongs to one specific country (or specific special use).

Compared to if you would have been acting on a web-category (which would translate to an IP-address/range) then adding two or more categories to the same rule would more likely create duplicates or IP-addresses/ranges who are within each other. Like a 1.2.3.0/24 and 1.2.0.0/16 can be replaced by a single 1.2.0.0/16.