How to handle large configuration sizes

[gblack]

I’ve got a project to look at building a open-source firewall. Along the way I encountered VyOS which seems to already do much of what I was planning any have been evaluating it. I’ve run into an issue with config size that I’m guessing some of you have hit before, and I’d like to know if this is just a known issue or if perhaps there’s a better way of doing this than what I’ve tried so far.

I started off with 10,000 rules per zone and 20 zones. I found no workable way of loading that via set commands, the merge command, etc. Scaled down to 1,000 rules per zone and 20 zones. Still had no success. Then I scaled down to 200 rules per zone and 20 zones (so 4,000 rules). This I managed to get loaded via the merge command, but it took much longer than I would have liked - but possibly livable. I did discover though that the config appears to be broke out into directories and files on a ram drive as part of the loading process. While innovative, and likely pretty flexible, I’m guessing this is also a real performance issue and may explain the performance I’ve been looking at. (I know NFT has much better performance than this when loading things directly)

I saved the config, rebooted, and (keep in mind this is a VM) it took around 15 minutes or so to reboot - a pretty clear indicator that config size impacts boot times. That’s a long time to wait on a reboot from a VM. Worse, if scale should grow beyond 200 rules per zone, or there are more zones, I might get into a situation with much worse times - could be a real problem if needing to reboot or restore in some fashion. Is there a better way of doing this?

If there’s not a better way of doing this, given the rules are all based around the firewall features and the rest of the config is likely to be small, does anyone have tips on how I might go about leveraging VyOS for the non-firewall parts and then just load NFT itself on the backside without interfering with VyOS or having one of the two run over the other?

[Apachez]

Unfortunately, if you want an open-source firewall I would look at OPNsense instead.

VyOS (still) have an issue with configs when the number of routing or firewall entries are more than a few 100.

The root cause seems to be that each line gets a massive overhead during validation. Where one or two seconds of processing for a single config line is fine when you add just a single line - but when you have 100s of them this spirals out of control and takes more time than the sum of all lines.

Your best bet is if/when someone would take a look at tasks such as this to get this issue resolved (which unfortunately I believe is a larger work to complete otherwise it would have been fixed by now):

T5388 Something is fishy with commit and boot times when more than a few...

In the meantime a possible workaround is to act on postconfig on boot or better yet act on post hooks scripts:

https://docs.vyos.io/en/latest/automation/command-scripting.html#postconfig-on-boot

https://docs.vyos.io/en/latest/automation/command-scripting.html#executing-pre-hooks-post-hooks-scripts

Im thinking if you create the overall firewall structure with VyOS but with close to no actual content other than the placeholders (as in the actual IP-addresses/ranges etc) and then use the /config/scripts/commit/post-hooks.d to call your script that will dump the current ruleset, inject your bulk of firewall config and then replace the current nftables with this output you should probably be able to dramactically shrink both boot and commit times.

The bad thing is of course that until ⚓ T5388 Something is fishy with commit and boot times when more than a few hundred static routes are being used gets resolved you wont have a single config for your VyOS boxes. You will have the VyOS-config + the script that goes into /config/scripts/commit/post-hooks.d along with whatever fileformat your script will use as input to “inject” the rules.

The steps for your script to operate at would be something along with:

#A: Export
sudo nft -s list ruleset > /config/ruleset.txt
#B: Fix for import
sudo sed -i '1s/^/flush ruleset\n\n/' /config/ruleset.txt
#C: Modify
*** modify /config/ruleset.txt to your liking ***
#D: Import
sudo nft -o -f /config/ruleset.txt

That is A will first dump current ruleset.

B will inject “flush ruleset” as first line, otherwise you wont be able to re-import the outputed file (you will end up with funny errors).

At C you do your custom modifications to the outputed file.

And finally at D you re-import this modified file. I prefer using -o so nft can do its own optimizations if needed.

If you need to do a dry-run to figure out what kind of optimizations (if any) there might be you can manually run this after you modified “ruleset.txt”:

sudo nft -c -t -o -f /config/ruleset.txt