Quick and Dirty Benchmark of Cores Vs MHz

Development
1

Hello,

I’m a lover of zone based firewalls, and I like to be very granular for security reasons, hence lots of rules…

We have a client on 1.3.5 with a bunch of zones and rules, the config file is about 34,000 lines of mostly firewall rules. NAT is handled by another router.

Their hardware is an old Dell R220 with a quad core Xeon @ 3.xGhz… The time it takes to go from “Mounting VyOS Config…done.” to an actual login prompt and a usable system is about 30 minutes.

Our of curiosity I wanted to see if it was a CPU thing and if it was core or MHz limited.

On my work laptop I have a Ryzen 5, I spun up a virtual guest and loaded the clients config… this is a rough benchmark because I was doing other things on the laptop but I think the results are still revealing.

@6 cores, 4GB RAM the load time was 22 minutes
@4 cores, 4GB RAM the load time was 23 minutes
@2 cores, 4GB RAM the load time was 25 minutes

so it all seems the same no matter the core count…

Then I dropped the CPU frequency from the average of 3.5GHz to 1GHz

@2 cores (1GHz), 4GB RAM the load time was 88 minutes and it brought me to the login prompt but it also said “migrate rl-system firewall failed”

So… the cores didn’t seem to make a real difference for loading all those firewall rules. Going from 1GHz to 3.5GHz made it load 3.5x faster… so here frequency on loading firewall rules seems king.

Anyway, I’m getting over a cold and am putzing about so ran this quick test (we’ll it took 4 hours to complete), and wanted to put it out there for all those with the same unanswered question. I ordered some used higher frequency CPUs and we’ll see what they do :slight_smile:

Next to see what 1.4 does… and thank you again for keeping zone based firewalls, if it had been dropped my confidence in my systems would have dropped.

2

There is this task reported over at vyos.dev:

Something is fishy with commit and boot times when more than a few hundred static routes are being used
https://vyos.dev/T5388

The sad part is that if you load the generated (once the device have booted) FRR config manually to FRR it goes in like less than a second, same with routes and nftables rules injected to the kernel directly from bash through a script. So there are room for improvement.

Hopefully this can get fixed in future…

3

Yes, hopefully, but until then telling the staff that it takes 30 minutes for the router to restart makes them both fear it and respect the hands that touch it… or so I like to imagine :slight_smile:

In reality I fear having to reboot it during the day but that only ever happens during a power outage. VyOS has been stable for me. I would like to add more rules/etc though so we’ll see what a faster CPU affords me.

1 Like
4

Unfortunately this is a dealbreaker for us to have VyOS anywhere close to production.

Waiting for an hour or two for a router to complete its boot its not an option.

5

@Apachez Just read your thorough testing and tweaking in ⚓ T5388 Something is fishy with commit and boot times when more than a few hundred static routes are being used. Nice work.

It seems to me that most cpu cycles are spent in userspace. E.g. the almost linear increase in commit time when for example adding zones to the zbf.

I agree with you that this is worrisome.

1 Like
6

If possible please try this as root (sudo bash) after all 34000 or so rules have been loaded:

# Export current ruleset.
sudo nft -s list ruleset > /config/ruleset.txt

# Add "flush ruleset" at top of the dump otherwise import will fail.
sudo sed -i '1s/^/flush ruleset\n\n/' /config/ruleset.txt

# Modify the exported ruleset...
*** modify the /config/ruleset.txt file ***

# Import the modified ruleset.
sudo nft -o -f /config/ruleset.txt

That is export, add “flush ruleset” and then import it again.

Please do this using “time” to get both kernel and userland time spent.

Would be interresting to see how much time it takes for the Linux kernel itself to process the nftables rules from a backup vs how long time VyOS configd spends doing the same thing.

7

For the import I had to omit -o as it said it wasn’t an option. This is the command I used

time nft -f /config/ruleset.txt
#and 
time sudo nft -f /config/ruleset.txt

The import was instantaneous so I feel I’m missing something, apologies.

This is VyOS 1.3.5, the upgrade to 1.4 failed over multiple attempts so that is on the back burner for now.

8

I missed that you were using 1.3.x.

The -o option is to “optimize” when importing rulesets, without the -o the ruleset will be exactly the same as when you first dumped it with “-s list ruleset”.

What did “time” say regarding kernel and user space time to perform the import operation?

9 
real     0m0.077s
user     0m0.054s
sys      0m0.022s
10

Thanks!

Yeah thats some difference between 25 minutes when runned through vyos-configd vs. runned with userland app nft which ended at 0.077 seconds.

1948051% or something like that as difference in performance…