Hello everyone,
I am working on optimizing the conntrack settings in VyOS for different VM sizes and would like to get the community’s opinion on the values I am considering. My goal is to use approximately 50% of the available RAM for conntrack, ensuring a balance between efficient memory usage and performance.
Configurations Considered
For a VM with 1GB of RAM (512MB for conntrack):
set system conntrack table-size 1146880
set system conntrack expect-table-size 102400
set system conntrack hash-size 143360
For a VM with 2GB of RAM (1GB for conntrack):
set system conntrack table-size 2291200
set system conntrack expect-table-size 204800
set system conntrack hash-size 286400
For a VM with 4GB of RAM (2GB for conntrack):
set system conntrack table-size 4582400
set system conntrack expect-table-size 409600
set system conntrack hash-size 572800
Discussion
- Do you think these settings provide a good balance between RAM usage and system performance?
- Are the proposed hash sizes adequate to maintain efficiency without causing excessive collisions?
- Any other configurations or adjustments you would recommend to further improve performance?
Thanks in advance for your help and opinions!
Best regards,
Roberto
First step “monitor” before you adjust the memory settings. you really need this memory for conntrack or it is another service that needs the memory.
As far as I know the tablesizes doesnt really take much space when they are not utilized.
Rumours on the internet also claims that once utilized it will occupy 320 bytes (on a x86-64 system) per connection.
Also for performance you want to have a 1:1 between the table and bucketsize otherwise (to conserve memory) one table entry could have more than 1 hash entry but this will yield additional CPU cycles spent to locate a particular connection.
The expect table size is as I recall it for related connections such as icmp echo-request vs icmp echo-reply.
Personally I set them all to the same size like so:
expect-table-size "10485760"
hash-size "10485760"
table-size "10485760"
The above would during full load utilize approx 6400MB of RAM (if I have got my maths correct).
That is hash-size equals table-size to get a 1:1 relation (one hash per table row). And then be able to deal with 10M established connections aswell as 10M related connections.
So the max utilization would become 2 (established + related) * 10485760 entries (of each kind) * 320 bytes (per connection) = 6710886400 bytes / 1024^2 = 6400MB.
The recommended hash size for conntrack is typically 1/8 of the table size, as mentioned here.
For a NAT/firewall setup with VyOS, I’m considering using 50% to 80% of the server’s RAM for conntrack. This approach is particularly relevant given that conntrack tables play a crucial role in handling SYN flood attacks and other DDoS scenarios. Larger conntrack tables can help resist such attacks more effectively.
In many sources, the default conntrack max limit is cited as 65536, but on a default VyOS 1.4 installation, the values are:
net.nf_conntrack_max = 262144net.netfilter.nf_conntrack_buckets = 32768net.netfilter.nf_conntrack_expect_max = 2048
Leaving these default values might not be ideal if you anticipate SYN flood attacks and prefer to allocate more RAM (which is relatively cheap nowadays) to your VyOS instances, rather than risk running into out-of-memory conditions.
For context, Fortigate 40F supports up to 700,000 concurrent sessions, the 80F supports 1.5 million, and the 200F supports up to 3 million. These values seem like good targets for VyOS to aim for.
3 Likes
Yes but many “recommendations” are broken.
For example they come from recommendations regarding a desktop system who rarely will use more than give or take 10k concurrent connections. Also the desktop system want to conserve avaialable RAM for other applications while a VyOS box will rarely use other applications since its core business is to route packets and when needed also NAT (both Source NAT and Destination NAT) of packets passing through (and by that match to connection tracking).
A VyOS router dealing with lets say CGNAT will easily consume millions of concurrent connections (or at least must be able to) and in that case I would prefer to have a 1:1 ratio instead of a 1:8 (table vs hash) ratio in order to optimize performance - simply since this box wont use the RAM to anything else than route and NAT (and firewalling).
Thank you, this is a good suggestion and change.
At day job, we’ve for a long time done exactly this with a firewall we maintain that very similar to VyOS can couple with LVS and keedalived.
The percentage you’ve suggested sounds a fair default.
Doing so makes life easy and is certainly helpful in relationship to infrastructure DDoS attack sizing.
In short (unless you do dynamic routing such as BGP or run various containers locally) 1GB for VyOS itself is more than enough for its local services. Anything above 1GB you could assign for conntrack tables (+ BGP when needed).
So I would say that this percentage could increase with the amount of RAM you got above 1GB. But at the same time if your router will track just about 100k concurrent connections then assigning lets say 10M as I do wont help much other than during a sudden increase of connections like a DDoS or some malware spreading etc.
The rule of thumb for connection tracking seems to be 320 bytes of RAM in total per connection to track.
Im not aware of any good numbers to account for when it comes to RIB and FIB when it comes to FRR (such as BGP) - perhaps someone on this forum might know?
A full transit peer (955k-odd v4 + ~204k v6 prefixes, no agg, full attributes) uses about 1.2GB, including ~370MB of RIB. A transit redundant pair with soft-reconfig inbound (common config I hope) bumps that up to about 2GB total each as they’re announcing to each other, same RIB. It’s all tunable with RRs and whatnot but I think that’s a good comparison baseline.
IMO it’s pretty safe to do something like you suggest, just give conntrack a RAM % up to a limit defined by how much baseline VyOS expects to use multiplied by a safety margin, then everything extra goes to conntrack. It shouldn’t reserve the whole table so it can still be used flexibly. If you’re running memory-hungry services it can be tuned down, enabling BGP might also benefit from bumping the “baseline limit”. There could be a tunable parameter set like “max-connections”, “max-services”, “balanced”, etc.
2 Likes