General NAT vs CGNAT questions, load, performance, optimization etc

Hello everybody

So as i have mentioned in my earlier threads we are trying to use VYOS on a server to provide NAT-services.

We are interested in using them to NAT north of 4000 clients, at the moment we are using rolling releases but if this makes it into production the plan of course is to buy in.

Here are my questions:

  • So far bandwidth have never been an issue, the CPU is what limits us.
    Is there any way to calculate relationship between sessions vs CPU needed?

  • We have only dabbled into the CGNAT-functions, which is more efficient? We have noticed that when logging NAT that takes a sizable amount of CPU-power so CGNAT should avoid that if i am not wrong.

  • Is there something special to keep in mind when designing hardware for this use? This far I’ve seen that the frequency matter more than amount of cores.

  • Running in Proxmox environment, this far I only tried running it as a VM. Linking PCIe directly to the VM gave a big boost in performance. Would it be even better running native or would that not make a difference in your opinion?

  • Would VPP in the future make a big difference? Considering the CGNAT in particular is a static routing-table, would that not be able to be implemented using VPP?

I am seeing alot of potential in this software and it would be great if we could use it in production so I am thankful for any responses.

tomastheswede you ask some good questions.

I manage an ISP in North Idaho. I’ve got 1k-Plus fiber customers and another 1k-Plus wireless customers ( using Mikrotik routers ).

Originally , 20+ years ago , I use Live IP address to customers ( 16 Class C networks ). Although this worked , my networks were always under constant never-ending probes and network attacks from the Internet. If/when I configured my core routers to also firewall these Internet probes to my customers , the result was a high CPU firewall load on my core equipment and slightly slower throughput.

Then later , I switched to NAT ( using A mikrotik vm CHR router ). This instantly dropped all of the Internet probes to my customers but it also created four problems. One was slower throughput to my customers and greater latency. Second was high CPU loads on my Proxmox and vm CHR routers. Third was when I received a notice about one of my NATted customers doing something bad - I had no way to cross reference who the customer was because I only had the outside NAT IP and port address to lookup. Fourth problem was some customers complained about things not working correctly. So NAT kinda worked but it did not work well in in ISP enviornment.

Then later I changed from a CHR Mikrotik NAT router to a CHR Mikrotik CGN-NAT router. Presto , things got better and faster - and now I was able cross reference outside IP addresses and ports to inside CGN-NAT customers. I ran this for years but my Proxmox vm Mikrotik CHR CGN-NAT router was getting CPU hammered big time.

So, then I tried VyOS and replaced my two vm Mikrotik CHR CGN-NAT routers with two VyOS routers that were configured to perform the exact same CGN-NAT functions to my customers. WOW !!! Every one of my CGN-NAT customers got a little faster and the latency to/from customers went way way down. All calls from CGN-NAT customers almost came to a complete silence - as in almost zip - nadda - everybody happy.

So , as a medium large ISP , I big time strongly suggest and recommend VyOS as an ISP CGN-NAT router(s).

In my environment , I have the following:
One VyOS CGN-NAT router for my wireless customes.
One VyOS CGN-NAT router for my Fiber/GPON customers.

Both/each VyOS router has the following conigurations:

  • 8 Live IP addresses ( to each CGN-NAT router )
  • I use 250 ports for/to each inside CGN-NAT customer.

Example:
IP-1 , ports 2000->2249 to 100.64.0.1
IP-1 , ports 2250->2499 to 100.64.0.2
… and continue in 250 blocks of ports.
Then continue to next Live IP address ports 2000->2249 and repeat again and continue adding more internal CGN-NAT IPs.

So as you can see , a single Live IP address can be used to give you an entire C block ( 255 IPs ) of CGN-NAT address space.
Eight Live IP address can be used to give you an entire /21 block of CGN-NAT address space.

EDIT - bonus - with CGN-NAT ( similar to the way I have mine configured ) , there is not need to —> log NAT , because it is easy/fast to cross-reference Live-IP address and port number to the inside CGN-NATted customer :slight_smile:

I myself use the Rolling release for my CGN-NAT routers , and BGP routers , and all core-NOC routers. It’s been stable and fast , and knowing what I know now - I would do it again :slight_smile:

North Idaho Tom Jones.

4 Likes

A little more info on my VyOS CGN-NAT setup.

  • I use Proxmox servers in a cluster configuration. No auto redundancy.
  • Hyper-Threading is disabled.
  • I use 20 CPUs per VyOS CGN-NAT router.
  • All of my vm VyOS routers use normal VirtIO network configurations ( no special config stuff ). ( I have not tried → Linking PCIe directly to a VM , I run mine in a native mode so that I can easily migrate a running VyOS router from one Proxmox server to a different Proxmox server ).
  • I have two 10-Gig BGP feeds ( two IPv4 BGP VyOS routers and two IPv6 BGP VyOS routers ). Total combined nightly peak throughput loads are around 6 to 9 Gigs.
  • I have several VyOS OSPF routers between my VyOS GGN-NAT routers and my VyOS BGP routers.
  • I also use VyOS distribution routers to my customers ( Trunks and Vlans ).
  • I use 40-Gig network cards on my Proxmux servers - into some 100-Gig switches.

** I am able to iperf3 test between two different VyOS routers ( each on a different Proxmox server ) and able to measure about 25 to 32-Gigs.

I still use Mikrotik CHR routers to shape/limit customer bandwidth to each customer. I’ve tried configuring VyOS routers to shape customer traffic so that I can get off of my Mikrotik CHR bandwidth shapers , but I have failed doing this. Soooo, one of my future VyOS projects will be how to bandwidth shape thousands of customers using these data items ( Customer MAC address and/or Customer inside CGN-NAT IP address — Customer UP speed and Customer down speed. ). I’m decent with VyOS & Ubuntu Linux but I’m not a software script programmer where I can cron-schedule to take/read a flat file , parse it and auto update a VyOS bandwidth shaping router. ( also , if there is no entry for the customer MAC or customer IP address , then block Internet to that customer and possibly redirect all http traffic for that customer to one of my web servers ( bla bla bla , your account is on HOLD , please call our office or click on this link to pay your bill and release your hold status ).
Mabey some day I hope…

North Idaho Tom Jones

2 Likes