ISP Edge Router CPU-NIC recommendations

Hi guys,

So it’s been a while since I’ve been back on these forums. I’ve now reached a new point where I now need to start routing more traffic across my network, rather, I’d like to route traffic faster.

So what I currently have in place, are a pair of Juniper MX5-T’s, they’re great, however for me to utilize the 10Gbe ports, I’m going to have to spend a lot of money on licensing. So that puts it out, I’m now looking at VyOS as a replacement, I considered a standard Linux OS with FRR, but why?

I currently have in place at each DC, a pair of Mikrotik CCR2004-1G-12S+2SX routers, however, as feature packed as the Mikrotik’s are, there are just a few things I don’t like about their methodology, etc. And there are a few niggly things that are kind of bugging me with it. I’d also love to move their usage as my broadband gateways rather than my edge router, seems more useful to have them there, since they can handle Q-in-Q a lot better etc.

So back to VyOS I’m drawn. So, here’s what I’m wanting to build, and recommendations from the community would be awesome!

I’m looking to build a pair of Edge Routers, I currently operate out of 2 separate Data Centers. Let’s just call them DC1 and DC2.

DC1 is the primary location, I have 4 Upstream transit providers, each providing with their full routing tables, around 1,266,000 routes in total are currently loaded. Additionally, DC1 is also connected to 2 local IX’s, and 2 international IX.
All 4 Upstream Transit providers are providing me with a 10Gbps. Local IX connections are 10Gbps, 2 international IX connections are 100Mbps

DC2 is the secondary location, here I have 2 Upstream Transit providers, again full routing table, similar number of routes. Here I also have 1 local IX connection and 2 international IX connections.
Both Upstream Transit provider connections are 10Gbps, local IX connection is 500Mbps, and 2 international IX connections are 100Mbps.

What would you guys recommend I look at, CPU and NIC wise? Obviously server will hopefully run on either a smallish SSD or NVMe drive, and will have 32GB of RAM, just cause, however, I know I’ll need to plan the NIC interfaces. What I was hoping to do, was install 4x Dual Port 10Gbe interface cards, offering a total of 80Gbps throughput. Or a Dual Port 40Gbe NIC with 2x 10Gbe breakout cable, and in both instances, bonding the connections into a single LAG port into the 10Gbe switch, which I’m planning for, in the new year to upgrade the core 10Gbe switches to 40Gbe.

Would love to hear about what others are doing when working with this sort of network traffic.

EDIT: I forgot to add. These routers will be performing pure BGP routing functions only. No NAT or anything fancy. Just BGP peering and traffic engineering.

1 Like

I am curious to see how did this turn out?

We use VyOS as our Production Router. Doing BGP and Traffic Shaping with multiple bonding interface. This is from one of our Routers.

We are using VyOS for past 2 Years, haven’t faced any issues. Currently have around 8 Bare Metal/x86 PCs running for my ISP.

1 Like

The CCR2004 is not a bad hardware platform but if you’re still using ROS6 and full tables you need to either update to (the not yet released) ROS7 or accept packet loss during BGP updates…BGP is still single threaded in version 6.x and the algorithm isn’t the best.

We were using one for a core router and gave up and replaced with a Vyos router on a Lanner NCA-5510 with Intel XL-710 based 40G NICs - much happier.

Not as much traffic as the above poster though - but only a E5-2620 not a 2690.

This page:

https://support.vyos.io/en/kb/articles/system-optimization-4

is your friend.

For Edge routers we like these:

Supermicro SYS-1019D-FRN8 but if I were buying again I’d get more cores.

Hi rgrant,

Thanks for the advice. Yeah, there a few niggly things that we’re finding are a bit of a pain to work with with the CCR2004’s. They are good routers, however, for what we’re wanting, we’re finding that working with them a bit painful at times.

You are correct, we are currently running ROS6, and ROS7 is still not mature enough. We’ve also found that, running both IPv4 and IPv6 stacks, we’re running close to running out of RAM. I think at the moment we’re down to around 500MB of available RAM.

We’ve decided to a pair set of HP DL360 G8’s with E5-2670’s with 64GB of RAM and a pair of a Mellanox CX314A 40Gbe NIC’s.

Once we’re out of lockdown and able to move freely around where we are, we’ll begin work on building those out.

Thanks for the link, will check it out!

Regards,
D

1 Like

Are you playing with the server before prod? Im interested in knowing what performance youre getting, as Im looking to upgrade from the exact same CPU. (CPU upgrade - looking for recommendations - vmware ~60 vm's)
I suppose your run vyos on bare metal and no hypervisor?
What are your single connection performance, PPS/Mbps etc?