Anyone using vyos for a 10G Router

General questions
1

Is anyone using Vyos for a 10G router? I would love to know the performance and pros/cons if so. I am considering testing it out in our dev environment as our core 10G router between all networks.

1 Like
2

Yep. I use it extensively, with VRRP, OSPFv2/v3 for a HA setup in a virtual environment.

With a few cores of E5-2640v4 processors, I am able to get 30Gbps+ routing. The VM on the D-1541 only handles about 16Gbps routing, and the D-1521 VM does around 18Gbps.

1 Like
3

Just for some baselining, and future references if someone is interested…

I have some VyOSes running on Intel Xeon E5-2670 @2.60GHz.
Theyre on VMware ESXI 6.5, on a HP ProLiant DL380 Gen8.

I do not manage to push 10G. I did some iPerf3 tests and got 6-7G at most.
2 cores, 4 streams.

4

Yeah i was wondering what real world throughput i could get if i set one up, much cheaper for our dev enviroment than a cisco 10G router. Most of the 10G routers i am seeing have 16 cores and a few gb ram so i wonder if i grab a supermicro board with a 16 core cpu and try that

5

Are you talking just routing? Or routing and firewalling. There’s definitely a difference when you roll in packet inspection.

6

Just routing at the moment

7

@snet2 I do firewall, but I do have these lines in my firewall, so the firewalling itself shouldn’t be too CPU intensive when doing iPerf3?
Some other info is that I use vCenter vDistributed Switchport with “route based on IP hash” if someone knows if that would impact performance at this scale.

vyos@vyos# show firewall state-policy 
 established {
     action accept
 }
 related {
     action accept
 }
8

Nah. Kernel packet filtering can’t really go 10Gb. You need some out-of-kernel solution like VPP/DPDK.

That’s why you can’t get 10Gb. Even with established/related accept rules, the packet still needs to be inspected.

2 Likes
9

Would it be possible to do some Qos if we wanted to say limit a interface to 8Gbit as long as the hardware supported it (CPU/ram). We are currently looking at asupermicro A2SDi-16C-TP8F as our dev test board.

All this is routed no NAT or firewall. We will try seperate boxes for the 10G firewall or NAT to see what kind of performance hit they take using those features

1 Like
10

…in 2016. I was testing some platforms (obsd,pfsense vyos…) and Vyos 1.1.7 was best routing performer. I was testing on PowerEdge R610/ 8 cpu/2,6 Mhz with two 10 Gb intterfaces and Vyos was in routing mode score cca 3 Mpps without performances degradations and packet loss… in FW mode - 1Mpps

capture_164_08032016_092023
capture_164_08032016_0920231642Ă—969 325 KB

2 Likes
11

We’ve started working on a 10Gb VyOS solution in our data center. It’s been a while since this topic has been active, and I’m curious if anyone’s made progress. So far our results are very mixed getting it to run on the network appliances we have on-hand (Supermicro with Chelsio SFP+ cards). I’ll be putting together a test lab this weekend and can start reporting back as I go.

The hardware compatibility aside, a major hurdle is still offloading packet inspection to XDP, correct? From what I understand, support for this is still under development and restricted to the 1.4 branch.

It’s been great running VyOS in hypervisors and test labs on 1Gb-capable gear, but as an ISP needing to route lots of optical traffic from a diverse outside plant, we need some better bare metal options. I saw last fall that a partnership with EPS Global would be making certified appliances available, and there look to be a few 10Gb-capable options like this one included, but they look more like CPE than DC options.

I’m hoping to learn more about that this week.

12

eronlloyd,

You may recall I also replied on your other thread about the strange SuperMicro UEFI issue. Interestingly enough, I am currently using Chelsio 10Gb SPF+ cards with VyOS in my SuperMicro chassis (MBR boot only, however, due to the other issue we both hit).

I’ve had multiple bare metal routers in production for some months using those 10Gb cards. If you don’t mind sharing your findings, I’m quite keen to see what results you get as well :slight_smile:

EDIT: I took another look at your post in the other thread, lo and behold one of my SuperMicro production routers is using the same exact card you’re testing with, the T540-CR. My experience has been “so far, so good” running under 1.2.7-LTS.

I only see one strange issue; stack traces in dmesg output reporting “hw csum failure”. Fortunately this does not seem to affect my traffic flows at all, and only occurs for traffic traversing one of my bond interface tagged vlans. If you have a chance to check out bonded interfaces on the T540-CR please do and let me know if you observe anything similar.

1 Like