Hello, we are planning on using BGP with two upstream peers and would like to setup a pair of VyOS routers as our core. Is VyOS production ready? Does anyone have any experience with VyOS in this kind of environment? We are planning on pushing about 1Gbps of traffic initially and will expand eventually to 10Gbps. We need a setup that is obviously stable. Any advice or suggestions or should we just consider using proprietary hardware/software like everyone else? 
2 Likes
We have been using Vyatta/VyOS for years in this type of capacity, it has worked very well. We easily push over 1Gbps with Xeon E3s.
At 10Gbps speeds, your enemy will be PPS. If you need resiliency under DDOS-type conditions or you have very high rate, small packet traffic, x86 won’t last long.
Vyos is very stable and we been using vyos / vyatta for ages without any issues, we have routers with 5 years uptime on them running vyatta/vyos and never had an issue.
My only recent concern is how quiet the project has suddenly become, movement / development on this project seems very slow and your only real support is these forums, which on the best of days, is slow. We are in the position now of replacing our old routers and deciding whether to continue with vyos and risk getting a year or 2 down the line with a product thats dead, or considering the Mikrotik CCR1072-1G-8S+ (over kill) or the Ubiquiti ER-Pro. In these cases they are still being developed on and their support base seems a lot larger.
That is exactly my concern as well. At an old datacenter I used to work with, they had Vyatta as their core routers. I would love to do the same but I don’t feel too confident in not having some type of commercial support for emergencies. At the very least, some more activity on the forum would be promising. I am certainly willing to donate to the project, but I think they are looking more for dev time rather than monetary donations.
We are using VyOS for bgp border routers with 10G interface (inter-vlan). 3xFullViews, several local IXs, near 3k internal routes. ~10 firewall input and ~5 output rules on edge interfaces
This part works well. Some problems with driver updating (eg mellanox version comes with vyos is very, very old and have no qinq support), but not critical.
And some stats from one of routers (i7-4771 CPU, Mellanox ConnectX3 10G adapter)
Uptime: 12:02:47 up 293 days, 21:31, 2 users, load average: 0.17, 0.22, 0.22
Route Source Routes FIB
connected 20 20
static 5 5
ospf 2369 2352
ebgp 562391 562390
ibgp 467 466
Totals 565252 565233
RIB entries 1031319, using 94 MiB of memory
Peers 32, using 143 KiB of memory
But if you dont have good linux specialists in team, maybe better look at paid solutions for soft routers (Cisco CSR1000v, Juniper vMX, Brocade v5400, Mikrotik RouterOS etc…)
That’s really impressive, and you haven’t noticed any performance issues with it being x86 hardware and all?
At this point we had no problems with dataplane (packet routing).
Our tests shows that we can route more than 10-12G on current hardware without any services degradation (or much more with fine tuning of kernel, network stack and drivers).
There is some minor points with control protocols like bgp, but they are mostly related to quagga`s single-thread event processing.
Once we tried to gather netflow with vyos’ default tools, but it was bad idea.
I would then assuming going with a higher frequency CPU (3.5Ghz) would give better performance than going with lower frequency, higher core cpu. (E3 vs E5).
For any doubts on using Vyos as a core ISP router. The cost of deploying a node versus getting a Cisco or Juniper router is so low that you can deploy multiple nodes in HA/cold-standby.
Here’s another active deployment example running on a Dell R320 E5-1410:
Version: VyOS 1.1.5
Uptime: 16:30:39 up 220 days, 15:16, 1 user, load average: 0.18, 0.23, 0.23
Route Source Routes FIB
connected 7 7
static 14 4
ospf 1507 1502
ebgp 46817 46817
ibgp 597486 597397
Totals 645831 645727
RIB entries 1156156, using 106 MiB of memory
Peers 195, using 868 KiB of memory
The downside is that there will be the occasional bug that would require a reboot to fix. But with every update, I have seen the bugs get fixed. (started from Vyatta 6.3 4years ago)
Also, do not use Edgerouter Pro for ISP deployment. It will hang every couple of months and die on DDoS attacks.
VyOS project is bit slow, it´s true, but we working on that part and I hope we will solve this issues in near future. We really like what we do, and there are lot of plans for 2.0 version which will be complete rewrite. Meanwhile lot of work done to move from Debian 6 base to Debian 8 starting from 1.8.x
I think the main issue with the VyOS project is the lack of information regarding its progression and the releases.
For example, the Lithium development roadmap ( http://vyos.net/wiki/Lithium ) hasn’t had a case marked as Resolved since about October last year.
There is no information about when the Next Lithium Beta 2 is going to be released. The nightly iso builds seemed to stop working in Jan (http://dev.packages.vyos.net/iso/lithium/amd64/)
What bug exactly is requiring a reboot to fix?
Edit : nevermind, saw the announcement here - http://forum.vyos.net/showthread.php?tid=26532&pid=29796#pid29796
Currently bugs that require reboot are mostly due to bgp/routing. An annoying one is when there is a stale BGP route that is no longer routable and that can’t be removed without a reboot. Quick fix is to override the stale route by poisoning your route table with a smaller prefix. My most recent bug which I had to reboot was SNMPD and BGPD processes crashing every few hours. After the reboot it’s been running fine.
Hi! I just wanted to post an update to this thread after it’s been about three years.
Here are our stats:
vyos01:~$ show version
Version: VyOS 1.1.6
Description: VyOS 1.1.6 (helium)
Copyright: 2015 VyOS maintainers and contributors
Built by: maintainers@vyos.net
Built on: Mon Aug 17 03:58:33 UTC 2015
Build ID: 1508170358-a3033d5
System type: x86 64-bit
Boot via: image
HW model: PowerEdge R620
Uptime: 00:33:22 up 527 days, 1:47, 1 user, load average: 0.00, 0.01, 0.05
It is being used as our core router for our ISP (about 100 Business Users). Full BGP routes from two providers.
My concern after I read Anton’s post is that we are using two Ubiquiti EdgeRouter pros for our edge devices and then use OSPF and iBGP for our internal routes. You have me worried with the Edgerouters dying after a few months of use. So far we have had no issues.