Alternatives to VyOS

[marekm72]

I’m currently running VyOS, would be happy to continue building stable from sources, it was fun while it lasted but it seems to be going away (I don’t have access to the new private repo, if I understand ⚓ T6781 Auto-close pull requests sent to LTS and stream branches correctly), rolling is too unpredictable for my fairly complex setup, so I plan to migrate my two routers to some other solution. This will take some time as I have to do this on a live network with minimum possible downtime.

What are your experiences with rolling your own routers on Debian or Alpine? Any issues that would make one or the other preferable? Debian is well known, I have more experience with it (since 0.93r6) and VyOS is based on it, but it has grown a bit heavy during these 30 years. Alpine is more lightweight, ifstate as a way to configure network interfaces looks promising, and no systemd.

My two routers are running BGP, OSPF and PPPoE servers, dual stack IPv4+IPv6, using 10GbE interfaces but far from saturated (CPU load rarely eceeds 10%, no need for VPP), fortunately no CGNAT as I’m small enough to have global static IPv4 /32 and IPv6 /56 for everyone. It’s a small local ISP, my own single-person business for a few hundreds of customers in rural area where I live.

BIRD seems to be less memory-hungry than FRR. There are probably no open source alternatives to accel-ppp (there is rp-pppoe but it doesn’t do the DHCPv6 PD part), unless there is something in the BSD land (no experience there).

MikroTik might be another option (CCR2116 costs less than subscription and includes ARM64 hardware), but I couldn’t get Delegated-IPv6-Prefix to work (the feature in RouterOS had been requested since about 2014, last time I checked it was not working in 2020, might work today though).

Any other thoughts? (Yes, I already know about the possibility to get free LTS images, but I’m unlikely to qualify - most of my open source contributions were 20-30 years ago, not in the last year that counts.)

[Apachez]

There is nothing that stops you from creating your own “VyOS” from scratch using Debian or Alpine and then add FRR or BIRD or whatever you might prefer.

The main difference is that you wont get a single ISO with unionfs and not a single config but rather need to manually alter all kind of config files (and other kernel parameters etc). The good thing on the other hand is that you wont be dependent on what vyos-configd currently supports (sure you can manually manipulate the frr.conf even in VyOS but still).

To me its the packaging aka “the whole experience” which I like with VyOS when it comes to a software router. I can do the same myself but I save alot of time by just downloading the latest nightly (and test it in my lab before deploying a particular version). That is grab the ISO and the current config - install, reboot - done! Doing the same with a Debian install from scratch will take much longer time…

Looking at competitors there is Mikrotik who have both hardware but also both a baremetal and VM solution (which you must pay to use more than a few days).

Goes for $250 per installation (the VM-edition aka CHR): RouterOS license keys - RouterOS - MikroTik Documentation

In that context OPNsense is also an alternative depending on your needs even if its more geared to be a firewall (with routing capabilities) rather than a router (with firewall capabilities). OPNsense exists both as a community edition (only community support) aswell as a business edition (with paid support). Along with just downloading the ISO vs getting their hardware from https://shop.opnsense.com

Other than that there are few other forks of Vyatta but most of them havent been updated for ages or costs even more if want a support contract (not to mention that they dont even have a free nightly to be downloaded).

But Im happy to listen to what other alternatives there are other than:

• VyOS.
• Do it yourself (Debian or Alpine as base along with FRR or BIRD or such etc - install on baremetal or VM).
• Mikrotik (hardware or baremetal or VM).
• OPNsense (hardware or baremetal or VM).
• Getting a “real” router from Arista or such (hardware or VM (vEOS/cEOS)).

Even if getting a hardware from Mikrotik or lets say Arista costs more than using their solutions as a VM there are also offloading involved so you can have for example 4x25 + 16x100G (CRS520-4XS-16XQ-RM) or
12x25G + 2x100G (CCR2216-1G-12XS-2XQ) which is somewhat tricky to push with a software router unless you have a DPDK/VPP edition of it (which VyOS is in progress to create).

So it boils down to what you want to do with your box. Just deal with BGP as a routereflector or route traffic up to 2x100G then a software router will be just fine.

But if you need more than that you would need to get the VPP edition of VyOS or change to someone with hardware offloading such as Mikrotik or Arista etc.

[tjh]

There’s also ipfire but it’s trying to be a firewall as well. Last I tried it (quite a few years ago) you were force-limited to 4 NICs. Which to me is insane and I instantly ruled it out based on that. It doesn’t appear to support nftables at all, so no flowtable offload etc either.

There’s also OpenWRT which is a robust project. I don’t know you’d want to run an ISP on it though.

I came from pfSense after they imploded with stupidity, I looked at a large number of projects and VyOS was far and away the most robust/functional of the lot.

Myself and a few of my friends keep our ears pretty close to the ground (or at least think we do!) for similar compeditors to VyOS and I’ve yet to find one that’s as good. OpnSense is good but having to do everything via a GUI really makes it a no-go for me, as well as the fact that FreeBSD under a Linux Hypervisor seems to chew a lot more CPU than Linux under a Linux Hypervisor.

Good luck.

[marc_s]

Same. I’ve tried a customized Red Hat Enterprise install as router, pfSense, OPNsense, and Ubiquiti ERs over the last decade and IMHO nothing beats VyOS. The fact that you can define a precise configuration, it has a CLI (also perfect for troubleshooting) and on proper hardware you get awesome performance and stability.
Plus, I like the community here

[artooro]

If you think the VyOS rolling is unpredictable, just wait till you try something like OPNsense which is a roller coaster in comparison

Wouldn’t VyOS Stream solve your issue potentially? We’re still waiting for what exactly that’ll look like, but it sounds like it might be a good middle ground for cases like yours.

[modzilla99]

I also want to give my 2 cents. For our bgp evpn Router / Firewall we originally planned to use vyos and also had it already running. But then all the news dropped…

We now switched to bare Ubuntu 24.04, which works very well. The interface configuration is being done by netplan, which has a nicer yaml configuration compared to systemd networkd, which it generates the config for. Be aware that it has problems in combination with FRR and removes all the nexthops when you apply a new config. For some reason FRR totally disregards that the nexthops are missing and either has to be restarted or the BGP sessions resetted. There is a fix for that in systemd-networkd, so that it won’t remove the nexthops, but that is not in noble unfortunately.

Apart from that minority (as it only happens on config changes that we only apply on the backup node first anyway) it works really well. We use keepalived for use with vrrp and wireguard in HA mode using vrrp, switching the external IP on failover and dropping all peers within the wg kernel. The IP is being announced via BGP.

To facilitate the firewall aspect, we use bare nftables with the config split out in multiple zones and a config file for each zone as they can be sourced within a nft script. We heavily make use of vmaps to jump into chains like you would do with the zone based firewall. (from_x_to_y / from_y_to_x )

Actually we do now prefer the way we do it compared to VyOS since we are puppet users, which makes rolling out updates to our routers really easy. Configuring the firewall has gotten much easier now as well as nftables is such a nice “language” to write rules in.

[spicefrenzy]

A couple of years ago I was planning a project where we would roll our own with Devuan or BSD as a base and manage the configs with Ansible. Unfortunately, the project we would have used this in didn’t get approved and we dropped the idea. Lately I’ve been giving that a second look.

[North-Idaho-Tom-Jone]

re: Alternatives to VyOS

IMO

• It all depends on what you are trying to do.
  Prior to myself implementing VyOS in my ISP networks , I used to use multiple Mikrotik CHR and PfSense and Cisco routers.

For me , VyOS is easier to configure and faster with lower latency than both CHR and PfSense.
I found that setting my BGP and OSPF and routing was easier to configure in VyOS.
For basic firewalls, I find that PfSense is easier to configure.
For customer bandwidth traffic shaping/bandwidth-limiters ( in-line devices ) , in my ISP networks that uses SONAR , I find that Mikrotik CHR routers work well. So far , I have been unable to get VyOS working with my SONAR ISP software.

For CGN NAT in a busy ISP network , then I would say 100-percent go VyOS.

So , for routers , I would say that VyOS & PfSense & Mikrotik CHR & Cisco routers are good. Each has it’s strengths and weakness ( cost , ease of use , throughput/latency , portability , compatibility , internal/external tech support , disaster recovery … ).

I have also tried and tested other routers , but found other routers were not to my liking.

[dtoux]

At this point I would start looking back at Juniper vSRX - way more advanced in terms of features and now it is comparable in price… I’ve also considered building ISOs using yocto project - much better choice than Debian and it is not that hard to do… a week of work by my estimate… but then I would be worried that they close source their VyOS repo as well… so, I’m currently still weighing this option against vSRX…

[marekm72]

Thank you all for your responses. It’s not an easy decision to make, there are always tradeoffs, especially as it all needs to be done carefully on a live network. I have even considered getting a subscription (Standard at 1500/year) but haven’t received any response in 2 weeks (otner than the auto-reply promising reply from the sales team on the next working day). It’s quite expensive but so are other alternatives (if not monetary cost then time spent, which is a scarce resource too).

VyOS Stream - it’s too early to tell, would certainly give it a try at a later more convenient time if the “build LTS from source” option (still available according to the website) was not taken away so hastily. My plan was to first upgrade from 1.3.8 to self-built 1.4.1 when it comes out, while migrating to newer hardware at the same time, then re-use the older hardware and test Stream on it, with an option to quickly fall back to the stable working setup if anything goes wrong. This would also make it easier for me to continue working on a fix for ⚓ T4600 Closing IPV6CP by client closes PPPoE link completely, even if IPv6 is optional and test the fixed accel-ppp well on lots of routers with different bugs before submitting the fix upstream.

One IXP in Germany is already using routers built on Alpine Linux, there was some discussion about making a router distro based on it, still in early stages but ooks promising, though there are still some rough edges (like I have to avoid using unnumbered interfaces as ifState doesn’t support them yet).

Thanks again for all the feedback, and goodbye!