Is there any chance that the VyOS team will change their mind on ISO builds?

There does appear to be a pretty significant community backlash on this decision, so I thought I would ask publically for a definitive response.

If the answer is ‘no, this is permanent, and we are not going to change our minds’, even after all the external developer responses, then I suspect the same thing is going to happen to VyOS as happened to Vyatta!

So, here’s the opportunity for you guys to admit you made a mistake and allow people to build ISOs again, or, double down and confirm that this was a decision that won’t change!

For those that are unaware of what this is about, it is no longer possible to build the stable branch of VyOS yourself - only the unstable branch. More information here: Community, Contributors, User Base and LTS builds

4 Likes

I was about to open a bug report, because of building of v1.4 failed… besides it worked a while ago :slight_smile:

The problem seem to be with access to 1.4 Sagitta Repo :

Reading package lists... Done
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/InRelease/binary-amd64/Packages  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/InRelease/binary-all/Packages  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/InRelease/i18n/Translation-en  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/sagitta/binary-amd64/Packages  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/sagitta/binary-all/Packages  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/sagitta/i18n/Translation-en  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/main/binary-amd64/Packages  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/main/binary-all/Packages  403  Forbidden [IP: 104.18.31.79 443]
E: Failed to fetch https://sagitta-packages.vyos.net/dists/sagitta/main/i18n/Translation-en  403  Forbidden [IP: 104.18.31.79 443]

It looked as an error, or some over-quotat to me… but as i came by the post you linked to, i’m a bit sad about that !

/joen

So here’s my thoughts on this, from a decently technical but non programmer’s view point.

but if you want to build LTS releases, the process of building custom packages, assembling them into repositories, and building images will be on you.

Honestly, I’m okay with using my own resources to build non rolling releases, and I’m also happy to run an instance of a rolling release on my test server, but as my main home router, I need the stability that an LTS provides. I don’t use VyOS for my job, I’m just an enthusiast / prosumer that loved the idea of this project over your competitors.

BUT, if I came to VyOS today, it’s too complicated to get a stable build off the bat, I would just go somewhere else. Heck, even my first build of VyOS a couple years back took time to learn, and that’s a road block to a lot of individuals, individuals who may have eventually become supporters or contributors.

I hope the VyOS team realizes that these changes just push more people to create non-official build sites for a VyOS ISO. And in realizing that, either help come up with easier to follow instructions on how to build our own ISO’s, OR reverse the decision they have made.
I truly believe this decision will prevent you from gaining more of your community, and will only isolate your product, if by no other way then preventing less technical people from trying VyOS out.

Finally, Id like to share the experiences with rolling releases that myself and a couple friends have had over the last 2 years or so. One friend has had very good luck with Rolling releases, and never had issues with them. Myself and another friend have never had a bug free release. My very first build was a rolling, and it had extreme speed issues, solved by going LTS. Not the best introduction to a platform I must say, but I understand rolling releases are basically like a beta, you expect to have bugs. Sadly, I have no knowledge on how to troubleshoot or figure out what was causing this bug, but luckily someone else had made an entry on the tracker already. Rolling releases are great, but not for the majority of people who want a product to just work.

A last second thought, I know VyOS accepts donations through open collective, but have you ever thought of adding more consumer level packages to your main store? Something for the home user, something for the small business owner who is a one man shop but still wants to support / purchase the builds he uses. Just a thought.

Anywho, thanks for reading, I hope the Dev’s get to read this and think on it.

1 Like

Its an misintepretation that the LTS versions would be somehow magically more stable than the rolling releases.

In fact the LTS releases are missing fixes from the past 3-6 months not only VyOS related but also the backend stuff such as Linux Kernel, FRR, Debian packages etc.

So in short if you need commercial support then go for the enterprise edition aka “LTS”, if you dont - then save some money and go for the community edition aka “rolling release” instead (and either donate or participate through reporting, fixing and document VyOS).

I was also thinking about some kind of “SOHO Liscence” or something in the like.

If i’m not wrong, entry price for “Stable” releases : from 1200 to 15 000€ / year… while, PfSense for example range from 130 to 800$/ year… :slight_smile:

This is maybe the root cause, of a small community : if no one can afford the price, what’s the matter in getting involved ? Somehow reminds me of VMWare early days… :slight_smile:

As some suggested : what’s the point in supporting them financially ?
Downloading dev release, and reporting bugs, seem a fair trade :slight_smile:

I really don’t want to make this into an argumentative thread, but saying LTS isn’t more stable then a rolling release because it doesn’t have as many patches is not accurate.
The whole point of LTS going through RC phases and extensive testing is to work out as many bugs as possible.

Yes, rolling releases will have more patches, features, etc, but each release is only minimally tested and bugs can easily creep in.

I 100% want to run either an LTS or atleast an RC/EP as my main home router, as I can’t have things breaking when I’m not home to fix them.

I run rolling on my dev server because it’s unimportant. if something breaks, I can update to the latest rolling, or if I have time / knowledge, try to figure out the issue and report it.

Saying home users should use Rolling as the defacto standard and they are just as stable is how you lose people thinking you have a crappy product, when in reality, VyOS is amazing.

While a Cisco Firepower who is based on opensource project Snort goes for $3000/year per device and upwards (where the VyOS pricing is for unlimited amount of devices).

Which gives that a 5 year subscription of VyOS for $6400/year gives you unlimited amount of on-premises deployments while just 3 of the smallest Cisco Firepower units will cost you more than $9000/year.

Let’s not pretend that the gentleman isn’t building, or attempting to build, a 1.4 Sagitta image today.
Stating:

backend stuff such as Linux Kernel, FRR, Debian packages etc.

is absolutely, completely useless because a new build of the 1.4 branch today will bring in these fixes.
I really hate reading this in the threads where this is discussed.

If a user were to still have access to the official Sagitta repos, they would pull in the recently compiled (within the last week) kernel, FRR, vyos-1x, and netfilter, for example, and as part of the routine build process, also pull in the latest debian packages.

No user using the official Sagitta repos was ever able to guarantee building the tagged/pinned/pseudo-clone of the official iso as even the official Sagitta repo is always rolling - at least it is today. Perhaps this slows post 1.4.0 GA.

Now, I do also believe there is a bit of misconception about the current branch vs n LTS branch. LTS would, by definition, include a subset of smoketests found in the current release, excluding removed/reworked featured (such as ISC-Kea), and the current branch is therefore tested equivalently via automation. The VyOS team even states that they use the rolling release within their infrastructure in some locations/functions.

Your quoting of pricing of these devices is exceedingly meaningless as well. The user is a stated home user. Many companies offer home user licenses for reduced cost, with the understanding that there is limited to no support. For example, a home user could run an NGFF Palo Alto device with full security updates/definitions for under $800/year. Granted, this is an NGFF, which is not what VyOS purports to be.

All this said, there are instructions floating around on this forum and elsewhere on building out your own repos and associated .iso images, some even include executing the suite of smoketests.

1 Like

Yes its accurate.

You can just look at when LTS 1.3.6 was compiled and then make a diff of all changes that Linux Kernel (stable), FRR, Debian packages and VyOS unique stuff for todays nightly release of 1.5 to find out that 1.3.6 have plenty of bugs thats fixed in 1.5 nightly.

This gives that the nightly IS more stable than LTS.

This gives that only time when LTS is as stable as the nightly is on its release date. Already the next day things have changed and nightly have moved forward and fixed stuff while the LTS is standing still and the bucket list of issues and bugs and security vulns are growing… until a new LTS is released and the story repeats itself.

I think people incorrectly read in things in the naming of “LTS” which just isnt correct.

Personally I would prefer if the releases were called “Enterprise edition” (with enterprise support which is what you pay for when it comes to the enterprise license - as with for example MySQL and MariaDB, aka paid support) and then “Community edition” who is free of charge with community support (and if you want something more than that like NBD response times or “within 1 hour” or whatelse then go for the “Enterprise edition”). But this is just a play with words - in reality nothing would change since the LTS already today have a paid support included with SLA of response times where the rolling edition is free of charge with community support. And that the community edition have newer releases including fixes not yet implemented in the LTS (aka Enterprise) edition.

If you build a 1.4 release today you are not building a LTS release since that was compiled weeks ago. And if you try to build a 1.3.5 LTS that was built months ago.

Only time you can build a LTS yourself is if you build it the same date as when the original LTS was built. Building it any other day will get you updated Debian packages which have fixed issues (otherwise new versions wouldnt have been released).

This gives that the LTS versions will ALWAYS lag in terms of fixes and by that be less stable than the nightly releases who use the latest stable of everything.

So again, people are incorrectly reading in things with LTS which is just not true when it comes to stability. Probably confused by the selected name “LTS”.

If you look at the content of 1.5-rolling you will see that all parts that builds 1.5-rolling are from current stable branches of Linux Kernel, FRR, Debian (Bookworm) etc.

It would be a different case if the Linux mainline release candidates were used or the alpha releases of FRR or the Debian testing were used as base.

If you build a 1.4 release today you are not building a LTS release since that was compiled weeks ago. And if you try to build a 1.3.5 LTS that was built months ago.

As I said, it’s exceedingly tone deaf to state this.
Folks are simply referring to building with the “LTS branch” and following the formerly documented steps Build VyOS — VyOS 1.4.x (sagitta) documentation, which, by using the official Sagitta repos, is effectively, a rolling LTS build. Even so, there’s still less of a chance in Sagitta than Circinus that the user is testing possibly odd changes in packages such as vyos-1x.

This gives that the LTS versions will ALWAYS lag in terms of fixes and by that be less stable than the nightly releases who use the latest stable of everything.

Yes, but the VyOS code from that LTS branch has been extensively tested, and that is what people want vs a rolling nightly build that may have huge bugs no one has caught yet. That’s my point.

I don’t need an enterprise license for $8000 a year, I need one instance for my home. So please stop comparing individual home users and huge businesses as the same. If my server that has VyOS on it dies tomorrow, I can just plug in my ISP modem/router combo and be back online in minutes.


All this said, there are instructions floating around on this forum and elsewhere on building out your own repos and associated .iso images, some even include executing the suite of smoketests.

This is good to know, I’m going to have to go looking for these, as I would like to upgrade my instance to 1.4ep2, while I test out 1.5 rolling on my dev server.
I really hope VyOS will make some official instructions on their docs. I would rather not have to rely on third party instructions, or repo’s when I can follow official instructions.
As another point to the above, after reading some other threads on this, the fact this change is causing so many people to host their own images and build chains, I believe will end up hurting VyOS and doing the opposite of what they wanted. All it takes is one person to add an extra tainted package to the instructions, and how many people will blindly copy and paste them?

Anyway, if there is one thing I would say to take away from this thread, it will be don’t hurt your community to get back at the few. Make VyOS easier to access for the regular joe, produce a great product, heck a great platform, and the community will come together.
Adding more and more roadblocks will just make your community more selective, and it will spread to less people. Being CLI only, already weeds out a lot of the public and shortens your reach. Don’t make it harder, please.

Hopefully you don’t have to look too hard.

As much as folks wish, the devs simply aren’t reverting the repo access change, and the steps to produce/seed a repo are challenging at best to document.

The good news is that, as one of the devs stated, while the initial set up is burdensome, it pays dividends once set up. Additionally, once you have it set up, you have full control over your router image and its contents.

What folks need to also understand is that the unofficial images an individual produces are wholly unsupported. Should you find a bug in your custom built image, the devs will not care, and they will ask you to reproduce the issue on the latest available-to-you official image.

2 Likes

I see this argument so often, but I never understand: Why does your home network need a Long Term Support version of Vyos? Are you planning on ringing up to get support if something goes wrong?

Yes, 1.5 may have breakages/changes in it if you’re updating all the time, but why would you be updating all the time if you’re also screaming for stability? If you want your own stable version, then don’t upgrade every 3 days. Deploy a 1.5 version that works and be happy.

People seem to forget/ignore that Vyos is built atop Linux - is your suggestion that the Linux 6.6.x kernel isn’t stable? That FRR in 1.5-rolling is core dumping all the time? What is it about a fixed-in-time version of 1.5 that’s “unstable”?

If 1.5-rolling was core dumping left right and centre, or the kernel was locking up every 3 days, I’d understand your “I need 1.4 for home” argument. But it’s not, so I don’t.

3 Likes

Yes, but as long as the ~775 or so Debian packages isnt pinned with the version on each and every one of them (which they are not when you look at the source code - only a few are) when the original lets say 1.3.6 LTS was built at around 14 feb 2024 your edition of 1.3.6 LTS even if you use github tags will NOT be the same when you build it today.

It will have the Debian packages available as of today 29 may 2024 which is similar to how the nightly 1.5-rolling is being built (difference from 1.3 is that Debian 12.x Bookworm is being used for 1.4 and 1.5 currently while Debian 10.x Buster is used for 1.3 series of VyOS but we can do this compare with 1.4.0 LTS once thats released).

So it boils down to do you trust the current stable Debian 12.x Bookworm more or less compared to the old stable Debian 10.x Buster?

And if the Debian packages are newer in your build of 1.3.6 LTS is that better or worser than the original 1.3.6 LTS built by VyOS themselves?

You really should read up on the changelogs from Linux Kernel stable, Debian Bookworm (or Buster) and FRR stable.

The bugfixes in newer branches ARE the differences compared to whatever version of LTS you are currently incorrectly thinking would be more stable.

Hey TJH, thanks for the reply, here are my thoughts (and many other’s I’m sure) on this subject.

I see this argument so often, but I never understand: Why does your home network need a L ong T erm S upport version of Vyos? Are you planning on ringing up to get support if something goes wrong?

Maybe the naming scheme that VyOS uses is a bit confusing in this regard. LTS could easily be compared to a “slow update” channel. Where as updates are extensively tested and ‘proven’ before they are pushed to clients where stability is paramount.
Microsoft would call this the Semi-Annual Enterprise Channel. (they also have Monthly Enterprise, and Current Channels)

Here is an excerp from microsoft on their three update channels:

We recommend Current Channel, because it provides your users with the newest Office features as soon as they’re ready. If you need more predictability of when these new Office features are released each month, we recommend Monthly Enterprise Channel. In those cases where you’ve select devices that require extensive testing before receiving new features, we recommend Semi-Annual Enterprise Channel.

Lets look at something a bit closer to us though, Debian.
Debian has atleast 3 active branches or sometimes called releases: Stable, Testing, and Unstable. They also add OldStable, aswell as LTS releases.

Some organizations call Unstable, the Develop branch, or the Nightly Branch.
here is the description Debian uses: “The unstable distribution is where active development of Debian occurs. Generally, this distribution is run by developers and those who like to live on the edge.”
VyOS uses this kind of branch for their Rolling Releases.

Then we have the Testing branch. Sometimes this can be called the Release Canididate.
“The testing distribution contains packages that haven’t been accepted into a stable release yet, but they are in the queue for that. The main advantage of using this distribution is that it has more recent versions of software.”
VyOS doesn’t really have a testing branch most of the time. The exception is when they are getting ready to put out an LTS, they start feature freezing, and doing more extensive testing on their code, backporting bug fixes as they are found.

The next branch, is the Stable: “This is the production release of Debian, the one which we primarily recommend using.”
This branch is what most ordinary users would use, and almost all businesses would want. These are closest to the VyOS LTS releases. They aren’t updated quickly, everything “just works” on them, and you don’t have to worry about “was it my configuration or the build that is making mystery bugs and breaking this feature”. Bug fixes and the occasional feature is backported, but you really aren’t getting anything new.

Finally we come to the Debian LTS branch.
“Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian Security team, but by a separate group of volunteers and companies interested in making it a success.”
Basically, Debian core developers have stopped working on this, The main security team have stopped working on this, and it’s only really around because companies take what feels like decades to upgrade their infrastructure.

Hey Apachez, you are right of course, that all the debian packages aren’t pinned with each version of VyOS. You are also right that one of these packages could introduce a bug within the vyos code, and therefore break something.
That is a much smaller possibility then the rolling release code breaking something.

Most members of the public would (and should) never use a rolling / beta / develop / nightly branch of software. They should always be on a stable release. Sure stable still has bugs, but those are much fewer and farther between, and usually not breaking like a nightly release’s bugs would be.

The fact VyOS only has a CLI interface (officially), raises the average knowledge of it’s users quite a lot, but you still shouldn’t expect users to have to try out 2-3 releases before finding one that works.

Perhaps 1.5-rolling has been really reliable recently. This wasn’t my experience with 1.4-rolling when I first started with VyOS. I built my first 1.4-rolling, installed it, set it all up according to the quick start Doc, didn’t deviate from the Doc at all, and I had issues, it was related to the firewall. At this point I had spent hours troubleshooting as I figured it was user error.
A few days later, I came back to the project, and then read the forums and found this was an issue but was fixed in one of the next nightly builds. Cool.
So I built another nightly, installed it, tested it and yup, firewall bug fixed. I continued setting up the config, but for my situation instead of the quick start, and ran into another bug. This time it had to do with upload / download speed. Once again, I tested and made sure it wasn’t user error, not quickly as I had to first learn how to configure everything properly before I could confirm it wasn’t my config.

I then Downgraded to 1.3LTS, adjusted my config for the firewall differences, and everything worked right out of the gate.

How many people would have this patience? How many people are technically inclined enough for this?
This is why VyOS has a small community, they have very high barriers to entry for the average joe.

Anywho, we are quite far off topic now, sorry for the two giant walls of text.

Yes, this is called out specifically in their blog post.

The era of the perpetually broken rolling release is long gone. Our test suite is not perfect, and we are constantly working to expand and improve it, but if an image doesn’t boot or can’t load a relatively wide selection of configs with various features, it’s not even published.

Why would you think there are these multiple DIFFERENT stable and longterm releases of Linux Kernel currently instead of a single one?

https://kernel.org/

mainline: 6.10-rc1 2024-05-26
stable: 6.9.2 2024-05-25
stable: 6.8.11 2024-05-25
longterm: 6.6.32 2024-05-25
longterm: 6.1.92 2024-05-25
longterm: 5.15.160 2024-05-25
longterm: 5.10.218 2024-05-25
longterm: 5.4.277 2024-05-25
longterm: 4.19.315 2024-05-25
linux-next: next-20240529 2024-05-29

And why do for example 5.10 have 218 releases?

Currently the latest longterm 6.6.x is being used for 1.5-rolling:

https://vyos.dev/T5887