Really dangerous 1.4.1 bug? Default password came back

Bugs
1

I updated a 1.4.0 box to 1.4.1

Before I started, I ran ‘save’ just to be sure my active configuration was saved.
Then I began the upgrade using the ‘add system image’ to add the new ISO. I accepted the default name and answered yes to all questions, including the configuration copy.

I rebooted and 1.4.1 was now live! My configuration seemed to be working and traffic began flowing as expected. I later found a VERY SERIOUS problem however. The ‘vyos’ user now had the default password of ‘vyos’, allowing anyone to login to the box with a default password.

This is 100% not the case on the pre-upgrade box (Confirmed by snapshot)

Looking into what might have happened, I notice two things:
Comparing my pre-upgrade config to my post-upgrade config, there is indeed some sort of syntax change that happened. The encrypted password now starts with ‘$6$rounds=656000’, where as it did not include the ‘rounds’ before.

Pre-Upgrade:
set system login user vyos authentication encrypted-password ‘$6$xxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxx’
set system login user vyos authentication plaintext-password ‘’

Post Upgrade:
set system login user vyos authentication encrypted-password ‘$6$rounds=656000$yyyyyyyyyyyyyyyyyyyy.yyyyyyyyy’

The second thing I noticed is that the pre-upgrade for some reason had two password lines for the ‘vyos’ user. It had the normal encrypted line, but then also had a plain-text line for reasons unknown. Maybe this is related? The end result was that the previous configuration had a strong password for the ‘vyos’ user, and the post-update configuration was wide open and exposed. I don’t know why the previous configuration had the extra plain-text line, but if anyone else had this too, your configuration may now be wide open to compromise.

My best guess of what likely happened is that it basically processed the commands twice, once setting the password to the encrypted string because it was first, then a second time, using the plain-text of null, which set the encrypted string to default. The final configuration then only contained the encrypted string, because that is all that should actually be there anyways.

This is really concerning as I may have very well not noticed this at all. I could have upgraded the system, logged back in with my own PERSONAL user, tested that everything on the network was working, and called the upgrade a success. Months later, the system has been fully compromised by the default password, the company has been pwnd, and I get fired for leaving a default password configured, even though I never did.

Once fixed, this probably deserves new ISOs to be pushed to prevent production systems from being compromised on upgrade.

2

I still have no idea why that plain-text string was in the config at all, but here is something I found in further testing.

On both 1.4.0 and 1.4.1,
You have a ‘vyos’ user configured with an encrypted string, representing a strong password. This password will indeed work to log in.

If you enter the command:
set system login user vyos authentication plaintext-password ‘NewGoodPassword’

Now before commit you do ‘show system login’

You will see both the old encrypted string and the new plain-text password exactly as you entered it, with a + next to it, indicating that it will be added on commit. If you commit, the plain-text line is consumed by the configuration parser and a single new encrypted password line is written out to the config. You now only have only one line, and it is the encrypted password.

If you do the same thing, but this time set the plain-text password to ‘’
You now commit and end up with an additional line. You have the original encrypted line, which still actually is the current and active password for the user. You also have an additional line that says the plain-text password is ‘’, which has no affect on anything and you cannot login to the system with null or ‘vyos’. The encrypted password is what you will need to login to the system.

It would seem however that when the system upgrade parser runs when you upgrade to the next version of Vyos, it will pickup that plain-text line and parse it into a wide open default ‘vyos’ encrypted password.

3

Hi @James2012,

That’s a scary post to see just after a release and during the holidays! Thanks for letting us know — I have to admit I was in a bar when I noticed it and ran to test as soon as I finished my drink.

To let you know: the plaintext-password '' bit is a red herring. That config node is a hack to allow entering plaintext passwords from the CLI, but it’s always converted to encrypted-password when you commit a password change. You will see it empty in all versions going back to 1.0.0. We should eventually get to making that part more intuitive, although I’m not yet sure what a good design might look like.

To the actual security issue. My findings so far:

  • The explicit rounds (the number of hashing rounds) bit was actually new to 1.4.0 (not 1.4.1) — in new installations it’s there.
  • Upgrading a newly-installed 1.4.0 to 1.4.1 didn’t reproduce the issue: I couldn’t login with vyos:vyos, only with a password I set at installation time.
  • Upgrading from 1.3.8 straight to 1.4.1 also didn’t reproduce it for me.
  • Installing 1.3.8 and upgrading it to 1.4.0 and then to 1.4.1 sequentially also didn’t do the trick — I still could log in only with the real password that I set when installing 1.3.8.

The interesting bit is that in my lab setup, passwords in configs carried over from 1.3.8 still used the old format, without explicit hashing rounds. Which likely means something happened during the upgrade (likely at the config migration stage) in your installation that didn’t happen in mine — for example, because your config uses some features that my lab setup didn’t.

Could you tell us more about the upgrade history of that box? We will also need a config from the last good version — you can send it in a support ticket, or attach an anonymized version here.

2 Likes
4

I was just able to reproduce this on a fresh install on a new VM.

Steps to reproduce:

  1. Create a new ESXi VM with a single NIC
  2. Install vyos-1.4.0-generic-amd64.iso
  3. Boot it up and install system image. Accept defaults and set password where required.
  4. Reboot now onto the installed system.
  5. Login with the password you created in the installer.
  6. Set an IP address on the interface. Set a hostname. Set SSH to listen on the IP address you just configured. Commit and save.

Here is what my whole config now looks like:

set interfaces ethernet eth0 address '10.0.200.51/24'
set interfaces ethernet eth0 hw-id '00:0c:29:3f:86:ac'
set interfaces loopback lo
set service ntp allow-client address '127.0.0.0/8'
set service ntp allow-client address '169.254.0.0/16'
set service ntp allow-client address '10.0.0.0/8'
set service ntp allow-client address '172.16.0.0/12'
set service ntp allow-client address '192.168.0.0/16'
set service ntp allow-client address '::1/128'
set service ntp allow-client address 'fe80::/10'
set service ntp allow-client address 'fc00::/7'
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set service ssh listen-address '10.0.200.51'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'vyos-test'
set system login user vyos authentication encrypted-password '$6$rounds=656000$yXnuLSXCfOspwIsM$nhFNU7.AwZzt/5fVLn.tXH0tQA9aAcGTrABnbNu9skGBkWAKXCp05g3LWlRm8wyibDpC34mYt9BrMfERKf31z0'
set system login user vyos authentication plaintext-password ''
set system syslog global facility all level 'info'
set system syslog global facility local7 level 'debug'
  1. SCP vyos-1.4.1-vmware-vsphere-amd64.iso to the /tmp/ on your new VYOS box.
  2. Install the new image:
vyos@vyos-test:~$ ad system image /tmp/vyos-1.4.1-vmware-vsphere-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4.1)
Would you like to set the new image as the default one for boot? [Y/n] y
An active configuration was found. Would you like to copy it to the new image? [Y/n] y
Copying configuration directory
Would you like to copy SSH host keys? [Y/n] y
Copying SSH host keys
Copying system image files
Cleaning up
Unmounting target filesystems
Removing temporary files
vyos@vyos-test:~$ reboot
Are you sure you want to reboot this system? [y/N] y
  1. When the system comes up, you may now log in with THE DEFAULT system password. Your old password will fail. The config now looks like this:
set interfaces ethernet eth0 address '10.0.200.51/24'
set interfaces ethernet eth0 hw-id '00:0c:29:3f:86:ac'
set interfaces ethernet eth0 mtu '1500'
set interfaces loopback lo
set service ntp allow-client address '127.0.0.0/8'
set service ntp allow-client address '169.254.0.0/16'
set service ntp allow-client address '10.0.0.0/8'
set service ntp allow-client address '172.16.0.0/12'
set service ntp allow-client address '192.168.0.0/16'
set service ntp allow-client address '::1/128'
set service ntp allow-client address 'fe80::/10'
set service ntp allow-client address 'fc00::/7'
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set service ssh client-keepalive-interval '180'
set service ssh listen-address '10.0.200.51'
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'vyos'
set system login user vyos authentication encrypted-password '$6$rounds=656000$pUR/DUKv1Zj09zpU$EWYw1wxs4jyq.58XlQgQTknY58JXCTit9quEmYyPpJp6u6gOejmmt/TshbwsSq/t/BniXLJ4NUhpCORM.H4IX/'
set system syslog global facility all level 'info'
set system syslog global facility local7 level 'debug'

You will see that the null plain-text password is gone. Seemingly processed into a new encrypted password, overwriting the old one.

You will also see that the hostname has been reset to ‘vyos’. Yes, totally unrelated to this otherwise very important topic, but it happened in my prod system and my test system… so noting it here.

5

I’m guessing the issue is something like this?

You install from 1.4.0 ISO and set the password at install-time. It seemingly adds this extra plain-text null thing after the encrypted password.

set system login user vyos authentication encrypted-password '$6$rounds=656000$yXnuLSXCfOspwIsM$nhFNU7.AwZzt/5fVLn.tXH0tQA9aAcGTrABnbNu9skGBkWAKXCp05g3LWlRm8wyibDpC34mYt9BrMfERKf31z0'
set system login user vyos authentication plaintext-password ''

If you EVER change the vyos password again, you have just mitigated the problem, because your new plain-text password will overwrite the old, and once commited, it turns into an encrypted password.

If you don’t change it again, you will still have the plain-text null password. When you upgrade, this turns into a default password of ‘vyos’ as an encrypted-password. You are now pwnd.

6

Ok, I have no idea. I assumed that setting a new plain-text password would fix this problem once commited, but I just tested it and it does not. I have no explanation at this time.

This time I did the install, rebooted, changed the password, confirmed that the plain-text password disappeared from the config (it did), then performed the upgrade.

I still have a default hostname and a default password after the upgrade.

7

I presently cannot come up with an upgrade scenario where this problem does not happen. That is to say it is happening 100% of the time. I end up with a default hostname and default password.

I am using the following ISOs, which came from official sources.

vyos@vyos:~$ sha256sum /tmp/vyos-1.4.0-generic-amd64.iso
b276d133cbc949507be6c08faa57ee035d2d6a6cd066cc41f0c0eff1fd5235e1  /tmp/vyos-1.4.0-generic-amd64.iso
vyos@vyos:~$ sha256sum /tmp/vyos-1.4.1-vmware-vsphere-amd64.iso
a2ef7fd2cbbbd3afc257352eb99c0cc575c24314b2321f00083553d1d4eab248  /tmp/vyos-1.4.1-vmware-vsphere-amd64.iso
8

OK, woke up in the the night thinking about this and resumed testing.

More info:
My original install from the vyos-1.4.0-generic-amd64.iso shows the following:

root@vyos-test:~# ls -l /etc/cloud*
ls: cannot access '/etc/cloud*': No such file or directory
root@vyos-test:~# systemctl status cloud-init
Unit cloud-init.service could not be found.

When I upgrade with the new platform-specific upgrade ISO /tmp/vyos-1.4.1-vmware-vsphere-amd64.iso
I now have:

root@vyos-test:~# systemctl status cloud-init
● cloud-init.service - Initial cloud-init job (metadata service crawler)
     Loaded: loaded (/lib/systemd/system/cloud-init.service; enabled; preset: enabled)
     Active: active (exited) since Fri 2024-12-27 04:33:39 UTC; 6min ago
   Main PID: 1265 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 2336)
     Memory: 4.7M
        CPU: 2.221s
     CGroup: /system.slice/cloud-init.service

Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: |   1   |  10.0.200.0 |  0.0.0.0   | 255.255.255.0 |    eth0   |   U   |
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: +-------+-------------+---------+-----------+-------+
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: | Route | Destination | Gateway | Interface | Flags |
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: +-------+-------------+---------+-----------+-------+
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: |   1   |  multicast  |    ::   |    eth0   |   U   |
Dec 27 04:33:38 vyos-test cloud-init[1268]: ci-info: +-------+-------------+---------+-----------+-------+
Dec 27 04:33:38 vyos-test cloud-init[1268]: 2024-12-27 04:33:38,559 - DataSourceVMware.py[ERROR]: failed to find a valid data access method
Dec 27 04:33:39 vyos-test systemd[1]: Finished cloud-init.service - Initial cloud-init job (metadata service crawler).
ro

I am not trying to use or configure anything with cloud-init. I am betting that this return of the default hostname and password has something to do with cloud-init suddenly coming into the picture.

Going back to bed, but wanted to leave this here.

9

To let you know: the plaintext-password '' bit is a red herring.

@dmbaturin I believe you are correct.

You will see that the null plain-text password is gone. Seemingly processed into a new encrypted password, overwriting the old one.

You will also see that the hostname has been reset to ‘vyos’. Yes, totally unrelated to this otherwise very important topic, but it happened in my prod system and my test system… so noting it here.

@James2012 neither of these assumptions were correct.

This morning I re-installed vyos-1.4.0-generic-amd64.iso, configured a handful of settings, commited, saved, then upgraded with vyos-1.4.1-vmware-vsphere-amd64.iso

Sure enough, my configuration was intact post upgrade, except for the fact that my hostname had been set back to ‘vyos’ and my vyos user password was now default again.

Now with an eye on cloud-init, there does seem to be a log showing this happening:

/var/log/cloud-init.log

2024-12-27 10:55:25,351 - cc_vyos.py[DEBUG]: Default user: vyos
2024-12-27 10:55:25,352 - cc_vyos.py[DEBUG]: Using configuration file: /opt/vyatta/etc/config/config.boot
2024-12-27 10:55:25,352 - cc_vyos.py[DEBUG]: Running migrations for: /opt/vyatta/etc/config/config.boot
2024-12-27 10:55:27,803 - cc_vyos.py[DEBUG]: Adding fallback user: vyos
2024-12-27 10:55:27,804 - cc_vyos.py[DEBUG]: Configuring plaintext password password for: vyos
2024-12-27 10:55:27,804 - cc_vyos.py[DEBUG]: Configuring network using Cloud-init networking config version 2
2024-12-27 10:55:27,804 - cc_vyos.py[DEBUG]: IP address dhcp have type: dhcp
2024-12-27 10:55:27,804 - cc_vyos.py[DEBUG]: IP address for interface eth0 already configured: ['10.0.200.49/24']
2024-12-27 10:55:27,804 - cc_vyos.py[DEBUG]: IP address 10.0.200.49/24 have type: ipv4
2024-12-27 10:55:27,804 - cc_vyos.py[ERROR]: IP address dhcp cannot be configured, because it conflicts with already exists: ['10.0.200.49/24']
2024-12-27 10:55:27,807 - cc_vyos.py[DEBUG]: Setting MTU for eth0: 1500
2024-12-27 10:55:27,816 - cc_vyos.py[DEBUG]: Setting MAC for eth0: 00:0c:29:3f:86:ac
2024-12-27 10:55:27,816 - cc_vyos.py[DEBUG]: Configuring SSH service
2024-12-27 10:55:27,818 - cc_vyos.py[DEBUG]: Configuring hostname to: vyos
2024-12-27 10:55:27,818 - cc_vyos.py[DEBUG]: Configuration file saved: /opt/vyatta/etc/config/config.boot

Release structure change — update ISOs are now available for every platform

#https://blog.vyos.io/vyos-1.4.1-release
Previously we had a generic ISO image that could be used for installing VyOS on bare metal and for updating various kinds of systems via add system image <ISO URL>, including KVM and VMware VMs, Google Cloud and Oracle cloud instances, and more. That structure always raised a lot of questions from users what image type they should use for a particular system. For Microsoft Azure, we always had a special update image, for Amazon Web Services we added it later, and for everything else we recommended the generic image. However, there is no reason to complicate things — update procedures should not require any reasoning about compatibility.

Now there is an update ISO for every platform, local and cloud alike. The generic ISO no longer contains any VM or cloud guest agents and is meant for use on bare metal. For updating KVM instances, you will find a KVM ISO with just qemu-guest agent built-in and no agents for other hypervisors that do not bring any benefits to a KVM guest — ditto for other platforms. From now on, the general rule is that if we support a platform, there will be a targeted image for it (OVA, qcow2, vhdx...) and an ISO image that you can use for updating already deployed instances.

I have virtualized installations on vSphere. As I understand the above blog post, I should now be using the new platform specific vyos-1.4.1-vmware-vsphere-amd64.iso rather than the generic ISO that I had originally installed from. This default password problem seems to stem from that and the introduction of cloud-init. I suspect that anyone who started out with a generic ISO install and is now upgrading to 1.4.1 with the new vSphere ISO is probably now running default passwords on their machine unless they noticed this.

I have not tested to see if this affects other platform-specific ISO combinations or not.

1 Like
10

@James2012 thank you for reporting.

I can recreate the issue while upgrading from vyos-1.4.0-generic-amd64.iso → vyos-1.4.1-vmware-vsphere-amd64.iso.

This magically changes back my hostname to vyos and add:

 user vyos {
     authentication {
         encrypted-password $6$rounds=656000$H.jyUxzbeTljjx8T$IULP./OhfIXqAEbDDPR35FHiEB0IUAvAFjIx5fM8O/AFKWtqC1zv6UZJdXng8pOvnCKf6oRs1MQy9tj72un471
     }
 }

to the active configuration

11

Ok, to summarize: it’s unsafe to update systems where cloud-init never ran to images that contain cloud-init.

Only these platform-specific flavors contain cloud-init: VMware, Nutanix, OpenStack. The rest only include VM guest agents and similar. The advisory should be that machines on those platforms that were deployed from a generic ISO must be re-deployed or updated to the generic image.

I’m not yet sure what’s the best solution: to produce special update flavors with everything except cloud-init or adjust the cloud-init module to detect whether the config was not the default. Need to think about it.

2 Likes
12

I’m not yet sure what’s the best solution: to produce special update flavors with everything except cloud-init or adjust the cloud-init module to detect whether the config was not the default. Need to think about it.

I would suggest that the latter be implemented. It could be in addition to the former, or independent of it.
I say this because the latter puts a technical control in place to prevent people from unknowingly leaving their box in a wide open state ripe for exploitation. Relying only on a human followed instruction(RTFM) would nearly certainly lead to additional systems being compromised.

1 Like
13

For now I made a flavor with open-vm-tools but without cloud-init and put it alongside the earlier two VMware flavors in the download page. Safeguards to prevent cloud-init from ruining existing configs will be the next step.

2 Likes