Vyos sometimes does not start properly

General questions
1

Good evening,
I have 3 vyos routers
2 are virtualized and one is a hardware router.
Now I have the following problem on all three routers
When I restart the system, there is a 2 to 1 chance that the system will not boot properly.
I get to the point “Mounting vyos config… Done.”
After that nothing happens.
The network adapters are not active and only a restart helps.
Most of the time it works, otherwise I have to repeat the process
I use the 1.5 rolling release on all 3 servers.
The 1.5-rolling-202410300007 is installed on the hardware
on the virtual servers an older version about 3 months old.

Does anyone have a similar problem or a solution?
There is no configuration on the hardware Vyos except a network adapter.

Thanks a lot

2

But you can log in to the box after it boots in that bad state?

3

Yes, ctrl+c takes me to the loggon screen

4

So it hangs on “Mounting vyos config… Done.”

5

Yes, it remains with “Mounting vyos config… Done.” hang

6

Can you share your config from any machine that have that behavior (you can remove passwords or VPN keys if any)

7

Hardware Vyos

vyos@vyos# show
 interfaces {
     ethernet eth0 {
         hw-id 00:1a:8c:17:d6:b8
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth1 {
         hw-id 00:1a:8c:17:d6:b9
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth2 {
         hw-id 00:1a:8c:17:d6:ba
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth3 {
:...skipping...
 interfaces {
     ethernet eth0 {
         hw-id 00:1a:8c:17:d6:b8
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth1 {
         hw-id 00:1a:8c:17:d6:b9
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth2 {
         hw-id 00:1a:8c:17:d6:ba
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth3 {
         hw-id 00:1a:8c:17:d6:bb
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth4 {
         hw-id 00:1a:8c:17:d6:bc
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth5 {
         hw-id 00:1a:8c:17:d6:bd
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth6 {
         hw-id 00:1a:8c:17:d6:be
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth7 {
         address 10.23.32.2/30
         hw-id 00:1a:8c:17:d6:bf
         offload {
             gro
             gso
             sg
             tso
         }
     }
     loopback lo {
     }
 }
 service {
     ntp {
         allow-client {
             address 127.0.0.0/8
             address 169.254.0.0/16
             address 10.0.0.0/8
             address 172.16.0.0/12
             address 192.168.0.0/16
             address ::1/128
             address fe80::/10
             address fc00::/7
         }
         server time1.vyos.net {
         }
         server time2.vyos.net {
         }
         server time3.vyos.net {
         }
     }
     ssh {
         port 22
     }
 }
 system {
     config-management {
         commit-revisions 100
     }
     console {
         device ttyS0 {
             speed 115200
         }
     }
     host-name vyos
     login {
         user vyos {
             authentication {
                 encrypted-password 
                 plaintext-password ""
             }
         }
     }
     syslog {
         global {
             facility all {
                 level info
             }
             facility local7 {
                 level debug
             }
         }
     }
 }

The other machines will follow shortly

8 
 dummy dum0 {
     address 172.16.10.1/32
 }
 ethernet eth0 {
     address xxx.xxx.xxx.xxx/32
     duplex auto
     offload {
         gro
         gso
         sg
         tso
     }
     speed auto
 }
 loopback lo {
 }
 vti vti0 {
     address 172.16.0.0/32
 }
 vti vti1 {
     address 172.16.2.0/32
 }
 vti vti2 {
     address 172.16.0.2/32
 }
 vti vti3 {
     address 172.16.0.4/32
     mtu 1400
 }
 vti vti4 {
     address 172.16.0.7/32
 }


 ospf {
     area 0.0.0.0 {
         area-type {
             normal
         }
         authentication md5
     }
     default-metric 10
     interface dum0 {
         area 0.0.0.0
         bfd {
         }
         passive {
         }
     }
     interface eth0 {
     }
     interface vti0 {
         area 0.0.0.0
         authentication {
             md5 {
                 key-id 1 {
                     md5-key ""
                 }
             }
         }
         bfd {
         }
         cost 30
         dead-interval 5
         hello-interval 2
         mtu-ignore
         network point-to-point
         retransmit-interval 5
     }
     interface vti1 {
         area 0.0.0.0
         authentication {
             md5 {
                 key-id 1 {
                     md5-key ""
                 }
             }
         }
         bfd {
         }
         cost 10
         dead-interval 5
         hello-interval 2
         mtu-ignore
         network point-to-point
         retransmit-interval 5
     }
     interface vti2 {
         area 0.0.0.0
         authentication {
             md5 {
                 key-id 1 {
                     md5-key ""
                 }
             }
         }
         bfd {
         }
         cost 30
         dead-interval 5
         hello-interval 2
         mtu-ignore
         network point-to-point
         retransmit-interval 5
     }
     interface vti3 {
         area 0.0.0.0
         authentication {
             md5 {
                 key-id 1 {
                     md5-key 
                 }
             }
         }
         bfd {
         }
         cost 30
         dead-interval 5
         hello-interval 2
         mtu-ignore
         network point-to-point
         retransmit-interval 5
     }
     parameters {
         router-id 1.0.0.1
     }
     redistribute {
     }
 }
 static {
     route 0.0.0.0/0 {
         next-hop 172.31.1.1 {
         }
     }
     route 172.16.0.1/32 {
         interface vti0 {
             distance 10
         }
     }
     route 172.16.0.3/32 {
         interface vti2 {
         }
     }
     route 172.16.0.5/32 {
         interface vti3 {
         }
     }
     route 172.16.0.8/32 {
         interface vti4 {
         }
     }
     route 172.16.2.1/32 {
         interface vti1 {
         }
     }
     route 172.31.1.1/32 {
         interface eth0 {
         }
     }
 }


 ntp {
     allow-client {
         address 127.0.0.0/8
         address 169.254.0.0/16
         address 10.0.0.0/8
         address 172.16.0.0/12
         address 192.168.0.0/16
         address ::1/128
         address fe80::/10
         address fc00::/7
     }
     server time1.vyos.net {
     }
     server time2.vyos.net {
     }
     server time3.vyos.net {
     }
 }
 snmp {
     listen-address 172.16.10.1 {
     }
     location Datacenter01
     v3 {
         engineid 000000000000000000000002
         group default {
             mode ro
             view default
         }
         user  {
             auth {
                 encrypted-password 
                 type sha
             }
             group default
             privacy {
                 encrypted-password 
                 type aes
             }
         }
         view default {
             oid 1 {
             }
         }
     }
 }
 ssh {
     port 22
 }


show system
 config-management {
     commit-revisions 100
 }
 console {
     device ttyS0 {
         speed 115200
     }
 }
 host-name xxxxx
 login {
     user  {
         authentication {
             encrypted-password          }
     }
     user  {
         authentication {
             encrypted-password 
         }
     }
 }
 name-server 8.8.8.8
 name-server 8.8.4.4
 syslog {
     global {
         facility all {
             level info
         }
         facility local7 {
             level debug
         }
     }
 }



 ipsec {
     authentication {
         psk v5 {
             
             id 
             secret 
         }
         psk v1 {
             id 
             secret          }
         psk v2 {
             id 
             id oijpuhpojpiuhp89hj98jp8076
             secret ZBGgHg7glL5w8O4xy721Ifionm
         }
         psk v3 {
             id gsefsdlQsedvges7OesdgbkjCcI2zyp
             id fpQdNMWeuKvKaSodYyJsz7wiVb&2wsdz
             secret fpQdNMWeuKvKaSodYyJsz7wiVb&2wsdz
         }
         psk v4 {
             id 
             secret          }
     }
     esp-group v1 {
         lifetime 14400
         mode tunnel
         pfs dh-group21
         proposal 1 {
             encryption aes256
             hash sha512
         }
     }
     esp-group v2 {
         lifetime 14400
         mode tunnel
         pfs dh-group21
         proposal 1 {
             encryption aes256
             hash sha512
         }
     }
     esp-group v3 {
         lifetime 14400
         mode tunnel
         pfs dh-group21
         proposal 1 {
             encryption aes256
             hash sha512
         }
     }
     esp-group v4 {
         lifetime 14400
         mode tunnel
         pfs dh-group21
         proposal 1 {
             encryption aes256
             hash sha512
         }
     }
     esp-group v5 {
         lifetime 14400
         mode tunnel
         pfs dh-group21
         proposal 1 {
             encryption aes256
             hash sha512
         }
     }
     ike-group v1 {
         close-action none
         dead-peer-detection {
             action trap
             interval 10
             timeout 6
         }
         key-exchange ikev2
         lifetime 28800
         proposal 1 {
             dh-group 21
             encryption aes256
             hash sha512
         }
     }
     ike-group v2 {
         close-action none
         dead-peer-detection {
             action trap
             interval 10
             timeout 6
         }
         key-exchange ikev2
         lifetime 28800
         proposal 1 {
             dh-group 21
             encryption aes256
             hash sha512
         }
     }
     ike-group v3 {
         close-action none
         dead-peer-detection {
             action trap
             interval 10
             timeout 6
         }
         key-exchange ikev2
         lifetime 28800
         proposal 1 {
             dh-group 21
             encryption aes256
             hash sha512
         }
     }
     ike-group v4 {
         close-action none
         dead-peer-detection {
             action trap
             interval 10
             timeout 6
         }
         key-exchange ikev2
         lifetime 28800
         proposal 1 {
             dh-group 21
             encryption aes256
             hash sha512
         }
     }
     ike-group v5 {
         close-action start
         dead-peer-detection {
             action trap
             interval 10
             timeout 6
         }
         key-exchange ikev2
         lifetime 28800
         proposal 1 {
             dh-group 21
             encryption aes256
             hash sha512
         }
     }
     interface eth0
     options {
         disable-route-autoinstall
     }
     site-to-site {
         peer AS01 {
             authentication {
                 local-id 
                 mode pre-shared-secret
                 remote-id 
             }
             default-esp-group v1
             ike-group v1
             local-address 
             remote-address 
             vti {
                 bind vti2
                 esp-group v1
             }
         }
         peer v2 {
             authentication {
                 local-id 
                 mode pre-shared-secret
                 remote-id              }
             default-esp-group v2
             ike-group v2
             local-address 
             remote-address
             vti {
                 bind vti0
                 esp-group v2
             }
         }
         peer v3 {
             authentication {
                 local-id 
                 mode pre-shared-secret
                 remote-id              }
             default-esp-group v3
             disable
             ike-group v3
             local-address 
             remote-address 
             vti {
                 bind vti4
                 esp-group v3
             }
         }
         peer v4 {
             authentication {
                 local-id 
                 mode pre-shared-secret
                 remote-id 
             }
             default-esp-group v4
             ike-group v4
             local-address 
             remote-address 
             vti {
                 bind vti1
                 esp-group v4
             }
         }
         peer v5 {
             authentication {
                 local-id 
                 mode pre-shared-secret
                 remote-id              }
             default-esp-group v5
             ike-group v5
             local-address 
             remote-address 
             vti {
                 bind vti3
                 esp-group v5
             }
         }
     }
 }

The config of the vms
Both are configured the same with the exception of the IPs

9

Thanks,
we will need to collect logs too
especially from that broken state

10

which logs are needed and how do i get them

11

I’ve been noticing similar issues on some of my builds and I wonder if it’s the same thing. I completely lose any console output after loading config. If I reboot repeatedly, maybe eventually I get in.

Looking at it, it appears what is happening to me is unexpected output is causing exception in vyos-configd, which causes the service to terminate. Then since maybe terminal can’t start because it has precondition of vyos-config. If you look at journalctl from failed boot, do you get similar output to me?

What’s interesting is a lot of the failed output appears to be nft commands, and it happens whether I have a firewall config in place or not, although if there is a firewall config anecdotally I have to reboot many more times to get to login.

Anyways, don’t mean to hijack your issue so if you don’t have same logs as me, feel free to ignore me. I’m going to keep digging, I think maybe I have a build where this doesn’t happen, maybe I will try to diff vyos-config between the 2.

Blockquote
sudo journalctl -b -2 | grep -Ei “vyos-config|getty”
…
Nov 12 16:20:48 test0 vyos-configd[1001]: Traceback (most recent call last):
Nov 12 16:20:48 test0 vyos-configd[1001]: File “/usr/libexec/vyos/services/vyos-configd”, line 326, in
Nov 12 16:20:48 test0 vyos-configd[1001]: message = json.loads(msg)
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^^^^^^^^^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: File “/usr/lib/python3.11/json/init.py”, line 346, in loads
Nov 12 16:20:48 test0 vyos-configd[1001]: return _default_decoder.decode(s)
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: File “/usr/lib/python3.11/json/decoder.py”, line 337, in decode
Nov 12 16:20:48 test0 vyos-configd[1001]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: File “/usr/lib/python3.11/json/decoder.py”, line 355, in raw_decode
Nov 12 16:20:48 test0 vyos-configd[1001]: raise JSONDecodeError(“Expecting value”, s, err.value) from None
Nov 12 16:20:48 test0 vyos-configd[1001]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Nov 12 16:20:48 test0 vyos-configd[1001]: noteworthy:
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘utmpdump /run/utmp’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: [2] [00000] [~~ ] [reboot ] [~ ] [6.6.56-vyos ] [0.0.0.0 ] [2024-11-12T16:20:12,405610+00:00]
Nov 12 16:20:48 test0 vyos-configd[1001]: [1] [00051] [~~ ] [runlevel] [~ ] [6.6.56-vyos ] [0.0.0.0 ] [2024-11-12T16:20:16,555084+00:00]
Nov 12 16:20:48 test0 vyos-configd[1001]: [5] [01320] [tty1] [ ] [tty1 ] [ ] [0.0.0.0 ] [2024-11-12T16:20:16,584667+00:00]
Nov 12 16:20:48 test0 vyos-configd[1001]: [8] [01321] [tyS0] [ ] [ttyS0 ] [ ] [0.0.0.0 ] [1970-01-01T00:00:00,000000+00:00]
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Utmp dump of /run/utmp
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth0” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth0 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth4” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth4 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘ethtool --pause eth3 autoneg on tx on rx on’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: netlink error: Invalid argument
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth3” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth3 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth5” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth5 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘ethtool --pause eth2 autoneg on tx on rx on’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: netlink error: Invalid argument
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth2” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth2 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth7” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth7 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth6” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth6 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 vyos-configd[1001]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “eth1” }’
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (out):
Nov 12 16:20:48 test0 vyos-configd[1001]: returned (err):
Nov 12 16:20:48 test0 vyos-configd[1001]: Error: Could not process rule: No such file or directory
Nov 12 16:20:48 test0 vyos-configd[1001]: delete element inet vrf_zones ct_iface_map { eth1 }
Nov 12 16:20:48 test0 vyos-configd[1001]: ^^^^
Nov 12 16:20:48 test0 python3[1001]: File “/usr/libexec/vyos/services/vyos-configd”, line 326, in
Nov 12 16:20:48 test0 systemd[1]: vyos-configd.service: Main process exited, code=exited, status=1/FAILURE
Nov 12 16:20:48 test0 systemd[1]: vyos-configd.service: Failed with result ‘exit-code’.
Nov 12 16:20:48 test0 systemd[1]: vyos-configd.service: Consumed 13.195s CPU time.
Nov 12 16:20:48 test0 systemd[1]: vyos-configd.service: Scheduled restart job, restart counter is at 1.
Nov 12 16:20:48 test0 systemd[1]: Stopped vyos-configd.service - VyOS configuration daemon.
Nov 12 16:20:48 test0 systemd[1]: vyos-configd.service: Consumed 13.195s CPU time.
Nov 12 16:20:48 test0 systemd[1]: Started vyos-configd.service - VyOS configuration daemon.
Nov 12 16:21:46 test0 systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 12 16:21:46 test0 systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 12 16:21:46 test0 systemd[1]: [email protected]: Scheduled restart job, restart counter is at 1.
Nov 12 16:21:46 test0 systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 12 16:21:46 test0 systemd[1]: Started [email protected] - Getty on tty1.
Nov 12 16:22:02 test0 systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 12 16:22:02 test0 systemd[1]: [email protected]: Failed with result ‘timeout’.

12

I have noticed that if there is no config in VYOS, the system does not start correctly
The more config I add the more often vyos does not start anymore

Enclosed my log

vyos@vyos# sudo journalctl -b -2 | grep -Ei “vyos-config|getty”
Nov 13 10:58:38 vyos systemd[1]: Created slice system-getty.slice - Slice /system/getty.
Nov 13 10:58:38 vyos systemd[1]: Started vyos-configd.service - VyOS configuration daemon.
Nov 13 10:58:40 vyos systemd[1]: getty-static.service - getty on tty2-tty6 if dbus and logind are not available was skipped because of an unmet condition check (ConditionPathExists=!/usr/bin/dbus-daemon).
Nov 13 10:58:42 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 10:58:42 vyos systemd[1]: Reached target getty.target - Login Prompts.
Nov 13 10:58:56 vyos vyos-configd[688]: Received message: {“type”: “init”}
Nov 13 10:58:56 vyos vyos-configd[688]: config session pid is 1499
Nov 13 10:58:56 vyos vyos-configd[688]: config session sudo_user is root
Nov 13 10:58:56 vyos vyos-configd[688]: commit_scripts: [‘system_host-name’, ‘system_console’, ‘nat’, ‘interfaces_loopback_lo’, ‘interfaces_bridge_br1900’, ‘interfaces_bridge_br1304’, ‘interfaces_bridge_br0022’, ‘interfaces_bridge_br1600’, ‘interfaces_bridge_br1901’, ‘interfaces_bridge_br1300’, ‘interfaces_bridge_br1002’, ‘interfaces_bridge_br0419’, ‘interfaces_bridge_br0800’, ‘interfaces_bridge_br0301’, ‘interfaces_bridge_br0900’, ‘interfaces_bridge_br2000’, ‘interfaces_bridge_br0805’, ‘interfaces_bridge_br1612’, ‘interfaces_bridge_br0300’, ‘interfaces_bridge_br0803’, ‘interfaces_bridge_br1000’, ‘interfaces_bridge_br1601’, ‘interfaces_bridge_br0804’, ‘interfaces_bridge_br0007’, ‘interfaces_bridge_br2001’, ‘interfaces_bridge_br0002’, ‘interfaces_bridge_br2510’, ‘interfaces_bridge_br2100’, ‘interfaces_bridge_br3898’, ‘interfaces_bridge_br3200’, ‘interfaces_bridge_br0050’, ‘interfaces_bridge_br1301’, ‘interfaces_bridge_br0008’, ‘interfaces_bridge_br0150’, ‘interfaces_bridge_br2012’, ‘interfaces_bridge_br1603’, ‘interfaces_bridge_br2500’, ‘interfaces_bridge_br1001’, ‘interfaces_bridge_br1332’, ‘interfaces_bridge_br0801’, ‘interfaces_bridge_br0040’, ‘interfaces_bridge_br2222’, ‘interfaces_bridge_br1302’, ‘interfaces_bridge_br1608’, ‘interfaces_bridge_br2301’, ‘interfaces_bridge_br0020’, ‘interfaces_bridge_br0021’, ‘interfaces_bridge_br1305’, ‘interfaces_bridge_br2210’, ‘interfaces_bridge_br1607’, ‘interfaces_bridge_br1606’, ‘interfaces_bridge_br1613’, ‘interfaces_bridge_br1604’, ‘interfaces_bridge_br1605’, ‘interfaces_bridge_br1611’, ‘interfaces_bridge_br3000’, ‘interfaces_bridge_br1602’, ‘interfaces_bridge_br1610’, ‘interfaces_bridge_br2013’, ‘interfaces_bridge_br1303’, ‘interfaces_bridge_br1903’, ‘interfaces_bridge_br0802’, ‘interfaces_bridge_br1614’, ‘interfaces_bridge_br1003’, ‘interfaces_bridge_br1331’, ‘interfaces_bridge_br1330’, ‘interfaces_ethernet_eth2’, ‘interfaces_ethernet_eth6’, ‘interfaces_ethernet_eth1’, ‘interfaces_ethernet_eth3’, ‘interfaces_ethernet_eth4’, ‘interfaces_ethernet_eth7’, ‘interfaces_ethernet_eth5’, ‘interfaces_ethernet_eth0’, ‘interfaces_bonding_bond0’, ‘system_login’, ‘system_config-management’, ‘system_syslog’, ‘service_ntp’, ‘service_ssh’]
Nov 13 10:58:56 vyos vyos-configd[688]: Received message: {“type”: “node”, “last”: false, “data”: “/usr/libexec/vyos/conf_mode/system_host-name.py”}
Nov 13 10:58:56 vyos vyos-configd[688]: Sending reply: 1
Nov 13 10:58:56 vyos vyos-configd[688]: Received request: msg_size
Nov 13 10:58:56 vyos vyos-configd[688]: Sending reply: 0
Nov 13 10:58:56 vyos vyos-configd[688]: Received message: {“type”: “node”, “last”: false, “data”: “/usr/libexec/vyos/conf_mode/system_console.py”}
Nov 13 10:58:57 vyos systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
Nov 13 10:58:57 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 10:58:57 vyos vyos-configd[688]: Sending reply: 1
Nov 13 10:58:57 vyos vyos-configd[688]: Received request: msg_size
Nov 13 10:58:57 vyos vyos-configd[688]: Sending reply: 0
Nov 13 10:58:57 vyos vyos-configd[688]: Received message: {“type”: “node”, “last”: false, “data”: “/usr/libexec/vyos/conf_mode/nat.py”}
Nov 13 10:58:58 vyos vyos-configd[688]: Sending reply: 1
Nov 13 10:58:58 vyos vyos-configd[688]: Received request: msg_size
Nov 13 10:58:58 vyos vyos-configd[688]: Sending reply: 70
Nov 13 10:58:58 vyos vyos-configd[688]: Received request: send
Nov 13 10:58:58 vyos vyos-configd[688]: Sending reply with output
Nov 13 10:58:58 vyos vyos-configd[688]: Received message: {“type”: “node”, “last”: false, “data”: “VYOS_TAGNODE_VALUE=lo/usr/libexec/vyos/conf_mode/interfaces_loopback.py”}
Nov 13 10:58:58 vyos vyos-configd[688]: Sending reply: 1
Nov 13 10:58:58 vyos vyos-configd[688]: Received request: msg_size
Nov 13 10:58:58 vyos vyos-configd[688]: Sending reply: 0
Nov 13 10:58:58 vyos vyos-configd[688]: Received message: send
Nov 13 10:58:58 vyos vyos-configd[688]: VyOS had an issue completing a command.
Nov 13 10:58:58 vyos vyos-configd[688]: We are sorry that you encountered a problem while using VyOS.
Nov 13 10:58:58 vyos vyos-configd[688]: There are a few things you can do to help us (and yourself):
Nov 13 10:58:58 vyos vyos-configd[688]: - Contact us using the online help desk if you have a subscription:
Nov 13 10:58:58 vyos vyos-configd[688]: https://support.vyos.io/
Nov 13 10:58:58 vyos vyos-configd[688]: - Make sure you are running the latest version of VyOS available at:
Nov 13 10:58:58 vyos vyos-configd[688]: VyOS Community
Nov 13 10:58:58 vyos vyos-configd[688]: - Consult the community forum to see how to handle this issue:
Nov 13 10:58:58 vyos vyos-configd[688]: https://forum.vyos.io
Nov 13 10:58:58 vyos vyos-configd[688]: - Join us on Slack where our users exchange help and advice:
Nov 13 10:58:58 vyos vyos-configd[688]: https://vyos.slack.com
Nov 13 10:58:58 vyos vyos-configd[688]: When reporting problems, please include as much information as possible:
Nov 13 10:58:58 vyos vyos-configd[688]: - do not obfuscate any data (feel free to contact us privately if your
Nov 13 10:58:58 vyos vyos-configd[688]: business policy requires it)
Nov 13 10:58:58 vyos vyos-configd[688]: - and include all the information presented below
Nov 13 10:58:58 vyos vyos-configd[688]: Report time: 2024-11-13 10:58:58
Nov 13 10:58:58 vyos vyos-configd[688]: Image version: VyOS 1.5-rolling-202411130006
Nov 13 10:58:58 vyos vyos-configd[688]: Release train: current
Nov 13 10:58:58 vyos vyos-configd[688]: Built by: [email protected]
Nov 13 10:58:58 vyos vyos-configd[688]: Built on: Wed 13 Nov 2024 00:07 UTC
Nov 13 10:58:58 vyos vyos-configd[688]: Build UUID: f299db9c-62ab-470c-b873-392840266546
Nov 13 10:58:58 vyos vyos-configd[688]: Build commit ID: 4d01efaacf195f
Nov 13 10:58:58 vyos vyos-configd[688]: Architecture: x86_64
Nov 13 10:58:58 vyos vyos-configd[688]: Boot via: installed image
Nov 13 10:58:58 vyos vyos-configd[688]: System type: bare metal
Nov 13 10:58:58 vyos vyos-configd[688]: Hardware vendor: Sophos
Nov 13 10:58:58 vyos vyos-configd[688]: Hardware model: UTM
Nov 13 10:58:58 vyos vyos-configd[688]: Hardware S/N: A1905D5963C47E2
Nov 13 10:58:58 vyos vyos-configd[688]: Hardware UUID: 00000000-0000-0000-0807-060504030201
Nov 13 10:58:58 vyos vyos-configd[688]: Traceback (most recent call last):
Nov 13 10:58:58 vyos vyos-configd[688]: File “/usr/libexec/vyos/services/vyos-configd”, line 326, in
Nov 13 10:58:58 vyos vyos-configd[688]: message = json.loads(msg)
Nov 13 10:58:58 vyos vyos-configd[688]: ^^^^^^^^^^^^^^^
Nov 13 10:58:58 vyos vyos-configd[688]: File “/usr/lib/python3.11/json/init.py”, line 346, in loads
Nov 13 10:58:58 vyos vyos-configd[688]: return _default_decoder.decode(s)
Nov 13 10:58:58 vyos vyos-configd[688]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 13 10:58:58 vyos vyos-configd[688]: File “/usr/lib/python3.11/json/decoder.py”, line 337, in decode
Nov 13 10:58:58 vyos vyos-configd[688]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov 13 10:58:58 vyos vyos-configd[688]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 13 10:58:58 vyos vyos-configd[688]: File “/usr/lib/python3.11/json/decoder.py”, line 355, in raw_decode
Nov 13 10:58:58 vyos vyos-configd[688]: raise JSONDecodeError(“Expecting value”, s, err.value) from None
Nov 13 10:58:58 vyos vyos-configd[688]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Nov 13 10:58:58 vyos vyos-configd[688]: noteworthy:
Nov 13 10:58:58 vyos vyos-configd[688]: cmd ‘utmpdump /run/utmp’
Nov 13 10:58:58 vyos vyos-configd[688]: returned (out):
Nov 13 10:58:58 vyos vyos-configd[688]: [2] [00000] [~~ ] [reboot ] [~ ] [6.6.56-vyos ] [0.0.0.0 ] [2024-11-13T10:58:37,156329+00:00]
Nov 13 10:58:58 vyos vyos-configd[688]: [1] [00051] [~~ ] [runlevel] [~ ] [6.6.56-vyos ] [0.0.0.0 ] [2024-11-13T10:58:42,495557+00:00]
Nov 13 10:58:58 vyos vyos-configd[688]: [5] [00961] [tty1] [ ] [tty1 ] [ ] [0.0.0.0 ] [2024-11-13T10:58:42,536384+00:00]
Nov 13 10:58:58 vyos vyos-configd[688]: returned (err):
Nov 13 10:58:58 vyos vyos-configd[688]: Utmp dump of /run/utmp
Nov 13 10:58:58 vyos vyos-configd[688]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “lo” }’
Nov 13 10:58:58 vyos vyos-configd[688]: returned (out):
Nov 13 10:58:58 vyos vyos-configd[688]: returned (err):
Nov 13 10:58:58 vyos vyos-configd[688]: Error: Could not process rule: No such file or directory
Nov 13 10:58:58 vyos vyos-configd[688]: delete element inet vrf_zones ct_iface_map { lo }
Nov 13 10:58:58 vyos vyos-configd[688]: ^^
Nov 13 10:58:58 vyos python3[688]: File “/usr/libexec/vyos/services/vyos-configd”, line 326, in
Nov 13 10:58:58 vyos systemd[1]: vyos-configd.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 10:58:58 vyos systemd[1]: vyos-configd.service: Failed with result ‘exit-code’.
Nov 13 10:58:58 vyos systemd[1]: vyos-configd.service: Consumed 1.755s CPU time.
Nov 13 10:58:58 vyos systemd[1]: vyos-configd.service: Scheduled restart job, restart counter is at 1.
Nov 13 10:58:58 vyos systemd[1]: Stopped vyos-configd.service - VyOS configuration daemon.
Nov 13 10:58:58 vyos systemd[1]: vyos-configd.service: Consumed 1.755s CPU time.
Nov 13 10:58:58 vyos systemd[1]: Started vyos-configd.service - VyOS configuration daemon.
Nov 13 11:00:12 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:00:12 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:00:12 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 1.
Nov 13 11:00:12 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:00:12 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:00:27 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:00:27 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:00:27 vyos systemd[1]: [email protected]: Unit process 1893 (sleep) remains running after unit stopped.
Nov 13 11:00:28 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 1.
Nov 13 11:00:28 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:00:28 vyos systemd[1]: [email protected]: Found left-over process 1893 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:00:28 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:01:42 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:01:42 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:01:42 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 2.
Nov 13 11:01:42 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:01:42 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:01:58 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:01:58 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:01:58 vyos systemd[1]: [email protected]: Unit process 2077 (sleep) remains running after unit stopped.
Nov 13 11:01:58 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 2.
Nov 13 11:01:58 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:01:58 vyos systemd[1]: [email protected]: Found left-over process 2077 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:01:58 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:03:12 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:03:12 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:03:12 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 3.
Nov 13 11:03:12 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:03:12 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:03:28 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:03:28 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:03:28 vyos systemd[1]: [email protected]: Unit process 2261 (sleep) remains running after unit stopped.
Nov 13 11:03:29 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 3.
Nov 13 11:03:29 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:03:29 vyos systemd[1]: [email protected]: Found left-over process 2261 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:03:29 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:04:43 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:04:43 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:04:43 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 4.
Nov 13 11:04:43 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:04:43 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:04:59 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:04:59 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:04:59 vyos systemd[1]: [email protected]: Unit process 2448 (sleep) remains running after unit stopped.
Nov 13 11:04:59 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 4.
Nov 13 11:04:59 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:04:59 vyos systemd[1]: [email protected]: Found left-over process 2448 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:04:59 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:06:13 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:06:13 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:06:13 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 5.
Nov 13 11:06:13 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:06:13 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:06:29 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:06:29 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:06:29 vyos systemd[1]: [email protected]: Unit process 2632 (sleep) remains running after unit stopped.
Nov 13 11:06:30 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 5.
Nov 13 11:06:30 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:06:30 vyos systemd[1]: [email protected]: Found left-over process 2632 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:06:30 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:07:43 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:07:43 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:07:43 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 6.
Nov 13 11:07:43 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:07:43 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:08:00 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:08:00 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:08:00 vyos systemd[1]: [email protected]: Unit process 2816 (sleep) remains running after unit stopped.
Nov 13 11:08:00 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 6.
Nov 13 11:08:00 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:08:00 vyos systemd[1]: [email protected]: Found left-over process 2816 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:08:00 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:09:13 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:09:13 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:09:13 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 7.
Nov 13 11:09:13 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:09:13 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:09:30 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:09:30 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:09:30 vyos systemd[1]: [email protected]: Unit process 3001 (sleep) remains running after unit stopped.
Nov 13 11:09:31 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 7.
Nov 13 11:09:31 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:09:31 vyos systemd[1]: [email protected]: Found left-over process 3001 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:09:31 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:10:43 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:10:43 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:10:43 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 8.
Nov 13 11:10:43 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:10:43 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:11:01 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:11:01 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:11:01 vyos systemd[1]: [email protected]: Unit process 3186 (sleep) remains running after unit stopped.
Nov 13 11:11:01 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 8.
Nov 13 11:11:01 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:11:01 vyos systemd[1]: [email protected]: Found left-over process 3186 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:11:01 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:12:14 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:12:14 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:12:14 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 9.
Nov 13 11:12:14 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:12:14 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:12:31 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:12:31 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:12:31 vyos systemd[1]: [email protected]: Unit process 3370 (sleep) remains running after unit stopped.
Nov 13 11:12:32 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 9.
Nov 13 11:12:32 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:12:32 vyos systemd[1]: [email protected]: Found left-over process 3370 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:12:32 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:13:44 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:13:44 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:13:44 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 10.
Nov 13 11:13:44 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:13:44 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:14:02 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:14:02 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:14:02 vyos systemd[1]: [email protected]: Unit process 3556 (sleep) remains running after unit stopped.
Nov 13 11:14:02 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 10.
Nov 13 11:14:02 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:14:02 vyos systemd[1]: [email protected]: Found left-over process 3556 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:14:02 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:15:14 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:15:14 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:15:14 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 11.
Nov 13 11:15:14 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:15:14 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:15:32 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:15:32 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:15:32 vyos systemd[1]: [email protected]: Unit process 3743 (sleep) remains running after unit stopped.
Nov 13 11:15:33 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 11.
Nov 13 11:15:33 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:15:33 vyos systemd[1]: [email protected]: Found left-over process 3743 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:15:33 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:16:44 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:16:44 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:16:44 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 12.
Nov 13 11:16:44 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:16:44 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:17:03 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:17:03 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:17:03 vyos systemd[1]: [email protected]: Unit process 3930 (sleep) remains running after unit stopped.
Nov 13 11:17:03 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 12.
Nov 13 11:17:03 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:17:03 vyos systemd[1]: [email protected]: Found left-over process 3930 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:17:03 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:18:15 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:18:15 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:18:15 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 13.
Nov 13 11:18:15 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:18:15 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:18:33 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:18:33 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:18:33 vyos systemd[1]: [email protected]: Unit process 4115 (sleep) remains running after unit stopped.
Nov 13 11:18:34 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 13.
Nov 13 11:18:34 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:18:34 vyos systemd[1]: [email protected]: Found left-over process 4115 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:18:34 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:19:45 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:19:45 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:19:45 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 14.
Nov 13 11:19:45 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:19:45 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:20:04 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:20:04 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:20:04 vyos systemd[1]: [email protected]: Unit process 4299 (sleep) remains running after unit stopped.
Nov 13 11:20:04 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 14.
Nov 13 11:20:04 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:20:04 vyos systemd[1]: [email protected]: Found left-over process 4299 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:20:04 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:21:15 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:21:15 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:21:15 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 15.
Nov 13 11:21:15 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:21:15 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:21:34 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.
Nov 13 11:21:34 vyos systemd[1]: [email protected]: Failed with result ‘timeout’.
Nov 13 11:21:34 vyos systemd[1]: [email protected]: Unit process 4484 (sleep) remains running after unit stopped.
Nov 13 11:21:35 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 15.
Nov 13 11:21:35 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:21:35 vyos systemd[1]: [email protected]: Found left-over process 4484 (sleep) in control group while starting unit. Ignoring.
Nov 13 11:21:35 vyos systemd[1]: Started [email protected] - Serial Getty on ttyS0.
Nov 13 11:22:21 vyos systemd[1]: [email protected]: Deactivated successfully.
Nov 13 11:22:21 vyos systemd[1]: [email protected]: Scheduled restart job, restart counter is at 16.
Nov 13 11:22:21 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:22:21 vyos systemd[1]: Started [email protected] - Getty on tty1.
Nov 13 11:22:38 vyos systemd[1]: Stopped target getty.target - Login Prompts.
Nov 13 11:22:38 vyos systemd[1]: Stopping [email protected] - Getty on tty1…
Nov 13 11:22:38 vyos systemd[1]: [email protected]: Deactivated successfully.
Nov 13 11:22:38 vyos systemd[1]: [email protected]: Unit process 4806 (sleep) remains running after unit stopped.
Nov 13 11:22:38 vyos systemd[1]: Stopped [email protected] - Serial Getty on ttyS0.
Nov 13 11:22:38 vyos systemd[1]: [email protected]: Deactivated successfully.
Nov 13 11:22:38 vyos systemd[1]: Stopped [email protected] - Getty on tty1.
Nov 13 11:22:38 vyos systemd[1]: [email protected]: Consumed 1.071s CPU time.
Nov 13 11:22:38 vyos systemd[1]: Removed slice system-getty.slice - Slice /system/getty.
Nov 13 11:22:38 vyos systemd[1]: system-getty.slice: Consumed 2.514s CPU time.
Nov 13 11:22:38 vyos systemd[1]: Removed slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
Nov 13 11:22:38 vyos systemd[1]: system-serial\x2dgetty.slice: Consumed 1.515s CPU time.

13

Ok, it looks like maybe we see similar issues? It appears the script issues nft commands that for whatever reason the device doesn’t think are valid:

Nov 13 10:58:58 vyos vyos-configd[688]: cmd ‘nft --check delete element inet vrf_zones ct_iface_map { “lo” }’
Nov 13 10:58:58 vyos vyos-configd[688]: returned (out):
Nov 13 10:58:58 vyos vyos-configd[688]: returned (err):
Nov 13 10:58:58 vyos vyos-configd[688]: Error: Could not process rule: No such file or directory
Nov 13 10:58:58 vyos vyos-configd[688]: delete element inet vrf_zones ct_iface_map { lo }
Nov 13 10:58:58 vyos vyos-configd[688]: ^^

Then tries to use json.load but gets a formatting error probably I guess because the command fails?

Nov 13 10:58:58 vyos vyos-configd[688]: raise JSONDecodeError(“Expecting value”, s, err.value) from None
Nov 13 10:58:58 vyos vyos-configd[688]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Then because getty service has prestart condition on vyos-config, it fails?

Nov 13 11:00:27 vyos systemd[1]: [email protected]: start-pre operation timed out. Terminating.

cat /etc/systemd/system/[email protected]/aftervyos.conf
[Service]
ExecStartPre=-/usr/libexec/vyos/init/vyos-config
StandardOutput=journal+console

I’m just guessing here though.

14

Ok, I cannot, for the life of me, figure out exactly what triggers it. In the meantime, I wrote a shell script to:
Look for vyos-configd exception in journal
Reboot if found
If it happens again on next boot, mask vyos-configd and reboot again.

Then run it with systemd on every boot.

It seems like once there is successful config load, it is safe to unmask vyos-configd and it will be okay on subsequent boots.

15

Likely related:

A recent change in vyos-configd may be obscuring any underlying issues: an initial fix will be ready shortly, which will help in further investigations.

16

Thanks for that, those symptoms do look remarkably similar. When I trigger the issue it’s always on new install and not an upgrade, but it definitely looks like it could be related.

17

The PR for ⚓ T6899 Return output from configd in multipart message to maintain resilience on error will resolve the immediate issue in vyos-configd, at which point we can re-investigate the original problem.

1 Like
18

Thanks, the update has solved the problem.
Since the installation I can restart Vyos without any problems even with config.
In addition, the problem that the commit command runs into the void is also gone.

2 Likes
19

This is fixed for you in image from today?

I’ll try to respin and see if it’s gone for me, too.

3 Likes
20

1.5-rolling-202411230007
This images has completely fixed my problems

2 Likes