@n.fort many thanks for looking into this.
I have upgraded to the version you mentioned, VyOS 1.4-rolling-202109050613
, and can verify that transition scripts are running.
For example,these are the logs of my secondary instance between booting the new version and rebooting the primary one:
Sep 07 18:45:46 vyos keepalived-fifo.py[3337]: GROUP sync changed state to BACKUP
Sep 07 18:45:46 vyos keepalived-fifo.py[3337]: Running the command: /config/scripts/disable-pppoe0.script
Sep 07 18:45:46 vyos keepalived-fifo.py[3337]: Received message: INSTANCE "int" BACKUP 100
Sep 07 18:45:46 vyos keepalived-fifo.py[3337]: INSTANCE int changed state to BACKUP
Sep 07 18:45:46 vyos keepalived-fifo.py[3337]: Received message: INSTANCE "int6" BACKUP 100
Sep 07 18:45:46 vyos keepalived-fifo.py[3337]: INSTANCE int6 changed state to BACKUP
Sep 07 18:46:01 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:46:16 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:46:31 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:46:46 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:47:01 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:47:16 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:47:31 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:47:46 vyos Keepalived_vrrp[3334]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 18:47:56 vyos Keepalived_vrrp[3334]: (int) Backup received priority 0 advertisement
Sep 07 18:47:56 vyos Keepalived_vrrp[3334]: (int6) Backup received priority 0 advertisement
Sep 07 18:47:57 vyos Keepalived_vrrp[3334]: (int6) Entering MASTER STATE
Sep 07 18:47:57 vyos Keepalived_vrrp[3334]: (int6) using locally configured advertisement interval (1000 milli-sec)
Sep 07 18:47:57 vyos Keepalived_vrrp[3334]: VRRP_Group(sync) Syncing instances to MASTER state
Sep 07 18:47:57 vyos Keepalived_vrrp[3334]: (int) Entering MASTER STATE
Sep 07 18:47:57 vyos conntrack-tools[4233]: vyatta-vrrp-conntracksync invoked at Tue 07 Sep 2021 06:47:57 PM UTC
Sep 07 18:47:57 vyos conntrack-tools[4236]: vyos transitioning to MASTER state for VRRP sync-group [sync]
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: Received message: INSTANCE "int6" MASTER 100
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: INSTANCE int6 changed state to MASTER
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: Received message: INSTANCE "int" MASTER 100
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: INSTANCE int changed state to MASTER
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: Received message: GROUP "sync" MASTER 0
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: GROUP sync changed state to MASTER
Sep 07 18:47:57 vyos keepalived-fifo.py[3337]: Running the command: /config/scripts/enable-pppoe0.script
Sep 07 18:47:57 vyos conntrack-tools[3355]: resync with master conntrack table
However, after rebooting the primary instance keepalived seems not to be running anymore. I manually delete the high-availability
configuration, then re-apply it and finally things are working:
-- Journal begins at Tue 2021-09-07 18:48:20 UTC, ends at Tue 2021-09-07 20:34:11 UTC. --
Sep 07 20:29:48 vyos systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Sep 07 20:29:48 vyos Keepalived[3192]: Starting Keepalived v2.1.5 (07/13,2020)
Sep 07 20:29:48 vyos Keepalived[3192]: Running on Linux 5.10.62-amd64-vyos #1 SMP Sat Sep 4 05:22:09 UTC 2021 (built for Linux 5.8.14)
Sep 07 20:29:48 vyos Keepalived[3192]: Command line: '/usr/sbin/keepalived' '--dont-fork' '--snmp'
Sep 07 20:29:48 vyos Keepalived[3192]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 07 20:29:48 vyos Keepalived[3192]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Sep 07 20:29:48 vyos Keepalived[3192]: Starting VRRP child process, pid=3214
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Registering Kernel netlink reflector
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Registering Kernel netlink command channel
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Starting SNMP subagent
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Warning: Failed to connect to the agentx master agent ([NIL]):
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: (int): entering FAULT state (interface eth1 down)
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: (int6): entering FAULT state (interface eth1 down)
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: VRRP_Group(sync): Syncing int to FAULT state
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: (int) entering FAULT state
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: (int6) entering FAULT state
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Registering gratuitous ARP shared channel
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: Registering gratuitous NDISC shared channel
Sep 07 20:29:48 vyos Keepalived_vrrp[3214]: bind unicast_src fd86::2 failed 99 - Cannot assign requested address
Sep 07 20:29:48 vyos conntrack-tools[3229]: vyatta-vrrp-conntracksync invoked at Tue 07 Sep 2021 08:29:48 PM UTC
Sep 07 20:29:48 vyos conntrack-tools[3234]: vyos transitioning to FAULT state for VRRP sync-group [sync]
Sep 07 20:29:48 vyos conntrack-tools[3236]: ERROR: failed to invoke conntrackd -t
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Starting FIFO pipe for Keepalived
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Loaded configuration: {'vrrp_groups': {'int': {'STOP': None, 'FAULT': None, 'BACKUP': None, 'MASTER': None}, 'int6': {'STOP': None, 'FAULT': None, 'BACKUP': None, 'M>
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: PIPE already exist: /run/keepalived_notify_fifo
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Message reading start
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Message processing start
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Received message: INSTANCE "int" FAULT 200
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: INSTANCE int changed state to FAULT
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Received message: INSTANCE "int6" FAULT 200
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: INSTANCE int6 changed state to FAULT
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: Received message: GROUP "sync" FAULT 0
Sep 07 20:29:48 vyos keepalived-fifo.py[3226]: GROUP sync changed state to FAULT
Sep 07 20:29:49 vyos Keepalived_vrrp[3214]: Netlink reports eth1 up
Sep 07 20:29:50 vyos systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Sep 07 20:29:50 vyos Keepalived[3192]: Stopping
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: Ending processing: Received SIGTERM signal
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: Closing FIFO pipe
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: Received message: INSTANCE "int" STOP 200
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: INSTANCE int changed state to STOP
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: Received message: INSTANCE "int6" STOP 200
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: INSTANCE int6 changed state to STOP
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: Received message: GROUP "sync" STOP 0
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: GROUP sync changed state to STOP
Sep 07 20:29:50 vyos keepalived-fifo.py[3226]: Terminating messages processing thread
Sep 07 20:29:51 vyos Keepalived_vrrp[3214]: Stopped
Sep 07 20:29:51 vyos Keepalived[3192]: Stopped Keepalived v2.1.5 (07/13,2020)
Sep 07 20:29:51 vyos systemd[1]: keepalived.service: Succeeded.
Sep 07 20:29:51 vyos systemd[1]: Stopped Keepalive Daemon (LVS and VRRP).
Sep 07 20:33:18 vyos systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Sep 07 20:33:18 vyos Keepalived[5743]: Starting Keepalived v2.1.5 (07/13,2020)
Sep 07 20:33:18 vyos Keepalived[5743]: Running on Linux 5.10.62-amd64-vyos #1 SMP Sat Sep 4 05:22:09 UTC 2021 (built for Linux 5.8.14)
Sep 07 20:33:18 vyos Keepalived[5743]: Command line: '/usr/sbin/keepalived' '--dont-fork' '--snmp'
Sep 07 20:33:18 vyos Keepalived[5743]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 07 20:33:18 vyos Keepalived[5743]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Sep 07 20:33:18 vyos Keepalived[5743]: Starting VRRP child process, pid=5750
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: Registering Kernel netlink reflector
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: Registering Kernel netlink command channel
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: Starting SNMP subagent
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: NET-SNMP version 5.9 AgentX subagent connected
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: Registering gratuitous ARP shared channel
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: Registering gratuitous NDISC shared channel
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: (int) Entering BACKUP STATE (init)
Sep 07 20:33:18 vyos Keepalived_vrrp[5750]: (int6) Entering BACKUP STATE (init)
Sep 07 20:33:18 vyos conntrack-tools[5759]: vyatta-vrrp-conntracksync invoked at Tue 07 Sep 2021 08:33:18 PM UTC
Sep 07 20:33:18 vyos conntrack-tools[5762]: vyos transitioning to BACKUP state for VRRP sync-group [sync]
Sep 07 20:33:18 vyos conntrack-tools[5764]: WARNING: conntrackd was not cleanly stopped.
Sep 07 20:33:18 vyos conntrack-tools[5765]: If you suspect that it has crashed:
Sep 07 20:33:18 vyos conntrack-tools[5766]: 1) Enable coredumps
Sep 07 20:33:18 vyos conntrack-tools[5767]: 2) Try to reproduce the problem
Sep 07 20:33:18 vyos conntrack-tools[5768]: 3) Post the coredump to [email protected]
Sep 07 20:33:18 vyos conntrack-tools[5770]: netlink event socket buffer size has been set to 262142 bytes
Sep 07 20:33:18 vyos conntrack-tools[5770]: initialization completed
Sep 07 20:33:18 vyos conntrack-tools[5772]: -- starting in daemon mode --
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: Starting FIFO pipe for Keepalived
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: Loaded configuration: {'vrrp_groups': {'int': {'STOP': None, 'FAULT': None, 'BACKUP': None, 'MASTER': None}, 'int6': {'STOP': None, 'FAULT': None, 'BACKUP': None, 'M
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: Message reading start
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: Message processing start
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: Received message: GROUP "sync" BACKUP 0
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: GROUP sync changed state to BACKUP
Sep 07 20:33:19 vyos keepalived-fifo.py[5756]: Running the command: /config/scripts/disable-pppoe0.script
Sep 07 20:33:19 vyos sudo[5906]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='eth0' /usr/libexec/vyos/conf_mode/interfaces-ethernet.py
Sep 07 20:33:19 vyos sudo[5906]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:21 vyos sudo[5906]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:21 vyos sudo[6181]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='pppoe0' /usr/libexec/vyos/conf_mode/interfaces-pppoe.py
Sep 07 20:33:21 vyos sudo[6181]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:21 vyos sudo[6181]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:21 vyos sudo[6226]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/mv /tmp/config.boot.6220 /opt/vyatta/etc/config/archive/config.boot
Sep 07 20:33:21 vyos sudo[6226]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:21 vyos sudo[6226]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:21 vyos sudo[6228]: root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/logrotate -f -s /opt/vyatta/etc/config/archive/lr.state /opt/vyatta/etc/config/archive/lr.conf
Sep 07 20:33:21 vyos sudo[6228]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:21 vyos sudo[6228]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:21 vyos commit[6231]: Successful change to active configuration by user root on unknown
Sep 07 20:33:21 vyos keepalived-fifo.py[5756]: Received message: INSTANCE "int" BACKUP 200
Sep 07 20:33:21 vyos keepalived-fifo.py[5756]: INSTANCE int changed state to BACKUP
Sep 07 20:33:21 vyos keepalived-fifo.py[5756]: Received message: INSTANCE "int6" BACKUP 200
Sep 07 20:33:21 vyos keepalived-fifo.py[5756]: INSTANCE int6 changed state to BACKUP
Sep 07 20:33:22 vyos Keepalived_vrrp[5750]: (int6) Entering MASTER STATE
Sep 07 20:33:22 vyos Keepalived_vrrp[5750]: (int6) using locally configured advertisement interval (1000 milli-sec)
Sep 07 20:33:22 vyos Keepalived_vrrp[5750]: VRRP_Group(sync) Syncing instances to MASTER state
Sep 07 20:33:22 vyos Keepalived_vrrp[5750]: (int) Entering MASTER STATE
Sep 07 20:33:22 vyos conntrack-tools[6238]: vyatta-vrrp-conntracksync invoked at Tue 07 Sep 2021 08:33:22 PM UTC
Sep 07 20:33:22 vyos conntrack-tools[6241]: vyos transitioning to MASTER state for VRRP sync-group [sync]
Sep 07 20:33:22 vyos conntrack-tools[5772]: resync with master conntrack table
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: Received message: INSTANCE "int6" MASTER 200
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: INSTANCE int6 changed state to MASTER
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: Received message: INSTANCE "int" MASTER 200
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: INSTANCE int changed state to MASTER
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: Received message: GROUP "sync" MASTER 0
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: GROUP sync changed state to MASTER
Sep 07 20:33:22 vyos keepalived-fifo.py[5756]: Running the command: /config/scripts/enable-pppoe0.script
Sep 07 20:33:22 vyos sudo[6340]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='eth0' /usr/libexec/vyos/conf_mode/interfaces-ethernet.py
Sep 07 20:33:22 vyos sudo[6340]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:23 vyos sudo[6340]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:23 vyos sudo[6453]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='pppoe0' /usr/libexec/vyos/conf_mode/interfaces-pppoe.py
Sep 07 20:33:23 vyos sudo[6453]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:23 vyos sudo[6453]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:23 vyos sudo[6608]: root : PWD=/ ; USER=root ; COMMAND=/usr/bin/mv /tmp/config.boot.6520 /opt/vyatta/etc/config/archive/config.boot
Sep 07 20:33:23 vyos sudo[6608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:23 vyos sudo[6608]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:23 vyos sudo[6610]: root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/logrotate -f -s /opt/vyatta/etc/config/archive/lr.state /opt/vyatta/etc/config/archive/lr.conf
Sep 07 20:33:23 vyos sudo[6610]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 07 20:33:23 vyos sudo[6610]: pam_unix(sudo:session): session closed for user root
Sep 07 20:33:23 vyos commit[6615]: Successful change to active configuration by user root on unknown
Sep 07 20:33:25 vyos Keepalived_vrrp[5750]: Interface name has changed from ppp0 to pppoe0
Any idea what went wrong there? I have to say that in the meantime I have also added some ipv6 in vrrp:
# show high-availability
vrrp {
group int {
hello-source-address 192.168.1.2
interface eth1
peer-address 192.168.1.3
preempt-delay 180
priority 200
virtual-address 192.168.1.1/24
vrid 201
}
group int6 {
hello-source-address fd86::2
interface eth1
peer-address fd86::3
preempt-delay 180
priority 200
virtual-address fd86::1/64
vrid 6
}
sync-group sync {
member int
member int6
transition-script {
backup /config/scripts/disable-pppoe0.script
master /config/scripts/enable-pppoe0.script
}
}
}
So, this is a bit messier, but still I think this is caused by something else, e.g.,
Sep 07 20:29:48 vyos conntrack-tools[3236]: ERROR: failed to invoke conntrackd -t
Any ideas?