VyOS 1.2.4 conntrack-sync issue

Bugs
, ,
1

Hi,

We have a HA VRRP pair of VyOS systems using conntrack but after upgrading 1.2.2 → 1.2.4 conntrack seems broken. While running 1.2.2 we’ve had no conntrack issues.

Steps taken:

  1. Added system image 1.2.4 and rebooted the VyOS
  2. After the first boot everything was good:
  • Logging visible of all conntrack (-sync) events
  • Systemd shows running conntrackd service
  1. After rebooting the VyOS (I wanted to add RAM so I shut it down again):
  • No logging regarding conntrack
  • Systemd shows failed / stopped conntrackd service
  • show log conntrack shows conntrackd.service entered failed state

  1. Again reboot the situation stays the same (failed).

  2. Performed the upgrade on the second unit and it shows the exact same behaviour: First boot on 1.2.4 all ok and after another reboot conntrackd failed.

Looking into the full boot log and journalctl is shows during boot and initial startup of the service:
conntrackd[15557]: lockfile `/var/lock/conntrack.lock’ exists, perhaps conntrackd already running?

I tried manually disabling the conntrackd.service and deleting the lock file before reboot. After the reboot this shows the same failed state and log message about the lock file.

Anybody any idea?

Kind regards,
Frank

2

Hmm

Looks like conntrack-sync does sync the connections and works despite the errors.

This might be issue during boot between conntrack-tools and conntrackd.

I have errors:
vyos@kvmvyosf02:~$ show log conntrack-sync
Mar 26 10:34:51 kvmvyosf02 systemd[1]: conntrackd.service: main process exited, code=exited, status=1/FAILURE
Mar 26 10:34:51 kvmvyosf02 systemd[1]: Unit conntrackd.service entered failed state.
Mar 26 10:36:10 kvmvyosf02 systemd[1]: conntrackd.service: main process exited, code=exited, status=1/FAILURE
Mar 26 10:36:10 kvmvyosf02 systemd[1]: Unit conntrackd.service entered failed state.

But the standby unit does seem to be in sync with the active:
vyos@vyos02:~$ show conntrack-sync external-cache
Source Destination Protocol

Main Table Entries:

|x.x.x.x|:58609 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:28110 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:3633 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:53604 |x.x.x.x|:53 udp [17]
|x.x.x.x|:31828 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:51214 |x.x.x.x|:22 tcp [6]
|x.x.x.x|:51215 |x.x.x.x|:22 tcp [6]
|x.x.x.x|:4500 |x.x.x.x|:4500 udp [17]
|x.x.x.x|:54702 |x.x.x.x|:3780 udp [17]
|x.x.x.x|:44690 |x.x.x.x|:3780 udp [17]
|x.x.x.x|:41840 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:51043 |x.x.x.x|:443 tcp [6]

vyos@vyos01:~$ show conntrack-sync internal-cache
Source Destination Protocol

Main Table Entries:

|x.x.x.x|:42859 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:35534 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:28733 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:51214 |x.x.x.x|:22 tcp [6]
|x.x.x.x|:51215 |x.x.x.x|:22 tcp [6]
|x.x.x.x|:4500 |x.x.x.x|:4500 udp [17]
|x.x.x.x|:9327 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:54702 |x.x.x.x|:3780 udp [17]
|x.x.x.x|:44690 |x.x.x.x|:3780 udp [17]
|x.x.x.x|:51043 |x.x.x.x|:443 tcp [6]
|x.x.x.x|:55886 |x.x.x.x|:443 tcp [6]

3

Boot log:

firewall-cfg[1783]: disable_fw_conntrack(iptables)
firewall-cfg[1838]: enable_fw_conntrack(iptables)
conntrack-tools[2438]: vyatta-vrrp-conntracksync invoked at Thu Mar 26 10:34:49 CET 2020
conntrack-tools[2443]: vyos02 transitioning to BACKUP state for VRRP sync-group [SYNC-GROUP]
conntrack-tools[2447]: reliable ctnetlink event delivery is ENABLED.
conntrack-tools[2447]: using user-space event filtering
conntrack-tools[2447]: netlink event socket buffer size has been set to 2097152 bytes
conntrack-tools[2447]: configuring helper tns' with queuenum=5 and queuelen=0 conntrack-tools[2447]: policy name=tns expect_timeout=300 expect_max=1 conntrack-tools[2447]: helper tns’ configured successfully
conntrack-tools[2447]: configuring helper rpc' with queuenum=4 and queuelen=0 conntrack-tools[2447]: policy name=rpc expect_timeout=300 expect_max=1 conntrack-tools[2447]: helper rpc’ configured successfully
conntrack-tools[2447]: configuring helper rpc' with queuenum=3 and queuelen=0 conntrack-tools[2447]: policy name=rpc expect_timeout=300 expect_max=1 conntrack-tools[2447]: helper rpc’ configured successfully
conntrack-tools[2447]: initialization completed
conntrack-tools[2453]: – starting in daemon mode –
conntrack-tools[2453]: flushing conntrack table in 60 secs
conntrack-tools[2453]: request resync
charon[2563]: 00[LIB] loaded plugins: charon test-vectors ldap pkcs11 tpm aes rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation
cs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac ctr ccm gcm curl
socket-default connmark stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc
th-pam tnc-tnccs dhcp lookip error-notify certexpire led addrblock counters
sudo[2750]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/vyatta/sbin/vyatta-conntrack-sync.pl --action=enable
conntrackd[2770]: lockfile `/var/lock/conntrack.lock’ exists, perhaps conntrackd already running?
systemd[1]: conntrackd.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit conntrackd.service entered failed state.

4

Hello @frank-nl, I saw you close your ticket in help desk. I need additional information.
Exactly output of commands:

sudo systemctl status conntrackd.service
sudo journalctl > /tmp/log
show configuration commands | strip-private

And provide please /tmp/log. I think will be better continue research this issue in our help desk.

5

Thanks will do it in the support ticket,

8

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.