Failed to start Conntrack Daemon

c0ff33h4x · November 6, 2022, 5:09pm

Continuing the discussion from Drive filling up?:

After applying the fix @a.srividya recommended and rebooting the system my conntrackd.service is failing to start. Is there a related bug fix for this?

vyos@Vy1:~$ sudo systemctl status conntrackd.service
● conntrackd.service - Conntrack Daemon
   Loaded: loaded (/lib/systemd/system/conntrackd.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/conntrackd.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Sun 2022-11-06 16:47:38 UTC; 7min ago
     Docs: man:conntrackd(8)
           man:conntrackd.conf(5)
 Main PID: 4585 (code=exited, status=1/FAILURE)

Nov 06 16:47:37 Vy1 systemd[1]: Failed to start Conntrack Daemon.
Nov 06 16:47:38 Vy1 systemd[1]: conntrackd.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 16:47:38 Vy1 systemd[1]: conntrackd.service: Scheduled restart job, restart counter is at 7.
Nov 06 16:47:38 Vy1 systemd[1]: Stopped Conntrack Daemon.
Nov 06 16:47:38 Vy1 systemd[1]: conntrackd.service: Start request repeated too quickly.
Nov 06 16:47:38 Vy1 systemd[1]: conntrackd.service: Failed with result 'exit-code'.
Nov 06 16:47:38 Vy1 systemd[1]: Failed to start Conntrack Daemon.

vyos@Vy1:~$ sudo systemctl restart conntrackd.service
Job for conntrackd.service failed because the control process exited with error code.
See "systemctl status conntrackd.service" and "journalctl -xe" for details.

vyos@Vy1:~$ sudo systemctl status conntrackd.service
● conntrackd.service - Conntrack Daemon
   Loaded: loaded (/lib/systemd/system/conntrackd.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/conntrackd.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Sun 2022-11-06 16:55:08 UTC; 5s ago
     Docs: man:conntrackd(8)
           man:conntrackd.conf(5)
  Process: 5745 ExecStart=/usr/sbin/conntrackd -C /run/conntrackd/conntrackd.conf (code=exited, status=1/FAILURE)
 Main PID: 5745 (code=exited, status=1/FAILURE)

Nov 06 16:55:08 Vy1 systemd[1]: conntrackd.service: Service RestartSec=100ms expired, scheduling restart.
Nov 06 16:55:08 Vy1 systemd[1]: conntrackd.service: Scheduled restart job, restart counter is at 5.
Nov 06 16:55:08 Vy1 systemd[1]: Stopped Conntrack Daemon.
Nov 06 16:55:08 Vy1 systemd[1]: conntrackd.service: Start request repeated too quickly.
Nov 06 16:55:08 Vy1 systemd[1]: conntrackd.service: Failed with result 'exit-code'.
Nov 06 16:55:08 Vy1 systemd[1]: Failed to start Conntrack Daemon.

See additional log output

Nov  6 16:47:27 Vy1 conntrack-tools: vyatta-vrrp-conntracksync invoked at Sun 06 Nov 2022 04:47:27 PM UTC
Nov  6 16:47:27 Vy1 conntrack-tools: Vy1 transitioning to BACKUP state for VRRP sync-group [syncgrp]
Nov  6 16:47:27 Vy1 systemd[1]: Condition check resulted in Conntrack Daemon being skipped.
Nov  6 16:47:27 Vy1 conntrack-tools: ERROR: failed to invoke conntrackd -t
Nov  6 16:47:27 Vy1 conntrack-tools: ERROR: failed to invoke conntrackd -n

Nov  6 16:47:36 Vy1 conntrack-tools: vyatta-vrrp-conntracksync invoked at Sun 06 Nov 2022 04:47:36 PM UTC
Nov  6 16:47:36 Vy1 conntrack-tools: Vy1 transitioning to BACKUP state for VRRP sync-group [syncgrp]
Nov  6 16:47:36 Vy1 systemd[1]: Starting Conntrack Daemon...
Nov  6 16:47:36 Vy1 kernel: [  147.512935] conntrackd[4517]: segfault at 10 ip 00007f9a95469979 sp 00007ffdfd2f4d40 error 4 in libnetfilter_conntrack.so.3.7.0[7f9a95469000+f000]
Nov  6 16:47:36 Vy1 kernel: [  147.512959] Code: 41 5c c3 66 0f 1f 44 00 00 48 89 ef e8 70 fa ff ff 48 89 d8 5b 5d 41 5c c3 0f 1f 84 00 00 00 00 00 55 53 48 89 fb 48 83 ec 08 <48> 8b 7f 10 48 85 ff 74 0d e8 f9 f7 ff ff 48 c7 43 10 00 00 00 00
Nov  6 16:47:36 Vy1 systemd[1]: conntrackd.service: Main process exited, code=killed, status=11/SEGV
Nov  6 16:47:36 Vy1 systemd[1]: conntrackd.service: Failed with result 'signal'.
Nov  6 16:47:36 Vy1 systemd[1]: Stopped Conntrack Daemon.
Nov  6 16:47:36 Vy1 systemd[1]: Starting Conntrack Daemon...
Nov  6 16:47:36 Vy1 conntrackd[4521]: [Sun Nov  6 16:47:36 2022] (pid=4521) [ERROR] lockfile `/var/lock/conntrack.lock' exists, perhaps conntrackd already running?
Nov  6 16:47:36 Vy1 conntrack-tools[4521]: lockfile `/var/lock/conntrack.lock' exists, perhaps conntrackd already running?
Nov  6 16:47:36 Vy1 systemd[1]: conntrackd.service: Main process exited, code=exited, status=1/FAILURE
Nov  6 16:47:36 Vy1 systemd[1]: conntrackd.service: Failed with result 'exit-code'.
Nov  6 16:47:36 Vy1 systemd[1]: Failed to start Conntrack Daemon.
Nov  6 16:47:36 Vy1 conntrack-tools: ERROR: cannot launch conntrackd
Nov  6 16:47:36 Vy1 vyos-configd[552]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/ssh.py"}
Nov  6 16:47:36 Vy1 systemd[1]: conntrackd.service: Service RestartSec=100ms expired, scheduling restart.
Nov  6 16:47:36 Vy1 systemd[1]: conntrackd.service: Scheduled restart job, restart counter is at 1.

Viacheslav · November 6, 2022, 5:40pm

Without configuration it’s difficult to get was is wrong

c0ff33h4x · November 8, 2022, 5:11pm

My apologies, below is the configuration relevant to conntrack. Please let me know if there is additional information you need.

set high-availability vrrp group FW2SWVRRP hello-source-address '172.16.6.66'
set high-availability vrrp group FW2SWVRRP interface 'eth0.2030'
set high-availability vrrp group FW2SWVRRP peer-address '172.16.6.67'
set high-availability vrrp group FW2SWVRRP priority '130'
set high-availability vrrp group FW2SWVRRP rfc3768-compatibility
set high-availability vrrp group FW2SWVRRP virtual-address '172.16.6.65/28'
set high-availability vrrp group FW2SWVRRP vrid '22'
set high-availability vrrp group FWINTVRRP hello-source-address '172.16.6.8'
set high-availability vrrp group FWINTVRRP interface 'eth0.2010'
set high-availability vrrp group FWINTVRRP peer-address '172.16.6.9'
set high-availability vrrp group FWINTVRRP priority '130'
set high-availability vrrp group FWINTVRRP rfc3768-compatibility
set high-availability vrrp group FWINTVRRP virtual-address '172.16.6.7/28'
set high-availability vrrp group FWINTVRRP vrid '20'
set high-availability vrrp sync-group syncgrp member 'FW2SWVRRP'
set high-availability vrrp sync-group syncgrp member 'FWINTVRRP'
set service conntrack-sync accept-protocol 'tcp'
set service conntrack-sync accept-protocol 'udp'
set service conntrack-sync accept-protocol 'icmp'
set service conntrack-sync event-listen-queue-size '8'
set service conntrack-sync failover-mechanism vrrp sync-group 'syncgrp'
set service conntrack-sync interface eth0.2030 peer '172.16.6.67'
set service conntrack-sync sync-queue-size '8'

zsdc · November 9, 2022, 2:01pm

According to logs, the daemon simply crash. Maybe this is a good moment for an update to 1.3.2?

c0ff33h4x · November 9, 2022, 5:46pm

@zsdc I appreciate your input. Not really my preferred route but will have to happen if there’s no other options.
I was hoping that since after 1 very specific change was made before this error starting occurring that there would be a clear path or explanation as to how to address it since everything else is working correctly.

c0ff33h4x · November 12, 2022, 8:43pm

@zsdc @Viacheslav is there a work around to get conntrack going again while we test out 1.3.2? I’d like to have conntrack functional if possible in the interim while we test our current config running on 1.3.2

zsdc · November 14, 2022, 1:19am

Honestly said - no idea. If that would be a bug in config logic, I would try to recommend something, but since this is a simple segfault, the only simple way is an update. Of course, if we do not want to debug conntrackd, which again does not make a big sense if it is OK in the latest stable release.