Changes to IPSEC configuration take forever to commit, errors in the log

Bugs
, ,
1

I’m testing Vyos Stream 2025.11 and getting weird behavior on one of the nodes. The setup is policy based IPSEC/GRE and it fails to pass traffic. One noticeable issue is that committing configuration takes long time and there are these errors in the log:

Dec 03 04:42:42 systemd[1]: Reloading strongSwan IPsec IKEv1/IKEv2 daemon using swanctl...
Dec 03 04:42:42 charon[3244]: 11[CFG] loaded 0 entries for attr plugin configuration
Dec 03 04:42:42 charon[3244]: 11[CFG] loaded 0 RADIUS server configurations
Dec 03 04:42:42 charon[3244]: 12[CFG] loaded IKE shared key with id 'ike-PSK-KEY-R1' for: 'R1', 'R5'
Dec 03 04:42:42 charon[3244]: 09[CFG] loaded IKE shared key with id 'ike-PSK-KEY-R23' for: 'R5', 'R23'
Dec 03 04:42:42 charon[3244]: 11[CFG] loaded IKE shared key with id 'ike-PSK-KEY-R24' for: 'R24', 'R5'
Dec 03 04:42:42 charon[3244]: 12[CFG] replaced vici connection: PEER-R1
Dec 03 04:42:42 charon[3244]: 12[CFG] uninstalling 'PEER-R1-tunnel-0'
Dec 03 04:42:42 charon[3244]: 12[DMN] thread 12 received 11
Dec 03 04:42:42 charon[3244]: 12[LIB]  dumping 20 stack frame addresses:
Dec 03 04:42:42 charon[3244]: 12[LIB]   /lib/x86_64-linux-gnu/libc.so.6 @ 0x7f7eef541000 [0x7f7eef57d050]
Dec 03 04:42:42 charon-systemd[7544]: sh: line 1: addr2line: command not found
Dec 03 04:42:42 charon[3244]: 12[LIB]     ->
Dec 03 04:42:42 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfee9b9]
Dec 03 04:42:42 charon-systemd[7545]: sh: line 1: addr2line: command not found
Dec 03 04:42:42 charon[3244]: 12[LIB]     ->
Dec 03 04:42:42 charon[3244]: 12[LIB]   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef80a099]
Dec 03 04:42:42 charon-systemd[7546]: sh: line 1: addr2line: command not found
Dec 03 04:42:42 charon[3244]: 12[LIB]     ->
Dec 03 04:42:42 charon[3244]: 12[LIB]   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef82881c]
Dec 03 04:42:42 charon-systemd[7547]: sh: line 1: addr2line: command not found
Dec 03 04:42:42 charon[3244]: 12[LIB]     ->
Dec 03 04:42:42 charon[3244]: 12[LIB]   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef82b338]
Dec 03 04:42:43 charon-systemd[7548]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef8394e7]
Dec 03 04:42:43 charon-systemd[7549]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef839af8]
Dec 03 04:42:43 charon-systemd[7550]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff4e11]
Dec 03 04:42:43 charon-systemd[7551]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff51e9]
Dec 03 04:42:43 charon-systemd[7552]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff5d90]
Dec 03 04:42:43 charon-systemd[7553]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfe8b2b]
Dec 03 04:42:43 charon-systemd[7554]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff1d2b]
Dec 03 04:42:43 charon-systemd[7555]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 (process_request+0xc5) [0x7f7eedfea275]
Dec 03 04:42:43 charon-systemd[7556]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfea605]
Dec 03 04:42:43 charon-systemd[7557]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfe6df9]
Dec 03 04:42:43 charon-systemd[7558]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/libstrongswan.so.0 @ 0x7f7eef895000 [0x7f7eef8d2cf2]
Dec 03 04:42:43 charon-systemd[7559]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/libstrongswan.so.0 @ 0x7f7eef895000 [0x7f7eef8d3656]
Dec 03 04:42:43 charon-systemd[7560]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /usr/lib/ipsec/libstrongswan.so.0 @ 0x7f7eef895000 [0x7f7eef8e746e]
Dec 03 04:42:43 charon-systemd[7561]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /lib/x86_64-linux-gnu/libc.so.6 @ 0x7f7eef541000 [0x7f7eef5ca1f5]
Dec 03 04:42:43 charon-systemd[7562]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon[3244]: 12[LIB]   /lib/x86_64-linux-gnu/libc.so.6 @ 0x7f7eef541000 [0x7f7eef64a8dc]
Dec 03 04:42:43 charon-systemd[7563]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:  dumping 20 stack frame addresses:
Dec 03 04:42:43 charon-systemd[3244]:   /lib/x86_64-linux-gnu/libc.so.6 @ 0x7f7eef541000 [0x7f7eef57d050]
Dec 03 04:42:43 charon[3244]: 12[LIB]     ->
Dec 03 04:42:43 charon-systemd[7564]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfee9b9]
Dec 03 04:42:43 charon-systemd[7565]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef80a099]
Dec 03 04:42:43 charon-systemd[7566]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef82881c]
Dec 03 04:42:43 charon-systemd[7567]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef82b338]
Dec 03 04:42:43 charon-systemd[7568]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef8394e7]
Dec 03 04:42:43 charon-systemd[7569]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libcharon.so.0 @ 0x7f7eef7f9000 [0x7f7eef839af8]
Dec 03 04:42:43 charon-systemd[7570]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff4e11]
Dec 03 04:42:43 charon-systemd[7571]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff51e9]
Dec 03 04:42:43 charon-systemd[7572]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff5d90]
Dec 03 04:42:43 charon-systemd[7573]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfe8b2b]
Dec 03 04:42:43 charon-systemd[7574]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedff1d2b]
Dec 03 04:42:43 charon-systemd[7575]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 (process_request+0xc5) [0x7f7eedfea275]
Dec 03 04:42:43 charon-systemd[7576]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfea605]
Dec 03 04:42:43 charon-systemd[7577]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/plugins/libstrongswan-vici.so @ 0x7f7eedfe2000 [0x7f7eedfe6df9]
Dec 03 04:42:43 charon-systemd[7578]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libstrongswan.so.0 @ 0x7f7eef895000 [0x7f7eef8d2cf2]
Dec 03 04:42:43 charon-systemd[7579]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libstrongswan.so.0 @ 0x7f7eef895000 [0x7f7eef8d3656]
Dec 03 04:42:43 charon-systemd[7580]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /usr/lib/ipsec/libstrongswan.so.0 @ 0x7f7eef895000 [0x7f7eef8e746e]
Dec 03 04:42:43 charon-systemd[7581]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /lib/x86_64-linux-gnu/libc.so.6 @ 0x7f7eef541000 [0x7f7eef5ca1f5]
Dec 03 04:42:43 charon-systemd[7582]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon-systemd[3244]:   /lib/x86_64-linux-gnu/libc.so.6 @ 0x7f7eef541000 [0x7f7eef64a8dc]
Dec 03 04:42:43 charon-systemd[7583]: sh: line 1: addr2line: command not found
Dec 03 04:42:43 charon-systemd[3244]:     ->
Dec 03 04:42:43 charon[3244]: 12[DMN] killing ourself, received critical signal
Dec 03 04:42:43 systemd[1]: strongswan.service: Main process exited, code=killed, status=6/ABRT
Dec 03 04:44:13 systemd[1]: strongswan.service: Reload operation timed out. Killing reload process.
Dec 03 04:44:13 systemd[1]: strongswan.service: Failed with result 'signal'.
Dec 03 04:44:13 systemd[1]: Reload failed for strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.
Dec 03 04:44:13 systemd[1]: strongswan.service: Consumed 1min 12.029s CPU time.
Dec 03 04:44:13 systemd[1]: strongswan.service: Scheduled restart job, restart counter is at 1.
Dec 03 04:44:13 systemd[1]: Stopped strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.
Dec 03 04:44:13 systemd[1]: strongswan.service: Consumed 1min 12.029s CPU time.
Dec 03 04:44:13 systemd[1]: Starting strongSwan IPsec IKEv1/IKEv2 daemon using swanctl...
Dec 03 04:44:13 charon[7586]: 00[DMN] Starting charon-systemd IKE daemon (strongSwan 5.9.11, Linux 6.6.114-vyos, x86_64)
Dec 03 04:44:13 charon[7586]: 00[CFG] PKCS11 module '<name>' lacks library path
Dec 03 04:44:13 charon[7586]: 00[PTS] TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Dec 03 04:44:13 charon[7586]: 00[LIB] plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Dec 03 04:44:13 charon[7586]: 00[LIB] providers loaded by OpenSSL: legacy default
Dec 03 04:44:13 charon[7586]: 00[CFG] install DNS servers in '/etc/resolv.conf'
Dec 03 04:44:13 charon[7586]: 00[KNL] unable to create IPv4 routing table rule
Dec 03 04:44:13 charon[7586]: 00[KNL] unable to create IPv6 routing table rule
Dec 03 04:44:13 charon[7586]: 00[NET] using forecast interface eth0
Dec 03 04:44:13 charon[7586]: 00[CFG] joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Dec 03 04:44:13 charon[7586]: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Dec 03 04:44:13 charon[7586]: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Dec 03 04:44:13 charon[7586]: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Dec 03 04:44:13 charon[7586]: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Dec 03 04:44:13 charon[7586]: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Dec 03 04:44:13 charon[7586]: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Dec 03 04:44:13 charon[7586]: 00[CFG] opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Dec 03 04:44:13 charon[7586]: 00[CFG] loaded 0 RADIUS server configurations
Dec 03 04:44:13 charon[7586]: 00[CFG] HA config misses local/remote address
Dec 03 04:44:13 charon[7586]: 00[LIB] loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Dec 03 04:44:13 charon[7586]: 00[LIB] dropped capabilities, running as uid 0, gid 0
Dec 03 04:44:13 charon[7586]: 00[JOB] spawning 16 worker threads
Dec 03 04:44:13 charon[7586]: 07[CFG] loaded IKE shared key with id 'ike-PSK-KEY-R1' for: 'R1', 'R5'
Dec 03 04:44:13 charon[7586]: 11[CFG] loaded IKE shared key with id 'ike-PSK-KEY-R23' for: 'R5', 'R23'
Dec 03 04:44:13 charon[7586]: 01[CFG] loaded IKE shared key with id 'ike-PSK-KEY-R24' for: 'R24', 'R5'
Dec 03 04:44:13 charon[7586]: 09[CFG] added vici connection: PEER-R1
Dec 03 04:44:13 charon[7586]: 09[CFG] initiating 'PEER-R1-tunnel-0'
Dec 03 04:44:13 charon[7586]: 09[IKE] <PEER-R1|1> initiating IKE_SA PEER-R1[1] to 184.75.211.181
Dec 03 04:44:13 charon-systemd[7586]: initiating IKE_SA PEER-R1[1] to 184.75.211.181
Dec 03 04:44:13 charon[7586]: 09[ENC] <PEER-R1|1> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Dec 03 04:44:13 charon[7586]: 09[NET] <PEER-R1|1> sending packet: from 104.167.216.115[500] to 184.75.211.181[500] (232 bytes)
Dec 03 04:44:13 charon[7586]: 11[CFG] added vici connection: PEER-R23
Dec 03 04:44:13 charon[7586]: 11[CFG] installing 'PEER-R23-tunnel-0'
Dec 03 04:44:13 charon[7586]: 11[KNL] policy already exists, try to update it
Dec 03 04:44:13 charon[7586]: 11[KNL] policy already exists, try to update it
Dec 03 04:44:13 charon[7586]: 11[KNL] policy already exists, try to update it
Dec 03 04:44:13 charon[7586]: 15[CFG] added vici connection: PEER-R24
Dec 03 04:44:13 charon[7586]: 15[CFG] installing 'PEER-R24-tunnel-0'
Dec 03 04:44:13 charon[7586]: 15[CFG] installing trap failed, remote address unknown
Dec 03 04:44:13 swanctl[7607]: loaded ike secret 'ike-PSK-KEY-R1'
Dec 03 04:44:13 swanctl[7607]: loaded ike secret 'ike-PSK-KEY-R23'
Dec 03 04:44:13 swanctl[7607]: loaded ike secret 'ike-PSK-KEY-R24'
Dec 03 04:44:13 swanctl[7607]: no authorities found, 0 unloaded
Dec 03 04:44:13 swanctl[7607]: no pools found, 0 unloaded
Dec 03 04:44:13 swanctl[7607]: loaded connection 'PEER-R1'
Dec 03 04:44:13 swanctl[7607]: loaded connection 'PEER-R23'
Dec 03 04:44:13 swanctl[7607]: loaded connection 'PEER-R24'
Dec 03 04:44:13 swanctl[7607]: successfully loaded 3 connections, 0 unloaded
Dec 03 04:44:13 charon[7586]: 10[NET] <PEER-R1|1> received packet: from 184.75.211.181[500] to 104.167.216.115[500] (232 bytes)
Dec 03 04:44:13 systemd[1]: Started strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.
Dec 03 04:44:13 charon[7586]: 10[ENC] <PEER-R1|1> parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(MULT_AUTH) ]
Dec 03 04:44:13 charon[7586]: 10[CFG] <PEER-R1|1> selected proposal: IKE:AES_GCM_12_128/PRF_HMAC_SHA2_256/CURVE_25519
Dec 03 04:44:13 charon[7586]: 10[IKE] <PEER-R1|1> authentication of 'R5' (myself) with pre-shared key
Dec 03 04:44:13 charon[7586]: 10[IKE] <PEER-R1|1> establishing CHILD_SA PEER-R1-tunnel-0{2}
Dec 03 04:44:13 charon-systemd[7586]: establishing CHILD_SA PEER-R1-tunnel-0{2}
Dec 03 04:44:13 charon[7586]: 10[ENC] <PEER-R1|1> generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Dec 03 04:44:13 charon[7586]: 10[NET] <PEER-R1|1> sending packet: from 104.167.216.115[4500] to 184.75.211.181[4500] (369 bytes)
Dec 03 04:44:13 charon[7586]: 07[NET] <PEER-R1|1> received packet: from 184.75.211.181[4500] to 104.167.216.115[4500] (255 bytes)
Dec 03 04:44:13 charon[7586]: 07[ENC] <PEER-R1|1> parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) ]
Dec 03 04:44:13 charon[7586]: 07[IKE] <PEER-R1|1> authentication of 'R1' with pre-shared key successful
Dec 03 04:44:13 charon-systemd[7586]: IKE_SA PEER-R1[1] established between 104.167.216.115[R5]...184.75.211.181[R1]
Dec 03 04:44:13 charon[7586]: 07[IKE] <PEER-R1|1> peer supports MOBIKE
Dec 03 04:44:13 charon[7586]: 07[IKE] <PEER-R1|1> IKE_SA PEER-R1[1] established between 104.167.216.115[R5]...184.75.211.181[R1]
Dec 03 04:44:13 charon[7586]: 07[IKE] <PEER-R1|1> scheduling rekeying in 27346s
Dec 03 04:44:13 charon[7586]: 07[IKE] <PEER-R1|1> maximum IKE_SA lifetime 30226s
Dec 03 04:44:13 charon[7586]: 07[CFG] <PEER-R1|1> selected proposal: ESP:AES_GCM_12_128/NO_EXT_SEQ
Dec 03 04:44:13 charon[7586]: 07[KNL] <PEER-R1|1> policy already exists, try to update it
Dec 03 04:44:13 charon-systemd[7586]: CHILD_SA PEER-R1-tunnel-0{2} established with SPIs cf85b378_i c81c4ab8_o and TS 10.100.100.5/32[gre] === 10.100.100.1/32[gre]
Dec 03 04:44:13 charon[7586]: 07[KNL] <PEER-R1|1> policy already exists, try to update it
Dec 03 04:44:13 charon[7586]: 07[KNL] <PEER-R1|1> policy already exists, try to update it
Dec 03 04:44:13 charon[7586]: 07[IKE] <PEER-R1|1> CHILD_SA PEER-R1-tunnel-0{2} established with SPIs cf85b378_i c81c4ab8_o and TS 10.100.100.5/32[gre] === 10.100.100.1/32[gre]
Dec 03 04:44:43 charon[7586]: 07[NET] <PEER-R1|1> received packet: from 184.75.211.181[4500] to 104.167.216.115[4500] (53 bytes)
Dec 03 04:44:43 charon[7586]: 07[ENC] <PEER-R1|1> parsed INFORMATIONAL request 0 [ ]
Dec 03 04:44:43 charon[7586]: 07[ENC] <PEER-R1|1> generating INFORMATIONAL response 0 [ ]
Dec 03 04:44:43 charon[7586]: 07[NET] <PEER-R1|1> sending packet: from 104.167.216.115[4500] to 184.75.211.181[4500] (53 bytes)

Strangely enough I only noticed it on one node that has 3 tunnels terminated, two other nodes have 2 tunnels each and I don’t sink i saw this error.

Vyos instance is running on proxmox with physical NIC (Intel i350) passed through.

I don’t think this error is the root cause of tunnel not working but it is the most noticeable issue.

Here is config of vpn ipsec

r5# show vpn ipsec | strip-private 
 authentication {
     psk PSK-KEY-R1 {
         id R1
         id R5
         secret xxxxxx
     }
     psk PSK-KEY-R23 {
         id R5
         id R23
         secret xxxxxx
     }
     psk PSK-KEY-R24 {
         id R24
         id R5
         secret xxxxxx
     }
 }
 esp-group ESP {
     pfs enable
     proposal 10 {
         encryption aes128gcm96
         hash sha256
     }
 }
 ike-group IKE-INITIATOR {
     close-action start
     dead-peer-detection {
         action restart
         interval 30
     }
     key-exchange ikev2
     proposal 10 {
         dh-group 31
         encryption aes128gcm96
         hash sha256
     }
 }
 ike-group IKE-RESPONDER {
     close-action none
     key-exchange ikev2
     proposal 10 {
         dh-group 31
         encryption aes128gcm96
         hash sha256
     }
 }
 options {
     interface br0
 }
 site-to-site {
     peer PEER-R1 {
         authentication {
             local-id R5
             mode pre-shared-secret
             remote-id R1
         }
         connection-type initiate
         default-esp-group ESP
         description TOR1-R1
         ike-group IKE-INITIATOR
         ikev2-reauth inherit
         local-address xxx.xxx.216.115
         remote-address xxx.xxx.211.181
         tunnel 0 {
             local {
                 prefix xxx.xxx.100.5/32
             }
             protocol gre
             remote {
                 prefix xxx.xxx.100.1/32
             }
         }
     }
     peer PEER-R23 {
         authentication {
             local-id R5
             mode pre-shared-secret
             remote-id R23
         }
         connection-type respond
         default-esp-group ESP
         description WBY1-R23
         ike-group IKE-RESPONDER
         ikev2-reauth inherit
         local-address xxx.xxx.216.115
         remote-address xxx.xxx.67.177
         tunnel 0 {
             local {
                 prefix xxx.xxx.100.5/32
             }
             protocol gre
             remote {
                 prefix xxx.xxx.100.23/32
             }
         }
     }
     peer PEER-R24 {
         authentication {
             local-id R5
             mode pre-shared-secret
             remote-id R24
         }
         connection-type respond
         default-esp-group ESP
         description WBY1-R24
         ike-group IKE-RESPONDER
         ikev2-reauth inherit
         local-address xxx.xxx.216.115
         remote-address any
         tunnel 0 {
             local {
                 prefix xxx.xxx.100.5/32
             }
             protocol gre
             remote {
                 prefix xxx.xxx.100.24/32
             }
         }
     }
 }

these are tunnel interfaces:

tunnel tun51 {
     address xxxx:xxxx:6000:cc00::1/128
     address xxx.xxx.128.125/31
     description "[r5-r1-ipsec] to tor1-r1"
     encapsulation gre
     mtu 1380
     remote xxx.xxx.100.1
     source-address xxx.xxx.100.5
 }
 tunnel tun523 {
     address xxx.xxx.128.173/31
     address xxxx:xxxx:6000:cc00::1/128
     description "[r5-r23-ipsec-gre] to wby1-r23 (via Bell Canada)"
     enable-multicast
     encapsulation gre
     mtu 1380
     remote xxx.xxx.100.23
     source-address xxx.xxx.100.5
 }
 tunnel tun524 {
     address xxxx:xxxx:6000:cc00::1/128
     address xxx.xxx.128.175/31
     description "[r5-r24-ipsec-gre] to wby1-r24 (via Starlink)"
     enable-multicast
     encapsulation gre
     remote xxx.xxx.100.24
     source-address xxx.xxx.100.5
 }

…and this is operational status:

r5:~$ show vpn ipsec sa | strip-private 
Connection         State    Uptime    Bytes In/Out    Packets In/Out    Remote address    Remote ID    Proposal
-----------------  -------  --------  --------------  ----------------  ----------------  -----------  --------------
PEER-R1-tunnel-0   up       19m13s    28K/0B          343/0             xxx.xxx.211.181    R1           AES_GCM_12_128
PEER-R23-tunnel-0  up       17m32s    25K/0B          304/0             xxx.xxx.67.177    R23          AES_GCM_12_128
PEER-R24-tunnel-0  up       17m30s    25K/0B          305/0             xxx.xxx.192.159   R24          AES_GCM_12_128

(notice, that counters are updated only on one side).

r5:~$ show vpn ike sa | strip-private 
Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.192.159 R24                     xxx.xxx.216.115 R5                     


    State  IKEVer  Encrypt      Hash          D-H Group      NAT-T  A-Time  L-Time
    -----  ------  -------      ----          ---------      -----  ------  ------
    up     IKEv2   AES_GCM_12_128 n/a           CURVE_25519    no     1115    26655  

Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.67.177 R23                      xxx.xxx.216.115 R5                     


    State  IKEVer  Encrypt      Hash          D-H Group      NAT-T  A-Time  L-Time
    -----  ------  -------      ----          ---------      -----  ------  ------
    up     IKEv2   AES_GCM_12_128 n/a           CURVE_25519    no     1117    25954  

Peer ID / IP                            Local ID / IP
------------                            -------------
xxx.xxx.211.181 R1                       xxx.xxx.216.115 R5                     


    State  IKEVer  Encrypt      Hash          D-H Group      NAT-T  A-Time  L-Time
    -----  ------  -------      ----          ---------      -----  ------  ------
    up     IKEv2   AES_GCM_12_128 n/a           CURVE_25519    no     1217    26129

Update 1

I’ve just realized that this system was installed from scratch and the other two were upgraded from 1.3.5. There are some differences in partition layouts between old and new ones but I don’t think it means anything.

I also tried removing one of 3 instances but it did not make any difference - errors remains and tunnels not working

2

Hi, @dtoux

In this case, it seems there are two likely unrelated issues. The first involves the error messages you saw, which result from a SIGSEGV signal received by the StrongSwan charon process. This is definitely not ideal, and the cause could vary. If you could share more logs, especially those leading up to the error, and not just from charon but from the entire system, that would help to tell more.

Despite this issue, the process restarts correctly, and sessions are established successfully. In other words, IPsec itself is not the problem. The issue you should focus on is likely in your routing configuration. There are no outgoing packets because nothing is probably being routed through these tunnels. I recommend double-checking this.

3

Do you get the same addr2line errors and faulty ipsec if you remove the IPv6 stuff regarding IPsec from your VyOS config?

4

I can reliably reproduce the error message by just doing reset ipsec, so the log begins from when command was issued. it appears that error occurs during shutdown and it prevents service from exiting cleanly… then systemd kills whatever remains from the process after about 30 second timeout and then restarts. This likely causes what commit command to hang for half a minute. Generally this would be harmless but considering that strongswan manipulates kernel state, the error may be preventing it from properly cleaning after itself.

As for the routing issue, I use /31 subnets for tunnels, so routes are added automatically as “connected”:

r5:~$ show ip route | strip-private 
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

L * xxx.xxx.100.5/32 is directly connected, dum0, weight 1, 16:44:46
C>* xxx.xxx.100.5/32 is directly connected, dum0, weight 1, 16:44:46
S>* xxx.xxx.128.0/24 [1/0] unreachable (blackhole), weight 1, 16:44:20
O   xxx.xxx.128.17/32 [110/0] is directly connected, lo, weight 1, 16:44:20
L * xxx.xxx.128.17/32 is directly connected, lo, weight 1, 16:44:46
C>* xxx.xxx.128.17/32 is directly connected, lo, weight 1, 16:44:46
O   xxx.xxx.128.24/29 [110/1] is directly connected, eth1, weight 1, 16:44:20
C>* xxx.xxx.128.24/29 is directly connected, eth1, weight 1, 16:44:42
L>* xxx.xxx.128.25/32 is directly connected, eth1, weight 1, 16:44:42
O   xxx.xxx.128.124/31 [110/10] is directly connected, tun51, weight 1, 16:44:20
C>* xxx.xxx.128.124/31 is directly connected, tun51, weight 1, 16:44:39
L>* xxx.xxx.128.125/32 is directly connected, tun51, weight 1, 16:44:39
O   xxx.xxx.128.172/31 [110/10] is directly connected, tun523, weight 1, 16:44:20
C>* xxx.xxx.128.172/31 is directly connected, tun523, weight 1, 16:44:40
L>* xxx.xxx.128.173/32 is directly connected, tun523, weight 1, 16:44:40
O   xxx.xxx.128.174/31 [110/50] is directly connected, tun524, weight 1, 16:44:20
C>* xxx.xxx.128.174/31 is directly connected, tun524, weight 1, 16:44:41
L>* xxx.xxx.128.175/32 is directly connected, tun524, weight 1, 16:44:41
C>* xxx.xxx.216.112/29 is directly connected, br0, weight 1, 16:44:40
L>* xxx.xxx.216.114/32 is directly connected, br0, weight 1, 16:44:40
S>* xxx.xxx.211.176/29 [1/0] is directly connected, br0, weight 1, 16:44:20

…so, those xxx/31 are properly routed to the tunnel. The node on the other end has identical configuration and packets are arriving from remote side but nothing flows back - counters on the opposite side are exact reverse. So, my leading theory is that the error somehow affects the process after restart… there is one flaw in this theory though - the tunnels shoud work after clean boot and they don’t… I will try to gather more data tonight by collecting logs right after reboot.

5

If you are referring to the tunnel that runs over IPv6 then I’ve already tried that. I generally prefer running ipsec via IPv6 when available, it is probably doesn’t matter that much but considering that IPSEC was designed as part of IPv6 may give it better karma or something :slight_smile:

But I generally wouldn’t worry about IPv6 on GRE tunnel. I use different addressing method and routes are created by ospfv3 and only come up when tunnel is operational. I used to use BFD for link monitoring but it creates more problems than it solves, so now I’m back to native ospfv3.

6

I did some rebooting today and error does not occur on initial boot but tunnels sill don’t come up, so the two issues are likely unrelated at this point.

7

I was thinking since you got shitloads of addr2line errors in the log perhaps some package have been renamed between bookworm and trixie and if addr2line is missing then perhaps IPv6 will fail and then it cascades from there for your IPsec config.

So using only IPv4 (remove all references to IPv6 addresses and ranges) works with your IPsec config?

What about the opposite - using only IPv6 (remove all references to IPv4 addresses and ranges) will still work with your IPsec config?

But having a dualstack config where both IPv4 and IPv6 is being used by the IPsec config will fail?

8

Found the solution for the second problem - apparently name resolution was not working properly (management network has no internet access) and as a result NTP failed to find servers causing time to drift by 10+ seconds… as soon as I’ve synchronized the clock - tunnels came up instantly.

…unfortunately it did not fix the error from the log, but it scaled down from potential problem to nuisance :wink:

9

addr2line is a debugging tool usually called when something’s crashed and to help you figure out what line in the program caused the crash:

addr2line translates addresses or symbol+offset into file names
and line numbers.  Given an address or symbol+offset in an
executable or an offset in a section of a relocatable object, it
uses the debugging information to figure out which file name and
line number are associated with it.
10

Yes but that seems to be missing according to the error message?

sh: line 1: addr2line: command not found

And according to Debian -- Package Search Results -- addr2line then this package should be included in VyOS?

librust-addr2line-dev

11

Yes, it does. But my point is, it’s only required/called after the process has fully crashed and now an attempt to assist with debugging it is called.
i.e. It’s shows up after the crash, the lack of it being there isn’t the cause of the crash.

12

After some further investigation it seems goes deeper than that.

  • I have static routes defined between two endpoints and they work as expected
  • However my nodes may be isolated from the internet until tunnels are established and having just static routes between nodes seems not to be enough to establish tunnel with pre-shared keys.
  • My suspicion is that strongswan tries to resolve endpoint identities using DNS and if DNS is unavailable - it may fail.

The way I come to this conclusion is that as soon as I add a static route to the internet the tunnel start working again. But if I remove the static route after that it drops after some time, likely when it times to re-key.

My theory is that strongswan (or Vyos) treat endpoint IDs as of undetermined type, and they try to establish if ID is a valid hostname name by using DNS. If DNS search fails - strong swan does no renew the key letting the tunnel expire even though local/remote IDs are not actual hostnames… in my case I assign them R1, R5, etc. as identifiers.

I switched my tunnels to use rsa keys and certificates instead of pre-shared keys but need more time to confirm these conclusions.

At this point it all I can conclude that this can be a bug or a feature depending how you look at it, but nevertheless - it seems like using pre-shared keys may have unexpected limitations that people should be informed about.

…and I also realize that these findings drifted away from the original subject, but in my head - they are all plausible causes of my problem :slight_smile:

13

Technically using certbased auth should give you more problems due to CRL/OSCP trying to resolve before connection and if your DNS doesnt work before the tunnel is up you are then up for a surprise.

14

Good point :+1: I ran into a similar issue on Juniper SRX but they require either explicitly disable CRL or add CRL to pki – otherwise configuration won’t even pass validation (not sure how they deals with OSCP though). Maybe Vyos could add some safety around it as well.