VyOS 1.4 DMVPN IPSEC not working?

lclements0 · July 16, 2023, 7:06pm

I’ve got a new VyOS 1.4 build I’ve been playing with for a home lab, running latest rolling release as of 2 days ago, that appears unable to build IPSec to a known-working set of Cisco DMVPN hubs. Copying the config from a functional VyOS 1.3 instance, making the necessary modifications to support 1.4, and committing results in GRE packets reaching the hub unencrypted. Running “show vpn ike sa” seems to always show the IPSec daemon as being down, even though it isn’t.

set interfaces tunnel tun253 address 'X.X.X.X/24'
set interfaces tunnel tun253 description 'DMVPN'
set interfaces tunnel tun253 disable-link-detect
set interfaces tunnel tun253 enable-multicast
set interfaces tunnel tun253 encapsulation 'gre'
set interfaces tunnel tun253 mtu '1400'
set interfaces tunnel tun253 parameters ip key '253'
set interfaces tunnel tun253 source-interface 'eth1'
set protocols nhrp tunnel tun253 cisco-authentication 'XXXXXXXX'
set protocols nhrp tunnel tun253 holding-time '300'
set protocols nhrp tunnel tun253 map X.X.X.X/24 cisco
set protocols nhrp tunnel tun253 map X.X.X.X/24 nbma-address 'X.X.X.X'
set protocols nhrp tunnel tun253 map X.X.X.X/24 register
set protocols nhrp tunnel tun253 map X.X.X.X/24 cisco
set protocols nhrp tunnel tun253 map X.X.X.X/24 nbma-address 'X.X.X.X'
set protocols nhrp tunnel tun253 map X.X.X.X/24 register
set protocols nhrp tunnel tun253 map X.X.X.X/24 cisco
set protocols nhrp tunnel tun253 map X.X.X.X/24 nbma-address 'X.X.X.X'
set protocols nhrp tunnel tun253 map X.X.X.X/24 register
set protocols nhrp tunnel tun253 multicast 'nhs'
set protocols nhrp tunnel tun253 redirect
set protocols nhrp tunnel tun253 shortcut
set vpn ipsec esp-group ESP-DMVPN254 lifetime '900'
set vpn ipsec esp-group ESP-DMVPN254 mode 'tunnel'
set vpn ipsec esp-group ESP-DMVPN254 pfs 'disable'
set vpn ipsec esp-group ESP-DMVPN254 proposal 1 encryption 'aes128'
set vpn ipsec esp-group ESP-DMVPN254 proposal 1 hash 'sha512'
set vpn ipsec ike-group IKE-DMVPN254 close-action 'restart'
set vpn ipsec ike-group IKE-DMVPN254 dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-DMVPN254 key-exchange 'ikev1'
set vpn ipsec ike-group IKE-DMVPN254 lifetime '86400'
set vpn ipsec ike-group IKE-DMVPN254 proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-DMVPN254 proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-DMVPN254 proposal 1 hash 'sha512'
set vpn ipsec ike-group IKE-DMVPN254 proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-DMVPN254 proposal 2 encryption 'aes256'
set vpn ipsec ike-group IKE-DMVPN254 proposal 2 hash 'sha512'
set vpn ipsec interface 'eth1'
set vpn ipsec log level '2'
set vpn ipsec options disable-route-autoinstall
set vpn ipsec profile DMVPN254 authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN254 authentication pre-shared-secret 'psk'
set vpn ipsec profile DMVPN254 bind tunnel 'tun253'
set vpn ipsec profile DMVPN254 esp-group 'ESP-DMVPN254'
set vpn ipsec profile DMVPN254 ike-group 'IKE-DMVPN254'

On the Cisco side, I just repeatedly see:

%CRYPTO-4-RECVD_PKT_NOT_IPSEC: Rec'd packet not an IPSEC packet. (ip) vrf/dest_addr=

Output of show version:

Version:          VyOS 1.4-rolling-202307141223
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Fri 14 Jul 2023 12:23 UTC
Build UUID:       3af12ceb-f3d4-4bf9-b647-48627e54223e
Build commit ID:  5a81df95612424

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Default string
Hardware model:   Default string
Hardware S/N:
Hardware UUID:    03000200-0400-0500-0006-000700080009

Copyright:        VyOS maintainers and contributors

My current question is, have there been any breaking changes to IPSec and GRE tunnels in the 1.4 release that aren’t well documented? Any configurations changes that are similarly undocumented that need to be input for this to work?

Figured I’d post here before I opened a bug on this.

a.apostoliuk · July 17, 2023, 6:40am

@lclements0
Can you show IPSEC logs from VyOS?
sudo journalctl -l | grep charon

lclements0 · July 17, 2023, 1:48pm

Sure can, these are the logs from a timestamp around a change to the NHRP protocol configuration.

Jul 16 13:11:19 vyos charon[2706]: 00[DMN] SIGTERM received, shutting down
Jul 16 13:11:19 vyos charon-systemd[2706]: SIGTERM received, shutting down
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] PKCS11 module '<name>' lacks library path
Jul 16 13:11:19 vyos charon-systemd[21031]: PKCS11 module '<name>' lacks library path
Jul 16 13:11:19 vyos charon[21031]: 00[PTS] TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Jul 16 13:11:19 vyos charon-systemd[21031]: TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Jul 16 13:11:19 vyos charon-systemd[21031]: plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] providers loaded by OpenSSL: legacy default
Jul 16 13:11:19 vyos charon-systemd[21031]: providers loaded by OpenSSL: legacy default
Jul 16 13:11:19 vyos charon[21031]: 00[NET] using forecast interface eth1
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Jul 16 13:11:19 vyos charon-systemd[21031]: using forecast interface eth1
Jul 16 13:11:19 vyos charon-systemd[21031]: joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading ca certificates from '/etc/ipsec.d/cacerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading aa certificates from '/etc/ipsec.d/aacerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading attribute certificates from '/etc/ipsec.d/acerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading crls from '/etc/ipsec.d/crls'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Jul 16 13:11:19 vyos charon-systemd[21031]: loading secrets from '/etc/ipsec.secrets'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loaded 0 RADIUS server configurations
Jul 16 13:11:19 vyos charon-systemd[21031]: opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] HA config misses local/remote address
Jul 16 13:11:19 vyos charon-systemd[21031]: loaded 0 RADIUS server configurations
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-
alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-g
eneric xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Jul 16 13:11:19 vyos charon-systemd[21031]: HA config misses local/remote address
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] dropped capabilities, running as uid 0, gid 0
Jul 16 13:11:19 vyos charon-systemd[21031]: loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-
alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-g
eneric xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Jul 16 13:11:19 vyos charon[21031]: 00[JOB] spawning 16 worker threads
Jul 16 13:11:19 vyos charon-systemd[21031]: dropped capabilities, running as uid 0, gid 0
Jul 16 13:11:19 vyos charon-systemd[21031]: spawning 16 worker threads
Jul 16 13:11:20 vyos charon[21031]: 15[CFG] loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Jul 16 13:11:20 vyos charon-systemd[21031]: loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Jul 16 13:11:20 vyos charon[21031]: 11[CFG] added vici connection: dmvpn-DMVPN254-tun253
Jul 16 13:11:20 vyos charon-systemd[21031]: added vici connection: dmvpn-DMVPN254-tun253

a.apostoliuk · July 18, 2023, 7:59am

Hello @lclements0
This is the working configurations of VyOS and Cisco routers
https://docs.vyos.io/en/latest/configuration/vpn/dmvpn.html
Check Hub and Spokes configurations.
I do not see any IPSEC conversations between spoke and hub.
Try to generate traffic through tunnel from spoke to hub.
Use the next commands to troubleshoot the issue.

show nhrp
show vpn ike sa
show vpn ipsec sa
sudo swanctl -l
sudo journalctl -l | grep charon

lclements0 · July 23, 2023, 7:16pm

From reviewing the configuration in the docs, and comparing to my configuration, outside of a few changes that the 1.4 CLI provides, they look solid to me. What appears to be happening is GRE is escaping the VyOS box without hitting IPsec, as this spoke is sending GRE traffic to the hub unencrypted. The same configuration on a 1.3.3 spoke works fine and encrypts the GRE traffic as it should. As I mentioned in my original post, the NHRP traffic is hitting the Cisco side and it’s dropping it as it’s arriving unencrypted.

I’ll look at playing with this some more next week and updating as I have some time. Just wasn’t sure if this was a known issue.

Jackwmtr · August 3, 2023, 6:52pm

Try to change

set vpn ipsec ike-group IKE-DMVPN254 close-action hold

From what I’ve read it keeps VyOS box from sending unencrypted traffic.

lclements0 · December 23, 2023, 2:45am

So I ended up coming back to this in the latest release candidate to see if this was resolved, and still seem to be having issues here.

Dec 22 21:19:41 vyos charon[24843]: 00[DMN] SIGTERM received, shutting down
Dec 22 21:19:41 vyos charon-systemd[24843]: SIGTERM received, shutting down
Dec 22 21:19:41 vyos charon[34476]: 00[DMN] Starting charon-systemd IKE daemon (strongSwan 5.9.11, Linux 6.1.69-amd64-vyos, x86_64)
Dec 22 21:19:41 vyos charon-systemd[34476]: Starting charon-systemd IKE daemon (strongSwan 5.9.11, Linux 6.1.69-amd64-vyos, x86_64)
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] PKCS11 module '<name>' lacks library path
Dec 22 21:19:41 vyos charon-systemd[34476]: PKCS11 module '<name>' lacks library path
Dec 22 21:19:41 vyos charon[34476]: 00[PTS] TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Dec 22 21:19:41 vyos charon[34476]: 00[LIB] plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Dec 22 21:19:41 vyos charon-systemd[34476]: TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Dec 22 21:19:41 vyos charon-systemd[34476]: plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Dec 22 21:19:41 vyos charon[34476]: 00[LIB] providers loaded by OpenSSL: legacy default
Dec 22 21:19:41 vyos charon-systemd[34476]: providers loaded by OpenSSL: legacy default
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] install DNS servers in '/etc/resolv.conf'
Dec 22 21:19:41 vyos charon-systemd[34476]: install DNS servers in '/etc/resolv.conf'
Dec 22 21:19:41 vyos charon[34476]: 00[NET] using forecast interface eth1
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Dec 22 21:19:41 vyos charon-systemd[34476]: using forecast interface eth1
Dec 22 21:19:41 vyos charon-systemd[34476]: joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Dec 22 21:19:41 vyos charon-systemd[34476]: loading ca certificates from '/etc/ipsec.d/cacerts'
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Dec 22 21:19:41 vyos charon-systemd[34476]: loading aa certificates from '/etc/ipsec.d/aacerts'
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Dec 22 21:19:41 vyos charon-systemd[34476]: loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Dec 22 21:19:41 vyos charon-systemd[34476]: loading attribute certificates from '/etc/ipsec.d/acerts'
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Dec 22 21:19:41 vyos charon-systemd[34476]: loading crls from '/etc/ipsec.d/crls'
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Dec 22 21:19:41 vyos charon-systemd[34476]: loading secrets from '/etc/ipsec.secrets'
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Dec 22 21:19:41 vyos charon-systemd[34476]: opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] loaded 0 RADIUS server configurations
Dec 22 21:19:41 vyos charon-systemd[34476]: loaded 0 RADIUS server configurations
Dec 22 21:19:41 vyos charon[34476]: 00[CFG] HA config misses local/remote address
Dec 22 21:19:41 vyos charon-systemd[34476]: HA config misses local/remote address
Dec 22 21:19:41 vyos charon[34476]: 00[LIB] loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Dec 22 21:19:41 vyos charon-systemd[34476]: loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Dec 22 21:19:41 vyos charon[34476]: 00[LIB] dropped capabilities, running as uid 0, gid 0
Dec 22 21:19:41 vyos charon-systemd[34476]: dropped capabilities, running as uid 0, gid 0
Dec 22 21:19:41 vyos charon[34476]: 00[JOB] spawning 16 worker threads
Dec 22 21:19:41 vyos charon-systemd[34476]: spawning 16 worker threads
Dec 22 21:19:41 vyos charon[34476]: 07[CFG] loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Dec 22 21:19:41 vyos charon-systemd[34476]: loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Dec 22 21:19:41 vyos charon[34476]: 11[CFG] added vici connection: dmvpn-DMVPN254-tun253
Dec 22 21:19:41 vyos charon-systemd[34476]: added vici connection: dmvpn-DMVPN254-tun253
Dec 22 21:22:21 vyos charon[34476]: 00[DMN] SIGTERM received, shutting down
Dec 22 21:22:21 vyos charon-systemd[34476]: SIGTERM received, shutting down
Dec 22 21:27:49 vyos charon[36837]: 00[DMN] Starting charon-systemd IKE daemon (strongSwan 5.9.11, Linux 6.1.69-amd64-vyos, x86_64)
Dec 22 21:27:49 vyos charon-systemd[36837]: Starting charon-systemd IKE daemon (strongSwan 5.9.11, Linux 6.1.69-amd64-vyos, x86_64)
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] PKCS11 module '<name>' lacks library path
Dec 22 21:27:49 vyos charon-systemd[36837]: PKCS11 module '<name>' lacks library path
Dec 22 21:27:49 vyos charon[36837]: 00[PTS] TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Dec 22 21:27:49 vyos charon[36837]: 00[LIB] plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Dec 22 21:27:49 vyos charon-systemd[36837]: TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Dec 22 21:27:49 vyos charon-systemd[36837]: plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Dec 22 21:27:49 vyos charon[36837]: 00[LIB] providers loaded by OpenSSL: legacy default
Dec 22 21:27:49 vyos charon-systemd[36837]: providers loaded by OpenSSL: legacy default
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] install DNS servers in '/etc/resolv.conf'
Dec 22 21:27:49 vyos charon-systemd[36837]: install DNS servers in '/etc/resolv.conf'
Dec 22 21:27:49 vyos charon[36837]: 00[NET] using forecast interface eth1
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Dec 22 21:27:49 vyos charon-systemd[36837]: using forecast interface eth1
Dec 22 21:27:49 vyos charon-systemd[36837]: joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Dec 22 21:27:49 vyos charon-systemd[36837]: loading ca certificates from '/etc/ipsec.d/cacerts'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Dec 22 21:27:49 vyos charon-systemd[36837]: loading aa certificates from '/etc/ipsec.d/aacerts'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Dec 22 21:27:49 vyos charon-systemd[36837]: loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Dec 22 21:27:49 vyos charon-systemd[36837]: loading attribute certificates from '/etc/ipsec.d/acerts'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Dec 22 21:27:49 vyos charon-systemd[36837]: loading crls from '/etc/ipsec.d/crls'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Dec 22 21:27:49 vyos charon-systemd[36837]: loading secrets from '/etc/ipsec.secrets'
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] loaded 0 RADIUS server configurations
Dec 22 21:27:49 vyos charon-systemd[36837]: opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Dec 22 21:27:49 vyos charon[36837]: 00[CFG] HA config misses local/remote address
Dec 22 21:27:49 vyos charon-systemd[36837]: loaded 0 RADIUS server configurations
Dec 22 21:27:49 vyos charon[36837]: 00[LIB] loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Dec 22 21:27:49 vyos charon-systemd[36837]: HA config misses local/remote address
Dec 22 21:27:49 vyos charon[36837]: 00[LIB] dropped capabilities, running as uid 0, gid 0
Dec 22 21:27:49 vyos charon-systemd[36837]: loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Dec 22 21:27:49 vyos charon[36837]: 00[JOB] spawning 16 worker threads
Dec 22 21:27:49 vyos charon-systemd[36837]: dropped capabilities, running as uid 0, gid 0
Dec 22 21:27:49 vyos charon-systemd[36837]: spawning 16 worker threads
Dec 22 21:27:49 vyos charon[36837]: 07[CFG] loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Dec 22 21:27:49 vyos charon-systemd[36837]: loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Dec 22 21:27:49 vyos charon[36837]: 11[CFG] added vici connection: dmvpn-DMVPN254-tun253
Dec 22 21:27:49 vyos charon-systemd[36837]: added vici connection: dmvpn-DMVPN254-tun253

NHRP seems to be working, and show nhrp tunnel reports back valid data.

vyos@vyos:~$ show nhrp tunnel 
Status: ok
Interface    Type    Protocol-Address    Alias-Address    Flags          NBMA-Address
-----------  ------  ------------------  ---------------  -------------  --------------
tun253       local   X.X.X.X/32                 X.X.X.X            up
tun253       local   X.X.X.X/32                                          up
tun253       local   X.X.X.X/32                 X.X.X.X            up
tun253       local   X.X.X.X/32                                          up
tun253       static  X.X.X.X/24                               used lower-up  X
tun253       static  X.X.X.X/24                               used lower-up  X

However, no matter how much traffic I throw at the tunnel, IPSec never comes up.

show vpn ipsec sa remains empty.

show vpn ipsec connections shows the connection state as “down”.

Connection             State    Type    Remote address    Local TS      Remote TS     Local id    Remote id    Proposal
---------------------  -------  ------  ----------------  ------------  ------------  ----------  -----------  ----------
dmvpn-DMVPN254-tun253  down     IKEv1   %any              -             -                                      -
dmvpn                  down     IPsec   %any              dynamic[gre]  dynamic[gre]                           -

A VyOS 1.3.X router brings this connection up using similar configuration syntax, and obviously Cisco joins up to the Cisco DMVPN hubs just fine.

Any further suggestions here for what to look at would be helpful.

Thanks!