I’ve got a new VyOS 1.4 build I’ve been playing with for a home lab, running latest rolling release as of 2 days ago, that appears unable to build IPSec to a known-working set of Cisco DMVPN hubs. Copying the config from a functional VyOS 1.3 instance, making the necessary modifications to support 1.4, and committing results in GRE packets reaching the hub unencrypted. Running “show vpn ike sa” seems to always show the IPSec daemon as being down, even though it isn’t.
set interfaces tunnel tun253 address 'X.X.X.X/24'
set interfaces tunnel tun253 description 'DMVPN'
set interfaces tunnel tun253 disable-link-detect
set interfaces tunnel tun253 enable-multicast
set interfaces tunnel tun253 encapsulation 'gre'
set interfaces tunnel tun253 mtu '1400'
set interfaces tunnel tun253 parameters ip key '253'
set interfaces tunnel tun253 source-interface 'eth1'
set protocols nhrp tunnel tun253 cisco-authentication 'XXXXXXXX'
set protocols nhrp tunnel tun253 holding-time '300'
set protocols nhrp tunnel tun253 map X.X.X.X/24 cisco
set protocols nhrp tunnel tun253 map X.X.X.X/24 nbma-address 'X.X.X.X'
set protocols nhrp tunnel tun253 map X.X.X.X/24 register
set protocols nhrp tunnel tun253 map X.X.X.X/24 cisco
set protocols nhrp tunnel tun253 map X.X.X.X/24 nbma-address 'X.X.X.X'
set protocols nhrp tunnel tun253 map X.X.X.X/24 register
set protocols nhrp tunnel tun253 map X.X.X.X/24 cisco
set protocols nhrp tunnel tun253 map X.X.X.X/24 nbma-address 'X.X.X.X'
set protocols nhrp tunnel tun253 map X.X.X.X/24 register
set protocols nhrp tunnel tun253 multicast 'nhs'
set protocols nhrp tunnel tun253 redirect
set protocols nhrp tunnel tun253 shortcut
set vpn ipsec esp-group ESP-DMVPN254 lifetime '900'
set vpn ipsec esp-group ESP-DMVPN254 mode 'tunnel'
set vpn ipsec esp-group ESP-DMVPN254 pfs 'disable'
set vpn ipsec esp-group ESP-DMVPN254 proposal 1 encryption 'aes128'
set vpn ipsec esp-group ESP-DMVPN254 proposal 1 hash 'sha512'
set vpn ipsec ike-group IKE-DMVPN254 close-action 'restart'
set vpn ipsec ike-group IKE-DMVPN254 dead-peer-detection action 'restart'
set vpn ipsec ike-group IKE-DMVPN254 key-exchange 'ikev1'
set vpn ipsec ike-group IKE-DMVPN254 lifetime '86400'
set vpn ipsec ike-group IKE-DMVPN254 proposal 1 dh-group '2'
set vpn ipsec ike-group IKE-DMVPN254 proposal 1 encryption 'aes256'
set vpn ipsec ike-group IKE-DMVPN254 proposal 1 hash 'sha512'
set vpn ipsec ike-group IKE-DMVPN254 proposal 2 dh-group '2'
set vpn ipsec ike-group IKE-DMVPN254 proposal 2 encryption 'aes256'
set vpn ipsec ike-group IKE-DMVPN254 proposal 2 hash 'sha512'
set vpn ipsec interface 'eth1'
set vpn ipsec log level '2'
set vpn ipsec options disable-route-autoinstall
set vpn ipsec profile DMVPN254 authentication mode 'pre-shared-secret'
set vpn ipsec profile DMVPN254 authentication pre-shared-secret 'psk'
set vpn ipsec profile DMVPN254 bind tunnel 'tun253'
set vpn ipsec profile DMVPN254 esp-group 'ESP-DMVPN254'
set vpn ipsec profile DMVPN254 ike-group 'IKE-DMVPN254'
On the Cisco side, I just repeatedly see:
%CRYPTO-4-RECVD_PKT_NOT_IPSEC: Rec'd packet not an IPSEC packet. (ip) vrf/dest_addr=
Output of show version:
Version: VyOS 1.4-rolling-202307141223
Release train: current
Built by: autobuild@vyos.net
Built on: Fri 14 Jul 2023 12:23 UTC
Build UUID: 3af12ceb-f3d4-4bf9-b647-48627e54223e
Build commit ID: 5a81df95612424
Architecture: x86_64
Boot via: installed image
System type: bare metal
Hardware vendor: Default string
Hardware model: Default string
Hardware S/N:
Hardware UUID: 03000200-0400-0500-0006-000700080009
Copyright: VyOS maintainers and contributors
My current question is, have there been any breaking changes to IPSec and GRE tunnels in the 1.4 release that aren’t well documented? Any configurations changes that are similarly undocumented that need to be input for this to work?
Figured I’d post here before I opened a bug on this.
@lclements0
Can you show IPSEC logs from VyOS?
sudo journalctl -l | grep charon
Sure can, these are the logs from a timestamp around a change to the NHRP protocol configuration.
Jul 16 13:11:19 vyos charon[2706]: 00[DMN] SIGTERM received, shutting down
Jul 16 13:11:19 vyos charon-systemd[2706]: SIGTERM received, shutting down
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] PKCS11 module '<name>' lacks library path
Jul 16 13:11:19 vyos charon-systemd[21031]: PKCS11 module '<name>' lacks library path
Jul 16 13:11:19 vyos charon[21031]: 00[PTS] TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Jul 16 13:11:19 vyos charon-systemd[21031]: TPM 2.0 - could not load "libtss2-tcti-tabrmd.so.0"
Jul 16 13:11:19 vyos charon-systemd[21031]: plugin 'tpm': failed to load - tpm_plugin_create returned NULL
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] providers loaded by OpenSSL: legacy default
Jul 16 13:11:19 vyos charon-systemd[21031]: providers loaded by OpenSSL: legacy default
Jul 16 13:11:19 vyos charon[21031]: 00[NET] using forecast interface eth1
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Jul 16 13:11:19 vyos charon-systemd[21031]: using forecast interface eth1
Jul 16 13:11:19 vyos charon-systemd[21031]: joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading ca certificates from '/etc/ipsec.d/cacerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading aa certificates from '/etc/ipsec.d/aacerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading crls from '/etc/ipsec.d/crls'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading attribute certificates from '/etc/ipsec.d/acerts'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loading secrets from '/etc/ipsec.secrets'
Jul 16 13:11:19 vyos charon-systemd[21031]: loading crls from '/etc/ipsec.d/crls'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Jul 16 13:11:19 vyos charon-systemd[21031]: loading secrets from '/etc/ipsec.secrets'
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] loaded 0 RADIUS server configurations
Jul 16 13:11:19 vyos charon-systemd[21031]: opening secrets file '/etc/ipsec.secrets' failed: No such file or directory
Jul 16 13:11:19 vyos charon[21031]: 00[CFG] HA config misses local/remote address
Jul 16 13:11:19 vyos charon-systemd[21031]: loaded 0 RADIUS server configurations
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-
alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-g
eneric xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Jul 16 13:11:19 vyos charon-systemd[21031]: HA config misses local/remote address
Jul 16 13:11:19 vyos charon[21031]: 00[LIB] dropped capabilities, running as uid 0, gid 0
Jul 16 13:11:19 vyos charon-systemd[21031]: loaded plugins: charon-systemd test-vectors pkcs11 aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs12 pgp dnskey sshkey pem openssl gcrypt pkcs8 af-
alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac kdf ctr ccm gcm drbg curl attr kernel-netlink resolve socket-default connmark forecast stroke vici updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-radius eap-tls eap-ttls eap-tnc xauth-g
eneric xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire addrblock counters
Jul 16 13:11:19 vyos charon[21031]: 00[JOB] spawning 16 worker threads
Jul 16 13:11:19 vyos charon-systemd[21031]: dropped capabilities, running as uid 0, gid 0
Jul 16 13:11:19 vyos charon-systemd[21031]: spawning 16 worker threads
Jul 16 13:11:20 vyos charon[21031]: 15[CFG] loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Jul 16 13:11:20 vyos charon-systemd[21031]: loaded IKE shared key with id 'ike-dmvpn-tun253' for: '%any'
Jul 16 13:11:20 vyos charon[21031]: 11[CFG] added vici connection: dmvpn-DMVPN254-tun253
Jul 16 13:11:20 vyos charon-systemd[21031]: added vici connection: dmvpn-DMVPN254-tun253
Hello @lclements0
This is the working configurations of VyOS and Cisco routers
https://docs.vyos.io/en/latest/configuration/vpn/dmvpn.html
Check Hub and Spokes configurations.
I do not see any IPSEC conversations between spoke and hub.
Try to generate traffic through tunnel from spoke to hub.
Use the next commands to troubleshoot the issue.
show nhrp
show vpn ike sa
show vpn ipsec sa
sudo swanctl -l
sudo journalctl -l | grep charon
From reviewing the configuration in the docs, and comparing to my configuration, outside of a few changes that the 1.4 CLI provides, they look solid to me. What appears to be happening is GRE is escaping the VyOS box without hitting IPsec, as this spoke is sending GRE traffic to the hub unencrypted. The same configuration on a 1.3.3 spoke works fine and encrypts the GRE traffic as it should. As I mentioned in my original post, the NHRP traffic is hitting the Cisco side and it’s dropping it as it’s arriving unencrypted.
I’ll look at playing with this some more next week and updating as I have some time. Just wasn’t sure if this was a known issue.
Try to change
set vpn ipsec ike-group IKE-DMVPN254 close-action hold
From what I’ve read it keeps VyOS box from sending unencrypted traffic.