vyOS DMVPN hub, Cisco Spokes - only allowing one spoke to connect

General questions
, , ,
1

Hello,
I’m building a PoC for VyOS hub routers in GNS3. All spoke routers are Cisco, with a VyOS hub behind a firewall. I can get the tunnel to come up on one spoke router, OSPF adjacency forms, traffic passes, etc. The second I try to bring up a second spoke, the first one drops and the second one establishes connection. I’m using vyOS 1.4 rolling 202212310809. Am I missing something in my VyOS config? The spokes work fine with a Cisco Hub. I’m wondering if it has something to do with NAT-T, but I can’t find a nat-traversal command on this image?

HUB config:
set interfaces ethernet eth0 address ‘10.100.1.3/26’
set interfaces tunnel tun0 address ‘172.16.21.254/24’
set interfaces tunnel encapsulation ‘gre’
set interfaces tunnel ip adjust-mss ‘1360’
set interfaces tunnel mtu ‘1400’
set interfaces tunnel tun0 multicast ‘enable’
set interfaces tunnel tun0 parameters ip key ‘1’
set interfaces tunnel tun0 source-address ‘0.0.0.0’

set protocols static route 0.0.0.0/0 next-hop 10.100.1.62

set protocols nhrp tunnel tun0 cisco-authentication ‘passkey’
set protocols nhrp tunnel tun0 holding-time ‘600’
set protocols nhrp tunnel tun0 multicast ‘dynamic’
set protocols nhrp tunnel tun0 ‘redirect’
set protocols nhrp tunnel tun0 ‘shortcut’

set vpn ipsec esp-group ESP-HUB lifetime ‘3600’
set vpn ipsec esp-group ESP-HUB mode ‘transport’
set vpn ipsec esp-group ESP-HUB pfs ‘disable’
set vpn ipsec esp-group ESP-HUB proposal 1 encryption ‘aes128’
set vpn ipsec esp-group ESP-HUB proposal 1 hash ‘sha1’
set vpn ipsec ike-group IKE-HUB key-exchange ‘ikev1’
set vpn ipsec ike-group IKE-HUB lifetime ‘86400’
set vpn ipsec ike-group IKE-HUB mode ‘aggressive’
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group ‘5’

set vpn ipsec interface ‘eth0’

set vpn ipsec profile NHRPVPN authentication mode ‘pre-shared-secret’
set vpn ipsec profile NHRPVPN authentication pre-shared-secret ‘PSK’
set vpn ipsec profile NHRPVPN bind tunnel ‘tun0’
set vpn ipsec profile NHRPVPN esp-group ‘ESP-HUB’
set vpn ipsec profile NHRPVPN ike-group ‘IKE-HUB’

Spoke Config (Works perfectly with Cisco Hub, or single router spoke to VyOS Hub):

crypto isakmp policy 100
encryption aes
authentication pre-share
group 5

crypto isakmp key PSK address 0.0.0.0

crypto ipsec transform-set xfrm esp-aes esp-sha-hmac
mode transport

crypto ipsec profile SPOKE
set transform-set xfrm

int tun0
ip addr 172.16.21.x 255.255.255.0
ip mtu 1400
ip nhrp authentication passkey
ip nhrp map 172.16.21.254 <*public IP of HUB>
ip nhrp map multicast <*public IP of HUB>
ip nhrp network-id 1
ip nhrp holdtime 600
ip nhrp nhs 172.16.21.254
ip tcp adjust-mss 1360
tunnel source fa0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile SPOKE

2

Also, the example included in the documentation for 1.4.x includes the command:
set interfaces tunnel tun100 local-ip ‘192.0.2.1’

No image I have downloaded has local-ip as a valid command under the tunnel interface. Just like none of them have the set vpn nat-traversal or set vpn ipsec nat-traversal.

3

So after a little more digging, both tunnels come up - but it tries to route everything through the first tunnel to connect - ie, if 172.16.21.2 connects first, traffic to all subsequent connections on that subnet (172.16.21.3, .4, etc) all go to .2 as the next hop. It’s weird.

4

what does OSPF routes look like? Don’t advertise default routes, or connected WAN networks.

5

This behavior happens even if OSPF is turned off - but basically all OSPF is doing is advertising routes that are redistributed from BGP, and redistributing the routes it learns from the spokes into BGP.

6

So I did some more labbing and have been able to narrow the issue down. It happens if the hub router is behind a firewall and NAT is in use.
If I move the hub out from behind the FW and assign it a public IP directly, the issue goes away.
Not sure if this is a config issue or an implementation issue, but the same setup works perfectly with a CSR1000v as the hub router behind the firewall.

7

ok, so I finally found the issue, and it had nothing to do with VyOS.
The firewall in the lab’s license had expired. I installed a new license, and it started working fine. The weird part is that the CSR didn’t exhibit the same behavior.
Who knows. I feel like an idiot.

Thanks guys!