Problems with mixed VyOS/Cisco DMVPN

[steveb]

Having some trouble with my mixed Cisco/VyOS DMVPN. In my GNS3 environment, I have a Cisco hub, two Cisco spoke routers, and two Vyos spoke routers.

-The Cisco spokes pass traffic to the hub and each other

-The Vyos spokes can also pass traffic to the hub and each other

-The Cisco spokes CANNOT successfully ping the Vyos spokes

-The Vyos spokes get ONE successful ping response from the Cisco spokes, and the CLI hangs up on subsequent pings (strangely, it never times out)

-The present DMVPN configuration is Phase 1

-IPSEC isn’t configured right now, just GRE and NHRP

(All configs are truncated for brevity)

===================
Cisco hub config:

hostname hub

ip dhcp pool hubclientLAN
network 192.168.0.0 255.255.255.252
default-router 192.168.0.1

interface Loopback0
description router ID
ip address 192.168.0.129 255.255.255.255

interface Tunnel0
ip address 172.16.0.254 255.255.255.0
no ip redirects
ip mtu 1476
ip nhrp authentication allrise
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip tcp adjust-mss 1436
ip ospf network point-to-multipoint
tunnel source 1.0.0.2
tunnel mode gre multipoint
tunnel key 99

interface GigabitEthernet0/0
description internet link
ip address 1.0.0.2 255.255.255.252

interface GigabitEthernet0/1
description hub client LAN
ip address 192.168.0.1 255.255.255.252

router ospf 100
network 172.16.0.0 0.0.0.255 area 0
network 192.168.0.0 0.0.0.3 area 0

no ip http server
no ip http secure-server
ip route 11.0.0.0 255.255.255.0 1.0.0.1
ip route 21.0.0.0 255.255.255.0 1.0.0.1
ip route 31.0.0.0 255.255.255.0 1.0.0.1
ip route 41.0.0.0 255.255.255.0 1.0.0.1

end

======================
Vyos 1.4 Spoke 1 config:

set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 description ‘WAN Link’
set interfaces ethernet eth1 address ‘192.168.1.1/30’
set interfaces ethernet eth1 description ‘spoke 1 client network’
set interfaces tunnel tun0 address ‘172.16.0.1/24’
set interfaces tunnel tun0 encapsulation ‘gre’
set interfaces tunnel tun0 ip ospf dead-interval ‘40’
set interfaces tunnel tun0 ip ospf hello-interval ‘10’
set interfaces tunnel tun0 ip ospf network ‘point-to-multipoint’
set interfaces tunnel tun0 ip ospf priority ‘1’
set interfaces tunnel tun0 ip ospf retransmit-interval ‘5’
set interfaces tunnel tun0 ip ospf transmit-delay ‘1’
set interfaces tunnel tun0 local-ip ‘0.0.0.0’
set interfaces tunnel tun0 mtu ‘1476’
set interfaces tunnel tun0 multicast ‘enable’
set interfaces tunnel tun0 parameters ip key ‘99’
set interfaces tunnel tun0 policy route ‘change-mss’
set policy route change-mss rule 1 protocol ‘tcp’
set policy route change-mss rule 1 set tcp-mss ‘1360’
set policy route change-mss rule 1 tcp flags ‘SYN’
set protocols nhrp tunnel tun0 cisco-authentication ‘allrise’
set protocols nhrp tunnel tun0 map 172.16.0.254/24 nbma-address ‘1.0.0.2’
set protocols nhrp tunnel tun0 map 172.16.0.254/24 register
set protocols nhrp tunnel tun0 multicast ‘nhs’
set protocols ospf area 0
set protocols ospf area 0.0.0.0 network ‘172.16.0.0/24’
set protocols ospf area 0.0.0.0 network ‘192.168.1.0/30’
set protocols static route 1.0.0.0/24 next-hop dhcp-interface ‘eth0’
set protocols static route 21.0.0.0/24 dhcp-interface ‘eth0’
set protocols static route 31.0.0.0/24 dhcp-interface ‘eth0’
set protocols static route 41.0.0.0/24 dhcp-interface ‘eth0’
set service dhcp-server shared-network-name spoke1clientLAN subnet 192.168.1.0/30 default-router ‘192.168.1.1’
set service dhcp-server shared-network-name spoke1clientLAN subnet 192.168.1.0/30 range 0 start ‘192.168.1.2’
set service dhcp-server shared-network-name spoke1clientLAN subnet 192.168.1.0/30 range 0 stop ‘192.168.1.2’
set system host-name ‘spoke1’

====================
Cisco Spoke 2 config:

hostname spoke2

ip dhcp pool spoke2clientLAN
network 192.168.2.0 255.255.255.252
default-router 192.168.2.1

interface Loopback0
description router ID
ip address 192.168.2.129 255.255.255.255

interface Tunnel0
ip address 172.16.0.2 255.255.255.0
ip mtu 1476
ip nhrp authentication allrise
ip nhrp map multicast dynamic
ip nhrp map 172.16.0.254 1.0.0.2
ip nhrp network-id 1
ip nhrp nhs 172.16.0.254
ip tcp adjust-mss 1436
ip ospf network point-to-multipoint
tunnel source GigabitEthernet0/0
tunnel destination 1.0.0.2
tunnel key 99

interface GigabitEthernet0/0
description WAN Link
ip address dhcp

interface GigabitEthernet0/1
description spoke 2 client network
ip address 192.168.2.1 255.255.255.252

router ospf 100
network 172.16.0.0 0.0.0.255 area 0
network 192.168.2.0 0.0.0.3 area 0

ip route 1.0.0.0 255.255.255.252 GigabitEthernet0/0
ip route 11.0.0.0 255.255.255.0 21.0.0.1
ip route 31.0.0.0 255.255.255.0 21.0.0.1
ip route 41.0.0.0 255.255.255.0 21.0.0.1

end

====================
Cisco Spoke 3 config:

hostname spoke3

ip dhcp pool spoke3clientLAN
network 192.168.3.0 255.255.255.252
default-router 192.168.3.1

interface Loopback0
description router ID
ip address 192.168.3.129 255.255.255.255

interface Tunnel0
ip address 172.16.0.3 255.255.255.0
ip mtu 1476
ip nhrp authentication allrise
ip nhrp map multicast dynamic
ip nhrp map 172.16.0.254 1.0.0.2
ip nhrp network-id 1
ip nhrp nhs 172.16.0.254
ip tcp adjust-mss 1436
ip ospf network point-to-multipoint
tunnel source GigabitEthernet0/0
tunnel destination 1.0.0.2
tunnel key 99

interface GigabitEthernet0/0
description WAN Link
ip address dhcp

interface GigabitEthernet0/1
description spoke 3 client network
ip address 192.168.3.1 255.255.255.252

router ospf 100
network 172.16.0.0 0.0.0.255 area 0
network 192.168.3.0 0.0.0.3 area 0

ip route 1.0.0.0 255.255.255.252 GigabitEthernet0/0
ip route 11.0.0.0 255.255.255.0 GigabitEthernet0/0
ip route 21.0.0.0 255.255.255.0 GigabitEthernet0/0
ip route 41.0.0.0 255.255.255.0 GigabitEthernet0/0

end

=====================
Vyos 1.4 Spoke 4 config:

set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 description ‘WAN Link’
set interfaces ethernet eth1 address ‘192.168.4.1/30’
set interfaces ethernet eth1 description ‘spoke 4 client network’
set interfaces tunnel tun0 address ‘172.16.0.4/24’
set interfaces tunnel tun0 encapsulation ‘gre’
set interfaces tunnel tun0 ip ospf dead-interval ‘40’
set interfaces tunnel tun0 ip ospf hello-interval ‘10’
set interfaces tunnel tun0 ip ospf network ‘point-to-multipoint’
set interfaces tunnel tun0 ip ospf priority ‘1’
set interfaces tunnel tun0 ip ospf retransmit-interval ‘5’
set interfaces tunnel tun0 ip ospf transmit-delay ‘1’
set interfaces tunnel tun0 local-ip ‘0.0.0.0’
set interfaces tunnel tun0 mtu ‘1476’
set interfaces tunnel tun0 multicast ‘enable’
set interfaces tunnel tun0 parameters ip key ‘99’
set interfaces tunnel tun0 policy route ‘change-mss’
set policy route change-mss rule 1 protocol ‘tcp’
set policy route change-mss rule 1 set tcp-mss ‘1360’
set policy route change-mss rule 1 tcp flags ‘SYN’
set protocols nhrp tunnel tun0 cisco-authentication ‘allrise’
set protocols nhrp tunnel tun0 map 172.16.0.254/24 nbma-address ‘1.0.0.2’
set protocols nhrp tunnel tun0 map 172.16.0.254/24 register
set protocols nhrp tunnel tun0 multicast ‘nhs’
set protocols ospf area 0
set protocols ospf area 0.0.0.0 network ‘172.16.0.0/24’
set protocols ospf area 0.0.0.0 network ‘192.168.4.0/30’
set protocols static route 1.0.0.0/24 dhcp-interface ‘eth0’
set protocols static route 11.0.0.0/24 dhcp-interface ‘eth0’
set protocols static route 21.0.0.0/24 dhcp-interface ‘eth0’
set protocols static route 31.0.0.0/24 dhcp-interface ‘eth0’
set service dhcp-server shared-network-name spoke4clientLAN subnet 192.168.4.0/30 default-router ‘192.168.4.1’
set service dhcp-server shared-network-name spoke4clientLAN subnet 192.168.4.0/30 range 0 start ‘192.168.4.2’
set service dhcp-server shared-network-name spoke4clientLAN subnet 192.168.4.0/30 range 0 stop ‘192.168.4.2’
set system host-name ‘spoke4’

[e.khudiyev]

Hi @steveb , could you please share the exact software version release that you’ve used for IOS and VyOS as well as network topology from your GNS3 lab?

[steveb]

Thanks for the reply @e.khudiyev!

My topology:

Cisco vers: Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.6(2)T, RELEASE SOFTWARE (fc2)

VyOS vers: VyOS 1.4-rolling-202101150632

[e.khudiyev]

@steveb could you please tell which interfaces are you trying to ping from Cisco to VyOS spokes and vice-versa?

Just built a similar topology with 1 Cisco HUB, 1 Cisco SPOKE and 1 VyOS SPOKE and everything seems to be working. The only difference is that I’ve used VyOS 1.4-rolling-202108110714. Could you please try this one and check if the result is the same or not? By the way, configuration part is little bit differs and based on the information you’ve provided looks like (Spoke-1):

set interfaces tunnel tun0 address ‘172.16.0.1/24’
set interfaces tunnel tun0 encapsulation ‘gre’
set interfaces tunnel tun0 mtu ‘1476’
set interfaces tunnel tun0 multicast ‘enable’
set interfaces tunnel tun0 parameters ip key ‘99’
set interfaces tunnel tun0 policy route ‘change-mss’
set interfaces tunnel tun0 source-address ‘0.0.0.0’
set policy route change-mss rule 1 protocol ‘tcp’
set policy route change-mss rule 1 set tcp-mss ‘1360’
set policy route change-mss rule 1 tcp flags ‘SYN’
set protocols nhrp tunnel tun0 cisco-authentication ‘allrise’
set protocols nhrp tunnel tun0 map 172.16.0.254/24 nbma-address ‘10.0.0.1’
set protocols nhrp tunnel tun0 map 172.16.0.254/24 register
set protocols nhrp tunnel tun0 multicast ‘nhs’
set protocols ospf area 0 network ‘172.16.0.0/24’
set protocols ospf area 0 network ‘192.168.1.0/30’
set protocols ospf interface tun0 dead-interval ‘120’
set protocols ospf interface tun0 hello-interval ‘30’
set protocols ospf interface tun0 network ‘point-to-multipoint’
set protocols ospf interface tun0 priority ‘1’
set protocols ospf interface tun0 retransmit-interval ‘5’
set protocols ospf interface tun0 transmit-delay ‘1’

Ping results from VyOS spoke 1 towards Cisco spoke 2:

vyos@spoke1:~$ ping 192.168.2.1 interval 0.1 count 100 quiet
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.

— 192.168.2.1 ping statistics —
100 packets transmitted, 100 received, 0% packet loss, time 10026ms
rtt min/avg/max/mdev = 0.663/1.252/1.754/0.257 ms

vyos@spoke1:~$ ping 172.16.0.2 interval 0.1 count 100 quiet
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.

— 172.16.0.2 ping statistics —
100 packets transmitted, 100 received, 0% packet loss, time 9980ms
rtt min/avg/max/mdev = 0.749/1.351/1.672/0.134 ms

Ping from Cisco to VyOS:
spoke2#ping 192.168.1.1 timeout 1 repeat 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 192.168.1.1, timeout is 1 seconds:
!!!
!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/2 ms

spoke2#ping 172.16.0.1 timeout 1 repeat 100
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 172.16.0.1, timeout is 1 seconds:
!!!
!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/2 ms

[steveb]

@e.khudiyev thanks, updating to the newer image resolved the problem. All tunnel paths are registering and passing traffic.

To answer your question, my failed pings were between the tunnel interfaces. I wasn’t attempting recursive pings.

Using Wireshark, I was seeing that the hub was rejecting the VyOS spokes’ NHRP join requests with the message: “Receive Registration Reply packet with error - unique address registered already” Even though I was definitely using unique addresses. Perhaps an OpenNHRP bug? Either way, it seems to be fixed in the 202108110714 build.

As I was troubleshooting this issue, I received the advice that I should configure “ip nhrp registration non-unique” in the spokes, causing the hub to stop validating protocol/NMBA address pairs for uniqueness. This is of course a Cisco command, but I wasn’t able to find a corresponding NHRP command in VyOS. Does VyOS have the equivalent of “ip nhrp registration non-unique”?

Another question: what are the functional differences between the commands:
set protocols nhrp tunnel tun0 map 172.16.0.254/24 nbma-address 1.0.0.2
set protocols nhrp tunnel tun0 dynamic-map 172.16.0.254/24 nbma-address 1.0.0.2
I couldn’t find anything in the documentation.

Lastly, what function does this command serve: set protocols nhrp tunnel tun0 map 172.16.0.254/24 cisco

Thank you so much for your help!!

[e.khudiyev]

@steveb great to hear that everything worked fine with the newer version.

“unique address registered already” message seems to be generated when spoke devices interface configured with DHCP, received new IP address and makes a register while cache entry on the NHS not timed-out yet. One of the possible solutions for that is to decrease nhrp registration time out to the lower values. Not sure about the bug as didn’t found anything similar to that.

“ip nhrp registration non-unique” - unfortunately, I didn’t found this option in VyOS available commands or existing documentation. I think you can create a feature request on https://phabricator.vyos.net/ for that.

set protocols nhrp tunnel tun0 dynamic-map is described here

set protocols nhrp tunnel tun0 map and “cisco” option here

Hope this will answer your questions and help.