DMVPN NHRP spoke stops sending resolution requests upon restart

Jackwmtr · April 21, 2023, 6:07pm

Hi!
Recently I’ve encountered a possible bug which renders DMVPN spokes almost unusable in DMVPN phase 2|3 scenario. (static map entries do the trick, though).
The lab to investigate is as follows:

HUB IP: NBMA: 192.168.77.66, tun IP: 10.160.17.131/25
Spoke1 IP: NBMA: 192.168.77.64, tun IP:10.160.17.141
Spoke2 IP: NBMA: 192.168.77.65, tun IP:10.160.17.151
No IPSEC, no NAT.
HUB config:

set interfaces ethernet eth0 address ‘dhcp’
set interfaces loopback lo
set interfaces tunnel tun1 address ‘10.160.17.131/25’
set interfaces tunnel tun1 description ‘DMVPN_HUB_interface’
set interfaces tunnel tun1 encapsulation ‘gre’
set interfaces tunnel tun1 multicast ‘enable’
set interfaces tunnel tun1 parameters ip key ‘17128’
set interfaces tunnel tun1 source-address ‘192.168.77.66’
set protocols nhrp tunnel tun1 cisco-authentication ‘LPDMVPN’
set protocols nhrp tunnel tun1 holding-time ‘300’
set protocols nhrp tunnel tun1 multicast ‘dynamic’

Spoke1 config:

set interfaces ethernet eth0 address ‘dhcp’
set interfaces tunnel tun10 address ‘10.160.17.141/25’
set interfaces tunnel tun10 encapsulation ‘gre’
set interfaces tunnel tun10 multicast ‘enable’
set interfaces tunnel tun10 parameters ip key ‘17128’
set interfaces tunnel tun10 source-address ‘192.168.77.64’
set protocols nhrp tunnel tun10 cisco-authentication ‘LPDMVPN’
set protocols nhrp tunnel tun10 holding-time ‘300’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 nbma-address ‘192.168.77.66’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 register
set protocols nhrp tunnel tun10 multicast ‘nhs’

Spoke2 config:

set interfaces ethernet eth0 address ‘dhcp’
set interfaces tunnel tun10 address ‘10.160.17.151/25’
set interfaces tunnel tun10 encapsulation ‘gre’
set interfaces tunnel tun10 multicast ‘enable’
set interfaces tunnel tun10 parameters ip key ‘17128’
set interfaces tunnel tun10 source-address ‘192.168.77.65’
set protocols nhrp tunnel tun10 cisco-authentication ‘LPDMVPN’
set protocols nhrp tunnel tun10 holding-time ‘300’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 nbma-address ‘192.168.77.66’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 register
set protocols nhrp tunnel tun10 multicast ‘nhs’

At the beginning all the tunnels are nice and fine:
HUB:>

vyos@HUB:~$ show nhrp tu
Status: ok

Interface: tun1
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.131
Flags: up

Interface: tun1
Type: local
Protocol-Address: 10.160.17.131/32
Flags: up

Interface: tun1
Type: dynamic
Protocol-Address: 10.160.17.141/32
NBMA-Address: 192.168.77.64
Flags: up
Expires-In: 4:01

Interface: tun1
Type: dynamic
Protocol-Address: 10.160.17.151/32
NBMA-Address: 192.168.77.65
Flags: up
Expires-In: 3:32

Spoke1:

vyos@S1:~$ show nhrp tu
Status: ok

Interface: tun10
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.141
Flags: up

Interface: tun10
Type: local
Protocol-Address: 10.160.17.141/32
Flags: up

Interface: tun10
Type: cached
Protocol-Address: 10.160.17.151/32
NBMA-Address: 192.168.77.65
Flags: used up
Expires-In: 4:56

Interface: tun10
Type: static
Protocol-Address: 10.160.17.131/25
NBMA-Address: 192.168.77.66
Flags: up

Spoke2:>

vyos@S2:~$ show nhrp tu
Status: ok

Interface: tun10
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.151
Flags: up

Interface: tun10
Type: local
Protocol-Address: 10.160.17.151/32
Flags: up

Interface: tun10
Type: cached
Protocol-Address: 10.160.17.141/32
NBMA-Address: 192.168.77.64
Flags: up
Expires-In: 2:27

Interface: tun10
Type: static
Protocol-Address: 10.160.17.131/25
NBMA-Address: 192.168.77.66
Flags: up

NHRP logs from Spoke1 (when it works:):

Apr 21 17:38:35 opennhrp[1779]: NL-ARP(tun10) who-has 10.160.17.151
Apr 21 17:38:35 opennhrp[1779]: NL-ARP(tun10) 10.160.17.151 is-at 192.168.77.66
Apr 21 17:38:35 opennhrp[1779]: Adding incomplete 10.160.17.151/32 dev tun10
Apr 21 17:38:35 opennhrp[1779]: Sending Resolution Request to 10.160.17.151
Apr 21 17:38:35 opennhrp[1779]: Sending packet 1, from: 10.160.17.141 (nbma 192.168.77.64), to: 10.160.17.151 (nbma 192.168.77.66)
Apr 21 17:38:35 opennhrp[1779]: Received Resolution Request from proto src 10.160.17.151 to 10.160.17.141
Apr 21 17:38:35 opennhrp[1779]: Sending Resolution Reply 10.160.17.141/32 is-at 192.168.77.64 (holdtime 300)
Apr 21 17:38:35 opennhrp[1779]: Sending packet 2, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:38:35 opennhrp[1779]: Received Resolution Reply 10.160.17.151/32 is at proto 10.160.17.151 nbma 192.168.77.65 (code 0)
Apr 21 17:38:35 opennhrp-script.py[2919]: Running script with arguments: [‘/etc/opennhrp/opennhrp-script.py’, ‘peer-up’], environment: environ({‘NHRP_TYPE’: ‘cached’, ‘NHRP_SRCADDR’: ‘10.160.17.141’, ‘NHRP_SRCNBMA’: ‘192.168.77.64’, ‘NHRP_DESTADDR’: ‘10.160.17.151’, ‘NHRP_DESTPREFIX’: ‘32’, ‘NHRP_DESTNBMA’: ‘192.168.77.65’, ‘NHRP_INTERFACE’: ‘tun10’, ‘NHRP_GRE_KEY’: ‘17128’, ‘LC_CTYPE’: ‘C.UTF-8’})
Apr 21 17:38:35 opennhrp-script.py[2919]: Peer UP event for spoke using IKE profile
Apr 21 17:38:35 opennhrp-script.py[2919]: NBMA addresses: local 192.168.77.64, remote 192.168.77.65
Apr 21 17:38:35 opennhrp[1779]: [10.160.17.151] Peer up script: success
Apr 21 17:38:35 opennhrp[1779]: NL-ARP(tun10) 10.160.17.151 is-at 192.168.77.65
At this point I see all the resolution requests and replies on the spoke side.

Steps to reproduce:
Modify any configuration on spoke-side with commit and optionally revert back.
(for example nhrp hold-time)
tear down connection between S-S|S-H peers or modify HUB config (works sometimes).
The Spoke stops sending resolution requests, while registration requests to hub works well. That can be seen in a traffic dump taken on the spoke/hub.

NHRP logs from Spoke1 (in failed state):

vyos@S1:~$ monitor log nhrp
– Journal begins at Sat 2023-01-28 14:10:46 UTC. –
Apr 21 17:45:51 opennhrp[3402]: Received Resolution Request from proto src 10.160.17.151 to 10.160.17.141
Apr 21 17:45:51 opennhrp[3402]: Sending Resolution Reply 10.160.17.141/32 is-at (unspecified) (holdtime 300)
Apr 21 17:45:51 opennhrp[3402]: Sending packet 2, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma (unspecified))
Apr 21 17:45:51 opennhrp[3402]: Cannot send packet to tun10(9): Invalid argument
Apr 21 17:45:56 opennhrp[3402]: Sending packet 1, from: 10.160.17.141 (nbma (unspecified)), to: 10.160.17.151 (nbma (unspecified))
Apr 21 17:45:56 opennhrp[3402]: Cannot send packet to tun10(9): Invalid argument
Apr 21 17:45:56 opennhrp[3402]: Received Resolution Request from proto src 10.160.17.151 to 10.160.17.141
Apr 21 17:45:56 opennhrp[3402]: Sending Resolution Reply 10.160.17.141/32 is-at (unspecified) (holdtime 300)
Apr 21 17:45:56 opennhrp[3402]: Sending packet 2, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma (unspecified))
Apr 21 17:45:56 opennhrp[3402]: Cannot send packet to tun10(9): Invalid argument
Apr 21 17:46:01 opennhrp[3402]: Failed to resolve 10.160.17.151: timeout (65535)
Apr 21 17:46:01 opennhrp[3402]: Removing incomplete 10.160.17.151/32 dev tun10 used

Spoke 2 in the meanwhile gets response timeouts because of S1 keeps silent about resolution replies:

vyos@S2:~$ monitor log nhrp
– Journal begins at Sat 2023-01-28 14:10:46 UTC. –
Apr 21 17:49:40 opennhrp[1613]: NL-ARP(tun10) who-has 10.160.17.141
Apr 21 17:49:40 opennhrp[1613]: NL-ARP(tun10) 10.160.17.141 is-at 192.168.77.66
Apr 21 17:49:40 opennhrp[1613]: Adding incomplete 10.160.17.141/32 dev tun10
Apr 21 17:49:40 opennhrp[1613]: Sending Resolution Request to 10.160.17.141
Apr 21 17:49:40 opennhrp[1613]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:49:45 opennhrp[1613]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:49:50 opennhrp[1613]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:49:55 opennhrp[1613]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:50:00 opennhrp[1613]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:50:05 opennhrp[1613]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)
Apr 21 17:50:10 opennhrp[1613]: Failed to resolve 10.160.17.141: timeout (65535)
Apr 21 17:50:10 opennhrp[1613]: Removing incomplete 10.160.17.141/32 dev tun10 used

If we modify any nhrp configuration on S2 it behaves the same as S1 and stops sending NHRP resolution requests:

Apr 21 17:55:26 opennhrp[3654]: NL-ARP(tun10) who-has 10.160.17.141
Apr 21 17:55:26 opennhrp[3654]: NL-ARP(tun10) 10.160.17.141 is-at 192.168.77.66
Apr 21 17:55:26 opennhrp[3654]: Adding incomplete 10.160.17.141/32 dev tun10
Apr 21 17:55:26 opennhrp[3654]: Sending Resolution Request to 10.160.17.141
Apr 21 17:55:26 opennhrp[3654]: Sending packet 1, from: 10.160.17.151 (nbma (unspecified)), to: 10.160.17.141 (nbma (unspecified))
No real traffic here on a wire
Apr 21 17:55:26 opennhrp[3654]: Cannot send packet to tun10(9): Invalid argument
Apr 21 17:55:31 opennhrp[3654]: Sending packet 1, from: 10.160.17.151 (nbma (unspecified)), to: 10.160.17.141 (nbma (unspecified))
Apr 21 17:55:31 opennhrp[3654]: Cannot send packet to tun10(9): Invalid argument

Restarting failed spoke always helps.
I’ve tested this issue on a plenty of 1.4 rolling versions as the issue persists for quite a long time during image upgrades. Here are some versions for example:

1: 1.4-rolling-202304130846 - latest
2: 1.4-rolling-202211290318
3: 1.4-rolling-202301280924

Observations:
There is no matter if we turn on nhrp redirect/shortcut on the hub/spoke. The spoke behaves the same in terms of nhrp resolution packets.
In the production env I use static S-S map entries which solves the problem, but scalability becomes a nightmare.

a.hajiyev · April 22, 2023, 11:53am

Hi @Jackwmtr,
According in your issue I have simulated DMVPN in my lab.
I have used VyOS 1.4-rolling-202304130846 and done configurations according VyOS 1.4.x sagitta manual DMVPN — VyOS 1.4.x (sagitta) documentation
Also used routing protocol (I have done simple bgp configuration)
Tested it and “DMVPN” is working without any problem.
I have changed some values in Spoke site (for example nhrp hold-time) observed no issue.

Here is lab details:

Hub Site:
dum1: 192.168.1.1/24
eth0: 192.168.122.11/24
eth1: 192.168.11.1/24
tun1: 10.160.17.131/25

Spoke-01 Site:
dum1: 192.168.2.1/24
eth0: dhcp
tun1: 10.160.17.141/25

Spoke-02 Site:
dum1: 192.168.3.1/24
eth0: dhcp
tun1: 10.160.17.151/25

Here is configuration:
============= HUB site configuration ===========================

set interfaces dummy dum1 address ‘192.168.1.1/24’
set interfaces ethernet eth0 address ‘192.168.122.11/24’
set interfaces ethernet eth1 address ‘192.168.11.1/24’

set interfaces tunnel tun1 address ‘10.160.17.131/25’
set interfaces tunnel tun1 description ‘DMVPN_HUB_interface’
set interfaces tunnel tun1 enable-multicast
set interfaces tunnel tun1 encapsulation ‘gre’
set interfaces tunnel tun1 parameters ip key ‘1’
set interfaces tunnel tun1 source-address ‘192.168.122.11’

set protocols nhrp tunnel tun1 cisco-authentication ‘secret’
set protocols nhrp tunnel tun1 holding-time ‘300’
set protocols nhrp tunnel tun1 multicast ‘dynamic’
set protocols nhrp tunnel tun1 redirect
set protocols nhrp tunnel tun1 shortcut

set vpn ipsec esp-group ESP-HUB lifetime ‘1800’
set vpn ipsec esp-group ESP-HUB mode ‘transport’
set vpn ipsec esp-group ESP-HUB pfs ‘dh-group2’
set vpn ipsec esp-group ESP-HUB proposal 1 encryption ‘aes256’
set vpn ipsec esp-group ESP-HUB proposal 1 hash ‘sha1’
set vpn ipsec esp-group ESP-HUB proposal 2 encryption ‘3des’
set vpn ipsec esp-group ESP-HUB proposal 2 hash ‘md5’

set vpn ipsec ike-group IKE-HUB key-exchange ‘ikev1’
set vpn ipsec ike-group IKE-HUB lifetime ‘3600’
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group ‘2’
set vpn ipsec ike-group IKE-HUB proposal 1 encryption ‘aes256’
set vpn ipsec ike-group IKE-HUB proposal 1 hash ‘sha1’
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group ‘2’
set vpn ipsec ike-group IKE-HUB proposal 2 encryption ‘aes128’
set vpn ipsec ike-group IKE-HUB proposal 2 hash ‘sha1’

set vpn ipsec interface ‘eth0’

set vpn ipsec profile NHRPVPN authentication mode ‘pre-shared-secret’
set vpn ipsec profile NHRPVPN authentication pre-shared-secret ‘secret’
set vpn ipsec profile NHRPVPN bind tunnel ‘tun1’
set vpn ipsec profile NHRPVPN esp-group ‘ESP-HUB’
set vpn ipsec profile NHRPVPN ike-group ‘IKE-HUB’

set protocols bgp address-family ipv4-unicast network 192.168.1.0/24
set protocols bgp address-family ipv4-unicast network 192.168.11.0/24
set protocols bgp neighbor 10.160.17.141 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp neighbor 10.160.17.141 ebgp-multihop ‘2’
set protocols bgp neighbor 10.160.17.141 remote-as ‘65522’
set protocols bgp neighbor 10.160.17.141 update-source ‘10.160.17.131’
set protocols bgp neighbor 10.160.17.151 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp neighbor 10.160.17.151 ebgp-multihop ‘2’
set protocols bgp neighbor 10.160.17.151 remote-as ‘65523’
set protocols bgp neighbor 10.160.17.151 update-source ‘10.160.17.131’
set protocols bgp parameters router-id ‘10.160.17.131’
set protocols bgp system-as ‘65521’

=============== Spoke-01 site configuration =======================

set interfaces dummy dum1 address ‘192.168.2.1/24’
set interfaces ethernet eth0 address ‘dhcp’

set interfaces tunnel tun1 address ‘10.160.17.141/25’
set interfaces tunnel tun1 enable-multicast
set interfaces tunnel tun1 encapsulation ‘gre’
set interfaces tunnel tun1 parameters ip key ‘1’
set interfaces tunnel tun1 source-address ‘0.0.0.0’

set protocols nhrp tunnel tun1 cisco-authentication ‘secret’
set protocols nhrp tunnel tun1 holding-time ‘300’
set protocols nhrp tunnel tun1 map 10.160.17.131/25 nbma-address ‘192.168.122.11’
set protocols nhrp tunnel tun1 map 10.160.17.131/25 register
set protocols nhrp tunnel tun1 multicast ‘nhs’
set protocols nhrp tunnel tun1 redirect
set protocols nhrp tunnel tun1 shortcut

set vpn ipsec esp-group ESP-HUB lifetime ‘1800’
set vpn ipsec esp-group ESP-HUB mode ‘transport’
set vpn ipsec esp-group ESP-HUB pfs ‘dh-group2’
set vpn ipsec esp-group ESP-HUB proposal 1 encryption ‘aes256’
set vpn ipsec esp-group ESP-HUB proposal 1 hash ‘sha1’
set vpn ipsec esp-group ESP-HUB proposal 2 encryption ‘3des’
set vpn ipsec esp-group ESP-HUB proposal 2 hash ‘md5’

set vpn ipsec ike-group IKE-HUB close-action ‘none’
set vpn ipsec ike-group IKE-HUB key-exchange ‘ikev1’
set vpn ipsec ike-group IKE-HUB lifetime ‘3600’
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group ‘2’
set vpn ipsec ike-group IKE-HUB proposal 1 encryption ‘aes256’
set vpn ipsec ike-group IKE-HUB proposal 1 hash ‘sha1’
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group ‘2’
set vpn ipsec ike-group IKE-HUB proposal 2 encryption ‘aes128’
set vpn ipsec ike-group IKE-HUB proposal 2 hash ‘sha1’

set vpn ipsec interface ‘eth0’

set vpn ipsec profile NHRPVPN authentication mode ‘pre-shared-secret’
set vpn ipsec profile NHRPVPN authentication pre-shared-secret ‘secret’
set vpn ipsec profile NHRPVPN bind tunnel ‘tun1’
set vpn ipsec profile NHRPVPN esp-group ‘ESP-HUB’
set vpn ipsec profile NHRPVPN ike-group ‘IKE-HUB’

set protocols bgp address-family ipv4-unicast network 192.168.2.0/24
set protocols bgp neighbor 10.160.17.131 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp neighbor 10.160.17.131 ebgp-multihop ‘2’
set protocols bgp neighbor 10.160.17.131 remote-as ‘65521’
set protocols bgp neighbor 10.160.17.131 update-source ‘10.160.17.141’
set protocols bgp parameters router-id ‘10.160.17.141’
set protocols bgp system-as ‘65522’

===================== Spoke-01 site configuration ==========================

set interfaces dummy dum1 address ‘192.168.3.1/24’
set interfaces ethernet eth0 address ‘dhcp’

set interfaces tunnel tun1 address ‘10.160.17.151/25’
set interfaces tunnel tun1 enable-multicast
set interfaces tunnel tun1 encapsulation ‘gre’
set interfaces tunnel tun1 parameters ip key ‘1’
set interfaces tunnel tun1 source-address ‘0.0.0.0’

set protocols nhrp tunnel tun1 cisco-authentication ‘secret’
set protocols nhrp tunnel tun1 holding-time ‘150’
set protocols nhrp tunnel tun1 map 10.160.17.131/25 nbma-address ‘192.168.122.11’
set protocols nhrp tunnel tun1 map 10.160.17.131/25 register
set protocols nhrp tunnel tun1 multicast ‘nhs’
set protocols nhrp tunnel tun1 redirect
set protocols nhrp tunnel tun1 shortcut

set vpn ipsec esp-group ESP-HUB lifetime ‘1800’
set vpn ipsec esp-group ESP-HUB mode ‘transport’
set vpn ipsec esp-group ESP-HUB pfs ‘dh-group2’
set vpn ipsec esp-group ESP-HUB proposal 1 encryption ‘aes256’
set vpn ipsec esp-group ESP-HUB proposal 1 hash ‘sha1’
set vpn ipsec esp-group ESP-HUB proposal 2 encryption ‘3des’
set vpn ipsec esp-group ESP-HUB proposal 2 hash ‘md5’

set vpn ipsec ike-group IKE-HUB close-action ‘none’
set vpn ipsec ike-group IKE-HUB key-exchange ‘ikev1’
set vpn ipsec ike-group IKE-HUB lifetime ‘3600’
set vpn ipsec ike-group IKE-HUB proposal 1 dh-group ‘2’
set vpn ipsec ike-group IKE-HUB proposal 1 encryption ‘aes256’
set vpn ipsec ike-group IKE-HUB proposal 1 hash ‘sha1’
set vpn ipsec ike-group IKE-HUB proposal 2 dh-group ‘2’
set vpn ipsec ike-group IKE-HUB proposal 2 encryption ‘aes128’
set vpn ipsec ike-group IKE-HUB proposal 2 hash ‘sha1’

set vpn ipsec interface ‘eth0’

set vpn ipsec profile NHRPVPN authentication mode ‘pre-shared-secret’
set vpn ipsec profile NHRPVPN authentication pre-shared-secret ‘secret’
set vpn ipsec profile NHRPVPN bind tunnel ‘tun1’
set vpn ipsec profile NHRPVPN esp-group ‘ESP-HUB’
set vpn ipsec profile NHRPVPN ike-group ‘IKE-HUB’

set protocols bgp address-family ipv4-unicast network 192.168.3.0/24
set protocols bgp neighbor 10.160.17.131 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp neighbor 10.160.17.131 ebgp-multihop ‘2’
set protocols bgp neighbor 10.160.17.131 remote-as ‘65521’
set protocols bgp neighbor 10.160.17.131 update-source ‘10.160.17.151’
set protocols bgp parameters router-id ‘10.160.17.151’
set protocols bgp system-as ‘65523’

Jackwmtr · April 22, 2023, 12:13pm

Thanks for your effort!
Could you post “show nhrp tun” output after spoke cfg change? I suspect that nhrp S2S resolution still breaks, just won’t affect S2H BGP connectivity as S2H traffic still persists.
Thanks!

a.hajiyev · April 22, 2023, 12:16pm

vyos@VPN-HUB# run show nhrp tunnel
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address Expires-In

tun1 local 10.160.17.255/32 10.160.17.131 up
tun1 local 10.160.17.131/32 up
tun1 dynamic 10.160.17.141/32 up 192.168.122.53 1:07
tun1 dynamic 10.160.17.151/32 up 192.168.122.68 1:42

a.hajiyev · April 22, 2023, 12:16pm

vyos@Spoke-01# run show nhrp tunnel
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address

tun1 local 10.160.17.255/32 10.160.17.141 up
tun1 local 10.160.17.141/32 up
tun1 static 10.160.17.131/25 up 192.168.122.11

@Jackwmtr if you need any other additional command outputs please write.

Jackwmtr · April 22, 2023, 12:30pm

Sounds weird
I’ll try your config and get back.

Jackwmtr · April 22, 2023, 1:37pm

I can’t see your “cached” entries (see below:) Because that’s the one I am having troubles with. It wasn’t an issue for you because you have only S2H connections and not the direct S2S. (BGP) The main issue is with that “cached” enries which utilis the “nhrf request packets” rather that “register requests” for S2H. THose packet’s are missing after opennhrp reload.
To add: If you do “set nhrp…” commit the S2S traffic still persist but it starts to flow via HUB (with additional hop, just the same as DMVPN ph1), because lack of the ddirect “caches” entry.

vyos@vyos:~$ show nhrp tu
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address Expires-In

tun1 local 10.160.17.255/32 10.160.17.161 up
tun1 local 10.160.17.161/32 up
tun1 cached 10.160.17.141/32 used up 192.168.77.64 4:55
tun1 static 10.160.17.131/25 up 192.168.77.66

a.hajiyev · April 22, 2023, 1:55pm

@Jackwmtr have you tried my config?

For redusing memory consumption on big NBMA subnets you can use:
set protocols nhrp tunnel tun1 non-caching

Jackwmtr · April 22, 2023, 1:58pm

My last test with brand new peer added:

No spoke resolution entry at the start, sinco no S2S traffic initiated yet:

show nhrp tu
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address

tun1 local 10.160.17.255/32 10.160.17.161 up
tun1 local 10.160.17.161/32 up
tun1 static 10.160.17.131/25 up 192.168.77.66
Let’s do some S2S traffic:
vyos@vyos:~$ ping 10.160.17.141
PING 10.160.17.141 (10.160.17.141) 56(84) bytes of data.
64 bytes from 10.160.17.141: icmp_seq=1 ttl=63 time=7.63 ms
64 bytes from 10.160.17.141: icmp_seq=2 ttl=64 time=4.01 ms
64 bytes from 10.160.17.141: icmp_seq=3 ttl=64 time=3.09 ms
^C
— 10.160.17.141 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 3.091/4.910/7.627/1.957 ms

See that new “cached” entry added? It tells to initiate direct S2S tunnel if we’d go to .141 rather than going via HUB:

vyos@vyos:~$ show nhrp tu
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address Expires-In

tun1 local 10.160.17.255/32 10.160.17.161 up
tun1 local 10.160.17.161/32 up
tun1 cached 10.160.17.141/32 used up 192.168.77.64 4:55
tun1 static 10.160.17.131/25 up 192.168.77.66

Change config/restart opennhrp process on the Spoke:

vyos@vyos# set proto nhrp tunnel tun1 holding-time 500
[edit]

ICMP still here, but now it goes through the HUB rather than directly:

vyos@vyos# ping 10.160.17.141
PING 10.160.17.141 (10.160.17.141) 56(84) bytes of data.
64 bytes from 10.160.17.141: icmp_seq=1 ttl=64 time=2.50 ms
64 bytes from 10.160.17.141: icmp_seq=2 ttl=64 time=1.97 ms
^C
— 10.160.17.141 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.973/2.238/2.504/0.265 ms

vyos@vyos# run traceroute 10.160.17.141
traceroute to 10.160.17.141 (10.160.17.141), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 1.530 ms 1.487 ms 1.357 ms
2 10.160.17.141 (10.160.17.141) 2.489 ms 3.510 ms 3.795 ms
new hop was added
Because the NHRP requests are now broken - we got this “incomplete” entries in NHRP neighbors:
vyos@vyos# run show nhrp tu
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address

tun1 local 10.160.17.255/32 10.160.17.161 up
tun1 local 10.160.17.161/32 up
tun1 incomplete 10.160.17.141/32 used
tun1 static 10.160.17.131/25 used up 192.168.77.66
[edit]
vyos@vyos#
That makes spoke to behave as a DMVPN phase 1.

Jackwmtr · April 22, 2023, 2:01pm

I tried your config and it works because it doesn’t need direct S2S communication.
The network scheme that I am using unfortunately require such a communication.

BTW you can test the issue by creating 1-hop BGP session between S2S and modify reload nhrp. You’ll probably see that S2S BGP sessions are not able to be established anymore because of an increased hopcount.

Jackwmtr · April 22, 2023, 2:52pm

Can you ping/traceroute .151 from .141 before and after nhrp commit on spoke and post “show nhrp tu” here?

a.hajiyev · April 22, 2023, 3:45pm

vyos@Spoke-01# run traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 1.365 ms 5.331 ms 5.272 ms
2 10.160.17.151 (10.160.17.151) 8.503 ms 8.430 ms 8.389 ms
[edit]
vyos@Spoke-01# set protocols nhrp tunnel tun1 holding-time ‘200’
[edit]
vyos@Spoke-01# commit
[edit]
vyos@Spoke-01#
[edit]
vyos@Spoke-01# run traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 2.587 ms 2.630 ms 2.481 ms
2 10.160.17.151 (10.160.17.151) 3.258 ms * *
[edit]
vyos@Spoke-01# run traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 1.229 ms 1.111 ms 1.959 ms
2 10.160.17.151 (10.160.17.151) 1.331 ms 5.030 ms 4.639 ms
[edit]
vyos@Spoke-01# run sh interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description

dum1 192.168.2.1/24 u/u
eth0 192.168.122.53/24 u/u
tun1 10.160.17.141/25 u/u

vyos@Spoke-01# run sh nhrp tunnel
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address

tun1 local 10.160.17.255/32 10.160.17.141 up
tun1 local 10.160.17.141/32 up
tun1 static 10.160.17.131/25 up 192.168.122.11

a.hajiyev · April 22, 2023, 3:51pm

vyos@Spoke-02# run traceroute 10.160.17.141
traceroute to 10.160.17.141 (10.160.17.141), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 3.077 ms 4.738 ms 4.739 ms
2 10.160.17.141 (10.160.17.141) 5.194 ms 5.146 ms 5.134 ms
[edit]
vyos@Spoke-02#
[edit]
vyos@Spoke-02# set protocols nhrp tunnel tun1 holding-time ‘250’
[edit]
vyos@Spoke-02# commit
[edit]
vyos@Spoke-02# run traceroute 10.160.17.141
traceroute to 10.160.17.141 (10.160.17.141), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 2.016 ms 1.796 ms 1.734 ms
2 10.160.17.141 (10.160.17.141) 4.950 ms 4.899 ms 4.886 ms

vyos@Spoke-02#
[edit]
vyos@Spoke-02# run sh nhrp tunnel
Status: ok
Interface Type Protocol-Address Alias-Address Flags NBMA-Address

tun1 local 10.160.17.255/32 10.160.17.151 up
tun1 local 10.160.17.151/32 up
tun1 static 10.160.17.131/25 up 192.168.122.11

vyos@Spoke-02# run sh interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description

dum1 192.168.3.1/24 u/u
eth0 192.168.122.68/24 u/u

tun1 10.160.17.151/25 u/u

Jackwmtr · April 22, 2023, 6:01pm

So, no spoke 2 spoke tunnels/dynamic entries whatsoever
Can you post the logs from spoke? (restart S1 S2 and ping spoke2spoke)

There should be an entry in the logs about asking a HUB to resolve the spoke’s address into NBMA like this:

Apr 21 14:53:34 opennhrp[3868]: **-ARP(tun10) who-has 10.160.17.141
Apr 21 14:53:34 opennhrp[3868]: NL-ARP(tun10) 10.160.17.141 is-at 192.168.77.66

^^Here we see that we don’t have an entry for .141 yet

Apr 21 14:53:34 opennhrp[3868]: Adding incomplete 10.160.17.141/32 dev tun10
Apr 21 14:53:34 opennhrp[3868]: Sending Resolution Request to 10.160.17.141

**^^Asking HUB about .141’s NBMA address and sending a resolution packet.

> Apr 21 14:53:34 opennhrp[3868]: Sending packet 1, from: 10.160.17.151 (nbma 192.168.77.65), to: 10.160.17.141 (nbma 192.168.77.66)

^^At this point you should see a NHRP resolution request towards the hub in traffic capture.(good_cap&bad_cap files attached) In a “bad case” there is an entry like this in the log with “nbma unspecified” but no real NHRP packets from spoke. **
In your last output, I can’t see the creation of dynamic tunnel at all. Given that your ‘sh nhrp tun’ is as if there were no attempts.

Apr 21 14:53:34 opennhrp[3868]: Traffic Indication from proto src 10.160.17.131; about packet to 10.160.17.141
^^This happens because HUB has configured ‘ip nhrp redirect’ for phase 3. It’s not mandatory for sending and receiving NHRP resolution requests/creating dynamic entries.
Apr 21 14:53:34 opennhrp[3868]: Received Resolution Request from proto src 10.160.17.141 to 10.160.17.151
Apr 21 14:53:34 opennhrp[3868]: Sending Resolution Reply 10.160.17.151/32 is-at 192.168.77.65 (holdtime 300)
Apr 21 14:53:34 opennhrp[3868]: Sending packet 2, from: 10.160.17.141 (nbma 192.168.77.64), to: 10.160.17.151 (nbma 192.168.77.66)
Apr 21 14:53:34 opennhrp[3868]: Received Resolution Reply 10.160.17.141/32 is at proto 10.160.17.141 nbma 192.168.77.64 (code 0)
Apr 21 14:53:34 opennhrp-script.py[4764]: Running script with arguments: [‘/etc/opennhrp/opennhrp-script.py’, ‘peer-up’], environment: environ({‘NHRP_TYPE’: ‘cached’, ‘NHRP_SRCADDR’: ‘10.160.17.151’, ‘NHRP_SRCNBMA’: ‘192.168.77.65’, ‘NHRP_DESTADDR’: ‘10.160.17.141’, ‘NHRP_DESTPREFIX’: ‘32’, ‘NHRP_DESTNBMA’: ‘192.168.77.64’, ‘NHRP_INTERFACE’: ‘tun10’, ‘NHRP_GRE_KEY’: ‘17128’, ‘LC_CTYPE’: ‘C.UTF-8’})
Apr 21 14:53:34 opennhrp-script.py[4764]: Peer UP event for spoke using IKE profile
Apr 21 14:53:34 opennhrp-script.py[4764]: NBMA addresses: local 192.168.77.65, remote 192.168.77.64
Apr 21 14:53:34 opennhrp[3868]: [10.160.17.141] Peer up script: success

Again, the issue is with creation of dynamic tunnels only. It suddenly stops working after a reload.

Jackwmtr · April 22, 2023, 8:35pm

JFI: Checked with stable 1.3.2. Observing the same behavior.

Jackwmtr · April 23, 2023, 6:53pm

@a.hajiyev with ver 1.2.9 dynamic S2S entries work as expected!
After opennhrp reload spoke keeps sending resolution requests about other spokes to hub.
It looks like some kind of regression bug started from 1.3

Jackwmtr · April 24, 2023, 11:28am

I tested further and it turned out the behavior is quite the same on ver 1.2.9. Spoke stops sending resolution requests after configuration modification
Sorry for the misleading info!
Actually it resolves after some time, but not every time. Needs more testing for 1.2.9 though.

a.hajiyev · April 25, 2023, 4:22am

Hi @Jackwmtr
Sorry for the late answer and thank you for testing all this.
For further investigation (or creating a bug report)
Could you please share your configurations (S2S and DMVPN together)
Thank you

Jackwmtr · April 28, 2023, 10:18am

Hi @a.hajiyev and all!
Sorry for the late reply (I’ve been out of my lab hands on for a while). To keep things as simple as possible I’ve made it without involving DMVPN phase 3 (no nhrp redirect/shortcut), as I believe the issue is with NHRP DMVPN phase2 packets. If there is a need to add it - let me know.

HUB config:

vyos@HUB# run show conf comm
set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 hw-id ‘50:f5:6f:00:07:00’
set interfaces ethernet eth1 hw-id ‘50:f5:6f:00:07:01’
set interfaces ethernet eth2 hw-id ‘50:f5:6f:00:07:02’
set interfaces ethernet eth3 hw-id ‘50:f5:6f:00:07:03’
set interfaces loopback lo
set interfaces tunnel tun1 address ‘10.160.17.131/25’
set interfaces tunnel tun1 description ‘DMVPN_HUB_interface’
set interfaces tunnel tun1 encapsulation ‘gre’
set interfaces tunnel tun1 multicast ‘enable’
set interfaces tunnel tun1 parameters ip key ‘17128’
set interfaces tunnel tun1 source-address ‘192.168.77.66’
set protocols nhrp tunnel tun1 cisco-authentication ‘LPDMVPN’
set protocols nhrp tunnel tun1 holding-time ‘300’
set protocols nhrp tunnel tun1 multicast ‘dynamic’
set service ntp allow-client address ‘0.0.0.0/0’
set service ntp allow-client address ‘::/0’
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set service ssh
set system config-management commit-revisions ‘100’
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed ‘115200’
set system host-name ‘HUB’
set system login user vyos authentication encrypted-password ‘$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/’
set system login user vyos authentication plaintext-password ‘’
set system syslog global facility all level ‘info’
set system syslog global facility protocols level ‘debug’

Spoke config:

vyos@S1:~$ show conf comm
set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 hw-id ‘50:da:ab:00:08:00’
set interfaces ethernet eth1 hw-id ‘50:da:ab:00:08:01’
set interfaces ethernet eth2 hw-id ‘50:da:ab:00:08:02’
set interfaces ethernet eth3 hw-id ‘50:da:ab:00:08:03’
set interfaces loopback lo
set interfaces tunnel tun10 address ‘10.160.17.141/25’
set interfaces tunnel tun10 encapsulation ‘gre’
set interfaces tunnel tun10 multicast ‘enable’
set interfaces tunnel tun10 parameters ip key ‘17128’
set interfaces tunnel tun10 source-address ‘192.168.77.64’
set protocols nhrp tunnel tun10 cisco-authentication ‘LPDMVPN’
set protocols nhrp tunnel tun10 holding-time ‘300’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 nbma-address ‘192.168.77.66’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 register
set protocols nhrp tunnel tun10 multicast ‘nhs’
set service ntp allow-client address ‘0.0.0.0/0’
set service ntp allow-client address ‘::/0’
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set service ssh
set system config-management commit-revisions ‘100’
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed ‘115200’
set system host-name ‘S1’
set system login user vyos authentication encrypted-password ‘$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/’
set system login user vyos authentication plaintext-password ‘’
set system syslog global facility all level ‘info’
set system syslog global facility protocols level ‘debug’
vyos@S1:~$

Spoke 2 config

vyos@S2:~$ show conf comm
set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 hw-id ‘50:27:f1:00:07:00’
set interfaces ethernet eth1 hw-id ‘50:27:f1:00:07:01’
set interfaces ethernet eth2 hw-id ‘50:27:f1:00:07:02’
set interfaces ethernet eth3 hw-id ‘50:27:f1:00:07:03’
set interfaces loopback lo
set interfaces tunnel tun10 address ‘10.160.17.151/25’
set interfaces tunnel tun10 encapsulation ‘gre’
set interfaces tunnel tun10 multicast ‘enable’
set interfaces tunnel tun10 parameters ip key ‘17128’
set interfaces tunnel tun10 source-address ‘192.168.77.65’
set protocols nhrp tunnel tun10 cisco-authentication ‘LPDMVPN’
set protocols nhrp tunnel tun10 holding-time ‘1234’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 nbma-address ‘192.168.77.66’
set protocols nhrp tunnel tun10 map 10.160.17.131/25 register
set protocols nhrp tunnel tun10 multicast ‘nhs’
set service ntp allow-client address ‘0.0.0.0/0’
set service ntp allow-client address ‘::/0’
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set service ssh
set system config-management commit-revisions ‘100’
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed ‘115200’
set system host-name ‘S2’
set system login user vyos authentication encrypted-password ‘$6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/’
set system login user vyos authentication plaintext-password ‘’
set system syslog global facility all level ‘info’
set system syslog global facility protocols level ‘debug’
vyos@S2:~$

Test methodology:

All peers are registered on the hub (.141, .151):

NHRP HUB registrations:
vyos@HUB:~$ show nhrp tu
Status: ok

Interface: tun1
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.131
Flags: up

Interface: tun1
Type: local
Protocol-Address: 10.160.17.131/32
Flags: up

Interface: tun1
Type: dynamic
Protocol-Address: 10.160.17.141/32
NBMA-Address: 192.168.77.64
Flags: used up
Expires-In: 4:37

Interface: tun1
Type: dynamic
Protocol-Address: 10.160.17.151/32
NBMA-Address: 192.168.77.65
Flags: used up
Expires-In: 19:48

vyos@HUB:~$

Tracing the route from S1 to S2:
vyos@S1:~$ traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 4.346 ms 1.145 ms 1.060 ms
2 10.160.17.151 (10.160.17.151) 8.335 ms 8.690 ms 8.075 ms
^^ at first resolution done via the HUB asking the HUB hor NBMA address of .151 to create dynamic tunnel:
vyos@S1:~$ traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.151 (10.160.17.151) 2.856 ms 2.952 ms *
^^ the second trace goes straight to S2 as we resolved NBMA of .151 and it’s in the cache
you can see it from NHRP entries below:

vyos@S1:~$ show nhrp tu
Status: ok

Interface: tun10
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.141
Flags: up

Interface: tun10
Type: local
Protocol-Address: 10.160.17.141/32
Flags: up

Interface: tun10
Type: cached
Protocol-Address: 10.160.17.151/32
NBMA-Address: 192.168.77.65
Flags: used up
Expires-In: 19:49

Interface: tun10
Type: static
Protocol-Address: 10.160.17.131/25
NBMA-Address: 192.168.77.66
Flags: up

Things are pretty nice so far. Let’s reload NHRP daemon on spoke1:

vyos@S1# set proto nhrp tunnel tun10 holding-time 1234
[edit]
vyos@S1# commit

NHRP caches are flushed after reload (no .151 entry):

vyos@S1:~$ show nhrp tu
Status: ok

Interface: tun10
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.141
Flags: up

Interface: tun10
Type: local
Protocol-Address: 10.160.17.141/32
Flags: up

Interface: tun10
Type: static
Protocol-Address: 10.160.17.131/25
NBMA-Address: 192.168.77.66
Flags: used up

Ping S1 to H OK:
vyos@S1:~$ ping 10.160.17.131
PING 10.160.17.131 (10.160.17.131) 56(84) bytes of data.
64 bytes from 10.160.17.131: icmp_seq=1 ttl=64 time=1.25 ms
^C
— 10.160.17.131 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.252/1.252/1.252/0.000 ms
Ping S1 to S2 OK:
vyos@S1:~$ ping 10.160.17.151
PING 10.160.17.151 (10.160.17.151) 56(84) bytes of data.
64 bytes from 10.160.17.151: icmp_seq=1 ttl=64 time=2.80 ms
64 bytes from 10.160.17.151: icmp_seq=2 ttl=64 time=2.42 ms
^C
— 10.160.17.151 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.417/2.609/2.801/0.192 ms
But trace rom S1 to S2 shows that traffic still goes through a HUB:
vyos@S1:~$ traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 2.336 ms 1.278 ms 1.320 ms
2 10.160.17.151 (10.160.17.151) 3.462 ms 2.629 ms 2.030 ms
vyos@S1:~$ traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 2.898 ms 3.387 ms 1.393 ms
2 10.160.17.151 (10.160.17.151) 3.327 ms * *
vyos@S1:~$ traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 1.082 ms 0.678 ms *
2 10.160.17.151 (10.160.17.151) 2.059 ms * *
vyos@S1:~$ traceroute 10.160.17.151
traceroute to 10.160.17.151 (10.160.17.151), 30 hops max, 60 byte packets
1 10.160.17.131 (10.160.17.131) 1.011 ms * *
2 10.160.17.151 (10.160.17.151) 1.577 ms * *

because it’s unable to resolve spoke this time:

vyos@S1:~$ show nhrp tu
Status: ok

Interface: tun10
Type: local
Protocol-Address: 10.160.17.255/32
Alias-Address: 10.160.17.141
Flags: up

Interface: tun10
Type: local
Protocol-Address: 10.160.17.141/32
Flags: up

Interface: tun10
Type: incomplete
Protocol-Address: 10.160.17.151/32
Flags: used

Interface: tun10
Type: static
Protocol-Address: 10.160.17.131/25
NBMA-Address: 192.168.77.66
Flags: up

The question is why reloading nhrp daemon stops spoke from sending resolution replies, even though the attempts entries are in the nhrp log. (logs are in previous messages and a packet captures too)

Guys, your help in nailing this problem will be really appreciated!

Jackwmtr · May 29, 2023, 10:12am

Hi! Any news on issue so far? Should I create a bug report?