Hi All,
Running 1.2.0-rolling+201904140337 on a AWS VM at the moment. It crashed for what I assume was a low memory situation similar to the issue I had here.
messages.txt (132.1 KB)
I have attached the log file which shows on May 3 the log file starts seeing a lot of activity.
May 4 was when the IPSEC tunnels went down, causing me to do a hard restart of the VM to bring it back up.
Do you have a large bgp import or very may frr rules to load? It looks to me that it crashed when frr was triggered via vtysh.
No currently just being used for IPSEC site to site VPN, its a hub in a hub and spoke topology.
No BGP or FRR rules.
Hmm, that’s weird. I think DMVPN inserts then routes into frr (staticd) which may have caused the issue.
You can try ‘vpn ipsec options disable-route-autoinstall’ and see if it still crashes. You would have to set the required routes then manually, so it’s just for testing.
hmmm is there a command which shows what routes are currently inserted into frr?
I can try disabling that, but will have to wait before its possible.
Also not sure if its related, but VPN tunnels are showing up with duplicates:
The ones grouped in red are duplicates
I suspect there is definitely going something sideway. You can directly check in frr via ‘vtysh show running-config staticd’, it should be possible to get the route insertions also via snmp (not sure if it can send a trap).
Running the command you provided, it just gave me a different shell with some different commands available, I ran the following:
vtysh -c ‘show running-config staticd’ or just vtysh -c ‘show running-config’
Got it, ok so heres the output of the command:
Hmm, weird, I was expecting routes being pushed from ipsec into it. can you reproduce the crash all the time? If so, how? I’d like to reproduce it locally in my environment.
No I cant unfortunately. It seem to happen more consistently on an older release, like every 1-2 weeks. Seems more stable now. Interesting thing is, I have other instances running fewer IPSEC tunnels and they have never had this issue.
Ive setup a script for now to monitor and notify when memory runs low.
