Run out of Memory Crash


#1

Hi All,

Running 1.2.0-rolling+201904140337 on a AWS VM at the moment. It crashed for what I assume was a low memory situation similar to the issue I had here.

messages.txt (132.1 KB)

I have attached the log file which shows on May 3 the log file starts seeing a lot of activity.
May 4 was when the IPSEC tunnels went down, causing me to do a hard restart of the VM to bring it back up.


#2

Do you have a large bgp import or very may frr rules to load? It looks to me that it crashed when frr was triggered via vtysh.


#3

No currently just being used for IPSEC site to site VPN, its a hub in a hub and spoke topology.

No BGP or FRR rules.


#4

Hmm, that’s weird. I think DMVPN inserts then routes into frr (staticd) which may have caused the issue.
You can try ‘vpn ipsec options disable-route-autoinstall’ and see if it still crashes. You would have to set the required routes then manually, so it’s just for testing.


#5

hmmm is there a command which shows what routes are currently inserted into frr?

I can try disabling that, but will have to wait before its possible.

Also not sure if its related, but VPN tunnels are showing up with duplicates:

The ones grouped in red are duplicates


#6

I suspect there is definitely going something sideway. You can directly check in frr via ‘vtysh show running-config staticd’, it should be possible to get the route insertions also via snmp (not sure if it can send a trap).


#7

Running the command you provided, it just gave me a different shell with some different commands available, I ran the following:


#8

vtysh -c ‘show running-config staticd’ or just vtysh -c ‘show running-config’


#9

Got it, ok so heres the output of the command:
screen


#10

Hmm, weird, I was expecting routes being pushed from ipsec into it. can you reproduce the crash all the time? If so, how? I’d like to reproduce it locally in my environment.


#11

No I cant unfortunately. It seem to happen more consistently on an older release, like every 1-2 weeks. Seems more stable now. Interesting thing is, I have other instances running fewer IPSEC tunnels and they have never had this issue.

Ive setup a script for now to monitor and notify when memory runs low.