Memory leak in 1.3.2?

jvilafe · December 15, 2022, 8:56am

Hi team,
we are facing large amount of memory usage in my DMVPN hub’s. I have 16 Gb RAM hub’s, consuming up to 95% ram (up to 15Gb!!!). When I make a reboot and all services are running, we are facing only 300-400 Mb of consumed RAM, but this value raised up and a couple of days after reboot we have monitoring alert for high RAM usage.

For example, we have this two hub’s for dual DMVPN access for the same scenario. Only two spokes connected at this time…
This morning, both Hub’s faces 90% ram usage. I made a reboot on one node.

Reloaded node, after 1 hour of work:

free -hm
total used free shared buff/cache available
Mem: 15Gi 346Mi 14Gi 11Mi 317Mi 14Gi
Swap: 0B 0B

The other node, not reloaded, after 57 days of work.
free -hm
total used free shared buff/cache available
Mem: 15Gi 13Gi 737Mi 151Mi 1.1Gi 1.3Gi
Swap: 0B 0B 0B

I take a look to find what processes are consuming ram…but I can’t found it.

ps -o pid,user,%mem,command ax | sort -b -k3 -r
PID USER %MEM COMMAND
763 root 0.6 /lib/systemd/systemd-journald
770 root 0.3 /usr/bin/python3 -u /usr/libexec/vyos/services/vyos-configd
2587 root 0.1 /usr/lib/ipsec/charon --use-syslog
769 root 0.1 /usr/bin/python3 -u /usr/libexec/vyos/services/vyos-hostsd
3030 root 0.1 /usr/bin/python3 -Es /usr/sbin/tuned -l -P
38 root 0.0 [writeback]
76 root 0.0 [watchdogd]
173 root 0.0 [vmw_pvscsi_wq_0]
98687 jordi 0.0 -vbash
2429 zabbix 0.0 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
2428 zabbix 0.0 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
2427 zabbix 0.0 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
2426 zabbix 0.0 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
2425 zabbix 0.0 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
2430 zabbix 0.0 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
2996 root 0.0 /usr/sbin/sshd -f /run/sshd/sshd_config -D

May be memory leak???

top shift + m

top - 10:03:21 up 57 days, 19:14, 1 user, load average: 0.07, 0.03, 0.01
Tasks: 215 total, 1 running, 214 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 91.5/15995.9 [ ]
MiB Swap: 0.0/0.0 [ ]

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
763 root 20 0 143236 103056 35480 S 0.3 0.6 209:29.31 systemd-journal
770 root 20 0 307228 61504 16880 S 0.0 0.4 0:06.24 python3
769 root 20 0 120100 26780 13592 S 0.0 0.2 0:02.88 python3
3030 root 20 0 332424 24940 13040 S 0.0 0.2 20:35.79 tuned
2587 root 20 0 1207704 17552 12300 S 0.0 0.1 21:38.25 charon
1113 frr 20 0 1070452 16008 4324 S 0.0 0.1 29:50.44 zebra
1121 frr 20 0 197176 14412 6384 S 0.0 0.1 22:10.16 bgpd
827 root 20 0 48640 11708 10164 S 0.0 0.1 113:47.44 vmtoolsd
1 root 20 0 170680 10684 8080 S 0.0 0.1 109:58.24 systemd
823 root 20 0 46200 10640 9288 S 0.0 0.1 0:00.65 VGAuthService
75014 root 0 -20 10748 10148 4280 S 0.0 0.1 0:10.10 atop
3287 jordi 20 0 21160 8968 7640 S 0.0 0.1 0:00.41 systemd
98619 root 20 0 14680 8120 6996 S 0.0 0.0 0:00.04 sshd
920 root 20 0 19764 7476 6312 S 0.0 0.0 53:40.53 systemd-logind
1134 frr 20 0 34172 7324 3212 S 0.0 0.0 4:36.16 ospfd
3029 root 20 0 234060 7244 6224 S 0.0 0.0 0:00.25 polkitd
2996 root 20 0 13808 7108 6244 S 0.0 0.0 0:00.17 sshd
1137 frr 20 0 33340 6920 3172 S 0.0 0.0 4:38.52 ospf6d
1128 frr 20 0 32892 6488 3052 S 0.0 0.0 4:36.53 ripd
1140 frr 20 0 12724 6140 3264 S 0.0 0.0 4:47.01 isisd
778 root 20 0 21756 5996 3976 S 0.0 0.0 0:13.67 systemd-udevd
2427 zabbix 20 0 23856 5724 4400 S 0.0 0.0 24:17.82 zabbix_agentd
2429 zabbix 20 0 23888 5724 4400 S 0.0 0.0 24:11.23 zabbix_agentd
2428 zabbix 20 0 23856 5720 4400 S 0.0 0.0 23:54.33 zabbix_agentd
98687 jordi 20 0 8244 5276 3532 S 0.0 0.0 0:00.20 vbash
1145 frr 20 0 12256 5108 2840 S 0.0 0.0 4:43.60 ldpd
1131 frr 20 0 11448 5020 2848 S 0.0 0.0 4:37.07 ripngd
1149 frr 20 0 11124 4912 2936 S 0.0 0.0 4:38.22 staticd
98686 jordi 20 0 14680 4912 3784 S 0.0 0.0 0:00.12 sshd
6096 frr 20 0 11380 4844 2884 S 0.7 0.0 859:04.43 bfdd
1143 frr 20 0 10708 4816 3920 S 0.0 0.0 0:02.80 ldpd
1144 frr 20 0 10564 4780 3888 S 0.0 0.0 0:00.00 ldpd
824 root 20 0 8076 4768 1644 S 0.0 0.0 0:00.28 haveged
2226 root 20 0 223768 4764 3376 S 0.0 0.0 42:21.76 rsyslogd
2430 zabbix 20 0 23752 4572 3372 S 0.0 0.0 7:22.27 zabbix_agentd
924 message+ 20 0 8808 4200 3640 S 0.0 0.0 27:26.02 dbus-daemon
99717 jordi 20 0 11232 3928 3152 R 0.7 0.0 0:00.07 top
2426 zabbix 20 0 23752 3676 2592 S 0.0 0.0 22:53.43 zabbix_agentd
2547 ntp 20 0 76472 3200 2676 S 0.0 0.0 9:43.38 ntpd
2425 zabbix 20 0 23752 3144 2076 S 0.0 0.0 0:00.00 zabbix_agentd
1073 root 20 0 10240 3104 1908 S 0.0 0.0 75:35.87 watchfrr
922 root 20 0 8496 2696 2436 S 0.0 0.0 0:16.44 cron
3288 jordi 20 0 106084 2644 16 S 0.0 0.0 0:00.00 (sd-pam)
949 daemon 20 0 5480 2056 1880 S 0.0 0.0 0:00.13 atd
99724 root 20 0 5376 1980 1864 S 0.0 0.0 0:00.00 agetty
7018 root 20 0 4040 1848 1608 S 0.3 0.0 44:48.70 opennhrp
2586 root 20 0 5248 1788 1404 S 0.0 0.0 0:00.00 starter
3190 root 20 0 5604 1768 1652 S 0.0 0.0 0:00.01 agetty
930 root 0 -20 2276 1628 1520 S 0.0 0.0 10:10.24 atopacctd
1011 root 20 0 2284 1440 1332 S 0.0 0.0 0:00.00 netplugd
923 root 20 0 2316 756 692 S 0.0 0.0 0:00.04 acpid
2 root 20 0 0 0 0 S 0.0 0.0 0:04.61 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-events_highpri
8 root 0 -20 0 0 0 I 0.0 0.0 2:09.84 kworker/0:1H-events_highpri
9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
10 root 20 0 0 0 0 S 0.0 0.0 11:11.91 ksoftirqd/0
11 root 20 0 0 0 0 I 0.3 0.0 91:34.89 rcu_sched
12 root rt 0 0 0 0 S 0.0 0.0 0:28.32 migration/0
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
16 root rt 0 0 0 0 S 0.0 0.0 0:26.56 migration/1
17 root 20 0 0 0 0 S 0.0 0.0 4:32.64 ksoftirqd/1
19 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/1:0H-kblockd
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/2

jvilafe · December 15, 2022, 9:06am

Viacheslav · December 15, 2022, 9:16am

You can find atop filed in ls -la /var/log/atop/
Try to read some of them

atop -r /var/log/atop/atop.log_xxxxxx

Press “t” for the next time period

I would start with altogether disabling SNMP config / zabbix-agent (not native)
and check if memory is still increasing

jvilafe · December 15, 2022, 9:58am

Hi Viacheslav

thanks for your answer.

We have Zabix_agent running for monitoring from our internal platforms.

root@dmvpn:/var/log/atop# systemctl status zabbix-agent
â— zabbix-agent.service - Zabbix Agent
Loaded: loaded (/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-12-15 09:42:25 CET; 1h 8min ago
Main PID: 2126 (zabbix_agentd)
Tasks: 6 (limit: 19160)
Memory: 5.2M
CGroup: /system.slice/zabbix-agent.service
â”œâ”€2126 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
â”œâ”€2127 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
â”œâ”€2128 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
â”œâ”€2129 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
â”œâ”€2130 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
â””â”€2131 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

We don’t have snmp service active

:~$ show snmp community configured
SNMP service is not configured

systemctl status snmpd
â— snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
Loaded: loaded (/lib/systemd/system/snmpd.service; disabled; vendor preset: enabled)
Active: inactive (dead)

I can’t see any service or process consuming ram in atop.

last time period

may be one important detail…we have the same Vyos version and base config ( with Zabbix-agent) in all dmvpn environment, but only Hub’s are facing the issue…spokes doesn’t. All spokes are running on bare metal platforms, but Hub’s are running on VMWare VM.

jvilafe · December 15, 2022, 10:00am

ram usage in reloaded node after 2 work hours

free -mh
total used free shared buff/cache available
Mem: 15Gi 399Mi 14Gi 27Mi 343Mi 14Gi
Swap: 0B 0B 0B

Ram raised up from 346 to 400 Mb in one hour

up to 436 in two hours

free -mh
total used free shared buff/cache available
Mem: 15Gi 436Mi 14Gi 35Mi 356Mi 14Gi
Swap: 0B 0B 0B

Dmitry · December 15, 2022, 1:28pm

@jvilafe if you have active subscription, please open a ticket in support.vyos.io and share atop files with us.

Also, try to stop openvm tools and unload modules on the router.

sudo systemctl stop open-vm-tools.service
sudo modprobe -r vmw_balloon
sudo modprobe -r vmw_vsock_vmci_transport
sudo modprobe -r vsock

After this changes show an output of the command

show system memory

jvilafe · December 15, 2022, 3:41pm

Hi Dmitry,

thanks for your answer.

I don’t have subscription services :-/

I made the config stoping vmw-tools and unloading the modules

Before stopping:
root@dmvpn:/home/jordi# sh sys memory
Total: 15995
Free: 15380
Used: 615

One hour after:
root@dmvpn:/home/jordi# sh sys memory
Total: 15995
Free: 15347
Used: 648

It doesn’t works.

regards

jvilafe · December 15, 2022, 3:48pm

Please

anybody knows how to locate where the memory goes to? 14Gb of used memory must leave a trace!

regards

Dmitry · December 23, 2022, 8:08am

Hi @jvilafe , try to run sudo atop command and order by memory.

jvilafe · December 23, 2022, 4:20pm

Hi Dmitry,

atop -m shows only systemd-journaol with 1% MEM and no other process above 0%. 1.2G for Zebra, 1 G for charon and all other process with only K. There is no reason for 8 Gb consumed RAM

Korikaze · December 26, 2022, 9:44pm

Random thought, VyOS has a lot of tmpfs file systems. Are there any with a lot of used space?