VyOS rolling + Full BGP + ssh ram usage

Hi ALL

I’m new to VyOS community.
These days I installed and set up vyos as DC router with full bgp from my provider(first of, next days I’ll get connected with second provider). After changed from default route to full bgp, RAM usage increased (as expected). But what wasn’t expected, I can’t connect more than one ssh session to vyos, because every ssh session increase RAM usage by ~1.3GB. My VM with vyos have 4GB of RAM.
When I try to connect second session, in log it gives:
sshd[4399]: fatal: fork of unprivileged child failed
I think this is because of insufficient RAM for reservation, I am right?
It’s a normal thing that every ssh-session “eat” 1.3 GB of RAM?

My VyOS setup: (sh ver)
Version: VyOS 1.3-rolling-202006150117
Release Train: equuleus

Built by: [email protected]
Built on: Mon 15 Jun 2020 01:17 UTC
Build UUID: ab90d6b2-8f3c-4d8e-8022-50606d6ffa9b
Build Commit ID: 1dfa9a3c7cce72

Architecture: x86_64
Boot via: installed image
System type: VMware guest

Thanks in advance.

Hi @AlexB

This does not normal. Can you share you atop files? You can find these files in /var/log/atop/

Ok, put the last, today’s atop file, because today the problem appear.atop_20200619.log (761.1 KB)

Also, I want to mention that if I logout from ssh, and watch htop by console, ram get back. So, only after login via ssh, over-ram get used.

Hello @AlexB, can you provide a screenshot with VM configuration? I see really odd situation in atop log file.

Hi,
it’s a 4 x core, 4GB RAM, 8GB HDD and 5 network cards.
But strange in screenshot, because I “killed” vyos user. I’m using another user to manage router.Screenshot_20200620_101529

Very odd, I can’t reach this behavior in my test VyOS router on ESXi, even with deleted vyos user.

Hi,

today, after do a reboot of vyos for testing some stuff, problem has gone away, unexpected as come :slight_smile:

I think it’s corelated somehow with activating full-bgp peering with provider. Because after they changed bgp from “default route” to Full BGP, the problem appear.

Hi,

sorry, but the problem re-appears again :expressionless:

Hello,

We have the same issue and didnt fiind any solution to solve this.

We only restrict the ssh to be pointed to one ip and changed the port of the ssh but still getting brute force to it and even when i login true ssh me and my colegs the ram get crazy.

Yesterday, my login to ssh was ok, but when I created another login for my colleague with another username, and he entered via ssh, his sshd fork eat 1.2 gb ram. And finished with restarting vyos :frowning:
Today I expand vyos ram to 6GB, will see if this help in any way.
P.S: On my vyos ssh port is another from 22, and I did’n see any bot scans or stuff like this.
P.P.S: I’ll attach log from /var/log/messages for the short period then vyos become unresponsive, hope this help DEVs to found the reason of this problem.vyos.log (10.1 KB)

Hi,
@AlexB, @CristianD are you use RADIUS for authentication users?

Hi,
Nope. Using simple/system users
set system login user admin1 authentication encrypted-password ‘$6$pxxxx…xxxxx’
set system login user admin1 authentication plaintext-password ‘’
set system login user admin2 authentication encrypted-password ‘6.gxxxx…xxxx’
set system login user admin2 authentication plaintext-password ‘’

1 Like

Hi,
no any idea what to do, or what to investigate? Because problem persist (in a lighter manner, but anyway…). As I seen, ram eater is bgpd /w one of zebra process. Any action made with bgp stuff (down/up bgp peers, route-maps, etc) involve ram eating by bgp daemon.
And, in a final have to do a router reboot. :frowning:

Followup in Phabricator https://phabricator.vyos.net/T2678

1 Like