SNMP service started and loaded but stops working until router reboot

Hi team,

I’m facing this issue on many platforms. All routers Vyos 1.3.1 or 1.3.2 version. SNMP set up with same config (more than units).

SNMP works fine but, sometimes, stops responding. I can see that snmp service is OK with show snmp or systemctl status snmpd. I tryed restarting snmp by vyos cli or by systemctl restart snmpd…doesn’t works.

The only way to restore snmp functionality is reloading the router with cli reboot command.

The config:

set service snmp community xxxxxxxx authorization 'ro'
set service snmp community xxxxxx network 'xxx.xxx.xxx.xxx/24'
set service snmp community xxxxxx network 'xxx.xxx.xxx.xxx/24'
set service snmp community xxxxxx network 'xxx.xxx.xxx.xxx/24'
set service snmp contact 'suport@xperientia.es'
set service snmp listen-address xxx.xxx.xxx.xxx
set service snmp location 'XPi204r-xxxxx'

nano snmpd.conf

### Autogenerated by snmp.py ###

# non configurable defaults
sysObjectID 1.3.6.1.4.1.44641
sysServices 14
master agentx
agentXPerms 0777 0777
pass .1.3.6.1.2.1.31.1.1.1.18 /opt/vyatta/sbin/if-mib-alias
smuxpeer .1.3.6.1.2.1.83
smuxpeer .1.3.6.1.2.1.157
smuxsocket localhost

# linkUp/Down configure the Event MIB tables to monitor
# the ifTable for network interfaces being taken up or down
# for making internal queries to retrieve any necessary information
iquerySecName vyos09605aa282a4dabb

# Modified from the default linkUpDownNotification
# to include more OIDs and poll more frequently
notificationEvent  linkUpTrap    linkUp   ifIndex ifDescr ifType ifAdminStatus ifOperStatus
notificationEvent  linkDownTrap  linkDown ifIndex ifDescr ifType ifAdminStatus ifOperStatus
monitor  -r 10 -e linkUpTrap   "Generate linkUp" ifOperStatus != 2
monitor  -r 10 -e linkDownTrap "Generate linkDown" ifOperStatus == 2

# Remove all old ifTable entries with the same ifName as newly appeared
# interface (with different ifIndex) - this is the case on e.g. ppp interfaces
interface_replace_old yes

########################
# configurable section #
########################

# Default system description is VyOS version
sysDescr VyOS Vyos Equuleus_1.3.1


# Listen
agentaddress unix:/run/snmpd.socket,udp:127.0.0.1:161,udp6:[::1]:161
# SNMP communities
rocommunity xxxxxx xxx.xxx.xxx.xxx/24
rocommunity xxxxxx xxx.xxx.xxx.xxx/24
rocommunity xxxxxx xxx.xxx.xxx.xxx/24

# system contact information
SysContact suport@xperientia.es

# system location information
SysLocation XPi204r-xxxxxxx



#
# SNMPv3 stuff goes here
#
# views

# access
#             context sec.model sec.level match  read    write  notif

# trap-target

# group

systemctl status snmpd

xxxx@XPi204r-xxxxxxx:~$ systemctl status snmpd
● snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
   Loaded: loaded (/lib/systemd/system/snmpd.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/snmpd.service.d
           └─override.conf
   Active: active (running) since Thu 2022-12-01 09:27:36 CET; 23min ago
  Process: 5164 ExecStartPre=/bin/mkdir -p /var/run/agentx (code=exited, status=0/SUCCESS)
 Main PID: 5165 (snmpd)
    Tasks: 1 (limit: 4915)
   Memory: 7.2M
   CGroup: /system.slice/snmpd.service
           └─5165 /usr/sbin/snmpd -LS0-5d -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -ipCidrRouteTable inetCidrRouteTable -f -p /run/snmpd.pid

Regards

Hi @jvilafe , can you find something in logs when SNMP stops responding?
What do you see in logs after restart snmp command?

Hi Dmitry,
I didn’t fins anything regarding snmpd in log ( journalctl) There is no entry for this services. I try to restart the snmp service from vyos cli or via systemctl restart snmpd with same results. No errors in journalctl, and snmp service started…but not responding to snmp gets. I verified that the get arrives to the router with monitor traffic filter udp…I can see the inbound packet, but there is no outbound response.

I can solve the issue restarting the router with reboot command, and snmp responds until the next issue.

Now, we have some units (so many units) unit responding to snmp gets but not showing anything in Vyos CLI command show snmp community xxxxx

xxxxx@XPi204r-xxxxxxx:~$ show snmp community xxxxxxx
Status of SNMP community xxxxxx on localhost
xxxxx@XPi204r-xxxxxxx:~$
xxxxx@XPi204r-xxxxxxx:~$
xxxxx@XPi204r-xxxxxxx:~$

but

xxxx@XPi204r-xxxxxx:~$ systemctl status snmpd
● snmpd.service - Simple Network Management Protocol (SNMP) Daemon.
Loaded: loaded (/lib/systemd/system/snmpd.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/snmpd.service.d
└─override.conf
Active: active (running) since Thu 2022-12-01 12:19:22 CET; 8h ago
Process: 15214 ExecStartPre=/bin/mkdir -p /var/run/agentx (code=exited, status=0/SUCCESS)
Main PID: 15215 (snmpd)
Tasks: 1 (limit: 4915)
Memory: 8.9M
CGroup: /system.slice/snmpd.service
└─15215 /usr/sbin/snmpd -LS0-5d -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -ipCidrRouteTable inetCidrRouteTable -f -p /run/snmpd.pid

Dec 01 12:19:22 XPi204r-xxxxxx systemd[1]: Starting Simple Network Management Protocol (SNMP) Daemon…
Dec 01 12:19:22 XPi204r-xxxxxx systemd[1]: Started Simple Network Management Protocol (SNMP) Daemon…

regards

Do you use any firewall or policy routing/wan-load-balancing/DNAT/etc?

I have Firewall from LAN to WAN, WAN to LAN and WAN to LOCAL, but I have specific rule to permit snmp (udp/161 and udp/162) from WAN to LOCAL.

set firewall name OUTSIDE_IN_LOCAL default-action ‘drop’
set firewall name OUTSIDE_IN_LOCAL description ‘Inbound Traffic from internet TO the firewall’
set firewall name OUTSIDE_IN_LOCAL rule 10 action ‘accept’
set firewall name OUTSIDE_IN_LOCAL rule 10 state established ‘enable’
set firewall name OUTSIDE_IN_LOCAL rule 10 state related ‘enable’
.
.
.
set firewall name OUTSIDE_IN_LOCAL rule 230 action ‘accept’
set firewall name OUTSIDE_IN_LOCAL rule 230 description ‘SNMP’
set firewall name OUTSIDE_IN_LOCAL rule 230 destination port ‘161’
set firewall name OUTSIDE_IN_LOCAL rule 230 protocol ‘udp’
set firewall name OUTSIDE_IN_LOCAL rule 230 state new ‘enable’
set firewall name OUTSIDE_IN_LOCAL rule 240 action ‘accept’
set firewall name OUTSIDE_IN_LOCAL rule 240 description ‘SNMP’
set firewall name OUTSIDE_IN_LOCAL rule 240 destination port ‘162’
set firewall name OUTSIDE_IN_LOCAL rule 240 protocol ‘udp’
set firewall name OUTSIDE_IN_LOCAL rule 240 state new ‘enable’
.
.
.
set interfaces pppoe pppoe0 firewall local name ‘OUTSIDE_IN_LOCAL’

I have outbound NAT (masquerade)

No policy routing or Wan load balancing. Only one WAN Internet access.

I think that the FW works fine, because snmp runs and responds with this config…until stops responding. Then, i made a reboot with the same config and snmp works fine again.

Just delete the firewall for several hours/days and check if you get the same issue.

I don’t have any router failing at this moment. All routers are production routers connected to internet, so, I don’t want to disable Inbound FW protection for all and wait until next issue…
I should wait until next issue and, then, disable FW inbound and check if this solves the issue.

regards