snmpd keeps crashing


#1

The snmpd service keeps crashing and this time I got an error in syslog.

KERNEL : [156341.707801] snmpd[2967]: segfault at 0 ip 00007f4e56bf3783 sp 00007fff8497ab70 error 4 in libnetsnmpmibs.so.30.0.2[7f4e56b8f000+15e000]


#2

Are you polling the device? If so, able you able to tie the crashes to specific events/polls? Which version of VyOS are you running? Is this a new install, or an existing where SNMP was not crashing initially? Any other log events at all around the same time?


#3

I am polling with observium every 5 mins. The version is 1.1.6. Yes this is a new install starting happening after I made it prod. The router does no nat/firewall, ospf just 1 static route and bgp. I can start the snmpd service and then as soon as observium polls it it will crash.


config

interfaces {
    ethernet eth0 {
        address xxxxxxxxxx/30
        address xxxxxxxx::1/64
        description link-to-fw-public
        duplex auto
        hw-id 00:50:56:98:55:36
        smp_affinity auto
        speed auto
    }
    ethernet eth1 {
        address xxxxxxxx/29
        description xxxxxx_fiberlink
        duplex auto
        hw-id 00:50:56:98:5e:77
        smp_affinity auto
        speed auto
    }
    ethernet eth2 {
        address xxxxxxxx/30
        description xxxxxx_fiberlink
        duplex auto
        hw-id 00:50:56:98:0f:3c
        smp_affinity auto
        speed auto
    }
    ethernet eth3 {
        address xxxxxxxxxxx/126
        description xxxxxx_ipv6
        duplex auto
        hw-id 00:50:56:98:19:05
        smp_affinity auto
        speed auto
    }
    ethernet eth4 {
        description spare
        disable
        duplex auto
        hw-id 00:50:56:98:2a:10
        smp_affinity auto
        speed auto
    }
    ethernet eth5 {
        description spare
        disable
        duplex auto
        hw-id 00:50:56:98:2b:ee
        smp_affinity auto
        speed auto
    }
    loopback lo {
    }
}
protocols {
    bgp xxxxx {
        address-family {
            ipv6-unicast {
                network xxxxxx:/32 {
                }
            }
        }
        neighbor xxxxxxx {
            remote-as xxxxx
            route-map {
            }
            soft-reconfiguration {
                inbound
            }
        }
        neighbor xxxxxx {
            remote-as xxxxx
            route-map {
            }
            soft-reconfiguration {
                inbound
            }
        }
        neighbor xxxxxxxx9 {
            address-family {
                ipv6-unicast {
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            remote-as 2711
            soft-reconfiguration {
                inbound
            }
        }
        network xxxxxx/20 {
        }
        network xxxxxxx0/20 {
        }
        network xxxxxxxxx/22 {
        }
        parameters {
            disable-network-import-check
        }
    }
    static {
        route xxxxx/19 {
            blackhole {
            }
        }
        route xxxxx/20 {
            blackhole {
            }
        }
        route xxxxx/20 {
            blackhole {
            }
        }
        route xxxxx/32 {
            blackhole {
            }
        }
        route xxxxxxx/19 {
            next-hop xxxxxx {
            }
        route6 xxxxx {
            next-hop xxxxx {
            }
        }
    }
}
service {
    snmp {
        community xxxx_public {
            authorization ro
        }
        listen-address xxxxxxx {
            port xx161
        }
    }
    ssh {
        listen-address xxxxxx
        listen-address xxxxxx
        listen-address xxxxxx
        port xxxx22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    host-name 301bgp-rt
    login {
        user vyos {
            authentication {
                encrypted-password  xxxxx
                plaintext-password ""
            }
            level admin
        }
    }
    name-server xxxxx
    name-server xxxxx
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    package {
        auto-sync 1
        repository community {
            components main
            distribution helium
            password ""
            url http://packages.vyos.net/vyos
            username ""
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
        host xxxxxx{
            facility all {
                level debug
            }
        }
    }
    time-zone US/Eastern
}

#4

Have you tried to remove polls one by one to see if you can find any relation?


#5

It is still a little soon but I disabled “ucd-mib” and I was able to poll 2x manual and will give it 15-20 mins to see if the auto poll crashes it.


#6

It crashed again over night wit the same error as above. This time there was also error about

SNMPD : ioctl 35123 returned -1

#7

Which OIDs are being polled? Is this a physical or virtual environment?


#8

I will have to run a debug poll tonight. This is hosting in vmware with e1000 nics on vmware 5.5.


#9

I know that this doesn’t help too much, but: happens to me as well. It started a few days ago, whereas the other router with the exact same VyOS version and hardware doesn’t have any problems. SNMP crashes suddenly with snmpd[3989]: segfault at 0 ip 00007f9b98eb1783 sp 00007fffbaa7c930 error 4 in libnetsnmpmibs.so.30.0.2[7f9b98e4d000+15e000]

Polling of the router didn’t change, it’s the same as before and on the other router. Did anyone find a solution for this crashing symptom?


#10

No I did not find a fixed. I just wrote a script that checks to see if snmpd is running if not restart it. Then made a cron job to run the script every 5 mins.