VyOS 1.5-rolling-202401260023 FRR BGP daemon crash

crycoco · June 11, 2024, 1:50am

We have a testing environment and built a set of MPLS using VYOS. LOG:
Jun 10 10:11:13 bgpd[2672750]: [N463T-4M950][EC 33554449] u1:s23 attributes Jun 10 10:11:13 bgpd[2672750]: [N463T-4M950][EC 33554449] u1:s23 attributes Jun 10 10:11:13 bgpd[2672750]: [N463T-4M950][EC 33554449] u1:s23 attributes Jun 10 10:11:13 BGP[2672750]: Received signal 11 at 1717985473 (si_addr Jun 10 10:11:13 BGP[2672750]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlo Jun 10 10:11:13 BGP[2672750]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlo Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: showing active Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Buffer Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Host config Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Command Tokens Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Command Token Text Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Command Token Help Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Command Argument Name Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: RCU thread Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: RCU sequence barrier Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Scripting Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: FRR POSIX Thread Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: POSIX sync primitives Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Graph Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Graph Node Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Hash Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Hash Bucket Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Hash Index Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Interface Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Connected Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Link List Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Link Node Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Temporary memory Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Bitfield memory Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Module loading name Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Nexthop Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Nexthop label Jun 10 10:11:13 BGP[2672750]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xf6001) Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Northbound Node Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Northbound Configuration Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Northbound Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Prefix Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Privilege information Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Ring buffer Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Skip List Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Skip Node Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Skiplist Counters Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Socket union Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Stream Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Stream FIFO Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Route table Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Thread Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Thread master Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Thread Poll Info Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Thread stats Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Typed-hash bucket Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Typed-heap array Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Vector Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Vector index Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: VRF Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: VRF bit-map Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: VTY Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: VTY server Jun 10 10:11:13 BGP[2672750]: /lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0) Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Work queue Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Work queue name string Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: YANG module Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Zclient Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Redistribution Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: log thread-local buffer Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: showing active Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: syslog target Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: showing active Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Peer KeepAlive Timer Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP Peer pthread Mutex Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Mac Hash Entry Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Mac Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP instance Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP listen socket details Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP peer Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP peer hostname Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Peer group Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP Peer group hostname Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP peer af Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP update group Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP update subgroup Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP packet Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP attribute Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP aspath Jun 10 10:11:13 BGP[2672750]: /usr/lib/frr/bgpd(bgp_advertise_clean_subgroup+0x1c) Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP aspath seg Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP aspath segment data Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP aspath str Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP table Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP node Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP route Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP ancillary route info Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP connected Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP adv attr Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP adv Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP synchronise Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP adj out Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP multipath info Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: community Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: community val Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: community str Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: extcommunity Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: extcommunity val Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: extcommunity str Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: community-list handler Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP nexthop Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP peer update interface Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP own address Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP EVPN MH Information Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: Software Version Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP Jun 10 10:11:13 BGP[2672750]: /usr/lib/frr/bgpd(subgroup_update_packet+0x583) Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP PBR Context Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP interface context Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: BGP Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: showing active Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: NVE Configuration Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: RFAPI Generic Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: RFAPI Import Table Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: RFAPI Monitor Encap Jun 10 10:11:13 frrinit.sh[2672750]: core_handler: memstats: RFAPI IT Extra Jun 10 10:11:13 BGP[2672750]: /usr/lib/frr/bgpd(bgp_generate_updgrp_packets+0x429) Jun 10 10:11:13 BGP[2672750]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(event_call+0x7d) Jun 10 10:11:13 BGP[2672750]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xc0) Jun 10 10:11:13 BGP[2672750]: /usr/lib/frr/bgpd(main+0x373) [0x555a241506b3 Jun 10 10:11:13 BGP[2672750]: /lib/x86_64-linux-gnu/libc.so.6(+0x271ca) Jun 10 10:11:13 BGP[2672750]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) Jun 10 10:11:13 BGP[2672750]: /usr/lib/frr/bgpd(_start+0x21) [0x555a2415243 Jun 10 10:11:13 BGP[2672750]: in thread (bgp_generate_updgrp_packets) Jun 10 10:11:13 watchfrr[1658]: [HD38Q-0HBRT][EC 268435457] bgpd state Jun 10 10:11:13 zebra[1677]: [VXKFG-8SJRV][EC 4043309121] Client ‘vnc’ Jun 10 10:11:13 zebra[1677]: [VXKFG-8SJRV][EC 4043309121] Client ‘bgp’ Jun 10 10:11:13 zebra[1677]: [YDZ55-W3VM6] release_daemon_table_chunks: Jun 10 10:11:13 zebra[1677]: [JPSA8-5KYEA] client 31 disconnected Jun 10 10:11:13 zebra[1677]: [S929C-NZR3N] client 31 disconnected Jun 10 10:11:13 zebra[1677]: [YDZ55-W3VM6] release_daemon_table_chunks: Jun 10 10:11:13 zebra[1677]: [JPSA8-5KYEA] client 17 disconnected Jun 10 10:11:13 zebra[1677]: [S929C-NZR3N] client 17 disconnected Jun 10 10:11:18 systemd[1]: serial-getty@ttyS0.service: Deactivated Jun 10 10:11:18 systemd[1]: serial-getty@ttyS0.service: Scheduled Jun 10 10:11:18 systemd[1]: Stopped serial-getty@ttyS0.service - Serial Jun 10 10:11:18 systemd[1]: Started serial-getty@ttyS0.service - Serial Jun 10 10:11:18 agetty[2932135]: /dev/ttyS0: not a tty
Jun 10 10:11:18 watchfrr[1658]: [YFT0P-5Q5YX] Forked background command Jun 10 10:11:18 frrinit.sh[2932136]: Cannot stop bgpd: pid 2672750 Jun 10 10:11:18 watchfrr.sh[2932142]: Cannot stop bgpd: pid 2672750 Jun 10 10:11:18 zebra[1677]: [V98V0-MTWPF] client 17 says hello and Jun 10 10:11:18 zebra[1677]: [V98V0-MTWPF] client 31 says hello and Jun 10 10:11:19 bgpd[2932144]: [VTVCM-Y2NW3] Configuration Read in Jun 10 10:11:19 bgpd[2932144]: [G6NKK-8C6DV] end_config: VTY:0x55c2117b6430, Jun 10 10:11:19 watchfrr[1658]: [QDG3Y-BY5TN] bgpd state → up : Recently, we found that the FRR daemon (bgp) of VYOS often crashes. Any suggestions, Thanks~~
too long, cannot send UPDATE
too long, cannot send UPDATE
too long, cannot send UPDATE
0x30, PC 0x555a241f192c); aborting…
g_backtrace_sigsafe+0x6f) [0x7f0c64e8322f]
g_signal+0xf5) [0x7f0c64e83435]
allocations in memory group libfrr
: 4 * 24
: 6 * (variably sized)
: 13161 * 72
: 9423 * (variably sized)
: 9423 * (variably sized)
: 2295 * (variably sized)
: 2 * 128
: 1 * 32
: 18 * (variably sized)
: 4 * (variably sized)
: 4 * (variably sized)
: 42 * 8
: 15485 * 32
: 581 * (variably sized)
: 415306 * 32
: 291 * (variably sized)
: 7 * 272
: 9 * 48
: 62 * 40
: 378 * 24
: 2349 * (variably sized)
: 2 * (variably sized)
: 1 * 5
: 16 * 152
: 16 * 8
[0x7f0c64eb8001]
: 258 * 1192
: 2 * 24
Configuration Entry: 2 * 1032
: 9 * 56
: 3 * (variably sized)
: 22 * (variably sized)
: 1192 * 56
: 3379 * (variably sized)
: 1192 * 68
: 18 * 112
: 7 * (variably sized)
: 22 * 64
: 105 * 56
: 47 * 160
: 12 * (variably sized)
: 6 * 8192
: 26 * 96
: 12 * (variably sized)
: 1 * (variably sized)
: 31055 * 24
: 31055 * (variably sized)
: 2 * 216
: 3 * 8
: 2 * (variably sized)
: 2 * 32
[0x7f0c64ae6fd0]
: 6 * 144
: 6 * (variably sized)
: 5 * 48
: 2 * 3144
instance IDs : 6 * 2
: 2 * 24608
allocations in memory group logging subsystem
: 1 * 56
allocations in memory group bgpd
: 9 * 24
Peer pthread Conditional : 1 * 48
: 1 * 40
: 4 * 16
Hash Entry Intf String : 5 * (variably sized)
: 4 * (variably sized)
: 2 * 144
: 12 * 20864
: 22 * (variably sized)
: 1 * 64
: 1 * 9
: 9 * 80
: 1 * 104
: 1 * 240
: 1 * 56
: 374243 * 320
: 38517 * 40
[0x555a241f192c]
: 38516 * 24
: 38516 * (variably sized)
: 38517 * (variably sized)
: 92 * 56
: 2364 * 192
: 4374 * 136
: 4374 * 432
: 3 * 4
: 1 * 24
: 1 * 64
: 1 * 48
: 1208 * 96
: 1879 * 48
: 13 * 40
: 13 * (variably sized)
: 13 * (variably sized)
: 6 * 40
: 6 * (variably sized)
: 6 * (variably sized)
: 1 * 120
: 9 * 184
: 10 * 5
: 3 * 64
: 1 * 56
: 9 * (variably sized)
Martian Addr Intf String : 3 * 5
[0x555a241f4163]
: 1 * 32
: 7 * 4
EVPN instance information : 1 * 56
allocations in memory group rfapi
: 1 * 2984
: 1 * 296
: 1 * 208
: 2187 * 40
: 1197 * 40
[0x555a241b19c9]
[0x7f0c64eca07d]
[0x7f0c64e7af00]
]
[0x7f0c64ad21ca]
[0x7f0c64ad2285]
1]
scheduled from …/bgpd/bgp_fsm.c:1006 bgp_adjust_routeadv()
→ down : read returned EOF
encountered an error and is shutting down.
encountered an error and is shutting down.
Released 0 table chunks
0 vnc routes removed from the rib
0 vnc nhgs removed from the rib
Released 0 table chunks
0 bgp routes removed from the rib
0 bgp nhgs removed from the rib
successfully.
restart job, restart counter is at 921838.
Getty on ttyS0.
Getty on ttyS0.
[pid 2932136]: /usr/lib/frr/watchfrr.sh restart bgpd
not running
not running
bids fair to announce only bgp routes vrf=0
bids fair to announce only vnc routes vrf=0
Took: 00:00:00
pending SET-CFG: 0
connect succeeded

BGP Config：
protocols {
bgp {
address-family {
ipv4-unicast {
redistribute {
connected {
}
static {
}
}
}
}
neighbor eth1.2303 {
interface {
peer-group PE_IPv4
}
}
neighbor vti4 {
interface {
peer-group CE_IPv4
}
update-source dum5
}
parameters {
graceful-restart {
}
graceful-shutdown
log-neighbor-changes
router-id 169.254.100.4
}
peer-group CE_IPv4 {
address-family {
ipv4-unicast {
allowas-in {
}
as-override
soft-reconfiguration {
}
}
}
ebgp-multihop 255
remote-as 65533
}
peer-group PE_IPv4 {
address-family {
ipv4-unicast {
allowas-in {
}
as-override
soft-reconfiguration {
}
}
}
ebgp-multihop 255
remote-as 9936
update-source eth1.2303
}
system-as 65511
}
}
table 2303

crycoco · June 11, 2024, 1:51am

I want to know what version of FRR corresponds to the this VYOS version?

talmakion · June 11, 2024, 2:18am

I want to know what version of FRR corresponds to the this VYOS version?

You can pull that up with vtysh and/or dpkg, for example I have a 1.5-rolling-202406020021 that shows:

$ vtysh -c 'show ver'
FRRouting 9.1 (TEST-VYOS-LEFT) on Linux(6.6.32-amd64-vyos).
[...big text blob snipped...]
$ dpkg -l frr
[...snip...]
||/ Name           Version            Architecture Description
+++-==============-==================-============-=============================================================
ii  frr            9.1-172-g923799172 amd64        FRRouting suite of internet protocols (BGP, OSPF, IS-IS, ...)

Received signal 11

A SIGSEGV is a serious fault that should probably be reported upstream if it hasn’t already been resolved. If you can confirm it’s still a problem with the most recent VyOS rolling (there have been changes to the FRR build since the date on your rolling), that would be a starting point.

If you want to check a point-in-time build process for the FRR package used, it’s in the vyos-build repo at: vyos-build/packages/frr at current · vyos/vyos-build · GitHub

tjh · June 11, 2024, 3:45am

This is excellent advice and exactly what I was going to post.

Please do keep us updated though!

Apachez · June 11, 2024, 9:01am

Since you are using VyOS 1.5-rolling-202401260023 I would start by trying the latest version which currently is VyOS 1.5-rolling-202406060020 to see if the problem remains.

The above can be downloaded from: Releases · vyos/vyos-rolling-nightly-builds · GitHub

Then as config the command mode is prefered since it makes it easier to copy the config elsewhere.

Like so:

show config commands | strip-private

You can leave out that “strip-private” part if you dont have anything sensitive in your config.

Info about hardware like what kind of CPU (vendor/model) and amount of RAM would be helpful aswell. Also which network cards and drivers you are using (output of lspci and lsmod).

marekm72 · June 12, 2024, 10:53am

If the crashes are often at the same address, this is likely a bug. If it happens at various random addresses, could likely be caused by hardware issues like bad RAM. ECC RAM helps, sadly not all hardware platforms support it. Try stressing the hardware with https://www.memtest.org/ for a few hours - boot-time RAM test by BIOS alone is not sufficient. Long long ago a good test was a Linux kernel build, GCC was good at detecting bad RAM due to heavy use of pointers, BGP may be simlar as it manages big routing tables.