OSPFv3 crash when around 700 routes comes

Bugs
,
1

When I peer with my router which send more than 700 routes in OSPFv3, OSPFv3 seems crashing.

When I restart OSPFv3, it doesn’t seem to help.

When I stop sending OSPFv3 route from my other router, it works.

In the logs, I can see :

Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791875, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 710
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791877, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 614
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791879, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 686
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791881, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 432
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791883, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 33
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791885, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 628
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791887, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 375
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791889, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 764
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791891, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 311
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791893, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 337
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791895, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 231
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791897, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 820
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791899, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 38
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791901, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 789
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791903, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 561
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791905, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 431
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791907, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 330
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791909, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 86
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791911, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 578
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791913, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 443
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791915, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 810
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791917, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 490
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791919, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 389
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791921, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 189
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791923, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 368
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791925, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 102
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=3791927, pid=3064360775
Oct 28 09:44:13 fw-brest-01 zebra[1073]: [EC 4043309108] LSP Install Failure: in-label 596
Oct 28 09:44:13 fw-brest-01 zebra[1073]: Extended Error: Label >= configured maximum in platform_labels

Is there any open issue about that ?

Best regards,

2

Can you add that string and check again?

set system sysctl custom net.mpls.platform_labels value 1048575

3

Hello !

Thank you a lot for your reply ! :slight_smile:

I added this string to my configuration.

So, errors disappears from logs. But when I check with

run show ipv6 ospfv3

I got :

OSPFv3 is not running

So, here my OSPFv3 configuration :

[edit protocols ospfv3]

show

area 0 {
interface eth4.1094
interface eth4.1091
range fd00:0:0:1091::/64 {
}
range fd00:0:0:1092::/64 {
}
range fd00:0:0:1093::/64 {
}
range fd00:0:0:1094::/64 {
}
range fd00:0:0:1095::/64 {
}
range fd00:0:0:1099::/64 {
}
}
parameters {
router-id 192.168.0.17
}
redistribute {
connected {
}
kernel {
}
static {
}
}

Do you have any idea of how I can increase OSPFv3 logging ?
I’m looking in /var/log/messages and /var/log/frr/frr.log, and I got only this :

Oct 28 14:10:12 fw-brest-01 ospf6d[23837]: snmp[info]: NET-SNMP version 5.7.3 AgentX subagent connected
Oct 28 14:10:12 fw-brest-01 zebra[1073]: client 51 says hello and bids fair to announce only ospf6 routes vrf=0
Oct 28 14:10:16 fw-brest-01 watchfrr[1035]: ospf6d state → up : connect succeeded

So, in next hour, I will reboot the firewall (maybe it could help ?), because for now I only commit changes, and I will say what is the results.

Thank you so much !

4

Do you restart ospf process via FRR?
This cannot be done.
There is 2 options now. Reboot router or run “show conf com | match ospf”
Delete protocol ospf, commit, and add commands again.

5

Hello,

I restart the whole firewall, OSPFv3 stays buggued.

I tried this procedure :

  1. Starting OSPFv3 in VyOS
  2. Starting OSPFv3 on my Juniper

What I can see in log :

OSPFv3 instanciated

Oct 28 21:58:23 fw-brest-01 ospf6d[9310]: recvmsg read full buffer size: 0
Oct 28 21:58:48 fw-brest-01 ospf6d[9310]: message repeated 5 times: [ recvmsg read full buffer size: 0]
Shutdown OSPFv3 on Juniper and Vyos
Oct 28 22:03:38 fw-brest-01 ospf6d[9310]: Terminating on signal SIGINT
Oct 28 22:03:38 fw-brest-01 zebra[1104]: [EC 4043309121] Client ‘ospf6’ encountered an error and is shutting down.
Oct 28 22:03:38 fw-brest-01 watchfrr[1068]: [EC 268435457] ospf6d state → down : read returned EOF
Oct 28 22:03:39 fw-brest-01 ospf6d[9615]: snmp[info]: NET-SNMP version 5.7.3 AgentX subagent connected
Oct 28 22:03:39 fw-brest-01 zebra[1104]: client 57 disconnected. 0 ospf6 routes removed from the rib
Oct 28 22:03:39 fw-brest-01 zebra[1104]: client 57 says hello and bids fair to announce only ospf6 routes vrf=0
Oct 28 22:03:43 fw-brest-01 watchfrr[1068]: ospf6d state → up : connect succeeded ← OSPF not working here
Step 1 : Starting OSPv3 in VyOS
Oct 28 22:12:10 fw-brest-01 ospf6d[9615]: SPF: Scheduled in 0 msec
Oct 28 22:12:10 fw-brest-01 ospf6d[9615]: SPF processing: # Areas: 1, SPF runtime: 0 sec 73 usec, Reason: R+, L+
Oct 28 22:12:11 fw-brest-01 ospf6d[9615]: SPF: Scheduled in 0 msec
Oct 28 22:12:11 fw-brest-01 ospf6d[9615]: SPF processing: # Areas: 1, SPF runtime: 0 sec 42 usec, Reason: L+
Oct 28 22:12:12 fw-brest-01 ospf6d[9615]: SPF: Scheduled in 0 msec
Oct 28 22:12:12 fw-brest-01 ospf6d[9615]: SPF processing: # Areas: 1, SPF runtime: 0 sec 59 usec, Reason: R+, R-
Step 2 : Starting OSPFv3 in Juniper router
Oct 28 22:15:11 fw-brest-01 ospf6d[9615]: SPF: Scheduled in 0 msec
Oct 28 22:15:11 fw-brest-01 ospf6d[9615]: SPF processing: # Areas: 1, SPF runtime: 0 sec 74 usec, Reason: R+
Oct 28 22:15:11 fw-brest-01 ospf6d[9615]: SPF: Scheduled in 13 msec
Oct 28 22:15:11 fw-brest-01 ospf6d[9615]: SPF processing: # Areas: 1, SPF runtime: 0 sec 48 usec, Reason: L+
Oct 28 22:15:11 fw-brest-01 ospf6d[9615]: SPF: Scheduled in 70 msec
Oct 28 22:15:11 fw-brest-01 ospf6d[9615]: SPF processing: # Areas: 1, SPF runtime: 0 sec 9290 usec, Reason: R+, R-, N+
Oct 28 22:15:14 fw-brest-01 zebra[1104]: [EC 4043309121] Client ‘ospf6’ encountered an error and is shutting down.
Oct 28 22:15:14 fw-brest-01 watchfrr[1068]: [EC 268435457] ospf6d state → down : read returned EOF
Oct 28 22:15:14 fw-brest-01 zebra[1104]: client 57 disconnected. 766 ospf6 routes removed from the rib
Oct 28 22:15:19 fw-brest-01 watchfrr[1068]: [EC 100663303] Forked background command [pid 10684]: /usr/lib/frr/watchfrr.sh restart ospf6d
Oct 28 22:15:19 fw-brest-01 zebra[1104]: client 57 says hello and bids fair to announce only ospf6 routes vrf=0
Oct 28 22:15:19 fw-brest-01 watchfrr[1068]: ospf6d state → up : connect succeeded
Oct 28 22:15:21 fw-brest-01 ospf6d[10692]: recvmsg read full buffer size: 0 ← OSPF not working here

Best regards,

7

Memory leak in FRR.

==613254== 788 (256 direct, 532 indirect) bytes in 1 blocks are definitely lost in loss record 17 of 20
==613254==    at 0x483AB65: calloc (vg_replace_malloc.c:760)
==613254==    by 0x48D65F5: qcalloc (memory.c:115)
==613254==    by 0x13FC70: ospf6_route_create (ospf6_route.c:402)
==613254==    by 0x13FD79: ospf6_route_copy (ospf6_route.c:427)
==613254==    by 0x1558A5: ospf6_area_route_hook_add (ospf6_area.c:122)
==613254==    by 0x12B3CA: ospf6_intra_route_calculation (ospf6_intra.c:1967)
==613254==    by 0x145F6A: ospf6_spf_calculation_thread (ospf6_spf.c:637)
==613254==    by 0x492112D: thread_call (thread.c:1628)
==613254==    by 0x48CFAC6: frr_run (libfrr.c:1099)
==613254==    by 0x116220: main (ospf6_main.c:232)

We communicate with FRR developers.

8

Hello,

Ok, do you think I can patch VyOS with another version of FRR ?

Best regards,

9

I’m not sure, I think all versions are affected.
Can you try clean debian with installed any frr version?

10

Hello,

Not immediatly but in some days yes, I will send you feedback about other FRR version.

Thank you so much !

11

Hello,

I tried a clean Debian 10 installation, with FRRouting 6.0.2

With this basic configuration of FRR :

frr version 6.0.2
frr defaults traditional
hostname vyos-debug
log syslog informational
service integrated-vtysh-config
!
interface ens18
ip ospf cost 10
ip ospf dead-interval 4
ip ospf hello-interval 1
!
router ospf
ospf router-id 192.168.0.70
redistribute connected
passive-interface lo
network 192.168.0.16/28 area 0.0.0.0
!
router ospf6
ospf6 router-id 192.168.0.70
redistribute connected
area 0.0.0.0 range fd00:0:0:1094::/64
interface ens18 area 0.0.0.0
!
line vty
!

I have all my 765 prefixes from my Juniper router with no bug.

Best regards,

12

I believe this is a bug in FRR 7.3 — which was fixed in FRR 7.5:

@c-po has backported the fix to FRR 7.3, and I’m sure can tell you more :slight_smile:

13

Hi, i’m new in this forum
I have a Vyos 1.3.4 LTS and a 1.2.9 S1: do both of them come with fixed FRR 1.3.1 ?
Or the OSPFv3 bug is still present?

14

dpkg -l | grep frr

Will tell you what version of FRR you have.

1.3.5 comes with 7.5.1-20231128 - I would assume 1.3.4 has a fairly modern (fixed) version as well seeing as this thread is ~3 years old now…

2 Likes
15

Hi tjh, thanks for your answer.
In 1.2.9-S1 version I have FRR 1.31
in 1.3.4 version I have FRR 7.5.1-20230910
so i guess that i’ll use the 1.3.4 version.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.