Static routes are no longer applied on 1.4 rolling

Bugs
, , , , ,
1

At some point on my 1.4 rolling, static routes operations are not longer taken into account.

Current routing table :

vyos@rc-kube-01:~$ sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] via 185.46.228.254, eth2, weight 1, 00:25:13
C * 10.10.50.0/24 is directly connected, eth1, 00:25:07
C>* 10.10.50.0/24 is directly connected, eth1, 00:25:16
S>* 10.20.5.0/24 [1/0] via 185.46.228.250, eth2, weight 1, 00:25:13
S>* 10.20.251.0/24 [1/0] via 185.46.228.251, eth2, weight 1, 00:25:13
S>* 91.132.252.0/24 [1/0] via 185.46.228.251, eth2, weight 1, 00:25:13
C>* 185.46.228.0/24 is directly connected, eth2, 00:25:16
S>* 192.168.2.0/24 [1/0] via 185.46.228.251, eth2, weight 1, 00:25:13
S>* 195.13.51.0/24 [1/0] unreachable (blackhole), weight 1, 00:25:13
B>* 195.13.51.1/32 [20/0] via 10.10.50.14, eth1, weight 1, 00:24:57

Current kernel routing table :

vyos@rc-kube-01:~$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         185.46.228.254  0.0.0.0         UG        0 0          0 eth2
10.10.50.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
10.20.5.0       185.46.228.250  255.255.255.0   UG        0 0          0 eth2
10.20.251.0     185.46.228.251  255.255.255.0   UG        0 0          0 eth2
91.132.252.0    185.46.228.251  255.255.255.0   UG        0 0          0 eth2
185.46.228.0    0.0.0.0         255.255.255.0   U         0 0          0 eth2
192.168.2.0     185.46.228.251  255.255.255.0   UG        0 0          0 eth2
195.13.51.0     0.0.0.0         255.255.255.0   U         0 0          0 *
195.13.51.1     10.10.50.14     255.255.255.255 UGH       0 0          0 eth1

Removing the route :

vyos@rc-kube-01# delete protocols static route 192.168.2.0/24 next-hop 185.46.228.251
[edit]
vyos@rc-kube-01# commit
[edit]

Route still present :

vyos@rc-kube-01:~$ sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] via 185.46.228.254, eth2, weight 1, 00:25:25
C * 10.10.50.0/24 is directly connected, eth1, 00:25:19
C>* 10.10.50.0/24 is directly connected, eth1, 00:25:28
S>* 10.20.5.0/24 [1/0] via 185.46.228.250, eth2, weight 1, 00:25:25
S>* 10.20.251.0/24 [1/0] via 185.46.228.251, eth2, weight 1, 00:25:25
S>* 91.132.252.0/24 [1/0] via 185.46.228.251, eth2, weight 1, 00:25:25
C>* 185.46.228.0/24 is directly connected, eth2, 00:25:28
S>* 192.168.2.0/24 [1/0] via 185.46.228.251, eth2, weight 1, 00:25:25
S>* 195.13.51.0/24 [1/0] unreachable (blackhole), weight 1, 00:25:25
B>* 195.13.51.1/32 [20/0] via 10.10.50.14, eth1, weight 1, 00:25:09

If I reboot, route is gone.

On this box, I get : BGP, a few static routes, VRRP and NAT.e

Exact version : 1.4-rolling-202105091233

Thanks

2

I found these logs in dmesg when touching routes :

[Tue May 18 14:26:09 2021] ------------[ cut here ]------------
[Tue May 18 14:26:09 2021] WARNING: CPU: 1 PID: 3495 at include/net/nexthop.h:354 fib_nh_match+0x1ed/0x3b0
[Tue May 18 14:26:09 2021] Modules linked in: af_packet vrf nft_masq nft_nat fuse nft_chain_nat xt_CT xt_tcpudp nft_compat nfnetlink_cthelper nft_counter nf_tables nfnetlink nf_nat_pptp nf_conntrack_pptp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper evdev pcspkr button mpls_iptunnel mpls_router ip_tunnel mpls_gso br_netfilter bridge stp llc xenfs xen_privcmd ip_tables x_tables autofs4 usb_storage ohci_hcd sd_mod t10_pi squashfs zstd_decompress lz4_decompress loop overlay ext4 crc32c_generic crc16 mbcache jbd2 nls_ascii hid_generic usbhid hid sr_mod cdrom ata_piix xen_netfront xen_blkfront crc32c_intel uhci_hcd libata i2c_piix4 scsi_mod ehci_hcd
[Tue May 18 14:26:09 2021] CPU: 1 PID: 3495 Comm: ip Tainted: G        W         5.10.35-amd64-vyos #1
[Tue May 18 14:26:09 2021] Hardware name: Xen HVM domU, BIOS 4.13 02/04/2021
[Tue May 18 14:26:09 2021] RIP: 0010:fib_nh_match+0x1ed/0x3b0
[Tue May 18 14:26:09 2021] Code: 7c 24 10 48 8b b5 90 00 00 00 e8 2e b6 f6 ff 48 8b 7c 24 10 41 89 c4 e8 21 b4 f6 ff 45 85 e4 0f 85 4e fe ff ff e9 66 ff ff ff <0f> 0b e9 35 ff ff ff 3c 0a 0f 85 7d fe ff ff 48 8b 8d 98 00 00 00
[Tue May 18 14:26:09 2021] RSP: 0018:ffffb7ff40ebf950 EFLAGS: 00010286
[Tue May 18 14:26:09 2021] RAX: 0000000000000000 RBX: ffffb7ff40ebfaa8 RCX: ffffb7ff40ebfbc8
[Tue May 18 14:26:09 2021] RDX: ffff99f80ae02000 RSI: 0000000000000000 RDI: ffffffffb36e8b40
[Tue May 18 14:26:09 2021] RBP: ffff99f80ae02000 R08: 00000000000000fe R09: 0000000000000000
[Tue May 18 14:26:09 2021] R10: 0000000000000008 R11: 00000000c0a80200 R12: ffffb7ff40ebfaa8
[Tue May 18 14:26:09 2021] R13: 0000000000000000 R14: ffff99f905bcf0c0 R15: ffff99f80ae02000
[Tue May 18 14:26:09 2021] FS:  00007f3798b7c800(0000) GS:ffff99f90aa40000(0000) knlGS:0000000000000000
[Tue May 18 14:26:09 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue May 18 14:26:09 2021] CR2: 00007f3798faa443 CR3: 000000000ac3e000 CR4: 0000000000350ee0
[Tue May 18 14:26:09 2021] Call Trace:
[Tue May 18 14:26:09 2021]  fib_table_delete+0x196/0x2f0
[Tue May 18 14:26:09 2021]  inet_rtm_delroute+0x86/0x100
[Tue May 18 14:26:09 2021]  rtnetlink_rcv_msg+0x127/0x380
[Tue May 18 14:26:09 2021]  ? __d_alloc+0x115/0x1c0
[Tue May 18 14:26:09 2021]  ? rtnl_calcit.isra.40+0x110/0x110
[Tue May 18 14:26:09 2021]  netlink_rcv_skb+0x4b/0xf0
[Tue May 18 14:26:09 2021]  netlink_unicast+0x1a0/0x280
[Tue May 18 14:26:09 2021]  netlink_sendmsg+0x213/0x430
[Tue May 18 14:26:09 2021]  ? netlink_unicast+0x280/0x280
[Tue May 18 14:26:09 2021]  ____sys_sendmsg+0x265/0x290
[Tue May 18 14:26:09 2021]  ? copy_msghdr_from_user+0x57/0x90
[Tue May 18 14:26:09 2021]  ___sys_sendmsg+0x77/0xb0
[Tue May 18 14:26:09 2021]  ? filemap_map_pages+0x2e3/0x410
[Tue May 18 14:26:09 2021]  ? __raw_spin_unlock+0x5/0x10
[Tue May 18 14:26:09 2021]  ? handle_mm_fault+0x15c8/0x1a10
[Tue May 18 14:26:09 2021]  ? __cgroup_bpf_run_filter_setsockopt+0xbb/0x350
[Tue May 18 14:26:09 2021]  __sys_sendmsg+0x52/0xa0
[Tue May 18 14:26:09 2021]  do_syscall_64+0x33/0x40
[Tue May 18 14:26:09 2021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[Tue May 18 14:26:09 2021] RIP: 0033:0x7f3798f38914
[Tue May 18 14:26:09 2021] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41 89 d4 55 48 89 f5 53
[Tue May 18 14:26:09 2021] RSP: 002b:00007ffeb53848d8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[Tue May 18 14:26:09 2021] RAX: ffffffffffffffda RBX: 0000000060a3ce82 RCX: 00007f3798f38914
[Tue May 18 14:26:09 2021] RDX: 0000000000000000 RSI: 00007ffeb5384940 RDI: 0000000000000003
[Tue May 18 14:26:09 2021] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000008
[Tue May 18 14:26:09 2021] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000001
[Tue May 18 14:26:09 2021] R13: 0000557447c49020 R14: 0000557447c26f70 R15: 00007ffeb5384b60
[Tue May 18 14:26:09 2021] ---[ end trace 98f99f8074967a22 ]---
3

Not reproducible with the latest rolling

vyos@r6-roll# run show ver

Version:          VyOS 1.4-rolling-202105170417
Release Train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Tue 18 May 2021 01:17 UTC
Build UUID:       f951a144-69c1-41d7-a0ab-9979f7917f0a
Build Commit ID:  fbf794f268bd2b

vyos@r6-roll# set protocols static route 192.0.2.0/24 next-hop 192.168.122.1
[edit]
vyos@r6-roll# commit
run show ip route[edit]
vyos@r6-roll# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] via 192.168.122.1, eth0, weight 1, 00:01:12
C>* 10.11.11.0/24 is directly connected, dum0, 00:01:15
C>* 100.64.0.0/30 is directly connected, eth1, 00:01:15
S>* 192.0.2.0/24 [1/0] via 192.168.122.1, eth0, weight 1, 00:00:02
C>* 192.168.122.0/24 is directly connected, eth0, 00:01:15
[edit]
vyos@r6-roll# 
[edit]
vyos@r6-roll# del protocols static route 192.0.2.0/24 next-hop 192.168.122.1
[edit]
vyos@r6-roll# commit
[edit]
vyos@r6-roll# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] via 192.168.122.1, eth0, weight 1, 00:01:28
C>* 10.11.11.0/24 is directly connected, dum0, 00:01:31
C>* 100.64.0.0/30 is directly connected, eth1, 00:01:31
C>* 192.168.122.0/24 is directly connected, eth0, 00:01:31