VyOS states "Network is unreachable" and refuses to send packets to a destination in the routing table

Here’s my Topology:

I’m trying to ping the loopback of vmrouter (top) from R1 (bottom) – but VyOS states “Network is unreachable” and refuses to send packets to the destination – even though the network is in the routing table. The next hops are reachable.

I have eBGP running between vmrouter and Exit1, and iBGP between exit1 and R1. Each router is redistributing connected networks to BGP.

The output from my router:

vyos@R1:~$ ping 1.2.10.1
connect: Network is unreachable
vyos@R1:~$ trace 1.2.10.1
traceroute to 1.2.10.1 (1.2.10.1), 30 hops max, 60 byte packets
connect: Network is unreachable
vyos@R1:~$ sh ip route 1.2.10.1
Routing entry for 1.2.10.1/32
  Known via "bgp", distance 200, metric 0, best
  Last update 03:14:18 ago
    192.168.203.5 (recursive), weight 1
  *   1.2.1.1, via eth0, weight 1

vyos@R1:~$ ping 192.168.203.5
PING 192.168.203.5 (192.168.203.5) 56(84) bytes of data.
64 bytes from 192.168.203.5: icmp_seq=1 ttl=63 time=3.22 ms
^C
--- 192.168.203.5 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.223/3.223/3.223/0.000 ms
vyos@R1:~$ ping 1.2.1.1
PING 1.2.1.1 (1.2.1.1) 56(84) bytes of data.
64 bytes from 1.2.1.1: icmp_seq=1 ttl=64 time=1.00 ms
^C
--- 1.2.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.003/1.003/1.003/0.000 ms
vyos@R1:~$ sh int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             1.2.1.2/30                        u/u  
eth1             -                                 u/u  
eth2             -                                 u/u  
eth3             -                                 u/u  
eth4             -                                 u/u  
eth5             -                                 u/D  
eth6             -                                 u/D  
eth7             -                                 u/D  
lo               127.0.0.1/8                       u/u  
                 ::1/128
vyos@R1:~$ sh ip bgp 1.2.10.1
BGP routing table entry for 1.2.10.1/32
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  10001
    192.168.203.5 from 1.2.1.1 (1.2.10.2)
      Origin incomplete, metric 0, localpref 100, valid, internal, best (First path received)
      Last update: Mon May 29 16:59:51 2023
vyos@R1:~$ sh ip bgp
BGP table version is 8, local router ID is 1.2.1.2, vrf id 0
Default local pref 100, local AS 10002
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 1.2.1.0/30       0.0.0.0                  0         32768 ?
* i                 1.2.1.1                  0    100      0 ?
*>i1.2.10.1/32      192.168.203.5            0    100      0 10001 ?
*>i1.2.10.2/32      1.2.1.1                  0    100      0 i
*>i192.168.0.0/24   192.168.203.5            0    100      0 10001 ?
*>i192.168.203.0/24 1.2.1.1                  0    100      0 ?

Displayed  5 routes and 6 total paths

Update

The IP is now pingable after adding and deleting a static route!

Step1: added a static route for 1.2.10.1 with next hop as 192.168.203.5 (basically replicating the BGP route with a static one).

C>* 1.2.1.0/30 is directly connected, eth0, 09:13:58
S>  1.2.10.1/32 [1/0] via 192.168.203.5 (recursive), 00:00:46
  *                     via 1.2.1.1, eth0, 00:00:46
B   1.2.10.1/32 [200/0] via 192.168.203.5 (recursive), 11:59:30
                          via 1.2.1.1, eth0, 11:59:30
B>* 1.2.10.2/32 [200/0] via 1.2.1.1, eth0, 09:13:57
B>  192.168.0.0/24 [200/0] via 192.168.203.5 (recursive), 11:59:30
  *                          via 1.2.1.1, eth0, 11:59:30
B>* 192.168.203.0/24 [200/0] via 1.2.1.1, eth0, 09:13:57

vyos@R1:~$ ping 1.2.10.1
PING 1.2.10.1 (1.2.10.1) 56(84) bytes of data.
64 bytes from 1.2.10.1: icmp_seq=1 ttl=63 time=1.72 ms
64 bytes from 1.2.10.1: icmp_seq=2 ttl=63 time=1.84 ms
^C
--- 1.2.10.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.720/1.781/1.843/0.074 ms

vyos@R1:~$ sh conf comm | grep static 
set protocols static route 1.2.10.1/32 next-hop 192.168.203.5 distance '1'

This proved that the problem was indeed that VyOS didn’t inject the BGP route into the global RT even though the sh ip route command said it did.

Then I tried deleting the route to see if the target becomes unreachable when I removed the route:

vyos@R1# del proto static route 1.2.10.1/32
[edit]

[OUTPUT OMITTED]

vyos@R1:~$ sh ip route 1.2.10.1
Routing entry for 1.2.10.1/32
  Known via "bgp", distance 200, metric 0, best
  Last update 12:02:25 ago
    192.168.203.5 (recursive), weight 1
  *   1.2.1.1, via eth0, weight 1

vyos@R1:~$ ping 1.2.10.1
PING 1.2.10.1 (1.2.10.1) 56(84) bytes of data.
64 bytes from 1.2.10.1: icmp_seq=1 ttl=63 time=2.08 ms
64 bytes from 1.2.10.1: icmp_seq=2 ttl=63 time=1.64 ms
^C
--- 1.2.10.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.644/1.863/2.083/0.223 ms

But this time it stayed reachable! Is this a bug? I even rechecked the route!

vyos@R1:~$ sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

C>* 1.2.1.0/30 is directly connected, eth0, 09:17:28
B>  1.2.10.1/32 [200/0] via 192.168.203.5 (recursive), 12:03:00
  *                       via 1.2.1.1, eth0, 12:03:00
B>* 1.2.10.2/32 [200/0] via 1.2.1.1, eth0, 09:17:27
B>  192.168.0.0/24 [200/0] via 192.168.203.5 (recursive), 12:03:00
  *                          via 1.2.1.1, eth0, 12:03:00
B>* 192.168.203.0/24 [200/0] via 1.2.1.1, eth0, 09:17:27

What does your routing table actually show?

Here’s my RT:

vyos@R1:~$ sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

C>* 1.2.1.0/30 is directly connected, eth0, 08:44:41
B>  1.2.10.1/32 [200/0] via 192.168.203.5 (recursive), 11:30:13
  *                       via 1.2.1.1, eth0, 11:30:13
B>* 1.2.10.2/32 [200/0] via 1.2.1.1, eth0, 08:44:40
B>  192.168.0.0/24 [200/0] via 192.168.203.5 (recursive), 11:30:13
  *                          via 1.2.1.1, eth0, 11:30:13
B>* 192.168.203.0/24 [200/0] via 1.2.1.1, eth0, 08:44:40

@giga1699 please check my update - it started working after adding and removing a static route! Is this a bug?!

What version are you using ? r1 has a ebgp neighbor with Exit1 or direct with VMroute , as I understand next-hop is Exit1, although this prefix is learned from Vmroute , if I’mcorrect.

Hi @fernando - Exit1 has iBGP with r1 and ebgp with VMRouter. The VyOS version is 1.2.9.S1(KVM).

Version:          VyOS 1.2.9-S1
Release Train:    crux

Built by:         autobuild@vyos.net
Built on:         Thu 23 Feb 2023 13:47 UTC
Build UUID:       31f37177-d137-4df8-93c6-32c4b609b834
Build Commit ID:  93090af8c36584-dirty

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:     Unknown
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors

I don’t think it’s a bug , if it wouldn’t work when you add static route (it might some process not was running perfectly) .however , this design can be covered perfectly by 1.4 nightly builds.

Not a BGP guru (only ancient CCNP)…but this topic makes me think about next-hop-self