vyOS 1.4 - L3VPN BGP import/export nexthop incorrect

I have a problem with L3VPNs and specifically with the import/export of routes in/out of VRF. The exported route from the VRF is declared as ‘inaccessible’ due to the fact that the route has a next hop of the global route table BGP loopback, although the RIB correctly states it is in the ‘customer’ VRF.

[VyOS 1.4-rolling-202211180318] - sh ver below

The following output shows:

  1. The route from within the VRF ‘customer’ which is correct and is in fact directly connected.

  2. The route from after being exported into a L3VPN, which shows ‘inaccessible’ with the IP of 10.255.254.2, which is in fact in the GRT as the GRT BGP process loopback/router-id.
    This L3VPN should be pointing to an address inside the VRF customer.

My config issue or bug ?

vyos@vyos-99:~$  **show bgp vrf customer ipv4 10.180.2.0/24**
BGP routing table entry for 10.180.2.0/24, version 1
Paths: (1 available, best #1, vrf customer)
  Not advertised to any peer
  Local
    0.0.0.0 from 0.0.0.0 (10.180.2.2)
      Origin IGP, metric 0, weight 32768, valid, sourced, local, best (First path received)
      Last update: Thu Nov 24 16:32:16 2022


vyos@vyos-99:~$  **show bgp ipv4 vpn 10.180.2.0/24**
BGP routing table entry for 1:2:10.180.2.0/24, version 0
not allocated
Paths: (1 available, no best path)
  Not advertised to any peer
  Local
    0.0.0.0 (**inaccessible**) from 0.0.0.0 (10.255.254.2) **vrf customer**(6) announce-nh-self
      Origin IGP, metric 0, weight 32768, invalid, sourced, local
      Extended Community: RT:2:1
      Originator: 10.255.254.2
      Remote label: 80
      Last update: Thu Nov 24 17:10:15 2022

Topology

The version I am using is

Version:          VyOS 1.4-rolling-202211180318
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Fri 18 Nov 2022 03:18 UTC
Build UUID:       7d903365-be97-4d56-865b-ca8560ae24aa
Build commit ID:  594c99e012fd66

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 08 c5 33 c1 d3 56 f5-83 1f 28 c4 b5 49 4d bc
Hardware UUID:    33c50842-d3c1-f556-831f-28c4b5494dbc

Copyright:        VyOS maintainers and contributors

The configuration is:

interfaces {
    dummy dum100 {
        address 10.255.254.2/32
    }
    ethernet eth0 {
        address 10.243.176.37/24
        description MGMT
        hw-id 00:50:56:88:36:22
        offload {
            gro
            gso
            sg
            tso
        }
        vrf mgmt
    }
    ethernet eth1 {
        address 10.180.0.1/25
        description CUSTOMER
        disable
        hw-id 00:50:56:88:f2:d5
        offload {
            gro
            gso
            sg
            tso
        }
        vrf customer
    }
    ethernet eth2 {
        address 10.180.2.2/24
        description RTN
        hw-id 00:50:56:88:55:cd
        offload {
            gro
            gso
            sg
            tso
        }
        vrf customer
    }
    ethernet eth3 {
        address 10.180.3.2/24
        description MPLS
        hw-id 00:50:56:88:82:43
        offload {
            gro
            gso
            sg
            tso
        }
    }
    loopback lo {
    }
}
protocols {
    bgp {
        neighbor 1.1.1.1 {
            address-family {
                ipv4-unicast {
                }
                ipv4-vpn {
                    soft-reconfiguration {
                        inbound
                    }
                }
            }
            description "MPLS c8kv"
            remote-as 64532
            update-source 10.255.254.2
        }
        parameters {
            log-neighbor-changes
            router-id 10.255.254.2
        }
        system-as 64532
    }
    mpls {
        interface eth3
        ldp {
            discovery {
                transport-ipv4-address 10.255.254.2
            }
            export {
                ipv4 {
                    explicit-null
                }
            }
            interface eth3
            router-id 10.255.254.2
        }
    }
    ospf {
        area 0 {
            network 0.0.0.0/0
        }
        parameters {
            abr-type cisco
            router-id 10.255.254.2
        }
    }
}
service {
    https {
        api {
            keys {
                id REST {
                    key ****************
                }
            }
        }
        vrf mgmt
    }
    ssh {
        client-keepalive-interval 180
        listen-address 10.243.176.37
        port 22
        vrf mgmt
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    domain-name prod.labs.haystacknetworks.com
    host-name vyos-99
    login {
        banner {
            post-login "EDP VRF Edge Security Appliance"
        }
        user vyos {
            authentication {
                encrypted-password ****************
            }
        }
    }
    name-server 192.168.10.1
    name-server 192.168.1.20
    ntp {
        server 10.243.176.254 {
        }
        vrf mgmt
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
}
vrf {
    name customer {
        protocols {
            bgp {
                address-family {
                    ipv4-unicast {
                        export {
                            vpn
                        }
                        import {
                            vpn
                        }
                        label {
                            vpn {
                                export auto
                            }
                        }
                        network 10.180.2.0/24 {
                        }
                        rd {
                            vpn {
                                export 1:2
                            }
                        }
                        route-target {
                            vpn {
                                export 2:1
                                import 1:1
                            }
                        }
                    }
                }
                neighbor 2.1.1.10 {
                    address-family {
                        ipv4-unicast {
                            nexthop-self {
                            }
                            soft-reconfiguration {
                                inbound
                            }
                        }
                    }
                    description "ACI L3out Customer Inside Loopback Peer A"
                    ebgp-multihop 3
                    remote-as 64530
                    update-source eth1
                }
                neighbor 2.1.1.11 {
                    address-family {
                        ipv4-unicast {
                            nexthop-self {
                            }
                            soft-reconfiguration {
                                inbound
                            }
                        }
                    }
                    description "ACI L3out Customer Inside Loopback Peer B"
                    ebgp-multihop 3
                    remote-as 64530
                    update-source eth1
                }
                parameters {
                    log-neighbor-changes
                    router-id 10.180.2.2
                }
                system-as 64531
            }
            static {
                route 2.1.1.10/32 {
                    next-hop 10.180.0.125 {
                    }
                }
                route 2.1.1.11/32 {
                    next-hop 10.180.0.125 {
                    }
                }
            }
        }
        table 120
    }
    name mgmt {
        protocols {
            static {
                route 0.0.0.0/0 {
                    next-hop 10.243.176.254 {
                    }
                }
            }
        }
        table 100
    }
}

Hi

I saw some points , it could be giving the issues , as you show over the topology draw , the BGP neigh in the address-family vpnv4 aren’t using the direct connected , try to apply this command :

    set protocols bgp neighbor 1.1.1.1 disable-connected-check 
    set protocols bgp neighbor 1.1.1.1 address-family ipv4-vpn nexthop-self

after it check if you next-hop in GRT is reachable by zebra :

sudo vtysh -c "show ip nht x.x.x.x"

also, you can remove explicti-null command in MPLS section , it shouldn’t be necessary.

regards

Hi

Unfortunately, that has not helped. The BGP peering and routes are stable and valid except for these routes from the vyOS customer VRF imported into VPNv4. So the peering between vyOS and 1.1.1.1 is good. I also removed explicit-null, that was as I read (somewhere) that there was an issue and had that from previous setup steps, anyway gone now.

So output from the vyOS customer VRF table for 10.180.2.0/24 and from the VPNv4 table the route has been exported to. Same issue on the VPNv4 RIB which has the correct nexthop VRF (customer) but the IP next hop is the dummy interface/loopback in the GRT.

vyos@vyos-99:~$ sh bgp vrf customer ipv4 10.180.2.0/24
BGP routing table entry for 10.180.2.0/24, version 4
Paths: (1 available, best #1, vrf customer)
  Not advertised to any peer
  Local
    0.0.0.0 from 0.0.0.0 (10.180.0.1)
      Origin IGP, metric 0, weight 32768, valid, sourced, local, best (First path received)
      Last update: Fri Nov 25 13:05:51 2022
vyos@vyos-99:~$
vyos@vyos-99:~$
vyos@vyos-99:~$ sh bgp ipv4 vpn 10.180.2.0/24
BGP routing table entry for 1:2:10.180.2.0/24, version 0
not allocated
Paths: (1 available, no best path)
  Not advertised to any peer
  Local
    0.0.0.0 (inaccessible) from 0.0.0.0 (10.255.254.2) vrf customer(6) announce-nh-self
      Origin IGP, metric 0, weight 32768, invalid, sourced, local
      Extended Community: RT:2:1
      Originator: 10.255.254.2
      Remote label: 80
      Last update: Fri Nov 25 13:05:50 2022


vyos@vyos-99:~$ sh int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
dum100           10.255.254.2/32                   u/u
eth0             10.243.176.37/24                  u/u  MGMT
eth1             10.180.0.1/25                     u/u  CUSTOMER
eth2             10.180.2.2/24                     u/u  RTN
eth3             10.180.3.2/24                     u/u  MPLS
lo               127.0.0.1/8                       u/u
                 ::1/128
vyos@vyos-99:~$
vyos@vyos-99:~$
vyos@vyos-99:~$ sh vrf
Name      State    MAC address        Flags                     Interfaces
--------  -------  -----------------  ------------------------  ------------
customer  up       46:5b:7a:25:1e:42  noarp,master,up,lower_up  eth1,eth2
mgmt      up       7e:ff:aa:70:75:22  noarp,master,up,lower_up  eth0

Looking at the show ip nht x.x.x.x command, I see what i would expect for the GRT with the 1.1.1.1 entry, and not entry for 10.180.2.0/24 which I “think” I would expect as the route is in the VPNv4 table as it’s a VPN route and not the GRT.

vyos@vyos-99:~$ sudo vtysh -c "show ip nht  10.180.2.0"
<no output>

vyos@vyos-99:~$ sudo vtysh -c "show ip nht"
1.1.1.1
 resolved via ospf
 via 10.180.3.254, eth3
 Client list: bgp(fd 32)

looks weird, however, in FRR use GRT to transport vpnv4 prefixes. it’s something different with regular vendors use vpnv4 without the necessity GRT . I’ll make a lab with it(it’s possible to be a bug)

Hello Simón

I’ve created a lab with his current configuration, it seems that command network 10.180.2.0/24 over the address-family ipv4 isn’t able to install direct connected prefix . however , if we use 'redistribute connected` it works properly.

set vrf name customer protocols bgp address-family ipv4-unicast export vpn
set vrf name customer protocols bgp address-family ipv4-unicast import vpn
set vrf name customer protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name customer protocols bgp address-family ipv4-unicast network 10.180.2.0/24 
set vrf name customer protocols bgp address-family ipv4-unicast  redistribute connected                                                                                               
set vrf name customer protocols bgp address-family ipv4-unicast rd vpn export '1:2'
set vrf name customer protocols bgp address-family ipv4-unicast route-target vpn export '1:2'
set vrf name customer protocols bgp address-family ipv4-unicast route-target vpn import '1:2'
set vrf name customer protocols bgp neighbor 2.1.1.10 address-family ipv4-unicast nexthop-self
set vrf name customer protocols bgp neighbor 2.1.1.10 address-family ipv4-unicast soft-reconfiguration inbound
set vrf name customer protocols bgp neighbor 2.1.1.10 ebgp-multihop '3'
set vrf name customer protocols bgp neighbor 2.1.1.10 remote-as '64530'
set vrf name customer protocols bgp neighbor 2.1.1.10 update-source 'eth1'
set vrf name customer protocols bgp neighbor 2.1.1.11 address-family ipv4-unicast soft-reconfiguration inbound
set vrf name customer protocols bgp neighbor 2.1.1.11 ebgp-multihop '3'
set vrf name customer protocols bgp neighbor 2.1.1.11 remote-as '64530'
set vrf name customer protocols bgp neighbor 2.1.1.11 update-source 'eth1'
set vrf name customer protocols bgp parameters log-neighbor-changes
set vrf name customer protocols bgp parameters router-id '10.180.2.2'

now we can see that it can install prefix :

vyos@cust-pe1:~$ show bgp ipv4 vpn 10.180.2.0/24
BGP routing table entry for 1:2:10.180.2.0/24, version 2
not allocated
Paths: (2 available, best #1)
  Advertised to non peer-group peers:
  1.1.1.1
  Local
    0.0.0.0 from 0.0.0.0 (10.255.254.2) vrf customer(7) announce-nh-self
      Origin incomplete, metric 0, weight 32768, valid, sourced, local, best (First path received)
      Extended Community: RT:1:2
      Originator: 10.255.254.2
      Remote label: 144
      Last update: Mon Nov 28 17:22:09 2022
  Local
    0.0.0.0 (inaccessible) from 0.0.0.0 (10.255.254.2) vrf customer(7) announce-nh-self
      Origin IGP, metric 0, weight 32768, invalid, sourced, local
      Extended Community: RT:1:2
      Originator: 10.255.254.2
      Remote label: 144
      Last update: Mon Nov 28 16:08:30 2022
vyos@cust-pe1:~$

also,I would suggest that used the command : set protocols bgp neighbor 1.1.1.1 disable-connected-check

here’s the ask on phabricator , I’m not sure that FRR decided this handle.

https://phabricator.vyos.net/T4846

Hi Fernando,

So not using the network statements and instead using redistribute connected does being a valid route in, I have the correct route as follows.

vyos@vyos-99:~$  show bgp ipv4 vpn 10.180.2.0/24
BGP routing table entry for 1:2:10.180.2.0/24, version 11
not allocated
Paths: (1 available, best #1)
  Advertised to non peer-group peers:
  1.1.1.1
  Local
    0.0.0.0 from 0.0.0.0 (10.255.254.6) vrf customer(6) announce-nh-self
      Origin incomplete, metric 0, weight 32768, valid, sourced, local, best (First path received)
      Extended Community: RT:2:1
      Originator: 10.255.254.6
      Remote label: 80
      Last update: Tue Nov 29 18:15:42 2022
vyos@vyos-99:~$

Which is good, but still leaves me with the same traffic issue, in that the traffic is not being bumped over to the VRF from MPLS on the vyOS router, this is still the same behavior I was seeing regardless of the route state.

As per the attached diagram with flows shown, red for the ICMP request and green for the ICMP reply. The reply gets back to the MPLS vyOS router but does not make it out of the eth2 interface, in fact it just enters eth3 then is dropped “somewhere”. The following is the diagram and after which a tcpdump for all interfaces with the relevant ICMP packets.

You will see that the ICMP request, comes in eth2 in the vrf customer, then goes out eth3 on MPLS, then the ICMP reply from 5.5.5.5 comes back in eth3, then nothing is seen again for that packet, we expect it should leave eth2 to the source of the ICMP request being 10.180.2.254 which is directly connected to vyOS eth2: 10.180.2.2.

Still doesnt feel right, config is as we have had with the expection of removing bgp network statements and using redistribute connected and also have got the disable-connected-check enabled.