1.4 to 1.5 upgrade problem - Multi-protocol IPv4 BGP sets weird IPv6 nexthop

Hi,

Upgraded a route reflector from 1.4 (202305300317), which used FRR 8.5.1, to 1.5 (202403020021), which uses FRR 9.1.0. The configuration was converted perfectly and things simply just worked, expect that IPv6 nexthops are altered when imported and subsequently reflected to route reflector clients.

Sessions are all iBGP, so I wouldn’t have expected the next hop to have been altered. Each participating router simply has an IPv6 address in a subnet common to the participating routers. I hope the following explains our setup:

A ----- RR ----- B

A and B are route reflector clients, with multiprotocol prefixes being exchanged over an IPv4 BGP session. RR is VyOS, where it has an IPv4 loopback IP assigned to ‘lo’ and it has it’s IPv6 IP in a bridge interface. This bridge is connected to other routers that also have their IPs in the same /64 subnet. A and B set the nexthop of IPv6 prefixes as their IPv6 address when sending these to RR.

This worked perfectly in VyOS 1.4 from May last year but VyOS 1.5 is changing the next hop address when reflecting these prefixes to other members.

Herewith an example of an IPv6 default gateway having been accepted by RR:

3710, (Received from a RR-client)
  2c0f:dead:beef:0:51:49:31:48 from 51.49.31.48 (51.49.31.48)
    Origin IGP, metric 0, localpref 140, valid, internal, best (Local Pref), rpki validation-state: not found
    Community: 55514:1013
    AddPath ID: RX 0, TX-All 38 TX-Best-Per-AS 0 TX-Best-Selected 0
    Advertised to: 51.49.31.48 51.49.31.49 51.49.31.52 51.49.31.50 51.49.31.51 51.49.31.55 51.49.32.45 51.49.32.46
    Last update: Tue Mar 12 11:09:25 2024

This however becomes a different next hop on other route reflector clients:

> /ipv6 route print detail where dst-address=::/0
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, o - ospf, b - bgp, U - unreachable
 0  Db  dst-address=::/0 gateway=::ffff:51.49.31.54 gateway-status=::ffff:51.49.31.54 unreachable distance=200 scope=40 target-scope=30 bgp-as-path="3710"
        bgp-local-pref=140 bgp-med=0 bgp-origin=igp bgp-communities=55514:1013 received-from=bwdur01-rr03

So VyOS installs the right IPv6 route and sets the correct next hop, as received from a route reflector client, but then changes the next hop to ::ffff:51.49.31.54. This is essentially ::ffff:< IPv4 address of peer that introduced prefix to RR >

Herewith redacted configs from the RR VyOS device:

set policy route-map rr_client-in rule 10 action 'permit'
set policy route-map rr_client-out rule 10 action 'permit'
set protocols bgp parameters cluster-id '51.49.31.50'
set protocols bgp parameters log-neighbor-changes
set protocols bgp parameters router-id '51.49.31.54'
set protocols bgp peer-group rr_client address-family ipv4-unicast route-map export 'rr_client-out'
set protocols bgp peer-group rr_client address-family ipv4-unicast route-map import 'rr_client-in'
set protocols bgp peer-group rr_client address-family ipv4-unicast route-reflector-client
set protocols bgp peer-group rr_client address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp peer-group rr_client address-family ipv6-unicast route-map export 'rr_client-out'
set protocols bgp peer-group rr_client address-family ipv6-unicast route-map import 'rr_client-in'
set protocols bgp peer-group rr_client address-family ipv6-unicast route-reflector-client
set protocols bgp peer-group rr_client address-family ipv6-unicast soft-reconfiguration inbound
set protocols bgp peer-group rr_client remote-as '55514'
set protocols bgp peer-group rr_client update-source 'lo'
set protocols bgp system-as '55514'

So you transport IPv4 and IPv6 over the same session?

Exactly, worked in VyOS 1.3 and 1.4 but the nexthop appears to get changed to an IPv4 address encoded as an IPv6 when sent to router regenerator clients, although VyOS installs the correct next hop for prefixes itself.

PS: We’re not wanting to do 6PE.

I presume this to be a bug, in that VyOS 1.5 rolling installs the correct gateway for IPv6 prefixes installed via an IPv4 BGP session (multi-protocol BGP) but then sets the gateway as an IPv4 address encoded as IPv6 when reflecting the prefix to other route reflect clients.

Any suggestions or should I open a Phabricator bug?

Can you try using extended nexthop statement?

Unfortunately doesn’t help. I believe this to possibly relate to PE6 (MPLS labelled paths for IPv6 by encoding IPv4 gateways as IPv6) having been forced on, breaking simple yet effective multi-protocol BGP:

I’ll try log a bug report in the upstream FRR project, thank you for your time and suggestions!

I logged the change in behaviour:

ton31337 has already provided a patch in response:

Is there a guide somewhere on how to rebuild VyOS incorporating an upstream (FRR) patch?

1 Like

I would like to suggest , create a bug report within :

https://vyos.dev/

you can add a patch , but you need re-build the version that are using , some example below :

Many thanks for your assistance, I have been able to rebuild the 1.5 nightly ISO but not sure if the .patch file I created (vyos-build/packages/frr/patches/revert_0325116.patch) is being processed. Purposefully putting something in the file to invalidate the match unfortunately doesn’t yield and error and the build again completes without problems.

I’ve created the following Phabiractor task, for VyOS developers to please consider reverting the change in behaviour in FRR 9.1 introduced in February: ⚓ T6202 Multi-Protocol BGP is broken by 6PE patch in upstream FRR 9.1

Herewith what I did, which appears to possibly silently ignore failures:

  • Debian 12 (Bookworm) minimal install
  • Install minimum required development tools:
    apt-get -y install devscripts git live-build make pbuilder python3-distutils python3-git python3-jinja2 python3-pystache python3-tomli qemu-utils sudo;
  • Clone VyOS 1.5:
    git clone -b current --single-branch https://github.com/vyos/vyos-build;
  • Created patch file:
cat > /root/vyos-build/packages/frr/patches/revert_0325116.patch <<EOF
diff -uNr frr/bgpd/bgp_updgrp_packet.c frr2/bgpd/bgp_updgrp_packet.c
--- frr/bgpd/bgp_updgrp_packet.c        2024-04-04 00:18:33.346905387 +0200
+++ frr2/bgpd/bgp_updgrp_packet.c       2024-04-04 00:20:04.487461666 +0200
@@ -523,16 +523,11 @@
                        gnh_modified = 1;
                }

-               if (peer->nexthop.v4.s_addr != INADDR_ANY &&
-                   (IN6_IS_ADDR_UNSPECIFIED(mod_v6nhg) ||
-                    (peer->connection->su.sa.sa_family == AF_INET &&
-                     paf->afi == AFI_IP6))) {
-                       /* set a IPv4 mapped IPv6 address if no global IPv6
-                        * address is found or if announcing IPv6 prefix
-                        * over an IPv4 BGP session.
-                        */
-                       ipv4_to_ipv4_mapped_ipv6(mod_v6nhg, peer->nexthop.v4);
-                       gnh_modified = 1;
+               if (IN6_IS_ADDR_UNSPECIFIED(mod_v6nhg)) {
+                       if (peer->nexthop.v4.s_addr != INADDR_ANY) {
+                               ipv4_to_ipv4_mapped_ipv6(mod_v6nhg,
+                                                        peer->nexthop.v4);
+                       }
                }

                if (IS_MAPPED_IPV6(&peer->nexthop.v6_global)) {
  • Retrieved FRR source:
cd /root/vyos-build/packages/frr;
git clone -b current --single-branch https://github.com/FRRouting/frr;
./build-frr.sh
I: Apply FRRouting patches not in main repository:
Patch format detection failed.

Is there possibly an idiot’s guide somewhere where I can learn how to rebuild the FRR package and the VyOS ISO with a custom patch for FRR? I presume that I’m missing an option or step somewhere to actually compile FRR when assembling VyOS 1.5 using the build script.

Did I mention that I have no development experience? :wink: Seriously though, many thanks for your assistance! I was able to recompile FRR with a custom patch but how to I get VyOS repackaged to include the custom FRR deb files?

Deployed a minimal Debian 12 host, then added minimum packages and figured out how to git clone the FRR source code. Looks like I need a newer version of libyang2-dev though.

  apt-get -y install devscripts git live-build make pbuilder python3-distutils python3-git python3-jinja2 python3-pystache python3-tomli qemu-utils sudo;
  git clone -b current --single-branch https://github.com/vyos/vyos-build;
  cd /root/vyos-build;
  make clean;
    apt-get -y install bison flex libc-ares-dev libelf-dev:native libpcre2-dev libreadline-dev libyang2-dev pkg-config python3-pytest:native python3-sphinx:native liblua5.3-dev;
    cd /root/vyos-build/packages/frr;
    git clone -b stable/9.1 --single-branch https://github.com/FRRouting/frr/;
    mkdir /root/vyos-build/packages/frr/patches;
    pico /root/vyos-build/packages/frr/patches/revert_0325116.patch;
      # See below
    export EMAIL=bbs2web@hotmail.com;
    ./build-frr.sh;
  cd /root/vyos-build;
  ./build-vyos-image iso --architecture amd64 --build-by "$EMAIL";

Patch file:

From 258409cfa05aaa378e4f120dc67fb226465fa829 Fri Apr 5 00:00:00 2024
From: David Herselman <bbs2web@hotmail.com>
Date: Fri, 5 Apr 2024 22:46:41 +0200
Subject: [PATCH] revert 0325116

Issue:
Revert FRR patch 0325116

Signed-off-by: David Herselman <bbs2web@hotmail.com>
---
 bgpd/bgp_updgrp_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff -uNr a/bgpd/bgp_updgrp_packet.c b/bgpd/bgp_updgrp_packet.c
--- a/bgpd/bgp_updgrp_packet.c  2024-04-04 00:49:02.669157448 +0200
+++ b/bgpd/bgp_updgrp_packet.c  2024-04-04 00:49:44.197402749 +0200
@@ -523,16 +523,11 @@
 			gnh_modified = 1;
 		}

-		if (peer->nexthop.v4.s_addr != INADDR_ANY &&
-		    (IN6_IS_ADDR_UNSPECIFIED(mod_v6nhg) ||
-		     (peer->connection->su.sa.sa_family == AF_INET &&
-		      paf->afi == AFI_IP6))) {
-			/* set a IPv4 mapped IPv6 address if no global IPv6
-			 * address is found or if announcing IPv6 prefix
-			 * over an IPv4 BGP session.
-			 */
-			ipv4_to_ipv4_mapped_ipv6(mod_v6nhg, peer->nexthop.v4);
-			gnh_modified = 1;
+		if (IN6_IS_ADDR_UNSPECIFIED(mod_v6nhg)) {
+			if (peer->nexthop.v4.s_addr != INADDR_ANY) {
+				ipv4_to_ipv4_mapped_ipv6(mod_v6nhg,
+							peer->nexthop.v4);
+			}
 		}

 		if (IS_MAPPED_IPV6(&peer->nexthop.v6_global)) {

The patch is applied as expected, but I can’t build without libyang2-dev >= 2.1.80

[root@debian-linux frr]# ./build-frr.sh
I: Apply FRRouting patches not in main repository:
Applying: revert 0325116
I: Ensure Debian build dependencies are met
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
chrpath is already the newest version (0.16-2+b1).
gawk is already the newest version (1:5.2.1-2).
install-info is already the newest version (6.8-6+b1).
libcap-dev is already the newest version (1:2.66-4).
libjson-c-dev is already the newest version (0.16-2).
librtr-dev is already the newest version (0.8.0-1+b1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'libpam0g-dev' instead of 'libpam-dev'
libpam0g-dev is already the newest version (1.5.2-6+deb12u1).
libprotobuf-c-dev is already the newest version (1.4.1-1+b1).
libpython3-dev is already the newest version (3.11.2-1+b1).
libsnmp-dev is already the newest version (5.9.3+dfsg-2).
protobuf-c-compiler is already the newest version (1.4.1-1+b1).
python3-dev is already the newest version (3.11.2-1+b1).
texinfo is already the newest version (6.8-6+b1).
lua5.3 is already the newest version (5.3.6-2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
I: Build Debian FRR Package
dch warning: Previous package version was Debian native whilst new version is not
dpkg-buildpackage: info: source package frr
dpkg-buildpackage: info: source version 9.1-117-gd292ccb5f
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by root <bbs2web@hotmail.com>
dpkg-buildpackage: info: host architecture amd64
 dpkg-source --before-build .
dpkg-checkbuilddeps: error: Unmet build dependencies: libyang2-dev (>= 2.1.80)
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)

Edit: Found and installed the newer libyang2 packages:

curl -O https://deb.frrouting.org/frr/pool/frr-9.1/liby/libyang2/libyang2_2.1.128-1~deb12u1_amd64.deb;
curl -O https://deb.frrouting.org/frr/pool/frr-9.1/liby/libyang2/libyang2-dev_2.1.128-1~deb12u1_amd64.deb;
dpkg -i libyang2*.deb;

Right, so I simply copy the compiled frr .deb packages to the root of the packages folder and the VyOS build script then uses those instead.

Herewith consolidated notes for anyone else needing to rebuild VyOS. The below notes apply against a newly deployed minimal Debian 12 (bookworm) instance:

apt-get -y install devscripts git live-build make pbuilder python3-distutils python3-git python3-jinja2 python3-pystache python3-tomli qemu-utils sudo;
git clone -b current --single-branch https://github.com/vyos/vyos-build;
cd /root/vyos-build;
curl -O https://deb.frrouting.org/frr/pool/frr-9.1/liby/libyang2/libyang2_2.1.128-1~deb12u1_amd64.deb;
curl -O https://deb.frrouting.org/frr/pool/frr-9.1/liby/libyang2/libyang2-dev_2.1.128-1~deb12u1_amd64.deb;
dpkg -i libyang2*.deb;
make clean;
  # Custom compiling FRR:
  apt-get -y install bison flex libc-ares-dev libelf-dev:native libpcre2-dev libreadline-dev libyang2-dev pkg-config python3-pytest:native python3-sphinx:native liblua5.3-dev;
  cd packages/frr;
  git clone -b stable/9.1 --single-branch https://github.com/FRRouting/frr/;
  mkdir patches;
  pico patches/revert_0325116.patch;
    # See above
  export EMAIL=bbs2web@hotmail.com;
  ./build-frr.sh;
  for f in frr_*.deb frr-pythontools_*.deb frr-rpki-rtrlib_*.deb frr-snmp_*.deb; do cp $f ..; done;
  cd ../..;
./build-vyos-image iso --architecture amd64 --build-by "$EMAIL";
2 Likes

@bbs2web have seen that it was revert on FRR , so , next version should have this regression . generally , we use containers flavor for bluid/patch in VyOS, to avoid installing all the dependency.