[VyOS 1.5 Stream] - VRF and PBR Not functioning correctly

gianluca.casella · April 15, 2025, 4:07pm

Hi Everyone!

I’ve got an interesting one here. I’ve been rattling my brain with this one and have tried multiple ways to try to make this work. Here is a high level description of what I’m trying to achieve.

I’ve created a simplified virtual lab to describe what I’m experiencing, i’ve also tested this on the latest version of the VyOS 1.5 rolling release as well with the same outcome.

LAN networks will use a specific default gateway for specific UDP traffic
Any LAN network requiring DNS / TCP (http / https) traffic will use the vrf named “INTERNET”
Using VETH interfaces to establish an iBGP connection between the vrfs “default” and “INTERNET”.
I have a very simple PBR in this lab that will redirect all Internet traffic out of the Internet VRF
ICMP ping works just fine reaching “8.8.8.8” or “1.1.1.2” but name resolution fails, and TCP fails too towards any url, in this I tried “ifconfig.me/ip”

VyOS 1.5 Stream Config:

set interfaces ethernet eth0 address '10.4.0.1/24'
set interfaces ethernet eth0 hw-id '0c:76:53:4a:00:00'
set interfaces ethernet eth1 hw-id '0c:76:53:4a:00:01'
set interfaces ethernet eth2 hw-id '0c:76:53:4a:00:02'
set interfaces ethernet eth3 hw-id '0c:76:53:4a:00:03'
set interfaces ethernet eth4 hw-id '0c:76:53:4a:00:04'
set interfaces ethernet eth5 hw-id '0c:76:53:4a:00:05'
set interfaces ethernet eth6 hw-id '0c:76:53:4a:00:06'
set interfaces ethernet eth7 hw-id '0c:76:53:4a:00:07'
set interfaces ethernet eth8 hw-id '0c:76:53:4a:00:08'
set interfaces ethernet eth9 hw-id '0c:76:53:4a:00:09'
set interfaces ethernet eth9 vif 35 vrf 'INTERNET'
set interfaces loopback lo
set interfaces pppoe pppoe0 authentication password '*****************'
set interfaces pppoe pppoe0 authentication username '*****************'
set interfaces pppoe pppoe0 description 'PPPoE'
set interfaces pppoe pppoe0 ip adjust-mss '1452'
set interfaces pppoe pppoe0 mru '1492'
set interfaces pppoe pppoe0 mtu '1492'
set interfaces pppoe pppoe0 no-peer-dns
set interfaces pppoe pppoe0 source-interface 'eth9.35'
set interfaces pppoe pppoe0 vrf 'INTERNET'
set interfaces virtual-ethernet veth0 address '10.255.255.0/31'
set interfaces virtual-ethernet veth0 peer-name 'veth1'
set interfaces virtual-ethernet veth1 address '10.255.255.1/31'
set interfaces virtual-ethernet veth1 peer-name 'veth0'
set interfaces virtual-ethernet veth1 vrf 'INTERNET'
set nat source rule 1 outbound-interface name 'pppoe0'
set nat source rule 1 source address '0.0.0.0/0'
set nat source rule 1 translation address 'masquerade'
set policy route PBR1 interface 'eth0'
set policy route PBR1 rule 20 action 'accept'
set policy route PBR1 rule 20 log
set policy route PBR1 rule 20 set table '150'
set policy route PBR1 rule 20 source address '10.4.0.0/24'
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp neighbor 10.255.255.1 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor 10.255.255.1 remote-as '65000'
set protocols bgp system-as '65000'
set service ntp allow-client address '127.0.0.0/8'
set service ntp allow-client address '169.254.0.0/16'
set service ntp allow-client address '10.0.0.0/8'
set service ntp allow-client address '172.16.0.0/12'
set service ntp allow-client address '192.168.0.0/16'
set service ntp allow-client address '::1/128'
set service ntp allow-client address 'fe80::/10'
set service ntp allow-client address 'fc00::/7'
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set system config-management commit-revisions '100'
set system console device ttyS0 speed '115200'
set system host-name 'vyos'
set system login user vyos authentication encrypted-password '$6$rounds=656000$7e7g5/6I9crYHF6i$2Bd/TUkoMUDY66aeOrOfMdDG7KN6Rnu9KkxeqPhCryFcRp2DcGf5OjTAHJmqIK2C/pZJmDIbVhYpOCn4dHa9c.'
set system login user vyos authentication plaintext-password ''
set system syslog global facility all level 'info'
set system syslog global facility local7 level 'debug'
set vrf name INTERNET protocols bgp address-family ipv4-unicast redistribute connected
set vrf name INTERNET protocols bgp neighbor 10.255.255.0 address-family ipv4-unicast nexthop-self
set vrf name INTERNET protocols bgp neighbor 10.255.255.0 remote-as '65000'
set vrf name INTERNET protocols bgp system-as '65000'
set vrf name INTERNET table '150'

VyOS Routing Table:

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF INTERNET:
S>* 0.0.0.0/0 [210/0] is directly connected, pppoe0, weight 1, 00:09:21
B>* 10.4.0.0/24 [200/0] via 10.255.255.0, veth1, weight 1, 00:09:01
C>* 10.255.255.0/31 is directly connected, veth1, 00:10:30
C>* 142.124.37.123/32 is directly connected, pppoe0, 00:09:21

VRF default:
C>* 10.4.0.0/24 is directly connected, eth0, 00:22:03
C>* 10.255.255.0/31 is directly connected, veth0, 00:22:05
B>* 142.124.37.123/32 [200/0] via 10.255.255.1, veth0, weight 1, 00:09:01
[edit]

PBR Log (ping 1.1.1.2):

Apr 15 16:02:02 vyos kernel: [ 1488.870210] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43612 DF PROTO=ICMP TYPE=8 CODE=0 ID=169 SEQ=1
Apr 15 16:02:03 vyos kernel: [ 1489.872255] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43848 DF PROTO=ICMP TYPE=8 CODE=0 ID=169 SEQ=2
Apr 15 16:02:04 vyos kernel: [ 1490.874009] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=44340 DF PROTO=ICMP TYPE=8 CODE=0 ID=169 SEQ=3

PBR Log (curl ifconfig.me/ip)

Apr 15 16:03:28 vyos kernel: [ 1575.145498] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26673 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
Apr 15 16:03:28 vyos kernel: [ 1575.145576] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26674 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
Apr 15 16:03:33 vyos kernel: [ 1580.150448] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26675 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
Apr 15 16:03:33 vyos kernel: [ 1580.150495] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26676 DF PROTO=UDP SPT=40384 DPT=53 LEN=37

PBR Log (ping google.ca)

Apr 15 16:02:50 vyos kernel: [ 1536.871047] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30648 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
Apr 15 16:02:50 vyos kernel: [ 1536.871145] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30649 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
Apr 15 16:02:55 vyos kernel: [ 1541.876231] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30650 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
Apr 15 16:02:55 vyos kernel: [ 1541.876291] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30651 DF PROTO=UDP SPT=56541 DPT=53 LEN=35

Results from Sample Host (10.4.0.200):

root@UbuntuDockerGuest-4:~# ping 1.1.1.2 -c 3
PING 1.1.1.2 (1.1.1.2) 56(84) bytes of data.
64 bytes from 1.1.1.2: icmp_seq=1 ttl=54 time=5.68 ms
64 bytes from 1.1.1.2: icmp_seq=2 ttl=54 time=5.62 ms
64 bytes from 1.1.1.2: icmp_seq=3 ttl=54 time=6.97 ms
--- 1.1.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 5.618/6.089/6.974/0.626 ms


root@UbuntuDockerGuest-4:~# ping google.ca
ping: google.ca: Temporary failure in name resolution


root@UbuntuDockerGuest-4:~# curl ifconfig.me/ip
curl: (6) Could not resolve host: ifconfig.me
root@UbuntuDockerGuest-4:~#

Now if i ended up doing this:

set protocols static route 0.0.0.0/0 next-hop 10.255.255.1 int veth0
delete policy route PBR1 rule 20 set table '150'

DNS and TCP will work normally – but this defeats the purpose… i can’t add static routes for ALL individual tcp/udp ip addresses in the main routing table:

Apr 15 16:05:56 vyos kernel: [ 1723.070514] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=52136 DF PROTO=UDP SPT=49392 DPT=53 LEN=35
Apr 15 16:05:56 vyos kernel: [ 1723.070606] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=52137 DF PROTO=UDP SPT=49392 DPT=53 LEN=35
Apr 15 16:05:56 vyos kernel: [ 1723.081009] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=142.251.41.35 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=19104 DF PROTO=ICMP TYPE=8 CODE=0 ID=170 SEQ=1
Apr 15 16:05:56 vyos kernel: [ 1723.085917] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=72 TOS=0x00 PREC=0x00 TTL=64 ID=49296 DF PROTO=UDP SPT=48706 DPT=53 LEN=52
Apr 15 16:05:57 vyos kernel: [ 1724.082562] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=142.251.41.35 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=19275 DF PROTO=ICMP TYPE=8 CODE=0 ID=170 SEQ=2
Apr 15 16:05:58 vyos kernel: [ 1725.084504] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=142.251.41.35 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=19406 DF PROTO=ICMP TYPE=8 CODE=0 ID=170 SEQ=3

Also, worth noting, if i used set protocols static table ## route without using VRF’s i won’t run into this issue.

Any insight on this would be greatly appreciated!!!

c-po · April 15, 2025, 7:42pm

The idea sounds interesting and legit to partially VRF-leak individual traffic. Some hints to probably ease your configuration:

No need to use veth pairs for iBGP, you can use:

set protocols bgp address-family ipv4-unicast import vrf INTERNET
set protocols bgp address-family ipv4-unicast route-map vrf import <optional-filter>

set vrf name INTERNET protocols bgp address-family ipv4-unicast import vrf default
set vrf name INTERNET protocols bgp address-family ipv4-unicast route-map vrf import <optional-filter>

You can also reference a VRF name in the PBR statement:

set policy route PBR1 rule 20 set vrf INTERNET

Sorry for just throwing in additional CLI commands - most likely not helping your issue.

gianluca.casella · April 15, 2025, 11:04pm

Hey @c-po
Yep! I’m aware of that part that you are mentioning in regards to not requiring the veth interfaces, the main purpose for this is because if I don’t do this than there is no next-hop IP address on the route and it messes around with the Locally routed traffic (for example pinging from one interface to another between VRFs) i’ve experimented a lot with that part.

As for set policy route PBR1 rule 20 set vrf INTERNET, this isn’t available in VyOS 1.5 Stream, but IS indeed available in the Rolling Releases of version 1.5 (which has been tested as well).

But nonetheless, thank you for the reponse!

gianluca.casella · April 21, 2025, 3:50pm

I ended up opening up this Bug Report