Vrf: default rout leaking

Hello,

I am trying to configure vrf in a very simple topology: eth0 is the wan interface in the default vrf and eth2 is in the pippo vrf. My goal is to leak the wan default route to the pippo vrf. Here is the configuration:

 interfaces {
     ethernet eth0 {
         address dhcp
     }
     ethernet eth2 {
         address 10.0.0.1/24
         vrf pippo
     }
 }
 nat {
     source {
         rule 10 {
             outbound-interface {
                 name eth0
             }
             translation {
                 address masquerade
             }
         }
     }
 }
 protocols {
     static {
         route 10.0.0.0/24 {
             interface eth2 {
                 vrf pippo
             }
         }
     }
 }
 vrf {
     name pippo {
         protocols {
             static {
                 route 0.0.0.0/0 {
                     next-hop 192.168.154.2 {
                         interface eth0
                         vrf default
                     }
                 }
                 route 192.168.154.0/24 {
                     interface eth0 {
                         vrf default
                     }
                 }
             }
         }
         table 1000
     }
 }

Here are the generated routing tables:

vyos@vyos:~$ show ip route vrf all
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF default:
S>* 0.0.0.0/0 [210/0] via 192.168.154.2, eth0, weight 1, 00:14:00
S>* 10.0.0.0/24 [1/0] is directly connected, eth2 (vrf pippo), weight 1, 01:31:56
C>* 192.168.154.0/24 is directly connected, eth0, 00:14:00

VRF pippo:
S>* 0.0.0.0/0 [1/0] via 192.168.154.2, eth0 (vrf default), weight 1, 00:14:00
C>* 10.0.0.0/24 is directly connected, eth2, 01:31:56
S>* 192.168.154.0/24 [1/0] is directly connected, eth0 (vrf default), weight 1, 00:14:01

And here is the information about the interfaces:

vyos@vyos:~$ show  interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address          MAC                VRF        MTU  S/L    Description
-----------  ------------------  -----------------  -------  -----  -----  -------------
eth0         192.168.154.128/24  xx:xx:xx:xx:xx:xx  default   1500  u/u
eth2         10.0.0.1/24         xx:xx:xx:xx:xx:xx  pippo     1500  u/u
lo           127.0.0.1/8         00:00:00:00:00:00  default  65536  u/u
             ::1/128

I can ping the internet from the default vrf, while I cannot from the pippo vrf:

vyos@vyos:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=128 time=13.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=128 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=128 time=13.7 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=128 time=13.7 ms
vyos@vyos:~$ ping 8.8.8.8 vrf pippo
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

Do you see anything wrong or missing in my configuration?
How can I debug this problem?

Forgotten to report the VyOS version I am using: 1.5-rolling-202401030023.

I have solved the issue using a pair of virtual ethernet interfaces to connect the two vrf.

Here is the config:

 interfaces {
     ethernet eth0 {
         address dhcp
     }
     ethernet eth2 {
         address 10.0.0.1/24
         vrf pippo
     }
     virtual-ethernet veth0 {
         address 10.1.1.0/31
         peer-name veth2
     }
     virtual-ethernet veth2 {
         address 10.1.1.1/31
         peer-name veth0
         vrf pippo
     }
 }
 vrf {
     name pippo {
         protocols {
             static {
                 route 0.0.0.0/0 {
                     next-hop 10.1.1.0 {
                         interface veth2
                     }
                 }
             }
         }
         table 1000
     }
 }

vth0 resides in the default vrf, while its peer, vth2 resides in the pippo vrf.
Then vth2 is used in the pippo vrf to define the default root.

Here are the routing tables:

VRF default:
S>* 0.0.0.0/0 [210/0] via 192.168.154.2, eth0, weight 1, 00:40:35
C>* 10.1.1.0/31 is directly connected, veth0, 00:15:54
C>* 192.168.154.0/24 is directly connected, eth0, 00:40:35

VRF pippo:
S>* 0.0.0.0/0 [1/0] via 10.1.1.0, veth2, weight 1, 00:09:15
C>* 10.0.0.0/24 is directly connected, eth2, 00:44:58
C>* 10.1.1.0/31 is directly connected, veth2, 00:15:54
3 Likes

Hi @giuppo77,

Thank you for sharing your solution! Your contribution to resolving the issue is greatly appreciated, and your solution will undoubtedly help others facing similar challenges.

Aloha.
Interesting solution.
Is this a requirement on Vyos to leak routes between 2 VRF?
the first solution with static routes looks correct for me and the control plane / routing tables looks also correct.
Thanks
Marcel

Thanks to you! I am far from being an expert in the field, but I am glad to contribute and help when I can.

Yes, the documentation says that route leaking in vrf is supported.

And indeed, at least in my case, route leaking works for sub-nets but not for the default route. I do not have any explanation for that.

I just had the same issue!

I created a VRF “LAN” and assigned an interface to it:

    vif 201 {
        address X.X.X.2/24
        vrf LAN
    }

Created an interface route on the default VRF pointing to the “LAN” VRF:

    interface-route X.X.X.0/24 {
        next-hop-interface bond0.201 {
            next-hop-vrf LAN
        }
    }

It did work as expected, as it created the correct route and did in fact route these packets to the “LAN” VRF:

aldemaro@sp1-r001-staging:~$ show ip route X.X.X.0/24
Routing entry for X.X.X.0/24
Known via “static”, distance 1, metric 0, best
Last update 00:42:08 ago

  • directly connected, bond0.201(vrf LAN), weight 1

Created a static route inside the VRF pointing the default route to the default vrf:

vrf LAN {
    static {
        interface-route 0.0.0.0/0 {
            next-hop-interface bond0 {
                next-hop-vrf default
            }
        }
    }
}

This generated the correct route, but it did not route packets at all between the VLANs:

aldemaro@sp1-r001-staging:~$ show ip route vrf LAN
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF LAN:
S>* 0.0.0.0/0 [1/0] is directly connected, bond0 (vrf default), weight 1, 00:10:17
C>* X.X.X.0/24 is directly connected, bond0.201, 00:45:02

It seems like that kind of configuration is valid, but it did not work.

@giuppo77

I had the same issue like you and applied the trick with virtual interface but it does not work…do you may have any idea? I copy my configuration below…the only difference is that I leak between 2 different VRF( and default is not one of them)

vyos@VYOS-02-032:~$ show ip route vrf WAN1
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF WAN1:
S>* 0.0.0.0/0 [210/0] via 10.10.10.1, eth0, weight 1, 2d19h18m
C>* 10.1.1.0/31 is directly connected, veth0, weight 1, 00:09:01
L>* 10.1.1.0/32 is directly connected, veth0, weight 1, 00:09:01
C>* 10.10.10.0/24 is directly connected, eth0, weight 1, 2d19h18m
K * 10.10.10.0/24 [0/0] is directly connected, eth0, weight 1, 2d19h18m
L>* 10.10.10.61/32 is directly connected, eth0, weight 1, 2d19h18m
K>* 127.0.0.0/8 [0/0] is directly connected, WAN1, weight 1, 2d19h18m
vyos@VYOS-02-032:~$ show ip route vrf DA
DATA_COMMUNITY DATA_HUMANIT DATA_ICT_STAFF
vyos@VYOS-02-032:~$ show ip route vrf DATA_ICT_STAFF
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF DATA_ICT_STAFF:
S 0.0.0.0/0 [1/0] via 10.1.1.0, veth2, weight 1, 00:04:48
C>* 10.1.1.0/31 is directly connected, veth2, weight 1, 00:09:08
L>* 10.1.1.1/32 is directly connected, veth2, weight 1, 00:09:08
C>* 10.249.92.0/24 is directly connected, eth4.111, weight 1, 2d19h18m
L>* 10.249.92.252/32 is directly connected, eth4.111, weight 1, 2d19h18m
K>* 127.0.0.0/8 [0/0] is directly connected, DATA_ICT_STAFF, weight 1, 2d19h18m
vyos@VYOS-02-032:~$ show fi
file firewall
vyos@VYOS-02-032:~$ show firewall group
Possible completions:
Execute the current command
Show firewall group
detail Show list view of firewall group

vyos@VYOS-02-032:~$ show firewall group
Firewall Groups

Name Type References Members


WAN interface_group nat-source-107 eth0
nat-source-111 eth1
nat-source-112 eth2
nat-source-199 eth3
nat-source-200
vyos@VYOS-02-032:~$ show nat source rules
Rule Source Destination Proto Out-Int Translation


107 10.251.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
111 10.249.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
112 10.250.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
199 10.252.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
200 10.200.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
vyos@VYOS-02-032:~$ ping 10.1.1.0 vrf DATA_ICT_STAFF
PING 10.1.1.0 (10.1.1.0) 56(84) bytes of data.
64 bytes from 10.1.1.0: icmp_seq=1 ttl=64 time=0.069 ms
^C
— 10.1.1.0 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.069/0.069/0.069/0.000 ms
vyos@VYOS-02-032:~$ ping 8.8.8.8 vrf DATA_ICT_STAFF
/bin/ping: connect: Network is unreachable
vyos@VYOS-02-032:~$ ping 8.8.8.8 vrf WAN1
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=1714 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=885 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=713 ms
^C
— 8.8.8.8 ping statistics —
4 packets transmitted, 3 received, 25% packet loss, time 3059ms
rtt min/avg/max/mdev = 713.097/1103.888/1713.597/436.801 ms, pipe 2

When you ping, it’s going to use the exit interface towards the route you’re using. You’re using this route right now:

S 0.0.0.0/0 [1/0] via 10.1.1.0, veth2, weight 1, 00:04:48

So when you try to do ping 8.8.8.8 vrf DATA_ICT_STAFF, you’re source address will be 10.1.1.1

You’ll notice that you don’t have a source NAT statement for that traffic, hence why that ping to the internet fails.

Your next problem is that you don’t have any return routes in your WAN1 VRF. You do have a source NAT statement for 10.249.92.0/24 traffic, but you need to make sure the WAN1 VRF can get back to the DATA_ICT_STAFF VRF. Just add another static route like you did before, but this time it’s from WAN1 to DATA_ICT_STAFF.

The last issue may reside here in your DATA_ICT_STAFF VRF:

S 0.0.0.0/0 [1/0] via 10.1.1.0, veth2, weight 1, 00:04:48

You’ll notice it’s neither a selected (>) or FIB (*) route, so the system may not use it. This may just be what you pasted in and it’s fine on the router.

You can quickly test if it’s working by doing:

ping 10.10.10.61 vrf DATA_ICT_STAFF

That won’t need to use NAT. If it doesn’t work, you may have an issue in your veth config. Just paste that in here if it’s still not working.

Thanks a lot for you answer…I just applied what you proposed…I can ping the 10.10.10.61 but still no internet

vyos@VYOS-02-032:~$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=594 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=853 ms
^C
— 1.1.1.1 ping statistics —
3 packets transmitted, 2 received, 33.3333% packet loss, time 1999ms
rtt min/avg/max/mdev = 594.072/723.587/853.102/129.515 ms

vyos@VYOS-02-032:~$ show ip route vrf DATA_ICT
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure

VRF DATA_ICT:
S>* 0.0.0.0/0 [1/0] via 10.1.1.0, veth2, weight 1, 00:20:00
C>* 10.1.1.0/31 is directly connected, veth2, weight 1, 00:20:06
L>* 10.1.1.1/32 is directly connected, veth2, weight 1, 00:20:06
C>* 10.249.92.0/24 is directly connected, eth4.111, weight 1, 00:20:05
L>* 10.249.92.252/32 is directly connected, eth4.111, weight 1, 00:20:05
K>* 127.0.0.0/8 [0/0] is directly connected, DATA_ICT, weight 1, 00:20:07
vyos@VYOS-02-032:~$ ping 1.1.1.1 vrf DATA_ICT
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
From 1.1.1.1 icmp_seq=1 Destination Host Unreachable
From 1.1.1.1 icmp_seq=2 Destination Host Unreachable
From 1.1.1.1 icmp_seq=3 Destination Host Unreachable
^C
— 1.1.1.1 ping statistics —
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4096ms
pipe 4
vyos@VYOS-02-032:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address MAC VRF MTU S/L Description


eth0 10.10.10.61/24 52:54:00:53:1d:ea default 1500 u/u WAN_1
eth1 - 52:54:00:96:3e:87 default 1500 A/D WAN_2
eth2 - 52:54:00:f7:ef:ff default 1500 A/D WAN_3
eth3 - 52:54:00:ab:3a:e3 default 1500 A/D WAN_4
eth4 - 52:54:00:41:77:11 default 1500 u/u
eth4.107 10.251.92.252/24 52:54:00:41:77:11 default 1500 u/u CONTROL_PLANE
eth4.111 10.249.92.252/24 52:54:00:41:77:11 DATA_ICT 1500 u/u DATA_ICT_STAFF
eth4.112 10.250.92.252/24 52:54:00:41:77:11 default 1500 u/u DATA_HUMANITARIANS
eth4.199 10.252.92.252/24 52:54:00:41:77:11 default 1500 u/u GLOBAL_MGMT
eth4.200 10.200.92.252/24 52:54:00:41:77:11 default 1500 u/u DATA_COMMUNITIES
lo 127.0.0.1/8 00:00:00:00:00:00 default 65536 u/u
::1/128
veth0 10.1.1.0/31 4e:39:96:65:4b:7e default 1500 u/u
veth2 10.1.1.1/31 86:87:37:3c:0a:b9 DATA_ICT 1500 u/u
vyos@VYOS-02-032:~$ ping 10.10.10.61 vrf DATA_ICT
PING 10.10.10.61 (10.10.10.61) 56(84) bytes of data.
64 bytes from 10.10.10.61: icmp_seq=1 ttl=64 time=0.055 ms
64 bytes from 10.10.10.61: icmp_seq=2 ttl=64 time=0.093 ms
64 bytes from 10.10.10.61: icmp_seq=3 ttl=64 time=0.056 ms
^C
— 10.10.10.61 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2018ms
rtt min/avg/max/mdev = 0.055/0.068/0.093/0.017 ms
vyos@VYOS-02-032:~$ show nat source rules
Rule Source Destination Proto Out-Int Translation


107 10.251.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
111 10.249.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
112 10.250.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
199 10.252.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
200 10.200.92.0/24 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any
300 10.1.1.0/31 0.0.0.0/0 IP @I_WAN masquerade
sport any dport any

What version are you on? It’s working fine for me on 1.4.1:

set interfaces dummy dum0 address '10.0.0.100/32'
set interfaces dummy dum0 vrf 'TEST'
set interfaces dummy dum1 address '10.0.0.1/32'
set interfaces dummy dum1 vrf 'WAN'

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 vrf 'WAN'

set interfaces virtual-ethernet veth0 address '10.1.1.0/31'
set interfaces virtual-ethernet veth0 peer-name 'veth1'
set interfaces virtual-ethernet veth0 vrf 'WAN'
set interfaces virtual-ethernet veth1 address '10.1.1.1/31'
set interfaces virtual-ethernet veth1 peer-name 'veth0'
set interfaces virtual-ethernet veth1 vrf 'TEST'

set nat source rule 10 outbound-interface name 'eth0'
set nat source rule 10 source address '10.0.0.100'
set nat source rule 10 translation address 'masquerade'
set nat source rule 20 outbound-interface name 'eth0'
set nat source rule 20 source address '10.1.1.0/31'
set nat source rule 20 translation address 'masquerade'

set vrf name TEST protocols static route 0.0.0.0/0 next-hop 10.1.1.0
set vrf name TEST table '101'
set vrf name WAN protocols static route 10.0.0.100/32 next-hop 10.1.1.1
set vrf name WAN table '100'
vyos@vyos:~$ ping 4.2.2.2 vrf TEST count 1 # Will use 10.1.1.1 as source
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
64 bytes from 4.2.2.2: icmp_seq=1 ttl=57 time=11.7 ms

--- 4.2.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 11.650/11.650/11.650/0.000 ms
vyos@vyos:~$ ping 4.2.2.2 vrf TEST count 1 source-address 10.0.0.100 # Hardcoding source to 10.0.0.100
PING 4.2.2.2 (4.2.2.2) from 10.0.0.100 : 56(84) bytes of data.
64 bytes from 4.2.2.2: icmp_seq=1 ttl=57 time=5.91 ms

--- 4.2.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.911/5.911/5.911/0.000 ms

I also recommend doing a tcpdump on eth0 to see if the traffic is actually getting natted properly.

1 Like

Sorry for my late reply. After my tests last year, I have decided to not follow the vrf path. My initial idea was to use a vrf for each VLAN in my network. At the same time, given the number of VLANs I have, creating a pair of VETH for each one was making the configuration cumbersome to maintain.