Questions about route-map or policy route based on ingress interface

, ,
1

I need help to use vyos’s policy route-map.

Follow to pics are what I want to do:

up
up1297×777 52.4 KB

  1. In this pic, if a ip packet with source ip(10.10.1.2) send to internet(like 8.8.8.8), I hope this packet passes through 1, 2, 3, and 4 in sequence, at vyos Router1’s eth0 the packet will do source nat, change it’s source ip address, then be sent to the internet.

down
down1302×753 53 KB

  1. In this pic, a data packet returned from 8.8.8.8 to 10.10.1.2. I hope the packet do nat in router 1, change it’s dest ip to 10.10.1.2, then passes through 2,3,4,5 in sequence.

My question is:
In vyos Router 1, can I use policy route-map to achieve my goal? If not, are there any other ways to achieve it?

Thanks for your patience

2

Do you want everything that comes through eth1 on vyos-router1 to go in the flow through vyos-router2 or only a particular client?

Why do you want to push traffic through that 2nd vyos-router?

I would probably resolve this by setting up two different VRF’s on vyos-router1, lets say INTERNAL and INTERNET.

Then bind eth1 and eth2 to vrf INTERNAL and eth0 and ether3 to vrf INTERNET.

This way the default route for vrf INTERNAL on vyos-router1 points to 10.10.195.2 so anything that comes from your clients connected to the switch will be pushed to the vyos-router2.

Then on vyos-router2 that would have its default route pointing to 10.10.196.1.

And finally the vrf INTERNET on vyos-router1 have its default route learned from the ISP.

This way you only need to do a single masquerade (SNAT) and thats on the eth0 interface towards your ISP.

3

Thanks, and good questions.

Actually, there are other lans connect to Router 1.
And only packets targets to internet should come through vyos-router2.

There is an application running on vyos-router2 to process internet traffic.

I know little about vrf.
Can I use policy route-map to achieve my goal?

4

No, use vrf to achieve your goal.

https://docs.vyos.io/en/latest/configuration/vrf/index.html

5

Thanks!
It almost works!

Following is my configuration:

  1. vyos router2
# show interfaces 
 ethernet eth0 {
     address 10.10.196.2/28
     hw-id 40:62:31:12:8c:1c
 }
 ethernet eth1 {
     address 10.10.195.2/28
     hw-id 40:62:31:12:8c:1d
 }
 loopback lo {
 }

# show protocols 
 static {
     route 0.0.0.0/0 {
         next-hop 10.10.196.1 {
         }
     }
     route 10.10.0.0/17 {
         next-hop 10.10.195.1 {
         }
     }
 }
  1. vyos router1
# show interfaces 
 ethernet eth0 {
     address 10.10.192.2/28
     hw-id 40:62:31:15:a9:87
     vrf INTERNET
 }
 ethernet eth1 {
     address 10.10.193.1/28
     hw-id 40:62:31:15:a9:88
     vrf INTERNAL
 }
 ethernet eth2 {
     address 10.10.195.1/28
     hw-id 40:62:31:15:a9:89
     vrf INTERNAL
 }
 ethernet eth3 {
     address 10.10.196.1/28
     hw-id 40:62:31:15:a9:8a
     vrf INTERNET
 }
 loopback lo {
 }

# show protocols 
 static {
     route 0.0.0.0/0 {
         next-hop 10.10.192.1 {
         }
     }
 }
 vrf INTERNAL {
     static {
         route 0.0.0.0/0 {
             next-hop 10.10.195.2 {
             }
         }
         route 10.10.0.0/17 {
             next-hop 10.10.193.2 {
             }
         }
     }
 }
 vrf INTERNET {
     static {
         route 0.0.0.0/0 {
             next-hop 10.10.192.1 {
             }
         }
         route 10.10.0.0/17 {
             next-hop 10.10.196.2 {
             }
         }
     }
 }
  1. and in router 2, I can see packets from user terminal:by tcpdump
tcpdump -i eth0 icmp -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:01:47.910790 IP 10.10.10.158 > 8.8.8.8: ICMP echo request, id 1, seq 1474, length 40
20:01:47.941679 IP 8.8.8.8 > 10.10.10.158: ICMP echo reply, id 1, seq 1474, length 40
20:01:48.926422 IP 10.10.10.158 > 8.8.8.8: ICMP echo request, id 1, seq 1475, length 40
20:01:48.957116 IP 8.8.8.8 > 10.10.10.158: ICMP echo reply, id 1, seq 1475, length 40

However there are still two problems:

  1. Router1 can’t access internet?
$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C>* 192.168.3.0/24 is directly connected, eth5, 6d04h08m

$ ping 8.8.8.8
connect: Network is unreachable
  1. How to access other lans?

I will go on tomorrow.
Thank you again!

6

it’s possible that you need consider more feature in this design ,for example created route-leaking where prefix in vrf INTERNAL be able to access default in INTERNET , there are many ways to do it .

https://docs.vyos.io/en/latest/configuration/vrf/index.html#vrf-route-leaking

only vrf is not enough to achieve this goal , also NAT or more techniques should be applied .

7

Thanks！

route-leaking makes user terminals(10.10.0.0/17) can access other lans.

But still I have the first problem: Router1 can’t access internet.

Here is my configuration in Router1:

# show protocols 
 static {
     route 0.0.0.0/0 {
         next-hop 10.10.192.1 {
             next-hop-vrf INTERNET
         }
     }
     route 10.10.195.0/28 {
         next-hop 10.10.195.2 {
             next-hop-vrf INTERNAL
         }
     }
     route 10.10.196.0/28 {
         next-hop 10.10.196.2 {
             next-hop-vrf INTERNET
         }
     }
 }
 vrf INTERNAL {
     static {
         route 0.0.0.0/0 {
             next-hop 10.10.195.2 {
             }
         }
         route 10.10.0.0/17 {
             next-hop 10.10.193.2 {
             }
         }
     }
 }
 vrf INTERNET {
     static {
         route 0.0.0.0/0 {
             next-hop 10.10.192.1 {
             }
         }
         route 10.10.0.0/17 {
             next-hop 10.10.196.2 {
             }
         }
         route 10.10.192.2/32 {
             next-hop 10.10.192.2 {
                 next-hop-vrf default
             }
         }
         route 10.10.193.0/28 {
             next-hop 10.10.193.2 {
                 next-hop-vrf INTERNAL
             }
         }
     }
 }

Then I can ping an internet ip:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=29.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=29.5 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=29.5 ms

but can’t ping an domain: Temporary failure in name resolution.

tcpdump shows Router1 response with icmp port unreachable:

tcpdump -i eth0 icmp or port 53  -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:22:00.500385 IP 10.10.192.2.56306 > 202.96.209.5.53: 19159+ A? www.vyos.io. (29)
02:22:00.503285 IP 202.96.209.5.53 > 10.10.192.2.56306: 19159 3/0/0 CNAME cname.vercel-dns.com., A 76.76.21.61, A 76.76.21.241 (95)
02:22:00.503357 IP 10.10.192.2 > 202.96.209.5: ICMP 10.10.192.2 udp port 56306 unreachable, length 131
02:22:05.505522 IP 10.10.192.2.42130 > 202.96.209.133.53: 19159+ A? www.vyos.io. (29)
02:22:05.512819 IP 202.96.209.133.53 > 10.10.192.2.42130: 19159 3/0/0 CNAME cname.vercel-dns.com., A 76.76.21.9, A 76.76.21.241 (95)
02:22:05.512885 IP 10.10.192.2 > 202.96.209.133: ICMP 10.10.192.2 udp port 42130 unreachable, length 131

02:22:14.356836 IP 10.10.192.2 > 8.8.8.8: ICMP echo request, id 26514, seq 1, length 64
02:22:14.386432 IP 8.8.8.8 > 10.10.192.2: ICMP echo reply, id 26514, seq 1, length 64
02:22:15.358643 IP 10.10.192.2 > 8.8.8.8: ICMP echo request, id 26514, seq 2, length 64
02:22:15.388070 IP 8.8.8.8 > 10.10.192.2: ICMP echo reply, id 26514, seq 2, length 64
02:22:16.359698 IP 10.10.192.2 > 8.8.8.8: ICMP echo request, id 26514, seq 3, length 64
02:22:16.388702 IP 8.8.8.8 > 10.10.192.2: ICMP echo reply, id 26514, seq 3, length 64

I can’t even ping itself in router1:

ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.192.2  netmask 255.255.255.240  broadcast 10.10.192.15
        inet6 fe80::4262:31ff:fe15:a987  prefixlen 64  scopeid 0x20<link>
        ether 40:62:31:15:a9:87  txqueuelen 1000  (Ethernet)
        RX packets 731397  bytes 239871423 (228.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 714112  bytes 159469861 (152.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdf500000-df57ffff  

vyos@vyos:/tmp$ 
vyos@vyos:/tmp$ ping 10.10.192.2
PING 10.10.192.2 (10.10.192.2) 56(84) bytes of data.
^C
--- 10.10.192.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 38ms

vyos@vyos:/tmp$ ping 10.10.192.1
PING 10.10.192.1 (10.10.192.1) 56(84) bytes of data.
64 bytes from 10.10.192.1: icmp_seq=1 ttl=64 time=0.312 ms
^C
--- 10.10.192.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.312/0.312/0.312/0.000 ms

What should I do to make Router1 self can access internet?
Thanks!

8

If the drawing is accurate then all lans seems to use 10.10.0.1 as their default gateway. The L3-switch then routes these packets towards vyos-router1 who then routes to vyous-router2 which then routes back to vyos-router1 and finally you have the ISP at eth0 on vyos-router1. This gives that the only place you need to apply NAT (source NAT) is at outgoing traffic on eth0 at vyos-router1 as long as you put eth1+eth2 in VRF INTERNAL and eth0+eth3 in VRF INTERNET.

What VyoS-version do you use?

Your VRF-config should look something like:

vrf {
    name INTERNAL {
        protocols {
            static {
                route 0.0.0.0/0 {
                    next-hop 10.10.195.2 {
                        distance 1
                    }
                }
                route 10.10.0.0/17 {
                    next-hop 10.10.193.2 {
                        distance 1
                    }
                }
            }
        }
        table 100
    }
    name INTERNET {
        protocols {
            static {
                route 0.0.0.0/0 {
                    next-hop <whatever IP your ISP says is their gw> {
                        distance 1
                    }
                }
                route 10.10.0.0/17 {
                    next-hop 10.10.196.2 {
                        distance 1
                    }
                }
                route 10.10.193.0/28 {
                    next-hop 10.10.196.2 {
                        distance 1
                    }
                }
            }
        }
        table 101
    }
}

That is you must define unique table number for each vrf.

Then for the INTERNAL vrf you have default towards vyos-router2 but you must also define the client IP-range towards switch as nexthop.

Then for INTERNET vrf you have default towards ISP gw but you must also define the client IP-range towards vyos-router2 along with the linknet you use between vyos-router1 and switch (vrf INTERNAL) so when the switch itself will try to do ntp, dns and whatelse its traffic will also flow the same path as the client-traffic.

And finally you attach each interface to a single VRF like so:

    ethernet eth0 {
        description ISP
        duplex auto
        hw-id xx:xx:xx:xx:xx:xx
        speed auto
        vrf INTERNET
    }
    ethernet eth1 {
        description SWITCH
        duplex auto
        hw-id xx:xx:xx:xx:xx:xx
        speed auto
        vrf INTERNAL
    }
    ethernet eth2 {
        description VYOS-ROUTER-2
        duplex auto
        hw-id xx:xx:xx:xx:xx:xx
        speed auto
        vrf INTERNAL
    }
    ethernet eth3 {
        description VYOS-ROUTER-2
        duplex auto
        hw-id xx:xx:xx:xx:xx:xx
        speed auto
        vrf INTERNET
    }

The above gives that the only point in your network where you must do source NAT (masquerade) is at eth0 on vyos-router1.

9

Thanks!

I’m using vyos 1.3.

Actually, there is another gateway above router1’s eth0. so router1 does not source nat:

image
image984×595 16 KB

  1. I can’t set table number in vrf xxx
vyos@vyos# set protocols vrf INTERNAL 
Possible completions:
 > static       Static route parameters
  1. I add a source to router1:
vyos@vyos# show nat
 source {
     rule 1 {
         outbound-interface eth0
         source {
             address 0.0.0.0/0
         }
         translation {
             address masquerade
         }
     }
 }
[edit]

But still I can ping a IP and udp or tcp communication will fail:

root@vyos:/tmp# nslookup qq.com
^C


root@vyos:~# tcpdump -i eth0 port 53 or icmp -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:00:06.848778 IP 10.10.192.2.35992 > 202.96.209.5.53: 45715+ A? qq.com. (24)
08:00:07.552875 IP 202.96.209.5.53 > 10.10.192.2.35992: 45715 3/0/0 A 183.3.226.35, A 123.151.137.18, A 61.129.7.47 (72)
08:00:07.552945 IP 10.10.192.2 > 202.96.209.5: ICMP 10.10.192.2 udp port 35992 unreachable, length 108

Following is my protocols configuration:

vyos@vyos# show protocols 
 static {
     route 0.0.0.0/0 {
         next-hop 10.10.192.1 {
             next-hop-vrf INTERNET
         }
     }
 }
 vrf INTERNAL {
     static {
         route 0.0.0.0/0 {
             next-hop 10.10.195.2 {
             }
         }
         route 10.10.0.0/17 {
             next-hop 10.10.193.2 {
             }
         }
         route 10.10.196.0/28 {
             next-hop 10.10.196.2 {
                 next-hop-vrf INTERNET
             }
         }
     }
 }
 vrf INTERNET {
     static {
         route 0.0.0.0/0 {
             next-hop 10.10.192.1 {
             }
         }
         route 10.10.0.0/17 {
             next-hop 10.10.196.2 {
             }
         }
         route 10.10.192.2/32 {
             next-hop 10.10.192.2 {
                 next-hop-vrf default
             }
         }
         route 10.10.193.0/28 {
             next-hop 10.10.193.2 {
                 next-hop-vrf INTERNAL
             }
         }
     }
 }
[edit]