Routing policy not working correctly

General questions
1

Hi all! Was endlessly looking for some help with an issue I’m facing in vyos, and thankfully came across this forum.

I’m facing an issue with policy routing
The objective is to route https traffic out through a different interface (eth2), and for any other traffic to route via eth1.

What I’ve got so far:

rule 1 {
         destination {
             address 0.0.0.0/0
             port https
         }
        
         protocol tcp
         set {
             table 103 //next-hop interface eth2
         }
     }

 rule 2 {
         destination {
             address 0.0.0.0/0
         }
         protocol all
         set {
             table 102 //next-hop gateway of eth1
         }
         source {
             address 172.16.0.0/24 //internal range for VM's
         }
     }

 table 102 {
     route 0.0.0.0/0 {
         next-hop x.x.x.x (hidden for thread) {
         }
     }
 }
 table 103 {
     interface-route 0.0.0.0/0 {
         next-hop-interface eth2 {
         }
     }
 }

The problem: VM’s trying to access https:// URL’s (either through wget, curl etc.) frequently see “no route to host” - if they keep trying the same command various times it eventually does work - Why is this happening? Is there something wrong with the rules above?

Workaround: I’ve just changed the policy to route everything through table 102 (i.e. all traffic via eth1), but as mentioned earlier I’d like https traffic to go out via eth2.

Thanks in advance :slight_smile:

2

I’ve done similar test, and it seems to be working as expected.
While doing curl/wget, you can do tcpdump in outgoing interface, and see if connection is being routed properly, and if response is received properly.
For example, this is a tcpdump:

vyos@vyos# sudo tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:48:50.527677 IP 10.1.1.2.55946 > 142.250.79.164.https: Flags [S], seq 1209830650, win 64240, options [mss 1460,sackOK,TS val 2382374897 ecr 0,nop,wscale 5], length 0
16:48:50.559336 IP 142.250.79.164.https > 10.1.1.2.55946: Flags [S.], seq 930274370, ack 1209830651, win 65535, options [mss 1412,sackOK,TS val 2232838053 ecr 2382374897,nop,wscale 8], length 0
16:48:50.560484 IP 10.1.1.2.55946 > 142.250.79.164.https: Flags [.], ack 1, win 2008, options [nop,nop,TS val 2382374930 ecr 2232838053], length 0
16:48:50.563003 IP 10.1.1.2.55946 > 142.250.79.164.https: Flags [P.], seq 1:518, ack 1, win 2008, options [nop,nop,TS val 2382374933 ecr 2232838053], length 517
16:48:50.594363 IP 142.250.79.164.https > 10.1.1.2.55946: Flags [.], ack 518, win 261, options [nop,nop,TS val 2232838088 ecr 2382374933], length 0
16:48:50.616005 IP 142.250.79.164.https > 10.1.1.2.55946: Flags [.], seq 1:1401, ack 518, win 261, options [nop,nop,TS val 2232838110 ecr 2382374933], length 1400
16:48:50.616005 IP 142.250.79.164.https > 10.1.1.2.55946: Flags [.], seq 1401:2801, ack 518, win 261, options [nop,nop,TS val 2232838110 ecr 2382374933], length 1400

Relevant config for reference:

vyos@vyos:~$ show config comm | grep policy
set interfaces ethernet eth3 policy route 'HTTPS-FILTER'             # eth3 is LAN interface, where linux host is located
set policy route HTTPS-FILTER rule 10 destination address '0.0.0.0/0'
set policy route HTTPS-FILTER rule 10 destination port 'https'
set policy route HTTPS-FILTER rule 10 protocol 'tcp'
set policy route HTTPS-FILTER rule 10 set table '103'

vyos@vyos:~$ show config comm | grep static
set protocols static route 0.0.0.0/0 next-hop 192.168.122.1
set protocols static table 103 route 0.0.0.0/0 next-hop 10.1.1.1

vyos@vyos:~$ show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.122.240/24                u/u  GRAL WAN
eth1             10.1.1.2/30                       u/u  HTTPS WAN
eth2             -                                 u/D  
eth3             192.168.50.1/24                   u/u  
lo               127.0.0.1/8                       u/u  
                 ::1/128                                
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] via 192.168.122.1, eth0, weight 1, 00:30:42
C>* 10.1.1.0/30 is directly connected, eth1, 00:30:43
C>* 192.168.50.0/24 is directly connected, eth3, 00:30:43
C>* 192.168.122.0/24 is directly connected, eth0, 00:30:43

vyos@vyos:~$ show ip route table 103
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 103:
S>* 0.0.0.0/0 [1/0] via 10.1.1.1, eth1, weight 1, 00:19:04

Also, in lab I have nat source configured

3

Why do you need “next-hop-interface eth2” ?
That will only work if there is a router on eth2 subnet, doing proxy arp

If eth2 is dhcp client, either use some script on ifup to setup next-hop address.

Or set distance on default route to some bad value, so dhcp interface route is best in main route table . Then you need a policy routes for eth1 traffic, and you can use main route table when you want to route via dhcp interface

4

Thanks, what if the interface you want https traffic to go out of is setup with DHCP? I essentially want all https traffic going via eth2 (DHCP)

5

I set it to next-hop-interface eth2 since eth2 is a dhcp client - we look to get new IPs every so often and always went https traffic going out of that. I haven’t heard of the ifup script solution you’ve mentioned before, is this the only way?