Getting Policy-Based Routing to work with IP AND Interface-based routing.

Hello,

I have been struggling for days with setting ip policy-based routing the way I want it.

Objective:
Getting traffic to be sent out of different interface depending on IP origin, with the following pitfalls:
I have 2 networks:
LAN (network 192.168.0.0/24) with gateway being interface eth1 on the router (192.168.0.254)
DMZ (172.22.0/24), with gateway being interface eth2 on the router (172.22.0.254)
I have 3 different interfaces with a public IP (replaced below with private addresses), that should be used for incoming traffic and outgoing traffic. They all belong to the same network and have the same gateway IP (say 10.0.0.62)
eth0 - 10.0.0.10/26
eth3 - 10.0.0.11/26
eth4 - 10.0.0.12/26
With current topology:
my machines on DMZ and LAN use standard network configuration, with 192.168.0.254 for LAN gateway and 172.22.0.254 for DMZ gateway
I have some source nat configured on the router for the LAN machines
Also DMZ devices should be able to access LAN devices (per current design).
the DMZ devices will first hit the eth2 interface / 172.22.0.254 as this is their gateway, but I want the router to then route the traffic through a different interface depending on the source IP, such as:
IPDMZ1 => eth3
IPDMZ2 => eth4
the LAN devices arrive on eth1 gateway and should then be sent out through eth0
My main question (maybe I should have started with that) is: How do I specify in VyOS policy routing / protocols route table that a specific route table should go through a specific interface?

I got it somewhat working after many hours, however my solution is not reboot persistent because I get wrong default routes for my custom route tables when setting this up with VyOS CLI.

The idea so far has been to set up route policy for eth2, associated with different custom route tables to send traffic towards the right output interfac

For example:

Code:

route PUBLIC {
       rule 20 {
           destination {
               address 0.0.0.0/0
           }
           set {
               table 4
           }
           source {
               address 172.22.0.100/32
           }
       }
       rule 21 {
           destination {
               address 0.0.0.0/0
           }
           set {
               table 2
           }
           source {
               address 172.22.0.101/32
           }
       }
       rule 30 {
           destination {
               address 192.168.51.0/24
           }
           set {
               table 10
           }
           source {
               address 172.22.0.0/24
           }
       }
   }
[...]

protocols {
   static {
       interface-route 192.168.0.0/24 {
           next-hop-interface eth1 {
           }
       }
       route 192.168.0.0/24 {
           next-hop 192.168.0.254 {
           }
       }
       table 1 {
           interface-route 10.0.0.0/26 {
               next-hop-interface eth0 {
               }
           }
           interface-route 192.168.0.0/24 {
               next-hop-interface eth1 {
               }
           }
           route 0.0.0.0/0 {
               next-hop 10.0.0.62 {
               }
           }
       }
       table 2 {
           interface-route 10.0.0.0/26 {
               next-hop-interface eth3 {
               }
           }
           interface-route 192.168.0.0/24 {
               next-hop-interface eth1 {
               }
           }
           route 0.0.0.0/0 {
               next-hop 10.0.0.62 {
               }
           }
       }
       table 3 {
           interface-route 10.0.0.0/26 {
               next-hop-interface eth4 {
               }
           }
           interface-route 192.168.0.0/24 {
               next-hop-interface eth1 {
               }
           }
           route 0.0.0.0/0 {
               next-hop 10.0.0.62 {
               }
           }
       }
       table 10 {
           interface-route 192.168.0.0/24 {
               next-hop-interface eth1 {
               }
           }
       }
       table 11 {
           interface-route 172.22.0.0/24 {
               next-hop-interface eth2 {
               }
           }
       }
   }

[...]

ethernet eth2 {
       address 172.22.0.254/24
       duplex auto
       hw-id xx:xx:xx:46:1d:21
       policy {
           route PUBLIC
       }
       smp_affinity auto
       speed auto
   }

I hope you get the idea of what I am trying to achieve.
But I end up with wrong default routes, that I can see from ip command in the form of:

Code:

# ip route sh table all | grep default
default via 10.0.0.62 dev eth0 table 1
default via 10.0.0.62 dev eth0 table 2
default via 10.0.0.62 dev eth0 table 3

So I manually change this to:
Code:

# ip route sh table all | grep default
default via 10.0.0.62 dev eth0 table 4
default via 10.0.0.62 dev eth3 table 2
default via 10.0.0.62 dev eth4 table 3

And then it works.

I have extra settings which may or may not explain why my route tables end up not the way I want, such as system gateway set up for eth0. I have been fiddling with those but with no results so far.