When i am doing policy based routing for one of my source ip(isp2 subnet) to ISP2, trace is going on both ISP1 & ISP2 from all the source IP’s (ISP1 & ISP2)
Below is the configuration done…
set policy route AppTraffic rule 1000 source add 38.x.x.x
set policy route AppTraffic rule 1000 set table 3
set protocols static table 3 route 0.0.0.0/0 next-hop 38.x.x.x7
set interfaces ethernet eth0 policy route AppTraffic
There is something missing from the information. Do you route public IP’s (without NAT)? Could you give the (perhaps anonymized network ranges)? (You can replace the first two bytes with x1.y1 and so on, but being consistent.)
If you do not use NAT, then then in source you should add that range of public IP’s you are routing that you want to go this separate way.
On the other side if you do use NAT, then the source IP you have to use in source should be the private one, not the public one you NAT onto.
Are you perhaps using some other features like WAN-load-balancing?
Could you paste what the sudo ip rule show and sudo ip route show table all output? (Perhaps anonymized, but carefully not to break the routing semantic of “most specific route”.)
VYOS-ISP:~$ sudo ip rul
0: from all lookup local
32763: from all fwmark 0x3 lookup 3
32764: from all fwmark 0x2 lookup 2
32765: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
VYOS-ISP# run sh ip route table 3
table 3:
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route
S>* 0.0.0.0/0 [1/0] via 3x.x.11.x7, eth1.300
[edit]
VYOS-ISP#
set policy route AppTraffic rule 1000 destination address 0.0.0.0/0
set policy route AppTraffic rule 1000 source add 3x.x.189.0/24
set policy route AppTraffic rule 1000 set table 3
set protocols static table 3 route 0.0.0.0/0 next-hop 3x.x.11.x7
set interfaces ethernet eth0 policy route AppTraffic
We are using wan load balancing only for interface health monitor
VYOS-ISP:~$ sh wan-load-balance
Interface: eth1.300
Status: active
Last Status Change: Thu Jul 12 16:20:54 2018
+Test: ping Target: 3x.x.11.x7
Last Interface Success: 1s
Last Interface Failure: n/a
# Interface Failure(s): 0
Interface: eth2.100
Status: active
Last Status Change: Thu Jul 12 16:20:54 2018
+Test: ping Target: 7x.x.0.85
Last Interface Success: 1s
Last Interface Failure: n/a
# Interface Failure(s): 0
Interface: eth2.200
Status: active
Last Status Change: Thu Jul 12 16:20:54 2018
+Test: ping Target: 7x.x.0.65
Last Interface Success: 1s
Last Interface Failure: n/a
# Interface Failure(s): 0
Well that’s the most probable cause, i.e. the WAN-load-balancing, because just like policy routing it uses iptables -t mangle rules to mark the packets with a given routing table (and this marking happens before the policy routing ones). (Therefore the actual order of your rules doesn’t matter, unless those tables are empty…)
It doesn’t even help if you exclude traffic from the WAN-load-balancing feature, as this “breaks” the policy routing by not letting those packets get marked.
Could you paste the output of sudo iptables -t mangle -S? (If you see any -j ACCEPT before your policy routing rules, then the WAN-load-balancing is causing your issues…)
I’ll point you to a previous post of mine that tackles the issue with WAN-load-balancing and its integration with the rest of VyOS features. (Even if you are not using WAN-load-balancing, the custom script I wrote, “fixes” the -j ACCEPT issue.)
Are you sure this is the entire output? (Have you truncated something?)
I’m asking because you have some iptables chains (i.e. from policy routing “tables”) that don’t seem to be actually “jumped to” anywhere…
Bellow is the re-ordered set of rules as the kernel passes through them. At the first encounter of -j ACCEPT any other rules are ignored, therefore any --set-mark (or related) will be ignored. (And this --set-mark is the one determining which routing table will be checked against.)
(Things marked with ???? seem out of place, see the !!!! for the problem.)
-A PREROUTING -j VYATTA_FW_IN_HOOK
???? -A VYATTA_FW_IN_HOOK -i eth0 -j cogenttraffic ## this chain is empty what is its purpose?
-A PREROUTING -j WANLOADBALANCE_PRE
-A WANLOADBALANCE_PRE -i eth0 -m state --state NEW -m statistic --mode random --probability 0.33333300008 -j ISP_eth1.300
-A ISP_eth1.300 -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth1.300 -j MARK --set-xmark 0x1/0xffffffff
-A ISP_eth1.300 -j ACCEPT
-A WANLOADBALANCE_PRE -i eth0 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth2.100
-A ISP_eth2.100 -j CONNMARK --set-xmark 0x2/0xffffffff
-A ISP_eth2.100 -j MARK --set-xmark 0x2/0xffffffff
-A ISP_eth2.100 -j ACCEPT
-A WANLOADBALANCE_PRE -i eth0 -m state --state NEW -j ISP_eth2.200
-A ISP_eth2.200 -j CONNMARK --set-xmark 0x3/0xffffffff
-A ISP_eth2.200 -j MARK --set-xmark 0x3/0xffffffff
-A ISP_eth2.200 -j ACCEPT
!!!! nothing will get passed this rule for `-i eth0`
-A WANLOADBALANCE_PRE -i eth0 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
???? this is not jumped in anywhere
-A VYATTA_PBR_3 -j MARK --set-xmark 0x80000002/0xffffffff
-A VYATTA_PBR_3 -j ACCEPT
???? this is not jumped in anywhere
-A AppTraffic -s 3x.x.189.0/24 -m comment --comment AppTraffic-1000 -j VYATTA_PBR_3
-A AppTraffic -m comment --comment “AppTraffic-10000 default-action accept” -j RETURN