DNAT with multiwan and policy routing, incoming connections only work on primary interface

Great.
Since you are matching source|destination addresses|ports, you should carefully analyze all patterns. Because in your example you are receiving connection to port 80 through one interface, and answer is delivered through other interface.
I’ll try to prepare a simplified example for this requirements

1 Like

Thanks that would be extremely helpful; for others to follow as well.

I tried a fresh install on 1.3 with minimal rules and I still could not get incoming DNAT on eth1 or eth2. I am now on 1.4 rolling.

Attached is the config I am working with currently. Incoming local connections (tested with ssh) on eth0, eth1, and eth2 work. Incoming connections on eth0 forwarded to 10.0.10.1 on port 80 work, but eth1 and eth2 do not.

config.boot.txt (76.2 KB)

Configuration B - Marking incoming packets on eth0/eth1/eth2

Packet arrives on eth1 from s.s.s.s and attempts to leave eth1 for 10.0.10.1

To explain why: That’s because tables 100 101 and 102 probably don’t have connected routes in them, only default route.
But no need to add those routes to the tables, PBR set table command should be applied to LAN not WAN interface

Here is a simple example. Hope all requirements are covered.
First of all, this lab differs from your config, and there is no active firewall.
Topology and requirements for this lab:

Router config:

set firewall group address-group GOOGLE-DNS address '8.8.8.8'
set firewall group address-group GOOGLE-DNS address '8.8.4.4'
set firewall group port-group WEB-PORTS port '80'
set firewall group port-group WEB-PORTS port '443'

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'WAN-01'
set interfaces ethernet eth1 address 'dhcp'
set interfaces ethernet eth1 description 'WAN-02'
set interfaces ethernet eth2 address 'dhcp'
set interfaces ethernet eth2 description 'WAN-03'
set interfaces ethernet eth3 address '192.168.200.1/24'
set interfaces ethernet eth3 address '192.168.100.1/24'
set interfaces ethernet eth3 description 'LAN'
set interfaces ethernet eth3 policy route 'POL-LAN'

set nat destination rule 10 description 'DST NAT 80 - WAN01'
set nat destination rule 10 destination port '80'
set nat destination rule 10 inbound-interface 'eth0'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 translation address '192.168.100.2'
set nat destination rule 20 description 'DST NAT 80 - WAN02'
set nat destination rule 20 destination port '80'
set nat destination rule 20 inbound-interface 'eth1'
set nat destination rule 20 protocol 'tcp'
set nat destination rule 20 translation address '192.168.100.2'
set nat destination rule 30 description 'DST NAT 80 - WAN03'
set nat destination rule 30 destination port '80'
set nat destination rule 30 inbound-interface 'eth2'
set nat destination rule 30 protocol 'tcp'
set nat destination rule 30 translation address '192.168.100.2'
set nat source rule 10 description 'MASQ WAN01'
set nat source rule 10 outbound-interface 'eth0'
set nat source rule 10 translation address 'masquerade'
set nat source rule 20 description 'MASQ WAN02'
set nat source rule 20 outbound-interface 'eth1'
set nat source rule 20 translation address 'masquerade'
set nat source rule 30 description 'MASQ WAN03'
set nat source rule 30 outbound-interface 'eth2'
set nat source rule 30 translation address 'masquerade'

set policy route POL-LAN rule 10 destination group address-group 'GOOGLE-DNS'
set policy route POL-LAN rule 10 set table '102'
set policy route POL-LAN rule 10 source address '192.168.100.2/32'
set policy route POL-LAN rule 20 description 'Server to 55 through WAN-03'
set policy route POL-LAN rule 20 destination address '5.5.5.55/32'
set policy route POL-LAN rule 20 set table '103'
set policy route POL-LAN rule 20 source address '192.168.100.2/32'
set policy route POL-LAN rule 30 description 'Server - default thorugh WAN-01'
set policy route POL-LAN rule 30 set table '101'
set policy route POL-LAN rule 30 source address '192.168.100.2/32'
set policy route POL-LAN rule 110 description 'Net 100/24 through WAN-01'
set policy route POL-LAN rule 110 set table '101'
set policy route POL-LAN rule 110 source address '192.168.100.0/24'
set policy route POL-LAN rule 210 description 'Net200 - web through WAN-01'
set policy route POL-LAN rule 210 destination group port-group 'WEB-PORTS'
set policy route POL-LAN rule 210 protocol 'tcp'
set policy route POL-LAN rule 210 set table '101'
set policy route POL-LAN rule 210 source address '192.168.200.0/24'
set policy route POL-LAN rule 220 description 'Net200 throguh WAN-02'
set policy route POL-LAN rule 220 set table '102'
set policy route POL-LAN rule 220 source address '192.168.200.0/24'

set protocols static table 101 route 0.0.0.0/0 dhcp-interface 'eth0'
set protocols static table 102 route 0.0.0.0/0 dhcp-interface 'eth1'
set protocols static table 103 route 0.0.0.0/0 dhcp-interface 'eth2'

set service ssh

  • DNAT working on three WANs
  • All requirements pointed in the graph were tested.
  • Remote SSH access to router from “internet” working on three WANs.

Hope you find this useful!

Thank you. I will test this tonight. Can I ask if you tested this on 1.3 or 1.4, or both?

Tested on VyOS 1.3.1-S1

I need to review config and make some corrections, because its working but not as it should. If possible, later I’ll post corrections

1 Like

Thanks so much. I tested this configuration and it is not working. In comparing to my example config.boot above it doesn’t appear different when I remove the local routes and disable firewall and load-balance, which doesn’t work. This could be a 1.4 issue. I will test on 1.3

One difference between our configurations is that my configuration has a default route out WAN1. Would that have an impact on preventing incoming port 80 DNAT connections on eth1 and eth2 from leaving on the proper interface?

set protocols static route 0.0.0.0/0 interface eth0 distance '200'

Also, I noticed that the load-balance has the sticky-connections inbound option. Does that impact all incoming connections and might it help what I am looking to do?

set load-balancing wan sticky-connections inbound

Yes, I would suggest moving forward to wan load balance and sticky connections.
Desired setup using purely PBR requires a more advanced rule processing on firewall (mangle)

Interesting - very much looking forward to seeing how this is possible, either with wan load balance or custom nftable rules. I tried wan load balance and it also did not work for me. I am somewhat relieved to learn that this is not trivial to configure - the examples online make this seem very easy.

I attempted this configuration with wan-load-balance but for some reason the packets are not being marked properly - load balance is ignored and incoming DNAT on eth1 and eth2 still do not work. Very strange. This is VyOS 1.4-rolling-202204230217

me@m:~$ show wan-load-balance status
Chain WANLOADBALANCE_PRE (1 references)
 pkts bytes target     prot opt in     out     source               destination
   22  1848 ISP_eth1   all  --  eth4   *       172.16.80.0/24       24.29.97.0/24        state NEW
    0     0 CONNMARK   all  --  eth4   *       172.16.80.0/24       24.29.97.0/24        CONNMARK restore
   60  5040 ISP_eth0   all  --  eth4   *       172.16.80.0/24       163.182.128.0/24     state NEW
    0     0 CONNMARK   all  --  eth4   *       172.16.80.0/24       163.182.128.0/24     CONNMARK restore
    0     0 ISP_eth0   all  --  eth4   *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth4   *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore

Here is a simple example. Hope all requirements are covered.

  • DNAT working on three WANs

When testing, make sure not to test DNAT from direct connected networks on eth0 eth1 and eth2. As

Can confirm did not test from any directly connected network

Just an update with this - I cannot get incoming connections to work properly with multiple WANs on VyOS, 1.3 or 1.4. I had to use iptables directly to mark the incoming packets and place them in the appropriate ip rule tables. My guess is this is an underutilized feature and so receives little testing.

It’s really a pity.

I had a very similar setup with 2 WAN connections (dymanic address, PPPoE). I can’t use the multi-wan load-balance feature because I’m using BGP dynamic routing for interconnection between sites. There are some services deployed on the LAN side and I definitely wanted to make both WAN addresses to be accessible to form a inbound failover setup. However, it seems not achievable using current versions of VyOS.

You must use local-route policy: https://docs.vyos.io/en/equuleus/configuration/policy/index.html#local-route