SNAT Rules generated by Load-Balancing

Hi !

Q to developers - is it possible somehow review/list source NAT rules generated by load-balancing? “Run show configuration commands” don’t list them. Looks like they are interfere with my own rules, causing glitches and artifacts. Its possible to disable them, yet I would prefer to keep originals and modify my own.

From VyOS wiki
https://wiki.vyos.net/wiki/WAN_load_balancing#Source_NAT_rules

Per default, interfaces used in a load balancing pool replace the source IP of each outgoing packet with its own address to ensure that replies arrive on the same interface. This works through automatically generated source NAT (SNAT) rules, these rules are only applied to balanced traffic. In cases where this behavior is not desired, the automatic generation of SNAT rules can be disabled:

# set load-balancing wan disable-source-nat

Thanks in advance.

Strictly replying to your question you can “review” (i.e. view) the iptables rules inserted by the load-balancing feature by using the following command:

iptables -t mangle -S

The chains introduced by this feature start with WANLOADBALANCE_.


However I also struggled with this feature myself and my solution was to use a custom script that “adapts” these rules after the load-balancing feature generates them. I’ve described my experience (and the solution) in the following thread. (It was triggered by the TCP-MSS clamping issue, but the underlying cause and solution is the same, namely poor integration between the load-balance feature an the rest of the VyOS routing features.)

These NAT reflection (hairpin) rules causes link to ISP #1 (eth0) behave in really awkward way - upload speed is just several KB/sec.
eth0 - ISP #1, eth1 - ISP #2, eth2 - DMZ, eth3 - Internal zone

set load-balancing wan rule 20 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 20 'exclude'
set load-balancing wan rule 20 inbound-interface 'eth3'
set load-balancing wan rule 20 destination address 'isp1.ip1.xx.xx'
set load-balancing wan rule 20 protocol all

set load-balancing wan rule 21 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 21 'exclude'
set load-balancing wan rule 21 inbound-interface 'eth3'
set load-balancing wan rule 21 destination address 'isp2.ip1.yy.yy'
set load-balancing wan rule 21 protocol all

set load-balancing wan rule 22 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 22 'exclude'
set load-balancing wan rule 22 inbound-interface 'eth2'
set load-balancing wan rule 22 destination address 'isp1.ip1.xx.xx'
set load-balancing wan rule 22 protocol all

set load-balancing wan rule 23 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 23 'exclude'
set load-balancing wan rule 23 inbound-interface 'eth2'
set load-balancing wan rule 23 destination address 'isp2.ip1.yy.yy'
set load-balancing wan rule 23 protocol all

set nat source rule 150 description 'NAT Reflection (Hairpin NAT) 192.168.0.0/24'
set nat source rule 150 destination address '192.168.0.0/24'
set nat source rule 150 source address '192.168.0.0/24'
set nat source rule 150 outbound-interface 'eth3'
set nat source rule 150 translation address 'masquerade'

set nat source rule 151 description 'NAT Reflection (Hairpin NAT) 192.168.1.0/24'
set nat source rule 151 destination address '192.168.1.0/24'
set nat source rule 151 source address '192.168.1.0/24'
set nat source rule 151 outbound-interface 'eth2'
set nat source rule 151 translation address 'masquerade'

Could you please also post the iptables rules present in nat and mangle tables? (You can obtain these by running sudo iptables -t nat -S and sudo iptables -t mangle -S.)


I don’t think that the load-balancing rules 21 through 23 actually cause any issues (with regard to WAN traffic). My suspicion is that you use another feature which inserts -j ACCEPT rules in either PREROUTING or POSTROUTING chains and thus interfere with either load-balancing or NAT.

As for the NAT rules, it seems that the destination and source addresses are the same, which doesn’t make much sense. Shouldn’t they be “symmetric” as I assume that you want to use those to be able to route between eth2 and eth3 and apply NAT between them.

Thanks a lot for help! Please find below.
192.168.1.3 is a DMZ server.
192.168.0.xx:8888 is local PC with one port natted from ouside for testing purposes. It will be removed after setup is complete.

# **************************************

vyos@linrt-aio:~$ sudo iptables -t nat -S 
-P PREROUTING ACCEPT                                                                                                                                         
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N VYATTA_PRE_DNAT_HOOK
-N VYATTA_PRE_SNAT_HOOK
-N WANLOADBALANCE
-A PREROUTING -j VYATTA_PRE_DNAT_HOOK
-A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-510 -j DNAT --to-destination 192.168.0.xxx:8888
-A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-511 -j DNAT --to-destination 192.168.0.xxx:8888
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-700 -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-710 -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport xxx1 -m comment --comment DST-NAT-720 -j DNAT --to-destination 192.168.1.3:yy1
-A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport xxx2 -m comment --comment DST-NAT-721 -j DNAT --to-destination 192.168.1.3:yy2
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport xxxx -m comment --comment DST-NAT-722 -j DNAT --to-destination 192.168.1.3:8080
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport xxxx -m comment --comment DST-NAT-723 -j DNAT --to-destination 192.168.1.3:8080
-A POSTROUTING -j VYATTA_PRE_SNAT_HOOK
-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 -o eth3 -m comment --comment SRC-NAT-150 -j MASQUERADE
-A VYATTA_PRE_DNAT_HOOK -j RETURN
-A VYATTA_PRE_SNAT_HOOK -j WANLOADBALANCE
-A VYATTA_PRE_SNAT_HOOK -j RETURN
-A WANLOADBALANCE -m connmark --mark 0x1 -j SNAT --to-source isp1.ip1.xx.xx
-A WANLOADBALANCE -m connmark --mark 0x2 -j SNAT --to-source isp2.ip2.yy.yy

# **************************************

vyos@linrt-aio:~$ sudo iptables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ISP_eth0
-N ISP_eth0_IN
-N ISP_eth1
-N ISP_eth1_IN
-N WANLOADBALANCE_PRE
-A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN
-A PREROUTING -i eth0 -m state --state NEW -j ISP_eth0_IN
-A PREROUTING -j WANLOADBALANCE_PRE
-A ISP_eth0 -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j MARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j ACCEPT
-A ISP_eth0_IN -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth1 -j CONNMARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j MARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j ACCEPT
-A ISP_eth1_IN -j CONNMARK --set-xmark 0x2/0xffffffff
-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth3 -j ACCEPT
-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth3 -j ACCEPT
-A WANLOADBALANCE_PRE -d 192.168.0.0/16 -i eth+ -j ACCEPT
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -j ISP_eth1
-A WANLOADBALANCE_PRE -i eth2 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -j ISP_eth1
-A WANLOADBALANCE_PRE -i eth3 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

# **************************************

Hairpin (NAT reflection) rules are working, however, they disrupt norml traffic flow behavior as I described in my post. This video have some brief explanation:

Additionally, these rules permit traffic between DMZ/Local.

set load-balancing wan rule 30 exclude
set load-balancing wan rule 30 inbound-interface eth+
set load-balancing wan rule 30 destination address '192.168.0.0/16'
set load-balancing wan rule 30 protocol all

I had more detailed definition before but its seems its not necessary after all.

set load-balancing wan rule 30 'exclude'
set load-balancing wan rule 30 inbound-interface 'eth3'
set load-balancing wan rule 30 source address '192.168.0.0/24'
set load-balancing wan rule 30 destination address '192.168.1.0/24'
set load-balancing wan rule 30 protocol 'all'

set load-balancing wan rule 32 'exclude'
set load-balancing wan rule 32 inbound-interface 'eth2'
set load-balancing wan rule 32 source address '192.168.1.0/24'
set load-balancing wan rule 32 destination address '192.168.0.0/24'
set load-balancing wan rule 32 protocol 'all'

I tried to “reorder” the iptables rules so that one can more easily “follow” through them and pasted them at the end. (I eliminated all -j SOME_CHAIN that don’t have additional conditions, and I’ve reordered the rules in the same order they are evaluated by iptables. I have also eliminated -j RETURN, and copy-pasted some chains that are called more than twice.)

From what I gather there shouldn’t be an interference between the hairpin rules and the load-balance feature.

But just to be on the safe-side, use the more “detailed” definition of the exclusion rules for the DMZ<->LAN, as that one is the only one which makes iptables jump over some load-balancing rules. (See the !!!! perhaps problematic !!!! comment in the rules.)


However could you specify more clearly which are the “glitches” and “artifacts” caused by your setup?

For example have you tcpdump-ed eth0 and eth1 (sudo tcpdump -n -n -N -i eth0) and see if:

  • on the outbound direction you see traffic which is not SNAT-ed;
  • create only one TCP connection from LAN to WAN and make it generate outbound traffic; (a good candidate would be something like while sleep 0.1 ; do echo . ; done | ssh some-server cat; ) tcpdump both WAN connections and see if the packets jump from one to another (they should stick to one WAN connection;)

Moreover, as said earlier please check your hairpin SNAT rule 150 which should be symmetric to SNAT rule 151 (even if you subsequently deleted rule 151, as I see it’s not present in the iptables rules). As it currently stands I doubt it works as it states “if traffic comes from an source LAN-IP and is towards a destination LAN-IP but goes out the DMZ interface then SNAT-it” (i.e. you can’t normally have packets with LAN-IP destination going out the DMZ interface).


## nat-PREROUTING

# DNAT WAN->LAN
-A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-510 -j DNAT --to-destination 192.168.0.xxx:8888
-A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-511 -j DNAT --to-destination 192.168.0.xxx:8888

# DNAT WAN->DMZ
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-700 -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3

# DNAT WAN->DMZ
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-710 -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3

# DNAT WAN->DMZ
-A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 20198 -m comment --comment DST-NAT-720 -j DNAT --to-destination 192.168.1.3:22
-A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 20198 -m comment --comment DST-NAT-721 -j DNAT --to-destination 192.168.1.3:22

# DNAT WAN->DMZ
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 8081 -m comment --comment DST-NAT-722 -j DNAT --to-destination 192.168.1.3:8080
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 8081 -m comment --comment DST-NAT-723 -j DNAT --to-destination 192.168.1.3:8080
## nat-POSTROUTING

# SNAT WAN
-A WANLOADBALANCE -m connmark --mark 0x1 -j SNAT --to-source isp1.ip1.xx.xx
-A WANLOADBALANCE -m connmark --mark 0x2 -j SNAT --to-source isp2.ip2.yy.yy

# SNAT LAN->DMZ
# !!!! perhaps problematic (actually wrong) !!!!
-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 -o eth3 -m comment --comment SRC-NAT-150 -j MASQUERADE
## mangle-PREROUTING

# WAN LB-inbound
-A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN
-A ISP_eth1_IN -j CONNMARK --set-xmark 0x2/0xffffffff
-A PREROUTING -i eth0 -m state --state NEW -j ISP_eth0_IN
-A ISP_eth0_IN -j CONNMARK --set-xmark 0x1/0xffffffff

# LAN hairpin exception
-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth3 -j ACCEPT
-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth3 -j ACCEPT

# LAN->DMZ exception
# !!!! perhaps problematic !!!!
-A WANLOADBALANCE_PRE -d 192.168.0.0/16 -i eth+ -j ACCEPT

# LAN->WAN LB-outbound
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
-A ISP_eth0 -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j MARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j ACCEPT
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -j ISP_eth1
-A ISP_eth1 -j CONNMARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j MARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j ACCEPT
-A WANLOADBALANCE_PRE -i eth2 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

# DMZ->WAN LB-outbound
-A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
-A ISP_eth0 -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j MARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j ACCEPT
-A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -j ISP_eth1
-A ISP_eth1 -j CONNMARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j MARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j ACCEPT
-A WANLOADBALANCE_PRE -i eth3 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

Hi !

I’m very sorry - I’m forgot that 6 hairpin rules (on 3rd post of this thread) are from system with artifacts, yet iptables rules are pasted from the same system which I get working a bit later by deleting these 3 rules, which correspond to hairpin NAT and DMZ 192.168.1.x :

set load-balancing wan rule 22 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 22 'exclude'
set load-balancing wan rule 22 inbound-interface 'eth2'
set load-balancing wan rule 22 destination address 'isp1.ip1.xx.xx'
set load-balancing wan rule 22 protocol all

set load-balancing wan rule 23 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 23 'exclude'
set load-balancing wan rule 23 inbound-interface 'eth2'
set load-balancing wan rule 23 destination address 'isp2.ip1.yy.yy'
set load-balancing wan rule 23 protocol all

set nat source rule 151 description 'NAT Reflection (Hairpin NAT) 192.168.1.0/24'
set nat source rule 151 destination address '192.168.1.0/24'
set nat source rule 151 source address '192.168.1.0/24'
set nat source rule 151 outbound-interface 'eth2'
set nat source rule 151 translation address 'masquerade'

Hairpin NAT now works from local 192.168.0.x only, not from DMZ 192.168.1.x, which is still acceptable. Upload speed through eth0/ISP1 is fine now, one small glitch remains - connecting via SSH to eth1 / ISP#2 happens instantly, yet to eth0 / ISP#1 it takes about a minute or so to establish connection, and often simply ends without password prompt. Have no idea why.

Its possible to paste 3 deleted rules again to the system to check iptables - yet I’m not sure we can make load-balancing and hairpin NAT to get working together as expected, at least without direct hacks into iptables rules generated by VyOS. There are samples around - yet no one which combines load-balancing and hairpin NAT.

Another strange thing - currently there are no explicit masquerading rules for local and DMZ (they were, but I removed ones during debugging), yet masquerading from local and DMZ works. I launched web browser on DMZ server and loaded youtube page:

vyos@linrt-aio# run show nat source translations
Pre-NAT              Post-NAT             Prot  Timeout 
192.168.1.3          isp1.ip1.xx.xx        tcp   431995  
192.168.1.3          isp2.ip2.yy.yy        tcp   431987  
216.244.66.245       isp2.ip2.yy.yy        udp   5       
192.168.1.3          isp2.ip2.yy.yy        udp   170     
192.168.1.3          isp2.ip2.yy.yy        udp   167     
192.168.1.3          isp2.ip2.yy.yy        tcp   95      
192.168.1.3          isp2.ip2.yy.yy        tcp   62      
192.168.1.3          isp2.ip2.yy.yy        udp   131     
192.168.1.3          isp1.ip1.xx.xx        tcp   431997  
192.168.1.3          isp2.ip2.yy.yy        tcp   431996  
192.168.1.3          isp2.ip2.yy.yy        udp   165     
192.168.1.3          isp2.ip2.yy.yy        tcp   62      
192.168.1.3          isp2.ip2.yy.yy        tcp   67      
192.168.1.3          isp2.ip2.yy.yy        tcp   62      
192.168.1.3          isp2.ip2.yy.yy        udp   108     
192.168.1.3          isp1.ip1.xx.xx        tcp   62 

Removed masquerading rules:

set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 source address '192.168.0.0/24'
set nat source rule 100 translation address 'masquerade'
set nat source rule 100 protocol all

set nat source rule 101 outbound-interface 'eth0'
set nat source rule 101 source address '192.168.1.0/24'
set nat source rule 101 translation address 'masquerade'
set nat source rule 101 protocol all

set nat source rule 120 outbound-interface 'eth1'
set nat source rule 120 source address '192.168.0.0/24'
set nat source rule 120 translation address 'masquerade'
set nat source rule 120 protocol all

set nat source rule 121 outbound-interface 'eth1'
set nat source rule 121 source address '192.168.1.0/24'
set nat source rule 121 translation address 'masquerade'
set nat source rule 121 protocol all

PS. Its a VyOS 1.1.8 under KVM host on Debian 9, running on Jetway fanless PC with 10 Ethernet interfaces, KVM network device model - virtio.

Before going into the reply, I have to ask why do you need the hairpin-NAT? What is your concrete use-case? Because when connecting from LAN->DMZ you don’t actually need the SNAT if the VyOS is the router for both networks.


When you say “connecting via SSH” you mean to the VyOS itself, the server in the DMZ or another one in LAN? Although this issue can easily be diagnosed with tcpdump and seeing exactly where the packets enter and where the packets exit through.

If you have the time you could try to paste both the VyOS WAN/NAT settings and the related iptables -S rules, for both the working system and the glitchy one.

Although it is highly likely that you’ll need some “scripting” hacks to make everything work…

They work because the WAN-load-balancing feature automatically introduces these rules itself. See the rule -A WANLOADBALANCE -m connmark --mark 0x1 -j SNAT --to-source isp1.ip1.xx.xx from the script in the previous posts.

Here they are:

*******************************

vyos@linrt-aio# sudo iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N VYATTA_PRE_DNAT_HOOK
-N VYATTA_PRE_SNAT_HOOK
-N WANLOADBALANCE
-A PREROUTING -j VYATTA_PRE_DNAT_HOOK
-A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-510 -j DNAT --to-destination 192.168.0.173:8888
-A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-511 -j DNAT --to-destination 192.168.0.173:8888

-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-700 -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp1.ip1.xx.xx/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3

-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-710 -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3
-A PREROUTING -d isp2.ip2.yy.yy/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3

-A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 10xxx -m comment --comment DST-NAT-720 -j DNAT --to-destination 192.168.1.3:22
-A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 10xxx -m comment --comment DST-NAT-721 -j DNAT --to-destination 192.168.1.3:22

-A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 10yy -m comment --comment DST-NAT-722 -j DNAT --to-destination 192.168.1.3:8080
-A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 10yy -m comment --comment DST-NAT-723 -j DNAT --to-destination 192.168.1.3:8080

-A POSTROUTING -j VYATTA_PRE_SNAT_HOOK
-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 -o eth3 -m comment --comment SRC-NAT-150 -j MASQUERADE
-A POSTROUTING -s 192.168.1.0/24 -d 192.168.1.0/24 -o eth2 -m comment --comment SRC-NAT-151 -j MASQUERADE

-A VYATTA_PRE_DNAT_HOOK -j RETURN
-A VYATTA_PRE_SNAT_HOOK -j WANLOADBALANCE
-A VYATTA_PRE_SNAT_HOOK -j RETURN

-A WANLOADBALANCE -m connmark --mark 0x1 -j SNAT --to-source isp1.ip1.xx.xx
-A WANLOADBALANCE -m connmark --mark 0x2 -j SNAT --to-source isp2.ip2.yy.yy

*******************************

vyos@linrt-aio# sudo iptables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ISP_eth0
-N ISP_eth0_IN
-N ISP_eth1
-N ISP_eth1_IN
-N WANLOADBALANCE_PRE
-A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN
-A PREROUTING -i eth0 -m state --state NEW -j ISP_eth0_IN
-A PREROUTING -j WANLOADBALANCE_PRE
-A ISP_eth0 -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j MARK --set-xmark 0x1/0xffffffff
-A ISP_eth0 -j ACCEPT
-A ISP_eth0_IN -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_eth1 -j CONNMARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j MARK --set-xmark 0x2/0xffffffff
-A ISP_eth1 -j ACCEPT
-A ISP_eth1_IN -j CONNMARK --set-xmark 0x2/0xffffffff
-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth3 -j ACCEPT
-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth3 -j ACCEPT
-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth2 -j ACCEPT
-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth2 -j ACCEPT
-A WANLOADBALANCE_PRE -d 192.168.0.0/16 -i eth+ -j ACCEPT
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -j ISP_eth1
-A WANLOADBALANCE_PRE -i eth2 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -j ISP_eth1
-A WANLOADBALANCE_PRE -i eth3 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

*******************************

And here are rules which cause upload eth0/ISP#1 glitch from DMZ server. Hairpin rules taken from samples floating around.

set load-balancing wan rule 22 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 22 'exclude'
set load-balancing wan rule 22 inbound-interface 'eth2'
set load-balancing wan rule 22 destination address 'isp1.ip1.xx.xx'
set load-balancing wan rule 22 protocol all

set load-balancing wan rule 23 description 'for NAT reflection (Hairpin NAT)'
set load-balancing wan rule 23 'exclude'
set load-balancing wan rule 23 inbound-interface 'eth2'
set load-balancing wan rule 23 destination address 'isp2.ip2.yy.yy'
set load-balancing wan rule 23 protocol all

set nat source rule 151 description 'NAT Reflection (Hairpin NAT) 192.168.1.0/24'
set nat source rule 151 destination address '192.168.1.0/24'
set nat source rule 151 source address '192.168.1.0/24'
set nat source rule 151 outbound-interface 'eth2'
set nat source rule 151 translation address 'masquerade'

I have number of services natted from DMZ to outside world, which should work from both internal and external network using external IP address. And while I can set split DNS for ISPConfig DMZ servers, this is too problematic for other services.

OK, great, thanks. I’m currently don’t do direct iptables scripting, configuring firewalls/routers is a casual task for me upon necessity.

Current config is not perfect yet its usable.

Anyway, its quite interesting what I have done wrong.

I mean connecting to router via SSH from outside world.
80.232.198.21 - temporary external IP, eth0, ISp #1 (problematic)
80.232.198.17 - gateway IP ISP #1

sudo tcpdump -i eth0 | grep 80.232.198.21

vyos@linrt-aio:~$ sudo tcpdump -i eth0 | grep 80.232.198.21

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:07:03.965717 IP 80.232.198.21.20271 > dns.balticom.lv.domain: 19183+ PTR? 95.181.108.95.in-addr.arpa. (44)
13:07:05.959530 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:05.999302 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:06.046294 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [S], seq 84938726, win 29200, options [mss 1460,sackOK,TS val 1108397486 ecr 0,nop,wscale 7], length 0
13:07:06.901849 IP 80.232.198.21.44887 > ns12.lattelecom.lv.domain: 61981+ [b2&3=0x182] PTR? 203.66.244.216.in-addr.arpa. (45)
13:07:07.055152 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [S], seq 84938726, win 29200, options [mss 1460,sackOK,TS val 1108398513 ecr 0,nop,wscale 7], length 0
13:07:07.055241 IP 80.232.198.21.10055 > balticom-228-190.balticom.lv.49142: Flags [S.], seq 3164082046, ack 84938727, win 28960, options [mss 1460,sackOK,TS val 15781950 ecr 1108397486,nop,wscale 7], length 0
13:07:07.067755 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [.], ack 1, win 229, options [nop,nop,TS val 1108398532 ecr 15781950], length 0
13:07:07.067860 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108398533 ecr 15781950], length 21
13:07:09.407088 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108400881 ecr 15781950], length 21
13:07:10.963351 ARP, Request who-has 80.232.198.17 tell 80.232.198.21, length 28
13:07:11.011771 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:11.014312 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:11.887343 IP 80.232.198.21.44887 > dns.balticom.lv.domain: 61981+ PTR? 203.66.244.216.in-addr.arpa. (45)
13:07:11.887470 IP 80.232.198.21.44887 > google-public-dns-a.google.com.domain: 61981+ PTR? 203.66.244.216.in-addr.arpa. (45)
13:07:11.948067 IP google-public-dns-a.google.com.domain > 80.232.198.21.44887: 61981 ServFail 0/0/0 (45)
13:07:12.735195 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108404209 ecr 15781950], length 21
13:07:16.027182 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:16.029850 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:19.135014 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108410609 ecr 15781950], length 21
13:07:21.043089 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:21.047480 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:22.011701 IP 80.232.198.21.ntp > ntp1.0x00.lv.ntp: NTPv4, Client, length 48
13:07:22.012314 IP 80.232.198.21.26611 > google-public-dns-a.google.com.domain: 54016+ PTR? 85.54.111.89.in-addr.arpa. (43)
13:07:22.014574 IP ntp1.0x00.lv.ntp > 80.232.198.21.ntp: NTPv4, Server, length 48
13:07:22.033622 IP google-public-dns-a.google.com.domain > 80.232.198.21.26611: 54016 1/0/0 PTR ntp1.0x00.lv. (69)
13:07:24.011826 IP 80.232.198.21.ntp > ntp1.0x00.lv.ntp: NTPv4, Client, length 48
13:07:24.014944 IP ntp1.0x00.lv.ntp > 80.232.198.21.ntp: NTPv4, Server, length 48
13:07:26.059213 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:26.061608 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:29.539210 IP census10.shodan.io.31743 > 80.232.198.21.chargen: UDP, length 1
13:07:29.539789 IP 80.232.198.21.16979 > google-public-dns-a.google.com.domain: 27523+ PTR? 6.105.221.82.in-addr.arpa. (43)
13:07:29.539974 IP 80.232.198.21.16979 > ns12.lattelecom.lv.domain: 27523+ PTR? 6.105.221.82.in-addr.arpa. (43)
13:07:29.543066 IP ns12.lattelecom.lv.domain > 80.232.198.21.16979: 27523 1/3/3 PTR census10.shodan.io. (180)
13:07:29.551934 IP google-public-dns-a.google.com.domain > 80.232.198.21.16979: 27523 1/0/0 PTR census10.shodan.io. (75)
13:07:31.073710 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:31.076538 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:31.935416 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108423409 ecr 15781950], length 21
13:07:32.011785 IP 80.232.198.21.ntp > ntp1.0x00.lv.ntp: NTPv4, Client, length 48
13:07:32.014994 IP ntp1.0x00.lv.ntp > 80.232.198.21.ntp: NTPv4, Server, length 48
13:07:36.088930 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:36.091790 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:36.905145 IP ns12.lattelecom.lv.domain > 80.232.198.21.44887: 61981 ServFail 0/0/0 (45)
13:07:41.103294 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:41.107357 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:42.274246 IP 80.232.198.21.47075 > google-public-dns-a.google.com.domain: 51410+ PTR? 149.47.125.74.in-addr.arpa. (44)
13:07:42.274373 IP 80.232.198.21.47075 > ns12.lattelecom.lv.domain: 51410+ PTR? 149.47.125.74.in-addr.arpa. (44)
13:07:42.294781 IP ns12.lattelecom.lv.domain > 80.232.198.21.47075: 51410 NXDomain 0/1/0 (104)
13:07:42.296004 IP google-public-dns-a.google.com.domain > 80.232.198.21.47075: 51410 NXDomain 0/1/0 (104)
13:07:42.296068 IP 80.232.198.21 > google-public-dns-a.google.com: ICMP 80.232.198.21 udp port 47075 unreachable, length 140
13:07:42.339250 IP 80.232.198.21.58603 > dns.balticom.lv.domain: 25972+ PTR? 84.73.125.74.in-addr.arpa. (43)
13:07:46.122805 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:46.125818 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:47.344739 IP 80.232.198.21.58603 > ns12.lattelecom.lv.domain: 25972+ PTR? 84.73.125.74.in-addr.arpa. (43)
13:07:47.344861 IP 80.232.198.21.58603 > dns.balticom.lv.domain: 25972+ PTR? 84.73.125.74.in-addr.arpa. (43)
13:07:47.364897 IP ns12.lattelecom.lv.domain > 80.232.198.21.58603: 25972 NXDomain 0/1/0 (103)
13:07:51.137330 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:51.140898 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:56.156649 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:07:56.159635 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:07:57.535024 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108449009 ecr 15781950], length 21
13:08:01.170961 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:01.173807 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:06.187774 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:06.190826 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:11.202360 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:11.206628 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:16.226918 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:16.230011 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:21.250175 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:21.252842 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:26.264517 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:26.267376 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:29.011730 IP 80.232.198.21.ntp > ntp1.0x00.lv.ntp: NTPv4, Client, length 48
13:08:29.014756 IP ntp1.0x00.lv.ntp > 80.232.198.21.ntp: NTPv4, Server, length 48
13:08:29.143854 IP 80.232.198.21.41659 > dns.balticom.lv.domain: 54158+ PTR? 147.66.249.66.in-addr.arpa. (44)
13:08:31.279203 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:31.283193 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:36.294596 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:36.298061 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:36.667565 IP 80.232.198.21.39288 > google-public-dns-a.google.com.domain: 47961+ PTR? 162.126.171.112.in-addr.arpa. (46)
13:08:37.285470 IP google-public-dns-a.google.com.domain > 80.232.198.21.39288: 47961 NXDomain 0/1/0 (107)
13:08:41.309279 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:41.312317 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:46.323962 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:46.326648 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:48.735333 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 1:22, ack 1, win 229, options [nop,nop,TS val 1108500210 ecr 15781950], length 21
13:08:48.735436 IP 80.232.198.21.10055 > balticom-228-190.balticom.lv.49142: Flags [.], ack 22, win 227, options [nop,nop,TS val 15792118 ecr 1108500210,nop,nop,sack 1 {1:22}], length 0
13:08:51.338112 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:51.341197 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:08:56.352864 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:08:56.357027 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:09:01.368079 IP 80.232.198.21 > 80.232.198.17: ICMP echo request, id 13274, seq 256, length 40
13:09:01.371944 IP 80.232.198.17 > 80.232.198.21: ICMP echo reply, id 13274, seq 256, length 40
13:09:04.993392 IP 80.232.198.21.10055 > balticom-228-190.balticom.lv.49142: Flags [P.], seq 1:826, ack 22, win 227, options [nop,nop,TS val 15793744 ecr 1108500210], length 825
13:09:04.997597 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [.], ack 826, win 251, options [nop,nop,TS val 1108516472 ecr 15793744], length 0
13:09:04.999545 IP balticom-228-190.balticom.lv.49142 > 80.232.198.21.10055: Flags [P.], seq 22:1382, ack 826, win 251, options [nop,nop,TS val 1108516473 ecr 15793744], length 1360
^C318 packets captured
318 packets received by filter
0 packets dropped by kernel

I’ve made a quick diff between the rules in earlier posts and the rules in your last message:

For NAT:

--- /tmp/nat-working	2018-06-24 20:40:26.950962472 +0300
+++ /tmp/nat-glitchy	2018-06-24 20:42:16.340855176 +0300
@@ -1,27 +1,28 @@
 -P PREROUTING ACCEPT
 -P INPUT ACCEPT
 -P OUTPUT ACCEPT
 -P POSTROUTING ACCEPT
 -N VYATTA_PRE_DNAT_HOOK
 -N VYATTA_PRE_SNAT_HOOK
 -N WANLOADBALANCE
 -A PREROUTING -j VYATTA_PRE_DNAT_HOOK
 -A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-510 -j DNAT --to-destination 192.168.0.xxx:8888
 -A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 88 -m comment --comment DST-NAT-511 -j DNAT --to-destination 192.168.0.xxx:8888
 -A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-700 -j DNAT --to-destination 192.168.1.3
 -A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3
 -A PREROUTING -d isp1.ip1.xx.xx/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-705 tcp_udp" -j DNAT --to-destination 192.168.1.3
 -A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m multiport --dports 80,443,25,465,110,995,143,993,21,40110:40210 -m comment --comment DST-NAT-710 -j DNAT --to-destination 192.168.1.3
 -A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3
 -A PREROUTING -d isp2.ip2.yy.yy/32 -p udp -m udp --dport 53 -m comment --comment "DST-NAT-715 tcp_udp" -j DNAT --to-destination 192.168.1.3
 -A PREROUTING -d isp1.ip1.xx.xx/32 -i eth0 -p tcp -m tcp --dport 10xxx -m comment --comment DST-NAT-720 -j DNAT --to-destination 192.168.1.3:22
 -A PREROUTING -d isp2.ip2.yy.yy/32 -i eth1 -p tcp -m tcp --dport 10xxx -m comment --comment DST-NAT-721 -j DNAT --to-destination 192.168.1.3:22
 -A PREROUTING -d isp1.ip1.xx.xx/32 -p tcp -m tcp --dport 10yy -m comment --comment DST-NAT-722 -j DNAT --to-destination 192.168.1.3:8080
 -A PREROUTING -d isp2.ip2.yy.yy/32 -p tcp -m tcp --dport 10yy -m comment --comment DST-NAT-723 -j DNAT --to-destination 192.168.1.3:8080
 -A POSTROUTING -j VYATTA_PRE_SNAT_HOOK
 -A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 -o eth3 -m comment --comment SRC-NAT-150 -j MASQUERADE
+-A POSTROUTING -s 192.168.1.0/24 -d 192.168.1.0/24 -o eth2 -m comment --comment SRC-NAT-151 -j MASQUERADE
 -A VYATTA_PRE_DNAT_HOOK -j RETURN
 -A VYATTA_PRE_SNAT_HOOK -j WANLOADBALANCE
 -A VYATTA_PRE_SNAT_HOOK -j RETURN
 -A WANLOADBALANCE -m connmark --mark 0x1 -j SNAT --to-source isp1.ip1.xx.xx
 -A WANLOADBALANCE -m connmark --mark 0x2 -j SNAT --to-source isp2.ip2.yy.yy

For mangle:

--- /tmp/mangle-working	2018-06-24 20:40:26.950962472 +0300
+++ /tmp/mangle-glitchy	2018-06-24 20:42:16.340855176 +0300
@@ -1,30 +1,32 @@
 -P PREROUTING ACCEPT
 -P INPUT ACCEPT
 -P FORWARD ACCEPT
 -P OUTPUT ACCEPT
 -P POSTROUTING ACCEPT
 -N ISP_eth0
 -N ISP_eth0_IN
 -N ISP_eth1
 -N ISP_eth1_IN
 -N WANLOADBALANCE_PRE
 -A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN
 -A PREROUTING -i eth0 -m state --state NEW -j ISP_eth0_IN
 -A PREROUTING -j WANLOADBALANCE_PRE
 -A ISP_eth0 -j CONNMARK --set-xmark 0x1/0xffffffff
 -A ISP_eth0 -j MARK --set-xmark 0x1/0xffffffff
 -A ISP_eth0 -j ACCEPT
 -A ISP_eth0_IN -j CONNMARK --set-xmark 0x1/0xffffffff
 -A ISP_eth1 -j CONNMARK --set-xmark 0x2/0xffffffff
 -A ISP_eth1 -j MARK --set-xmark 0x2/0xffffffff
 -A ISP_eth1 -j ACCEPT
 -A ISP_eth1_IN -j CONNMARK --set-xmark 0x2/0xffffffff
 -A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth3 -j ACCEPT
 -A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth3 -j ACCEPT
+-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth2 -j ACCEPT
+-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth2 -j ACCEPT
 -A WANLOADBALANCE_PRE -d 192.168.0.0/16 -i eth+ -j ACCEPT
 -A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
 -A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -j ISP_eth1
 -A WANLOADBALANCE_PRE -i eth2 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
 -A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_eth0
 -A WANLOADBALANCE_PRE -i eth3 -m state --state NEW -j ISP_eth1
 -A WANLOADBALANCE_PRE -i eth3 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

Unfortunately the three rules that seem to be extra (in the “glitchy” version) don’t seem to be the culprit in glitchy WAN behaviour…

Perhaps you’ve missed the configuration that actually caused the issue. (Next time you encounter issues make a snapshot of both the config and the iptables -t xxx -S output.)

I’ve tried to make some sense from the textual output but it’s quite hard. (There is a lot of mixed traffic in there…)

If you encounter these issues again please do the following:

  • note the (public) IP of the host you are connecting from (and the local port for each problematic connection);
  • do a tcpdump on both network interfaces; (as it might happen that traffic from your router goes the wrong interface;)
  • use the following tcpdump arguments: tcpdump -v -n -n -N -i ethX;
  • (alternatively the most helpful would be two pcap files, one for each interface, perhaps filtered with Wireshark only for the SSH connection (which could be obtained by filtering for the source-ip+port and destination-ip+port tuple; this has only a slight privacy issue as it makes public encrypted SSH traffic;)
  • (alternatively instead of SSH you can test this with socat tcp-listen:222 stdio </dev/urandom >/dev/null on the server and socat tcp-connect:IP:222 stdio </dev/urandom >/dev/null on the client which will generate random traffic; socat I don’t think is available on VyOS;)

I verified behavior of the rooter before posting. Adding hairpin NAT rules for 192.168.1.x breaks upload speed for eth0 / ISP #1. Even with hairpin NAT rules only for 192.16.0.xx there are some artifacts, which are still acceptable. I assume combination of these features have not been thoughtfully tested.

+-A POSTROUTING -s 192.168.1.0/24 -d 192.168.1.0/24 -o eth2 -m comment --comment SRC-NAT-151 -j MASQUERADE
+-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth2 -j ACCEPT
+-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth2 -j ACCEPT

I’ts quite strange that these three rules generate glitches because:

  • the first set of rules in iptables -t nat (pasted bellow) shouldn’t actually do anything, that is because they would apply for traffic destined to the eth3 network, but comming from the same IP range as eth3 network, which normally shouldn’t happen as devices in that network should communicate directly not via the router;
 -A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.0/24 -o eth3 -m comment --comment SRC-NAT-150 -j MASQUERADE
+-A POSTROUTING -s 192.168.1.0/24 -d 192.168.1.0/24 -o eth2 -m comment --comment SRC-NAT-151 -j MASQUERADE
  • (if instead the IP range 192.168.0.0/24 is for eth2, then that rule is even more broken, as those packets will never exit eth3; as said in earlier posts these two rules should be symmetric, i.e. you can’t have the same source and destination in the same rule;)

  • thus the only rule that might make troubles is the one in iptables -t mangle that was generated by exclusions; however I doubt this could be the issue because those rules don’t exit any of the WAN interfaces, and moreover these rules are in PREROUTING, thus any hair-pin NAT would be applied afterwards in iptables -t nat PREROUTING chain;

 -A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth3 -j ACCEPT
 -A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth3 -j ACCEPT
+-A WANLOADBALANCE_PRE -d isp1.ip1.xx.xx/32 -i eth2 -j ACCEPT
+-A WANLOADBALANCE_PRE -d isp2.ip2.yy.yy/32 -i eth2 -j ACCEPT
 -A WANLOADBALANCE_PRE -d 192.168.0.0/16 -i eth+ -j ACCEPT