[solved] WAN load-balancing with `policy route` rules [previously: WAN load-balancing with 2 PPPoE connections with TCP-MSS clamping]

ciprian.craciun · May 30, 2018, 3:41pm

Update

It seems I was able to “hack” my way through this problem and solve it as described in my last comment.

Original

It seems that WAN load-balancing and TCP-MSS clamping policies don’t mix well. Has anyone managed to correctly configure such a setup?

Is there an alternative setup? (Perhaps with different routing tables?)

From what I gather the WAN load-balancing feature introduces -j ACCEPT rules in the -t mangle table, which come before the TCP-MSS clamping rules generated by the policy routing.

ciprian.craciun · May 31, 2018, 1:09pm

The following is the iptables -t mangle -S script that pertains to WAN load-balancing enabled for a single IP. The rules that “break” the TCP-MSS clamping are marked with !!.

The rules that enable TCP-MSS clamping are called from within VYATTA_FW_IN_HOOK (from within PREROUTING), which however are not reached anymore because inside ISP_pppoe0_IN there is an -j ACCEPT.

-A PREROUTING -i pppoe1 -m state --state NEW -j ISP_pppoe1_IN
-A PREROUTING -i pppoe0 -m state --state NEW -j ISP_pppoe0_IN
-A PREROUTING -j WANLOADBALANCE_PRE
-A PREROUTING -j VYATTA_FW_IN_HOOK

-A WANLOADBALANCE_PRE -d 216.239.38.21/32 -i eth0.3 -m state --state NEW -m statistic --mode random --probability 0.50000000000 -j ISP_pppoe0
-A WANLOADBALANCE_PRE -d 216.239.38.21/32 -i eth0.3 -m state --state NEW -j ISP_pppoe1
-A WANLOADBALANCE_PRE -d 216.239.38.21/32 -i eth0.3 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

-A ISP_pppoe0 -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_pppoe0 -j MARK --set-xmark 0x1/0xffffffff
!! -A ISP_pppoe0 -j ACCEPT

-A ISP_pppoe1 -j CONNMARK --set-xmark 0x2/0xffffffff
-A ISP_pppoe1 -j MARK --set-xmark 0x2/0xffffffff
!! -A ISP_pppoe1 -j ACCEPT

-A ISP_pppoe0_IN -j CONNMARK --set-xmark 0x1/0xffffffff
-A ISP_pppoe1_IN -j CONNMARK --set-xmark 0x2/0xffffffff

-A VYATTA_FW_IN_HOOK -i eth0.1 -j pppoe-mangle-out
-A VYATTA_FW_IN_HOOK -i pppoe1 -j pppoe-mangle-in
-A VYATTA_FW_IN_HOOK -i pppoe0 -j pppoe-mangle-in
-A VYATTA_FW_IN_HOOK -i eth0.3 -j pppoe-mangle-out

If one is to remove the two problematic rules, everything works just fine:

sudo iptables -t mangle -D ISP_pppoe0 -j ACCEPT
sudo iptables -t mangle -D ISP_pppoe1 -j ACCEPT

The following is the whole output of iptables -t mangle -L -n:

Chain PREROUTING (policy ACCEPT)
target              prot opt source               destination
ISP_pppoe1_IN       all  --  0.0.0.0/0            0.0.0.0/0            state NEW
ISP_pppoe0_IN       all  --  0.0.0.0/0            0.0.0.0/0            state NEW
WANLOADBALANCE_PRE  all  --  0.0.0.0/0            0.0.0.0/0
VYATTA_FW_IN_HOOK   all  --  0.0.0.0/0            0.0.0.0/0

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
VYATTA_FW_OUT_HOOK  all  --  0.0.0.0/0            0.0.0.0/0

Chain ISP_pppoe0 (1 references)
target     prot opt source               destination
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0            CONNMARK set 0x1
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK set 0x1
!! ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain ISP_pppoe0_IN (1 references)
target     prot opt source               destination
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0            CONNMARK set 0x1

Chain ISP_pppoe1 (1 references)
target     prot opt source               destination
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0            CONNMARK set 0x2
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK set 0x2
!! ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

Chain ISP_pppoe1_IN (1 references)
target     prot opt source               destination
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0            CONNMARK set 0x2

Chain VYATTA_FW_IN_HOOK (1 references)
target            prot opt source               destination
pppoe-mangle-out  all  --  0.0.0.0/0            0.0.0.0/0
pppoe-mangle-in   all  --  0.0.0.0/0            0.0.0.0/0
pppoe-mangle-in   all  --  0.0.0.0/0            0.0.0.0/0
pppoe-mangle-out  all  --  0.0.0.0/0            0.0.0.0/0

Chain VYATTA_FW_OUT_HOOK (1 references)
target     prot opt source               destination

Chain WANLOADBALANCE_PRE (1 references)
target      prot opt source               destination
ISP_pppoe0  all  --  0.0.0.0/0            216.239.38.21        state NEW statistic mode random probability 0.50000000000
ISP_pppoe1  all  --  0.0.0.0/0            216.239.38.21        state NEW
CONNMARK    all  --  0.0.0.0/0            216.239.38.21        CONNMARK restore

Chain pppoe-mangle-in (2 references)
target     prot opt source               destination
TCPMSS     tcp  --  0.0.0.0/0            0.0.0.0/0            /* pppoe-mangle-in-1 */ tcp flags:0x06/0x02 TCPMSS set 1452
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            /* pppoe-mangle-in-10000 default-action accept */

Chain pppoe-mangle-out (2 references)
target     prot opt source               destination
TCPMSS     tcp  --  0.0.0.0/0            0.0.0.0/0            /* pppoe-mangle-out-1 */ tcp flags:0x06/0x02 ! match-set lan dst TCPMSS set 1452
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            /* pppoe-mangle-out-10000 default-action accept */

I tried to grep the VyOS WAN load-balancing repository and it seems that this issue was previously identified and solved (by replacing the ACCEPT with RETURN), however I don’t think it made it into VyOS 1.1.8…

ciprian.craciun · May 31, 2018, 3:12pm

The solution (although it’s more a “hack”)

So I managed to “hack” the config so that it works OK, by doing the following:

create a custom script with the contents given bellow:
nano -w /config/scripts/wan-lb-hook.script
configure this script as the WAN load-balancing hook with:
set load-balancing wan hook '/config/scripts/wan-lb-hook.script'
disable source-NAT (this has to be done explicitly via edit nat source):
set load-balancing wan 'disable-source-nat'
never use the set rule x exclude to exclude load-balanced traffic! (if you do so it breaks again the whole setup due to the usage of -j ACCEPT rules;)
if you need to exclude some traffic use the following policy route table and de-comment the script lines which contain wan-lb-exclude (and the few lines before and after):

edit policy route wan-lb-exclude
set rule 1 set mark 2147483647
set rule 1 ... # other criteria

Update (1): if your routing is “simple”, thus just WAN and a single LAN, then you don’t need the exclusion policy as described above. However if you route also between multiple LAN’s, then all traffic that is destined to those LAN’s has to be excluded. (This can easily be achieved by defining a firewall network group, listing all local networks (or networks that shouldn’t be WAN-load-balanced), and just add an exclusion rule as described above.)

Update (2): Policy routing, especially blackhole routes (used sometimes as a “poor-man-outbound-firewall” to be sure no “private” traffic leaks through WAN), won’t work for load-balanced traffic. This is due to the fact that in order for WAN-load-balancing to work, separate routing tables are created behind the scenes which don’t feature the blackhole routes.

#!/bin/bash

set -e -E -u -o pipefail -o noclobber -o noglob +o braceexpand || exit 1
trap 'printf -- "[ee] failed: %s\n" "${BASH_COMMAND}" >&2' ERR || exit 1

test "${#}" -eq 0
test -n "${WLB_INTERFACE_NAME:?}"
test -n "${WLB_INTERFACE_STATE:?}"

case "${WLB_INTERFACE_STATE}" in
        ( ACTIVE )
                iptables -t mangle -D "ISP_${WLB_INTERFACE_NAME}" -j ACCEPT
                iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 1 -j RETURN -m connmark ! --mark 0
                # iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 2 -j wan-lb-exclude
                # iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 3 -j CONNMARK --save-mark
                # iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 4 -j RETURN -m connmark ! --mark 0
                iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}_IN" -j RETURN -m connmark ! --mark 0
        ;;
        ( FAILED )
        ;;
esac

exit -- 0

How does it work?

How does it fix the `-j ACCEPT` issue?

It seems that the -j ACCEPT was used so that once a candidate interface was chosen, it won’t fallback to the next one. (I.e. if we just delete the -j ACCEPT it would have always used the last configured interface.)

However we need to emulate this behaviour by adding a rule which states that if a packet is already marked we should skip this chain.

How does it fix the exclusion issue?

We add a few rules to each interface chain so that:

we jump to the wan-lb-exclude chain, which via policy route wan-lb-exclude rule ... sets packet marks for those packets (and thus connections) that have to be excluded; (the value is not important, it just needs to be different than 0, and given how WAN load-balancing works, I have choosen the largest addmitted value;)
we then take any packet mark and save it to the connection mark; (else traffic won’t behave as expected;)
as in the previous case we just -j RETURN if there was a connection mark configured;

system · June 5, 2018, 12:29pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.