[solved] WAN load-balancing with `policy route` rules [previously: WAN load-balancing with 2 PPPoE connections with TCP-MSS clamping]

General questions
,
3

The solution (although it’s more a “hack”)

So I managed to “hack” the config so that it works OK, by doing the following:

  • create a custom script with the contents given bellow:
    nano -w /config/scripts/wan-lb-hook.script
  • configure this script as the WAN load-balancing hook with:
    set load-balancing wan hook '/config/scripts/wan-lb-hook.script'
  • disable source-NAT (this has to be done explicitly via edit nat source):
    set load-balancing wan 'disable-source-nat'
  • never use the set rule x exclude to exclude load-balanced traffic! (if you do so it breaks again the whole setup due to the usage of -j ACCEPT rules;)
  • if you need to exclude some traffic use the following policy route table and de-comment the script lines which contain wan-lb-exclude (and the few lines before and after):
edit policy route wan-lb-exclude
set rule 1 set mark 2147483647
set rule 1 ... # other criteria

Update (1): if your routing is “simple”, thus just WAN and a single LAN, then you don’t need the exclusion policy as described above. However if you route also between multiple LAN’s, then all traffic that is destined to those LAN’s has to be excluded. (This can easily be achieved by defining a firewall network group, listing all local networks (or networks that shouldn’t be WAN-load-balanced), and just add an exclusion rule as described above.)

Update (2): Policy routing, especially blackhole routes (used sometimes as a “poor-man-outbound-firewall” to be sure no “private” traffic leaks through WAN), won’t work for load-balanced traffic. This is due to the fact that in order for WAN-load-balancing to work, separate routing tables are created behind the scenes which don’t feature the blackhole routes.

#!/bin/bash

set -e -E -u -o pipefail -o noclobber -o noglob +o braceexpand || exit 1
trap 'printf -- "[ee] failed: %s\n" "${BASH_COMMAND}" >&2' ERR || exit 1

test "${#}" -eq 0
test -n "${WLB_INTERFACE_NAME:?}"
test -n "${WLB_INTERFACE_STATE:?}"

case "${WLB_INTERFACE_STATE}" in
        ( ACTIVE )
                iptables -t mangle -D "ISP_${WLB_INTERFACE_NAME}" -j ACCEPT
                iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 1 -j RETURN -m connmark ! --mark 0
                # iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 2 -j wan-lb-exclude
                # iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 3 -j CONNMARK --save-mark
                # iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 4 -j RETURN -m connmark ! --mark 0
                iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}_IN" -j RETURN -m connmark ! --mark 0
        ;;
        ( FAILED )
        ;;
esac

exit -- 0

How does it work?

How does it fix the -j ACCEPT issue?

It seems that the -j ACCEPT was used so that once a candidate interface was chosen, it won’t fallback to the next one. (I.e. if we just delete the -j ACCEPT it would have always used the last configured interface.)

However we need to emulate this behaviour by adding a rule which states that if a packet is already marked we should skip this chain.

How does it fix the exclusion issue?

We add a few rules to each interface chain so that:

  • we jump to the wan-lb-exclude chain, which via policy route wan-lb-exclude rule ... sets packet marks for those packets (and thus connections) that have to be excluded; (the value is not important, it just needs to be different than 0, and given how WAN load-balancing works, I have choosen the largest addmitted value;)
  • we then take any packet mark and save it to the connection mark; (else traffic won’t behave as expected;)
  • as in the previous case we just -j RETURN if there was a connection mark configured;