Questions and issues with the built in load balancing feature

Bugs
1

My primary wan finally started working today and I was able to test this feature. I’ll list all the issues I have with this.

For this post, WAN1 is primary, WAN2 s failover. I am on,

PRETTY_NAME="VyOS 1.5-rolling-202310090023 (current)"

  1. On recovery, It wipes connection on WAN2. This behavior should be configurable. I am trying to minimize interruptions on my home network. When it fails over, it has to wipe connections because there is no other option. On recovery, I’ll much prefer it keep the existing connections on WAN2 and new connections should be setup over WAN1. Some people may not want this so maybe this can be configurable. I personally really do want this. I have reasonably fast 5G failover and I do not want any thing to be interrupted if it doesn’t has to be.

  2. It wipes connection table even when I update a completely unrelated firewall rule. I ran, set firewall ipv4 input filter default-action accept; commit and it wiped the table :man_facepalming: I’ll really prefer if if it does not do this.

  3. It wipes connections between local devices. There is no reason it should be doing this and this is just interruption for no reason at all. For example, When I committed the firewall rule in 2, These are all the connections it wiped.

icmp     1 29 src=10.0.150.148 dst=46.16.133.44 type=8 code=0 id=12042 src=46.16.133.44 dst=103.83.146.116 type=0 code=0 id=12042 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.99.14 dst=8.8.4.4 sport=34210 dport=853 src=8.8.4.4 dst=100.90.79.102 sport=853 dport=34210 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32814 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32814 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.30.12 dst=18.164.242.42 sport=38636 dport=443 src=18.164.242.42 dst=100.90.79.102 sport=443 dport=38636 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 27 src=0.0.0.0 dst=255.255.255.255 sport=68 dport=67 [UNREPLIED] src=255.255.255.255 dst=0.0.0.0 sport=67 dport=68 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32858 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32858 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.150.148 dst=10.0.99.14 sport=34991 dport=53 src=10.0.99.14 dst=10.0.150.148 sport=53 dport=34991 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431995 ESTABLISHED src=10.0.99.1 dst=10.0.10.34 sport=22 dport=54628 src=10.0.10.34 dst=10.0.99.1 sport=54628 dport=22 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.35 dst=10.0.99.14 sport=28568 dport=53 src=10.0.99.14 dst=10.0.10.35 sport=53 dport=28568 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.50.14 dst=95.89.104.191 sport=51411 dport=48042 [UNREPLIED] src=95.89.104.191 dst=100.90.79.102 sport=48042 dport=51411 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.30.16 dst=10.0.99.14 sport=36717 dport=53 src=10.0.99.14 dst=10.0.30.16 sport=53 dport=36717 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.99.14 dst=8.8.4.4 sport=34252 dport=853 src=8.8.4.4 dst=100.90.79.102 sport=853 dport=34252 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.10.34 dst=172.67.73.1 sport=38637 dport=443 [UNREPLIED] src=172.67.73.1 dst=100.90.79.102 sport=443 dport=38637 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=53095 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=53095 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.10.23 dst=255.255.255.255 sport=56235 dport=6667 [UNREPLIED] src=255.255.255.255 dst=10.0.10.23 sport=6667 dport=56235 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 298 ESTABLISHED src=10.0.10.34 dst=54.211.224.118 sport=44978 dport=443 [UNREPLIED] src=54.211.224.118 dst=100.90.79.102 sport=443 dport=44978 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32804 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32804 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 25 src=139.84.164.110 dst=103.83.146.116 type=8 code=0 id=42649 src=103.83.146.116 dst=139.84.164.110 type=0 code=0 id=42649 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.160.148 dst=192.36.148.17 sport=48650 dport=53 src=192.36.148.17 dst=103.83.146.116 sport=53 dport=48650 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=53148 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=53148 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 299 ESTABLISHED src=10.0.50.3 dst=10.0.10.34 sport=443 dport=41798 src=10.0.10.34 dst=10.0.50.3 sport=41798 dport=443 mark=0 use=2
Oct 11 19:19:27 wan_lb[61995]: tcp      6 298 ESTABLISHED src=10.0.20.11 dst=157.240.16.52 sport=36278 dport=443 src=157.240.16.52 dst=103.83.146.116 sport=443 dport=36278 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 28 src=10.0.160.148 dst=82.98.65.254 type=8 code=0 id=28713 [UNREPLIED] src=82.98.65.254 dst=103.83.146.116 type=0 code=0 id=28713 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32800 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32800 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.99.4 dst=255.255.255.255 sport=16720 dport=16720 [UNREPLIED] src=255.255.255.255 dst=10.0.99.4 sport=16720 dport=16720 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431996 ESTABLISHED src=10.0.30.12 dst=10.0.99.14 sport=50336 dport=853 src=10.0.99.14 dst=10.0.30.12 sport=853 dport=50336 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32840 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32840 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431998 ESTABLISHED src=10.0.99.14 dst=8.8.4.4 sport=34242 dport=853 src=8.8.4.4 dst=100.90.79.102 sport=853 dport=34242 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=52278 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=52278 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 299 ESTABLISHED src=10.0.30.16 dst=35.154.140.128 sport=48552 dport=5222 src=35.154.140.128 dst=100.90.79.102 sport=5222 dport=48552 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=46723 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=46723 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 119 TIME_WAIT src=10.1.1.3 dst=10.0.50.7 sport=58142 dport=5432 src=10.0.50.7 dst=10.1.1.3 sport=5432 dport=58142 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.10.20 dst=3.110.250.34 sport=43956 dport=8886 src=3.110.250.34 dst=100.90.79.102 sport=8886 dport=43956 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431998 ESTABLISHED src=10.0.99.14 dst=8.8.4.4 sport=34220 dport=853 src=8.8.4.4 dst=100.90.79.102 sport=853 dport=34220 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 26 src=10.0.150.148 dst=156.154.79.254 type=8 code=0 id=28708 [UNREPLIED] src=156.154.79.254 dst=103.83.146.116 type=0 code=0 id=28708 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.40.14 dst=10.0.99.14 sport=38022 dport=853 src=10.0.99.14 dst=10.0.40.14 sport=853 dport=38022 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=52819 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=52819 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431998 ESTABLISHED src=10.0.99.14 dst=1.0.0.1 sport=35200 dport=853 src=1.0.0.1 dst=100.90.79.102 sport=853 dport=35200 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32834 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32834 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=45517 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=45517 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 25 src=10.0.20.11 dst=10.0.99.14 sport=27418 dport=53 src=10.0.99.14 dst=10.0.20.11 sport=53 dport=27418 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.150.148 dst=78.46.48.134 sport=20494 dport=33443 [UNREPLIED] src=78.46.48.134 dst=103.83.146.116 sport=33443 dport=20494 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32784 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32784 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.10.34 dst=10.0.99.14 sport=48586 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=48586 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.40.14 dst=157.240.239.11 sport=34744 dport=443 src=157.240.239.11 dst=100.90.79.102 sport=443 dport=34744 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=139.84.164.110 dst=103.83.146.116 sport=51820 dport=51820 src=103.83.146.116 dst=139.84.164.110 sport=51820 dport=51820 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32864 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32864 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 299 ESTABLISHED src=10.0.50.3 dst=10.0.10.34 sport=443 dport=41796 src=10.0.10.34 dst=10.0.50.3 sport=41796 dport=443 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.10.34 dst=3.7.13.58 sport=54856 dport=443 src=3.7.13.58 dst=100.90.79.102 sport=443 dport=54856 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=35179 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=35179 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.10.34 dst=65.1.97.76 sport=47726 dport=443 src=65.1.97.76 dst=100.90.79.102 sport=443 dport=47726 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 295 ESTABLISHED src=10.0.10.18 dst=10.0.70.9 sport=40370 dport=514 src=10.0.70.9 dst=10.0.10.18 sport=514 dport=40370 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 297 ESTABLISHED src=10.0.99.8 dst=10.0.50.8 sport=46620 dport=8086 src=10.0.50.8 dst=10.0.99.8 sport=8086 dport=46620 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 295 ESTABLISHED src=10.0.50.3 dst=10.0.10.34 sport=443 dport=41136 src=10.0.10.34 dst=10.0.50.3 sport=41136 dport=443 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.99.14 dst=1.0.0.1 sport=35190 dport=853 src=1.0.0.1 dst=100.90.79.102 sport=853 dport=35190 [ASSURED] maconntrack v1.4.6 (conntrack-tools): 74 flow entries have been deleted.
Oct 11 19:19:27 wan_lb[61995]: rk=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431998 ESTABLISHED src=10.0.99.14 dst=8.8.4.4 sport=34228 dport=853 src=8.8.4.4 dst=100.90.79.102 sport=853 dport=34228 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 29 src=139.84.164.110 dst=103.83.146.116 type=8 code=0 id=11907 src=103.83.146.116 dst=139.84.164.110 type=0 code=0 id=11907 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 29 src=10.0.160.148 dst=192.161.132.1 type=8 code=0 id=28694 [UNREPLIED] src=192.161.132.1 dst=103.83.146.116 type=0 code=0 id=28694 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32856 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32856 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 28 src=10.0.10.34 dst=10.0.99.14 sport=33907 dport=53 src=10.0.99.14 dst=10.0.10.34 sport=53 dport=33907 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.99.14 dst=1.0.0.1 sport=35210 dport=853 src=1.0.0.1 dst=100.90.79.102 sport=853 dport=35210 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 299 ESTABLISHED src=10.0.40.14 dst=157.240.239.11 sport=34746 dport=443 src=157.240.239.11 dst=100.90.79.102 sport=443 dport=34746 [ASSURED] mark=201 use=2
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.30.23 dst=49.44.59.15 sport=14500 dport=4500 [UNREPLIED] src=49.44.59.15 dst=100.90.79.102 sport=4500 dport=14500 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 299 ESTABLISHED src=10.0.10.34 dst=18.205.222.128 sport=52912 dport=443 [UNREPLIED] src=18.205.222.128 dst=100.90.79.102 sport=443 dport=52912 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.40.14 dst=157.240.13.14 sport=43345 dport=443 src=157.240.13.14 dst=100.90.79.102 sport=443 dport=43345 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.10.34 dst=54.151.204.41 sport=44352 dport=443 src=54.151.204.41 dst=100.90.79.102 sport=443 dport=44352 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 299 ESTABLISHED src=10.0.10.34 dst=10.0.99.14 sport=39726 dport=443 src=10.0.99.14 dst=10.0.10.34 sport=443 dport=39726 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431999 ESTABLISHED src=10.0.40.14 dst=157.240.15.19 sport=38184 dport=443 src=157.240.15.19 dst=100.90.79.102 sport=443 dport=38184 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 29 src=100.90.79.102 dst=1.0.0.1 type=8 code=0 id=5449 src=1.0.0.1 dst=100.90.79.102 type=0 code=0 id=5449 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: icmp     1 29 src=103.83.146.116 dst=1.0.0.1 type=8 code=0 id=5449 src=1.0.0.1 dst=103.83.146.116 type=0 code=0 id=5449 mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: udp      17 29 src=10.0.99.12 dst=255.255.255.255 sport=16720 dport=16720 [UNREPLIED] src=255.255.255.255 dst=10.0.99.12 sport=16720 dport=16720 mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32792 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32792 [ASSURED] mark=0 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 118 TIME_WAIT src=10.0.50.14 dst=171.101.117.139 sport=45101 dport=65531 src=171.101.117.139 dst=100.90.79.102 sport=65531 dport=45101 [ASSURED] mark=201 use=1
Oct 11 19:19:27 wan_lb[61995]: tcp      6 431997 ESTABLISHED src=10.1.1.3 dst=10.0.50.3 sport=32828 dport=9001 src=10.0.50.3 dst=10.1.1.3 sport=9001 dport=32828 [ASSURED] mark=0 use=1

Questions:

  1. What is enable-local-traffic exactly ?

I enabled this but I still had to add rules to avoid sending local traffic into load-balancing config.

2

WLB is still using legacy vyatta code and hasn’t been fully ported to our new XML/Python standard yet.

If it’s just failover you’re after, you could alternatively use: Failover — VyOS 1.4.x (sagitta) documentation

3

I did try this initially but I was a little bit confused and it wasn’t working well so I went back to using the load balancing feature.

If I do this, I’ll need separate scripts to clear conntrack and probably more firewall rules to keep track of connections on WAN2 so they can continue to be routed over WAN2 even when WAN1 recovers.

I’ll probably try to figure that out in the next few days.

4

Unfortunately, if your ISPs provide dynamic addresses (ie. DHCP or PPPoE), you can’t use failover routes as they are currently.

I requested expanding the feature to support this but it didn’t seem to get any traction.

5

It could be extended with load-balancing logic.
Feel free to add a feature request or extend the current.

6

Another issue I ran into!

I have a rule to force all DNS(udp/tcp 53) traffic towards my DNS server. The rule looks like this,

nat {
    destination {
        rule 1 {
            destination {
                group {
                    address-group "!dns_servers"
                }
                port "53"
            }
            disable
            protocol "tcp_udp"
            source {
                group {
                    address-group "!dns_servers"
                }
            }
            translation {
                address "10.0.99.14"
                port "53"
            }
        }
    }
}

This does not work with the load balancing feature enabled and any traffic to public resolver(that was redirected to my local resolver) times out.

(I have made a separate post for config review / feedback)