Multi-wan load balancing -- configurations have no effect

General questions
1

We have 3 ISPs (eth0, eth1, eth2) and our LAN is on eth4. wg101 is a wireguard interface. The below configuration has no effect – all traffic is routed through eth0.

Pinging “'24.29.97.15” for example goes through eth0 instead of eth1.

Default route specified on the gateway, but removing this does not change the behavior of load-balancing.

set protocols static route 0.0.0.0/0 interface eth0 distance '200'

set load-balancing wan interface-health eth0 failure-count '5'
set load-balancing wan interface-health eth0 nexthop 'dhcp'
set load-balancing wan interface-health eth0 success-count '1'
set load-balancing wan interface-health eth0 test 10 resp-time '5'
set load-balancing wan interface-health eth0 test 10 target '1.1.1.1'
set load-balancing wan interface-health eth0 test 10 ttl-limit '1'
set load-balancing wan interface-health eth0 test 10 type 'ttl'
set load-balancing wan interface-health eth0 test 20 resp-time '5'
set load-balancing wan interface-health eth0 test 20 target '1.0.0.1'
set load-balancing wan interface-health eth0 test 20 ttl-limit '1'
set load-balancing wan interface-health eth0 test 20 type 'ping'
set load-balancing wan interface-health eth1 failure-count '5'
set load-balancing wan interface-health eth1 nexthop 'dhcp'
set load-balancing wan interface-health eth1 success-count '1'
set load-balancing wan interface-health eth1 test 10 resp-time '5'
set load-balancing wan interface-health eth1 test 10 target '8.8.8.8'
set load-balancing wan interface-health eth1 test 10 ttl-limit '1'
set load-balancing wan interface-health eth1 test 10 type 'ttl'
set load-balancing wan interface-health eth1 test 20 resp-time '5'
set load-balancing wan interface-health eth1 test 20 target '8.8.4.4'
set load-balancing wan interface-health eth1 test 20 ttl-limit '1'
set load-balancing wan interface-health eth1 test 20 type 'ping'
set load-balancing wan interface-health eth2 failure-count '5'
set load-balancing wan interface-health eth2 nexthop 'dhcp'
set load-balancing wan interface-health eth2 success-count '1'
set load-balancing wan interface-health eth2 test 10 resp-time '5'
set load-balancing wan interface-health eth2 test 10 target '9.9.9.9'
set load-balancing wan interface-health eth2 test 10 ttl-limit '1'
set load-balancing wan interface-health eth2 test 10 type 'ttl'
set load-balancing wan interface-health eth2 test 20 resp-time '5'
set load-balancing wan interface-health eth2 test 20 target '149.112.112.112'
set load-balancing wan interface-health eth2 test 20 ttl-limit '1'
set load-balancing wan interface-health eth2 test 20 type 'ping'
set load-balancing wan interface-health wg101 failure-count '5'
set load-balancing wan interface-health wg101 nexthop 'dhcp'
set load-balancing wan interface-health wg101 success-count '1'
set load-balancing wan interface-health wg101 test 10 resp-time '5'
set load-balancing wan interface-health wg101 test 10 target '9.9.9.10'
set load-balancing wan interface-health wg101 test 10 ttl-limit '1'
set load-balancing wan interface-health wg101 test 10 type 'ttl'
set load-balancing wan interface-health wg101 test 20 resp-time '5'
set load-balancing wan interface-health wg101 test 20 target '149.112.112.10'
set load-balancing wan interface-health wg101 test 20 ttl-limit '1'
set load-balancing wan interface-health wg101 test 20 type 'ping'


set load-balancing wan rule 100 description 'SpeedTest - Spectrum'
set load-balancing wan rule 100 destination address '24.29.97.0/24'
set load-balancing wan rule 100 inbound-interface 'eth4'
set load-balancing wan rule 100 interface eth1 weight '1'

set load-balancing wan rule 110 description 'SpeedTest - NaturalWireless'
set load-balancing wan rule 110 destination address '163.182.128.0/24'
set load-balancing wan rule 110 inbound-interface 'eth4'
set load-balancing wan rule 110 interface eth2 weight '1'

set load-balancing wan rule 1000 description 'DEFAULT FAILOVER RULE'
set load-balancing wan rule 1000 failover
set load-balancing wan rule 1000 inbound-interface 'eth4'
set load-balancing wan rule 1000 interface eth0 weight '3'
set load-balancing wan rule 1000 interface eth1 weight '2'
set load-balancing wan rule 1000 interface eth2 weight '1'
set load-balancing wan rule 1000 protocol 'all'

set load-balancing wan sticky-connections inbound
2

Hello @ajgnet,

Which version of VyOS are you using?

3

Hello, *vyos-1.4-rolling-202204060217-amd64.iso

4

hi

I think you need to set the source traffic , so wan-load balance feature can apply the rule . here is an example:

set load-balancing wan rule 100 description 'SpeedTest - Spectrum'
set load-balancing wan rule 100 destination address '24.29.97.0/24'
set load-balancing wan rule 100 inbound-interface 'eth3'
set load-balancing wan rule 100 interface eth1 weight '1'
set load-balancing wan rule 100 protocol 'all'
set load-balancing wan rule 100 source address '172.16.80.0/24'

if everything is ok , you should see match on this rule:

vyos@test-wan:~$ show wan-load-balance status
Chain WANLOADBALANCE_PRE (1 references)
 pkts bytes target     prot opt in     out     source               destination
   32  2688 ISP_eth1   all  --  eth3   *       172.16.80.0/24       24.29.97.0/24        state NEW
5

The rule gets matched in show wan-load-balance status by both the default rule, and the destination ip rule. But it is never applied

6

I’ve replicated this behavior and opened a case where I describe this issue :

https://phabricator.vyos.net/T4352

you can add any comment or test , if you want

regards

7

Can you try adding this rule to firewall.

 sudo iptables-nft -t nat -A POSTROUTING -j VYOS_PRE_SNAT_HOOK
8

The rule already exists at the end of the POSTROUTING chain. When re-adding there is no impact.

This appears to be an issue with VyOS not creating the policy tables:

me@gw01# sudo ip rule
0:	from all lookup local
98:	from all fwmark 0xcc lookup 204
99:	from all fwmark 0xcb lookup 203
100:	from all fwmark 0xca lookup 202
101:	from all fwmark 0xc9 lookup 201
102:	from all fwmark 0x7fffff99 lookup 102
114:	from all fwmark 0x7fffff8d lookup 114
32766:	from all lookup main
32767:	from all lookup default

me@gw01# sudo ip r show table 202
Error: ipv4: FIB table does not exist.
Dump terminated
[edit]
me@gw01# sudo ip r show table 203
Error: ipv4: FIB table does not exist.
Dump terminated
[edit]
me@gw01# sudo ip r show table 204
Error: ipv4: FIB table does not exist.
Dump terminated

Are there currently smoke tests for wan load balancing and policy routing with multiple external interfaces? There appear to be frequent breaking changes in these two areas and maybe this is a way to improve reliability with the current branch.