Scenario: I have two interfaces, eth0 (cable modem) and wwan0 (cellular modem). When the cable modem is operational, it should be used, only failing to cellular modem when necessary.
This mostly works fine, except when the cable modem experiences a failure without the link going down. In this case, the DHCP-learned default route stays around even while the wan load balancing has detected the failure.
Any ideas?
Config:
interfaces {
ethernet eth0 {
address dhcp
hw-id 00:0c:29:aa:aa:1e
}
ethernet eth1 {
address 10.254.254.1/24
hw-id 00:0c:29:aa:aa:28
}
loopback lo {
}
}
load-balancing {
wan {
flush-connections
interface-health eth0 {
failure-count 1
nexthop dhcp
success-count 1
test 1 {
resp-time 1
target 8.8.8.8
ttl-limit 1
}
}
interface-health wwan0 {
failure-count 1
nexthop dhcp
success-count 1
test 1 {
resp-time 1
target 1.1.1.1
ttl-limit 1
}
}
rule 1 {
failover
inbound-interface eth1
interface eth0 {
weight 250
}
interface wwan0 {
weight 1
}
protocol all
}
}
}
protocols {
static {
interface-route 0.0.0.0/0 {
next-hop-interface wwan0 {
distance 240
}
}
route 0.0.0.0/0 {
dhcp-interface eth0
}
}
}
service {
dhcp-server {
shared-network-name lan {
name-server 1.1.1.1
subnet 10.254.254.0/24 {
default-router 10.254.254.1
name-server 1.1.1.1
range 0 {
start 10.254.254.10
stop 10.254.254.250
}
}
}
}
ssh {
}
}
system {
config-management {
commit-revisions 100
}
conntrack {
modules {
ftp
h323
nfs
pptp
sip
sqlnet
tftp
}
}
console {
device ttyS0 {
speed 115200
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password ...
plaintext-password ""
}
}
}
ntp {
server time1.vyos.net {
}
server time2.vyos.net {
}
server time3.vyos.net {
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
}
[edit]
When both interfaces are up:
vyos@vyos:~$ show wan-load-balance
Interface: eth0
Status: active
Last Status Change: Fri Sep 2 00:58:28 2022
+Test: ping Target: 8.8.8.8
Last Interface Success: 0s
Last Interface Failure: 1m11s
# Interface Failure(s): 0
Interface: wwan0
Status: active
Last Status Change: Fri Sep 2 00:58:28 2022
+Test: ping Target: 1.1.1.1
Last Interface Success: 0s
Last Interface Failure: 1m11s
# Interface Failure(s): 0
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
S 0.0.0.0/0 [240/0] is directly connected, wwan0, weight 1, 00:00:13
S>* 0.0.0.0/0 [1/0] via 192.168.17.1, eth0, weight 1, 00:01:17
S 0.0.0.0/0 [210/0] via 192.168.17.1, eth0, weight 1, 00:01:17
S>* 8.8.8.8/32 [1/0] via 192.168.17.1, eth0, weight 1, 00:01:17
C>* 10.254.254.0/24 is directly connected, eth1, 00:01:43
C>* 167.20.XX.XX/30 is directly connected, wwan0, 00:01:21
C>* 192.168.17.0/24 is directly connected, eth0, 00:01:17
vyos@vyos:~$ show wan-load-balance status
Chain WANLOADBALANCE_PRE (1 references)
pkts bytes target prot opt in out source destination
118 15149 ISP_eth0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW
440 75908 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
When eth0 is hard down:
vyos@vyos:~$ show wan-load-balance
Interface: eth0
Status: failed
Last Status Change: Fri Sep 2 01:01:21 2022
-Test: ping Target: 8.8.8.8
Last Interface Success: 56s
Last Interface Failure: 0s
# Interface Failure(s): 5
Interface: wwan0
Status: active
Last Status Change: Fri Sep 2 00:58:28 2022
+Test: ping Target: 1.1.1.1
Last Interface Success: 0s
Last Interface Failure: 3m43s
# Interface Failure(s): 0
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
S>* 0.0.0.0/0 [240/0] is directly connected, wwan0, weight 1, 00:02:47
C>* 10.254.254.0/24 is directly connected, eth1, 00:04:17
C>* 167.20.XX.XX/30 is directly connected, wwan0, 00:03:55
vyos@vyos:~$ show wan-load-balance status
Chain WANLOADBALANCE_PRE (1 references)
pkts bytes target prot opt in out source destination
92 13032 ISP_wwan0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW
48 12168 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
When eth0 is up but cannot reach target:
vyos@vyos:~$ show wan-load-balance
Interface: eth0
Status: failed
Last Status Change: Fri Sep 2 01:04:39 2022
-Test: ping Target: 8.8.8.8
Last Interface Success: 20s
Last Interface Failure: 0s
# Interface Failure(s): 2
Interface: wwan0
Status: active
Last Status Change: Fri Sep 2 00:58:28 2022
+Test: ping Target: 1.1.1.1
Last Interface Success: 0s
Last Interface Failure: 6m26s
# Interface Failure(s): 0
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
S 0.0.0.0/0 [210/0] via 192.168.17.1, eth0, weight 1, 00:00:21
S>* 0.0.0.0/0 [1/0] via 192.168.17.1, eth0, weight 1, 00:00:21
S 0.0.0.0/0 [240/0] is directly connected, wwan0, weight 1, 00:05:25
S>* 8.8.8.8/32 [1/0] via 192.168.17.1, eth0, weight 1, 00:00:21
C>* 10.254.254.0/24 is directly connected, eth1, 00:06:55
C>* 167.20.XX.XX/30 is directly connected, wwan0, 00:06:33
C>* 192.168.17.0/24 is directly connected, eth0, 00:00:21
vyos@vyos:~$ show wan-load-balance status
Chain WANLOADBALANCE_PRE (1 references)
pkts bytes target prot opt in out source destination
54 14863 ISP_wwan0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW
4 468 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore