Failover loadbalance with PPPOE

General questions
1

i’m continue my experiments with vyos
now i try move test configuration to production environment with include:

  • eth0 - my lan, DHCP and DNS places on dedicated server (subnet 192.168.1.0/24)
  • eth1 - my ISP1 with DCHP and PPPOE over it
  • eth2 - my ISP2 with static IP
  1. i try to connect ISP2 and all works perfectly - internet available on router and network computers
  2. i try to connect ISP1 and again all works perfectly - PPPOE connection established in several seconds, and restore automatic after reboot
  3. i try to remove ISP1 cable for emulate ISP fail. I expect that wan balancer check it and switch to ISP2, but than i see wan balancer status i see that pppoe0 hve status ACTIVE (but i really can’t ping target (8.8.8.8)). I wait for a minute but pppoe0 still have active state

this is my load balancer config
load-balancing {
wan {
interface-health eth2 {
failure-count 1
nexthop 83.167.84.161
success-count 1
test 10 {
resp-time 5
target 8.8.8.8
ttl-limit 1
type ping
}
}
interface-health pppoe0 {
failure-count 1
nexthop dhcp
success-count 1
test 10 {
resp-time 5
target 8.8.8.8
ttl-limit 1
type ping
}
}
}

what i was miss in my configuration?

2

Your config seems correct (i.e. I compared it to my own), however I do see you are missing the actual rule items, that would make the load-balancing actually work. (I’m not sure why the connection appears ACTIVE, but it might be that the load-balancing is not activated without any rules actually using it?)

Regarding your interface-health configuration, I would increase failure-count to something like 3 as only one “glitch” would take your connection down and reset your connections. (The same should apply also for success-count.)

Moreover I would suggest to also apply flush-connections (especially since you also do NAT), because without it your connections would just “stall” and timeout once the failover happens.

3

i’m already have rule section (just missed it in post):
rule 10 {
failover
inbound-interface eth0
interface eth2 {
weight 1
}
interface pppoe0 {
weight 10
}
protocol all
}

about flush-connection i haven’t read yet, thanks. i will search documentation about it

4

Try to do the following experiment: disconnect the main cable, let it settle for a minute and see what the command sudo ip route get 9.9.9.9 gives. (The ip 9.9.9.9 does not matter.)

The command shows you the route the packet will take. Perhaps the routes are updated accordingly, it’s just that the ACTIVE status is somehow not “updated”.

Also try a traceroute to an external IP.

5

thank you, but i can try this only on Tuesday. i will report about result.

6

ok, as i wrote i try your suggestion and… all work fine! i think, that my mistake in previous experiment is that second ISP already down too… now i connect both ISP and when i disconnect one of them status changed and route changed too. so, it’s work!

but now i have different issue. and this is very strange for me.

now i have configuration that allow me two use two ISP in failover mode. On the main ISP (which established over pppoe) i have l2tp vpn with radius server. Radius server located on my win domain machine. On this machine a have primary dns too (secondary dns located on other machine)
so, with addresses i have follow infrastructure:
vyos internet server:
eth0 - 192.168.0.1/24 - local adapter with static address
eth1 + pppoe0 - main ISP
eth2 - some statis ip to second ISP
192.168.0.10 - primary dns, ldap, radius, dhcp
192.168.0.11 - secondary dns, secondary domain controller

wan-balancer works - i see that both connections are alive and i can ping some external ip, traceroute to them
now i open browser on my work machine (in same subnet 192.168.0.0/24, with gateway 192.168.0.1 and dns 192.168.0.10,192.168.0.11). I try to ping for example vyos.net

ping vyos. net (space because i have limit to post link, of course i try to ping and traceroute without it))

Pinging vyos. net [46.253.202.165] with 32 bytes of data:
Reply from 46.253.202.165: bytes=32 time=55ms TTL=49
Reply from 46.253.202.165: bytes=32 time=55ms TTL=49

Ping statistics for 46.253.202.165:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 55ms, Maximum = 55ms, Average = 55ms
Control-C
^C

now i want to traceroute to them:

tracert vyos.net

Tracing route to v y o s . n e t [46.253.202.165]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms is.singularis.local [192.168.0.1]
2 <1 ms <1 ms 1 ms 10.93.255.254
3 5 ms <1 ms 8 ms … [88.87.67.46]
4 18 ms 16 ms 17 ms … [87.245.243.62]
5 16 ms 16 ms 16 ms … [87.245.243.61]
6 44 ms 44 ms 44 ms … [87.245.233.74]
7 44 ms 45 ms 44 ms … [194.68.123.194]
8 45 ms 50 ms 44 ms … [80.67.4.187]
9 53 ms 53 ms 53 ms … [80.67.0.226]
10 55 ms 55 ms 55 ms 46.253.202.165

Trace complete.

looks good
but now, i want to open it in browser… and nothing. i get message that site cannot reach. and same go on with other sites. but i see that some of sites open correctly (for example google. com), but i cant find only two of such sites. in my opinion there are some problems with dns, but i don’t understand what exactly, because when i switch server back to current solution (based on win 2008 + RAAS) all works perfectly!

may be you can give me some advise for direction of search?
vyos look’s very good and i want to switch to it, because raas have less feature

7

one more question that i forgot:
when client established l2tp connection it can ping resources inside LAN, but cannot access to internet resources at all (event ping it). i think i miss some configuration
this is that i have:
vpn {
ipsec {
ipsec-interfaces {
interface pppoe0
}
nat-networks {
allowed-network 0.0.0.0/0
}
nat-traversal enable
}
l2tp {
remote-access {
authentication {
mode radius
radius server 192.168.0.10 {
key ***
}
}
client-ip-pool {
start 192.168.0.150
stop 192.168.0.200
}
dns-servers {
server-1 192.168.0.10
server-2 192.168.0.11
}
ipsec-settings {
authentication {
mode pre-shared-secret
pre-shared-secret ***
}
ike-lifetime 3600
}
outside-address ...
}

8

There are two most likely possibilities:

  • (A) You are using the WAN-loadbalancer in an actual “load-balancing” mode (i.e. without failover), and thus some packets in the TCP stream exit via one provider and others via another.
  • (B) Your DNS servers are not updated when you are switching between the two ISP connections.

Please check both of the above.

Also try to tcpdump the ISP connections and make sure that traffic exits only via one of them.

9

Regarding the L2TP / IPSec setup, I don’t have experience with those, however I would advise against using such features in combination with WAN-loadbalancing…

If you have such a complex setup I would advise having one router just handling the WAN-loadbalancing, and another router handling the rest.

[internet-ISP1] --/-- [WAN-vyos] -- [internal-vyos] -\-- [LAN1]
[internet-ISP2] -/                                    \- [LAN2]
1 Like
10

i try to move in another direction - i disabled all services and keep only wan-balancing
also, i change my dns settings on client to use google dns (8.8.8.8,.8.8.4.4)

now, i try to ping some resources from vyos router - all works fine
i try to traceroute site from vyos router - all works fine
i try to curl page and also works fine (i can see page contents)

now i switch to client computer
i still can ping site
also i can traceroute to them
but i can’t execute curl (and browse from browser app)

i think, that there are some problems with nat/firewall, but i didn’t configure it (i didn’t find any note in wiki about load balancing)

there are my current config:
interfaces {
ethernet eth0 {
address 192.168.0.1/24
description INTERNAL-LAN
duplex auto
hw-id 14:da:e9:da:bd:3f
smp_affinity auto
speed auto
}
ethernet eth1 {
address dhcp
description WAN1-KOT
duplex auto
hw-id 00:40:f4:34:c0:cb
pppoe 0 {
default-route auto
local-address ...
mtu 1492
name-server auto
password ****
user-id ****
}
smp_affinity auto
speed auto
}
ethernet eth2 {
address .../30
description WAN2-ALEKS
duplex auto
hw-id 00:e0:4c:9f:05:bb
smp_affinity auto
speed auto
}
loopback lo {
}
}
load-balancing {
wan {
flush-connections
interface-health eth2 {
failure-count 1
nexthop ...
success-count 1
test 10 {
resp-time 5
target 8.8.8.8
ttl-limit 1
type ping
}
}
interface-health pppoe0 {
failure-count 1
nexthop dhcp
success-count 1
test 10 {
resp-time 5
target 8.8.8.8
ttl-limit 1
type ping
}
}
rule 10 {
failover
inbound-interface eth0
interface eth2 {
weight 1
}
interface pppoe0 {
weight 10
}
protocol all
}
}
}
service {
ssh {
port 22
}
}
system {
config-management {
commit-revisions 20
}
console {
device ttyS0 {
speed 9600
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password ******
plaintext-password “”
}
level admin
}
}
ntp {
server 0 . pool . ntp . org {
}
server 1 . pool . ntp . org {
}
server 2 . pool . ntp . org {
}
}
package {
auto-sync 1
repository community {
components main
distribution helium
password “”
url http :// packages . vyos . net / vyos
username “”
}
}
syslog {
global {
facility all {
level notice
}
facility protocols {
level debug
}
}
}
time-zone UTC
}

/* Warning: Do not remove the following line. /
/ === vyatta-config-version: “cluster@1:config-management@1:conntrack-sync@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@4:nat@4:qos@1:quagga@2:system@6:vrrp@1:wanloadbalance@3:webgui@1:webproxy@1:zone-policy@1” === /
/ Release version: VyOS 1.1.8 */

11

Sorry I was not paying attention and missed that you are using PPPoE, which most likely means you to should TCP-MSS-clamping, else TCP will fail to work with most sites. (Google with Chrome might work as they use HTTP over QUIC, but others will fail)…

Basically if you do a tcpdump on the outgoing interface you’ll see that most TCP connections will stall and retransmit constantly before they eventually timeout and reset.

And now the bad news… TCP MSS clamping and WAN load-balancing don’t mix well together. In fact I stumbled into this issue at first and had to do quite a bit of hacking to make it right.

I’ve described the whole problem and solution in the following thread.

After you read the above (and apply it) you’ll also have to configure TCP-MSS clamping as bellow:

policy {
    route pppoe-mangle-in {
        rule 1 {
            protocol tcp
            set {
                tcp-mss 1452
            }
            tcp {
                flags SYN,!RST
            }
        }
    }
    route pppoe-mangle-out {
        rule 1 {
            destination {
                group {
                    network-group !lan
                }
            }
            protocol tcp
            set {
                tcp-mss 1452
            }
            tcp {
                flags SYN,!RST
            }
        }
    }
...
}

Then apply it to your pppoe interfaces like so:

        pppoe 0 {
            ....
            policy {
                route pppoe-mangle-in
            }

Then also to all your “lan” interfaces like so:

        eth0 {
            ....
            policy {
                route pppoe-mangle-out
            }

Also notice that there is a network-group called lan which should contain all your local networks… (You’ll also have to adjust the MTU of 1452 if the PPPoE overhead in your case is larger.

However if you are doing this in a larger deployment, and especially if you are using VM’s, I would strongly suggest that you use two separate VyOS routers, one for WAN balancing, and the other one for NAT, DHCP, TCP MSS clamping, etc.

This will keep your setup “sane”, especially since WAN-load-balancing doesn’t play nice with almost none other feature…

13

ok… i try your advice today and… no luck… at all
my current config looks like this:

interfaces {
    ethernet eth0 {
        address 192.168.0.1/24
        description INTERNAL-LAN
        duplex auto
        hw-id 14:da:e9:da:bd:3f
        policy {
            route pppoe-mangle-out
        }
        smp_affinity auto
        speed auto
    }
    ethernet eth1 {
        address dhcp
        description WAN1-KOT
        duplex auto
        hw-id 00:40:f4:34:c0:cb
        pppoe 0 {
            default-route auto
            local-address *.*.*.*
            mtu 1492
            name-server auto
            password ****
            policy {
                route pppoe-mangle-in
            }
            user-id ****
        }
        smp_affinity auto
        speed auto
    }
    ethernet eth2 {
        address *.*.*.*/30
        description WAN2-ALEKS
        duplex auto
        hw-id 00:e0:4c:9f:05:bb
        smp_affinity auto
        speed auto
    }
    loopback lo {
    }
}
load-balancing {
    wan {
        disable-source-nat
        flush-connections
        hook /config/scripts/wan-lb-hook.script
        interface-health eth2 {
            failure-count 1
            nexthop *.*.*.*
            success-count 1
            test 10 {
                resp-time 5
                target 8.8.8.8
                ttl-limit 1
                type ping
            }
        }
		interface-health pppoe0 {
            failure-count 1
            nexthop dhcp
            success-count 1
            test 10 {
                resp-time 5
                target 8.8.8.8
                ttl-limit 1
                type ping
            }
        }
        rule 10 {
            failover
            inbound-interface eth0
            interface eth2 {
                weight 1
            }
            interface pppoe0 {
                weight 10
            }
            protocol all
        }
    }
}
policy {
    route pppoe-mangle-in {
        rule 10 {
            protocol tcp
            set {
                tcp-mss 1452
			}
            tcp {
                flags SYN,!RST
            }
        }
    }
    route pppoe-mangle-out {
        rule 10 {
            destination {
                address !192.168.0.0/24
            }
            protocol tcp
            set {
                tcp-mss 1452
            }
            tcp {
                flags SYN,!RST
            }
        }
    }
}
service {
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password ****
                plaintext-password ""
            }
            level admin
        }
    }
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    package {
        auto-sync 1
        repository community {
            components main
            distribution helium
            password ""
            url http://packages.vyos.net/vyos
            username ""
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "cluster@1:config-management@1:conntrack-sync@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@4:nat@4:qos@1:quagga@2:system@6:vrrp@1:wanloadbalance@3:webgui@1:webproxy@1:zone-policy@1" === $
/* Release version: VyOS 1.1.8 */

and my hook script
#!/bin/bash

set -e -E -u -o pipefail -o noclobber -o noglob +o braceexpand || exit 1
trap 'printf -- "[ee] failed: %s\n" "${BASH_COMMAND}" >&2' ERR || exit 1

test "${#}" -eq 0
test -n "${WLB_INTERFACE_NAME:?}"
test -n "${WLB_INTERFACE_STATE:?}"

case "${WLB_INTERFACE_STATE}" in
        ( ACTIVE )
                iptables -t mangle -D "ISP_${WLB_INTERFACE_NAME}" -j ACCEPT
                iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}" 1 -j RETURN -m connmark ! --mark 0
                iptables -t mangle -I "ISP_${WLB_INTERFACE_NAME}_IN" -j RETURN -m connmark ! --mark 0
        ;;
        ( FAILED)
        ;;
esac

exit -- 0

now if my primary ISP is ok than i can ping internet resources from router, but can’t do it from client!
load balance status
vyos@vyos:~$ show wan-load-balance
Interface: eth2
Status: active
Last Status Change: Fri Aug 24 06:07:06 2018
+Test: ping Target: 8.8.8.8
Last Interface Success: 0s
Last Interface Failure: n/a
# Interface Failure(s): 0

Interface:  pppoe0
  Status:  active
  Last Status Change:  Fri Aug 24 06:10:56 2018
  +Test:  ping  Target: 8.8.8.8
    Last Interface Success:  0s
    Last Interface Failure:  4m47s
    # Interface Failure(s):  0

tcpdump
vyos@vyos:~$ tcpdump
06:17:17.035538 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [.], ack 200124, win 2053, length 0
06:17:17.035747 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [.], ack 202052, win 2053, length 0
06:17:17.035867 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [.], ack 203980, win 2053, length 0
06:17:17.035937 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [.], ack 205272, win 2048, length 0
06:17:17.038497 IP sad.domain.local.56918 > google-public-dns-a . google . com . domain: 2774+ A? tsfe. trafficshaping . dsp . mp . microsoft . com. (58)
06:17:17.038592 IP is.domain.local.44458 > ad.domain.local.domain: 32552+ PTR? 8.8.8.8.in-addr.arpa. (38)
06:17:17.039466 IP ad.domain.local.62901 > google-public-dns-a . google . com . domain: 542+ PTR? 8.8.8.8.in-addr.arpa. (38)
06:17:17.050090 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [P.], seq 409:493, ack 205272, win 2048, length 84
06:17:17.050108 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [P.], seq 493:545, ack 205272, win 2048, length 52
06:17:17.050127 IP is.domain.local.ssh > pc-34.domain.local.49754: Flags [.], ack 545, win 821, length 0
06:17:17.050152 IP is.domain.local.ssh > pc-34.domain.local.49754: Flags [P.], seq 205272:205440, ack 545, win 821, length 168
06:17:17.050174 IP is.domain.local.ssh > pc-34.domain.local.49754: Flags [P.], seq 205440:206228, ack 545, win 821, length 788
06:17:17.050958 ARP, Request who-has is.domain.local (14:da:e9:da:bd:3f (oui Unknown)) tell pc-24.domain.local, length 46
06:17:17.050970 ARP, Reply is.domain.local is-at 14:da:e9:da:bd:3f (oui Unknown), length 28
06:17:17.051042 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [.], ack 206228, win 2053, length 0
06:17:17.053385 IP pc-15.domain.local.49753 > 157.55.56.142.40027: Flags [S], seq 3111884987, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
06:17:17.058432 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [P.], seq 545:629, ack 206228, win 2053, length 84
06:17:17.058450 IP pc-34.domain.local.49754 > is.domain.local.ssh: Flags [P.], seq 629:681, ack 206228, win 2053, length 52
06:17:17.058470 IP is.domain.local.ssh > pc-34.domain.local.49754: Flags [.], ack 681, win 821, length 0
06:17:17.058489 IP is.domain.local.ssh > pc-34.domain.local.49754: Flags [P.], seq 206228:206264, ack 681, win 821, length 36
06:17:17.064049 IP pc-32.domain.local.49906 > 185.72.247.32.socks: Flags [S], seq 3224682067, win 17520, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
06:17:17.067752 IP pc-32.domain.local.49907 > 185.72.247.32.socks: Flags [S], seq 540582427, win 17520, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
06:17:17.087120 IP pc-24.domain.local.62251 > google-public-dns-a . google . com . domain: 15578+ A? google . com. (28)
06:17:17.087665 IP ad.domain.local.61357 > google-public-dns-a . google . com . domain: 37763+ A? google . com. (28)
06:17:17.089233 IP pc-24.domain.local.60287 > 104.130.209.20.9095: Flags [S], seq 771173741, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
06:17:17.089458 IP pc-24.domain.local.60288 > 104.130.211.185.1457: Flags [S], seq 698906952, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
^C^C
^C06:17:17.089669 IP pc-24.domain.local.60289 > 23.253.156.234.3398: Flags [S], seq 2839844023, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

1204 packets captured
6644 packets received by filter
5410 packets dropped by kernel

messages from log
vyos@vyos:~$ tail /var/log/messages
Aug 24 06:10:56 vyos wan_lb: Interface pppoe0 has changed state to ACTIVE
Aug 24 06:10:56 vyos wan_lb: executing script: /config/scripts/wan-lb-hook.script
Aug 24 06:11:07 vyos wan_lb: wan_lb, rechecking interfaces…
Aug 24 06:18:02 vyos wan_lb: wan_lb, rechecking interfaces…

ip route
vyos@vyos:~$ ip route get 8.8.8.8
8.8.8.8 dev pppoe0 src 88.87.90.54
cache

iptables
vyos@vyos:~$ iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
VYATTA_PRE_FW_IN_HOOK all – anywhere anywhere
VYATTA_POST_FW_IN_HOOK all – anywhere anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
VYATTA_PRE_FW_FWD_HOOK  all  --  anywhere             anywhere
VYATTA_POST_FW_FWD_HOOK  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
VYATTA_PRE_FW_OUT_HOOK  all  --  anywhere             anywhere
VYATTA_POST_FW_OUT_HOOK  all  --  anywhere             anywhere

Chain VYATTA_POST_FW_FWD_HOOK (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain VYATTA_POST_FW_IN_HOOK (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain VYATTA_POST_FW_OUT_HOOK (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain VYATTA_PRE_FW_FWD_HOOK (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain VYATTA_PRE_FW_IN_HOOK (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain VYATTA_PRE_FW_OUT_HOOK (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

now i turn off ISP1 and now i can’t ping anything in internet even from router
load balancer status
vyos@vyos:~$ show wan-load-balance
Interface: eth2
Status: active
Last Status Change: Fri Aug 24 06:07:06 2018
+Test: ping Target: 8.8.8.8
Last Interface Success: 0s
Last Interface Failure: n/a
# Interface Failure(s): 0

Interface:  pppoe0
  Status:  failed
  Last Status Change:  Fri Aug 24 06:21:05 2018
  -Test:  ping  Target: 8.8.8.8
    Last Interface Success:  7s
    Last Interface Failure:  0s
    # Interface Failure(s):  1

ip route
vyos@vyos:~$ ip route get 8.8.8.8
8.8.8.8 dev pppoe0 src 88.87.90.54
cache

iptables
vyos@vyos:~$ iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
VYATTA_PRE_FW_IN_HOOK all – anywhere anywhere
VYATTA_POST_FW_IN_HOOK all – anywhere anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
VYATTA_PRE_FW_FWD_HOOK  all  --  anywhere             anywhere
VYATTA_POST_FW_FWD_HOOK  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
VYATTA_PRE_FW_OUT_HOOK  all  --  anywhere             anywhere
VYATTA_POST_FW_OUT_HOOK  all  --  anywhere             anywhere

Chain VYATTA_POST_FW_FWD_HOOK (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain VYATTA_POST_FW_IN_HOOK (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain VYATTA_POST_FW_OUT_HOOK (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain VYATTA_PRE_FW_FWD_HOOK (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain VYATTA_PRE_FW_IN_HOOK (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain VYATTA_PRE_FW_OUT_HOOK (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

messages from log
Aug 24 06:21:05 vyos wan_lb: Interface pppoe0 has changed state to FAILED
Aug 24 06:21:05 vyos wan_lb: executing script: /config/scripts/wan-lb-hook.script
Aug 24 06:22:14 vyos pppd[4208]: Serial link appears to be disconnected.
Aug 24 06:22:14 vyos zebra[2635]: interface pppoe0 index 5 changed <POINTOPOINT,NOARP,MULTICAST>.
Aug 24 06:22:15 vyos pluto[4269]: shutting down
Aug 24 06:22:15 vyos pluto[4269]: forgetting secrets
Aug 24 06:22:15 vyos pluto[4269]: “remote-access-mac-zzz”: deleting connection
Aug 24 06:22:15 vyos pluto[4269]: “remote-access-win-aaa”: deleting connection
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface lo/lo ::1
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface lo/lo 127.0.0.1
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface lo/lo 127.0.0.1
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface eth0/eth0 192.168.0.1
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface eth0/eth0 192.168.0.1
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface eth2/eth2 ...
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface eth2/eth2 ...
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface pppoe0/pppoe0 ...
Aug 24 06:22:15 vyos pluto[4269]: shutting down interface pppoe0/pppoe0 ...
Aug 24 06:22:15 vyos ipsec_starter[4268]: pluto stopped after 20 ms
Aug 24 06:22:15 vyos ipsec_starter[4268]: charon stopped after 200 ms
Aug 24 06:22:15 vyos ipsec_starter[4268]: ipsec starter stopped
Aug 24 06:22:16 vyos wan_lb: wan_lb: error on sending icmp packet: 101
Aug 24 06:22:18 vyos ipsec_starter[5577]: Starting strongSwan 4.5.2 IPsec [starter]…
Aug 24 06:22:18 vyos ipsec_starter[5577]: no default route - cannot cope with %defaultroute!!!
Aug 24 06:22:18 vyos pluto[5597]: Starting IKEv1 pluto daemon (strongSwan 4.5.2) THREADS SMARTCARD VENDORID CISCO_QUIRKS
Aug 24 06:22:18 vyos pluto[5597]: including NAT-Traversal patch (Version 0.6c)
Aug 24 06:22:18 vyos pluto[5597]: failed to load pkcs11 module ‘/usr/lib/opensc-pkcs11.so’
Aug 24 06:22:18 vyos ipsec_starter[5596]: pluto (5597) started after 20 ms
Aug 24 06:22:18 vyos ipsec_starter[5596]: charon (5647) started after 20 ms
Aug 24 06:22:18 vyos pluto[5597]: Changing to directory ‘/etc/ipsec.d/crls’
Aug 24 06:22:18 vyos pluto[5597]: loading secrets from “/etc/ipsec.secrets”
Aug 24 06:22:18 vyos pluto[5597]: loading secrets from “/etc/dmvpn.secrets”
Aug 24 06:22:18 vyos pluto[5597]: loaded PSK secret for 88.87.90.54 %any
Aug 24 06:22:18 vyos pluto[5597]: Changing to directory ‘/etc/ipsec.d/crls’
Aug 24 06:22:18 vyos pluto[5597]: listening for IKE messages
Aug 24 06:22:18 vyos pluto[5597]: adding interface eth2/eth2 ...:500
Aug 24 06:22:18 vyos pluto[5597]: adding interface eth2/eth2 ...:4500
Aug 24 06:22:18 vyos pluto[5597]: adding interface eth0/eth0 192.168.0.1:500
Aug 24 06:22:18 vyos pluto[5597]: adding interface eth0/eth0 192.168.0.1:4500
Aug 24 06:22:18 vyos pluto[5597]: adding interface lo/lo 127.0.0.1:500
Aug 24 06:22:18 vyos pluto[5597]: adding interface lo/lo 127.0.0.1:4500
Aug 24 06:22:18 vyos pluto[5597]: adding interface lo/lo ::1:500
Aug 24 06:22:18 vyos pluto[5597]: forgetting secrets
Aug 24 06:22:18 vyos pluto[5597]: loading secrets from “/etc/ipsec.secrets”
Aug 24 06:22:18 vyos pluto[5597]: loading secrets from “/etc/dmvpn.secrets”
Aug 24 06:22:18 vyos pluto[5597]: loaded PSK secret for ... %any
Aug 24 06:22:18 vyos pluto[5597]: added connection description “remote-access-win-aaa”
Aug 24 06:22:18 vyos pluto[5597]: added connection description “remote-access-mac-zzz”
Aug 24 06:22:18 vyos pluto[5597]: the protocol must be the same for leftport and rightport
Aug 24 06:22:18 vyos ntpd[4390]: ntpd exiting on signal 15
Aug 24 06:22:20 vyos ntpd[5716]: ntpd 4.2.6p2@1.2194-o Fri Oct 13 03:32:58 UTC 2017 (1)
Aug 24 06:22:20 vyos ntpd[5717]: proto: precision = 0.106 usec
Aug 24 06:22:21 vyos pppd[4208]: Connection terminated: no multilink.
Aug 24 06:22:21 vyos zebra[2635]: interface pppoe0 index 5 deleted.
Aug 24 06:22:21 vyos ripd[2637]: interface delete pppoe0 index 5 flags 0x1090 metric 1 mtu 1492
Aug 24 06:22:21 vyos ripngd[2639]: interface delete pppoe0 index 5 flags 0x1090 metric 1 mtu 1492
Aug 24 06:22:21 vyos pppd[4208]: Modem hangup
Aug 24 06:22:27 vyos wan_lb: wan_lb: failure to bind to interface: pppoe0
Aug 24 06:23:00 vyos wan_lb: last message repeated 3 times
Aug 24 06:23:26 vyos wan_lb: last message repeated 2 times
Aug 24 06:23:26 vyos pppd[4208]: Timeout waiting for PADO packets
Aug 24 06:23:26 vyos pppd[4208]: Unable to complete PPPoE Discovery
Aug 24 06:23:28 vyos ntpd_intres[5735]: host name not found: 0.pool.ntp.org
Aug 24 06:23:33 vyos wan_lb: wan_lb: failure to bind to interface: pppoe0