DNS forwarding issue, Wan load balancing issue

Running 1.3.4, and having two issues.

First when visiting new sites dns resolution takes a long time, sometimes up to 3 seconds, and some websites will not load at all. Looking at firefox web console I’m seeing errors such as NS_ERROR_NET_TIMEOUT and NS_ERROR_NET_INTERRUPT. It seems to function fine when using dig from the terminal, but having the above issues when using a browser across multiple clients.

Second, I have configured wan load balancing in failover mode and whenever I connect the primary connection (eth1) I lose internet connectivity from lan and also lose ssh access from lan to the router. I can ping the router but I cannot ssh into it. If I remove the primary connection it falls back to the secondary (pppoe0) and everything works again.

config.boot

firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name lan-local {
        default-action accept
        enable-default-log
    }
    name lan-wan {
        default-action accept
        enable-default-log
    }
    name local-lan {
        default-action accept
        enable-default-log
    }
    name local-wan {
        default-action accept
        enable-default-log
    }
    name wan-lan {
        default-action drop
        enable-default-log
        rule 5 {
            action accept
            description "Allow EST/Related Traffic"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            protocol icmp
            state {
                new enable
            }
        }
    }
    name wan-local {
        default-action drop
        enable-default-log
        rule 5 {
            action accept
            description "Allow EST/Related Traffic"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            protocol icmp
            state {
                new enable
            }
        }
        rule 30 {
            action accept
            description "Allow Wireguard Traffic"
            destination {
                port 51820
            }
            protocol udp
            source {
            }
            state {
                new enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
    twa-hazards-protection disable
}
interfaces {
    ethernet eth0 {
        hw-id 00:1a:8c:45:4a:b0
    }
    ethernet eth1 {
        address dhcp
        hw-id 00:1a:8c:45:4a:b1
    }
    ethernet eth2 {
        disable
        hw-id 00:1a:8c:45:4a:b2
    }
    ethernet eth3 {
        address 10.5.10.1/24
        hw-id 00:1a:8c:45:4a:b3
        vif 520 {
            address 10.5.20.1/24
        }
        vif 521 {
            address 10.5.21.1/24
        }
        vif 522 {
            address 10.5.22.1/24
        }
        vif 530 {
            address 10.5.30.1/24
        }
        vif 531 {
            address 10.5.31.1/24
        }
        vif 532 {
            address 10.5.32.1/24
        }
        vif 542 {
            address 10.5.42.1/24
        }
    }
    loopback lo {
    }
    pppoe pppoe0 {
        authentication {
            password <REDACTED>
            user <REDACTED>
        }
        description "Centurylink PPPOE"
        source-interface eth0
    }
    wireguard wg01 {
        address 172.16.10.2/24
        peer to-iphone {
            allowed-ips 172.16.10.101/32
            pubkey <REDACTED>
        }
        peer to-r1 {
            address <REDACTED>
            allowed-ips 172.16.10.1/32
            allowed-ips 10.4.0.0/16
            allowed-ips 10.68.0.0/16
            persistent-keepalive 15
            port 51820
            pubkey <REDACTED>
        }
        port 51820
    }
    wireless wlan0 {
        disable
        hw-id d0:ab:d5:45:43:db
        physical-device phy0
    }
    wireless wlan1 {
        disable
        hw-id 00:0e:8e:69:a1:62
        physical-device phy0
    }
}
load-balancing {
    wan {
        enable-local-traffic
        flush-connections
        interface-health eth1 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        interface-health pppoe0 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        rule 520 {
            failover
            inbound-interface eth3.520
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
        rule 521 {
            failover
            inbound-interface eth3.521
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
        rule 522 {
            failover
            inbound-interface eth3.522
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
        rule 530 {
            failover
            inbound-interface eth3.530
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
        rule 531 {
            failover
            inbound-interface eth3.531
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
        rule 532 {
            failover
            inbound-interface eth3.532
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
        rule 542 {
            failover
            inbound-interface eth3.542
            interface eth1 {
                weight 9
            }
            interface pppoe0 {
                weight 1
            }
            protocol all
        }
    }
}
nat {
    source {
        rule 100 {
            outbound-interface pppoe0
            source {
                address 10.5.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    static {
        interface-route 172.16.10.1/32 {
            next-hop-interface wg01 {
            }
        }
        route 10.4.0.0/16 {
            next-hop 172.16.10.1 {
            }
        }
    }
}
service {
    dhcp-relay {
        interface eth3.530
        interface eth3.531
        interface eth3.532
        server 10.4.30.15
    }
    dhcp-server {
        shared-network-name HOME-INT {
            subnet 10.5.20.0/24 {
                default-router 10.5.20.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                ping-check
                range 0 {
                    start 10.5.20.210
                    stop 10.5.20.250
                }
            }
        }
        shared-network-name HOME-GUEST-WIFI {
            subnet 10.5.42.0/24 {
                default-router 10.5.42.1
                name-server 10.5.10.1
                range 0 {
                    start 10.5.42.10
                    stop 10.5.42.199
                }
            }
        }
        shared-network-name HOME-INT-CLIENT {
            name-server 10.5.10.1
            subnet 10.5.21.0/24 {
                default-router 10.5.21.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                ping-check
                range 0 {
                    start 10.5.21.10
                    stop 10.5.21.250
                }
            }
        }
        shared-network-name HOME-INT-CLIENT-WIFI {
            name-server 10.5.10.1
            subnet 10.5.22.0/24 {
                default-router 10.5.22.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                range 0 {
                    start 10.5.22.10
                    stop 10.5.22.250
                }
            }
        }
    }
    dns {
        dynamic {
            interface pppoe0 {
                service cloudflare {
                    host-name r2.example.com
                    login <REDACTED>
                    password <REDACTED>
                    protocol cloudflare
                    server www.cloudflare.com
                    zone example.com
                }
            }
        }
        forwarding {
            allow-from 10.4.0.0/16
            allow-from 10.5.0.0/16
            dnssec log-fail
            domain 10.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 20.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 21.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 22.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 30.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain 31.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.12
                server 10.4.30.11
            }
            domain 32.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain ad.example.com {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain int.example.com {
                addnta
                server 10.4.20.12
                server 10.4.20.11
            }
            domain k8s.example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain mgmt.example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain roc.example.com {
                server 10.4.20.11
                server 10.4.20.12
            }
            listen-address 10.5.10.1
            source-address 10.5.10.1
            system
        }
    }
    mdns {
    }
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name r2.example.com
    ipv6 {
        disable
    }
    login {
        user vyos {
            authentication {
                encrypted-password <REDACTED>
            }
        }
    }
    name-server 8.8.8.8
    ntp {
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
zone-policy {
    zone lan {
        default-action drop
        from local {
            firewall {
                name local-lan
            }
        }
        from wan {
            firewall {
                name wan-lan
            }
        }
        interface eth3
        interface eth3.520
        interface eth3.521
        interface eth3.522
        interface eth3.530
        interface eth3.531
        interface eth3.532
        interface eth3.542
        interface wg01
    }
    zone local {
        default-action drop
        from lan {
            firewall {
                name lan-local
            }
        }
        from wan {
            firewall {
                name wan-local
            }
        }
        local-zone
    }
    zone wan {
        default-action drop
        from lan {
            firewall {
                name lan-wan
            }
        }
        from local {
            firewall {
                name local-wan
            }
        }
        interface eth0
        interface eth0.201
        interface pppoe0
        interface eth1
    }
}


// Warning: Do not remove the following line.
// vyos-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack@3:conntrack-sync@2:container@1:dhcp-relay@2:dhcp-server@6:dhcpv6-server@1:dns-forwarding@3:firewall@5:https@2:interfaces@22:ipoe-server@1:ipsec@5:isis@1:l2tp@3:lldp@1:mdns@1:nat@5:ntp@1:pppoe-server@5:pptp@2:qos@1:quagga@8:rpki@1:salt@1:snmp@2:ssh@2:sstp@3:system@21:vrrp@2:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2:zone-policy@1"
// Release version: 1.3.4

Little more digging and while the primary wan (eth1) is connected i can ping the internet, so it seems that it’s just that while primary wan is connected I’m just getting denied access to from lan to local so I can’t ssh or do dns forwarding.

Also saw in my config that the load balancing rules had pppoe as the interface instead of pppoe0 and I updated that with no change in behavior. I’ve updated the above config with the changes.

Your failover configuration is wrong.

As stated in docs https://docs.vyos.io/en/equuleus/configuration/loadbalancing/index.html#failover there must be one primary interface. In your config they are only failover interfaces.

It should looks like:

load-balancing {
    wan {
        enable-local-traffic
        flush-connections
        interface-health eth1 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        interface-health pppoe0 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        rule 10 {
            inbound-interface eth3.520
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 11 {
            failover
            inbound-interface eth3.520
            interface pppoe0 {
                weight 1
            }
            protocol all
        }

       ...

The other issue with the sites taking long to load or failing to load reeks of MTU troubles.

Try manually lowering MTU on the pppoe0 interface or use MSS clamping, for CLI examples see the suggestion by @tjh in VyOS in DNS forwarding mode resolving some hosts but not others - #6 by tjh.

2 Likes

Thank you both!

For now I’ve reverted my secondary modem to nat and removed all pppoe config. That has resolved the slow loading pages when using that connection so it must be the MTU/TCP issues as @marc_s wrote. I’ll try fixing pppoe once everything else is resloved.

I’ve updated my load-balancing rules to reflect the correct configuration as shown by @pepe.

The remaining issue seems to be that client networks can’t traverse the site-to-site wireguard vpn with load-balancing wan enabled. I can ping into the other site from the vyos router but cannot ping from clients on this site. Any idea why that might be? I tried adding load balancing rules for the wireguard interface but that had no effect. Running a traceroute shows that attempting to ping an address in the 10.4.0.0/16 space from a client is routed out the current gateway instead of over the wireguard tunnel. A traceroute on the router correctly follows the static routes configured for wireguard.

updated config.boot

firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name lan-local {
        default-action accept
        enable-default-log
    }
    name lan-wan {
        default-action accept
        enable-default-log
    }
    name local-lan {
        default-action accept
        enable-default-log
    }
    name local-wan {
        default-action accept
        enable-default-log
    }
    name wan-lan {
        default-action drop
        enable-default-log
        rule 5 {
            action accept
            description "Allow EST/Related Traffic"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            protocol icmp
            state {
                new enable
            }
        }
    }
    name wan-local {
        default-action drop
        enable-default-log
        rule 5 {
            action accept
            description "Allow EST/Related Traffic"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            protocol icmp
            state {
                new enable
            }
        }
        rule 30 {
            action accept
            description "Allow Wireguard Traffic"
            destination {
                port 51820
            }
            protocol udp
            source {
            }
            state {
                new enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
    twa-hazards-protection disable
}
interfaces {
    ethernet eth0 {
        address dhcp
        hw-id 00:1a:8c:45:4a:b0
    }
    ethernet eth1 {
        address dhcp
        hw-id 00:1a:8c:45:4a:b1
    }
    ethernet eth2 {
        disable
        hw-id 00:1a:8c:45:4a:b2
    }
    ethernet eth3 {
        address 10.5.10.1/24
        hw-id 00:1a:8c:45:4a:b3
        vif 520 {
            address 10.5.20.1/24
        }
        vif 521 {
            address 10.5.21.1/24
        }
        vif 522 {
            address 10.5.22.1/24
        }
        vif 530 {
            address 10.5.30.1/24
        }
        vif 531 {
            address 10.5.31.1/24
        }
        vif 532 {
            address 10.5.32.1/24
        }
        vif 542 {
            address 10.5.42.1/24
        }
    }
    loopback lo {
    }
    wireguard wg01 {
        address 172.16.10.2/24
        peer to-iphone {
            allowed-ips 172.16.10.101/32
            pubkey <redacted>
        }
        peer to-r1 {
            address <redacted>
            allowed-ips 172.16.10.1/32
            allowed-ips 10.4.0.0/16
            allowed-ips 10.68.0.0/16
            persistent-keepalive 15
            port 51820
            pubkey <redacted>
        }
        port 51820
    }
    wireless wlan0 {
        disable
        hw-id d0:ab:d5:45:43:db
        physical-device phy0
    }
    wireless wlan1 {
        disable
        hw-id 00:0e:8e:69:a1:62
        physical-device phy0
    }
}
load-balancing {
    wan {
        flush-connections
        interface-health eth0 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        interface-health eth1 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        rule 10 {
            inbound-interface eth3
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 11 {
            failover
            inbound-interface eth3
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 20 {
            inbound-interface eth3.520
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 21 {
            failover
            inbound-interface eth3.520
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 30 {
            inbound-interface eth3.521
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 31 {
            failover
            inbound-interface eth3.521
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 40 {
            inbound-interface eth3.522
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 41 {
            failover
            inbound-interface eth3.522
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 50 {
            inbound-interface eth3.530
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 51 {
            failover
            inbound-interface eth3.530
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 60 {
            inbound-interface eth3.531
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 61 {
            failover
            inbound-interface eth3.531
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 70 {
            inbound-interface eth3.532
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 71 {
            failover
            inbound-interface eth3.532
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 80 {
            inbound-interface eth3.542
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 81 {
            failover
            inbound-interface eth3.542
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 90 {
            inbound-interface wg01
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 91 {
            failover
            inbound-interface wg01
            interface eth0 {
                weight 1
            }
            protocol all
        }
    }
}
nat {
    source {
        rule 101 {
            outbound-interface eth1
            source {
                address 10.5.0.0/16
            }
            translation {
                address masquerade
            }
        }
        rule 102 {
            outbound-interface eth0
            source {
                address 10.5.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
protocols {
    static {
        interface-route 172.16.10.1/32 {
            next-hop-interface wg01 {
            }
        }
        route 10.4.0.0/16 {
            next-hop 172.16.10.1 {
            }
        }
    }
}
service {
    dhcp-relay {
        interface eth3.530
        interface eth3.531
        interface eth3.532
        server 10.4.30.15
    }
    dhcp-server {
        shared-network-name HOME-INT {
            subnet 10.5.20.0/24 {
                default-router 10.5.20.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                ping-check
                range 0 {
                    start 10.5.20.210
                    stop 10.5.20.250
                }
            }
        }
        shared-network-name HOME-GUEST-WIFI {
            subnet 10.5.42.0/24 {
                default-router 10.5.42.1
                name-server 10.5.10.1
                range 0 {
                    start 10.5.42.10
                    stop 10.5.42.199
                }
            }
        }
        shared-network-name HOME-INT-CLIENT {
            name-server 10.5.10.1
            subnet 10.5.21.0/24 {
                default-router 10.5.21.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                ping-check
                range 0 {
                    start 10.5.21.10
                    stop 10.5.21.250
                }
            }
        }
        shared-network-name HOME-INT-CLIENT-WIFI {
            name-server 10.5.10.1
            subnet 10.5.22.0/24 {
                default-router 10.5.22.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                range 0 {
                    start 10.5.22.10
                    stop 10.5.22.250
                }
            }
        }
    }
    dns {
        forwarding {
            allow-from 10.4.0.0/16
            allow-from 10.5.0.0/16
            dnssec log-fail
            domain 10.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 20.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 21.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 22.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 30.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain 31.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.12
                server 10.4.30.11
            }
            domain 32.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain ad.example.com {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain int.example.com {
                addnta
                server 10.4.20.12
                server 10.4.20.11
            }
            domain k8s.example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain mgmt.example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain roc.example.com {
                server 10.4.20.11
                server 10.4.20.12
            }
            listen-address 10.5.10.1
            source-address 10.5.10.1
            system
        }
    }
    mdns {
    }
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name r2.example.com
    ipv6 {
        disable
    }
    login {
        user vyos {
            authentication {
                encrypted-password <redacted>
            }
        }
    }
    name-server 8.8.8.8
    ntp {
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
zone-policy {
    zone lan {
        default-action drop
        from local {
            firewall {
                name local-lan
            }
        }
        from wan {
            firewall {
                name wan-lan
            }
        }
        interface eth3
        interface eth3.520
        interface eth3.521
        interface eth3.522
        interface eth3.530
        interface eth3.531
        interface eth3.532
        interface eth3.542
        interface wg01
    }
    zone local {
        default-action drop
        from lan {
            firewall {
                name lan-local
            }
        }
        from wan {
            firewall {
                name wan-local
            }
        }
        local-zone
    }
    zone wan {
        default-action drop
        from lan {
            firewall {
                name lan-wan
            }
        }
        from local {
            firewall {
                name local-wan
            }
        }
        interface eth0
        interface eth1
    }
}


// Warning: Do not remove the following line.
// vyos-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack@3:conntrack-sync@2:container@1:dhcp-relay@2:dhcp-server@6:dhcpv6-server@1:dns-forwarding@3:firewall@5:https@2:interfaces@22:ipoe-server@1:ipsec@5:isis@1:l2tp@3:lldp@1:mdns@1:nat@5:ntp@1:pppoe-server@5:pptp@2:qos@1:quagga@8:rpki@1:salt@1:snmp@2:ssh@2:sstp@3:system@21:vrrp@2:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2:zone-policy@1"
// Release version: 1.3.4

You must exclude wireguard interface form load-balancing, try this:

set load-balancing wan rule 1 inbound-interface wg01
set load-balancing wan rule 1 exclude

1 Like

The issue persists after replacing my wg01 load balance rules with an exclude rule.

vyos@r2.example.com# show load-balancing wan 
 flush-connections
 interface-health eth0 {
     failure-count 1
     nexthop dhcp
     success-count 1
 }
 interface-health eth1 {
     failure-count 1
     nexthop dhcp
     success-count 1
 }
 rule 1 {
     exclude
     inbound-interface wg01
 }
...
vyos@r2.example.com:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [210/0] via 192.168.0.1, eth0, weight 1, 00:22:34
  *                   via 192.168.4.1, eth1, weight 1, 00:22:34
S>  10.4.0.0/16 [1/0] via 172.16.10.1 (recursive), weight 1, 3d19h49m
  *                     via 172.16.10.1, wg01 onlink, weight 1, 3d19h49m
C>* 10.5.10.0/24 is directly connected, eth3, 3d19h49m
C>* 10.5.20.0/24 is directly connected, eth3.520, 3d19h49m
C>* 10.5.21.0/24 is directly connected, eth3.521, 3d19h49m
C>* 10.5.22.0/24 is directly connected, eth3.522, 3d19h49m
C>* 10.5.30.0/24 is directly connected, eth3.530, 3d19h49m
C>* 10.5.31.0/24 is directly connected, eth3.531, 3d19h49m
C>* 10.5.32.0/24 is directly connected, eth3.532, 3d19h49m
C>* 10.5.42.0/24 is directly connected, eth3.542, 3d19h49m
C>* 172.16.10.0/24 is directly connected, wg01, 3d19h49m
S>* 172.16.10.1/32 [1/0] is directly connected, wg01, weight 1, 3d19h49m
C>* 192.168.0.0/24 is directly connected, eth0, 20:26:29
C>* 192.168.4.0/22 is directly connected, eth1, 00:22:34

Is this working if you don’t use load-balancing ?

1 Like

Yes, if I delete the entire load-balancing wan block it works immediately.

current config.boot

firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name lan-local {
        default-action accept
        enable-default-log
    }
    name lan-wan {
        default-action accept
        enable-default-log
    }
    name local-lan {
        default-action accept
        enable-default-log
    }
    name local-wan {
        default-action accept
        enable-default-log
    }
    name wan-lan {
        default-action drop
        enable-default-log
        rule 5 {
            action accept
            description "Allow EST/Related Traffic"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            protocol icmp
            state {
                new enable
            }
        }
    }
    name wan-local {
        default-action drop
        enable-default-log
        rule 5 {
            action accept
            description "Allow EST/Related Traffic"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            protocol icmp
            state {
                new enable
            }
        }
        rule 30 {
            action accept
            description "Allow Wireguard Traffic"
            destination {
                port 51820
            }
            protocol udp
            source {
            }
            state {
                new enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
    twa-hazards-protection disable
}
interfaces {
    ethernet eth0 {
        address dhcp
        hw-id 00:1a:8c:45:4a:b0
    }
    ethernet eth1 {
        address dhcp
        hw-id 00:1a:8c:45:4a:b1
    }
    ethernet eth2 {
        disable
        hw-id 00:1a:8c:45:4a:b2
    }
    ethernet eth3 {
        address 10.5.10.1/24
        hw-id 00:1a:8c:45:4a:b3
        vif 520 {
            address 10.5.20.1/24
        }
        vif 521 {
            address 10.5.21.1/24
        }
        vif 522 {
            address 10.5.22.1/24
        }
        vif 530 {
            address 10.5.30.1/24
        }
        vif 531 {
            address 10.5.31.1/24
        }
        vif 532 {
            address 10.5.32.1/24
        }
        vif 542 {
            address 10.5.42.1/24
        }
    }
    loopback lo {
    }
    wireguard wg01 {
        address 172.16.10.2/24
        peer to-iphone {
            allowed-ips 172.16.10.101/32
            pubkey <REDACTED>
        }
        peer to-r1 {
            address <REDACTED>
            allowed-ips 172.16.10.1/32
            allowed-ips 10.4.0.0/16
            allowed-ips 10.68.0.0/16
            persistent-keepalive 15
            port 51820
            pubkey <REDACTED>
        }
        port 51820
    }
    wireless wlan0 {
        disable
        hw-id d0:ab:d5:45:43:db
        physical-device phy0
    }
    wireless wlan1 {
        disable
        hw-id 00:0e:8e:69:a1:62
        physical-device phy0
    }
}
load-balancing {
    wan {
        flush-connections
        interface-health eth0 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        interface-health eth1 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        rule 1 {
            exclude
            inbound-interface wg01
            protocol all
        }
        rule 10 {
            inbound-interface eth3
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 11 {
            failover
            inbound-interface eth3
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 20 {
            inbound-interface eth3.520
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 21 {
            failover
            inbound-interface eth3.520
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 30 {
            inbound-interface eth3.521
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 31 {
            failover
            inbound-interface eth3.521
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 40 {
            inbound-interface eth3.522
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 41 {
            failover
            inbound-interface eth3.522
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 50 {
            inbound-interface eth3.530
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 51 {
            failover
            inbound-interface eth3.530
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 60 {
            inbound-interface eth3.531
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 61 {
            failover
            inbound-interface eth3.531
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 70 {
            inbound-interface eth3.532
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 71 {
            failover
            inbound-interface eth3.532
            interface eth0 {
                weight 1
            }
            protocol all
        }
        rule 80 {
            inbound-interface eth3.542
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 81 {
            failover
            inbound-interface eth3.542
            interface eth0 {
                weight 1
            }
            protocol all
        }
    }
}
nat {
    source {
        rule 101 {
            outbound-interface eth1
            source {
                address 10.5.0.0/16
            }
            translation {
                address masquerade
            }
        }
        rule 102 {
            outbound-interface eth0
            source {
                address 10.5.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
policy {
}
protocols {
    static {
        interface-route 172.16.10.1/32 {
            next-hop-interface wg01 {
            }
        }
        route 10.4.0.0/16 {
            next-hop 172.16.10.1 {
            }
        }
    }
}
service {
    dhcp-relay {
        interface eth3.530
        interface eth3.531
        interface eth3.532
        server 10.4.30.15
    }
    dhcp-server {
        shared-network-name HOME-INT {
            subnet 10.5.20.0/24 {
                default-router 10.5.20.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                ping-check
                range 0 {
                    start 10.5.20.210
                    stop 10.5.20.250
                }
            }
        }
        shared-network-name HOME-GUEST-WIFI {
            subnet 10.5.42.0/24 {
                default-router 10.5.42.1
                name-server 10.5.10.1
                range 0 {
                    start 10.5.42.10
                    stop 10.5.42.199
                }
            }
        }
        shared-network-name HOME-INT-CLIENT {
            name-server 10.5.10.1
            subnet 10.5.21.0/24 {
                default-router 10.5.21.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                ping-check
                range 0 {
                    start 10.5.21.10
                    stop 10.5.21.250
                }
            }
        }
        shared-network-name HOME-INT-CLIENT-WIFI {
            name-server 10.5.10.1
            subnet 10.5.22.0/24 {
                default-router 10.5.22.1
                domain-name int.example.com
                domain-search int.example.com
                name-server 10.5.10.1
                range 0 {
                    start 10.5.22.10
                    stop 10.5.22.250
                }
            }
        }
    }
    dns {
        forwarding {
            allow-from 10.4.0.0/16
            allow-from 10.5.0.0/16
            dnssec log-fail
            domain 10.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 20.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 21.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 22.4.10.in-addr.arpa. {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain 30.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain 31.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.12
                server 10.4.30.11
            }
            domain 32.4.10.in-addr.arpa. {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain ad.example.com {
                addnta
                server 10.4.30.11
                server 10.4.30.12
            }
            domain example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain int.example.com {
                addnta
                server 10.4.20.12
                server 10.4.20.11
            }
            domain k8s.example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain mgmt.example.com {
                addnta
                server 10.4.20.11
                server 10.4.20.12
            }
            domain roc.example.com {
                server 10.4.20.11
                server 10.4.20.12
            }
            listen-address 10.5.10.1
            source-address 10.5.10.1
            system
        }
    }
    mdns {
    }
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name r2.example.com
    ipv6 {
        disable
    }
    login {
        user vyos {
            authentication {
                encrypted-password <REDACTED>
            }
        }
    }
    name-server 8.8.8.8
    ntp {
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
zone-policy {
    zone lan {
        default-action drop
        from local {
            firewall {
                name local-lan
            }
        }
        from wan {
            firewall {
                name wan-lan
            }
        }
        interface eth3
        interface eth3.520
        interface eth3.521
        interface eth3.522
        interface eth3.530
        interface eth3.531
        interface eth3.532
        interface eth3.542
        interface wg01
    }
    zone local {
        default-action drop
        from lan {
            firewall {
                name lan-local
            }
        }
        from wan {
            firewall {
                name wan-local
            }
        }
        local-zone
    }
    zone wan {
        default-action drop
        from lan {
            firewall {
                name lan-wan
            }
        }
        from local {
            firewall {
                name local-wan
            }
        }
        interface eth0
        interface eth1
    }
}


// Warning: Do not remove the following line.
// vyos-config-version: "broadcast-relay@1:cluster@1:config-management@1:conntrack@3:conntrack-sync@2:container@1:dhcp-relay@2:dhcp-server@6:dhcpv6-server@1:dns-forwarding@3:firewall@5:https@2:interfaces@22:ipoe-server@1:ipsec@5:isis@1:l2tp@3:lldp@1:mdns@1:nat@5:ntp@1:pppoe-server@5:pptp@2:qos@1:quagga@8:rpki@1:salt@1:snmp@2:ssh@2:sstp@3:system@21:vrrp@2:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2:zone-policy@1"
// Release version: 1.3.4

Show output of command show wan-load-balance status

I checked my old config where I used LB. You must exclude destination 10.4.0.0/16 (and probably 172.16.10.0/24) on all input interfaces.

        rule 2 {
            destination {
                address 10.4.0.0/16
            }
            exclude
            inbound-interface eth3
            protocol all
        }
        rule 3 {
            destination {
                address 10.4.0.0/16
            }
            exclude
            inbound-interface eth3.520
            protocol all
        }

Should I then have four rules for each inbound-interface? e.g. -

 rule 30 {
     inbound-interface eth3.521
     interface eth1 {
         weight 9
     }
     protocol all
 }
 rule 31 {
     failover
     inbound-interface eth3.521
     interface eth0 {
         weight 1
     }
     protocol all
 }
 rule 32 {
     destination {
         address 10.4.0.0/16
     }
     exclude
     inbound-interface eth3.521
     protocol all
 }
 rule 33 {
     destination {
         address 172.16.10.0/24
     }
     exclude
     inbound-interface eth3.521
     protocol all
}

I added the above rules to just the one inbound-interface, the client I’m testing with is on that vlan. Still getting the same result, traffic going out the gateway. Below results with added config.

vyos@r2.example.com:~$ show wan-load-balance  status
Chain WANLOADBALANCE_PRE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1   143 ACCEPT     all  --  wg01   *       0.0.0.0/0            0.0.0.0/0           
   17 12858 ISP_eth1   all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth1   all  --  eth3.520 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.520 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.520 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.520 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
  167 18158 ISP_eth1   all  --  eth3.521 *       0.0.0.0/0            0.0.0.0/0            state NEW
 4659  574K CONNMARK   all  --  eth3.521 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.521 *       0.0.0.0/0            0.0.0.0/0            state NEW
 4659  574K CONNMARK   all  --  eth3.521 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ACCEPT     all  --  eth3.521 *       0.0.0.0/0            10.4.0.0/16         
    0     0 ACCEPT     all  --  eth3.521 *       0.0.0.0/0            172.16.10.0/24      
    0     0 ISP_eth1   all  --  eth3.522 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.522 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.522 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.522 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth1   all  --  eth3.530 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.530 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.530 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.530 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth1   all  --  eth3.531 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.531 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.531 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.531 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth1   all  --  eth3.532 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.532 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.532 *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 CONNMARK   all  --  eth3.532 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
   30  2124 ISP_eth1   all  --  eth3.542 *       0.0.0.0/0            0.0.0.0/0            state NEW
   60 56342 CONNMARK   all  --  eth3.542 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3.542 *       0.0.0.0/0            0.0.0.0/0            state NEW
   60 56342 CONNMARK   all  --  eth3.542 *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore

I don’t think I could switch to using a failover route since my primary link (starlink) is behind a another nat router so the link will not go down even if internet is out.

Yes, for each inbound-interface.

All exclude rules must be first, before output interfaces definition.

2 Likes

Moved the exclude rules up font and it’s working now. Thank you for your help!!

Done a bit more tinkering to get the wireguard tunnel itself to make use of the wan failover. Seems that it’s all working fine but I’m lacking some understanding and am not sure if there are some issues with the my config I may not be aware of.

I set enable-local-traffic and checked the peer on the far side and it had switched to the primary wan. Forced a failover by disconnecting the primary and then reconnecting it. The peer on the far side showed the secondary and then back to the primary as the failovers happened. This is my desired behavior but I’m not exactly sure how local traffic is selecting the wan connection to use since there is no rules in the config for local traffic?

I removed the exclude inbound-interface wg01 rule from comment #6 and it had no effect, everything still functioned correctly. I would think that that rule would only effect traffic incoming from the other site over the tunnel? I tried a traceroute from a host on the other site to a client on the local site and it worked just fine without the inclusion.

Additionally I tried using wildcards for the vlan interfaces to reduce the number of rules in the config. Seems to be working fine but it’s not documented as possible so there could be issues there I’m unaware of.

load-balancing {
    wan {
        enable-local-traffic
        flush-connections
        interface-health eth0 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        interface-health eth1 {
            failure-count 1
            nexthop dhcp
            success-count 1
        }
        rule 10 {
            destination {
                address 10.0.0.0/8
            }
            exclude
            inbound-interface eth3+
            protocol all
        }
        rule 11 {
            destination {
                address 172.16.0.0/12
            }
            exclude
            inbound-interface eth3+
            protocol all
        }
        rule 12 {
            destination {
                address 192.168.0.0/16
            }
            exclude
            inbound-interface eth3+
            protocol all
        }
        rule 13 {
            inbound-interface eth3+
            interface eth1 {
                weight 9
            }
            protocol all
        }
        rule 14 {
            failover
            inbound-interface eth3+
            interface eth0 {
                weight 1
            }
            protocol all
        }
    }
}
vyos@r2.example.com# run show wan-load-balance status
Chain WANLOADBALANCE_PRE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    8   565 ACCEPT     all  --  eth3+  *       0.0.0.0/0            10.0.0.0/8          
    0     0 ACCEPT     all  --  eth3+  *       0.0.0.0/0            172.16.0.0/12       
    0     0 ACCEPT     all  --  eth3+  *       0.0.0.0/0            192.168.0.0/16      
   13  3638 ISP_eth1   all  --  eth3+  *       0.0.0.0/0            0.0.0.0/0            state NEW
   50  7755 CONNMARK   all  --  eth3+  *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
    0     0 ISP_eth0   all  --  eth3+  *       0.0.0.0/0            0.0.0.0/0            state NEW
   46  6865 CONNMARK   all  --  eth3+  *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore

The wireguard tunnel eventually switched back to using the secondary without intervention so I’m not sure how local traffic load balancing is being managed . I added another set of the same rules for the inbound-interface lo and it seems to be working so far.

wan-load-balance status is not showing any packets of the lo rules so I assume they’re not doing anything which makes sense as traffic originating locally would not be sent to lo.

Have you seen ⚓ T2747 "enable-local-traffic" has no effect in load-balancing to redirect local traffic?
Long standing issue with traffic originating from the router itself.

Thanks, I had not seen it. Maybe I can keep two tunnels up, one on each wan, then use failover routes for the destination networks?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.