Unsure how to achieve dual wan setup


#1

I have an original post listed here and was given a solution: “you must setup a routing policy to be ensure that all traffic for same connection will be forwarded through one external interface”. I tried setting this up and was not successful. Should I be using route-interface or just plain route? For start, I would only like VLAN 7 to run traffic through eth3 and everything else route through eth0. I initially added

set protocols static interface-route 192.168.7.0/24 eth3
set protocols static interface-route 192.168.10.0/24 eth0
set protocols static interface-route 192.168.20.0/24 eth0
set protocols static interface-route 192.168.30.0/24 eth0
set protocols static interface-route 192.168.40.0/24 eth0
set protocols static interface-route 192.168.50.0/24 eth0
set protocols static interface-route 192.168.60.0/24 eth0
set protocols static interface-route 192.168.70.0/24 eth0
set protocols static interface-route 192.168.80.0/24 eth0
set protocols static interface-route 192.168.110.0/24 eth0 

and it still gave issues while both eth0 and eth3 (the WAN’s) were connected. Thanks for any help or information.


#2

Hi, @joshsharrer!
Routing policy must be configured in set policy route section. After you configure policies, you must apply them to Ethernet interfaces.


#3
set policy route ISP1 rule 100 protocol all
set policy route ISP1 rule 100 set table 100
set protocols static table 100 description ISP1
set protocols static table 100 route 0.0.0.0/0 next-hop <IP address of eth3>
set interfaces ethernet eth1 vif 7 policy route ISP1


set policy route ISP2 rule 200 description ISP2
set policy route ISP2 rule 200 protocol all
set policy route ISP2 rule 200 set table 200
set protocols static table 200 description ISP2
set protocols static table 200 route 0.0.0.0/0 next-hop <IP address of eth0>
set interfaces ethernet eth1 vif 10 policy route ISP2
set interfaces ethernet eth1 vif 20 policy route ISP2
set interfaces ethernet eth1 vif 30 policy route ISP2
set interfaces ethernet eth1 vif 40 policy route ISP2
set interfaces ethernet eth1 vif 50 policy route ISP2
set interfaces ethernet eth1 vif 60 policy route ISP2
set interfaces ethernet eth1 vif 70 policy route ISP2
set interfaces ethernet eth1 vif 80 policy route ISP2
set interfaces ethernet eth1 vif 110 policy route ISP2
set interfaces ethernet eth2 policy route ISP2

I have tried the above with no success. Any idea what Im doing wrong? Im still getting packet loss and massive TCP re-transmissions with that addition. @zsdc
Thanks.


#4

@hagbard Any suggestions?


#5

Hi, @joshsharrer!
Show, please, full current config and output of next commands:

show ip route
show ip route table 100
show ip route table 200
ip rule show

#6
josh@ROUTER:~$ sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S>* 0.0.0.0/0 [210/0] via 158.x40.x2.1, eth0
    *                   via 74.X2.x56.1, eth3
S>* 10.1.1.0/24 [1/0] via 192.168.90.2 (recursive via 158.x40.x2.1)
C>* 74.82.156.0/23 is directly connected, eth3
C>* 127.0.0.0/8 is directly connected, lo
C>* 158.140.32.0/21 is directly connected, eth0
C>* 192.168.7.0/24 is directly connected, eth1.7
C>* 192.168.10.0/24 is directly connected, eth1.10
C>* 192.168.20.0/24 is directly connected, eth1.20
C>* 192.168.30.0/24 is directly connected, eth1.30
C>* 192.168.40.0/24 is directly connected, eth1.40
C>* 192.168.50.0/24 is directly connected, eth1.50
C>* 192.168.60.0/24 is directly connected, eth1.60
C>* 192.168.70.0/24 is directly connected, eth1.70
C>* 192.168.80.0/24 is directly connected, eth1.80
C>* 192.168.110.0/24 is directly connected, eth1.110
josh@ROUTER:~$ sh ip route table 100
table 100:

Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S>* 192.168.7.0/24 [1/0] via 74.x2.x57.197, eth3
josh@ROUTER:~$ sh ip route table 200
table 200:

Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S>* 192.168.10.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.20.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.30.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.40.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.50.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.60.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.70.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.80.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.90.0/24 [1/0] via 158.x40.x4.8, eth0
S>* 192.168.110.0/24 [1/0] via 158.x40.x4.8, eth0
josh@ROUTER:~$ ip rule show
0:      from all lookup local
100:    from all fwmark 0x80000063 lookup 100
200:    from all fwmark 0x800000c7 lookup 200
220:    from all lookup 220
32766:  from all lookup main
32767:  from all lookup default

#7

Thanks, @joshsharrer.
Routes in your tables is completely wrong. It looks like typos in configuration or bug in software. Without configuration we cant find out what it is exactly. You can sent it to me (without login section, of course) with private message, if you don’t want to show it to all.


#8
josh@ROUTER:~$ sh conf
firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name OUTSIDE-IN {
        default-action drop
        rule 10 {
            action accept
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            description OpenVPN
            destination {
                address 192.168.10.13
                port 1194
            }
            protocol udp
            state {
                established enable
                new enable
                related enable
            }
        }
        rule 30 {
            action accept
            description Plex
            destination {
                address 192.168.10.16
                port 32400
            }
            protocol tcp_udp
            state {
                established enable
                new enable
                related enable
            }
        }
        rule 40 {
            action accept
            description "Ethan Plex"
            destination {
                address 192.168.10.4
                port 32400
            }
            protocol tcp_udp
            state {
                established enable
                new enable
                related enable
            }
        }
    }
    name OUTSIDE-LOCAL {
        default-action drop
        rule 10 {
            action accept
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            icmp {
                type-name echo-request
            }
            protocol icmp
            state {
                new enable
            }
        }
        rule 30 {
            action drop
            destination {
                port 22
            }
            log enable
            protocol tcp
            state {
                new enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
    twa-hazards-protection disable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description ValleyWAN
        duplex auto
        firewall {
            in {
                name OUTSIDE-IN
            }
            local {
                name OUTSIDE-LOCAL
            }
        }
        hw-id d4:ae:52:a6:26:75
        smp-affinity auto
        speed auto
    }
    ethernet eth1 {
        duplex auto
        hw-id d4:ae:52:a6:26:76
        smp-affinity auto
        speed auto
        vif 7 {
            address 192.168.7.1/24
            description TEST
            policy {
                route ISP1
            }
        }
        vif 10 {
            address 192.168.10.1/24
            description Servers
            policy {
                route ISP2
            }
        }
        vif 20 {
            address 192.168.20.1/24
            description Desktops
            policy {
                route ISP2
            }
        }
        vif 30 {
            address 192.168.30.1/24
            description WiFi
            policy {
                route ISP2
            }
        }
        vif 40 {
            address 192.168.40.1/24
            description "Smart Home"
            policy {
                route ISP2
            }
        }
        vif 50 {
            address 192.168.50.1/24
            description Management
            policy {
                route ISP2
            }
        }
        vif 60 {
            address 192.168.60.1/24
            description Surveillance
            policy {
                route ISP2
            }
        }
        vif 70 {
            address 192.168.70.1/24
            description "Guest WiFi"
            policy {
                route ISP2
            }
        }
        vif 80 {
            address 192.168.80.1/24
            description HTB
            policy {
                route ISP2
            }
        }
        vif 110 {
            address 192.168.110.1/24
            description Smoqueed
            policy {
                route ISP2
            }
        }
    }
    ethernet eth2 {
        address 192.168.90.1/30
        description "Lab Environment"
        duplex auto
        hw-id d4:ae:52:a6:26:77
        policy {
            route ISP2
        }
        smp-affinity auto
        speed auto
    }
    ethernet eth3 {
        address dhcp
        description RiseWAN
        disable
        duplex auto
        firewall {
            in {
                name OUTSIDE-IN
            }
            local {
                name OUTSIDE-LOCAL
            }
        }
        hw-id d4:ae:52:a6:26:78
        smp-affinity auto
        speed auto
    }
    loopback lo {
        description Loopback
    }
}
nat {
    destination {
        rule 20 {
            description OpenVPN
            destination {
                address <ISP2_IP>
                port 1194
            }
            inbound-interface eth0
            protocol udp
            translation {
                address 192.168.10.13
                port 1194
            }
        }
        rule 30 {
            description Plex
            destination {
                address <ISP2_IP>
                port 32400
            }
            inbound-interface eth0
            protocol tcp_udp
            translation {
                address 192.168.10.16
                port 32400
            }
        }
        rule 40 {
            description "Ethan Plex"
            destination {
                address <ISP2_IP>
                port 32500
            }
            inbound-interface eth0
            protocol tcp_udp
            translation {
                address 192.168.10.4
                port 32400
            }
        }
    }
    source {
        rule 7 {
            outbound-interface eth3
            source {
                address 192.168.7.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 10 {
            outbound-interface eth0
            source {
                address 192.168.10.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 20 {
            outbound-interface eth0
            source {
                address 192.168.20.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 30 {
            outbound-interface eth0
            source {
                address 192.168.30.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 40 {
            outbound-interface eth0
            source {
                address 192.168.40.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 50 {
            outbound-interface eth0
            source {
                address 192.168.50.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 60 {
            outbound-interface eth0
            source {
                address 192.168.60.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 70 {
            outbound-interface eth0
            source {
                address 192.168.70.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 80 {
            outbound-interface eth0
            source {
                address 192.168.80.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 110 {
            outbound-interface eth0
            source {
                address 192.168.110.0/24
            }
            translation {
                address masquerade
            }
        }
    }
}
policy {
    route ISP1 {
        rule 100 {
            destination {
                address <ISP1_IP>
            }
            protocol all
            set {
                table 100
            }
        }
    }
    route ISP2 {
        rule 200 {
            destination {
                address <ISP2_IP>
            }
            protocol all
            set {
                table 200
            }
        }
    }
}
protocols {
    static {
        route 10.1.1.0/24 {
            next-hop 192.168.90.2 {
            }
        }
        table 100 {
            route 192.168.7.0/24 {
                next-hop <ISP1_IP> {
                }
            }
        }
        table 200 {
            route 192.168.10.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.20.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.30.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.40.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.50.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.60.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.70.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.80.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.90.0/24 {
                next-hop <ISP2_IP> {
                }
            }
            route 192.168.110.0/24 {
                next-hop <ISP2_IP> {
                }
            }
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name Desktops {
            authoritative disable
            subnet 192.168.20.0/24 {
                default-router 192.168.20.1
                dns-server 192.168.10.7
                dns-server 192.168.10.14
                dns-server 1.1.1.1
                domain-name sharrer.us
                lease 86400
                start 192.168.20.11 {
                    stop 192.168.20.254
                }
            }
        }
        shared-network-name Guest_WiFi {
            authoritative disable
            subnet 192.168.70.0/24 {
                default-router 192.168.70.1
                dns-server 192.168.10.14
                domain-name sharrer.us
                lease 86400
                start 192.168.70.10 {
                    stop 192.168.70.254
                }
            }
        }
        shared-network-name HTB {
            authoritative disable
            subnet 192.168.80.0/24 {
                default-router 192.168.80.1
                dns-server 192.168.10.14
                domain-name sharrer.us
                lease 86400
                start 192.168.80.10 {
                    stop 192.168.80.20
                }
            }
        }
        shared-network-name Smart_Home {
            authoritative disable
            subnet 192.168.40.0/24 {
                default-router 192.168.40.1
                dns-server 192.168.10.14
                domain-name sharrer.us
                lease 86400
                start 192.168.40.10 {
                    stop 192.168.40.254
                }
            }
        }
        shared-network-name Surveillance {
            authoritative disable
            subnet 192.168.60.0/24 {
                default-router 192.168.60.1
                dns-server 192.168.10.14
                domain-name sharrer.us
                lease 86400
                start 192.168.60.10 {
                    stop 192.168.60.254
                }
            }
        }
        shared-network-name WiFi {
            authoritative disable
            subnet 192.168.30.0/24 {
                default-router 192.168.30.1
                dns-server 192.168.10.14
                domain-name sharrer.us
                lease 86400
                start 192.168.30.5 {
                    stop 192.168.30.254
                }
            }
        }
    }
    dns {
        forwarding {
            cache-size 0
            listen-on eth1.10
            listen-on eth1.20
            listen-on eth1.30
            listen-on eth1.40
            listen-on eth1.50
            listen-on eth1.60
            listen-on eth1.70
            listen-on eth1.80
            listen-on eth1.110
            listen-on eth2
            name-server 192.168.10.14
        }
    }
    lldp {
        interface eth1 {
        }
        interface eth1.10 {
        }
        interface eth1.20 {
        }
        interface eth2 {
        }
    }
    mdns {
        repeater {
            interface eth1.20
            interface eth1.30
            interface eth1.40
        }
    }
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    domain-name sharrer.us
    host-name ROUTER
    ip {
        arp {
            table-size 8192
        }
    }
    ipv6 {
        disable
    }
    login {
        user josh {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
            }
            level admin
        }
    }
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone US/Pacific
}

#9

Hi, @joshsharrer! Thank you for config.

  1. Clear tables 100 and 200 and add to each of them only 0.0.0.0/0 route with ISP gateways:
set protocols static table 100 description ISP1
set protocols static table 100 route 0.0.0.0/0 next-hop dhcp-interface eth3
set protocols static table 200 description ISP2
set protocols static table 200 route 0.0.0.0/0 next-hop dhcp-interface eth0
  1. Remove destination parameter from policy route ISPx rules.

Check again after this.


#10

It seems to be working with those changes with one exception. Now my internal traffic destined for other subnets is trying to access them via the WAN links.