Dual WAN Setup Issues

Hello,

I am planing to change how my connection works and I need some help to figure out what parts of VyOS I need to setup and change.

What I had done:
I started with a single Cable ISP which only provided DHCP IP addresses but they rarely change. So I setup my VyOS as a VM on VMware ESXi 5.5 as follows:
eth0 -> Cable DHCP
eth1 -> Internal LAN

Then later when I needed a second IP so I could provide two HTTPS sites I added the following:
eth2 -> Cable DHCP

Basically I created a second NIC in VMware and have both go to the same virtual and physical switch which links them to the modem. This caused the modem to see two different MACs and so each NIC got a different IP address and they changed less then once a year.
Thing is, in order to make this configuration work I had to setup the load-balancing section of VyOS and used it to make connections sticky.

What I did now:
So now I’ve also bought a Business DSL service which provides me with 14 static IP addresses. So I’ve setup the below as well:
eth3 -> Connection to the Kasda DSL modem/router (in bridged mode)
eth4 -> Internal network to be used for DMZ
pppoe0 -> Gets the gateway IP for the DSL link and is used to point to my /28 IP range.

I currently have the following sections configured:

  • SNAT - for masquerading for the DHCP addresses
  • DNAT - for each port I want filtered from the Cable ISP to different servers
  • Load-Balancing - Setup to balance eth0 and eth2 and allow eth1 through.
  • Firewall - Rules to filter as needed for the outside connections.
  • Traffic-Policy - Used for QoS of my connection ( needed for VoIP purposes)

What I want to switch:

  • I’d like each IP from my 14 DSL IP addresses to have either a 1:1 NAT to a specific server in my LAN (eth1), or be given to the DMZ facing NIC on a server (eth4).
  • I’d like any connection coming into one of the 14 static DSL IPs to come in through pppoe0 and go back out the same way.
  • I’d like any traffic generated from the inside to be sent out eth0 so that any web surfing traffic goes in and out of my Cable DHCP IPs.
  • Using the above I should be able to split my connection so that servers run on DSL and surfing runs on Cable.

Here is my current Firewall config:

firewall { all-ping enable broadcast-ping disable config-trap disable group { network-group DMZ { network 104.234.237.144/28 } network-group LAN { network 10.10.10.0/24 } network-group VPN { network 10.9.10.0/24 network 10.0.1.0/24 network 10.0.2.0/24 network 10.0.3.0/24 } network-group iSCSI { network 10.10.5.0/24 } } ipv6-receive-redirects disable ipv6-src-route disable ip-src-route disable log-martians enable name DMZ-IN { default-action drop enable-default-log rule 1000 { action accept state { established enable related enable } } rule 1001 { action drop state { invalid enable } } rule 9000 { action accept source { group { network-group DMZ } } state { new enable } } } name DMZ-LOCAL { default-action drop rule 10 { action accept state { established enable related enable } } rule 11 { action drop state { invalid enable } } rule 20 { action accept icmp { type-name echo-request } protocol icmp state { new enable } } rule 30 { action accept destination { port 67 } protocol udp state { new enable } } rule 40 { action accept destination { port 53 } protocol tcp_udp state { new enable } } rule 50 { action accept destination { port 123 } protocol udp state { new enable } } } name DMZ-OUT { default-action drop enable-default-log rule 1000 { action accept state { established enable related enable } } rule 1001 { action drop state { invalid enable } } rule 1010 { action accept icmp { type-name echo-request } protocol icmp } rule 2010 { action accept source { group { network-group VPN } } } rule 2020 { action accept source { group { network-group LAN } } } rule 3000 { action accept description "Web Server" destination { address 104.234.237.146 port 22,80,443 } protocol tcp state { new enable } } } name Exchange { default-action drop rule 10 { action accept state { established enable related enable } } rule 20 { action accept destination { port 443 } log enable protocol tcp } rule 21 { action accept destination { port 80 } log enable protocol tcp } rule 22 { action accept destination { port 25 } log enable protocol tcp } rule 23 { action accept destination { port 993 } log enable protocol tcp } rule 24 { action accept destination { port 995 } log enable protocol tcp } rule 25 { action accept destination { port 22 } log enable protocol tcp } } name Inbound { default-action drop rule 9 { action accept log enable protocol gre } rule 10 { action accept state { established enable related enable } } rule 11 { action accept destination { port 80 } log enable protocol tcp } rule 12 { action accept destination { port 22 } log enable protocol tcp } rule 13 { action accept destination { port 443 } log enable protocol tcp } rule 14 { action accept destination { port 500 } log enable protocol udp } rule 15 { action accept destination { port 3389 } log enable protocol tcp } rule 16 { action accept destination { port 5721 } log enable protocol tcp } rule 17 { action accept destination { port 3074 } log enable protocol tcp } rule 18 { action accept destination { port 5070 } log enable protocol udp } rule 19 { action accept destination { port 5060 } log enable protocol udp } rule 20 { action accept destination { port 10000-20000 } log enable protocol udp } rule 21 { action accept log enable protocol esp } rule 22 { action accept destination { port 1701 } ipsec { match-ipsec } protocol udp } rule 23 { action accept destination { port 161 } log enable protocol udp } rule 24 { action accept destination { port 4500 } protocol udp } rule 25 { action accept icmp { type 8 } log enable protocol icmp } rule 26 { action accept icmp { type 0 } log enable protocol icmp } rule 27 { action accept destination { port 587 } log enable protocol tcp } rule 28 { action accept destination { port 465 } log enable protocol tcp } rule 29 { action accept icmp { type 3 } log enable protocol icmp } rule 31 { action accept icmp { type 11 } log enable protocol icmp } rule 32 { action accept destination { port 1194 } log enable protocol udp } rule 33 { action accept destination { port 8080 } log enable protocol tcp } rule 34 { action accept destination { port 6001 } log enable protocol tcp } rule 50 { action accept destination { address 10.10.10.0/24 } log enable } rule 51 { action accept destination { address 10.10.5.0/24 } log enable } } name LAN-IN { default-action drop enable-default-log rule 1000 { action accept state { established enable related enable } } rule 1001 { action drop state { invalid enable } } rule 9000 { action accept source { group { network-group LAN } } state { new enable } } } name LAN-OUT { default-action drop enable-default-log rule 1000 { action accept state { established enable related enable } } rule 1001 { action drop state { invalid enable } } rule 1010 { action accept icmp { type-name echo-request } protocol icmp } rule 2000 { action accept source { group { network-group VPN } } state { new enable } } rule 2020 { action accept source { group { network-group DMZ } } } } name Outbound { default-action drop rule 10 { action accept log enable state { new enable } } } name Router { default-action drop rule 10 { action accept state { established enable related enable } } rule 20 { action accept destination { port 1723 } protocol tcp } rule 21 { action accept destination { port 500 } log enable protocol udp } rule 22 { action accept destination { port 1701 } ipsec { match-ipsec } log enable protocol udp } rule 23 { action accept destination { port 4500 } log enable protocol udp } rule 24 { action accept log enable protocol esp } rule 25 { action accept destination { port 1194 } log enable protocol udp } } receive-redirects disable send-redirects enable source-validation disable state-policy { established { action accept } related { action accept } } syn-cookies enable twa-hazards-protection disable }

Here’s my current Interface config:

interfaces { ethernet eth0 { address dhcp description Outside duplex auto firewall { in { name Inbound } local { name Router } } hw-id 00:50:56:a1:20:08 smp_affinity auto speed auto traffic-policy { out Outbound } } ethernet eth1 { address 10.10.10.2/24 description Inside duplex auto hw-id 00:50:56:a1:4e:7d smp_affinity auto speed auto } ethernet eth2 { address dhcp description Exchange duplex auto firewall { in { name Exchange } } hw-id 00:50:56:a1:2c:57 smp_affinity auto speed auto traffic-policy { out Outbound } } ethernet eth3 { description "DSL Modem" duplex auto hw-id 00:50:56:a0:64:dc pppoe 0 { default-route auto mtu 1492 name-server auto password <cleaned> user-id <cleaned> } smp_affinity auto speed auto } ethernet eth4 { description DMZ duplex auto hw-id 00:50:56:a0:1b:fe smp_affinity auto speed auto } loopback lo { } openvpn vtun0 { description "OpenVPN Server Tunnel" encryption aes256 hash sha512 local-host 135.0.94.79 local-port 1194 mode server protocol udp server { client site3 { ip 10.9.10.4 push-route 10.0.1.0/24 push-route 10.0.2.0/24 subnet 10.0.3.0/24 } client site2 { ip 10.9.10.3 push-route 10.0.1.0/24 push-route 10.0.3.0/24 subnet 10.0.2.0/24 } client remote1 { ip 10.9.10.5 push-route 10.0.1.0/24 push-route 10.0.2.0/24 push-route 10.0.3.0/24 } client site1 { ip 10.9.10.2 push-route 10.0.2.0/24 push-route 10.0.3.0/24 subnet 10.0.1.0/24 } domain-name hecint.com name-server 10.10.10.12 name-server 10.10.10.13 push-route 10.10.10.0/24 push-route 198.251.51.0/24 push-route 74.125.69.188/32 push-route 173.194.46.98/32 push-route 216.58.216.78/32 push-route 115.29.160.153/32 subnet 10.9.10.0/24 } tls { ca-cert-file /config/auth/keys/ca.crt cert-file /config/auth/keys/vyatta.crt dh-file /config/auth/keys/dh1024.pem key-file /config/auth/keys/vyatta.key } } }

Here’s my Load Balancer config:

load-balancing { wan { interface-health eth0 { failure-count 5 nexthop dhcp success-count 1 test 10 { resp-time 5 target 135.0.199.31 ttl-limit 1 type ping } test 20 { resp-time 5 target 135.0.199.49 ttl-limit 1 type ping } } interface-health eth2 { failure-count 5 nexthop dhcp success-count 1 test 10 { resp-time 5 target 135.0.199.31 ttl-limit 1 type ping } test 20 { resp-time 5 target 135.0.199.49 ttl-limit 1 type ping } } rule 1 { destination { address 10.9.10.0/24 } exclude inbound-interface eth1 protocol all } rule 2 { destination { address 10.0.1.0/24 } exclude inbound-interface eth1 protocol all } rule 3 { destination { address 10.10.5.0/24 } exclude inbound-interface eth1 protocol all } rule 4 { destination { address 10.0.2.0/24 } exclude inbound-interface eth1 protocol all } rule 5 { destination { address 10.0.3.0/24 } exclude inbound-interface eth1 protocol all } rule 6 { destination { address 104.234.237.144/28 } exclude inbound-interface eth4 protocol all } rule 100 { failover inbound-interface eth1 interface eth0 { weight 255 } interface eth2 { weight 1 } protocol all } sticky-connections { inbound } } }

Here’s my DNAT config:

nat {
    destination {
        rule 20 {
            destination {
                port 80
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.60
                port 80
            }
        }
        rule 21 {
            destination {
                port 22
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.60
                port 22
            }
        }
        rule 22 {
            destination {
                port 443
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.60
                port 443
            }
        }
        rule 23 {
            destination {
                port 80
            }
            inbound-interface eth2
            protocol tcp
            translation {
                address 10.10.10.29
                port 80
            }
        }
        rule 24 {
            destination {
                port 3389
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.64
                port 3389
            }
        }
        rule 25 {
            destination {
                port 5721
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.67
                port 5721
            }
        }
        rule 26 {
            destination {
                port 3074
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.14
                port 3074
            }
        }
        rule 27 {
            destination {
                port 5070
            }
            inbound-interface eth0
            protocol udp
            translation {
                address 10.10.10.10
                port 5070
            }
        }
        rule 28 {
            destination {
                port 5060
            }
            inbound-interface eth0
            protocol udp
            translation {
                address 10.10.10.10
                port 5060
            }
        }
        rule 29 {
            destination {
                port 10000-20000
            }
            inbound-interface eth0
            protocol udp
            translation {
                address 10.10.10.10
                port 10000-20000
            }
        }
        rule 30 {
            destination {
                port 25
            }
            inbound-interface eth2
            protocol tcp
            translation {
                address 10.10.10.29
                port 25
            }
        }
        rule 31 {
            destination {
                port 22
            }
            inbound-interface eth2
            protocol tcp
            translation {
                address 10.10.10.64
                port 3389
            }
        }
        rule 32 {
            destination {
                port 443
            }
            inbound-interface eth2
            protocol tcp
            translation {
                address 10.10.10.29
                port 443
            }
        }
        rule 41 {
            destination {
                port 993
            }
            inbound-interface eth2
            protocol tcp
            translation {
                address 10.10.10.29
                port 993
            }
        }
        rule 42 {
            destination {
                port 995
            }
            inbound-interface eth2
            protocol tcp
            translation {
                address 10.10.10.29
                port 995
            }
        }
        rule 43 {
            destination {
                port 587
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.15
                port 587
            }
        }
        rule 44 {
            destination {
                port 465
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.15
                port 465
            }
        }
        rule 46 {
            destination {
                port 8080
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.67
                port 8080
            }
        }
        rule 47 {
            destination {
                port 6001
            }
            inbound-interface eth0
            protocol tcp
            translation {
                address 10.10.10.15
                port 6001
            }
        }
        rule 100 {
            destination {
                address 104.234.237.146
            }
            inbound-interface pppoe0
            protocol tcp
            translation {
                address 10.10.10.66
            }
        }
    }

Here’s my SNAT config:

    source {
        rule 100 {
            outbound-interface eth0
            source {
                address 10.10.10.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 101 {
            outbound-interface eth0
            source {
                address 10.10.5.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 200 {
            outbound-interface eth2
            source {
                address 10.10.10.0/24
            }
            translation {
                address masquerade
            }
        }
        rule 201 {
            outbound-interface eth2
            source {
                address 10.10.5.0/24
            }
            translation {
                address masquerade
            }
        }
    }
}

Here’s my Protocol’s config:

protocols {
    static {
        route 10.0.1.0/24 {
            next-hop 10.9.10.2 {
            }
        }
        route 10.0.2.0/24 {
            next-hop 10.9.10.3 {
            }
        }
        route 10.0.3.0/24 {
            next-hop 10.9.10.4 {
            }
        }
        route 10.10.5.0/24 {
            next-hop 10.10.10.25 {
            }
        }
        route 192.168.100.0/24 {
            next-hop 135.0.94.79 {
            }
        }
    }
}

Here’s my Traffic Policy config:

traffic-policy { shaper Outbound { bandwidth 10mbps class 10 { bandwidth 20% burst 15k ceiling 100% description "VoIP-RTP Traffic" match VOIP-RTP { ip { dscp 46 } } queue-type fair-queue } class 20 { bandwidth 5% burst 15k ceiling 100% description "VoIP-SIP Traffic" match VOIP-SIP { ip { dscp 26 } } queue-type fair-queue } default { bandwidth 75% burst 15k ceiling 100% queue-type fair-queue } description "QoS Policy for Home, Outbound" } }

Anyone know what I would need to change?

As an update, it seems to be working fine now, my big issue was I noticed in my routing able that 0.0.0.0/0 only was being sent out via the Cable ISP on eth0/eth2.

So I checked over my PPPoE configuration and found the ‘default-route’ was set to ‘auto’. I tested setting it to ‘force’ instead of ‘auto’ and this made my PPPoE connection force getting a gateway for itself and force adding that route to my routing table.

For anyone who wants to check their routing table, it’s ‘show ip route’.

Now it is working.

Funny story, as I get this working my Cable ISP does something messed up to my account and breaks my link, but I didn’t notice until I had to re-route traffic through my new DSL ISP.

not to kick up an old thread but i don’t see how you made the pppoe connection work successfully in a loadbalancing scenario which a dhcp address from your cable dsl connections.
the reason i ask is we face a similar setup but the ip table routes don’t work with both interfaces receiving their ip’s over dhcp.

It used policy based routing in the end.

can you post examples of this config? we’ve never managed to get this working.

Here’s my example I just pulled and sanitized from my current setup doing this:

Basically, I have two connections:
Cable for my wired and wireless clients.
DSL for the servers.

I created a subnet in the 10.10.x.x range for my WiFi, another for my wired clients, and another for my servers.

My eth0 is my LAN side NIC which leads to all clients, WiFi and servers.
My eth1 goes to my DMZ.
My eth2 goes to my DSL modem. Doing it this way allows my VyOS edge router to do the connecting and that’s required for it to be able to handle and route the multiple public IPs I pay for.
My eth3 goes to my Cable modem and gets and public DHCP for clients and WiFi.

I set the Cable modem as my default route since it takes a gateway anyways when I connects and gets the public IP from the DHCP my cable provider uses. So the entire router’s default gateway becomes the Cable internet.

If you look under my eth2, you’ll see the pppoe0 setup there including the line ‘default-route none’ which say to not take a gateway from the DSL link.

You should then notice under my eth0 (LAN side NIC) that I have the link ‘policy {route Servers_Out }’ which tells the VyOS router to apply the policies in the brackets to everything coming into this link. So anything coming outbound. Specifically it applies the route policy called ‘Servers_Out’.

If you then scroll down to the Policy area, you’ll see one called ‘Servers_Out’ which applies a rule to data from the source address range of any ‘0.0.0.0/0’ and destination address range of ‘10.0.0.0/26’. So here you see how it applies something different to your data based on the policy of where it comes from or is going to. Applying your own rules based on which subnet and to where is the trick.

Then further down under ‘Protocols’ you’ll see the ‘Static’ protocols which include the two tables I use and table 1 routes any IP to my gateway from the DSL link. This causes it to now route out the pppoe0 link and then funnel through the NAT to choose which address in my range it’ll be labeled as.

i see you moved away from the loadbalancing you initially used. did that cause issues?

tables 1 and 3. where did you get the next-hop from? i’m assuming if both the cable and pppoe connections are assigned via dhcp by the isp, these values would change dynamically as well.
do you know if interface routes would work this way with the default route of pppoe being set as none?

Load Balancing turned out to be for a different purpose. If you want two connections to both be used by your network at the same time in the same way then you’ll want to Load Balance.

I ended up having one for clients and one for servers, so it made sense to direct the different subnets out different links for me.

I got the next-hop addresses from the gateway address of the WAN interfaces. Cable is DHCP but PPPoE is static actually, so I know ahead of time the IP on the other side of my link to the PPPoE provider and that’s the same as a gateway for this purpose. It’s an IP I can route my traffic two which both will handle the traffic and allow my routing table to change the outbound interface based on using that IP here.

If both of your connections are DHCP then I would use an Interface Route instead. I’m not sure if that’ll work the same but you can try it.

The reason the default route for my PPPoE is set to none is because it would other add it to the default routes of my VyOS router, so it would have a default route going out two different WAN links and that would just start a scenario where it round-robins the WAN links equally based on how Debian handles two default gateways. Having more then one defeats the purpose of it being ‘default’. You should never have two, so one is assigned into there by the DHCP of the cable link and the other is routed to by these rules.

This also means that everything is assumed to go out my Cable link unless I tell the router to send it out the business DSL link, so anything new comes onto the network, it would have to figure out an IP on my server subnet (no DHCP there) and only then it could get out my business DSL link, if it guesses the right range to use and gateway.

P.S. my gateway to get out is not x.x.x.1 like most (i.e. 10.0.0.1), as this makes it harder for others to guess.