MultiWAN setup - only on one wan interface router is reachable from ouside

General questions
1

hello there!

i just migrated from vyatta to vyos this weekend, main driver was support for policy based routing. my vyos box is connected to three ISP (ADSL pppoe, CABLE, FIBER). issue i had with vyatta for years is router is accessible from outside (internet) only on one of these three interfaces.
i figure out it is because default route (there can be only one) due to which router will always try to send traffic to outside via one wan interface.
policy based routing was supposed to fix scenarios as this one, but i was not successful last two days. :frowning:

these are my interfaces:

ethernet eth0 { description "WAN with ADSL" duplex auto hw-id 00:0c:29:39:c7:44 pppoe 5 { default-route none mtu 1492 name-server auto password xxxxxxxxxx73 policy { route ADSL-IN } user-id xxxxxxxxxxxxx3@xxxxxxxxxxx } smp_affinity auto speed auto } ethernet eth1 { address 192.168.1.9/24 description LAN duplex auto hw-id 00:0c:29:39:c7:58 smp_affinity auto speed auto } ... ... ... ethernet eth5 { description FIBER-Internet duplex auto hw-id 00:0c:29:39:c7:76 smp_affinity auto speed auto vif 315 { address FF.FFF.FFF.FFF/30 policy { route FIBER-IN } } } ... ethernet eth7 { address dhcp description CABLE disable duplex auto hw-id 00:0c:29:39:c7:8a mac 00:1d:7e:4b:5d:99 policy { route CABLE-IN } smp_affinity auto speed auto } loopback lo { }

these are my policies:

 route ADSL-IN {
     rule 10 {
         destination {
             address AAA.AA.AA.AAA
         }
         protocol all
         set {
             table 1
         }
         source {
             address 0.0.0.0/0
         }
         state {
             established enable
             new enable
             related enable
         }
     }
 }
 route CABLE-IN {
     enable-default-log
     rule 10 {
         destination {
             address CC.CCC.CCC.CCC
         }
         log enable
         protocol all
         set {
             table 3
         }
         source {
             address 0.0.0.0/0
         }
         state {
             established enable
             new enable
             related enable
         }
     }
 }
 route FIBER-IN {
     rule 10 {
         set {
             table 2
         }
     }
 }

and my protocols:

 static {
     route 192.168.3.0/24 {
         next-hop 192.168.33.4 {
             distance 50
         }
     }
     table 1 {
         interface-route 0.0.0.0/0 {
             next-hop-interface pppoe5 {
                 distance 200
             }
         }
     }
     table 2 {
         route 0.0.0.0/0 {
             next-hop FG.FGG.FGG.FGG {
                 distance 50
             }
         }
     }
     table 3 {
         route 0.0.0.0/0 {
             next-hop CG.CGG.CGG.CGG {
                 distance 50
             }
         }
     }
 }

run show ip route (and tables)

vyos@vyos-yang# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

C>* 127.0.0.0/8 is directly connected, lo
C>* 192.168.1.0/24 is directly connected, eth1
C>* 192.168.2.0/24 is directly connected, eth2
S>* 192.168.3.0/24 [50/0] via 192.168.33.4, eth6.316
C>* 192.168.33.0/24 is directly connected, eth6.316
C>* 192.168.55.0/24 is directly connected, eth3
C>* AGG.AGG.AGG.AG/32 is directly connected, pppoe5
[edit]
vyos@vyos-yang# run show ip route table 1
table 1:

Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S>* 0.0.0.0/0 [200/0] is directly connected, pppoe5
[edit]
vyos@vyos-yang# run show ip route table 2
table 2:

Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S   0.0.0.0/0 [50/0] via FG.FGG.FGG.FGG inactive
[edit]
vyos@vyos-yang# run show ip route table 3
table 3:

Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

S   0.0.0.0/0 [50/0] via CG.CGG.CGG.CGG inactive
[edit]

legend:
AAA.AA.AA.AAA - ADSL static address (pppoe)
CC.CCC.CCC.CCC - CABLE static address, assigned via dhcp
FF.FFF.FFF.FFF/30 - FIBER static address, on vlan 315
AGG.AGG.AGG.AG - ADSL gateway
CG.CGG.CGG.CGG - CABLE gateway
FG.FGG.FGG.FGG - FIBER gateway

i disabled eth7 (CABLE) connection to force route via ADSL in my troubleshooting effort but i was not successful. In this setup when eth7 is enabled default route appear and router is reachable from the outside of course only on CABLE interface.

examples i have found are for edgemax and vyatta mostly for outgoing traffic. my needs are completely opposite - i just want possibility to reach router from internet via any of ISP interfaces.

2

Hi,
my vyatta/vyos configuration is a bit different than yours, but it is working very good in multiwan configuration.
First difference is that I am using wan load balancing. I had also issues with accessing vyatta from outside using two wan links at the same time. I found a solution on non-existent nowadays vyatta community forums.
It is similar to this wiki article:
http://vyos.net/wiki/How_to_make_inbound_WAN_connections_sticky_to_the_interface

To the point:
I had to use a script (code below), create a file in:
/config/scripts/nameofthecript
give it chmod 755 permissions and run it, then restart a vyos machine.

I had to disable source nat in load-balancing wan and create my own source nat rules instead.
I’ve been using it in production on vyatta 6.5R1 and vyos 1.0.4.
Hope this helps you a bit.
Regards

#!/bin/bash
# WAN Load-Balancing Symmetry workaround
# steven.kath@vyatta.com 2012-09-22
# See https://bugzilla.vyatta.com/show_bug.cgi?id=6245 for background.

if [[ $UID != 0 ]]; then
  echo -e "This script must be run with root permissions. Try:\n  sudo $0"
  exit
fi

# Get list of configured WAN-LB interfaces
cli-shell-api inSession && ACTION=listNodes || ACTION=listEffectiveNodes
WAN_INTERFACES=$(cli-shell-api $ACTION load-balancing wan interface-health)

# Clean out our rules from previous runs, if any.
for CHAIN in PREROUTING OUTPUT; do
  RULES=$(iptables -t mangle -nL $CHAIN --line-numbers)
  RULES=$(grep WLB_SYMMETRY <<< "$RULES" | cut -d' ' -f 1)
  for RULE in $(sort -r <<< "$RULES"); do
    echo "Deleting old rule $RULE from chain $CHAIN..." 
    iptables -t mangle -D $CHAIN $RULE
  done
done

if [[ -z "$WAN_INTERFACES" ]]; then
  echo "No configured WAN Load-Balancing interfaces found."
  exit
fi

until iptables -t mangle -nL WANLOADBALANCE_PRE &>/dev/null; do
  echo "Waiting for WLB to insert its hook..."; sleep 0.1
done

INDEX=1 # Starting index for route tables and marks
TAG="-m comment --comment WLB_SYMMETRY " 

for INTERFACE in $WAN_INTERFACES; do 
  INTERFACE=${INTERFACE//\'} # strip quotes
  MARK=$[INDEX++]
  PARAMS="-t mangle -I PREROUTING -i $INTERFACE "
  PARAMS+="-j CONNMARK --set-xmark $MARK "
  PARAMS+="$TAG "
  echo "Inserting PREROUTING rule, connmark $MARK for packets in $INTERFACE"
  iptables $PARAMS
done

echo "Inserting rule in OUTPUT chain to mark locally-generated packets"
iptables -t mangle -I OUTPUT -j CONNMARK --restore-mark $TAG

INITSCRIPTS=(
  "/etc/init.d/vyatta-wanloadbalance"
  "/opt/vyatta/sbin/vyatta-wanloadbalance.init"
)
for SCRIPT in ${INITSCRIPTS[@]}; do
  if [[ -e $SCRIPT ]] && ! grep -q WLB_SYMMETRY $SCRIPT; then
    echo "Hook not found in $SCRIPT, adding one ..."
    echo -e "\n$0 | logger -p notice -t $0 # WLB_SYMMETRY" >> $SCRIPT
  fi
done

# End
3

i just discovered something weird… on legacy vyatta i left only CABLE and FIBER connections and both vyatta and nat-ed services are reachable from internet… configuration was not changed (same for years) and it does not include anything with PBR. my current conclusion pppoe setup is to blame for messing up routing table on vyatta to prevent reaching it from outside…
can someone help how to configure pppoe connection without it changing default route? gateway does not seems to be one the same network as assigned address and cannot be pinged from router…