VYOS 1.2.RC5-AMD64 WAN Load Balancing - The clients can not open webpage

General questions
1

Dear all,
I have vyatta/vyos for a few years, it always work perfectly when it handle only one PPPOE connection or DSL static connection.
Now I have two pppoe connection from ISP, and I want to make it work together. So I did set the load balancing to check it and it doesn’t work.
The clients only can ping the website ip address, they can’t open the website. I am sure this is not DNS problem. (It can work perfectly when I change it on one PPPoE connection)
Could anyone help me to fix the problem? Thank you very much!
Below is the config code:
vyos@vyos# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route

K>* 0.0.0.0/0 [0/0] is directly connected, pppoe3, 00:29:05
C>* 100.64.0.1/32 is directly connected, pppoe3, 00:29:05
C>* 113.109.160.1/32 is directly connected, pppoe1, 00:29:02
S>* 114.114.114.114/32 [1/0] is directly connected, pppoe1, 00:04:51
  *                          is directly connected, pppoe3, 00:04:51
C>* 192.168.0.0/20 is directly connected, eth0, 00:29:04
----------------------------------------------

vyos@vyos# run show wan-load-balance 
Interface:  pppoe1
  Status:  active
  Last Status Change:  Wed Oct 31 12:52:08 2018
  +Test:  ping  Target: 114.114.114.114
    Last Interface Success:  0s 
    Last Interface Failure:  n/a                
    # Interface Failure(s):  0

Interface:  pppoe3
  Status:  active
  Last Status Change:  Wed Oct 31 12:52:08 2018
  +Test:  ping  Target: 114.114.114.114
    Last Interface Success:  0s 
    Last Interface Failure:  n/a                
    # Interface Failure(s):  0
----------------------------------------------
	
vyos@vyos# show int
 ethernet eth0 {
     address 192.168.0.176/20
     duplex auto
     hw-id 92:fd:26:c7:a3:12
     policy {
         route pppoeOut
     }
     smp-affinity auto
     speed auto
 }
 ethernet eth5 {
     duplex auto
     hw-id ba:13:86:6e:cb:ac
     pppoe 1 {
         default-route auto
         mtu 1492
         name-server auto
         password 'xxxxxxx'
         user-id 'xxxxxxxx'
     }
     smp-affinity auto
     speed auto
 }
 ethernet eth6 {
     duplex auto
     hw-id 9e:48:71:a5:dd:37     
     pppoe 3 {
         default-route auto
         mtu 1492
         name-server auto
         password 'xxxxxx'
         user-id 'xxxxxxxx'
     }
     smp-affinity auto
     speed auto
 }
----------------------------------------------
 
vyos@vyos# show policy 
 route pppoeOut {
     rule 1 {
         protocol tcp
         set {
             tcp-mss 1452
         }
         tcp {
             flags SYN
         }
     }
 }
----------------------------------------------

vyos@vyos# show nat 
source {
     rule 1 {
         outbound-interface pppoe1
         translation {
             address masquerade
         }
     }
     rule 3 {
         outbound-interface pppoe3
         translation {
             address masquerade
         }
     }
 }
----------------------------------------------

 vyos@vyos# show load-balancing 
wan {
     flush-connections
     interface-health pppoe1 {
         failure-count 1
         nexthop dhcp
         success-count 1
         test 1 {
             resp-time 5
             target 114.114.114.114
             ttl-limit 1
			 type ping		
         }
     }
     interface-health pppoe3 {
         failure-count 1
         nexthop dhcp
         success-count 1
         test 1 {
             resp-time 5
             target 114.114.114.114
             ttl-limit 1
			 type ping
         }
     }
     rule 1 {
         inbound-interface eth0
         interface pppoe1 {
             weight 10
         }
         interface pppoe3 {
             weight 10
         }
         protocol all
     }
 }
2

Hello, @coolvir!
Show, please, output of next commands:

ip l
ip r
3

ip left:eth0 (lan)
ip right:eth5(pppoe1),eth6(pppoe3)
and now it looks can ping everywhere,but still can not open the website.

4

I did try to setup 3 vyos system to check it (each 2 vyos point to each 2 ISP pppoe connection, and the third vyos setup three interface and route by other 2 vyos.), and I found it work normally.
Maybe the problem is policy route of the interface?Who can tell me why and how to fix the problem?

5

I am very sorry I miss it " [[solved] WAN load-balancing with policy route rules [previously: WAN load-balancing" in forum, But the method looks like ‘hack script’, Does the offical have the better solution?

6

Hello, @coolvir! I have mean ip link and ip route commands. :slightly_smiling_face: Example:

vyos@vyos:~$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:4b:33:e5 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:01:da:89 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 08:00:27:2d:07:cb brd ff:ff:ff:ff:ff:ff
vyos@vyos:~$ ip r
default via 10.0.2.2 dev eth0 proto static metric 20 
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.5 
192.168.20.0/24 dev eth1 proto kernel scope link src 192.168.20.60

Can you show your output of this commands?

7

I found the problem that is the policy route can not work on dual pppoe wan load balancing,do you have any better solution?

8

@coolvir The simplest solution is to balance based on a connection mark and not based on a packet.
There are couple approaches to this but the naive one is to use a “round robin” like algorithm which iptables will mark at prerouting for ingress traffic from the LAN.
To loadbalance the traffic from the VyOS machine itself I believe the settings would be a bit different.
I know how to set it all up using plain linux and iptables but not in VyOS.
It can work with dual pppoe connections

9

Thanks for your reply, Could you write down your config commands ? I think it can help a lot of people.

10

@coolvir if @syncer or @dmbaturin will not be able to answer faster then me I will try to see how it should be done on vyos.

edit:
@coolvir have you tried looking at: Load-balancing — VyOS 1.4.x (sagitta) documentation
?

11

Thank you very much for taking the time to reply, I wish this product more and more popular, thanks you for your efforts.

12

Hi elico,
I did try to set up load balancing by per packet mode and it not work.

13

@coolvir You need per flow not per packet.
In a per packet mode the syn will transmitted from one src IP while the syn-ack or ack or any other packet in the sequence will have different src IP.
Have you tried per flow?