Non Stable Internet Connection WAN-Load-Balancing

blacktux88 · October 10, 2019, 1:44pm

Dear Commuity

I have a strange “problem” with my vyos Setup (AlixBoard)

I had configured WB with my USG Pro , after i recogized that the Unifi Components could not Wan Load Balance wan interfaces, i ordered an Alix Board and installed Vyos 1.2.3 LTS.

So I have configured an simple Firewall Config.
172.16.0.0/24 is allowed to connect to the Internet through 2x PPPOE Internet Connections
nothing spezial. ( 1x PPPOE 150 MBIT and the second with 80 MBIT)
DHCP is enabled and DNS handels my PI Hole.

BUT there is always an BUT =)

I have problems with the internet connections , not every second connection but every X connection seems to be corruped , i have to refresh the Browser. After that it works.
Speedtest.net Connections works perfectly (220MBIT Down)
Steam Connections works also perfectly (25-30 MB per Seconds)

Thanks

s.lorente · October 11, 2019, 12:06am

Hi blacktux88,

What do you mean by corrupted?
What happens if you leave a continuous ping to an IP address? (That would also tell us if the problem could have anything to do with your Pi Hole)
Have you only felt that problem when browsing?
Would you share some logs? Specially covering when the problem happens.

zsdc · October 11, 2019, 12:50pm

@blacktux88, show please your configuration and the output of show interfaces detail command. Such problems can occur in case if part of your outgoing connections from the same device goes via one uplink, and part via other, and those uplinks has different characteristics.

blacktux88 · October 11, 2019, 3:02pm

Sorry for no answering , but my account was disabled

@s.lorente .
i have 3 routing Settings
One for PPPOE0 wan-loadbalance Test to IP 8.8.4.4
second for PPPOE1 wan-loadbalance Test to 1.1.1.1

And 0.0.0.0/0 with distance interface-pppoe0 distance "1!
if you say me how i can upload my config file.

Alll three of them are without packet loss during the hole time .
i have also make an video because its very difficult to explain this issue.

But you can understand it like this:

"I browser google , news.google.at or lteforum.at.
One time it works perfectly a second or third time it could run in an timeout ( But ping works perfectly)
If its not working you have to refresh the browser 1 or 2 times after that it works.
This extremly sounds like dns problem but i have also tried to configure manuel 8.8.8.8 in my NIC… this “works” the same way.

vyos@vyos:~$ show interfaces detail
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0d:b9:42:d7:30 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.254/24 brd 172.16.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20d:b9ff:fe42:d730/64 scope link
valid_lft forever preferred_lft forever

RX:  bytes    packets     errors    dropped    overrun      mcast
2773705255    5475541          0         10          0      38729
TX:  bytes    packets     errors    dropped    carrier collisions
12137073778    9934802          0          0          0          0

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0d:b9:42:d7:31 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20d:b9ff:fe42:d731/64 scope link
valid_lft forever preferred_lft forever
Description: BONDING

RX:  bytes    packets     errors    dropped    overrun      mcast
6855096095    5476074          0      71426          0      36064
TX:  bytes    packets     errors    dropped    carrier collisions
1435392951    2901608          0          0          0          0

eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0d:b9:42:d7:32 brd ff:ff:ff:ff:ff:ff
inet6 fe80::20d:b9ff:fe42:d732/64 scope link
valid_lft forever preferred_lft forever
Description: A1

RX:  bytes    packets     errors    dropped    overrun      mcast
5378322070    4726477          0          0          0      35931
TX:  bytes    packets     errors    dropped    carrier collisions
1347244533    2336822          0          0          0          0

lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

RX:  bytes    packets     errors    dropped    overrun      mcast
      2876         26          0          0          0          0
TX:  bytes    packets     errors    dropped    carrier collisions
      2876         26          0          0          0          0

pppoe0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN group default qlen 3
link/ppp
inet 178.251.70.1 peer 100.64.96.1/32 scope global pppoe0
valid_lft forever preferred_lft forever

RX:  bytes    packets     errors    dropped    overrun      mcast
5461545286    4267525          0          0          0          0
TX:  bytes    packets     errors    dropped    carrier collisions
 938294992    2158399          0          0          0          0

pppoe1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN group default qlen 3
link/ppp
inet 193.81.39.72 peer 91.115.231.254/32 scope global pppoe1
valid_lft forever preferred_lft forever

RX:  bytes    packets     errors    dropped    overrun      mcast
4491551941    4009405          0          0          0          0
TX:  bytes    packets     errors    dropped    carrier collisions
1103626392    1959577          0          0          0          0

thanks

blacktux88 · October 11, 2019, 3:03pm

set firewall all-ping ‘enable’
set firewall broadcast-ping ‘disable’
set firewall config-trap ‘disable’
set firewall group network-group NET-CLIENTS network ‘172.16.0.0/24’
set firewall ip-src-route ‘disable’
set firewall ipv6-receive-redirects ‘disable’
set firewall ipv6-src-route ‘disable’
set firewall log-martians ‘enable’
set firewall name LAN-IN default-action ‘drop’
set firewall name LAN-IN rule 10 action ‘accept’
set firewall name LAN-IN rule 10 state established ‘enable’
set firewall name LAN-IN rule 10 state related ‘enable’
set firewall name LAN-IN rule 100 action ‘accept’
set firewall name LAN-IN rule 100 source group network-group ‘NET-CLIENTS’
set firewall name LAN-IN rule 100 state new ‘enable’
set firewall name WAN-IN default-action ‘drop’
set firewall name WAN-IN rule 10 action ‘accept’
set firewall name WAN-IN rule 10 state established ‘enable’
set firewall name WAN-IN rule 10 state related ‘enable’
set firewall name WAN-IN rule 100 action ‘accept’
set firewall name WAN-IN rule 100 destination address ‘172.16.0.251’
set firewall name WAN-IN rule 100 destination port ‘8443’
set firewall name WAN-IN rule 100 protocol ‘tcp’
set firewall name WAN-IN rule 100 state new ‘enable’
set firewall name WAN-LOCAL default-action ‘drop’
set firewall name WAN-LOCAL rule 10 action ‘accept’
set firewall name WAN-LOCAL rule 10 state established ‘enable’
set firewall name WAN-LOCAL rule 10 state related ‘enable’
set firewall name WAN-LOCAL rule 20 action ‘accept’
set firewall name WAN-LOCAL rule 20 icmp type-name ‘echo-request’
set firewall name WAN-LOCAL rule 20 protocol ‘icmp’
set firewall name WAN-LOCAL rule 20 state new ‘enable’
set firewall name WAN-LOCAL rule 30 action ‘drop’
set firewall name WAN-LOCAL rule 30 destination port ‘2222’
set firewall name WAN-LOCAL rule 30 protocol ‘tcp’
set firewall name WAN-LOCAL rule 30 recent count ‘4’
set firewall name WAN-LOCAL rule 30 recent time ‘60’
set firewall name WAN-LOCAL rule 30 state new ‘enable’
set firewall name WAN-LOCAL rule 31 action ‘accept’
set firewall name WAN-LOCAL rule 31 destination port ‘2222’
set firewall name WAN-LOCAL rule 31 protocol ‘tcp’
set firewall name WAN-LOCAL rule 31 state new ‘enable’
set firewall name WAN-LOCAL rule 40 action ‘accept’
set firewall name WAN-LOCAL rule 40 destination port ‘1194’
set firewall name WAN-LOCAL rule 40 protocol ‘udp’
set firewall name WAN-LOCAL rule 40 state new ‘enable’
set firewall name WAN-LOCAL rule 50 action ‘accept’
set firewall name WAN-LOCAL rule 50 destination port ‘500’
set firewall name WAN-LOCAL rule 50 protocol ‘udp’
set firewall name WAN-LOCAL rule 50 state new ‘enable’
set firewall name WAN-LOCAL rule 51 action ‘accept’
set firewall name WAN-LOCAL rule 51 destination port ‘1701’
set firewall name WAN-LOCAL rule 51 protocol ‘udp’
set firewall name WAN-LOCAL rule 51 state new ‘enable’
set firewall name WAN-LOCAL rule 52 action ‘accept’
set firewall name WAN-LOCAL rule 52 destination port ‘4500’
set firewall name WAN-LOCAL rule 52 protocol ‘udp’
set firewall name WAN-LOCAL rule 52 state new ‘enable’
set firewall receive-redirects ‘disable’
set firewall send-redirects ‘enable’
set firewall source-validation ‘disable’
set firewall syn-cookies ‘enable’
set firewall twa-hazards-protection ‘disable’
set interfaces ethernet eth0 address ‘172.16.0.254/24’
set interfaces ethernet eth0 duplex ‘auto’
set interfaces ethernet eth0 firewall in name ‘LAN-IN’
set interfaces ethernet eth0 hw-id ‘HIDEME’
set interfaces ethernet eth0 smp-affinity ‘auto’
set interfaces ethernet eth0 speed ‘auto’
set interfaces ethernet eth1 description ‘BONDING’
set interfaces ethernet eth1 duplex ‘auto’
set interfaces ethernet eth1 hw-id ‘HIDEME’
set interfaces ethernet eth1 pppoe 0 default-route ‘auto’
set interfaces ethernet eth1 pppoe 0 firewall in name ‘WAN-IN’
set interfaces ethernet eth1 pppoe 0 firewall local name ‘WAN-LOCAL’
set interfaces ethernet eth1 pppoe 0 mtu ‘1492’
set interfaces ethernet eth1 pppoe 0 name-server ‘auto’
set interfaces ethernet eth1 pppoe 0 password ‘HIDEME’
set interfaces ethernet eth1 pppoe 0 policy route ‘MSS’
set interfaces ethernet eth1 pppoe 0 user-id ‘HIDEME’
set interfaces ethernet eth1 smp-affinity ‘auto’
set interfaces ethernet eth1 speed ‘auto’
set interfaces ethernet eth2 description ‘A1’
set interfaces ethernet eth2 duplex ‘auto’
set interfaces ethernet eth2 hw-id ‘HIDEME’
set interfaces ethernet eth2 pppoe 0 default-route ‘auto’
set interfaces ethernet eth2 pppoe 0 mtu ‘1492’
set interfaces ethernet eth2 pppoe 0 name-server ‘auto’
set interfaces ethernet eth2 pppoe 1 firewall in name ‘WAN-IN’
set interfaces ethernet eth2 pppoe 1 firewall local name ‘WAN-LOCAL’
set interfaces ethernet eth2 pppoe 1 password ‘HIDEME’
set interfaces ethernet eth2 pppoe 1 policy route ‘MSS’
set interfaces ethernet eth2 pppoe 1 user-id ‘HIDEME’
set interfaces ethernet eth2 smp-affinity ‘auto’
set interfaces ethernet eth2 speed ‘auto’
set interfaces loopback lo
set interfaces wireless wlan0 expunge-failing-stations ‘false’
set interfaces wireless wlan0 hw-id ‘HIDEME’
set interfaces wireless wlan0 isolate-stations ‘false’
set interfaces wireless wlan0 mgmt-frame-protection ‘disabled’
set interfaces wireless wlan0 mode ‘g’
set interfaces wireless wlan0 physical-device ‘phy0’
set interfaces wireless wlan0 type ‘monitor’
set load-balancing wan interface-health pppoe0 failure-count ‘5’
set load-balancing wan interface-health pppoe0 nexthop ‘dhcp’
set load-balancing wan interface-health pppoe0 success-count ‘1’
set load-balancing wan interface-health pppoe0 test 10 resp-time ‘5’
set load-balancing wan interface-health pppoe0 test 10 target ‘8.8.4.4’
set load-balancing wan interface-health pppoe0 test 10 ttl-limit ‘1’
set load-balancing wan interface-health pppoe0 test 10 type ‘ping’
set load-balancing wan interface-health pppoe1 failure-count ‘5’
set load-balancing wan interface-health pppoe1 nexthop ‘dhcp’
set load-balancing wan interface-health pppoe1 success-count ‘1’
set load-balancing wan interface-health pppoe1 test 10 resp-time ‘5’
set load-balancing wan interface-health pppoe1 test 10 target ‘1.1.1.1’
set load-balancing wan interface-health pppoe1 test 10 ttl-limit ‘1’
set load-balancing wan interface-health pppoe1 test 10 type ‘ping’
set load-balancing wan rule 100 inbound-interface ‘eth0’
set load-balancing wan rule 100 interface pppoe0 weight ‘1’
set load-balancing wan rule 100 interface pppoe1 weight ‘1’
set load-balancing wan rule 100 protocol ‘all’
set load-balancing wan sticky-connections inbound
set nat destination rule 100 description ‘ALL UNIFI’
set nat destination rule 100 destination port ‘8443’
set nat destination rule 100 inbound-interface ‘pppoe0’
set nat destination rule 100 protocol ‘tcp’
set nat destination rule 100 translation address ‘172.16.0.251’
set nat destination rule 100 translation port ‘8443’
set nat source rule 100 outbound-interface ‘pppoe0’
set nat source rule 100 source address ‘172.16.0.0/24’
set nat source rule 100 translation address ‘masquerade’
set nat source rule 110 outbound-interface ‘pppoe0’
set nat source rule 110 source address ‘172.16.9.0/24’
set nat source rule 110 translation address ‘masquerade’
set nat source rule 200 outbound-interface ‘pppoe1’
set nat source rule 200 source address ‘172.16.0.0/24’
set nat source rule 200 translation address ‘masquerade’
set policy route MSS description ‘TCP MSS clamping for PPPoE’
set policy route MSS rule 5 protocol ‘tcp’
set policy route MSS rule 5 set tcp-mss ‘1452’
set policy route MSS rule 5 tcp flags ‘SYN’
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe0 distance ‘1’
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe1 distance ‘1’
set protocols static interface-route 1.1.1.1/32 next-hop-interface pppoe1
set protocols static interface-route 8.8.4.4/32 next-hop-interface pppoe0
set service dns forwarding allow-from ‘0.0.0.0/0’
set service dns forwarding cache-size ‘1000’
set service dns forwarding listen-address ‘172.16.0.1’
set service dns forwarding name-server ‘8.8.8.8’
set service dns forwarding name-server ‘1.1.1.1’
set service ssh port ‘2222’
set system config-management commit-revisions ‘100’
set system console device ttyS0 speed ‘115200’
set system host-name ‘vyos’
set system login user vyos authentication encrypted-password ‘HIDEME’
set system login user vyos authentication plaintext-password ‘’
set system login user vyos level ‘admin’
set system name-server ‘9.9.9.9’
set system ntp server 0.pool.ntp.org
set system ntp server 1.pool.ntp.org
set system ntp server 2.pool.ntp.org
set system syslog global facility all level ‘info’
set system syslog global facility protocols level ‘debug’
set system time-zone ‘UTC’
set vpn ipsec ipsec-interfaces interface ‘pppoe0’
set vpn ipsec nat-networks allowed-network 0.0.0.0/0
set vpn ipsec nat-traversal ‘enable’
set vpn l2tp remote-access authentication local-users username HIDEMEa password ‘HIDEME’
set vpn l2tp remote-access authentication mode ‘local’
set vpn l2tp remote-access client-ip-pool start ‘172.16.9.2’
set vpn l2tp remote-access client-ip-pool stop ‘172.16.9.10’
set vpn l2tp remote-access idle ‘1800’
set vpn l2tp remote-access ipsec-settings authentication mode ‘pre-shared-secret’
set vpn l2tp remote-access ipsec-settings authentication pre-shared-secret ‘HIDEME’
set vpn l2tp remote-access ipsec-settings ike-lifetime ‘3600’
set vpn l2tp remote-access ipsec-settings lifetime ‘3600’
set vpn l2tp remote-access outside-address ‘HIDEME’

zsdc · October 11, 2019, 5:38pm

Thank you for the information. Check, please, additionally show wan-load-balance. It looks like with your config both links should always be treated as “failed”, because of ttl-limit '1'.

blacktux88 · October 11, 2019, 8:05pm

hello ,

@zsdc - “1” is the default value in the ttl-limit section

sorry for the late response.
here is my log … whats the perfekt ttl-limit ?

vyos@vyos:~$ show wan-load-balance
Interface: pppoe0
Status: active
Last Status Change: Thu Oct 10 20:23:14 2019
+Test: ping Target: 8.8.4.4
Last Interface Success: 1s
Last Interface Failure: 28m51s
# Interface Failure(s): 0

Interface: pppoe1
Status: active
Last Status Change: Thu Oct 10 20:23:09 2019
+Test: ping Target: 1.1.1.1
Last Interface Success: 1s
Last Interface Failure: 1h40m39s
# Interface Failure(s): 0

blacktux88 · October 12, 2019, 9:25am

I have uploaded my SmokePing Summary startet yesterday evening.

Should i upload an video ,which shows the problem ? could you tell me how to do this ? where i can upload an video

s.lorente · October 12, 2019, 9:52am

According to the output of your show interfaces detail, there are incoming packet drops at eth1.

Please check your eth2 configuration, it has 2 different PPOE unit numbers.

If that was not the problem, and if it is ok with you, please send us the logs (show log) covering the problem.

And, did you check with both flow and packet-based balancing?

blacktux88 · October 12, 2019, 2:32pm

Hello

Thanks

show log is no problem for me , but how should i attach it , in Chat or as attachement.

i have tried several things
I have added an exclusoin for the dns servers in Load Balance -> no change
i have set default route for both PPPOE in interface section to none
and after that i hav set the default route through the “set protocol” part --> no change.

i created an Video an also there is a way to check the connection with the firefox or chrome network debug settings. It Seems to be an dns resolf error but HOW

i have testet
-> google dns in Network settings
-> google dns through pihole
-> google dns through the vyos box

so give me constrution how i am allowed to add video files and config files

Thanks guys for helping

s.lorente · October 12, 2019, 5:52pm

When replying, check the buttons above, not all of them are format buttons, there is an upload button among them.

Or you can paste the logs here, everything is fine, whatever is easier for you.

blacktux88 · October 12, 2019, 6:35pm

Hello here the Log.

The video will be uploaded later

https://drive.google.com/open?id=1NjPSyG0cNLr_Jij1BgT1_7ts5vhs7ZpZ
log.txt (179.2 KB)

s.lorente · October 13, 2019, 12:07am

You probably tested some of these things and concluded something, but I need your confirmation in order to be in the same page.

Do you mean you configured it so that name resolution is not done by the Pi Hole anymore (but directly by 8.8.8.8 for instance) and you experienced the same problem?

So, before, when you wrote ‘This extremly sounds like dns problem but i have also tried to configure manuel 8.8.8.8 in my NIC… this “works” the same way.’

How exactly was that test? Did you test the load-balanced connection taking the Pi Hole out of the network and using 8.8.8.8 directly as your name resolver and still you experienced the same problem?

If the Pi Hole is not completely discarded yet as a problem-maker candidate, if possible, it’d be useful to delete the load-balancing configuration and test only one WAN link letting the Pi Hole be the DNS server. And also test separately one WAN at a time having the Pi Hole out of the network (just to confirm the problem does not belong to the ISP).

Please let us know the outcomes.

blacktux88 · October 14, 2019, 6:59am

So guys

Sorry for late answer , but the “time” Problem beteween Europe <-> your time is hard =)

U are alright , i have ment that it could be an DNS issue , but for now i know its not !

i have testet it how you discripted…

DNS configured on my clients is the pi-hole (pi-hole ask google for resolf) -> Same issue
DNS configuried on my clients is the vyos box (vyos Box ask google and quad 1 for resolf ) -> same issue
DNS configuried on my clients is 8.8.8.8 -> Same issue

I have tried to show with tcpdump --> but for that im to bad in debugging.

So i have tried the setup with pfsense , and there is no issue ( pi-hole as DNS)

because i really like the project vyos i want to use it.

i have tried an other idea

i have delete my default route and have added pppoe0 this

set interfaces ethernet eth1 pppoe 0 default-route ‘force’

after that i have set pppoe 1 default-route to none
the interesting part is that with this config the pppoe0 cant get up ,

Thanks

s.lorente · October 21, 2019, 8:47pm

Hi,

Do you have the same problem with
set load-balancing wan rule 100 per-packet-balancing ?

And does the following make any difference?
set load-balancing wan disable-source-nat
(By default load-balancing internally uses SNAT as a mechanism to behave as a unique interface)

Maybe you can try to check the logs in exactly the same second as the problem occurs, and also check what you see on Wireshark in exactly that same second.