I met a problem about ping issue from outside to wan load balance.
when I enabled wan load balancing for my two ISPs on vyos, I found that there is problem for ping IP of either ISPs from outside, the drop rate of ICMP ping is very high, I probably realized that it may be caused by two default gateways in routing table, but how to resolve it? because I need to check ISP’s healthy from outside, so I must be able to ping these two ISPs.
actually, I tried with policy routing as below, when ICMP destination hit the ISP IP, then go to the according routing table with correct gateway:
set policy route isp1_icmp rule 1 destination address <ISP1_IP>
set policy route isp1_icmp rule 1 protocol ‘icmp’
set policy route isp1_icmp rule 1 set table ‘100’
set protocols static table 100 route 0.0.0.0/0 next-hop <ISP1_gateway>
set interface ethernet eth0 policy route isp1_icmp
set policy route isp2_icmp rule 1 destination address <ISP2_IP>
set policy route isp2_icmp rule 1 protocol ‘icmp’
set policy route isp2_icmp rule 1 set table ‘200’
set protocols static table 200 route 0.0.0.0/0 next-hop <ISP2_gateway>
set interface ethernet eth1 policy route isp2_icmp
but it didn’t work, anybody can help me to solve it? thanks in advance.
the configuration for load balancing is as below:
set protocols static route 0.0.0.0/0 next-hop <ISP1_gateway>
set protocols static route 0.0.0.0/0 next-hop <ISP2_gateway>
set protocols static route 8.8.8.8/32 next-hop <ISP1_gateway>
set protocols static route 8.8.8.8/32 next-hop <ISP2_gateway>
set load-balancing wan ‘enable-local-traffic’
set load-balancing wan ‘flush-connections’
set load-balancing wan interface-health eth0 failure-count ‘5’
set load-balancing wan interface-health eth0 nexthop <ISP1_gateway>
set load-balancing wan interface-health eth0 success-count ‘1’
set load-balancing wan interface-health eth0 test 1 resp-time ‘5’
set load-balancing wan interface-health eth0 test 1 target ‘8.8.8.8’
set load-balancing wan interface-health eth0 test 1 ttl-limit ‘1’
set load-balancing wan interface-health eth0 test 1 type ‘ping’
set load-balancing wan interface-health eth1 failure-count ‘5’
set load-balancing wan interface-health eth1 nexthop <ISP2_gateway>
set load-balancing wan interface-health eth1 success-count ‘1’
set load-balancing wan interface-health eth1 test 1 resp-time ‘5’
set load-balancing wan interface-health eth1 test 1 target ‘8.8.8.8’
set load-balancing wan interface-health eth1 test 1 ttl-limit ‘1’
set load-balancing wan interface-health eth1 test 1 type ‘ping’
set load-balancing wan rule 1 description ‘network for load balancing’
set load-balancing wan rule 1 inbound-interface ‘eth3’
set load-balancing wan rule 1 interface eth0 weight ‘30’
set load-balancing wan rule 1 interface eth1 weight ‘70’
set load-balancing wan rule 1 protocol ‘all’
set load-balancing wan rule 1 source address ‘192.168.0.0/24’
if I disable load balancing, then problem is gone, as you know, load balancing for two ISPs require two default gateway defined, so when ping from internet come to vyos, then it faces problem that which gateway the ping response will go, I actually observed that ping response is distributed to two interface where ISP IPs locate by capturing packets.
for example, I ping ISP1 IP from somewhere, I can observe response both on eth0(ISP1 IP) and eth1(ISP2 IP), and I finally received the ping response through eth0, all response through eth1 was dropped somewhere, and one interesting thing is that all ping response(no matter through eth0 or eth1, its source IP always is ISP1 IP), so my guess is that rest of non-received response was dropped on ISP2’s network appliance because of some security reason, it’s easy to understand that ISP2 always assume the source IP of its outbound traffic should be its own IP, but now it’s ISP1 IP, it’s alien and dangerous with high risk, of course it will be dropped.
based on that, so I’m looking for a way that ping response always is through the same interface as ping request, as I said I have tried policy routing, but it didn’t work, so anyone have thoughts about it?
In normal condition load-balancing will affect only transit traffic that you describe in rules - for this traffic router create new routing tables. Locally generated traffic (e.g. ICMP from/to router) use main routing table. That why you don’t need to setup two default routes by hand.
But, if you just leave one default route, you would not able to access router through second ISP (transit traffic will be not affected). In this case you can use some workaround - script, that change default route when primary interface is down, for example.
P.S. In clean Linux system you can set rules, that allow to use both interfaces for input/output at same time, but as I know in VyOS this configuration is unsupported from scratch.
Thanks for your reply, I think I have already find the solution even it’s not elegant but it works for me.
Actually, it still use policy routing and with manual iptable settings.
As all you know, load balancing in vyos actually also use policy routing, with default settings, you will see two routing tables(for two ISP load balancing):
ip route show table 1
default via x.x.x.x.x(ISP1 GW) dev eth0
ip route show table 2
default via x.x.x.x.x(ISP2 GW) dev eth1
Here I reused these two tables and direct desired traffic to according routing tables. I choose to use OUTPUT chain in mangle table of iptable, if you are familiar with iptable, then you will know this chain is only used for locally generated traffic which is perfect for my requirement, here are rules:
sudo iptables -t mangle -I OUTPUT -s (ISP1 IP) -j MARK --set-mark 1
sudo iptables -t mangle -I OUTPUT -s (ISP2 IP) -j MARK --set-mark 2
note: the mark in the rule is mapping to corresponding routing tables
with settings above, then all locally generated traffic(no matter request or response) will be directed to desired routing table according to their source address
There is also a little trick, when you add iptable rules, remember use “Insert” rather then “Append” action if load balancing is used at same time since there is “-A OUTPUT -j WANLOADBALANCE_OUT” in the table, make sure that your rules is prior to this load balancing rule, otherwise it won’t take effect.
I had same problem in the past with load-balancing. Due to another issue (incompatibility with internet banking) had to disband WAN load-balancing altogether.
Anyway, it would be nice if mods listed by MapleWang will be added to VyOS rolling and stable 1.2.
