Load balancing transit links

[senseivita]

I’m trying to convert my pfSense edge/main router/firewall to VyOS. Back on pfSense, the main router connected to a remote gateway for its fixed public IPv4 addresses. It used two tunnels to place-shift the addresses, and since these doubled as extra WANs, it also load-balanced the tunnels, this made up for the massive difference in speed between the links on each side.

For outbound traffic, load balancing was done locally as I mentioned. On the remote side, there wasn’t much to do other than assigning it’s own gateway to each tunneled interface.

For inbound traffic, most incoming traffic was natted full cone to the reverse proxy from the remote side all the way to the local intranet, without NAT between the sites. Since it’s more than one link between the sites, that makes it an ECMP route, thus a routing protocol is needed.

But there’s a note in the docs that says: “WAN Load Balacing should not be used when dynamic routing protocol is used/needed. This feature creates customized routing tables and firewall rules, that makes it incompatible to use with routing protocols.”.

I replaced both pfSense routers with VyOS, established the links, fine-tuned MTUs, established OSPF neighbor adjacencies, but only then it occurred to me to read the load balancing documentation, so I got kinda stuck (and currently am offline). I assume this is doable since I’ve done it for a while, if it is, could you guide a me little please? Just the big picture, a link or two. If there’s anything to research I’m happy to do my own homework; I just need a little push in the right direction.

If it’s not though, could you confirm as well? I can always double-NAT, I rather not to but it seems like VyOS can do full cone so I think it will be alright.

Almost forgot, the diagram above it’s a simplified illlustration inter-site, and intranet-Internet traffic flows, I was doing that just for fun, totally unrelated, but it worked out well for this too. It’s too colorful/distracting for documentation so it’s the only chance I’d ever get to use it too.

Thanks.

[senseivita]

Nevermind… I spoke too soon.

As it turns out both the remote and local VyOS instances are sort of massively dropping packets. Over SSH, I redirected the tcpdump output to Wireshark and, while I don’t know what to make of Wireshark captures, I know that retransmissions and random tons of connection resets should not be there.

This only happens whenever VyOS is involved. Neither Microtik CHR, nor OpenWRT, nor the 'senses have this issue, I tried setting MTU lower than what’s usable to an ultra safe value and testing with packets a third of that. It doesn’t matter. Setting the full sizing (MTU, MRU, MSSes in both v4 and v6) on my invariable always rock-solid PPP fiber link. no difference.

For comparison, above, there’s a running ping with more than a dozen echo requests successfully making the round trip. On VyOS, a dozen is too many; only a handful succeed, followed by about half that handful in failing requests. It’s not an old picture BTW, Mikrotik CHR’s UI is kinda hideous um… “retro”.

I really want to use VyOS but I have no time to troubleshoot the most basic connectivity, which again, it’s only an issue when VyOS is involved, be it vSphere, OpenStack, Hyper-V or sitting on the thing raw.

Agh! I can’t even laugh at that.

Maybe next year. Thanks anyway.

[senseivita]

I forgot I had this still going:

I was going to say ten packets, a ten, but it doesn’t sound as good in English as it does in what I speak around here so I went with a dozen. As it turned out, I was exactly right. So, at least I got good news there. I guess that’s the silver lining.

[16again]

What path is taken? Ping times are doubled, suggesting a single ping goes twice back and forth.
Also, is routing protocol constantly flapping?
Like tunnel comes up, OSPF neigbor comes up, routes are learned, and these routes kill the tunnel.