HA - via Two basic ISP's + bgp, vrrp, tunnel?

General questions
, , ,
1

Hello,

looking to determine how i would build the following topology:

  • two standard isp’s connect via separate tunnels to cloud vps to advertise our arin subnets (we have an ASN)
  • two vyos routers each advertising the same subnet public v4 & v6 subnets
  • somehow allow wan failover between vyos routers if one router goes offline (isp’s are not mpls or bgp)
  • we have 2 ISP’s terminated on a High availability core switch into two vlans
    1 is dhcp 1 is pppoe
  • 2 vyos vm’s
    *ibgp and ebgp?

we would like the following to occur, vyos establishes a tunnel via wireguard/ipsec etc. via each of the isp’s (for failover) - towards our 2 cloud vps bgp tunnel providers (bgptunnel, vultur, etc.)

if one of the isp’s goes down the subnet continues to be advertised, if one of the vyos routers goes down, subnet continues to be advertised, if one of the cloud vps hosts goes down subnet continues to advertise.

BGP Design
BGP Design388×823 22.6 KB

2

Are the public prefixes you’re announcing at the provider end terminating on-prem (inside the “internal network”), or is this an access path into backend services from prem to the provider end? I assume the former, just double checking.

What is running at the VPS end?

3

thinking the vps’s will be vyos as well.

the former, just need to be able to route the prefixes into the network, however we’d also like outbound traffic to identify via the prefix as well.

4

In rough strokes:

  • Establish a VPN mesh
    • I would use IPsec + DMVPNs through force of habit, independently hubbed on each cloud VPS. Each on-prem router would connect to both DMVPNs.
    • Anything that can carry OSPF and routed traffic is fine - Wireguard is well regarded for this as well.
  • BGP at the prem end would be in a private (or at least different) ASN
    • Originate advertised prefixes on the prem routers, so the VPS end can only announce what it’s learned (ie: the link is up and working)
    • Both prem routers should be peered together in iBGP
    • You likely don’t want to learn any routes from the VPS end BGP providers, if your ISPs accept outbound traffic from any source. Otherwise, routing becomes more exciting and you’ll need some statics or VRFs. This may complicate the VPN mesh.
  • BGP at the VPS end would be in your public ASN
    • I would probably configure one of the VPS routers to prepend, this encourages traffic to take the other path. If for some reason someone sees both paths as equally costed, you don’t want packets hitting both paths, unless you’re a fan of out-of-order packets. There is no chance 2 separate ISPs delivering separate tail technologies have tightly sync’d underlying timing.
    • Strip the private ASN out of the path when announcing to your BGP providers. It avoids a bit of stuffing around with the VPS<->prem sessions and any need to have VPS<->VPS for full meshing.
  • Either OSPF to internal prem routing or VRRP if static.
  • It’s a good idea to establish dummy interfaces for BGP sessions and advertise them using OSPF over the VPN, especially if you plan to expand sites or providers in future
  • Keep in mind you’ll be running below standard ethernet MTU for traffic crossing the mesh due to VPN overhead - this may affect some hosting and firewalling setups. Try not to break PMTUD.
5

on the on-prem routers is there a way to establish failover for each isp?
say R1 can receive both isp 1 & isp 2 wan ip’s?

6

If everything was static ethernet and you had a /29 or wider from each ISP, yes. (I’m aware this is not the case as you’ve detailed it).

If you can bring up multiple PPPoE sessions to the PPPoE ISP, and it’ll give a valid WAN IP on each session, partial yes.

If you can bring up multiple MACs facing the DHCP ISP and their DHCP will offer each one a unique valid IP, the other part of the provisional yes.

You would want to exclude all inside and outside VPN traffic from any kind of LB/failover.

To have sessions to each VPS hub over each WAN on each prem router, you would need to either use unique hub IPs on the VPS end and static routes on the prem end (one static and VPS IP per ISP path), engage in a bit of PBR and bind the VPNs to appropriate source interfaces or use a VPN with complete VRF support, containing each WAN link in its own VRF. These are all some level of brittle, complex or impossible for some VPN types.

However - the part of the solution most likely to see outages (and having the most points of failure) are the WAN links. Having a router fail at the same time as a WAN link is very unlikely.

If using WAN failover at all, I would have a different ISP as “primary” on each prem router and just run a single outbound pair of tunnels over that upstream. WAN-LB can be used for local traffic failover as long as the VPN traffic is excluded, and you understand that for prem clients, changing upstream means a connection reset or timeout.

Under no circumstance would I recommend putting VPN traffic through the WAN failover mechanism, it should be excluded and follow a static path. For dynamic routing to do its job effectively, you want dud links to die and reroute by OSPF/BGP, not cough, splutter and stutter while VPNs try to re-establish from changing external WAN IPs and underlying pathing. If all VPN tunnels are up and working from the beginning, traffic just switches to a different clean path until the original returns. No fuss.

7

Alternatively, I forgot to mention - if local traffic failover is all that is needed, and you can’t bring up all ISP interfaces on both routers, if one router loses upstream WAN, configure dynamic routing so that it just routes local traffic to the partner router for NAT outwards. This will be safe if VRRP still points at one prem router with a dead upstream, while the other keeps going.

Don’t do this for the VPNs, they will be very confused.

8

That makes sense.

Vrrp would be for the local traffic default gw internally?
Bgp or ospf for the default route learning outbound to the VPS prefixes?

How would snat work for allocating certain ips from the prefix locally? Or would it just all sit in pbr?

Dmvpn just allows a more efficient way to build the tunnels between all the nodes?

Any benefit between dmvpn vs. IPsec/gre tunnels, vs wireguard?

Would using vxlan be beneficial?

I would like it to be as dynamic as possible for scalability.

My level of understanding in networking is a few tiers under this setup however I’ve used ospf, vrrp before.

9

Correct.

I’d use OSPF to show BGP where to go - the VPN mesh and dummy interface IPs only. BGP->VPS for your prefix announcement (obviously), and it’s up to you what you want to import back to prem. Under most circumstances, I wouldn’t push outbound traffic over tunnels into the VPSes if at all possible. If the ISPs will allow traffic sourced from your prefixes going outbound without filtering, do that.

Normally, with an entire /24 or larger for servers, I’d allocate IPs directly, avoid DNAT and firewall if required. If you want to use it, you can use interface groups and conntrack sync so that NATs can shift around and fail over between prem routers.

I can’t help much with this on specifics, I’ve done a lot of basic NAT with VyOS but not since the nftables change and not with conntrack sync.

That’s the idea, yes. One configuration at the hub end, then one simple config per hub on each spoke. Under the hood, it’s effectively dynamically established IPsec/GRE.

DMVPN simplifies configuration a bit, especially if you have many spokes. Not super relevant for your use case, but it also allows spokes to establish direct tunnels to bypass the hub as a full mesh. NHRP dynamic multipoint VPN originated in Cisco-land and is supported by a few other vendors, including VyOS.

IPsec/GRE is roughly the same as far as your needs go, exactly the same for performance and security, but it requires manual configuration on both ends for each point to point link.

Wireguard I’m not familiar with - it’s not supported by much vendor tin and I’ve never had the need. It’s well regarded and meant to be easy to configure & secure, at least on par with DMVPN or OpenVPN.

IMO for your usage, it’d just be overhead. Do you need layer 2 connectivity between prem and VPS ends? Might you want to bridge something from VPS into prem in future? You’ll be shrinking your MTU to get it. You can of course use the L3 VPN mesh as a VXLAN underlay with a little effort.

Honestly, overall you’d be better off getting direct services that can deliver BGP to a prem tail, or even diverse L2 into co-lo and finding peering partner(s). I’m guessing you’re aware of that already and they’re either not available or too expensive for your usage. You’re dealing in a lot of complexity for something that should be fairly straightforward.

10

Part of the use case was essentially being able to be highly dynamic with ISP’s. Starlink, cell, fibre, etc.

As far as I’m aware it’s not possible to do IP transit over cell or satellite.

We did want to have both outbound and inbound via our subnets since we’re going to be using this for V6 and v4.

Overhead would just be the tunnel?
I’m reading that wireguard is 1420 or 1440 bytes v4/v6
IPsec being 100bytes so MTU 1400

I assume the main reason to avoid using the VPS on outbound would be to just avoid fragmentation and clients that don’t have ICMP pmtud enabled?

11

That, plus shaving off unneeded latency and avoiding double-dipping on any traffic charges.

Pretty much. For DMVPN, it’s transport mode IPsec with a GRE header inside it, so don’t use calcuations for plain tunnel mode. We normally set ours a little lower, occasionally some mobile towers adjust MTU down (I assume they’re tunnelling themselves for some reason).

It does depend on location and what is available at wholesale. I wouldn’t have much more trouble delivering either as private wholesale tails in AU than I would with direct access fibre or metro ethernet, though it wouldn’t be anywhere near cost-comparable to consumer internet. Fortunately, our customers are into that.

From what I’ve seen, it’s a lot more difficult to get wholesale last mile and backhaul like that in the EU or US - carriers there want to sell you an entire managed WAN, they don’t want to interoperate with other carriers or break up their offerings into useful pieces.

12

Yes. traffic charges would be quite low so that’s not a factor with all but our cell modem backup.
ideally we would be using 2 active-active links (gpon) with 2 backup links (cell, starlink).

still trying to wrap my head around the config. yes it’s quite complex however our other option is via Cloudflare, totaluptime, gcs, which would all just advertise our prefix via our asn and provide us with tunnels back to our servers. and we would still need to have two options for diversity.

would need multiple routing tables and pbr to take into account the prefixes vs the default route.

default route > vpn to vps> vrrp on vps? > ospf > vps bgp

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.