HA project /w two residential ISPs

[trmg]

I am working on designing the topology above. All the routers in the diagram will be running VyOS. Yes, this is overly complicated, but it’s a personal project to attempt to configure active/active residential ISPs with high availability. The end goal is to be able to take any one element down (one ISP edge, one core router) with minimal downtime.

The switch and clients represented on the diagram is for illustration purposes. The focus is the routers.

• Two ISPs, each will provide DHCP WAN addresses
• ISP1-EDGE and ISP2-EDGE will handle NAT.
• ISP1-EDGE and ISP2-EDGE will iBGP (/w BFD) peer with CORE-HA.
• ISP1-EDGE and ISP2-EDGE will advertise a default route.
• CORE-HA will advertise its respective prefixes (172.27.0.1/24 per the example).
• CORE-HA will be a VRRP HA pair.

I have a decent idea on most of the configuration. However, there is a scenario that I could use some advice on:

When a given client initiates a given flow, BGP will choose an egress path. By nature of the how the ISPs are arranged, the reply traffic will follow the same path. BGP should maintain the path for an established flow and things should Just Work in this scenario.

However, if a connection originates from the outside, I don’t believe there is any guarantee that the return traffic will egress via the same path. I believe there is a 50/50 chance BGP will elect to egress via the opposing ISP edge path. I think this might be where policy routing can help, but I’m not sure.

Curious on y’all’s thoughts!

[trmg]

I think I have an idea that I will try to test soon.

I can build a policy to connection-mark packets from one of the edges like so:

set policy route mark-test interface '[INTERFACE-TO-ISP2-EDGE]'
set policy route mark-test rule 10 set connection-mark '1337'

Then I can build a policy route like below to send traffic to the appropriate router:

set protocols static table 1337 route 0.0.0.0/0 next-hop [ISP2-EDGE]

set policy route mark-test-nexthop rule 10 mark 1337
set policy route mark-test-nexthop rule 10 set table 1337

I would then rinse and repeat for ISP1 if I don’t want to influence BGP via local preference on CORE-HA, but I’m thinking in this scenario I’d likely prefer ISP 1 and ISP 2 would simply be for failover purposes (but if I mark both, they could be ECMP’d).