Since the network between your nodes is static through your ISP (each router have its default gateway statically pointed to the ISP CPE) I would go for:
-
As first level (what the ISP will see if wiretapping) use wireguard to encrypt traffic between your routers. Make sure that multicore en/decryption is enabled in the configuration of wireguard (I dont recall if there is an explicit setting for this in vyos-config or if this is now the default).
-
Then use EVPN/VXLAN which means that inside the wireguard encrypted tunnel your routers will do MP-BGP between each other (to exchange the EVPN information).
-
Also inside this wireguard encrypted tunnel there will be VXLAN traffic which is what your downstream hosts of each router will end at (well your routers will do the en/decapsulation of VXLAN).
-
Inside the VXLAN tunnels there will be the packets your hosts connected to each node are sending to each other.
With the above the routers will have one part which is the underlay where the wireguard and inside of that the MP-BGP will occur. And one overlay which is what the hosts will see. Underlay and overlay will be two different VRF’s (preferly with VyOS (Linux) lingo also NETNS for full isolation).
This way your hosts at each site can do IPv4, IPv6 and plain Layer2 ethernet - all three will be encapsulated in VXLAN (4789/UDP) by your routers which along with the signaling of EVPN (made by MP-BGP) will go through that encrypted wireguard tunnel.
One drawback, since its internet, is that you are limited to 1500 bytes MTU to be sent through your ISP.
This means that since wireguard uses 60 bytes if IPv4 is used or 80 bytes if IPv6:
https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html
The overhead of WireGuard breaks down as follows:
- 20-byte IPv4 header or 40 byte IPv6 header
- 8-byte UDP header
- 4-byte type
- 4-byte key index
- 8-byte nonce
- N-byte encrypted data
- 16-byte authentication tag
This gives that if the network between your nodes is IPv4 wireguard will eat up 60 bytes while if its IPv6 then it will consume 80 bytes.
Then VXLAN on its own will eat up 50 bytes.
So the MTU your hosts will be able to use will become (assuming IPv4 through ISP) 1500-60-50 = 1390 bytes.
So make sure that you allow for path mtu discovery between the hosts unless you tweak their MTU setting to 1390 bytes (the later will also be needed for UDP traffic).
If you would instead have a dedicated wan or wavelengths/darkfiber between your nodes you could enable jumboframes and make it the other way around like so: 1500+60+50 = 1610 bytes. That is using 1610 bytes as jumboframes between your nodes which then would mean that a MTU of 1500 bytes on the hosts would work.
Things to add to the above design is to make sure that you enable BFD between the routers - this way they can much faster and with less CPU (compared to lowering BGP timers) detect if a path no longer works.
Reason why I would select wireguard over IPsec today is easier configuration and higher performance. On the other hand if you will involve routers of other vendors, who current doesnt support wireguard, you will be stuck with IPsec if you want to have “one design to fit them all”. You could of course for those cases have IPsec just towards those routers and use wireguard between the routers who do support this.
The above design will also give that you can select on your own which node should be speaking to which node. Like if you want a star-network where everything passes R4 or if you want a fullmesh where if the R4 is gone then R1 and R2 can still exchange packets (since you have your ISP in between your nodes as I understood it).
