Need advice whether to go for IPsec/DMVPN/Wireguard/GRE/BGP or Wireguard/DMVPN/BGP

Hi Team,

Could someone please provide guidance on analyzing the diagram? I am in need of setting up a Vyos for approximately 11 branches, each of which will have two ISP links. Additionally, certain branches will need to communicate with each other and with R1. The diagram enclosed is the one I am currently working with is a simulation in my lab. Can consider

  • R1 as Hub
  • R2 and R3 are Spokes
  • And R4 is Internet

Wondering whether I should go for considering even

  1. IPsec/Gre/DMVPN/BGP OR
  2. Wireguard/DMVPN/GRE/BGP OR
  3. Wireguard/BGP OR
  4. IPsec/VTi/BGP OR
  5. OSPF will be better one than managing BGP?

Or any other mechanism you guys can suggest?

Wireguard +bgp or netbird on vyos

Since the network between your nodes is static through your ISP (each router have its default gateway statically pointed to the ISP CPE) I would go for:

  1. As first level (what the ISP will see if wiretapping) use wireguard to encrypt traffic between your routers. Make sure that multicore en/decryption is enabled in the configuration of wireguard (I dont recall if there is an explicit setting for this in vyos-config or if this is now the default).

  2. Then use EVPN/VXLAN which means that inside the wireguard encrypted tunnel your routers will do MP-BGP between each other (to exchange the EVPN information).

  3. Also inside this wireguard encrypted tunnel there will be VXLAN traffic which is what your downstream hosts of each router will end at (well your routers will do the en/decapsulation of VXLAN).

  4. Inside the VXLAN tunnels there will be the packets your hosts connected to each node are sending to each other.

With the above the routers will have one part which is the underlay where the wireguard and inside of that the MP-BGP will occur. And one overlay which is what the hosts will see. Underlay and overlay will be two different VRF’s (preferly with VyOS (Linux) lingo also NETNS for full isolation).

This way your hosts at each site can do IPv4, IPv6 and plain Layer2 ethernet - all three will be encapsulated in VXLAN (4789/UDP) by your routers which along with the signaling of EVPN (made by MP-BGP) will go through that encrypted wireguard tunnel.

One drawback, since its internet, is that you are limited to 1500 bytes MTU to be sent through your ISP.

This means that since wireguard uses 60 bytes if IPv4 is used or 80 bytes if IPv6:

The overhead of WireGuard breaks down as follows:

  • 20-byte IPv4 header or 40 byte IPv6 header
  • 8-byte UDP header
  • 4-byte type
  • 4-byte key index
  • 8-byte nonce
  • N-byte encrypted data
  • 16-byte authentication tag

This gives that if the network between your nodes is IPv4 wireguard will eat up 60 bytes while if its IPv6 then it will consume 80 bytes.

Then VXLAN on its own will eat up 50 bytes.

So the MTU your hosts will be able to use will become (assuming IPv4 through ISP) 1500-60-50 = 1390 bytes.

So make sure that you allow for path mtu discovery between the hosts unless you tweak their MTU setting to 1390 bytes (the later will also be needed for UDP traffic).

If you would instead have a dedicated wan or wavelengths/darkfiber between your nodes you could enable jumboframes and make it the other way around like so: 1500+60+50 = 1610 bytes. That is using 1610 bytes as jumboframes between your nodes which then would mean that a MTU of 1500 bytes on the hosts would work.

Things to add to the above design is to make sure that you enable BFD between the routers - this way they can much faster and with less CPU (compared to lowering BGP timers) detect if a path no longer works.

Reason why I would select wireguard over IPsec today is easier configuration and higher performance. On the other hand if you will involve routers of other vendors, who current doesnt support wireguard, you will be stuck with IPsec if you want to have “one design to fit them all”. You could of course for those cases have IPsec just towards those routers and use wireguard between the routers who do support this.

The above design will also give that you can select on your own which node should be speaking to which node. Like if you want a star-network where everything passes R4 or if you want a fullmesh where if the R4 is gone then R1 and R2 can still exchange packets (since you have your ISP in between your nodes as I understood it).

1 Like

Ooops that would too much of work I believe :slight_smile:
Yes I am gonna choose wireguard and BGP since its a pretty simple setup while evaluating netbird since I have never heard of that before and sounds interesting.

Hi Team,

Here is the issue I have with wireguard with multiple ISPs are multiple lcoations. In my scenarion

R110.10 <=> R230.10
R110.10 <=> R350.10
R120.10 <=> R240.10
R120.10<=> R360.10

What will be the wgN interfaces then? Do I need to configure separate Interface numbers for every connection?

R110.10 [wg10] 51820 <=> R230.10 wg10
R110.10 [wg20] 51821 <=> R350.10 wg20 ??? 

Do I need to configure separate wg20 interface or can I use same wg10 interface and peer different IP and if so then what about ports then? commit is failing
And then

R120.10 51823<=> R240.10 wg30
R120.10 51824<=> R360.10 wg40??

Wireguard supports both but I havent found any good pro/cons with each method. Perhaps somebody else in this forum have some available?

That is either you have a single wireguard interface (such as wg0) and configure multiple peers to that (each peer with its own public/private key pair and allowedips etc).

Or you configure one wgX interface per peer.

The practical difference is that with the first case the peers will talk to your box at (default) 51820 while when having one wgX interface per peer each will use its own port (51820, 51821, 51822 and so on).

I can imagine that the later is handy if you got several different vrf’s but other than that using a single wireguard interface seems to be the more common thing to do.

Some details on this subject:

If you compare to IPsec you would normally just use a single physical interface for all peers towards your box (or a loopback interface as source-interface in case you do dynamic routing).

If these are meant to be site-to-site tunnels, you’ll have to use different interfaces. Each interface can only use as an allowed IP for 1 peer.

Ok - that means if I have 10 peers with 2 ISP links on each spoke as well as on Hub. Then I believe I’ll have 20 wgX interfaces?
How do I keep one same interface on Hub and then I have spokes talking to Hub on port 51820?

So if I need to configure Wireguard+DMVPN is it necessary that I need to use gre tunnel?
Again I am so confused on wireguard tunnel with multiple peers.