I have been playing around with DMVPN in a virtual lab. I have everything configured but have noticed some oddities. For starters the output of ‘show vpn ipsec sa’ does not seem right with a dual hub configuration nor is there anything in ‘show vpn ike sa’ so not sure how one is supposed to validate ipsec connectivity? The bigger problem I am noticing is when simulating IPSec connectivity issues (it happens in the real world) that traffic is routed out the ‘internet’ as clear text GRE when the IPSec tunnel is down. If this was to be put into production across the actual internet this would be a HUGE security breach. In a DMVPN scenario how do you stop GRE from routing out to the ‘internet’ when IPSec is not established? And how can you properly verify all IPSec is established, especially with multiple hubs?
For the curious I found this workaround to stop GRE from routing to/from the internet without being encrypted:
sudo iptables -I VYATTA_PRE_FW_OUT_HOOK 1 -m policy --pol ipsec --dir out -p gre -j ACCEPT
sudo iptables -I VYATTA_PRE_FW_OUT_HOOK 2 -p gre -j DROP
sudo iptables -I VYATTA_PRE_FW_IN_HOOK 1 -m policy --pol ipsec --dir in -p gre -j ACCEPT
sudo iptables -I VYATTA_PRE_FW_IN_HOOK 2 -p gre -j DROP
This allows gre packets in/out as long as it’s via an ipsec policy but then blocks all other gre. You could also modify for specific peer IP to only block gre to/from specific peers. Add these to /config/scripts/vyos-postconfig-bootup.script so they survive a reboot. This works for VyOS 1.3 (and probably 1.2) which still uses iptables, not sure how this would work using VyOS 1.4 which I think has switched to nftables)
Putting the rules in VYATTA_PRE_FW_* chains seems to allow them to survive even when configuring firewall rules from the vyos configuration cli. You could also accomplish something similar from the vyos configuration cli for local inbound but not for outbound.
*** NOTE: I have only played with this workaround in a lab and cannot speak to its stability in a production situation ***
As for ‘show vpn ipsec sa’ on a DMVPN spoke with multiple hubs it still only shows one hub/tunnel so not sure on a workaround there.
Just to close this off, ‘show vpn ipsec sa verbose’ shows all the sa’s of dmvpn tunnels. Though, it’s not as “pretty” as it would be if ‘show vpn ipsec sa’ worked.
New VyOS user here about to deploy VyOS for its DMVPN capabilities… Can you help me understand why firewall rules cannot be added through the regular VyOS config to block outbound (unencrypted) GRE packets?
I would think there should be some sort of (optional) automatic kill switch that should prevent the unencrypted traffic through in the event that the IPSec goes down. Maybe that’s a feature request?
Edit: I’m looking to replace a site-to-site OpenVPN with DMVPN. Only a single hub, but three spoke sites (three different family houses). I want to make sure I’m keeping everything locked down.
Edit 2: I have tested this in my lab environment, and discovered that one of my spokes was already not communicating using IPSec to the hub (I didn’t even realize this), and allowing cleartext GRE traffic out. I tried to add a firewall rule to block outbound GRE packets (using the VyOS command line), but this did not work. I’m on 1.4.
If you used site to site try Wireguard, I’m sure you’ll like it.
Regarding firewall, auto generated rule it is a wrong thing. What will be if you use regular gre tunnels together with Dmvpn?
The simple full-mesh Wireguard is Tailscale which has a couple of major downsides for me. I want to learn IPSec and BGP, and I might want to use some Cisco routers at my spoke sites in the future - so this solution makes perfect sense for me. I found a tutorial on the VyOS blog and love this solution, I just have this security concern.
I specifically mentioned an “optional” rule, as I understand not everyone will want that behavior. However, there should be some way to block cleartext communications over the internet for those to do want that behavior.
I was looking to create a firewall rule that would block GRE packets outbound from the router, but it looks like I can only block inbound to the local router, or outbound from a routed packet behind the router. Is this true? There’s no way to firewall outbound traffic from the firewall itself?
Someone more knowledgeable in VyOS might have a better answer, but, as far as I am aware there is currently no way to block outbound traffic from the router itself using VyOS CLI. You need to use Linux iptables/nftables commands and put them in your vyos-postconfig-bootup.script to survive reboot. If you at least block clear text gre inbound from DMVPN peers then at least the tunnels/NHRP won’t work as they are blocked by the other end. However, that doesn’t solve the fact that some traffic could be sent clear text outbound (though it would get denied inbound on the peer side). The workaround I pasted above has continued to work well in my lab for blocking gre clear text outbound.
Maybe a disclaimer/warning should be explicitly stated in the documentation for anyone looking to implement the VyOS DMVPN solution in a production/enterprise environment with potential workarounds so people can take steps to safeguard their data. Even if the possibility is small, I would hate to have to explain to a CISO how data got leaked. Such things can be devastating to a company.
This to me may honestly be a deal breaker, and I’m not using this in a business environment. If I was, this would be an immediate no-go. The security implications are massive here for business environments. I messed around with a bunch of different rules trying to get the traffic blocked but it definitely does not work from the VyOS CLI. I’m on 1.4 (I don’t think your workaround will work, but I’ll play with it tonight), so I’m trying to learn nftables so I can make some rules to block the traffic at that level, but I’m still disappointed. Part of the draw to VyOS for me is the ability to have a single config file that I can keep in a git repo and restore to a new install if needed. This is light-years ahead of pfSense (where I’m coming from), which requires that everything be done through the web interface.
Maybe I’m naive, but I would think that if IPSec is configured on an interface, there’s a reasonable expectation that the traffic on that interface should be secured, and if the traffic is not secured, it probably shouldn’t leave that interface and travel over a public medium. Am I wrong?
I would agree. What would be nice is a tunnel interface command that can be manually set on gre tunnels that you expect to be protected by ipsec which would then automatically insert iptables/nftables rules to block unprotected gre to/from the destination tunnel peer (for ptp gre), or, as in the case of DMVPN, block all unprotected gre to/from the tunnel source. E.g.
…
set interfaces tunnel tun0 encapsulation ‘gre’
set interfaces tunnel tun0 multicast ‘enable’
set interfaces tunnel tun0 source-address ‘100.64.10.1’ set interfaces tunnel tun0 'protected’
…
Which could then result in some iptables rules like this:
iptables -I VYATTA_PRE_FW_OUT_HOOK 1 -m policy --pol ipsec --dir out -p gre -j ACCEPT
iptables -I VYATTA_PRE_FW_OUT_HOOK 2 -s 100.64.10.1/32 -p gre -j DROP
iptables -I VYATTA_PRE_FW_IN_HOOK 1 -m policy --pol ipsec --dir in -p gre -j ACCEPT
iptables -I VYATTA_PRE_FW_IN_HOOK 2 -d 100.64.10.1/32 -p gre -j DROP
Not sure if that would be possible with the vyos cli architecture. Maybe I’ll have to figure out how to submit a feature request.
Yes - that’s exactly what I’m looking for. That should definitely be possible for someone more knowledgeable with the code for the VyOS CLI. This is probably more of a bug-fix than a feature request, but it definitely has to be put in.
I have a couple of weird things going on in my testing…
I tried your iptables rules (started with outbound only) and they actually work to block outbound GRE traffic on VyOS 1.4, but it seems this has the effect of disconnecting that node from the DMVPN completely (though I could be wrong, sometimes the routers just don’t come back up at all after a reboot, and need another reboot).
This is what I put in the startup script:
sudo iptables -I VYOS_POST_FW 1 -m policy --pol ipsec --dir out -p gre -j ACCEPT
sudo iptables -I VYOS_POST_FW 2 -p gre -j drop
Additionally
It seems to be very easy to bring down the IPSec tunnels. I don’t know if you’re seeing this, but when I reboot everything, sometimes the tunnels all come up. When they do, I can ping the hub site from either of the spoke sites no problem. As soon as I ping a spoke site from a spoke site, the spoke site getting pinged loses its IPSec tunnel to the hub, and it doesn’t appear to attempt to reconnect. All traffic over that tunnel is clear GRE after that. Additionally, after a reboot, the IPSec tunnels seem to take a few seconds to get started, so a lot of the initial traffic, like BGP route advertisements, are sent in cleartext.
I’m currently in the middle of some experiments with 1.3, and that seems MUCH more stable (at least without BGP configured yet). I still have to configure BGP then I’ll post more results here.
My labs have all been VyOS 1.3 which has been stable using DMVPN/IPSec/BGP (only thing missing is it doesn’t have BGP neighbor listen range command). I have not attempted VyOS 1.4 yet. As @Viacheslav mentioned 1.4 uses nftables and I’m just not familiar enough with it yet.
I found that 1.4 still has the iptables command, and it seems that I can add rules to nftables with the iptables command. (Running the iptables commands then checking sudo nft list ruleset confirms this) I don’t think this had the end desired effect due to differences in chain configuration in 1.3 and 1.4. I tried it only because I’m not familiar with nftables yet.
In other news, I have setup my DMVPN / BGP lab on 1.3 and it is significantly more stable. as far as IPSec tunnels not constantly going down. Unfortunately, spokes don’t come back online after the hub reboots despite having configured dead peer detection on the spokes and hub, but that’s another topic.
I have tested adding start_action = trap to /etc/swanctl/swanctl.conf, and that works! (On a selfbuild iso of 1.4 from a couple of days ago). I’ve only tested this on a single spoke, but will test another spoke and report back when I’m off this week.
Specifically, here’s what this fixes:
When the spoke site initially establishes a connection to the hub, at least one GRE packet is sent before the IPSec is established. This no longer happens
When a hub reboots, the spoke reestablishes IPSec connectivity almost immediately (it may not be immediately, but it does reestablish the moment I try to ping the hub from the spoke). No GRE packets are seen between the hub and spoke.
Thank you so much for this solution, this is exactly what I’ve been looking for! Now it just needs to be added to the CLI so the change can be perpetuated across reboots.
I did some looking in the code, and I figured out what needs to be changed. I can make the fix and submit a patch, but I’m just waiting on my phabricator account to be approved so I can create the issue ticket.
I would still highly recommend having some sort of protection with iptables/nftables. I do not have my DMVPN lab configured anymore but I did a quick test using PTP GRE with IPSec and the GRE packets still try to escape clear text when IPSec is not established even with routed/trap mode. The option to manually set some sort of ‘protected’ keyword config to stop GRE escaping clear text when IPSec is intended would be highly desirable as mentioned above.