DMVPN - GRE routed clear text when IPSec down

Someone more knowledgeable in VyOS might have a better answer, but, as far as I am aware there is currently no way to block outbound traffic from the router itself using VyOS CLI. You need to use Linux iptables/nftables commands and put them in your vyos-postconfig-bootup.script to survive reboot. If you at least block clear text gre inbound from DMVPN peers then at least the tunnels/NHRP won’t work as they are blocked by the other end. However, that doesn’t solve the fact that some traffic could be sent clear text outbound (though it would get denied inbound on the peer side). The workaround I pasted above has continued to work well in my lab for blocking gre clear text outbound.

Maybe a disclaimer/warning should be explicitly stated in the documentation for anyone looking to implement the VyOS DMVPN solution in a production/enterprise environment with potential workarounds so people can take steps to safeguard their data. Even if the possibility is small, I would hate to have to explain to a CISO how data got leaked. Such things can be devastating to a company.

This to me may honestly be a deal breaker, and I’m not using this in a business environment. If I was, this would be an immediate no-go. The security implications are massive here for business environments. I messed around with a bunch of different rules trying to get the traffic blocked but it definitely does not work from the VyOS CLI. I’m on 1.4 (I don’t think your workaround will work, but I’ll play with it tonight), so I’m trying to learn nftables so I can make some rules to block the traffic at that level, but I’m still disappointed. Part of the draw to VyOS for me is the ability to have a single config file that I can keep in a git repo and restore to a new install if needed. This is light-years ahead of pfSense (where I’m coming from), which requires that everything be done through the web interface.

Maybe I’m naive, but I would think that if IPSec is configured on an interface, there’s a reasonable expectation that the traffic on that interface should be secured, and if the traffic is not secured, it probably shouldn’t leave that interface and travel over a public medium. Am I wrong?

I would agree. What would be nice is a tunnel interface command that can be manually set on gre tunnels that you expect to be protected by ipsec which would then automatically insert iptables/nftables rules to block unprotected gre to/from the destination tunnel peer (for ptp gre), or, as in the case of DMVPN, block all unprotected gre to/from the tunnel source. E.g.


set interfaces tunnel tun0 encapsulation ‘gre’
set interfaces tunnel tun0 multicast ‘enable’
set interfaces tunnel tun0 source-address ‘100.64.10.1’
set interfaces tunnel tun0 'protected’

Which could then result in some iptables rules like this:

iptables -I VYATTA_PRE_FW_OUT_HOOK 1 -m policy --pol ipsec --dir out -p gre -j ACCEPT
iptables -I VYATTA_PRE_FW_OUT_HOOK 2 -s 100.64.10.1/32 -p gre -j DROP

iptables -I VYATTA_PRE_FW_IN_HOOK 1 -m policy --pol ipsec --dir in -p gre -j ACCEPT
iptables -I VYATTA_PRE_FW_IN_HOOK 2 -d 100.64.10.1/32 -p gre -j DROP

Not sure if that would be possible with the vyos cli architecture. Maybe I’ll have to figure out how to submit a feature request.

Yes - that’s exactly what I’m looking for. That should definitely be possible for someone more knowledgeable with the code for the VyOS CLI. This is probably more of a bug-fix than a feature request, but it definitely has to be put in.

Create a feature request at phabricator

This sounds more like a routing issue than a GRE/IPSec issue. Would you be willing to post steps to reproduce what you’ve observed?

I have a couple of weird things going on in my testing…

I tried your iptables rules (started with outbound only) and they actually work to block outbound GRE traffic on VyOS 1.4, but it seems this has the effect of disconnecting that node from the DMVPN completely (though I could be wrong, sometimes the routers just don’t come back up at all after a reboot, and need another reboot).

This is what I put in the startup script:

sudo iptables -I VYOS_POST_FW 1 -m policy --pol ipsec --dir out -p gre -j ACCEPT
sudo iptables -I VYOS_POST_FW 2 -p gre -j drop

Additionally

It seems to be very easy to bring down the IPSec tunnels. I don’t know if you’re seeing this, but when I reboot everything, sometimes the tunnels all come up. When they do, I can ping the hub site from either of the spoke sites no problem. As soon as I ping a spoke site from a spoke site, the spoke site getting pinged loses its IPSec tunnel to the hub, and it doesn’t appear to attempt to reconnect. All traffic over that tunnel is clear GRE after that. Additionally, after a reboot, the IPSec tunnels seem to take a few seconds to get started, so a lot of the initial traffic, like BGP route advertisements, are sent in cleartext.

I’m currently in the middle of some experiments with 1.3, and that seems MUCH more stable (at least without BGP configured yet). I still have to configure BGP then I’ll post more results here.

Small note, VyOS 1.4 don’t use iptables
It uses nftables

My labs have all been VyOS 1.3 which has been stable using DMVPN/IPSec/BGP (only thing missing is it doesn’t have BGP neighbor listen range command). I have not attempted VyOS 1.4 yet. As @Viacheslav mentioned 1.4 uses nftables and I’m just not familiar enough with it yet.

I found that 1.4 still has the iptables command, and it seems that I can add rules to nftables with the iptables command. (Running the iptables commands then checking sudo nft list ruleset confirms this) I don’t think this had the end desired effect due to differences in chain configuration in 1.3 and 1.4. I tried it only because I’m not familiar with nftables yet.

In other news, I have setup my DMVPN / BGP lab on 1.3 and it is significantly more stable. as far as IPSec tunnels not constantly going down. Unfortunately, spokes don’t come back online after the hub reboots despite having configured dead peer detection on the spokes and hub, but that’s another topic.

Can someone test statrt_action = trap ? It can help

I have tested adding start_action = trap to /etc/swanctl/swanctl.conf, and that works! (On a selfbuild iso of 1.4 from a couple of days ago). I’ve only tested this on a single spoke, but will test another spoke and report back when I’m off this week.

Specifically, here’s what this fixes:

  • When the spoke site initially establishes a connection to the hub, at least one GRE packet is sent before the IPSec is established. This no longer happens

  • When a hub reboots, the spoke reestablishes IPSec connectivity almost immediately (it may not be immediately, but it does reestablish the moment I try to ping the hub from the spoke). No GRE packets are seen between the hub and spoke.

Thank you so much for this solution, this is exactly what I’ve been looking for! Now it just needs to be added to the CLI so the change can be perpetuated across reboots.

I did some looking in the code, and I figured out what needs to be changed. I can make the fix and submit a patch, but I’m just waiting on my phabricator account to be approved so I can create the issue ticket.

I would still highly recommend having some sort of protection with iptables/nftables. I do not have my DMVPN lab configured anymore but I did a quick test using PTP GRE with IPSec and the GRE packets still try to escape clear text when IPSec is not established even with routed/trap mode. The option to manually set some sort of ‘protected’ keyword config to stop GRE escaping clear text when IPSec is intended would be highly desirable as mentioned above.

Good observation- I haven’t tested point to point tunnels.

For DMVPN on VyOS 1.4, a quick fix in the template file /usr/share/vyos/templates/ipsec/swanctl/profile.j2 will stop all but one or two GRE packets

I think the full fix should be as simple as having a firewall chain / table that can catch the outbound traffic.

Hopefully someone with some actual nftables knowledge can weigh in here - this is outside my area.

I installed VyOS 1.4 and have tried to figure out how to use nftables to allow gre ipsec packets and block non ipsec protected gre but I can’t seem to figure it out. nftables logic does not seem to work like iptables.

OK, here is what I finally came up with to block clear text gre outbound using nftables on VyOS 1.4. This seems to work in my virtual lab. Normal VyOS firewall rules can be used to allow ipsec/gre inbound and block all other gre clear text inbound.

# Allow GRE outbound if attached to ipsec policy and block all other clear text GRE
nft add table ip custom
nft add chain ip custom output '{ type filter hook output priority -10 ; }'
nft add rule ip custom output ip protocol gre ipsec out reqid * counter accept
nft add rule ip custom output ip protocol gre counter drop

I am not sure if matching ipsec reqid with a wildcard is really proper but it seems to work (though it looks weird in nft list output). Another option is to match any reqid that is not 0 (i.e. anything above 0). E.g.

# Allow GRE outbound if attached to ipsec policy and block all other clear text GRE
nft add table ip custom
nft add chain ip custom output '{ type filter hook output priority -10 ; }'
nft add rule ip custom output ip protocol gre ipsec out reqid != 0 counter accept
nft add rule ip custom output ip protocol gre counter drop

This also seems to work in my virtual lab.

1 Like

Ok, so I’ve been messing around with VyOS some more, and I came across zone policy. From my testing, it looks like I can block traffic outbound from the firewall itself using zone policy. I have created a firewall that looks like this

name LOCAL-WAN {
    default-action "accept"
    rule 5 {
        action "drop"
        protocol "gre"
    }
}

and applied it to my WAN zone from traffic coming from my LOCAL zone. This does solve the problem of preventing cleartext gre outbound, but also has the unintended effect of preventing the IPSec tunnel from actually coming up. At least, I think that is what is happening. This is what I get when I do a tcpdump on the “WAN” bridge

07:11:41.740912 IP 192.168.7.101.isakmp > 192.168.7.100.isakmp: isakmp: phase 2/others I inf[E]
07:11:41.741697 IP 192.168.7.101.isakmp > 192.168.7.100.isakmp: isakmp: phase 1 I ident
07:11:41.742137 IP 192.168.7.100.isakmp > 192.168.7.101.isakmp: isakmp: phase 2/others R inf[E]
07:11:41.742965 IP 192.168.7.100.isakmp > 192.168.7.101.isakmp: isakmp: phase 1 R ident
07:11:41.745304 IP 192.168.7.101.isakmp > 192.168.7.100.isakmp: isakmp: phase 1 I ident
07:11:41.748848 IP 192.168.7.100.isakmp > 192.168.7.101.isakmp: isakmp: phase 1 R ident
07:11:41.751049 IP 192.168.7.101.isakmp > 192.168.7.100.isakmp: isakmp: phase 1 I ident[E]
07:11:41.751977 IP 192.168.7.100.isakmp > 192.168.7.101.isakmp: isakmp: phase 1 R ident[E]
07:11:41.754859 IP 192.168.7.101.isakmp > 192.168.7.100.isakmp: isakmp: phase 2/others I oakley-quick[E]
07:11:41.760351 IP 192.168.7.100.isakmp > 192.168.7.101.isakmp: isakmp: phase 2/others R oakley-quick[E]
07:11:41.763177 IP 192.168.7.101.isakmp > 192.168.7.100.isakmp: isakmp: phase 2/others I oakley-quick[E]

Then a second or two later

07:11:42.713122 IP 192.168.7.100.isakmp > 192.168.7.101.isakmp: isakmp: phase 2/others R inf[E]

192.168.7.100 is the hub while 192.168.7.101 is the spoke


I think I understand the first two and last lines of your nftables rules, but I don’t quite understand the third one. My understanding of IPSec is that the payload is completely encrypted, so the firewall shouldn’t be able to match a gre packet inside of an IPSec packet, no? I don’t understand the reqid *, reqid != 0, and counter bits at all. I briefly looked up reqid as it pertains to nftables and IPSec, but couldn’t find much information. Could you point me towards a resource that would explain it?

Is it possible to make a rule that includes the same parameters using the VyOS firewall?

I ask because I’d much prefer to have the rules as part of the VyOS config itself. That is partly for simpler administration and partly to minimize risk of accidentally not moving the startup script file during a migration or upgrade - especially since there’s no warning or failsafe if the file doesn’t run and apply the firewall “patch”)

Thank you for your help!

See this link for info on how iptables/nftables works with ipsec. Basically, meta data is attached to packets as it is processed and this includes ipsec data for packets that are to be encrypted by ipsec. This ipsec meta data is attached to the packet before it is actually encrypted, and it is this meta data you match on. So in my example I am matching a gre packet that also has ipsec policy attached (based on reqid) and then I drop all other gre without ipsec meta data. The linked article also mentions that you can match based on peer src/dst IP but I found this does not work, I am guessing this is because with DMVPN I am using ipsec transport mode (which is normal for DMVPN) so there wouldn’t be IPSec peer IP information, thus I used reqid to match.

Nice find on being able to use firewall zone policies to match outbound local traffic. Based on the above info above, rule 5 would need to be action “accept” and match would be “protocol gre” AND “ipsec match-ipsec” and then you would need a following rule that blocks all other gre. The problem with this is, it doesn’t work in nftables for output hooks (it works for input though). When using “ipsec match-ipsec” VyOS is trying to configure the nftables rule with “meta ipsec exists” which, for whatever reason, is not allowed on output. You have to use the ipsec syntax I mentioned in my last post.

Not being able to match ipsec traffic on outbound traffic is unfortunate. There’s some kind of firewall rework going on - I’ll ask if this is something that can be fixed as a part of that.


To anyone coming across this with the same problem, you should know that p252’s nftables rules need to go in /config/scripts/vyos-preconfig-bootup.script. This is different from vyos-postconfig-bootup.script, which is created automatically. If you put the rules in the postconfig file, NHRP will start before the script is run and the firewall rules are applied. This will result in at least your NHRP registration request packet being leaked in the cleartext, which contains your cisco-authentication password.

Putting the rules in the preconfig script applies the rules before anything else starts, and successfully prevents any cleartext GRE from escaping once services start coming up. Also, you need to escape the * with a \ when putting the rules into a file - otherwise it will try to insert file names from the working directory into the command.