Working (almost) vpn ipsec site to site, both sides behind NAT, not working in 1.2.0

Hi All,

Based on a Vyos’s blog post by Daniil Baturin, I’ve accomplished this configuration for interconnecting one head office (H1) and two other branch offices (B1, B2) on Vyos 1.1.8. The almost in the subject means that at H1, whenever the Vyos box is restarted, the vpn only starts working after entering “restart vpn”.

So, I’ve build an ISO based on Crux branch, tested it without success, used a rolling release from the day before yesterday, also with no luck.

The first thing I’ve found out is that version 1.2.0 doesn’t like rsa-keys generated with previous versions, it simply doesn’t load the vpn section from the config file. After regenerating new rsa-keys at all sides and exchanged them, the vpn simply doesn’t get up with no logs whatsoever, at least that I can find them.

So, if you could take a look at the following trimmed config files for H1 and B2 and give some tips, heads up, improvement, misconception, hell something, I would be high appreciated.

H1

interfaces {
    dummy dum0 {
        address xxx.xxx.254.1/32
    }
    dummy dum1 {
        address xxx.xxx.254.9/32
    }
    ethernet eth0 {
        address xxx.xxx.3.246/22
        duplex auto
        hw-id XX:XX:XX:c1:9d:c9
        smp_affinity auto
        speed auto
    }
    loopback lo {
    }
    tunnel tun0 {
        address xxx.xxx.0.1/29
        encapsulation gre
        local-ip xxx.xxx.254.1
        multicast disable
        remote-ip xxx.xxx.254.6
    }
    tunnel tun1 {
        address xxx.xxx.0.9/29
        encapsulation gre
        local-ip xxx.xxx.254.9
        multicast disable
        remote-ip xxx.xxx.254.14
    }
}
protocols {
    ospf {
        area 0 {
            network xxx.xxx.0.0/29
            network xxx.xxx.35.0/24
            network xxx.xxx.0.0/22
        }
        area 1 {
            network xxx.xxx.0.8/29
            network xxx.xxx.97.0/24
        }
        log-adjacency-changes {
        }
        parameters {
            abr-type cisco
            router-id xxx.xxx.254.1
        }
        redistribute {
            static {
                metric-type 2
            }
        }
    }
}

...

vpn {
    ipsec {
        esp-group e-default {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes256
                hash sha256
            }
        }
        ike-group i-default {
            dead-peer-detection {
                action restart
                interval 30
                timeout 60
            }
            ikev2-reauth no
            key-exchange ikev1
            lifetime 28800
            proposal 1 {
                dh-group 14
                encryption aes256
                hash sha256
            }
        }
        ipsec-interfaces {
            interface dum0
            interface dum1
        }
        nat-traversal enable
        site-to-site {
            peer @b1 {
                authentication {
                    id h1
                    mode rsa
                    rsa-key-name b1
                }
                connection-type respond
                default-esp-group e-default
                ike-group i-default
                ikev2-reauth inherit
                local-address any
                tunnel 2 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    local {
                        prefix xxx.xxx.254.9/32
                    }
                    remote {
                        prefix xxx.xxx.254.14/32
                    }
                }
            }
            peer @b2 {
                authentication {
                    id h1
                    mode rsa
                    rsa-key-name b2
                }
                connection-type respond
                default-esp-group e-default
                ike-group i-default
                ikev2-reauth inherit
                local-address any
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    local {
                        prefix xxx.xxx.254.1/32
                    }
                    remote {
                        prefix xxx.xxx.254.6/32
                    }
                }
            }
        }
    }
    rsa-keys {
        rsa-key-name b1 {
        }
        rsa-key-name b2 {
        }
    }
}

B2

interfaces {
    dummy dum0 {
        address xxx.xxx.254.6/32
    }
    ethernet eth0 {
        address xxx.xxx.97.247/24
        duplex auto
        hw-id XX:XX:XX:af:d5:cf
        smp_affinity auto
        speed auto
    }
    loopback lo {
    }
    tunnel tun0 {
        address xxx.xxx.0.6/29
        encapsulation gre
        local-ip xxx.xxx.254.6
        multicast disable
        remote-ip xxx.xxx.254.1
    }
}
protocols {
    ospf {
        area 0 {
            network xxx.xxx.97.0/24
            network xxx.xxx.0.0/29
        }
        log-adjacency-changes {
        }
        parameters {
            abr-type cisco
            router-id xxx.xxx.254.6
        }
        redistribute {
            static {
                metric-type 2
            }
        }
    }
}
...
vpn {
    ipsec {
        esp-group e-default {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes256
                hash sha256
            }
        }
        ike-group i-default {
            dead-peer-detection {
                action restart
                interval 30
                timeout 60
            }
            ikev2-reauth no
            key-exchange ikev1
            lifetime 28800
            proposal 1 {
                dh-group 14
                encryption aes256
                hash sha256
            }
        }
        ipsec-interfaces {
            interface dum0
        }
        nat-traversal enable
        site-to-site {
            peer xxxxx.tld {
                authentication {
                    id @b2
                    mode rsa
                    remote-id h1
                    rsa-key-name h1
                }
                connection-type initiate
                default-esp-group e-default
                ike-group i-default
                ikev2-reauth inherit
                local-address any
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    local {
                        prefix xxx.xxx.254.6/32
                    }
                    remote {
                        prefix xxx.xxx.254.1/32
                    }
                }
            }
        }
    }
    rsa-keys {
        rsa-key-name h1 {
        }
    }
}

Cheers,

Joao

sudo swanctl --log
sudo swanctl --log --pretty

Open two shells to your router and have one of them running this.

With both sides behind NAT you may need xxxxx.tld to be resolvable in DNS or you may see some error about not matching any keys. You might have this but I couldn’t tell from your post or config.

Your peer config @b2 on H1 does not have a remote-id specified.

IKEv1 not so good at recovering, so here is a good script, you can save this as /config/scripts/vpncheck.sh


#!/bin/bash

/bin/ping -c 3 -W 2 10.3.2.1 1>/dev/null 2>&1

if [ $? -ne 0 ]; then
echo “Failed”
/bin/vbash -ic ‘restart vpn’
fi


Make sure you chmod it to make it executable and set it to run with:

task-scheduler {
    task vpncheck {
        executable {
            path /config/scripts/vpncheck.sh

Hi chief_dan,

Thank you very much for your time and support.

The “xxxxx.tld” is actually a public IP, the script that anonymized the config changed it to this.

Is this what’s missing in H1 config?

set vpn ipsec site-to-site peer @b2 authentication remote-id @b2

As at both sides are VyOS routers, what should I change to use IKEv2 instead of IKEv1? I’m not a vpn expert and I’ve researched here and at Ubiquiti forums to get some clues, but couldn’t arrive at a conclusion.

Thanks for the script! I’ll use it if I cannot setup an IKEv2 config.

Cheers,

João

To change key exchange:

set vpn ipsec ike-group i-default key-exchange ‘ikev2’

If both ends are VyOS it works easy, but I’ve had challenges getting IKEv2 working on VyOS with other router or firewall vendors (PAN, Cisco, Ubiquiti). Some have mentioned DPD not working at all: ⚓ T349 Does the IKEv1 Dead-peer-detection work?

Yes, that remote-id is missing. I’ve seen the “@” used in examples but I’m not sure if that is just a name or performing a function.

I had perceived both sides were behind NAT, if you have the correct public IP in the peer config you may be ok without DNS. My use case was a private network to Azure and both ends were dynamic public IP and behind NAT.

Use the swanctl logging to see what the error is if any, that is going to be the definitive source for why it won’t come up.

OK, I’ll try that tomorrow and I’ll get back for feedback.

Once again, thank you!

No way :frowning:

Using “sudo swanctl --log --pretty” at B2 I can observer the initiator trying sending packets.

Using “sudo swanctl --log --pretty” at H1 it shows nothing.

Using tcpdump at H1 I can see packets arriving:

vyos@vpn2-h1:~$ sudo tcpdump -i eth0 port 500
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:42:34.885504 IP 197.xx.xxx.94.isakmp > 192.xx.xxx.246.isakmp: isakmp: parent_sa ikev2_init[I]
10:42:47.845843 IP 197.xx.xxx.94.isakmp > 192.xx.xxx.246.isakmp: isakmp: parent_sa ikev2_init[I]

Any hint?

What drives me crazy i that this configuration works in VyOS 1.1.8

Cheers,

João

Not sure.

Can you post some output from the swanctl log? run restart vpn while logging on both sides? show vpn debug before and after?

Is interesting traffic being generated? I would think just having the gre tunnel would create some.

@chief_dan

When you don’t know how to walk in high heels, someone has to give some hand. :slight_smile: Just kidding

Just to context: besides the dummy interfaces, which I believe I’m using them correctly to route traffic, I only have one (1) real interface per VyOS box, either physical or virtual, “eth0”.

“show vpn debug” give the click, the ipsec daemon was only listening on dum0 and dum1, not on ETH0.

Once I added “set vpn ipsec ipsec-interfaces interface eth0” on both sides and commit, the magic started showing.

@chief_dan You’re the man!

Just one last multiple question: based on your experience, does the config I builded with the help of enumerous contributors looks good? Anything you would do it different/better/more robust?

Kind regards,

João

That explains why the strongswan did not react to the ipsec traffic, and you might not need the ipsec interfaces on your dummies.

I usually only use one real interface as most of mine are VMs.

Do you have OSPF across that tunnel? Does the multicast disable impact that?

I don’t know about “more robust”. Are you using firewall? Do you have SSH locked down to only internal addresses or trusted IP space? Is there a use case for B1 traffic to reach B2 through H1?

OSPF somehow seems to be working. As this isn’t in production yet, it will replace an OpenVPN solution, also behind NAT and also based on VyOS, I cannot really sure if it’s properly working (some lack of resources in the far end to test it properly).
Does the multicast disable impact that?” Don’t understand your question, sorry :frowning:

Yes, FW is in place and SSH is only reachable from management workstations.

I do hope that OSPF will do some magic and I can reach B2 from B1 and vice versa.

My questions are related to that I wasn’t sure if this setup was indeed protecting my traffic.

OSPF typically communicates over multicast, but since your tunnel endpoints are virtually in the same broadcast domain that can be why it is working.

If your routing table shows the B1 networks as reachable through the tunnel, and the tunnel only establishes through the IPSEC policy, I’d be reasonably confident the traffic is encrypted. A packet capture on an upstream device could reinforce that assumption.

Dan,

I’ll try to find a way to capture some traffic and analyze it just to make sure.

In the meanwhile, I’ll adjust B1, updating and adjusting the configuration to see if in the end all works as I expected.

Once again, thank you very much for your time, help and support.

Hope this config helps others trying to achieve what I needed.

Best regards,

João