Site to site IPSec, problem with traffic flow

General questions
,
1

Hi, try to setup site to site VPN between two sites. IPSec is established, but cannot get traffic over the VPN. Not sure how the routing works on VYOS and how to force traffic over the tunnel/IPSec. Added some basic info and config under, someone that see my issue?

Setup is with two sites as described:

Site A
VYOS → Firewall → ISP/Internet
VYOS OUTSIDE IP: 10.10.1.18
Firewall INSIDE IP: 10.10.1.1
Firewall OUTSIDE PublicIP: 193.10.10.10
Internal Network: 10.10.1.0/24

Site B
Palo Alto Firewall → ISP/Internet
PublicIP: vpn.siteb.com
Internal Network: 10.77.0.0/16
Tunnel IP: 192.168.99.2

Config from Site A VYOS:

firewall {
    all-ping enable
}
interfaces {
    ethernet eth0 {
        address 10.10.1.18/24
        description OUTSIDE
    }
    ethernet eth1 {
        address 10.10.1.19/24
        description INSIDE
    }
    tunnel tun0 {
        address 192.168.99.1/29
        encapsulation ipip
        local-ip 10.10.1.18
        mtu 1400
        remote-ip 193.10.10.10
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 10.10.1.1 {
            }
        }
    }
}
vpn {
    ipsec {
        esp-group pa-esp {
            compression disable
            lifetime 3600
            mode tunnel
            pfs dh-group14
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        ike-group pa-ike {
            close-action none
            ikev2-reauth no
            key-exchange ikev2
            lifetime 86400
            proposal 1 {
                dh-group 14
                encryption aes256
                hash sha256
            }
        }
        ipsec-interfaces {
            interface eth0
        }
        nat-traversal enable
        site-to-site {
            peer vpn.siteb.com {
                authentication {
                    id 193.10.10.10
                    mode pre-shared-secret
                    pre-shared-secret ****************
                    remote-id vpn.siteb.com
                }
                connection-type initiate
                ike-group pa-ike
                ikev2-reauth inherit
                local-address 10.10.1.18
                tunnel 0 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group pa-esp
                    local {
                        prefix 10.10.1.0/24
                    }
                    remote {
                        prefix 10.77.0.0/16
                    }
                }
            }
        }
    }
}
2

If your IPSec tunnel is up and working you have to add a static route for your remote network 10.77.0.0/16 with tun0 as destination interface, e.g.:

set protocols static route 10.77.0.0/16 interface tun0

to tell your VyOS router which networks should be routed through the IPSec tunnel tun0.

3

Added it but still no traffic going over VPN. Interface tun0 was something i just added for a test, is it correctly configured?

This it the routing i have now:

S>* 0.0.0.0/0 [1/0] via 10.10.1.1, eth0, weight 1, 16:49:08
C * 10.10.1.0/24 is directly connected, eth1, 16:49:12
C>* 10.10.1.0/24 is directly connected, eth0, 16:49:12
S>* 10.77.0.0/16 [1/0] is directly connected, tun0, weight 1, 00:00:37
C>* 192.168.99.0/29 is directly connected, tun0, 16:49:11
4

Have you taken a look into the docs? There are some examples about how to configure a S2S VPN.

Does this help?

5

Changed from tunnel to VTI interface, still same issue. Followed the articale before also, but how the routing is working with IPSec is not clear for me. What i understand is that use of tunnel as my first config should fix the routing automatic and just work.Can be that VTI is better way to setup IPSec, but still i cant get traffic over the VPN.

interfaces {
    ethernet eth0 {
        address 10.10.1.18/24
        description OUTSIDE
    }
    ethernet eth1 {
        address 10.10.1.19/24
        description INSIDE
    }
    loopback lo {
    }
    vti vti1 {
        address 192.168.99.1/29
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 10.10.1.1 {
            }
        }
        route 10.77.0.0/16 {
            interface vti1 {
            }
        }
    }
}
vpn {
    ipsec {
        authentication {
            psk peer_vpn-siteb-com {
                id 193.10.10.10
                id vpn.siteb.com
                id 10.0.1.18
                secret ****************
            }
        }
        esp-group pa-esp {
            lifetime 3600
            mode tunnel
            pfs dh-group14
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        ike-group pa-ike {
            close-action none
            key-exchange ikev2
            lifetime 86400
            proposal 1 {
                dh-group 14
                encryption aes256
                hash sha256
            }
        }
        interface eth0
        options {
            disable-route-autoinstall
        }
        site-to-site {
            peer peer_vpn-stotvigveien-no {
                authentication {
                    local-id 193.10.10.10
                    mode pre-shared-secret
                    remote-id vpn.siteb.com
                }
                connection-type initiate
                ike-group pa-ike
                local-address 10.10.1.18
                remote-address vpn.siteb.com
                vti {
                    bind vti1
                    esp-group pa-esp
                }
            }
        }
    }
}

Route table looks like this:

S>* 0.0.0.0/0 [1/0] via 10.10.1.1, eth0, weight 1, 00:24:52
C * 10.10.1.0/24 is directly connected, eth1, 00:24:56
C>* 10.10.1.0/24 is directly connected, eth0, 00:24:56
S>* 10.77.0.0/16 [1/0] is directly connected, vti1, weight 1, 00:24:52
C>* 192.168.99.0/29 is directly connected, vti1, 00:24:55
6

The local address and local tunnel prefix belong to the same subnet, is the correct gateway defined on end hosts ?
Please share the output of this command
$ show vpn ipsec sa
$ show version
and also check the charon logs related to phase II:
$ show log vpn

You can also troubleshoot by taking packet capture at the ipsec interface eth0 to check for the incoming packets.
On which platform have you installed the VyOS ? and can you describe your topology a bit.

7

Topology is like this: VYOS → Cisco FW → Internet → PaloAlto SiteB
VYOS running on VMware and behind Cisco FW with NAT.

Check output under for the commands.

show vpn ipsec sa
Connection                    State    Uptime    Bytes In/Out    Packets In/Out    Remote address    Remote ID            Proposal
----------------------------  -------  --------  --------------  ----------------  ----------------  -------------------  ------------------------
peer_vpn-stotvigveien-no-vti  up       23m57s    0B/13K          0/170             85.166.175.154    vpn.stotvigveien.no  AES_CBC_256/HMAC_SHA1_96

show version
Version:          VyOS 1.4-rolling-202308240020
Release train:    current

Built by:         [email protected]
Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform

show log vpn (85.0.0.0 is siteB public IP)
Aug 28 13:01:29 charon[2280]: 06[NET] <peer_vpn-siteb-com|1> sending packet: from 10.10.1.18[500] to 85.0.0.0[500] (80 bytes)
Aug 28 13:01:29 charon-systemd[2280]: sending packet: from 10.10.1.18[500] to 85.0.0.0[500] (80 bytes)
Aug 28 13:01:34 charon[2280]: 16[NET] <peer_vpn-siteb-com|1> received packet: from 85.0.0.0[500] to 10.10.1.18[500] (80 bytes)
Aug 28 13:01:34 charon-systemd[2280]: received packet: from 85.0.0.0[500] to 10.10.1.18[500] (80 bytes)
Aug 28 13:01:34 charon[2280]: 16[ENC] <peer_vpn-siteb-com|1> parsed INFORMATIONAL request 311 [ ]
Aug 28 13:01:34 charon-systemd[2280]: parsed INFORMATIONAL request 311 [ ]
Aug 28 13:01:34 charon[2280]: 16[ENC] <peer_vpn-siteb-com|1> generating INFORMATIONAL response 311 [ ]
Aug 28 13:01:34 charon-systemd[2280]: generating INFORMATIONAL response 311 [ ]
Aug 28 13:01:34 charon[2280]: 16[NET] <peer_vpn-siteb-com|1> sending packet: from 10.10.1.18[500] to 85.0.0.0[500] (80 bytes)
Aug 28 13:01:34 charon-systemd[2280]: sending packet: from 10.10.1.18[500] to 85.0.0.0[500] (80 bytes)
8

According to you latest entry, tunnel is up and you can see outoing traffic though the tunnel, but no data is comming back. See counters in cero for Bytes In and Packets In.
Check if traffic is being received on remote end, and why is not comming back.

9

Traffic is not received on SiteB. Also SiteB has same numbers with packets out and no packet in.
Tried both way to ping the tunnel it self, but it not work. (192.168.99.0/29) This should work regardless of routing issue?

Checked the traffic flow with monitor traffic interface vti1 and can see that traffic hits the interface when i try a ping to SiteB. I never see packets when i send ping FROM SiteB. In logs of SiteB i can see that traffic sent over tunnel interface

monitor traffic interface vti1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vti1, link-type RAW (Raw IP), snapshot length 262144 bytes

16:23:05.454165 IP 192.168.99.1 > 192.168.99.2: ICMP echo request, id 41729, seq 1, length 64
16:23:06.517630 IP 192.168.99.1 > 192.168.99.2: ICMP echo request, id 41729, seq 2, length 64
16:23:07.541602 IP 192.168.99.1 > 192.168.99.2: ICMP echo request, id 41729, seq 3, length 64
16:23:08.565619 IP 192.168.99.1 > 192.168.99.2: ICMP echo request, id 41729, seq 4, length 64
16:23:09.589628 IP 192.168.99.1 > 192.168.99.2: ICMP echo request, id 41729, seq 5, length 64
16:23:10.613619 IP 192.168.99.1 > 192.168.99.2: ICMP echo request, id 41729, seq 6, length 64
16:23:16.476492 IP 192.168.99.1 > 10.77.0.1: ICMP echo request, id 17144, seq 1, length 64
16:23:17.525646 IP 192.168.99.1 > 10.77.0.1: ICMP echo request, id 17144, seq 2, length 64
10

I was also struggling a lot when I set up my IPSec VPN. Maybe the following bash shell commands help you to get more details about what’s happening when the IPSec session will be established:

swanctl --list-conns

swanctl --initiate --ike <CONNECTION_NAME>

swanctl --log
11

What does that Cisco FW tell you in its logs when it comes to this vpn-tunnel, does that even see and allow the traffic?

Do you have UDP encapsulation enabled?

That is UDP500 will be used for keyexchange and UDP4500 will be used for the traffic itself to pass through NAT etc.

Other option is to use wireguard instead if the destination supports that (will probably bring better performance aswell).

12

Use tcpdump to see if ESP packets are sent…and received on remote end
I have encountered situation, where ESP wasn’t passed by ISP
Forcing NAT-T (udp 4500) is a way around this