VyOS VTI Static Route Tunnel issue

[josue.mouco]

VyOS version: VyOS 1.4-rolling-202205250217

Hello,

I setup a VyOS router and an ipsec tunnel between VyOS and my Firewall Palo Alto which works, the tunnel is up.

From my LAN I can ping the VyOS router but I cannot ping local clients.

I setup the routes on both side, VyOS and Palo.

It worked for weeks and now suddenly it doesn’t work anymore.

My VyOS config is as follows:

set interfaces ethernet eth0 address ‘dhcp’
set interfaces ethernet eth0 description ‘OUTSIDE’
set interfaces ethernet eth0 hw-id ‘xx:xx:xx:xx:xx:d4’
set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth1 address ‘xxx.xxx.0.1/24’
set interfaces ethernet eth1 description ‘INSIDE’
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces loopback lo
set interfaces vti vti0 address ‘xxx.xxx.0.254/24’
set interfaces vti vti0 description ‘pa-tunnel-address’
set protocols static route xxx.xxx.0.0/0 interface eth0
set protocols static route xxx.xxx.0.0/16 interface vti0
set protocols static route xxx.xxx.0.0/16 interface vti0
set protocols static route xxx.xxx.0.0/16 interface vti0
set protocols static route xxx.xxx.0.0/20 interface vti0
set protocols static route xxx.xxx.3.0/24 interface vti0
set service ssh client-keepalive-interval ‘180’
set service ssh port ‘22’
set system config-management commit-revisions ‘100’
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication public-keys [email protected] key xxxxxx
set system login user xxxxxx authentication public-keys [email protected] type ssh-xxx
set system name-server ‘eth0’
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system syslog global facility all level ‘notice’
set system syslog global facility protocols level ‘debug’
set system time-zone ‘Europe/Zurich’
set vpn ipsec esp-group vyos-pa-esp compression ‘disable’
set vpn ipsec esp-group vyos-pa-esp lifetime ‘3600’
set vpn ipsec esp-group vyos-pa-esp mode ‘tunnel’
set vpn ipsec esp-group vyos-pa-esp pfs ‘dh-group14’
set vpn ipsec esp-group vyos-pa-esp proposal 1 encryption ‘aes256gcm128’
set vpn ipsec esp-group vyos-pa-esp proposal 1 hash ‘sha256’
set vpn ipsec ike-group vyos-pa-ike close-action ‘none’
set vpn ipsec ike-group vyos-pa-ike dead-peer-detection action ‘clear’
set vpn ipsec ike-group vyos-pa-ike dead-peer-detection interval ‘30’
set vpn ipsec ike-group vyos-pa-ike dead-peer-detection timeout ‘90’
set vpn ipsec ike-group vyos-pa-ike ikev2-reauth ‘no’
set vpn ipsec ike-group vyos-pa-ike key-exchange ‘ikev2’
set vpn ipsec ike-group vyos-pa-ike lifetime ‘86400’
set vpn ipsec ike-group vyos-pa-ike proposal 1 dh-group ‘14’
set vpn ipsec ike-group vyos-pa-ike proposal 1 encryption ‘aes256’
set vpn ipsec ike-group vyos-pa-ike proposal 1 hash ‘sha256’
set vpn ipsec interface ‘eth0’
set vpn ipsec log level ‘2’
set vpn ipsec log subsystem ‘any’
set vpn ipsec options disable-route-autoinstall
set vpn ipsec site-to-site peer xxxxx.tld authentication id ‘xxx.xxx.160.66’
set vpn ipsec site-to-site peer xxxxx.tld authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer xxxxx.tld authentication pre-shared-secret xxxxxx
set vpn ipsec site-to-site peer xxxxx.tld authentication remote-id ‘xxx.xxx.10.66’
set vpn ipsec site-to-site peer xxxxx.tld connection-type ‘initiate’
set vpn ipsec site-to-site peer xxxxx.tld default-esp-group ‘vyos-pa-esp’
set vpn ipsec site-to-site peer xxxxx.tld ike-group ‘vyos-pa-ike’
set vpn ipsec site-to-site peer xxxxx.tld ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer xxxxx.tld local-address ‘xxx.xxx.160.66’
set vpn ipsec site-to-site peer xxxxx.tld vti bind ‘vti0’
set vpn ipsec site-to-site peer xxxxx.tld vti esp-group ‘vyos-pa-esp’

vyos@vpn-endpoint:~$ show ip route
S>* 0.0.0.0/0 [1/0] is directly connected, eth0, weight 1, 00:01:56
S 0.0.0.0/0 [210/0] via 85.217.160.1, eth0, weight 1, 00:01:58
S>* x.x.0.0/16 [1/0] is directly connected, vti0, weight 1, 00:01:56
S>* x.x.0.0/16 [1/0] is directly connected, vti0, weight 1, 00:01:56
C * x.x.0.0/24 is directly connected, vti0, 00:01:59
C>* x.x.0.0/24 is directly connected, eth1, 00:02:00
S>* x.x.0.0/16 [1/0] is directly connected, vti0, weight 1, 00:01:56
S>* x.x.0.0/20 [1/0] is directly connected, vti0, weight 1, 00:01:56
C>* x.x.x.x/23 is directly connected, eth0, 00:01:59
S>* x.x.x.0/24 [1/0] is directly connected, vti0, weight 1, 00:01:56

Palo Alto side:

        <entry name="Tunnel-Exoscale">
          <auto-key>
            <ike-gateway>
              <entry name="IKE_Gateway-Exoscale"/>
            </ike-gateway>
            <ipsec-crypto-profile>IPSec_Crypto-Exoscale-Phase2</ipsec-crypto-profile>
            <proxy-id>
              <entry name="xxx">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.x.0/24</local>
                <remote>x.x.0.254</remote>
              </entry>
              <entry name="xxx">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/20</local>
                <remote>x.x.0.254</remote>
              </entry>
              <entry name="xxx">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/16</local>
                <remote>x.x.0.254</remote>
              </entry>
              <entry name="xxx">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/16</local>
                <remote>x.x.0.254</remote>
              </entry>
              <entry name="xxx">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/16</local>
                <remote>x.x.0.254</remote>
              </entry>
            </proxy-id>
          </auto-key>
          <tunnel-monitor>
            <enable>no</enable>
          </tunnel-monitor>
          <tunnel-interface>tunnel.1</tunnel-interface>
          <disabled>no</disabled>
        </entry>
      </ipsec>

Do you know what is wrong?

Thank you for your help,

[e.khudiyev]

Hi @josue.mouco , on the Palo side within the policy for the remote networks you provided the host address:

<remote>x.x.0.254</remote>

Try to replace it with INSIDE subnet of your VyOS ‘xxx.xxx.0.0/24’ and see if that helps.

[josue.mouco]

Hi @e.khudiyev ,

I modified as suggested

Now, I can ping the local eth1 interface of the VyOS router which was not working before.

But, I still can’t ping clients on both LAN sides.

[e.khudiyev]

Where do you check the pings? Do you mean client to client connecivity?

[josue.mouco]

I try to ping from 2 different Windows clients inside 2 different networks included in the routes, to different clients inside the VyOS Router network. x.x.0.0/24, like x.x.0.1 or 0.16, etc.

Ping x.x.0.254 (VyOS router VTI address) from x.x.x.0/24 (Windows client) works
Ping x.x.0.1 (VyOS router ETH1 address) from x.x.x.0/24 (Windows client) works
Ping x.x.0.16 (VyOS client) from x.x.x.0/24 (Windows client) doesn’t work: “Reply from x.x.0.254: Destination host unreachable.”

[e.khudiyev]

how does the static route looks like on your Palo? check if VyOS can ping the Windows host and it has arp entry for that host in its arp table. Also Try to create additional rules for remote x.x.0.254 and same local nets (keep previous change as it is) if windows host ia reachable from the VyOS vti0 interface.

[josue.mouco]

Hello,
The static route on Palo side is like this:

        <entry name="Tunnel-Exoscale">
          <auto-key>
            <ike-gateway>
              <entry name="IKE_Gateway-Exoscale"/>
            </ike-gateway>
            <ipsec-crypto-profile>IPSec_Crypto-Exoscale-Phase2</ipsec-crypto-profile>
            <proxy-id>
              <entry name="VPN">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.x.0/24</local>
                <remote>x.x.0.0/24</remote>
              </entry>
              <entry name="ISG-Intranet">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/20</local>
                <remote>x.x.0.0/24</remote>
              </entry>
              <entry name="FDN-Servers">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/16</local>
                <remote>x.x.0.0/24</remote>
              </entry>
              <entry name="FDN-DMZ">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/16</local>
                <remote>x.x.0.0/24</remote>
              </entry>
              <entry name="LGB-Admin">
                <protocol>
                  <any/>
                </protocol>
                <local>x.x.0.0/16</local>
                <remote>x.x.0.0/24</remote>
              </entry>
            </proxy-id>
          </auto-key>

VyOS cannot ping Windows hosts and I don’t see IPs in arp table other than exoscale LAN IPs:

Palo static route side after changes:

VyOS static route side after changes:

[e.khudiyev]

@josue.mouco remove the next-hop IP address from VyOS please. Also, try to ping from the hos behind VyOS to the host behind the Palo to see if that connectivity works.

[josue.mouco]

@e.khudiyev : ok I removed the next-hop ip addresses.
I tried to ping from a host behind VyOS to a host behind the Palo without success:

[e.khudiyev]

what is the source address of the VM you try to reach? It seems to be wrong routing here. Check traceroute and post the brief topology with the devices and network for a better understanding of your connections and traffic flow…

[josue.mouco]

I tried to reach 10.5.0.21 from my LAN:

The traceroute shows that the ping access the Exoscale side from my LAN:

[e.khudiyev]

Does your VyOS has 10.5.0.1/24 on eth1 and 10.5.0.254/24 on vti0 interface? If so, change the vti0 address (as it shouldn’t be from the same network as your LAN) to any other IP address (for example 10.10.10.1/30) and check. As you do so, remove the VPN policy on Palo including that .254 address as it’s not necessary and check host to host connectivity again. Traceroute must pass 10.5.0.1 address and then reach the host.

[josue.mouco]

Yes you are right, it’s the same VLAN. I changed vti0 to:
vti vti0 {
address 10.5.255.254/30
description pa-tunnel-address
}

I removed from Palo the proxy IDs for static route:

I try to ping host to host but without success, from my LAN:

from host behind VyOS:

[e.khudiyev]

Okay, maybe I didn’t clearly explain what is required So here is the last step you need to complete in order to get things work - bring back policy rules on your Palo that are containing the following:

1. Local x.x.x.x/24 networks behind Palo
2. Remote 10.5.0.0/24 network on VyOS

As you totally removed them all from Palo, all packets are dropped.

[josue.mouco]

OK I put back settings on Palo side:

It still doesn’t work.

Is that setting correct on Palo side?

[e.khudiyev]

On your screenshot, 10.5.255.254/32 doesn’t have the outgoing interface. Anyway, this shouldn’t affect the transit traffic between local and remote subnets. Please check step 7 here to ensure that the incoming/outgoing traffic is allowed under the Palo firewall

[josue.mouco]

Palo policies are correct: traffic is allowed as in the example.

I also added the route:

[16again]

Never touched Palo myself…
But from the topic start, I raise eyebrows on Palo config. It resembles policy based ipsec VPN , not route based: On all Palo tunnels, I see local/remote subnet declarations, whereas this should be 0.0.0.0/0 <-> 0.0.0.0/0 for route based (VTI) ipsec tunnel. Use that as proxy-id on Palo

Also, look into ipsec logging, something like:
sudo swanctl --log

[josue.mouco]

@16again sudo swanctl --log shows:

IPs are both WAN IP.

[e.khudiyev]

@josue.mouco as per the previous post by @16again , remove all proxy IDs and add 0.0.0.0/0 for both remote and local subnets. Unfortunately, it has been a long time since I’m not working with Palo and I assumed that the VyOS route-based will work with Palo policy-based… but it seems that they have different logic. You also may see logs on Palo that there is a proxy ID mismatch and Phase-2 basically won’t come up.