VTI is hung

Hi All!

I try to use VyOS router for realizing IPsec vpn backup scheme.

Vyos-1 connected to VyOS-3 across two links, eth1 with ip 10.1.3.1/24 and eth3 with ip 10.2.4.1/24.
Vyos-3 connected to VyOS-1 across two links, eth1 with ip 10.1.3.3/24 and eth3 with ip 10.2.4.3/24.

I have two IPsec connection via VTI binding. First connestion used eth1-eth1 and vti0 interfaces, second connection used eth3-eth3 and vti1 interfaces.

For backup I use static interface route with different distances.

Here is my configuration:

vyos@VyOS-1# show interfaces { ethernet eth0 { address xxx description Inet duplex auto hw-id 00:50:56:01:00:4e smp_affinity auto speed auto } ethernet eth1 { address 10.1.3.1/24 description "Link to VyOS-3" duplex auto hw-id 00:50:56:01:00:4f smp_affinity auto speed auto } ethernet eth3 { address 10.2.4.1/24 hw-id 00:50:56:01:00:59 } loopback lo { } vti vti0 { address 10.10.0.1/30 } vti vti1 { address 10.10.1.1/30 } } protocols { static { interface-route 192.168.34.0/24 { next-hop-interface vti0 { distance 10 } next-hop-interface vti1 { distance 20 } } route 0.0.0.0/0 { next-hop 185.96.86.17 { } } } } service { ssh { port 22 } } vpn { ipsec { auto-update 30 esp-group test1-esp { compression disable lifetime 30 mode tunnel pfs enable proposal 1 { encryption aes256 hash sha1 } } ike-group test1-ike { dead-peer-detection { action restart interval 30 timeout 90 } ikev2-reauth no key-exchange ikev1 lifetime 30 proposal 1 { encryption aes256 hash sha1 } } ipsec-interfaces { interface eth1 interface eth3 } site-to-site { peer 10.1.3.3 { authentication { mode pre-shared-secret pre-shared-secret Test1Test2Test3 } connection-type initiate ike-group test1-ike ikev2-reauth inherit local-address 10.1.3.1 vti { bind vti0 esp-group test1-esp } } peer 10.2.4.3 { authentication { mode pre-shared-secret pre-shared-secret Test1Test2Test3 } connection-type initiate ike-group test1-ike ikev2-reauth inherit local-address 10.2.4.1 vti { bind vti1 esp-group test1-esp } } } } }

Analog config is on VyOS-3. IPsec connections is established and work well, traffic goes accross vti0, then I disable eth1 for testing backup mechanism traffic goes to vti1, via second ipsec connection. But then I also enable eth1, link goes up, but traffic doesn’t go via first ipsec connection. The cause of it is vti0 is Administrativly Down state, and I can fix it only using “restart vpn” or disable and enable vti0 via config using commit.

[code]vyos@VyOS-1# run sh int vti
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description


vti0 10.10.0.1/30 A/D
vti1 10.10.1.1/30 u/u

[edit]
vyos@VyOS-1# run res
reset restart

[edit]
vyos@VyOS-1# run restart vpn
Restarting IPsec process…

[edit]
vyos@VyOS-1# run sh int vti
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description


vti0 10.10.0.1/30 u/u
vti1 10.10.1.1/30 u/u [/code]

I think it is some bug with vti hunging. Do somebody have any idea about this issue? Or may be someone know another variant of doing IPsec failover backup with two peers?

Since you are using VTI, you can run OSPF on top of the tunnels. It works very well.

Note that you need to tune MTU on the VTI interfaces and perform MSS clamping as well.

Ok, try to use OSPF, but it doesn’t resolve the problem. VTI is hung after IPsec is fel. My diagnostic info is below:

[code]vyos@VyOS-1# run sh vpn ipsec sa
Peer ID / IP Local ID / IP


10.1.3.3 10.1.3.1

Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
------  -----  -------------  -------  ----    -----  ------  ------  -----
vti     up     68.0/68.0      aes256   sha1    no     18      30      all

Peer ID / IP Local ID / IP


10.2.4.3 10.2.4.1

Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
------  -----  -------------  -------  ----    -----  ------  ------  -----
vti     up     0.0/0.0        aes256   sha1    no     26      30      all

vyos@VyOS-1# set interfaces ethernet eth1 disable
[edit]
vyos@VyOS-1# commit

vyos@VyOS-1# run sh vpn ipsec sa
Peer ID / IP Local ID / IP


10.1.3.3 10.1.3.1

Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
------  -----  -------------  -------  ----    -----  ------  ------  -----
vti     down   n/a            n/a      n/a     no     0       30      all

Peer ID / IP Local ID / IP


10.2.4.3 10.2.4.1

Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
------  -----  -------------  -------  ----    -----  ------  ------  -----
vti     up     300.0/300.0    aes256   sha1    no     26      30      all

vyos@VyOS-1# delete int ethernet eth1 disable
[edit]
vyos@VyOS-1# commit
[edit]
vyos@VyOS-1# run ping 10.1.3.3 count 5 interval 0.2
PING 10.1.3.3 (10.1.3.3) 56(84) bytes of data.
64 bytes from 10.1.3.3: icmp_req=1 ttl=64 time=0.430 ms
64 bytes from 10.1.3.3: icmp_req=2 ttl=64 time=0.342 ms
64 bytes from 10.1.3.3: icmp_req=3 ttl=64 time=0.350 ms
64 bytes from 10.1.3.3: icmp_req=4 ttl=64 time=0.357 ms
64 bytes from 10.1.3.3: icmp_req=5 ttl=64 time=0.462 ms

— 10.1.3.3 ping statistics —
5 packets transmitted, 5 received, 0% packet loss, time 799ms
rtt min/avg/max/mdev = 0.342/0.388/0.462/0.050 ms

vyos@VyOS-1# run sh int vti
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description


vti0 10.10.0.1/30 A/D
vti1 10.10.1.1/30 u/u

vyos@VyOS-1# run restart vpn
Restarting IPsec process…
[edit]
vyos@VyOS-1# run sh int vti
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description


vti0 10.10.0.1/30 u/u
vti1 10.10.1.1/30 u/u

[/code]

As you can see, VTI is hung, and only one opportunity to fix - to do “restart vpn”.

I have a Vyos with 2 VTI tunnels active, and ran into similar problems.
Ended up in using a cron job , repeating after 5 minutes, testing reachability of both vti neighbors, and restarting vpn when both are down.

I’ve got similar problem. Between VyOS 1.2.1 and EdgeRouter IPsec VPN Tunnel with VTI. VTI Interface on VyOS site goes down ( A/ D). After reset vpn ipsec-peer it comes backup up…

For those, who will meet with this problem: currently, there is no reliable way to (quickly) track VTI interface state and IPSec SA in VyOS without some tricks.
The tested workaround is:

  • make an event-handler for quick catching IPSec disconnection;
  • check IPSec states periodically.
    And run the VTI restart script using these two triggers.