ARP fails globally for upstream hosts when add site-to-site with VTI peer

Morning!

I have a challenge that may be a bug.
When adding IPSEC site-to-site with VTI interfaces to our Vyos.
ARPs to directly connected hosts are not responded to by Vyos once the IPSEC Peer is committed.
Deleting the PEER block restores service.

We can replicate this at will and have confirmed with TCPDUMP ARP requests are normal with the IPSEC config committed / deleted - but no ARP responses are returned from VYOS when IPSEC config is applied.

Our Vyos version is:

Version: VyOS 1.4-rolling-202103011828
Release Train: sagitta
Built by: autobuild@vyos.net
Built on: Mon 01 Mar 2021 18:28 UTC
Build UUID: e1e1673c-2e53-490c-9f9a-c3d422cf5ea7
Build Commit ID: 39ab069e421c6e
Architecture: x86_64
Boot via: installed image
System type: KVM guest

Our starting point / existing working solution without IPSEC.
We have VYOS deployed in a KVM connecting to our provider network on v.801, we take public IPs and create 1to1 NAT / firewall zones / to other Guest virtual machines across many vLans, all directly connected. This works well, we can ingress to guests from the internet on allowed ports, and guests can egress to the internet.

By way of example the attached image

A starting / anonymized config:

set firewall all-ping ‘enable’
set firewall broadcast-ping ‘disable’
set firewall config-trap ‘disable’
set firewall group address-group outside-v2174-ad-10 address ‘10.128.15.110’
set firewall group address-group outside-v2174-ad-10 description ‘hosts allowed to be hit on port 22’
set firewall group address-group v2174-outside-ad-10 address ‘10.128.15.110’
set firewall group port-group outside-v2174-pr-10 description ‘Description for port group’
set firewall group port-group outside-v2174-pr-10 port ‘22’
set firewall ipv6-receive-redirects ‘disable’
set firewall ipv6-src-route ‘disable’
set firewall ip-src-route ‘disable’
set firewall log-martians ‘enable’
set firewall name outside-local default-action ‘drop’
set firewall name outside-local rule 1 action ‘accept’
set firewall name outside-local rule 1 state established ‘enable’
set firewall name outside-local rule 1 state related ‘enable’
set firewall name outside-local rule 2 action ‘drop’
set firewall name outside-local rule 2 log ‘enable’
set firewall name outside-local rule 2 state invalid ‘enable’
set firewall name outside-local rule 10 action ‘accept’
set firewall name outside-local rule 10 destination port ‘22’
set firewall name outside-local rule 10 protocol ‘tcp’
set firewall name outside-local rule 10 state new ‘enable’
set firewall name outside-local rule 20 action ‘accept’
set firewall name outside-local rule 20 icmp type-name ‘echo-reply’
set firewall name outside-local rule 20 protocol ‘icmp’
set firewall name outside-local rule 20 state new ‘enable’
set firewall name outside-v2174 default-action ‘drop’
set firewall name outside-v2174 rule 1 action ‘accept’
set firewall name outside-v2174 rule 1 state established ‘enable’
set firewall name outside-v2174 rule 1 state related ‘enable’
set firewall name outside-v2174 rule 2 action ‘drop’
set firewall name outside-v2174 rule 2 log ‘enable’
set firewall name outside-v2174 rule 2 state invalid ‘enable’
set firewall name outside-v2174 rule 10 action ‘accept’
set firewall name outside-v2174 rule 10 description ‘SSH’
set firewall name outside-v2174 rule 10 destination group address-group ‘outside-v2174-ad-10’
set firewall name outside-v2174 rule 10 destination group port-group ‘outside-v2174-pr-10’
set firewall name outside-v2174 rule 10 protocol ‘tcp’
set firewall name outside-v2174 rule 10 source address ‘0.0.0.0/0’
set firewall name outside-v2187 default-action ‘drop’
set firewall name outside-v2187 rule 1 action ‘accept’
set firewall name outside-v2187 rule 1 state established ‘enable’
set firewall name outside-v2187 rule 1 state related ‘enable’
set firewall name outside-v2187 rule 2 action ‘drop’
set firewall name outside-v2187 rule 2 log ‘enable’
set firewall name outside-v2187 rule 2 state invalid ‘enable’
set firewall name v2174-outside default-action ‘drop’
set firewall name v2174-outside rule 1 action ‘accept’
set firewall name v2174-outside rule 1 state established ‘enable’
set firewall name v2174-outside rule 1 state related ‘enable’
set firewall name v2174-outside rule 2 action ‘drop’
set firewall name v2174-outside rule 2 log ‘enable’
set firewall name v2174-outside rule 2 state invalid ‘enable’
set firewall name v2174-outside rule 10 action ‘accept’
set firewall name v2174-outside rule 10 description ‘ANY’
set firewall name v2174-outside rule 10 destination address ‘0.0.0.0/0’
set firewall name v2174-outside rule 10 source group address-group ‘v2174-outside-ad-10’
set firewall name v2187-outside default-action ‘drop’
set firewall name v2187-outside rule 1 action ‘accept’
set firewall name v2187-outside rule 1 state established ‘enable’
set firewall name v2187-outside rule 1 state related ‘enable’
set firewall name v2187-outside rule 2 action ‘drop’
set firewall name v2187-outside rule 2 log ‘enable’
set firewall name v2187-outside rule 2 state invalid ‘enable’
set firewall receive-redirects ‘disable’
set firewall send-redirects ‘enable’
set firewall source-validation ‘disable’
set firewall syn-cookies ‘enable’
set firewall twa-hazards-protection ‘disable’
set interfaces ethernet eth0 firewall in name ‘outside-local’
set interfaces ethernet eth0 hw-id ‘50:6b:8d:d8:bd:8c’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.252/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.240/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.241/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.242/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.243/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.244/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.245/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.246/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.247/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.248/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.249/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.250/24’
set interfaces ethernet eth0 vif 801 address ‘1.1.1.251/24’
set interfaces ethernet eth0 vif 801 description ‘vLAN801 - outside’
set interfaces ethernet eth0 vif 2174 address ‘10.128.15.1/24’
set interfaces ethernet eth0 vif 2174 description ‘Customer uuid: c001 name: company1’
set interfaces ethernet eth0 vif 2187 address ‘10.128.28.1/24’
set interfaces ethernet eth0 vif 2187 description ‘Customer uuid: c001 name: company1’
set interfaces loopback lo
set nat destination rule 21742 description ‘1-to-1 NAT - 2174 - 64a07112-98bf-45ac-aee7-be2c1a5dccb0 - ingress’
set nat destination rule 21742 destination address ‘1.1.1.248’
set nat destination rule 21742 inbound-interface ‘eth0.801’
set nat destination rule 21742 translation address ‘10.128.15.110’
set nat destination rule 21743 description ‘NAT Reflection: INSIDE - 2174 - 64a07112-98bf-45ac-aee7-be2c1a5dccb0’
set nat destination rule 21743 destination address ‘1.1.1.248’
set nat destination rule 21743 inbound-interface ‘eth0.2174’
set nat destination rule 21743 translation address ‘10.128.15.110’
set nat destination rule 21872 description ‘1-to-1 NAT - 2187 - ab84139d-2636-48de-acf5-68939de10232 - ingress’
set nat destination rule 21872 destination address ‘1.1.1.245’
set nat destination rule 21872 inbound-interface ‘eth0.801’
set nat destination rule 21872 translation address ‘10.128.28.11’
set nat destination rule 21873 description ‘NAT Reflection: INSIDE - 2187 - ab84139d-2636-48de-acf5-68939de10232’
set nat destination rule 21873 destination address ‘1.1.1.245’
set nat destination rule 21873 inbound-interface ‘eth0.2187’
set nat destination rule 21873 translation address ‘10.128.28.11’
set nat source rule 2174 description ‘pass traffic to the Internet for vlan 2174’
set nat source rule 2174 outbound-interface ‘eth0.801’
set nat source rule 2174 source address ‘10.128.15.0/24’
set nat source rule 2174 translation address ‘masquerade’
set nat source rule 2187 description ‘pass traffic to the Internet for vlan 2187’
set nat source rule 2187 outbound-interface ‘eth0.801’
set nat source rule 2187 source address ‘10.128.28.0/24’
set nat source rule 2187 translation address ‘masquerade’
set nat source rule 21742 outbound-interface ‘eth0.801’
set nat source rule 21742 source address ‘10.128.15.110’
set nat source rule 21742 translation address ‘1.1.1.248’
set nat source rule 21743 description ‘NAT Reflection: INSIDE - 2174 - 64a07112-98bf-45ac-aee7-be2c1a5dccb0’
set nat source rule 21743 destination address ‘10.128.15.0/24’
set nat source rule 21743 outbound-interface ‘eth0.2174’
set nat source rule 21743 source address ‘10.128.15.0/24’
set nat source rule 21743 translation address ‘masquerade’
set nat source rule 21748 description ‘1-to-1 NAT - 2174 - 8a34c0a8-d861-43a8-a294-b71a7c634595 - ergess’
set nat source rule 21748 outbound-interface ‘eth0.801’
set nat source rule 21748 source address ‘10.128.15.85’
set nat source rule 21748 translation address ‘1.1.1.247’
set nat source rule 21749 description ‘NAT Reflection: INSIDE - 2174 - 8a34c0a8-d861-43a8-a294-b71a7c634595’
set nat source rule 21749 destination address ‘10.128.15.0/24’
set nat source rule 21749 outbound-interface ‘eth0.2174’
set nat source rule 21749 source address ‘10.128.15.0/24’
set nat source rule 21749 translation address ‘masquerade’
set nat source rule 21872 description ‘1-to-1 NAT - 2187 - ab84139d-2636-48de-acf5-68939de10232 - ergess’
set nat source rule 21872 outbound-interface ‘eth0.801’
set nat source rule 21872 source address ‘10.128.28.11’
set nat source rule 21872 translation address ‘1.1.1.245’
set nat source rule 21873 description ‘NAT Reflection: INSIDE - 2187 - ab84139d-2636-48de-acf5-68939de10232’
set nat source rule 21873 destination address ‘10.128.28.0/24’
set nat source rule 21873 outbound-interface ‘eth0.2187’
set nat source rule 21873 source address ‘10.128.28.0/24’
set nat source rule 21873 translation address ‘masquerade’
set protocols static route 0.0.0.0/0 next-hop 1.1.1.254
set service lldp
set service ssh disable-password-authentication
set service ssh port ‘22’
set system config-management commit-revisions ‘1000’
set system console device ttyS0 speed ‘115200’
set zone-policy zone outside default-action ‘drop’
set zone-policy zone outside from v2174 firewall name ‘v2174-outside’
set zone-policy zone outside from v2187 firewall name ‘v2187-outside’
set zone-policy zone outside interface ‘eth0.801’
set zone-policy zone v2174 from outside firewall name ‘outside-v2174’
set zone-policy zone v2174 interface ‘eth0.2174’
set zone-policy zone v2176 default-action ‘drop’
set zone-policy zone v2176 from outside firewall name ‘outside-v2176’
set zone-policy zone v2176 interface ‘eth0.2176’
set zone-policy zone v2187 default-action ‘drop’
set zone-policy zone v2187 from outside firewall name ‘outside-v2187’
set zone-policy zone v2187 interface 'eth0.2187

When apply the following IPSEC config VYOS no longer responds to ARP requests from any host on any the private network vLANS.

ESP Proposal

set vpn ipsec esp-group remote_site_vpn compression ‘disable’
set vpn ipsec esp-group remote_site_vpn lifetime 3600
set vpn ipsec esp-group remote_site_vpn mode ‘tunnel’
set vpn ipsec esp-group remote_site_vpn pfs dh-group2
set vpn ipsec esp-group remote_site_vpn proposal 1 encryption aes128
set vpn ipsec esp-group remote_site_vpn proposal 1 hash sha1

IKE Profile

set vpn ipsec ike-group remote_site_vpn dead-peer-detection action ‘restart’
set vpn ipsec ike-group remote_site_vpn dead-peer-detection interval ‘15’
set vpn ipsec ike-group remote_site_vpn dead-peer-detection timeout ‘30’
set vpn ipsec ike-group remote_site_vpn ikev2-reauth ‘yes’
set vpn ipsec ike-group remote_site_vpn key-exchange ikev2
set vpn ipsec ike-group remote_site_vpn lifetime 28800
set vpn ipsec ike-group remote_site_vpn proposal 1 dh-group 2
set vpn ipsec ike-group remote_site_vpn proposal 1 encryption aes128
set vpn ipsec ike-group remote_site_vpn proposal 1 hash sha1

Tunnel Interface

set interfaces vti vti1 address 169.254.142.233/30
set interfaces vti vti1 description ‘remote_site_vpn Tunnel Interface’

MSS to manage encapsulation overhead

set firewall options interface vti1 adjust-mss 1350

IPSEC on Customer interface

set vpn ipsec ipsec-interfaces interface eth0.801

Configure Tunnel

set vpn ipsec site-to-site peer 2.2.2.186 authentication id 1.1.1.245
set vpn ipsec site-to-site peer 2.2.2.186 authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer 2.2.2.186 authentication pre-shared-secret apretendsecret
set vpn ipsec site-to-site peer 2.2.2.186 authentication remote-id 2.2.2.186
set vpn ipsec site-to-site peer 2.2.2.186 connection-type respond
set vpn ipsec site-to-site peer 2.2.2.186 description ‘remote_site_vpn PRIMARY TUNNEL’
set vpn ipsec site-to-site peer 2.2.2.186 ike-group ‘remote_site_vpn’
set vpn ipsec site-to-site peer 2.2.2.186 ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer 2.2.2.186 local-address 1.1.1.245
set vpn ipsec site-to-site peer 2.2.2.186 vti bind vti1
set vpn ipsec site-to-site peer 2.2.2.186 vti esp-group remote_site_vpn

Configure Routes

set protocols static interface-route 192.168.88.0/24 interface vti1

NAT Exemption

set nat source rule 10 destination address 192.168.88.0/24
set nat source rule 10 ‘exclude’
set nat source rule 10 outbound-interface eth0.801
set nat source rule 10 source address 10.128.28.0/24
set nat source rule 20 destination address 10.128.28.0/24
set nat source rule 20 ‘exclude’
set nat source rule 20 outbound-interface eth0.2187
set nat source rule 20 source address 192.168.88.0/24
set nat source rule 30 destination address 10.128.28.0/24
set nat source rule 30 ‘exclude’
set nat source rule 30 outbound-interface eth0.801
set nat source rule 30 source address 192.168.88.0/24

From setting/deleting/committing sections of the VPN config with loop clearing the arp-cache and pinging we know its when commit the following PEER section that ARP fails. On deleting this section ARP replies normally.

Configure Tunnel

set vpn ipsec site-to-site peer 2.2.2.186 authentication id 1.1.1.245
set vpn ipsec site-to-site peer 2.2.2.186 authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer 2.2.2.186 authentication pre-shared-secret apretendsecret
set vpn ipsec site-to-site peer 2.2.2.186 authentication remote-id 2.2.2.186
set vpn ipsec site-to-site peer 2.2.2.186 connection-type respond
set vpn ipsec site-to-site peer 2.2.2.186 description ‘remote_site_vpn PRIMARY TUNNEL’
set vpn ipsec site-to-site peer 2.2.2.186 ike-group ‘remote_site_vpn’
set vpn ipsec site-to-site peer 2.2.2.186 ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer 2.2.2.186 local-address 1.1.1.245
set vpn ipsec site-to-site peer 2.2.2.186 vti bind vti1
set vpn ipsec site-to-site peer 2.2.2.186 vti esp-group remote_site_vpn

Can you test it with the latest 1.4 rolling image? Ipsec has changed a lot since the build you’re currently running.

1 Like

Hello @sarthurdev , did you try to add disable-route-autoinstall option?
set vpn ipsec options disable-route-autoinstall

1 Like

Thanks for the responses / suggestions.
I will try these and return with findings.

i have same issue ! ipsec up ,router can not ping direct vlan gateway !

Hello @crazycen,
As @Dmitry suggested:
did you try to add disable-route-autoinstall option?
set vpn ipsec options disable-route-autoinstall

Can you also provide the config?
show configuration commands | strip-private