Facing issue with BGP, any idea what could be the issue?

General questions
,
1

Hi Team,

My vyos version is 1.2. and have 2 ISP links configured on Vyos. I have two vti tunnels configured with Azure with BGP configured. What I observed is if one of the link flaps my entire BGP stops and I lose connectivity.

I started BGP logging - and I noticed below errors. Any clue?

Aug 14 20:09:16 xx-xxxx bgpd[1212]: 169.254.21.1 [Event] Connect failed 110(Connection timed out)
Aug 14 20:09:16 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] TCP_connection_open_failed (Connect->Active), fd 27
Aug 14 20:09:16 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0
Aug 14 20:09:16 xx-xxxx bgpd[1212]: 169.254.21.1 went from Connect to Active
Aug 14 20:09:48 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] BGP_Stop (Active->Idle), fd -1
Aug 14 20:09:48 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Idle established_peers 0
Aug 14 20:09:48 xx-xxxx bgpd[1212]: 169.254.21.1 went from Active to Idle
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (start timer expire).
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] BGP_Start (Idle->Connect), fd -1
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Waiting for NHT
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Connect established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 went from Idle to Connect
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] TCP_connection_open_failed (Connect->Active), fd -1
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 went from Connect to Active
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] ConnectRetry_timer_expired (Active->Connect), fd -1
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [Event] Connect start to 169.254.21.1 fd 27
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Non blocking connect waiting result, fd 27
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Connect established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 went from Active to Connect
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] TCP_connection_open (Connect->OpenSent), fd 27
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 open active, local address 169.254.21.13
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 Sending hostname cap with hn = xx-xxxx, dn = (null)
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 sending OPEN, version 4, my as 65506, holdtime 30, id 111.125.226.237
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: OpenSent established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 went from Connect to OpenSent
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] ConnectRetry_timer_expired (Active->Connect), fd -1
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [Event] Connect start to 169.254.22.1 fd 30
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Non blocking connect waiting result, fd 30
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Connect established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 went from Active to Connect
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] TCP_connection_open (Connect->OpenSent), fd 30
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 open active, local address 169.254.21.9
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 Sending hostname cap with hn = xx-xxxx, dn = (null)
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 sending OPEN, version 4, my as 65506, holdtime 30, id 111.125.226.237
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: OpenSent established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 went from Connect to OpenSent
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 rcv OPEN, version 4, remote-as (in open) 65515, holdtime 180, id 10.11.44.132
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 rcv OPEN w/ OPTION parameter len: 34
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 rcvd OPEN w/ optional parameter type 2 (Capability) len 32
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has MultiProtocol Extensions capability (1), length 4
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has MP_EXT CAP for afi/safi: IPv4/unicast
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has MultiProtocol Extensions capability (1), length 4
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has MP_EXT CAP for afi/safi: IPv6/unicast
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has Route Refresh capability (2), length 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has 4-octet AS number capability (65), length 4
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has Graceful Restart capability (64), length 10
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 OPEN has Graceful Restart capability
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 Peer has not restarted. Restart Time : 120
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 Address family IPv4 Unicast is not preserved
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 Addr-family IPv6/unicast(afi/safi) not enabled. Ignore the Graceful Restart capability
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm), fd 27
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: OpenConfirm established_peers 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 went from OpenSent to OpenConfirm
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established), fd 27
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Established established_peers 1
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 went from OpenConfirm to Established
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (routeadv timer expire)
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 rcv OPEN, version 4, remote-as (in open) 65515, holdtime 180, id 10.11.44.133
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 rcv OPEN w/ OPTION parameter len: 34
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 rcvd OPEN w/ optional parameter type 2 (Capability) len 32
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has MultiProtocol Extensions capability (1), length 4
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has MP_EXT CAP for afi/safi: IPv4/unicast
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has MultiProtocol Extensions capability (1), length 4
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has MP_EXT CAP for afi/safi: IPv6/unicast
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has Route Refresh capability (2), length 0
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has 4-octet AS number capability (65), length 4
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has Graceful Restart capability (64), length 10
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 OPEN has Graceful Restart capability
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 Peer has not restarted. Restart Time : 120
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 Address family IPv4 Unicast is not preserved
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 Addr-family IPv6/unicast(afi/safi) not enabled. Ignore the Graceful Restart capability
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm), fd 30
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: OpenConfirm established_peers 1
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 went from OpenSent to OpenConfirm
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Receive_KEEPALIVE_message (OpenConfirm->Established), fd 30
Aug 14 20:09:50 xx-xxxx bgpd[1212]: bgp_fsm_change_status : vrf default(0), Status: Established established_peers 2
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 went from OpenConfirm to Established
Aug 14 20:09:50 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Timer (routeadv timer expire)
Aug 14 20:09:51 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:09:51 xx-xxxx bgpd[1212]: send End-of-RIB for IPv4 Unicast to 169.254.21.1
Aug 14 20:09:51 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (routeadv timer expire)
Aug 14 20:09:51 xx-xxxx bgpd[1212]: send End-of-RIB for IPv4 Unicast to 169.254.22.1
Aug 14 20:09:51 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Timer (routeadv timer expire)
Aug 14 20:09:55 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:09:59 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:00 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:00 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:03 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:07 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:10 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:10 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:12 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:16 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:20 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:20 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:20 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:24 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:29 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:30 xx-xxxx bgpd[1212]: 169.254.21.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:30 xx-xxxx bgpd[1212]: 169.254.22.1 [FSM] Timer (keepalive timer expire)
Aug 14 20:10:33 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
Aug 14 20:10:38 xx-xxxx bgpd[1212]: [Event] 10.11.44.132 connection rejected - not configured and not valid for dynamic
2

I think a little more info is needed, like your interfaces, IPs and routing table.

Long shot, both BGP sessions are running over the same tunnel or something like that.

3

Here is config

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall options interface vti2 adjust-mss '1350'
set firewall options interface vti4 adjust-mss '1350'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 address 'xxx.xxx.97.110/29'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 firewall local name 'blockssh'
set interfaces ethernet eth0 hw-id 'XX:XX:XX:XX:XX:e0'
set interfaces ethernet eth0 smp-affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address 'xxx.xxx.226.237/28'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 firewall local name 'blockssh'
set interfaces ethernet eth1 hw-id 'XX:XX:XX:XX:XX:e1'
set interfaces ethernet eth1 smp-affinity 'auto'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 hw-id 'XX:XX:XX:XX:XX:e2'
set interfaces ethernet eth2 smp-affinity 'auto'
set interfaces ethernet eth2 speed 'auto'
set interfaces ethernet eth3 duplex 'auto'
set interfaces ethernet eth3 hw-id 'XX:XX:XX:XX:XX:e3'
set interfaces ethernet eth3 smp-affinity 'auto'
set interfaces ethernet eth3 speed 'auto'
set interfaces ethernet eth4 duplex 'auto'
set interfaces ethernet eth4 hw-id 'XX:XX:XX:XX:XX:e4'
set interfaces ethernet eth4 smp-affinity 'auto'
set interfaces ethernet eth4 speed 'auto'
set interfaces ethernet eth5 duplex 'auto'
set interfaces ethernet eth5 hw-id 'XX:XX:XX:XX:XX:e5'
set interfaces ethernet eth5 smp-affinity 'auto'
set interfaces ethernet eth5 speed 'auto'
set interfaces ethernet eth6 duplex 'auto'
set interfaces ethernet eth6 hw-id 'XX:XX:XX:XX:XX:e6'
set interfaces ethernet eth6 smp-affinity 'auto'
set interfaces ethernet eth6 speed 'auto'
set interfaces ethernet eth7 address 'xxx.xxx.144.17/28'
set interfaces ethernet eth7 duplex 'auto'
set interfaces ethernet eth7 hw-id 'XX:XX:XX:XX:XX:e7'
set interfaces ethernet eth7 smp-affinity 'auto'
set interfaces ethernet eth7 speed 'auto'
set interfaces loopback lo
set interfaces vti vti2 address 'xxx.xxx.21.9/30'
set interfaces vti vti2 description 'Azure WAI Vodfone Tunnel'
set interfaces vti vti4 address 'xxx.xxx.21.13/30'
set interfaces vti vti4 description 'Azure WAI Inspire Tunnel'
set policy prefix-list accept-only-s4 rule 2 action 'permit'
set policy prefix-list accept-only-s4 rule 2 prefix 'xxx.xxx.11.0/24'
set policy prefix-list accept-only-s4 rule 3 action 'permit'
set policy prefix-list accept-only-s4 rule 3 prefix 'xxx.xxx.10.0/24'
set policy prefix-list low-pref-vodafone rule 2 action 'permit'
set policy prefix-list low-pref-vodafone rule 2 prefix 'xxx.xxx.44.0/22'
set policy route-map accept-only-s4 rule 2 action 'deny'
set policy route-map accept-only-s4 rule 2 description 'Accept only S4 route and deny GBL subnets'
set policy route-map accept-only-s4 rule 2 match ip address prefix-list 'accept-only-s4'
set policy route-map accept-only-s4 rule 4 action 'permit'
set policy route-map as-path-vodafone description 'Give Higher pref to Vodfone'
set policy route-map as-path-vodafone rule 5 action 'permit'
set policy route-map as-path-vodafone rule 5 set as-path-prepend '65515'
set policy route-map low-pref-vodafone rule 2 action 'permit'
set policy route-map low-pref-vodafone rule 2 match ip address prefix-list 'low-pref-vodafone'
set policy route-map low-pref-vodafone rule 2 set as-path-prepend '65515 65515 65515'
set protocols bgp XXXXXX address-family ipv4-unicast network xxx.xxx.40.0/23
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 address-family ipv4-unicast route-map import 'accept-only-s4'
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 address-family ipv4-unicast weight '100'
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 disable-connected-check
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 remote-as '65515'
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 timers holdtime '30'
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 timers keepalive '15'
set protocols bgp XXXXXX neighbor xxx.xxx.21.1 update-source 'xxx.xxx.21.13'
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 address-family ipv4-unicast route-map export 'low-pref-vodafone'
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 disable-connected-check
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 remote-as '65515'
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 timers holdtime '30'
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 timers keepalive '15'
set protocols bgp XXXXXX neighbor xxx.xxx.22.1 update-source 'xxx.xxx.21.9'
set protocols static interface-route xxx.xxx.21.1/32 next-hop-interface vti4
set protocols static interface-route xxx.xxx.22.1/32 next-hop-interface vti2
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.226.225
set protocols static route xxx.xxx.153.181/32 next-hop xxx.xxx.97.105
set protocols static route xxx.xxx.43.36/32 next-hop xxx.xxx.97.105
set protocols static route xxx.xxx.10.0/24 next-hop xxx.xxx.144.18
set protocols static route xxx.xxx.11.0/24 next-hop xxx.xxx.144.18
set protocols static route xxx.xxx.40.0/23 next-hop xxx.xxx.144.18
set vpn ipsec auto-update '60'
set vpn ipsec esp-group WXX-AZURE compression 'disable'
set vpn ipsec esp-group WXX-AZURE lifetime '3600'
set vpn ipsec esp-group WXX-AZURE mode 'tunnel'
set vpn ipsec esp-group WXX-AZURE proposal 1 encryption 'aes256'
set vpn ipsec esp-group WXX-AZURE proposal 1 hash 'sha1'
set vpn ipsec ike-group WXX-AZURE dead-peer-detection action 'clear'
set vpn ipsec ike-group WXX-AZURE dead-peer-detection interval '15'
set vpn ipsec ike-group WXX-AZURE dead-peer-detection timeout '30'
set vpn ipsec ike-group WXX-AZURE ikev2-reauth 'yes'
set vpn ipsec ike-group WXX-AZURE key-exchange 'ikev2'
set vpn ipsec ike-group WXX-AZURE lifetime '28800'
set vpn ipsec ike-group WXX-AZURE proposal 1 dh-group '2'
set vpn ipsec ike-group WXX-AZURE proposal 1 encryption 'aes256'
set vpn ipsec ike-group WXX-AZURE proposal 1 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec ipsec-interfaces interface 'eth1'
set vpn ipsec site-to-site peer xxxxx.tld authentication id 'xxx.xxx.226.237'
set vpn ipsec site-to-site peer xxxxx.tld authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer xxxxx.tld authentication pre-shared-secret xxxxxx
set vpn ipsec site-to-site peer xxxxx.tld authentication remote-id 'xxx.xxx.16.236'
set vpn ipsec site-to-site peer xxxxx.tld connection-type 'initiate'
set vpn ipsec site-to-site peer xxxxx.tld description 'WAI Azure SAP Inspire Tunnel'
set vpn ipsec site-to-site peer xxxxx.tld ike-group 'WXX-AZURE'
set vpn ipsec site-to-site peer xxxxx.tld ikev2-reauth 'inherit'
set vpn ipsec site-to-site peer xxxxx.tld local-address 'xxx.xxx.226.237'
set vpn ipsec site-to-site peer xxxxx.tld vti bind 'vti4'
set vpn ipsec site-to-site peer xxxxx.tld vti esp-group 'WXX-AZURE'
set vpn ipsec site-to-site peer xxxxx.tld authentication id 'xxx.xxx.97.110'
set vpn ipsec site-to-site peer xxxxx.tld authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer xxxxx.tld authentication pre-shared-secret xxxxxx
set vpn ipsec site-to-site peer xxxxx.tld authentication remote-id 'xxx.xxx.153.181'
set vpn ipsec site-to-site peer xxxxx.tld connection-type 'initiate'
set vpn ipsec site-to-site peer xxxxx.tld description 'WAI Azure SAP Vodafone Tunnel'
set vpn ipsec site-to-site peer xxxxx.tld ike-group 'WXX-AZURE'
set vpn ipsec site-to-site peer xxxxx.tld ikev2-reauth 'inherit'
set vpn ipsec site-to-site peer xxxxx.tld local-address 'xxx.xxx.97.110'
set vpn ipsec site-to-site peer xxxxx.tld vti bind 'vti2'
set vpn ipsec site-to-site peer xxxxx.tld vti esp-group 'WXX-AZURE'

And nope tunnels are configured on separate links

show ip bgp
BGP table version is 6426, local router ID is 111.125.226.237, vrf id 0
Default local pref 100, local AS 65506
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*  10.1.0.0/16      169.254.22.1                           0 65515 i
*>                  169.254.21.1                         100 65515 i
*  10.11.44.0/22    169.254.22.1                           0 65515 i
*>                  169.254.21.1                         100 65515 i
*> 192.168.10.0/24  169.254.22.1                           0 65515 65505 i
*> 192.168.11.0/24  169.254.22.1                           0 65515 i
*> 192.168.40.0/23  0.0.0.0                  0         32768 i

Displayed  5 routes and 7 total paths
4

When it is working, what is the output of:-

show ip bgp neighbors 169.254.21.1
show ip bgp neighbors 169.254.22.1

and does it give any clues?

5

Nah - not much; Do you notice anything?

show ip bgp neighbors 169.254.21.1
BGP neighbor is 169.254.21.1, remote AS 65515, local AS 65506, external link
  BGP version 4, remote router ID 10.11.44.132, local router ID 111.125.226.237
  BGP state = Established, up for 00:28:05
  Last read 00:00:00, Last write 00:00:05
  Hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised IPv4 Unicast
    Route refresh: advertised and received(new)
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: received
    Hostname Capability: advertised (name: isn-mum-vpnx-wai,domain name: n/a) not received
    Graceful Restart Capabilty: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        IPv4 Unicast(not preserved)
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received:
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                551        422
    Notifications:        160        126
    Updates:             4014       1269
    Keepalives:       1124235    1281455
    Route Refresh:          6         19
    Capability:             0          0
    Total:            1128966    1283291
  Minimum time between advertisement runs is 0 seconds
  Update source is 169.254.21.13

 For address family: IPv4 Unicast
  Update group 1324, subgroup 1409
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  Inbound path policy configured
  Route map for incoming advertisements is *accept-only-s4
  2 accepted prefixes

  Connections established 418; dropped 417
  Last reset 06:26:17,  Waiting for peer OPEN
Local host: 169.254.21.13, Local port: 40661
Foreign host: 169.254.21.1, Foreign port: 179
Nexthop: 169.254.21.13
Nexthop global: fe80::200:5efe:6f7d:e2ed
Nexthop local: fe80::200:5efe:6f7d:e2ed
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 15 ms
Read thread: on  Write thread: on  FD used: 30

 show ip bgp neighbors 169.254.22.1
BGP neighbor is 169.254.22.1, remote AS 65515, local AS 65506, external link
  BGP version 4, remote router ID 10.11.44.133, local router ID 111.125.226.237
  BGP state = Established, up for 00:28:10
  Last read 00:00:05, Last write 00:00:10
  Hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised IPv4 Unicast
    Route refresh: advertised and received(new)
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: received
    Hostname Capability: advertised (name: isn-mum-vpnx-wai,domain name: n/a) not received
    Graceful Restart Capabilty: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        IPv4 Unicast(not preserved)
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received:
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:               1189       1016
    Notifications:       1258         30
    Updates:             2543       3564
    Keepalives:       1091644    1242044
    Route Refresh:          6         18
    Capability:             0          0
    Total:            1096640    1246672
  Minimum time between advertisement runs is 0 seconds
  Update source is 169.254.21.9

 For address family: IPv4 Unicast
  Update group 1323, subgroup 1408
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  Outbound path policy configured
  Route map for outgoing advertisements is *low-pref-vodafone
  4 accepted prefixes

  Connections established 1009; dropped 1008
  Last reset 06:26:45,  Waiting for peer OPEN
Local host: 169.254.21.9, Local port: 42473
Foreign host: 169.254.22.1, Foreign port: 179
Nexthop: 169.254.21.9
Nexthop global: fe80::200:5efe:2a68:616e
Nexthop local: fe80::200:5efe:2a68:616e
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 14 ms
Read thread: on  Write thread: on  FD used: 27
6

That might be the problem - it’s going from the wrong source address, and maybe therefore via the wrong tunnel, as previously suggested…

7

Hmmm… There is a route added

Local host: 169.254.21.13, Local port: 40661
Foreign host: 169.254.21.1, Foreign port: 179
Nexthop: 169.254.21.13


Local host: 169.254.21.9, Local port: 42473
Foreign host: 169.254.22.1, Foreign port: 179
Nexthop: 169.254.21.9

See these are routes added

set protocols static interface-route xxx.xxx.21.1/32 next-hop-interface vti4
set protocols static interface-route xxx.xxx.22.1/32 next-hop-interface vti2
8

Here is the output when issue occured -

Peer ID / IP                            Local ID / IP
------------                            -------------
20.204.xx.xx                           111.125.xx.xx

    Description: 

    State  IKEVer  Encrypt  Hash    D-H Group      NAT-T  A-Time  L-Time
    -----  ------  -------  ----    ---------      -----  ------  ------
    up     IKEv2   aes256   sha1_96 2(MODP_1024)   no     3600    28800


Peer ID / IP                            Local ID / IP
------------                            -------------
20.219.xx.xx                          xx.xx.xx.xx 

    Description: 

    State  IKEVer  Encrypt  Hash    D-H Group      NAT-T  A-Time  L-Time
    -----  ------  -------  ----    ---------      -----  ------  ------
    down   IKEv2   n/a      n/a     n/a(n/a)       no     0       n/a

My secondary tunnel went down and I lost both my bgp neighbors

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
169.254.21.1    4      65515 1334270 1173801        0    0    0 00:31:43       Active
169.254.22.1    4      65515 1297265 1141169        0    0    0 00:32:05       Active
9

Here is something I noticed -

When I changed the local address as per config my BGP neighbor ship did not come up at all. my VPN IKE & IPsec was up for sure but unable to reach my peers.

set vpn ipsec site-to-site peer 20.204.xx.xx local-address 169.254.21.13
set vpn ipsec site-to-site peer 20.219.xx.xx local-address 169.254.21.9

Then again I had to revert to my old config

set vpn ipsec site-to-site peer 20.204.xx.xx local-address '111.125.x.x'
[edit]
set vpn ipsec site-to-site peer 20.219.xx.xx local-address '42.104.xx.xx'

And it came back without any issues. What could be the problem.

10

Try it

set vpn ipsec options disable-route-autoinstall
11

Tried that - However same issues happened once. Though I would wait for some more time to confirm if the issue is fixed.

12

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.