BGP issues and unable to identify- Can someone please help?

blason · June 7, 2021, 4:30am

Hi Team,

In my below scenario both my tunnels are up

vyos@R4# run show vpn ipsec sa
Peer ID / IP                            Local ID / IP
------------                            -------------
100.1.1.60                              100.1.1.50

    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    vti     up     205.0K/14.3K   aes256   sha256  no     2146    3600    all


Peer ID / IP                            Local ID / IP
------------                            -------------
200.1.1.60                              200.1.1.50

    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    vti     up     13.8K/198.0K   aes256   sha256  no     2021    3600    all

And BGP is being learned from both the interfaces

vyos@R4# run show ip bgp
BGP table version is 0, local router ID is 192.168.5.50
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.10.11.0/24    169.254.254.11           1             0 65002 i
*                   169.254.252.11           1             0 65002 i
*> 192.168.47.0     0.0.0.0                  1         32768 i

Total number of prefixes 2

Here are my interfaces

vyos@R4# run show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.47.50/24                  u/u
eth1             200.1.1.50/24                     u/u
eth2             100.1.1.50/24                     u/u
eth3             192.168.5.50/24                   u/u
lo               127.0.0.1/8                       u/u
                 ::1/128
vti5             169.254.254.10/32                 u/u
vti10            169.254.252.10/32                 u/u
[edit]

Now my ping from 192.168.47.128 to 10.10.11.129 works fine through link 1 or through 169.254.254.10

however if I shut eth2 [100.1.1.50] even though bgp show

vyos@R4# set interfaces ethernet eth2 disable
[edit]
vyos@R4# commit
vyos@R4# run show ip bgp
BGP table version is 0, local router ID is 192.168.5.50
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*  10.10.11.0/24    169.254.252.11           1             0 65002 i
*> 192.168.47.0     0.0.0.0                  1         32768 i

Total number of prefixes 2

My ping never comes up

Tunnel even shows properly

vyos@R5# run show vpn ipsec sa
Peer ID / IP                            Local ID / IP
------------                            -------------
100.1.1.50                              100.1.1.60

    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    vti     down   n/a            n/a      n/a     no     0       3600    all


Peer ID / IP                            Local ID / IP
------------                            -------------
200.1.1.50                              200.1.1.60

    Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
    ------  -----  -------------  -------  ----    -----  ------  ------  -----
    vti     up     229.2K/19.5K   aes256   sha256  no     1887    3600    all

scene3
Any clue why?

vyos@R4# run show ip bgp neighbors 169.254.252.11
BGP neighbor is 169.254.252.11, remote AS 65002, local AS 65001, external link
  BGP version 4, remote router ID 192.168.5.51
  BGP state = Established, up for 00:38:43
  Last read 22:15:58, hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Route refresh: advertised and received(old & new)
    Address family IPv4 Unicast: advertised and received
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                 20          0
    Notifications:          0          9
    Updates:                5          3
    Keepalives:           233        232
    Route Refresh:          0          0
    Capability:             0          0
    Total:                258        244
  Minimum time between advertisement runs is 30 seconds
  Update source is 169.254.252.10

 For address family: IPv4 Unicast
  Community attribute sent to this neighbor(both)
  0 accepted prefixes

  Connections established 1; dropped 0
  Last reset never
Local host: 169.254.252.10, Local port: 179
Foreign host: 169.254.252.11, Foreign port: 34037
Nexthop: 169.254.252.10
Nexthop global: ::
Nexthop local: ::
BGP connection: non shared network
Read thread: on  Write thread: off

I can even reach 169.254.252.11 from R4

vyos@R4# ping 169.254.252.11
PING 169.254.252.11 (169.254.252.11) 56(84) bytes of data.
64 bytes from 169.254.252.11: icmp_req=1 ttl=64 time=0.407 ms
64 bytes from 169.254.252.11: icmp_req=2 ttl=64 time=0.435 ms
^C
--- 169.254.252.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.407/0.421/0.435/0.014 ms

Here is my entire config for R4

set interfaces ethernet eth0 address '192.168.47.50/24'
set interfaces ethernet eth0 hw-id '00:0c:29:ef:bb:2c'
set interfaces ethernet eth1 address '200.1.1.50/24'
set interfaces ethernet eth1 hw-id '00:0c:29:ef:bb:36'
set interfaces ethernet eth2 address '100.1.1.50/24'
set interfaces ethernet eth2 'disable'
set interfaces ethernet eth2 hw-id '00:0c:29:ef:bb:40'
set interfaces ethernet eth3 address '192.168.5.50/24'
set interfaces ethernet eth3 hw-id '00:0c:29:ef:bb:4a'
set interfaces loopback 'lo'
set interfaces vti vti5 address '169.254.254.10/32'
set interfaces vti vti10 address '169.254.252.10/32'
set protocols bgp 65001 neighbor 169.254.252.11 'disable-connected-check'
set protocols bgp 65001 neighbor 169.254.252.11 ebgp-multihop '10'
set protocols bgp 65001 neighbor 169.254.252.11 remote-as '65002'
set protocols bgp 65001 neighbor 169.254.252.11 timers holdtime '30'
set protocols bgp 65001 neighbor 169.254.252.11 timers keepalive '10'
set protocols bgp 65001 neighbor 169.254.252.11 update-source '169.254.252.10'
set protocols bgp 65001 neighbor 169.254.254.11 'disable-connected-check'
set protocols bgp 65001 neighbor 169.254.254.11 ebgp-multihop '10'
set protocols bgp 65001 neighbor 169.254.254.11 remote-as '65002'
set protocols bgp 65001 neighbor 169.254.254.11 timers holdtime '30'
set protocols bgp 65001 neighbor 169.254.254.11 timers keepalive '10'
set protocols bgp 65001 neighbor 169.254.254.11 update-source '169.254.254.10'
set protocols bgp 65001 network '192.168.47.0/24'
set protocols static interface-route 169.254.252.11/32 next-hop-interface 'vti10'
set protocols static interface-route 169.254.254.11/32 next-hop-interface 'vti5'
set service 'ssh'
set system config-management commit-revisions '20'
set system console device ttyS0 speed '9600'
set system host-name 'R4'
set system login user vyos authentication encrypted-password '$1$FNYTiiQ7$JdybrnisWCqxGWgOIbgTx/'
set system login user vyos authentication plaintext-password ''
set system login user vyos level 'admin'
set system ntp server '0.pool.ntp.org'
set system ntp server '1.pool.ntp.org'
set system ntp server '2.pool.ntp.org'
set system package repository community components 'main'
set system package repository community distribution 'helium'
set system package repository community url 'http://packages.vyos.net/vyos'
set system syslog global facility all level 'notice'
set system syslog global facility protocols level 'debug'
set vpn ipsec esp-group r4-r5-l1-esp compression 'disable'
set vpn ipsec esp-group r4-r5-l1-esp lifetime '3600'
set vpn ipsec esp-group r4-r5-l1-esp mode 'tunnel'
set vpn ipsec esp-group r4-r5-l1-esp pfs 'enable'
set vpn ipsec esp-group r4-r5-l1-esp proposal 5 encryption 'aes256'
set vpn ipsec esp-group r4-r5-l1-esp proposal 5 hash 'sha256'
set vpn ipsec esp-group r4-r5-l2-esp compression 'disable'
set vpn ipsec esp-group r4-r5-l2-esp lifetime '3600'
set vpn ipsec esp-group r4-r5-l2-esp mode 'tunnel'
set vpn ipsec esp-group r4-r5-l2-esp pfs 'enable'
set vpn ipsec esp-group r4-r5-l2-esp proposal 5 encryption 'aes256'
set vpn ipsec esp-group r4-r5-l2-esp proposal 5 hash 'sha256'
set vpn ipsec ike-group r4-r5-l1-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group r4-r5-l1-ike dead-peer-detection interval '15'
set vpn ipsec ike-group r4-r5-l1-ike dead-peer-detection timeout '30'
set vpn ipsec ike-group r4-r5-l1-ike ikev2-reauth 'no'
set vpn ipsec ike-group r4-r5-l1-ike key-exchange 'ikev1'
set vpn ipsec ike-group r4-r5-l1-ike lifetime '28800'
set vpn ipsec ike-group r4-r5-l1-ike proposal 10 dh-group '2'
set vpn ipsec ike-group r4-r5-l1-ike proposal 10 encryption 'aes256'
set vpn ipsec ike-group r4-r5-l1-ike proposal 10 hash 'sha256'
set vpn ipsec ike-group r4-r5-l2-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group r4-r5-l2-ike dead-peer-detection interval '15'
set vpn ipsec ike-group r4-r5-l2-ike dead-peer-detection timeout '30'
set vpn ipsec ike-group r4-r5-l2-ike ikev2-reauth 'no'
set vpn ipsec ike-group r4-r5-l2-ike key-exchange 'ikev1'
set vpn ipsec ike-group r4-r5-l2-ike lifetime '28800'
set vpn ipsec ike-group r4-r5-l2-ike proposal 10 dh-group '2'
set vpn ipsec ike-group r4-r5-l2-ike proposal 10 encryption 'aes256'
set vpn ipsec ike-group r4-r5-l2-ike proposal 10 hash 'sha256'
set vpn ipsec ipsec-interfaces interface 'eth2'
set vpn ipsec ipsec-interfaces interface 'eth1'
set vpn ipsec site-to-site peer 100.1.1.60 authentication id '100.1.1.50'
set vpn ipsec site-to-site peer 100.1.1.60 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer 100.1.1.60 authentication pre-shared-secret 'admin@123'
set vpn ipsec site-to-site peer 100.1.1.60 authentication remote-id '100.1.1.60'
set vpn ipsec site-to-site peer 100.1.1.60 connection-type 'initiate'
set vpn ipsec site-to-site peer 100.1.1.60 ike-group 'r4-r5-l1-ike'
set vpn ipsec site-to-site peer 100.1.1.60 local-address '100.1.1.50'
set vpn ipsec site-to-site peer 100.1.1.60 vti bind 'vti5'
set vpn ipsec site-to-site peer 100.1.1.60 vti esp-group 'r4-r5-l1-esp'
set vpn ipsec site-to-site peer 200.1.1.60 authentication id '200.1.1.50'
set vpn ipsec site-to-site peer 200.1.1.60 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer 200.1.1.60 authentication pre-shared-secret 'admin@123'
set vpn ipsec site-to-site peer 200.1.1.60 authentication remote-id '200.1.1.60'
set vpn ipsec site-to-site peer 200.1.1.60 connection-type 'initiate'
set vpn ipsec site-to-site peer 200.1.1.60 ike-group 'r4-r5-l2-ike'
set vpn ipsec site-to-site peer 200.1.1.60 local-address '200.1.1.50'
set vpn ipsec site-to-site peer 200.1.1.60 vti bind 'vti10'
set vpn ipsec site-to-site peer 200.1.1.60 vti esp-group 'r4-r5-l2-esp'

And here is for R5

set interfaces ethernet eth0 address '10.10.11.60/24'
set interfaces ethernet eth0 hw-id '00:0c:29:e3:b7:c7'
set interfaces ethernet eth1 address '100.1.1.60/24'
set interfaces ethernet eth1 hw-id '00:0c:29:e3:b7:d1'
set interfaces ethernet eth2 address '200.1.1.60/24'
set interfaces ethernet eth2 hw-id '00:0c:29:e3:b7:db'
set interfaces ethernet eth3 address '192.168.5.51/24'
set interfaces ethernet eth3 hw-id '00:0c:29:e3:b7:e5'
set interfaces loopback 'lo'
set interfaces vti vti5 address '169.254.254.11/32'
set interfaces vti vti10 address '169.254.252.11/32'
set protocols bgp 65002 neighbor 169.254.252.10 'disable-connected-check'
set protocols bgp 65002 neighbor 169.254.252.10 ebgp-multihop '10'
set protocols bgp 65002 neighbor 169.254.252.10 remote-as '65001'
set protocols bgp 65002 neighbor 169.254.252.10 timers holdtime '30'
set protocols bgp 65002 neighbor 169.254.252.10 timers keepalive '10'
set protocols bgp 65002 neighbor 169.254.252.10 update-source '169.254.252.11'
set protocols bgp 65002 neighbor 169.254.254.10 'disable-connected-check'
set protocols bgp 65002 neighbor 169.254.254.10 ebgp-multihop '10'
set protocols bgp 65002 neighbor 169.254.254.10 remote-as '65001'
set protocols bgp 65002 neighbor 169.254.254.10 timers holdtime '30'
set protocols bgp 65002 neighbor 169.254.254.10 timers keepalive '10'
set protocols bgp 65002 neighbor 169.254.254.10 update-source '169.254.254.11'
set protocols bgp 65002 network '10.10.11.0/24'
set protocols static interface-route 169.254.252.10/32 next-hop-interface 'vti10'
set protocols static interface-route 169.254.254.10/32 next-hop-interface 'vti5'
set service 'ssh'
set system config-management commit-revisions '20'
set system console device ttyS0 speed '9600'
set system host-name 'R5'
set system login user vyos authentication encrypted-password '$1$.0aLHoPW$8Ks/gmja3BMyg7cInbBZz.'
set system login user vyos authentication plaintext-password ''
set system login user vyos level 'admin'
set system ntp server '0.pool.ntp.org'
set system ntp server '1.pool.ntp.org'
set system ntp server '2.pool.ntp.org'
set system package repository community components 'main'
set system package repository community distribution 'helium'
set system package repository community url 'http://packages.vyos.net/vyos'
set system syslog global facility all level 'notice'
set system syslog global facility protocols level 'debug'
set vpn ipsec esp-group r5-r4-l1-esp compression 'disable'
set vpn ipsec esp-group r5-r4-l1-esp lifetime '3600'
set vpn ipsec esp-group r5-r4-l1-esp mode 'tunnel'
set vpn ipsec esp-group r5-r4-l1-esp pfs 'enable'
set vpn ipsec esp-group r5-r4-l1-esp proposal 5 encryption 'aes256'
set vpn ipsec esp-group r5-r4-l1-esp proposal 5 hash 'sha256'
set vpn ipsec esp-group r5-r4-l2-esp compression 'disable'
set vpn ipsec esp-group r5-r4-l2-esp lifetime '3600'
set vpn ipsec esp-group r5-r4-l2-esp mode 'tunnel'
set vpn ipsec esp-group r5-r4-l2-esp pfs 'enable'
set vpn ipsec esp-group r5-r4-l2-esp proposal 5 encryption 'aes256'
set vpn ipsec esp-group r5-r4-l2-esp proposal 5 hash 'sha256'
set vpn ipsec ike-group r5-r4-l1-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group r5-r4-l1-ike dead-peer-detection interval '15'
set vpn ipsec ike-group r5-r4-l1-ike dead-peer-detection timeout '30'
set vpn ipsec ike-group r5-r4-l1-ike ikev2-reauth 'no'
set vpn ipsec ike-group r5-r4-l1-ike key-exchange 'ikev1'
set vpn ipsec ike-group r5-r4-l1-ike lifetime '28800'
set vpn ipsec ike-group r5-r4-l1-ike proposal 5 dh-group '2'
set vpn ipsec ike-group r5-r4-l1-ike proposal 5 encryption 'aes256'
set vpn ipsec ike-group r5-r4-l1-ike proposal 5 hash 'sha256'
set vpn ipsec ike-group r5-r4-l2-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group r5-r4-l2-ike dead-peer-detection interval '15'
set vpn ipsec ike-group r5-r4-l2-ike dead-peer-detection timeout '30'
set vpn ipsec ike-group r5-r4-l2-ike ikev2-reauth 'no'
set vpn ipsec ike-group r5-r4-l2-ike key-exchange 'ikev1'
set vpn ipsec ike-group r5-r4-l2-ike lifetime '28800'
set vpn ipsec ike-group r5-r4-l2-ike proposal 5 dh-group '2'
set vpn ipsec ike-group r5-r4-l2-ike proposal 5 encryption 'aes256'
set vpn ipsec ike-group r5-r4-l2-ike proposal 5 hash 'sha256'
set vpn ipsec ipsec-interfaces interface 'eth1'
set vpn ipsec ipsec-interfaces interface 'eth2'
set vpn ipsec site-to-site peer 100.1.1.50 authentication id '100.1.1.60'
set vpn ipsec site-to-site peer 100.1.1.50 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer 100.1.1.50 authentication pre-shared-secret 'admin@123'
set vpn ipsec site-to-site peer 100.1.1.50 authentication remote-id '100.1.1.50'
set vpn ipsec site-to-site peer 100.1.1.50 ike-group 'r5-r4-l1-ike'
set vpn ipsec site-to-site peer 100.1.1.50 local-address '100.1.1.60'
set vpn ipsec site-to-site peer 100.1.1.50 vti bind 'vti5'
set vpn ipsec site-to-site peer 100.1.1.50 vti esp-group 'r5-r4-l1-esp'
set vpn ipsec site-to-site peer 200.1.1.50 authentication id '200.1.1.60'
set vpn ipsec site-to-site peer 200.1.1.50 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer 200.1.1.50 authentication pre-shared-secret 'admin@123'
set vpn ipsec site-to-site peer 200.1.1.50 authentication remote-id '200.1.1.50'
set vpn ipsec site-to-site peer 200.1.1.50 ike-group 'r5-r4-l2-ike'
set vpn ipsec site-to-site peer 200.1.1.50 local-address '200.1.1.60'
set vpn ipsec site-to-site peer 200.1.1.50 vti bind 'vti10'
set vpn ipsec site-to-site peer 200.1.1.50 vti esp-group 'r5-r4-l2-esp'

acrane · June 7, 2021, 4:53am

Hi, blason!

Could you please provide me with you VyOS version?

blason · June 7, 2021, 7:16am

Its 1.1.8 and i know its outdated - and 1.2 has a bug for VTI due to which I can put that live traffic since its not allowing me to create VTI

acrane · June 7, 2021, 7:22am

What exactly with VTI in 1.2.x do you mean? Maybe it’s already fixed. Tell me and I’ll try to dind it out.
Also there’s 1.3 LTS coming soon (end of August)

blason · June 7, 2021, 7:27am

Please have a look at this.

Viacheslav · June 7, 2021, 7:30am

It affected only 1.4 release, not 1.3 or 1.2
It’s already fixed in the latest rolling 1.4 releases

blason · June 7, 2021, 7:30am

I built 1.2 and struck with this issue. May be I sync current github that i had built 1.4

Dmitry · June 7, 2021, 7:33am

Hi, @blason are you sure that you build 1.2.x? I saw in your some posts that you build 1.4 but with the name crux-xxxx. Provide show version output

blason · June 7, 2021, 7:37am

I unfortunately deleted that machine. Let me install that ISO again.
By they is there any procedure to build 1.2.x, pls?

Dmitry · June 7, 2021, 7:40am

Use VyOS docs Building VyOS — VyOS 1.2.x (crux) documentation

acrane · June 8, 2021, 8:08am

It’s 220 table blocks ping.
Try set vpn ipsec options disable-route-autoinstall
After that commit-save-reboot

blason · June 8, 2021, 8:39am

okies @acrane let me try running those.

blason · June 8, 2021, 6:18pm

Actually just a reboot has resolved the issue. Been struggling on it for almost for 2 days and didn’t reboot.

system · June 10, 2021, 6:18pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.