Struggling with iBGP to a Kube metallb instance

turbo_kebab_GTR · January 29, 2023, 11:21am

Hi,

I’m using 1.1.4 VyOS and Im quite new to this newer version using FRR.

I can’t get any semblance of BGP up with a local neighbor which is a metallb.

Wondering if I’m doing anything wrong as looks like maybe I’m missing a flag that could magically get everything online.

Had no issue getting this working with FRR on pfsense in the past and FRR directly on Linux it just seems to be with Vyos.

I also have the local firewall allowing all to itself on the required interface.

I have the following config:

set address-family ipv4-unicast network 10.0.0.0/8
set address-family ipv4-unicast network 172.16.0.0/12
set address-family ipv4-unicast network 192.168.0.0/16
set neighbor 10.89.10.44 address-family ipv4-unicast nexthop-self
set neighbor 10.89.10.44 address-family ipv4-unicast soft-reconfiguration inbound
set neighbor 10.89.10.44 capability dynamic
set neighbor 10.89.10.44 capability extended-nexthop
set neighbor 10.89.10.44 description 'rancher'
set neighbor 10.89.10.44 remote-as '65098'
set system-as '65098'

I have these logs

vyos@vyos-wizznet# run show log tail 300 | grep bgpd
Jan 29 11:11:33 vyos-wizznet bgpd[819]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Jan 29 11:11:33 vyos-wizznet bgpd[819]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [YTARA-Q9ZD1] [Event] BGP connection from host 10.89.10.44 fd 31
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from Idle to Active
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] TCP_connection_open (Active->OpenSent), fd 31
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [WECS1-Q4P17] 10.89.10.44 passive open
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [XKJ09-9VTZ7] 10.89.10.44 Sending hostname cap with hn = vyos-wizznet, dn = (null)
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [JFFAN-DEGED] 10.89.10.44 sending OPEN, version 4, my as 65098, holdtime 180, id 172.16.254.2
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: OpenSent established_peers 0
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from Active to OpenSent
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [WNM1E-D314G] 10.89.10.44 rcv OPEN, version 4, remote-as (in open) 65098, holdtime 90, id 10.89.10.44
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [QG29C-5TSVS] 10.89.10.44 rcv OPEN w/ OPTION parameter len: 20
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [NVZPF-5ST3B] 10.89.10.44 rcvd OPEN w/ optional parameter type 2 (Capability) len 18
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [SCW43-WN4M1] 10.89.10.44 OPEN has MultiProtocol Extensions capability (1), length 4
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [VXVH9-3MXR0] 10.89.10.44 OPEN has MultiProtocol Extensions capability for afi/safi: IPv4/unicast
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [SCW43-WN4M1] 10.89.10.44 OPEN has MultiProtocol Extensions capability (1), length 4
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [VXVH9-3MXR0] 10.89.10.44 OPEN has MultiProtocol Extensions capability for afi/safi: IPv6/unicast
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [SCW43-WN4M1] 10.89.10.44 OPEN has 4-octet AS number capability (65), length 4
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm), fd 31
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: OpenConfirm established_peers 0
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from OpenSent to OpenConfirm
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [NJ2F2-2W769] 10.89.10.44 [Event] BGP connection closed fd 31
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [NTX3S-9Q8YV] 10.89.10.44 [Event] BGP error 5 on fd 31
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] TCP_connection_closed (OpenConfirm->Idle), fd 31
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Deleted established_peers 0
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from OpenConfirm to Deleted
Jan 29 11:12:37 vyos-wizznet bgpd[819]: [JFMSW-YMBC7] 10.89.10.44 [FSM] Timer (connect timer expire)
Jan 29 11:12:37 vyos-wizznet bgpd[819]: [Z195V-FNKRK] 10.89.10.44 [Event] Connect start to 10.89.10.44 fd 28
Jan 29 11:12:37 vyos-wizznet bgpd[819]: [G0837-S7QES] 10.89.10.44 [FSM] Non blocking connect waiting result, fd 28
Jan 29 11:13:14 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] BGP_Stop (Connect->Idle), fd 28
Jan 29 11:13:14 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Idle established_peers 0
Jan 29 11:13:14 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from Connect to Idle
Jan 29 11:13:14 vyos-wizznet bgpd[819]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Jan 29 11:13:14 vyos-wizznet bgpd[819]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Jan 29 11:13:15 vyos-wizznet bgpd[819]: [ZQTB5-H8522] 10.89.10.44 [FSM] Timer (start timer expire).
Jan 29 11:13:15 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] BGP_Start (Idle->Connect), fd -1
Jan 29 11:13:15 vyos-wizznet bgpd[819]: [Z195V-FNKRK] 10.89.10.44 [Event] Connect start to 10.89.10.44 fd 28
Jan 29 11:13:15 vyos-wizznet bgpd[819]: [G0837-S7QES] 10.89.10.44 [FSM] Non blocking connect waiting result, fd 28
Jan 29 11:13:15 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Connect established_peers 0
Jan 29 11:13:15 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from Idle to Connect
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [YTARA-Q9ZD1] [Event] BGP connection from host 10.89.10.44 fd 31
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from Idle to Active
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] TCP_connection_open (Active->OpenSent), fd 31
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [WECS1-Q4P17] 10.89.10.44 passive open
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [XKJ09-9VTZ7] 10.89.10.44 Sending hostname cap with hn = vyos-wizznet, dn = (null)
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [JFFAN-DEGED] 10.89.10.44 sending OPEN, version 4, my as 65098, holdtime 180, id 172.16.254.2
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: OpenSent established_peers 0
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from Active to OpenSent
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [WNM1E-D314G] 10.89.10.44 rcv OPEN, version 4, remote-as (in open) 65098, holdtime 90, id 10.89.10.44
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [QG29C-5TSVS] 10.89.10.44 rcv OPEN w/ OPTION parameter len: 20
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [NVZPF-5ST3B] 10.89.10.44 rcvd OPEN w/ optional parameter type 2 (Capability) len 18
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [SCW43-WN4M1] 10.89.10.44 OPEN has MultiProtocol Extensions capability (1), length 4
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [VXVH9-3MXR0] 10.89.10.44 OPEN has MultiProtocol Extensions capability for afi/safi: IPv4/unicast
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [SCW43-WN4M1] 10.89.10.44 OPEN has MultiProtocol Extensions capability (1), length 4
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [VXVH9-3MXR0] 10.89.10.44 OPEN has MultiProtocol Extensions capability for afi/safi: IPv6/unicast
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [SCW43-WN4M1] 10.89.10.44 OPEN has 4-octet AS number capability (65), length 4
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] Receive_OPEN_message (OpenSent->OpenConfirm), fd 31
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: OpenConfirm established_peers 0
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from OpenSent to OpenConfirm
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [NTX3S-9Q8YV] 10.89.10.44 [Event] BGP error 7 on fd 31
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZWCSR-M7FG9] 10.89.10.44 [FSM] TCP_fatal_error (OpenConfirm->Idle), fd 31
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Deleted established_peers 0
Jan 29 11:14:17 vyos-wizznet bgpd[819]: [ZQHFG-DQGX1] 10.89.10.44 went from OpenConfirm to Deleted
[edit]

turbo_kebab_GTR · January 29, 2023, 11:22am

bgp neigh

BGP neighbor is 10.89.10.44, remote AS 65098, local AS 65098, internal link
  Local Role: undefined
  Remote Role: undefined
 Description: rancher
  BGP version 4, remote router ID 0.0.0.0, local router ID 172.16.254.2
  BGP state = Connect
  Last read 00:11:25, Last write never
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  Not part of any update group
  Inbound soft reconfiguration allowed
  NEXT_HOP is always this router
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:11:25,  Waiting for peer OPEN
  Internal BGP neighbor may be up to 255 hops away.
BGP Connect Retry Timer in Seconds: 120
Next connect timer due in 74 seconds
Read thread: off  Write thread: off  FD used: 28

[edit]```

turbo_kebab_GTR · January 29, 2023, 11:24am

tcpdump on kube instance

root@rancher2:~# tcpdump -i eth0 -nn port 179 -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:24:16.372609 IP (tos 0x0, ttl 64, id 65436, offset 0, flags [DF], proto TCP (6), length 60)
    10.89.10.44.42147 > 10.89.10.1.179: Flags [S], cksum 0x290d (incorrect -> 0x92f6), seq 1369507035, win 42340, options [mss 1460,sackOK,TS val 4030462645 ecr 0,nop,wscale 9], length 0
11:24:16.372831 IP (tos 0xc0, ttl 255, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.89.10.1.179 > 10.89.10.44.42147: Flags [S.], cksum 0x290d (incorrect -> 0x0130), seq 1365069206, ack 1369507036, win 65160, options [mss 1460,sackOK,TS val 2688809307 ecr 4030462645,nop,wscale 7], length 0
11:24:16.372859 IP (tos 0x0, ttl 64, id 65437, offset 0, flags [DF], proto TCP (6), length 52)
    10.89.10.44.42147 > 10.89.10.1.179: Flags [.], cksum 0x2905 (incorrect -> 0x2e32), seq 1, ack 1, win 83, options [nop,nop,TS val 4030462645 ecr 2688809307], length 0
11:24:16.372914 IP (tos 0x0, ttl 64, id 65438, offset 0, flags [DF], proto TCP (6), length 101)
    10.89.10.44.42147 > 10.89.10.1.179: Flags [P.], cksum 0x2936 (incorrect -> 0x9756), seq 1:50, ack 1, win 83, options [nop,nop,TS val 4030462645 ecr 2688809307], length 49: BGP
        Open Message (1), length: 49
          Version 4, my AS 65098, Holdtime 90s, ID 10.89.10.44
          Optional parameters, length: 20
            Option Capabilities Advertisement (2), length: 18
              Multiprotocol Extensions (1), length: 4
                AFI IPv4 (1), SAFI Unicast (1)
                0x0000:  0001 0001
              Multiprotocol Extensions (1), length: 4
                AFI IPv6 (2), SAFI Unicast (1)
                0x0000:  0002 0001
              32-Bit AS Number (65), length: 4
                 4 Byte AS 65098
                0x0000:  0000 fe4a
11:24:16.373472 IP (tos 0xc0, ttl 255, id 59909, offset 0, flags [DF], proto TCP (6), length 164)
    10.89.10.1.179 > 10.89.10.44.42147: Flags [P.], cksum 0x2975 (incorrect -> 0x28b7), seq 1:113, ack 1, win 510, options [nop,nop,TS val 2688809307 ecr 4030462645], length 112: BGP
        Open Message (1), length: 112
          Version 4, my AS 65098, Holdtime 180s, ID 172.16.254.2
          Optional parameters, length: 83
            Option Capabilities Advertisement (2), length: 6
              Multiprotocol Extensions (1), length: 4
                AFI IPv4 (1), SAFI Unicast (1)
                0x0000:  0001 0001
            Option Capabilities Advertisement (2), length: 2
              Route Refresh (Cisco) (128), length: 0
            Option Capabilities Advertisement (2), length: 2
              Route Refresh (2), length: 0
            Option Capabilities Advertisement (2), length: 2
              Enhanced Route Refresh (70), length: 0
                no decoder for Capability 70
            Option Capabilities Advertisement (2), length: 6
              32-Bit AS Number (65), length: 4
                 4 Byte AS 65098
                0x0000:  0000 fe4a
            Option Capabilities Advertisement (2), length: 2
              Unknown (6), length: 0
                no decoder for Capability 6
            Option Capabilities Advertisement (2), length: 6
              Multiple Paths (69), length: 4
                AFI IPv4 (1), SAFI Unicast (1), Send/Receive: Receive
                0x0000:  0001 0101
            Option Capabilities Advertisement (2), length: 2
              Unknown (66), length: 0
                no decoder for Capability 66
            Option Capabilities Advertisement (2), length: 2
              Dynamic Capability (67), length: 0
                no decoder for Capability 67
            Option Capabilities Advertisement (2), length: 16
              Unknown (73), length: 14
                no decoder for Capability 73
                0x0000:  0c76 796f 732d 7769 7a7a 6e65 7400
            Option Capabilities Advertisement (2), length: 4
              Graceful Restart (64), length: 2
                Restart Flags: [none], Restart Time 120s
                0x0000:  4078
            Option Capabilities Advertisement (2), length: 9
              Unknown (71), length: 7
                no decoder for Capability 71
                0x0000:  0001 0180 0000 00
11:24:16.373486 IP (tos 0x0, ttl 64, id 65439, offset 0, flags [DF], proto TCP (6), length 52)
    10.89.10.44.42147 > 10.89.10.1.179: Flags [.], cksum 0x2905 (incorrect -> 0x2d90), seq 50, ack 113, win 83, options [nop,nop,TS val 4030462646 ecr 2688809307], length 0
11:24:16.373496 IP (tos 0xc0, ttl 255, id 59910, offset 0, flags [DF], proto TCP (6), length 52)
    10.89.10.1.179 > 10.89.10.44.42147: Flags [.], cksum 0x2905 (incorrect -> 0x2be5), seq 113, ack 50, win 510, options [nop,nop,TS val 2688809308 ecr 4030462645], length 0
11:24:16.373618 IP (tos 0xc0, ttl 255, id 59911, offset 0, flags [DF], proto TCP (6), length 71)
    10.89.10.1.179 > 10.89.10.44.42147: Flags [P.], cksum 0x2918 (incorrect -> 0x27b6), seq 113:132, ack 50, win 510, options [nop,nop,TS val 2688809308 ecr 4030462646], length 19: BGP
        Keepalive Message (4), length: 19
11:24:16.373624 IP (tos 0x0, ttl 64, id 65440, offset 0, flags [DF], proto TCP (6), length 52)
    10.89.10.44.42147 > 10.89.10.1.179: Flags [.], cksum 0x2905 (incorrect -> 0x2d7c), seq 50, ack 132, win 83, options [nop,nop,TS val 4030462646 ecr 2688809308], length 0
11:24:16.376805 IP (tos 0x0, ttl 64, id 65441, offset 0, flags [DF], proto TCP (6), length 52)
    10.89.10.44.42147 > 10.89.10.1.179: Flags [R.], cksum 0x2905 (incorrect -> 0x2d75), seq 50, ack 132, win 83, options [nop,nop,TS val 4030462649 ecr 2688809308], length 0

a.apostoliuk · January 30, 2023, 8:23am

@turbo_kebab_GTR
Can you show the configuration from the other side, or the working FRR configuration?

Viacheslav · January 30, 2023, 9:16am

Also take a look at the timers

Jan 29 11:12:17 vyos-wizznet bgpd[819]: [JFFAN-DEGED] 10.89.10.44 sending OPEN, version 4, my as 65098, holdtime 180, id 172.16.254.2
Jan 29 11:12:17 vyos-wizznet bgpd[819]: [WNM1E-D314G] 10.89.10.44 rcv OPEN, version 4, remote-as (in open) 65098, holdtime 90, id 10.89.10.44

turbo_kebab_GTR · January 30, 2023, 9:33am

Sure the working FRR config from my old Pfsense firewall is here

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname pf-firewall1.wizznet.co.uk
service integrated-vtysh-config
!
password ____
!
router bgp 65089
 bgp log-neighbor-changes
 no bgp ebgp-requires-policy
 bgp deterministic-med
 update-delay 1
 bgp graceful-shutdown
 bgp bestpath as-path multipath-relax
 no bgp network import-check
 timers bgp 1 3
 neighbor 10.89.10.44 remote-as 65098
 neighbor 10.89.10.44 description rancher2
 neighbor 10.89.10.44 disable-connected-check
 neighbor 10.89.10.44 timers 5 15
 neighbor 10.89.10.44 timers connect 5
 !
 address-family ipv4 unicast
  neighbor 10.89.10.44 addpath-tx-bestpath-per-AS
  neighbor 10.89.10.44 next-hop-self
  no neighbor 10.89.10.44 send-community
  neighbor 10.89.10.44 soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor 10.89.10.44 activate
  neighbor 10.89.10.44 addpath-tx-bestpath-per-AS
  neighbor 10.89.10.44 next-hop-self
  no neighbor 10.89.10.44 send-community
  neighbor 10.89.10.44 soft-reconfiguration inbound
 exit-address-family
!
line vty
!
end

turbo_kebab_GTR · January 30, 2023, 9:35am

pf-firewall1.wizznet.co.uk# show ip bgp neigh
BGP neighbor is 10.89.10.44, remote AS 65098, local AS 65089, external link
 Description: rancher2
  BGP version 4, remote router ID 10.89.10.44, local router ID 172.16.255.2
  BGP state = Established, up for 00:01:33
  Last read 00:00:03, Last write 00:00:03
  Hold time is 15, keepalive interval is 5 seconds
  Configured hold time is 15, keepalive interval is 5 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: TX advertised IPv4 Unicast
      IPv4 Unicast: RX advertised IPv4 Unicast
      IPv6 Unicast: TX advertised IPv6 Unicast
      IPv6 Unicast: RX advertised IPv6 Unicast
    Route refresh: advertised
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: advertised and received
    Hostname Capability: advertised (name: pf-firewall1.wizznet.co.uk,domain name: n/a) not received
    Graceful Restart Capability: advertised
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: Disable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                1         12
    Keepalives:            19         19
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 21         32
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  Update group 1, subgroup 1
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Advertise bestpath per AS via addpath
  NEXT_HOP is always this router
  Community attribute sent to this neighbor(large)
  12 accepted prefixes

 For address family: IPv6 Unicast
  Update group 2, subgroup 2
  Packet Queue length 0
  Inbound soft reconfiguration allowed
  Advertise bestpath per AS via addpath
  NEXT_HOP is always this router
  Community attribute sent to this neighbor(large)
  0 accepted prefixes

  Connections established 1; dropped 0
  Last reset 00:03:01,  No AFI/SAFI activated for peer
Local host: 10.89.10.1, Local port: 179
Foreign host: 10.89.10.44, Foreign port: 38475
Nexthop: 10.89.10.1
Nexthop global: fe80::bcb9:b5ff:fea4:9e5c
Nexthop local: fe80::bcb9:b5ff:fea4:9e5c
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 5
Estimated round trip time: 2 ms
Read thread: on  Write thread: on  FD used: 25

turbo_kebab_GTR · January 30, 2023, 9:35am

pf-firewall1.wizznet.co.uk# show ip bgp
BGP table version is 12, local router ID is 172.16.255.2, vrf id 0
Default local pref 100, local AS 65089
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.89.198.1/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.2/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.3/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.4/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.5/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.6/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.8/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.9/32   10.89.10.44                     0      0 65098 ?
*> 10.89.198.10/32  10.89.10.44                     0      0 65098 ?
*> 10.89.198.11/32  10.89.10.44                     0      0 65098 ?
*> 10.89.198.12/32  10.89.10.44                     0      0 65098 ?
*> 10.89.198.13/32  10.89.10.44                     0      0 65098 ?

Displayed  12 routes and 12 total paths

Viacheslav · January 30, 2023, 9:54am

  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds

vs

  Hold time is 15, keepalive interval is 5 seconds
  Configured hold time is 15, keepalive interval is 5 seconds

turbo_kebab_GTR · January 30, 2023, 11:24am

sorry forgot to say I already tried timers

vyos@vyos-wizznet# show | commands
set address-family ipv4-unicast network 10.0.0.0/8
set address-family ipv4-unicast network 172.16.0.0/12
set address-family ipv4-unicast network 192.168.0.0/16
set neighbor 10.89.10.44 address-family ipv4-unicast nexthop-self force
set neighbor 10.89.10.44 address-family ipv4-unicast soft-reconfiguration inbound
set neighbor 10.89.10.44 capability dynamic
set neighbor 10.89.10.44 capability extended-nexthop
set neighbor 10.89.10.44 description 'rancher'
set neighbor 10.89.10.44 disable-connected-check
set neighbor 10.89.10.44 remote-as '65098'
set neighbor 10.89.10.44 timers connect '5'
set neighbor 10.89.10.44 timers holdtime '15'
set neighbor 10.89.10.44 timers keepalive '5'
set parameters deterministic-med
set system-as '65098'
[edit protocols bgp]

turbo_kebab_GTR · January 30, 2023, 11:33am

Jan 30 10:00:50 vyos-wizznet bgpd[820]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer
Jan 30 10:02:50 vyos-wizznet bgpd[820]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer
Jan 30 10:04:50 vyos-wizznet bgpd[820]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer
Jan 30 10:06:50 vyos-wizznet bgpd[820]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer
Jan 30 10:10:50 vyos-wizznet bgpd[820]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer
Jan 30 10:12:50 vyos-wizznet bgpd[820]: [MJ4D6-VBJKV][EC 33554454] 10.89.10.44 [Error] bgp_read_packet error: Connection reset by peer

turbo_kebab_GTR · January 30, 2023, 12:52pm

sorry its an AS number problem 65098 vs 65089

turbo_kebab_GTR · January 30, 2023, 12:56pm

sorted

vyos@vyos-wizznet# run show ip bgp
BGP table version is 15, local router ID is 172.16.254.2, vrf id 0
Default local pref 100, local AS 65089
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.0.0.0/8       0.0.0.0                  0         32768 i
*> 10.89.198.1/32   10.89.10.44                            0 65098 ?
*> 10.89.198.2/32   10.89.10.44                            0 65098 ?
*> 10.89.198.3/32   10.89.10.44                            0 65098 ?
*> 10.89.198.4/32   10.89.10.44                            0 65098 ?
*> 10.89.198.5/32   10.89.10.44                            0 65098 ?
*> 10.89.198.6/32   10.89.10.44                            0 65098 ?
*> 10.89.198.8/32   10.89.10.44                            0 65098 ?
*> 10.89.198.9/32   10.89.10.44                            0 65098 ?
*> 10.89.198.10/32  10.89.10.44                            0 65098 ?
*> 10.89.198.11/32  10.89.10.44                            0 65098 ?
*> 10.89.198.12/32  10.89.10.44                            0 65098 ?
*> 10.89.198.13/32  10.89.10.44                            0 65098 ?
*> 172.16.0.0/12    0.0.0.0                  0         32768 i
*> 192.168.0.0/16   0.0.0.0                  0         32768 i

Displayed  15 routes and 15 total paths
[edit]