1.4 - BGP OPEN messages being ignored, RST/ACK

kcslb92 · November 1, 2021, 12:28am

Hi Guys,

I’m not sure if I have discovered a bug or if I’ve missed something quite obvious but I have been having issues standing up BGP peering relationships between my vyos router and NSX-T routers.

For reference:

vyos image:

1.4-rolling-202110240217

vyos peering interfaces

eth0.2010 - 10.255.217.33/28
eth0.2011 - 10.255.217.49/28

nsx peering interfaces

VLAN 2010 - 10.255.217.35/28, 10.255.217.36/28
VLAN 2011 - 10.255.217.51/28, 10.255.217.52/28

vyos AS: 65001
nsx AS: 65000

Here is a snippet of the packet capture to show what I’m seeing:

NSX router sends a SYN to the vyos router, tcp port 179 for bgp
vyos responds with a SYN, ACK to open the tcp stream
nsx sends an ack to vyos
nsx sends a BGP OPEN message
vyos sends back an ACK for the OPEN message
vyos sends a RST, ACK to close the tcp stream immediately thereafter

The closest thing to this behaviour that I could find online was here, indicating that it could be an issue with one of my BGP parameters.

I’ve enabled all of the bgp debug monitoring modes to see what the logs would show (as per below) but found that the only one that generated additional logs messages was the updates debug:

[email protected]:/var/log$ monitor protocol bgp enable
Possible completions:
  allow-martians
                Enable BGP martians next hops debugging
  as4           Enable BGP allow AS4 actions debugging
  bestpath      Enable BGP allow best path debugging
  flowspec      Enable BGP allow flowspec debugging
  keepalives    Enable BGP keepalives debugging
  labelpool     Enable BGP label pool debugging
  neighbor-events
                Enable BGP Neighbor events debugging
  nht           Enable BGP next hop tracking debugging
  pbr           Enable BGP policy based routing debugging
  rib           Enable BGP rib debugging
  update-groups Enable BGP update groups debugging
  updates       Enable BGP updates debugging
  vnc           Enable BGP VNC debugging

Here are the activated monitoring options:

[email protected]:/var/log$ show monitoring
Zebra debugging status:

RIP debugging status:

RIPng debugging status:

OSPF6 debugging status:

LDP debugging status:

BGP debugging status:
  BGP as4 debugging is on
  BGP keepalives debugging is on
  BGP neighbor-events debugging is on
  BGP next-hop tracking debugging is on
  BGP update-groups debugging is on
  BGP updates debugging is on (inbound)
  BGP updates debugging is on (outbound)
  BGP zebra debugging is on
  BGP allow martian next hop debugging is on
  BGP flowspec debugging is on
  BGP labelpool debugging is on
  BGP policy based routing debugging is on

isis debugging status:

Staticd debugging status

BFD debugging status:

Here’s some the of the relevant logs:

[email protected]:/var/log$ grep -r bgpd .
grep: ./btmp: Permission denied
./frr/frr-reload.log:2021-11-01 11:06:13,651  INFO: Called via "Namespace(input=None, reload=True, test=False, debug=False, log_level='info', stdout=False, pathspace=None, filename='/tmp/tmp6sxn6e2n', overwrite=False, bindir='/usr/bin', confdir='/etc/frr', rundir='/var/run/frr', vty_socket=None, daemon='bgpd')"
grep: ./vmware-vmsvc-root.log: Permission denied
./messages:Nov  1 11:05:57 a-vrouter-01 watchfrr[911]: bgpd state -> down : initial connection attempt failed
./messages:Nov  1 11:05:57 a-vrouter-01 watchfrr.sh: Cannot stop bgpd: pid file not found
./messages:Nov  1 11:06:00 a-vrouter-01 watchfrr[911]: bgpd state -> up : connect succeeded
./messages:Nov  1 11:06:06 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth0.192 in VRF 0
./messages:Nov  1 11:06:06 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_VRF_UPDATE: Cannot find IF eth0.192 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth0.2020 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_VRF_UPDATE: Cannot find IF eth0.2020 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth0.2021 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_VRF_UPDATE: Cannot find IF eth0.2021 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth0.2030 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_VRF_UPDATE: Cannot find IF eth0.2030 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth0.2031 in VRF 0
./messages:Nov  1 11:06:07 a-vrouter-01 bgpd[962]: [EC 100663301] INTERFACE_VRF_UPDATE: Cannot find IF eth0.2031 in VRF 0
./messages:Nov  1 11:08:46 a-vrouter-01 bgpd[962]: 10.255.217.36 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:46 a-vrouter-01 bgpd[962]: 10.255.217.35 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:47 a-vrouter-01 bgpd[962]: 10.255.217.51 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:48 a-vrouter-01 bgpd[962]: 10.255.217.52 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:56 a-vrouter-01 bgpd[962]: 10.255.217.36 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:56 a-vrouter-01 bgpd[962]: 10.255.217.35 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:57 a-vrouter-01 bgpd[962]: 10.255.217.51 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:08:58 a-vrouter-01 bgpd[962]: 10.255.217.52 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:09:06 a-vrouter-01 bgpd[962]: 10.255.217.36 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:09:06 a-vrouter-01 bgpd[962]: 10.255.217.35 - incoming conn rejected - no AF activated for peer
./messages:Nov  1 11:09:07 a-vrouter-01 bgpd[962]: 10.255.217.51 - incoming conn rejected - no AF activated for peer
grep: ./private: Permission denied
grep: ./vmware-vmtoolsd-root.log: Permission denied

Those incoming conn rejected - no AF activated for peer messages are the ones that get generated by having updates debug logging enabled as per above.

What I also find odd is this:

[email protected]:~$ show ip bgp summary
% No BGP neighbors found
[email protected]:~$ show ip bgp neighbors
BGP neighbor is 10.255.217.35, remote AS 65000, local AS 65001, external link
  BGP version 4, remote router ID 0.0.0.0, local router ID 10.255.217.49
  BGP state = Idle
  Last read 00:21:03, Last write never
  Hold time is 180, keepalive interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds

  Connections established 0; dropped 0
  Last reset 00:21:03,  No AFI/SAFI activated for peer
BGP Connect Retry Timer in Seconds: 120
Read thread: off  Write thread: off  FD used: -1

BGP neighbor is 10.255.217.36, remote AS 65000, local AS 65001, external link
  BGP version 4, remote router ID 0.0.0.0, local router ID 10.255.217.49
  BGP state = Idle
  Last read 00:21:03, Last write never
  Hold time is 180, keepalive interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds

  Connections established 0; dropped 0
  Last reset 00:21:03,  No AFI/SAFI activated for peer
BGP Connect Retry Timer in Seconds: 120
Read thread: off  Write thread: off  FD used: -1

BGP neighbor is 10.255.217.51, remote AS 65000, local AS 65001, external link
  BGP version 4, remote router ID 0.0.0.0, local router ID 10.255.217.49
  BGP state = Idle
  Last read 00:21:03, Last write never
  Hold time is 180, keepalive interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds

  Connections established 0; dropped 0
  Last reset 00:21:03,  No AFI/SAFI activated for peer
BGP Connect Retry Timer in Seconds: 120
Read thread: off  Write thread: off  FD used: -1

BGP neighbor is 10.255.217.52, remote AS 65000, local AS 65001, external link
  BGP version 4, remote router ID 0.0.0.0, local router ID 10.255.217.49
  BGP state = Idle
  Last read 00:21:03, Last write never
  Hold time is 180, keepalive interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*
    Remote GR Mode: NotApplicable
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds

  Connections established 0; dropped 0
  Last reset 00:21:03,  No AFI/SAFI activated for peer
BGP Connect Retry Timer in Seconds: 120
Read thread: off  Write thread: off  FD used: -1

I would have expected the summary command to show the peers?

I’ve uploaded a packet capture with everything on tcp.179 here.

Any help would be greatly appreciated.

Cheers,
Kane.

Nikolay · November 1, 2021, 2:42am

Hi @kcslb92
Looks like AF commands are missing:
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast
Can you provide the BGP commands?
sh config commands | match bgp

kcslb92 · November 1, 2021, 2:56am

Hi @Nikolay,

set protocols bgp local-as '65001'
set protocols bgp neighbor 10.255.217.35 remote-as '65000'
set protocols bgp neighbor 10.255.217.36 remote-as '65000'
set protocols bgp neighbor 10.255.217.51 remote-as '65000'
set protocols bgp neighbor 10.255.217.52 remote-as '65000'

I have now added the address family statement, that was a complete overlook on my part; my apologies for wasting your time .

set protocols bgp local-as '65001'
set protocols bgp neighbor 10.255.217.35 address-family ipv4-unicast
set protocols bgp neighbor 10.255.217.35 remote-as '65000'
set protocols bgp neighbor 10.255.217.36 address-family ipv4-unicast
set protocols bgp neighbor 10.255.217.36 remote-as '65000'
set protocols bgp neighbor 10.255.217.51 address-family ipv4-unicast
set protocols bgp neighbor 10.255.217.51 remote-as '65000'
set protocols bgp neighbor 10.255.217.52 address-family ipv4-unicast
set protocols bgp neighbor 10.255.217.52 remote-as '65000'

It appears to now be working, I will report back here if I have any more issues

Thank you!

Nikolay · November 1, 2021, 3:01am

Good news, @kcslb92
Have a great day!

system · November 3, 2021, 3:01am

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.