No buffer space available - rolling 031021

Been whacking at this all day to no avail… I’m having tons of connection reset by peer for one of my BGP neighbors also…

Mar 11 02:14:05 vyos ntpd[2893]: routing socket reports: No buffer space available
Mar 11 02:14:06 vyos ntpd[2893]: message repeated 4 times: [ routing socket reports: No buffer space available]
Mar 11 02:14:06 vyos ntpd[2893]: routing socket reports: No buffer space available
Mar 11 02:14:07 vyos ntpd[2893]: message repeated 3 times: [ routing socket reports: No buffer space available]

Suggestions/debug info from other threads (Yes, googling like mad)

(conf-strip-private)
set interfaces ethernet eth0 hw-id ‘XX:XX:XX:XX:XX:2c’
set interfaces ethernet eth1 hw-id ‘XX:XX:XX:XX:XX:2d’
set interfaces ethernet eth2 hw-id ‘XX:XX:XX:XX:XX:2e’
set interfaces ethernet eth3 hw-id ‘XX:XX:XX:XX:XX:2f’
set interfaces ethernet eth4 disable
set interfaces ethernet eth4 hw-id ‘XX:XX:XX:XX:XX:30’
set interfaces ethernet eth5 hw-id ‘XX:XX:XX:XX:XX:31’
set interfaces ethernet eth6 address ‘xxx.xxx.204.66/29’
set interfaces ethernet eth6 address ‘xxxx:xxxx:0:0:0:0:0:1/36’
set interfaces ethernet eth6 address ‘xxx.xxx.203.1/24’
set interfaces ethernet eth6 address ‘xxx.xxx.6.1/24’
set interfaces ethernet eth6 address ‘xxx.xxx.203.5/24’
set interfaces ethernet eth6 address ‘xxxx:xxxx:c010:a991::2/64’
set interfaces ethernet eth6 hw-id ‘XX:XX:XX:XX:XX:12’
set interfaces ethernet eth7 description ‘Uplink to WI’
set interfaces ethernet eth7 hw-id ‘XX:XX:XX:XX:XX:13’
set interfaces ethernet eth7 vif 314 address ‘xxx.xxx.12.35/24’
set interfaces ethernet eth7 vif 314 address ‘xxxx:xxxx:98::35/64’
set interfaces ethernet eth7 vif 314 description ‘STLIX’
set interfaces ethernet eth7 vif 713 address ‘xxx.xxx.136.17/24’
set interfaces ethernet eth7 vif 713 address ‘xxxx:xxxx:9E::17/64’
set interfaces ethernet eth7 vif 713 description ‘HOUIX’
set interfaces ethernet eth7 vif 816 address ‘xxx.xxx.7.119/25’
set interfaces ethernet eth7 vif 816 address ‘xxxx:xxxx:1B:1::119/64’
set interfaces ethernet eth7 vif 816 description ‘KCIX’
set interfaces loopback lo
set nat source rule 1 destination address ‘xxx.xxx.69.0/24’
set nat source rule 1 exclude
set nat source rule 1 outbound-interface ‘any’
set nat source rule 100 outbound-interface ‘any’
set nat source rule 100 source address ‘xxx.xxx.6.0/24’
set nat source rule 100 translation address ‘xxx.xxx.203.5’
set policy as-path-list LocalRoutes rule 10 action ‘permit’
set policy as-path-list LocalRoutes rule 10 regex ‘^' set policy as-path-list own-as rule 10 action 'permit' set policy as-path-list own-as rule 10 regex '^
set policy as-path-list own-as rule 20 action ‘deny’
set policy as-path-list own-as rule 20 regex ‘.+’
set policy route-map Out rule 10 action ‘permit’
set policy route-map Out rule 10 match as-path ‘LocalRoutes’
set policy route-map ixgw rule 2 action ‘permit’
set policy route-map ixgw rule 2 match interface ‘eth7.816’
set policy route-map ixgw rule 2 set ip-next-hop ‘xxx.xxx.7.119’
set policy route-map ixgw rule 3 action ‘permit’
set policy route-map ixgw rule 3 match interface ‘eth7.713’
set policy route-map ixgw rule 3 set ip-next-hop ‘xxx.xxx.136.17’
set policy route-map ixgw rule 4 action ‘permit’
set policy route-map ixgw rule 4 match interface ‘eth7.314’
set policy route-map ixgw rule 4 set ip-next-hop ‘xxx.xxx.12.35’
set policy route-map setmet rule 2 action ‘permit’
set policy route-map setmet rule 2 set as-path-prepend ‘2 2 2’
set protocols bgp XXXXXX address-family ipv4-unicast network xxx.xxx.203.0/24
set protocols bgp XXXXXX address-family ipv6-unicast network xxxx:xxxx::/36
set protocols bgp XXXXXX neighbor xxx.xxx.204.65 address-family ipv4-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxx.xxx.204.65 address-family ipv4-unicast route-map
set protocols bgp XXXXXX neighbor xxx.xxx.204.65 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.204.65 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.204.65 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxx.xxx.204.65 update-source ‘xxx.xxx.204.66’
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 address-family ipv4-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 address-family ipv4-unicast nexthop-self
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 address-family ipv4-unicast route-map export ‘Out’
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 peer-group ‘hurricane’
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 remote-as ‘6939’
set protocols bgp XXXXXX neighbor xxx.xxx.7.5 update-source ‘xxx.xxx.7.119’
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 address-family ipv4-unicast route-map export ‘ixgw’
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 address-family ipv4-unicast route-server-client
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 address-family ipv6-unicast route-server-client
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 description ‘KCIX Route Server’
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 remote-as ‘40542’
set protocols bgp XXXXXX neighbor xxx.xxx.7.126 update-source ‘xxx.xxx.7.119’
set protocols bgp XXXXXX neighbor xxx.xxx.12.5 address-family ipv4-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxx.xxx.12.5 address-family ipv4-unicast route-map export ‘Out’
set protocols bgp XXXXXX neighbor xxx.xxx.12.5 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.12.5 peer-group ‘hurricane’
set protocols bgp XXXXXX neighbor xxx.xxx.12.5 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxx.xxx.12.5 update-source ‘xxx.xxx.12.35’
set protocols bgp XXXXXX neighbor xxx.xxx.12.254 address-family ipv4-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxx.xxx.12.254 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.12.254 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.12.254 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxx.xxx.12.254 update-source ‘xxx.xxx.12.35’
set protocols bgp XXXXXX neighbor xxx.xxx.136.5 address-family ipv4-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxx.xxx.136.5 address-family ipv4-unicast route-map export ‘Out’
set protocols bgp XXXXXX neighbor xxx.xxx.136.5 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.136.5 peer-group ‘hurricane’
set protocols bgp XXXXXX neighbor xxx.xxx.136.5 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxx.xxx.136.5 update-source ‘xxx.xxx.136.17’
set protocols bgp XXXXXX neighbor xxx.xxx.136.254 address-family ipv4-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxx.xxx.136.254 address-family ipv4-unicast route-server-client
set protocols bgp XXXXXX neighbor xxx.xxx.136.254 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.136.254 capability dynamic
set protocols bgp XXXXXX neighbor xxx.xxx.136.254 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxx.xxx.136.254 update-source ‘xxx.xxx.136.17’
set protocols bgp XXXXXX neighbor xxxx:xxxx:1B:1::126 address-family ipv6-unicast route-server-client
set protocols bgp XXXXXX neighbor xxxx:xxxx:1B:1::126 remote-as ‘40542’
set protocols bgp XXXXXX neighbor xxxx:xxxx:1B:1::126 update-source ‘xxxx:xxxx:1B:1::119’
set protocols bgp XXXXXX neighbor xxxx:xxxx:1b:1::5 address-family ipv6-unicast
set protocols bgp XXXXXX neighbor xxxx:xxxx:1b:1::5 capability dynamic
set protocols bgp XXXXXX neighbor xxxx:xxxx:1b:1::5 remote-as ‘6939’
set protocols bgp XXXXXX neighbor xxxx:xxxx:1b:1::5 update-source ‘xxxx:xxxx:1B:1::119’
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::5 address-family ipv6-unicast
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::5 capability dynamic
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::5 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::5 update-source ‘xxxx:xxxx:9E::17’
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::254 address-family ipv6-unicast route-server-client
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::254 capability dynamic
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::254 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxxx:xxxx:9E::254 update-source ‘xxxx:xxxx:9E::17’
set protocols bgp XXXXXX neighbor xxxx:xxxx:9e::5 remote-as ‘6939’
set protocols bgp XXXXXX neighbor xxxx:xxxx:98::5 address-family ipv6-unicast
set protocols bgp XXXXXX neighbor xxxx:xxxx:98::5 capability dynamic
set protocols bgp XXXXXX neighbor xxxx:xxxx:98::5 remote-as ‘6939’
set protocols bgp XXXXXX neighbor xxxx:xxxx:98::5 update-source ‘xxxx:xxxx:98::35’
set protocols bgp XXXXXX neighbor xxxx:xxxx:c010:a991::1 address-family ipv6-unicast filter-list export ‘own-as’
set protocols bgp XXXXXX neighbor xxxx:xxxx:c010:a991::1 address-family ipv6-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxxx:xxxx:c010:a991::1 capability dynamic
set protocols bgp XXXXXX neighbor xxxx:xxxx:c010:a991::1 remote-as ‘external’
set protocols bgp XXXXXX neighbor xxxx:xxxx:c010:a991::1 update-source ‘xxxx:xxxx:c010:a991::2’
set protocols bgp XXXXXX parameters router-id ‘xxx.xxx.203.1’
set protocols bgp XXXXXX peer-group hurricane
set protocols static route xxx.xxx.0.0/18 blackhole distance ‘254’
set service router-advert interface eth6
set service router-advert interface eth6.6 default-preference ‘high’
set service router-advert interface eth6.6 hop-limit ‘64’
set service router-advert interface eth6.6 interval max ‘600’
set service router-advert interface eth6.6 name-server ‘xxxx:xxxx::1’
set service router-advert interface eth6.6 other-config-flag
set service router-advert interface eth6.6 prefix xxxx:xxxx::/36 valid-lifetime ‘259200’
set service router-advert interface eth6.6 reachable-time ‘0’
set service router-advert interface eth6.6 retrans-timer ‘0’
set service snmp community router authorization ‘ro’
set service snmp community router network ‘xxx.xxx.6.0/24’
set service snmp contact ‘aromberg@gmail.com’
set service snmp listen-address xxx.xxx.6.1 port ‘161’
set service snmp listen-address xxx.xxx.254.36
set service snmp location xxxxxx City, Mo’
set service ssh disable-password-authentication
set service ssh port ‘22’
set system config-management commit-revisions ‘100’
set system conntrack expect-table-size ‘2048’
set system conntrack hash-size ‘32768’
set system conntrack modules sip disable
set system conntrack table-size ‘262144’
set system console device ttyS0 speed ‘115200’
set system host-name xxxxxx
set system ipv6 neighbor table-size ‘32768’
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx full-name xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx key xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx type ssh-xxx
set system name-server ‘xxx.xxx.1.1’
set system ntp listen-address ‘xxx.xxx.203.1’
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system sysctl custom net.core.rmem_default value ‘425984’
set system sysctl custom net.ipv4.tcp_rmem value ‘4096 212992 6291456’
set system syslog global facility all level ‘warning’
set system syslog host xxx.xxx.6.95 facility all level ‘info’
set system syslog host xxx.xxx.6.95 facility all protocol ‘tcp’
set system syslog host xxx.xxx.6.95 port ‘514’
set vpn ipsec esp-group central-rtr-esp compression ‘disable’
set vpn ipsec esp-group central-rtr-esp lifetime ‘3600’
set vpn ipsec esp-group central-rtr-esp mode ‘tunnel’
set vpn ipsec esp-group central-rtr-esp pfs ‘enable’
set vpn ipsec esp-group central-rtr-esp proposal 1 encryption ‘aes256’
set vpn ipsec esp-group central-rtr-esp proposal 1 hash ‘sha1’
set vpn ipsec esp-group chi-rtr-esp compression ‘disable’
set vpn ipsec esp-group chi-rtr-esp lifetime ‘3600’
set vpn ipsec esp-group chi-rtr-esp mode ‘tunnel’
set vpn ipsec esp-group chi-rtr-esp pfs ‘enable’
set vpn ipsec esp-group chi-rtr-esp proposal 1 encryption ‘aes256’
set vpn ipsec esp-group chi-rtr-esp proposal 1 hash ‘sha256’
set vpn ipsec ike-group central-rtr-ike close-action ‘none’
set vpn ipsec ike-group central-rtr-ike ikev2-reauth ‘no’
set vpn ipsec ike-group central-rtr-ike key-exchange ‘ikev1’
set vpn ipsec ike-group central-rtr-ike lifetime ‘3600’
set vpn ipsec ike-group central-rtr-ike proposal 1 dh-group ‘14’
set vpn ipsec ike-group central-rtr-ike proposal 1 encryption ‘aes256’
set vpn ipsec ike-group central-rtr-ike proposal 1 hash ‘sha1’
set vpn ipsec ike-group chi-rtr-ike close-action ‘none’
set vpn ipsec ike-group chi-rtr-ike ikev2-reauth ‘no’
set vpn ipsec ike-group chi-rtr-ike key-exchange ‘ikev1’
set vpn ipsec ike-group chi-rtr-ike lifetime ‘3600’
set vpn ipsec ike-group chi-rtr-ike proposal 1 dh-group ‘14’
set vpn ipsec ike-group chi-rtr-ike proposal 1 encryption ‘aes256’
set vpn ipsec ike-group chi-rtr-ike proposal 1 hash ‘sha1’
set vpn ipsec ipsec-interfaces interface ‘eth6’
set vpn ipsec nat-networks allowed-network xxx.xxx.0.0/0
set vpn ipsec nat-traversal ‘enable’
set vpn ipsec site-to-site peer xxxxx.tld authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer xxxxx.tld authentication pre-shared-secret xxxxxx
set vpn ipsec site-to-site peer xxxxx.tld connection-type ‘initiate’
set vpn ipsec site-to-site peer xxxxx.tld ike-group ‘chi-rtr-ike’
set vpn ipsec site-to-site peer xxxxx.tld ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer xxxxx.tld local-address ‘xxx.xxx.203.1’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 allow-nat-networks ‘disable’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 allow-public-networks ‘disable’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 esp-group ‘chi-rtr-esp’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 local prefix ‘xxx.xxx.6.0/24’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 remote prefix ‘xxx.xxx.69.0/24’
set vpn ipsec site-to-site peer xxxxx.tld authentication mode ‘pre-shared-secret’
set vpn ipsec site-to-site peer xxxxx.tld authentication pre-shared-secret xxxxxx
set vpn ipsec site-to-site peer xxxxx.tld connection-type ‘initiate’
set vpn ipsec site-to-site peer xxxxx.tld ike-group ‘central-rtr-ike’
set vpn ipsec site-to-site peer xxxxx.tld ikev2-reauth ‘inherit’
set vpn ipsec site-to-site peer xxxxx.tld local-address ‘xxx.xxx.203.1’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 allow-nat-networks ‘disable’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 allow-public-networks ‘disable’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 esp-group ‘central-rtr-esp’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 local prefix ‘xxx.xxx.6.0/24’
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 remote prefix ‘xxx.xxx.69.0/24’
set vpn l2tp remote-access authentication local-users username xxxxxx password xxxxxx
set vpn l2tp remote-access authentication mode ‘local’
set vpn l2tp remote-access client-ip-pool start ‘xxx.xxx.6.160’
set vpn l2tp remote-access client-ip-pool stop ‘xxx.xxx.6.169’
set vpn l2tp remote-access gateway-address ‘xxx.xxx.6.1’
set vpn l2tp remote-access ipsec-settings authentication mode ‘pre-shared-secret’
set vpn l2tp remote-access ipsec-settings authentication pre-shared-secret xxxxxx
set vpn l2tp remote-access name-server ‘xxx.xxx.1.1’
set vpn l2tp remote-access outside-address ‘xxx.xxx.203.5’

eth6/7 are the same.

Settings for eth7:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
driver: ixgbe
version: 5.10.22-amd64-vyos
firmware-version: 0x800005b9
expansion-rom-version:
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Motherboard is a supermicro 5018D-FN8T

Hello @starblazer, did you try to increase ipv6 max route size and rmem/vmem?

configure
set system sysctl custom net.ipv6.route.max_size value 655360
set system sysctl custom net.core.wmem_max value 425984
set system sysctl custom net.core.rmem_max value 446464
commit

Applied, I’ll update if I see anything else weird.