Problem after update to latest rolling

merlinios · February 15, 2021, 12:20pm

Hello all ,

I have upgrade yesterday one of my 2 route reflector servers from rolling 1.2.xxx to latest 1.4-rolling-202102130218 . It seems that it start to have packet loss every 30-40 icmp packets . In the logs i see the below event ,

Feb 15 14:09:00 ef-1-routeserver2 ntpd[2015]: routing socket reports: No buffer space available
Feb 15 14:09:00 ef-1-routeserver2 ntpd[2015]: message repeated 2 times: [ routing socket reports: No buffer space available]
Feb 15 14:09:00 ef-1-routeserver2 ntpd[2015]: routing socket reports: No buffer space available
Feb 15 14:09:01 ef-1-routeserver2 ntpd[2015]: routing socket reports: No buffer space available
Feb 15 14:09:01 ef-1-routeserver2 ntpd[2015]: message repeated 16 times: [ routing socket reports: No buffer space available]
Feb 15 14:09:01 ef-1-routeserver2 ntpd[2015]: routing socket reports: No buffer space available
Feb 15 14:09:02 ef-1-routeserver2 ntpd[2015]: message repeated 3 times: [ routing socket reports: No buffer space available]
Feb 15 14:09:02 ef-1-routeserver2 ntpd[2015]: routing socket reports: No buffer space available
Feb 15 14:09:02 ef-1-routeserver2 ntpd[2015]: message repeated 8 times: [ routing socket reports: No buffer space available]

Any ideas ?

Dmitry · February 15, 2021, 12:55pm

Hello @merlinios, is it bare-metal or VM? Could you show configuration?

show configuration commands | strip-private

merlinios · February 15, 2021, 1:04pm

It is a bare-metal installation ,

superadmin@ef-1-routeserver2.interworks.cloud:~$ show configuration commands | strip-private
set interfaces bonding bond0 description ‘Trunk_to_Z9100’
set interfaces bonding bond0 hash-policy ‘layer2’
set interfaces bonding bond0 member interface ‘eth2’
set interfaces bonding bond0 member interface ‘eth3’
set interfaces bonding bond0 mode ‘802.3ad’
set interfaces bonding bond0 vif 702
set interfaces bonding bond0 vif 704
set interfaces bonding bond0 vif 707 address ‘xxx.xxx.79.185/30’
set interfaces bonding bond0 vif 707 address ‘xxxx:xxxx:1::a/126’
set interfaces bonding bond0 vif 708 address ‘xxx.xxx.79.189/30’
set interfaces bonding bond0 vif 708 address ‘xxxx:xxxx:1::e/126’
set interfaces bonding bond0 vif 999 address ‘xxx.xxx.0.4/29’
set interfaces ethernet eth0 address ‘xxx.xxx.0.42/24’
set interfaces ethernet eth0 description ‘Management’
set interfaces ethernet eth0 duplex ‘auto’
set interfaces ethernet eth0 hw-id ‘XX:XX:XX:XX:XX:e0’
set interfaces ethernet eth0 speed ‘auto’
set interfaces ethernet eth1 duplex ‘auto’
set interfaces ethernet eth1 hw-id ‘XX:XX:XX:XX:XX:e1’
set interfaces ethernet eth1 speed ‘auto’
set interfaces ethernet eth2 duplex ‘auto’
set interfaces ethernet eth2 hw-id ‘XX:XX:XX:XX:XX:50’
set interfaces ethernet eth2 speed ‘auto’
set interfaces ethernet eth3 duplex ‘auto’
set interfaces ethernet eth3 hw-id ‘XX:XX:XX:XX:XX:52’
set interfaces ethernet eth3 speed ‘auto’
set interfaces loopback lo address ‘xxx.xxx.100.4/32’
set interfaces loopback lo address ‘xxxx:xxxx::4/128’
set protocols bgp XXXXXX neighbor xxx.xxx.100.5 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.100.5 remote-as ‘50919’
set protocols bgp XXXXXX neighbor xxx.xxx.100.5 update-source ‘xxx.xxx.100.4’
set protocols bgp XXXXXX neighbor xxx.xxx.100.6 address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp XXXXXX neighbor xxx.xxx.100.6 remote-as ‘50919’
set protocols bgp XXXXXX neighbor xxx.xxx.100.6 update-source ‘xxx.xxx.100.4’
set protocols bgp XXXXXX neighbor xxxx:xxxx::1 address-family ipv6-unicast
set protocols bgp XXXXXX neighbor xxxx:xxxx::1 remote-as ‘50919’
set protocols bgp XXXXXX neighbor xxxx:xxxx::1 update-source ‘xxxx:xxxx::4’
set protocols bgp XXXXXX neighbor xxxx:xxxx::2 address-family ipv6-unicast
set protocols bgp XXXXXX neighbor xxxx:xxxx::2 remote-as ‘50919’
set protocols bgp XXXXXX neighbor xxxx:xxxx::2 update-source ‘xxxx:xxxx::4’
set protocols ospf area 0 network ‘xxx.xxx.0.0/29’
set protocols ospf area 0 network ‘xxx.xxx.100.4/32’
set protocols ospf area 0 network ‘xxx.xxx.79.188/30’
set protocols ospf area 0 network ‘xxx.xxx.79.184/30’
set protocols ospf parameters abr-type ‘cisco’
set protocols ospf parameters router-id ‘xxx.xxx.100.4’
set protocols ospf passive-interface ‘default’
set protocols ospf passive-interface-exclude ‘bond0.999’
set protocols ospf passive-interface-exclude ‘bond0.708’
set protocols ospf passive-interface-exclude ‘bond0.707’
set protocols ospfv3 area xxx.xxx.0.0 interface bond0.707
set protocols ospfv3 area xxx.xxx.0.0 interface bond0.708
set protocols ospfv3 area xxx.xxx.0.0 range xxxx:xxxx:1::8/126
set protocols ospfv3 area xxx.xxx.0.0 range xxxx:xxxx:1::c/126
set protocols ospfv3 parameters router-id ‘xxx.xxx.100.4’
set protocols ospfv3 redistribute connected
set protocols static route xxx.xxx.2.0/24 next-hop xxx.xxx.0.120
set protocols static route xxx.xxx.40.0/24 next-hop xxx.xxx.0.120
set service snmp community interworks authorization ‘ro’
set service snmp community interworks network ‘xxx.xxx.2.0/24’
set service ssh port ‘2222’
set system config-management commit-revisions ‘20’
set system console device ttyS0 speed ‘9600’
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system name-server ‘xxx.xxx.8.8’
set system name-server ‘xxx.xxx.4.4’
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system syslog global facility all level ‘notice’
set system syslog global facility protocols level ‘debug’
set system syslog host xxx.xxx.2.130 facility all level ‘warning’
set system time-zone ‘Europe/Athens’

Dmitry · February 15, 2021, 1:11pm

Could you show an output of the command sudo top and press 1
Also, it will be interesting to get NICs information

show interfaces ethernet eth0 physical 
show interfaces ethernet eth1 physical 
show interfaces ethernet eth2 physical 
show interfaces ethernet eth3 physical

And check sudo dmesg output, try to confirm that this is not conntrack table overflow issue

merlinios · February 15, 2021, 2:05pm

Ofcourse no problem

top - 15:13:27 up  2:04,  2 users,  load average: 3.44, 3.18, 3.12
Tasks: 127 total,   3 running, 124 sleeping,   0 stopped,   0 zombie
%Cpu0  : 54.9 us, 19.5 sy,  0.0 ni, 12.7 id,  0.0 wa,  0.0 hi, 12.9 si,  0.0 st
%Cpu1  : 30.6 us, 23.8 sy,  0.0 ni, 33.1 id,  0.0 wa,  0.0 hi, 12.5 si,  0.0 st
%Cpu2  : 57.7 us, 13.0 sy,  0.0 ni, 21.9 id,  0.0 wa,  0.0 hi,  7.4 si,  0.0 st
%Cpu3  : 39.7 us, 21.9 sy,  0.0 ni, 30.3 id,  0.0 wa,  0.0 hi,  8.1 si,  0.0 st
MiB Mem :  15851.6 total,  12453.4 free,   3016.9 used,    381.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  12296.8 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                   
  991 frr       20   0 1619224 587804   5492 R 155.4   3.6 188:29.44 zebra                                                                     
 1039 frr       20   0   12100   4048   1836 S  66.1   0.0  80:32.70 ldpd                                                                      
  999 frr       20   0 2284264   2.0g   7892 R  51.6  13.0  61:27.32 bgpd                                                                      
 2015 ntp       20   0   76472   3164   2644 S  18.7   0.0  22:10.13 ntpd                                                                      
  674 root      20   0   63452  26268  23072 S   0.5   0.2   0:35.73 systemd-journal                                                           
   10 root      20   0       0      0      0 S   0.3   0.0   0:19.14 ksoftirqd/0                                                               
   11 root      20   0       0      0      0 I   0.3   0.0   0:04.13 rcu_sched                                                                 
   17 root      20   0       0      0      0 S   0.3   0.0   0:19.84 ksoftirqd/1                                                               
   22 root      20   0       0      0      0 S   0.3   0.0   0:18.04 ksoftirqd/2                                                               
   27 root      20   0       0      0      0 S   0.3   0.0   0:17.90 ksoftirqd/3                                                               
    1 root      20   0  170604  10532   7984 S   0.0   0.1   0:02.24 systemd                                                                   
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd                                                                  
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                                                                    
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp                                                                
    6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri                                               
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq                                                              
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace                                                           
   12 root      rt   0       0      0      0 S   0.0   0.0   0:00.01 migration/0                                                               
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0                                                                   
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1                                                                   
   16 root      rt   0       0      0      0 S   0.0   0.0   0:00.08 migration/1                                                               
   19 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/1:0H-events_highpri                                               
   20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2                                                                   
   21 root      rt   0       0      0      0 S   0.0   0.0   0:00.08 migration/2                                                               
   23 root      20   0       0      0      0 I   0.0   0.0   0:00.09 kworker/2:0-mm_percpu_wq                                                  
   24 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/2:0H-events_highpri                                               
   25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3                                                                   
   26 root      rt   0       0      0      0 S   0.0   0.0   0:00.08 migration/3                                                               
   29 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/3:0H-events_highpri                                               
   30 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmpfs                                                                 
   31 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns                                                                     
   32 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kauditd                                                                   
   33 root      20   0       0      0      0 S   0.0   0.0   0:00.00 khungtaskd                                                                
   34 root      20   0       0      0      0 S   0.0   0.0   0:00.00 oom_reaper                                                                
   35 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 writeback                                                                 
   36 root      20   0       0      0      0 S   0.0   0.0   0:00.06 kcompactd0                                                                
   37 root      25   5       0      0      0 S   0.0   0.0   0:00.00 ksmd                                                                      
   38 root      39  19       0      0      0 S   0.0   0.0   0:00.61 khugepaged                                                                
   43 root      20   0       0      0      0 I   0.0   0.0   0:00.06 kworker/1:1-cgroup_destroy                                                
   58 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kintegrityd                                                               
   59 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kblockd                                                                   
   60 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 blkcg_punt_bio                                                            
   61 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 md                                                                        
   62 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 edac-poller                                                               
   63 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 devfreq_wq                                                                
   64 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 watchdogd                                                                 
   66 root       0 -20       0      0      0 I   0.0   0.0   0:00.01 kworker/3:1H-kblockd                                                      
   68 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kswapd0                                                                   
   69 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kthrotld                                                                  
   70 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/25-aerdrv                                                             
   71 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/26-aerdrv                                                             
   72 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/27-aerdrv                                                             
   73 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/28-aerdrv                                                             
   74 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 irq/29-aerdrv                                                             
   75 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 ipv6_addrconf                                                             
   79 root       0 -20       0      0      0 I   0.0   0.0   0:00.01 kworker/0:1H-kblockd                                                      
  144 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 i40e                                                                      
  145 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_0                                                                 
  146 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_0                                                                
  147 root      20   0       0      0      0 I   0.0   0.0   0:00.36 kworker/1:2-events_power_efficient                                        
  148 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 ata_sff                                                                   
  150 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_1                                                                 
  151 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_1                                                                
  152 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_2                                                                 
  153 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_2                                                                
  154 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_3                                                                 
  155 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_3                                                                
  156 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_4                                                                 
  157 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_4                                                                
  158 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_5                                                                 
  159 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_5                                                                
  160 root      20   0       0      0      0 S   0.0   0.0   0:00.00 scsi_eh_6                                                                 
  161 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 scsi_tmf_6                                                                
  167 root      20   0       0      0      0 I   0.0   0.0   0:00.06 kworker/2:2-events                                                        
  168 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 poll_megasas0_s                                                           
  170 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/2:1H-kblockd                                                      
  175 root       0 -20       0      0      0 I   0.0   0.0   0:00.01 kworker/1:1H-kblockd                                                      
  268 root      20   0       0      0      0 S   0.0   0.0   0:00.06 jbd2/sda1-8                                                               
  269 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 ext4-rsv-conver                                                           
  312 root       0 -20       0      0      0 S   0.0   0.0   0:00.21 loop0                                                                     
  677 root      20   0  308308  61652  16972 S   0.0   0.4   0:01.65 python3                                                                   
  679 root      20   0  121764  26148  13196 S   0.0   0.2   0:00.64 python3                                                                   
  685 root      20   0   21484   5580   3916 S   0.0   0.0   0:00.12 systemd-udevd                                                             
  687 root      20   0    8088   7524   1648 S   0.0   0.0   0:00.49 haveged                                                                   
  796 message+  20   0    8692   3668   3336 S   0.0   0.0   0:00.36 dbus-daemon                                                               
  803 root       0 -20    2288     72      0 S   0.0   0.0   0:00.38 atopacctd             




superadmin@ef-1-routeserver2.interworks.cloud:~$ show interfaces ethernet eth1 physical 
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                             100baseT/Half 100baseT/Full 
                                             1000baseT/Full 
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 2
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: yes
driver: tg3
version: 5.10.14-amd64-vyos
firmware-version: FFV20.6.18 bc 5720-v1.39
expansion-rom-version: 
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
superadmin@ef-1-routeserver2.interworks.cloud:~$ show interfaces ethernet eth2 physical 
Settings for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full 
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
driver: i40e
version: 5.10.14-amd64-vyos
firmware-version: 5.40 0x80002e8d 18.0.14
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
superadmin@ef-1-routeserver2.interworks.cloud:~$ show interfaces ethernet eth3 physical 
Settings for eth3:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full 
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
driver: i40e
version: 5.10.14-amd64-vyos
firmware-version: 5.40 0x80002e8d 18.0.14
expansion-rom-version: 
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
superadmin@ef-1-routeserver2.interworks.cloud:~$

merlinios · February 15, 2021, 1:28pm

Can i revert to previous version without having any problems ? I have a lot of complaints and no time for troubleshoot unfortunately

Thanks

Dmitry · February 15, 2021, 2:06pm

Sure you can, use operational mode command

 set system image default-boot vyos-1.2....

and reboot router

I guess the main issue related to i40e firmware and i40e kernel module, but on the other hand I see high cpu load by frr components, like zebra, mpls, etc.

merlinios · February 15, 2021, 3:17pm

Thanks Dmitry ! I will revert to the one that works