Ping packet loss on vyos 1.5

1

Hi everyone,

I’ve recently deployed VyOS instances running on Dell PowerEdge R630
CPU: E5-2680 v4 @ 2.40GHz
VyOS is running directly on the bare metal, no virtualization layer.
I’ve packet loss after every half day
I’ve setting offload and ring buffer but the problem still happens
This problem happens on ipv4 but no problem occurred on ipv6

R-vyos:~$ sudo netstat -i
Kernel Interface table
Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0             1500  2700398   2965      0 0       2607261      0      0      0 BMRU
eth1             1500        0      0      0 0             0      0      0      0 BMU
eth2             1500        0      0      0 0             0      0      0      0 BMU
eth3             1500        0      0      0 0             0      0      0      0 BMU
eth4             1500 32999432095      0      0 0      21087339797      0      0      0 BMRU
eth5             1500 22389784423      0      0 0      33724142400      0      0      0 BMRU
lo              65536 590826158      0      0 0      590826158      0      0      0 LRU
pim6reg          1452        0      0      0 0             0      0      0      0 ORU

R-vyos:~$ sudo ethtool -i eth4
driver: mlx4_en
version: 4.0-0
firmware-version: 2.43.7028
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

 parameter net.ipv4.icmp_ratelimit {
     value 10000
 }
 parameter net.ipv4.icmp_ratemask {
     value 4120
 }

IPV4

2024-10-21_104218
2024-10-21_104218700×337 19 KB

2

Are you seeing packet loss through the router? Or just pinging the router?

3

I using smokeping (ping) to monitoring passing through the router because losses happen after long hours usually during the day (5-10 times rto) and its happen on ipv4 not ipv6.
this the configuration firewall

 global-options {
     all-ping enable
     broadcast-ping disable
     ipv6-receive-redirects disable
     ipv6-src-route disable
     ip-src-route disable
     log-martians enable
 }


 ipv4 {
     forward {
         filter {
             rule 1 {
                 action accept
                 description Allow-from-SMTP-25
                 destination {
                     port 25
                 }
                 protocol tcp
                 source {
                     group {
                         address-group Allow-SMTP-25
                     }
                 }
             }
             rule 2 {
                 action accept
                 description Allow-to-SMTP-25
                 destination {
                     group {
                         address-group Allow-SMTP-25
                     }
                     port 25
                 }
                 protocol tcp
             }
             rule 3 {
                 action drop
                 description Deny-SMTP-25-CUST
                 destination {
                     port 25
                 }
                 protocol tcp
             }
             rule 4 {
                 action accept
                 protocol icmp
             }
         }
     }
     input {
         filter {
             default-action drop
             rule 1 {
                 action accept
                 destination {
                     port 9999
                 }
                 protocol tcp
                 source {
                     group {
                         address-group IP-SSH
                     }
                 }
             }
             rule 2 {
                 action accept
                 protocol ospf
             }
             rule 3 {
                 action accept
                 protocol icmp
             }
             rule 4 {
                 action accept
                 protocol tcp
                 source {
                     port 179
                 }
             }
             rule 5 {
                 action accept
                 protocol udp
             }
         }
     }
 }
4

i’ve a lot of log like this, but I’m not sure if this is the cause

Oct 21 12:57:32 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 12:57:57 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 12:58:17 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 13:07:32 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 13:08:48 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 13:29:42 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 13:30:02 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
Oct 21 13:30:42 kernel: IPv4: martian source x.x.x.x from 0.0.0.0, on dev eth4
5

Does

ethtool -S <eth0/1/2/3/4> show you anything else of interest?

I very much doubt that net.ipv4.icmp_ratelimit will have any impact of packets being forwarded by your router.

The only thing you could try is to enable software flow offload (not the NIC offloads) and see if that helps with performance. Personally I doubt that it will, it feels more like a kernel/NIC problem than a nftables one, but it can’t hurt to at least try.

Do those martian source messages align with when you see the packet loss?

6

Thanks for your response, at first i think its a physical problems like qsfp, cable or nic but i was change the different devices (qsfp, cable) and the problem not solved. This Vyos device direct connect with juniper and i was checked no fcs error on interface.

For the note this problem if i use ipv4 and no problem with ipv6

Can you any idea to solved or mitigation this problem?

R-vyos:~$ ethtool -S eth5
NIC statistics:
     rx_packets: 23462690834
     tx_packets: 35485742154
     rx_bytes: 8554930444029
     tx_bytes: 36420991049619
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 3018
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_fifo_errors: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0


R-vyos:~$ ethtool -S eth4
NIC statistics:
     rx_packets: 34749337414
     tx_packets: 22128735162
     rx_bytes: 36484460504846
     tx_bytes: 8428891418875
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 238674
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_fifo_errors: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0


R-vyos:~$ sh version
Version:          VyOS 1.5-rolling-202409120007
Release train:    current
Release flavor:   generic

Built by:         [email protected]
Built on:         Thu 12 Sep 2024 00:07 UTC
Build UUID:       ab6885a1-1970-4021-a315-dbbce3831676
Build commit ID:  55c17f9283f32d

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  Dell Inc.
Hardware model:   PowerEdge R630
Hardware S/N:     1BMF642
Hardware UUID:    4c4c4544-0042-4d10-8046-b1c04f363432

Copyright:        VyOS maintainers and contributors