Lets Discuss optimization VYOS in server with dual Xeon

Talks
, , , , ,
1

Hello Everyone

Before i must say sorry if my english bad, and i hope you can understand what i’am say in this discusstion.

I Want to share my expirience use VYOS handle 30++ Gbps with 2nd server dell630 with Xeon(R) CPU E5-2673 ( total setup under $2000 ).and i want your opinion about this.

Total throughput ( IN + OUT )

30Gbps

Runing only BGP VLAN

Device Information ::::

Processor

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 40
On-line CPU(s) list: 0-39
Thread(s) per core: 1
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz

Network card

04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
05:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
05:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)

OS

ersion: VyOS 1.3.4
Release train: equuleus

Built by: xxxxxxxx@gmail.com
Built on: Sun 18 Feb 2024 13:34 UTC
Build UUID: 86b07879-5593-470d-bf9c-f49abc06426a
Build commit ID: 1488e906e26d6b

Architecture: x86_64
Boot via: installed image
System type: bare metal

Hardware vendor: Dell Inc.
Copyright: VyOS maintainers and contributors

My Kernel Optimization

sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728
sysctl -w net.ipv4.tcp_rmem=“4096 87380 67108864”
sysctl -w net.ipv4.tcp_wmem=“4096 65536 67108864”
sysctl -w net.core.default_qdisc=fq

sysctl -w net.core.rmem_max=“223232”
sysctl -w net.core.rmem_default=31457280
sysctl -w net.core.rmem_max=33554432
sysctl -w net.core.wmem_default=31457280
sysctl -w net.core.wmem_max=33554432

sysctl -w net.core.somaxconn=65535
sysctl -w net.core.netdev_max_backlog=65536
sysctl -w net.core.optmem_max=25165824
sysctl -w net.ipv4.tcp_mem=“786432 1048576 26777216”
sysctl -w net.ipv4.udp_mem=“65536 131072 262144”
sysctl -w net.ipv4.tcp_rmem=“8192 87380 33554432”
sysctl -w net.ipv4.udp_rmem_min=16384
sysctl -w net.ipv4.tcp_wmem=“8192 65536 33554432”
sysctl -w net.ipv4.udp_wmem_min=16384
sysctl -w net.ipv4.tcp_max_tw_buckets=1440000
sysctl -w net.ipv4.tcp_tw_reuse=2

echo 134217728 > /proc/sys/net/core/rmem_max
echo 134217728 > /proc/sys/net/core/wmem_max
echo 6711296 > /proc/sys/net/core/rmem_default
echo 6711296 > /proc/sys/net/core/wmem_default
echo 40960 > /proc/sys/net/core/optmem_max

Refference

My Firewall

firewall {
all-ping enable
broadcast-ping disable
config-trap disable
ipv6-receive-redirects disable
ipv6-src-route disable
ip-src-route disable
log-martians enable
receive-redirects disable
send-redirects disable
source-validation disable
syn-cookies enable
twa-hazards-protection disable
}

INCREASE BUFFER RING AND RX TX

In this device i use ETH7 for INPUT and ETH8 for OUTPUT ( QSFP+ )

Ring parameters for eth7:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192

ethtool -g eth8
Ring parameters for eth8:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192

Refference

Allocation SMP Affinity ETH7 AND ETH8 To One Processor ( CPU0-CPU19 )

I’ve read in forum if Traffict cross dual processor will reduce performance but I forgot where to read it so sorry I can’t attach the reference.

For allocation SMP Affinity you can read in this link

problems I got while use this setup :::

  • IF ENBALE OFFLOAD will Intermitten ( Maybe because Cross Procesor ?? Idk )
  • Only Processor 1 work, if we allocation SMP Affinity to Processor 2 if will intermitten ( after traffcit exceed 10Gbps )
  • Some Times 1 CPU 100% and its randomly ( in top ksoftirqd/x )
  • some times, for save config or commit will take 30 second

If you’ve idea how to use second processor feel free to reply this thread for discusstion / send PM