Hi has anyone tried using Cisco TREX here to test throughput of their VYOS instance? What were the results? What’s the maximum UDP PPS you can pass through VYOS that is DUT?
I noticed when I do my test, even at low PPS, the drop rate always starts at 10-sec test mark. While if I test my Juniper MX80, it doesn’t drop anything.
I have VYOS on bare-metal, Dell PowerEdge R320
I have used it for testing on mine. Running a recent build of 1.4 on a supermicro X10SLH-N6-ST031 with 32G of ram and a E3-1230v3 i was able to push around 10M pps bidirectional before seeing packet loss with basic offloading enabled on the interfaces. Was just basic routing, no ACLs ect on the interfaces.
I have seen similar behavior in testing with trex when the routing is misconfigured on the DUT. Have you done tcpdumps on the interfaces of the DUT to see if any traffic is actually being routed?
Hi Kyle, do you see the same problem wherein the drop rate started to happen after few seconds of test? Mine starts at exactly 10 seconds. I will try doing tcpdump on outbound interface on the DUT, but I believe it is still routing because, I see few RX traffic on the Trex side.
At first I thought it was the TREX, because I tested it on VYOS running on bare metal, and Mikrotik CHR running on VMware and I see results. But when I pushed the same traffic over our Juniper MX80, I don’t see drop rate until 2 or 3 mins or so of testing.
I tested it. The DUT really stops forwarding traffic at 10 second mark.
Tested on UDP (including DNS)
Tested using TCP (smtp), 2 way flow is still ongoing even after 10 second mark but there’s a huge drop on traffic.
You can read this thread https://forum.vyos.io/t/udp-pkt-s-performance-issue/
hey man,
I tried executing
nft add flowtable ip filter f { hook ingress priority 0; devices = { eth4, eth2 }; counter; }
nft add chain ip filter forward { type filter hook forward priority 0; }
nft add rule ip filter forward ip protocol udp flow add @f
It doesn’t help. I get packet drops at 10sec test mark…
Iv run it for hours without issue. Il see if i cant post the configs here for you in a bit.
thanks man, looking forward. I’ve switched to 1.4 and I get the same issue 
So went through my notes and there is one thing i did have to do to prevent the issue you all are seeing. If the trex ports are not in service mode they do not respond to arp requests which will cause vyos to show the port as failed and not know where to send the traffic like below. As to why you dont see this issue on other devices i have no idea, maybe vyos doesnt retain the arp entry as long?
[email protected]:~$ show arp
Address Interface Link layer address State
--------------- ----------- -------------------- ----------
172.16.0.2 eth2 0c:c4:7a:58:21:ac STALE
23.136.233.33 bond0 00:1c:73:00:00:99 REACHABLE
192.168.253.106 eth6 FAILED
192.168.254.106 eth7 INCOMPLETE
23.136.233.43 bond0 12:fd:55:f2:eb:23 REACHABLE
The solution to this is to manually define the arp entry so we dont need trex to respond to arp. (Or force the tests to run in service mode but it is my understanding that doing so limits what trex can handle)
set protocols static arp interface eth6 address 192.168.253.106 mac '0c:c4:7a:58:21:b0'
set protocols static arp interface eth7 address 192.168.254.106 mac '0c:c4:7a:58:21:b1'
The Tests:
Im running trex 3.0.2 on debian 11.
The DUT is VyOS 1.4-rolling-202303260706.
Both the trex box and vyos box are identical hardware wise.
Motherboard: X10SLH-N6-ST031
CPU: E3-1230 v3
RAM: 4x Micron DDR 1600 8G ECC
Extra NIC: HP NC552SFP (Not used for testing)
All tests are done via one of the intel X540T2 nics onboard.
This is a small test that should give you around 1M PPS without over taxing the DUT.
start -f stl/udp_multi_simple_list_test.py -p0 -m 1200mbps
trex>stats
Global Statistics
connection : localhost, Port 4501 total_tx_L2 : 1.19 Gbps
version : STL @ v3.02 total_tx_L1 : 1.37 Gbps
cpu_util. : 3.49% @ 6 cores (6 per dual port) total_rx : 1.09 Gbps
rx_cpu_util. : 0.0% / 0 pps total_pps : 1.12 Mpps
async_util. : 0% / 4.47 bps drop_rate : 0 bps
total_cps. : 0 cps queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | IDLE |
speed | 10 Gb/s | 10 Gb/s |
CPU util. | 3.49% | 0.0% |
-- | | |
Tx bps L2 | 1.19 Gbps | 0 bps | 1.19 Gbps
Tx bps L1 | 1.37 Gbps | 0 bps | 1.37 Gbps
Tx pps | 1.12 Mpps | 0 pps | 1.12 Mpps
Line Util. | 13.75 % | 0 % |
--- | | |
Rx bps | 0 bps | 1.09 Gbps | 1.09 Gbps
Rx pps | 0 pps | 1.03 Mpps | 1.03 Mpps
---- | | |
opackets | 361046051 | 0 | 361046051
ipackets | 0 | 142594067 | 142594067
obytes | 48019124783 | 0 | 48019124783
ibytes | 0 | 18965011044 | 18965011044
tx-pkts | 361.05 Mpkts | 0 pkts | 361.05 Mpkts
rx-pkts | 0 pkts | 142.59 Mpkts | 142.59 Mpkts
tx-bytes | 48.02 GB | 0 B | 48.02 GB
rx-bytes | 0 B | 18.97 GB | 18.97 GB
----- | | |
oerrors | 0 | 0 | 0
ierrors | 0 | 0 | 0
VyOS Config
[email protected]# show
interfaces {
bonding bond0 {
address 23.136.233.58/27
hash-policy layer3+4
member {
interface eth0
interface eth1
}
mode 802.3ad
mtu 1400
}
ethernet eth0 {
hw-id 40:a8:f0:b3:f5:78
offload {
gro
gso
rps
sg
tso
}
}
ethernet eth1 {
hw-id 40:a8:f0:b3:f5:7c
offload {
gro
gso
rps
sg
tso
}
}
ethernet eth2 {
address 172.16.0.1/16
hw-id 0c:c4:7a:1f:97:7a
offload {
gro
gso
rps
sg
tso
}
}
ethernet eth3 {
disable
hw-id 0c:c4:7a:1f:97:7b
offload {
gro
gso
rps
sg
tso
}
}
ethernet eth4 {
hw-id 0c:c4:7a:1f:97:7c
}
ethernet eth5 {
hw-id 0c:c4:7a:1f:97:7d
}
ethernet eth6 {
address 192.168.253.107/24
hw-id 0c:c4:7a:1f:97:7e
ipv6 {
address {
autoconf
}
}
offload {
gro
gso
rfs
rps
sg
tso
}
vrf testing
}
ethernet eth7 {
address 192.168.254.107/24
hw-id 0c:c4:7a:1f:97:7f
ipv6 {
address {
autoconf
}
}
offload {
gro
gso
rfs
rps
sg
tso
}
vrf testing
}
loopback lo {
}
}
nat {
source {
rule 100 {
description testing
outbound-interface bond0
source {
address 172.16.0.0/16
}
translation {
address masquerade
}
}
}
}
protocols {
static {
arp {
interface eth6 {
address 192.168.253.106 {
mac 0c:c4:7a:58:21:b0
}
}
interface eth7 {
address 192.168.254.106 {
mac 0c:c4:7a:58:21:b1
}
}
}
route 0.0.0.0/0 {
next-hop 23.136.233.33 {
}
}
route 0.0.0.0/24 {
}
}
}
service {
lldp {
interface all {
}
}
ntp {
allow-client {
address 0.0.0.0/0
address ::/0
}
server time1.vyos.net {
}
server time2.vyos.net {
}
server time3.vyos.net {
}
}
ssh {
}
}
system {
config-management {
commit-revisions 100
}
conntrack {
modules {
ftp
h323
nfs
pptp
sip
sqlnet
tftp
}
}
console {
device ttyS0 {
speed 115200
}
}
host-name test1.serverforge.org
login {
user vyos {
authentication {
encrypted-password $6$g2Q7nR.B$7DpwnQQw3AGqwMiSR8qT6dIw3HkKFOviffmL1dkAAxB8TA4ksWOv.HrIzKbtlbOB2XxIbFm6o/dXPbwFefue2.
plaintext-password ""
public-keys idenifier {
key AAAAB3NzaC1yc2EAAAADAQABAAABAQC03wh5YEajBF5ZyfP/OYG5tJwyxdHxgNhySHd9IknCN2ANcg2ncEH6nVbS42QoYTPoQ11HMItiyn4u16FIMRzo/kVn/xsfZ7NdAiaKwvOqFc3X/EKEx75+BCFefbbAsdRkBNdCXp8aeSTGmeidw7B4mRI3PHwjF+YV0S6SOZBbX51uskWOe3UzoEEMqdY7WjgGhy7dU4Dk5uhTNdt3tFnM9PG9ywwkqHMbIx1CAjvEPntnFVcNRTQ434Ydjl2unRbDD/3MFPPsIjR8KtFkgdERhkw46tizevV8C30wlwD4LSb6mtoVHtuVlmgxf4ONRn1GN5cjOIr1Bi5z6N8B90oJ
type ssh-rsa
}
}
}
}
name-server 1.1.1.1
option {
performance throughput
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
}
vrf {
name testing {
protocols {
static {
route 16.0.0.0/8 {
next-hop 192.168.253.106 {
interface eth6
}
}
route 48.0.0.0/8 {
next-hop 192.168.254.106 {
interface eth7
}
}
}
}
table 100
}
}
TREX Config
root@yeet-cannon:/opt/v3.02# cat /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###
- version: 2
interfaces: ['06:00.0', '06:00.1']
port_info:
- ip: 192.168.253.106
default_gw: 192.168.253.107
- ip: 192.168.254.106
default_gw: 192.168.254.107
platform:
master_thread_id: 0
latency_thread_id: 1
dual_if:
- socket: 0
threads: [2,3,4,5,6,7]
that is such a good find. and yes it was the arp entry that’s causing my test to drop.
On my juniper MX, I have statically set the arp to timeout at 60seconds, which is why I can get to test up to 60 seconds w/out any issue.
I have statically set the arp now on vyos and it works fine. thank you so much for sharing your experience!