I am testing the performance of dmvpn recently, through the qat 8970 accelerator card (C62XX)
Single-threaded testing is normal：
root@vyos:/home/vyos# iperf -c 172.0.0.1
------------------------------------------------------------
Client connecting to 172.0.0.1, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 172.0.0.2 port 59340 connected with 172.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 2.09 GBytes 1.79 Gbits/sec
root@vyos:/home/vyos# show system acceleration qat device qat_dev0 flows
+------------------------------------------------+
| FW Statistics for Qat Device |
+------------------------------------------------+
| Firmware Requests [AE 0]: 43573 |
| Firmware Responses[AE 0]: 43573 |
+------------------------------------------------+
| Firmware Requests [AE 1]: 43572 |
| Firmware Responses[AE 1]: 43572 |
+------------------------------------------------+
| Firmware Requests [AE 2]: 43573 |
| Firmware Responses[AE 2]: 43573 |
+------------------------------------------------+
| Firmware Requests [AE 3]: 43573 |
| Firmware Responses[AE 3]: 43573 |
+------------------------------------------------+
| Firmware Requests [AE 4]: 43572 |
| Firmware Responses[AE 4]: 43572 |
+------------------------------------------------+
| Firmware Requests [AE 5]: 43573 |
| Firmware Responses[AE 5]: 43573 |
+------------------------------------------------+
| Firmware Requests [AE 6]: 43573 |
| Firmware Responses[AE 6]: 43573 |
+------------------------------------------------+
| Firmware Requests [AE 7]: 43572 |
| Firmware Responses[AE 7]: 43572 |
+------------------------------------------------+
| Firmware Requests [AE 8]: 43572 |
| Firmware Responses[AE 8]: 43572 |
+------------------------------------------------+
| Firmware Requests [AE 9]: 43573 |
| Firmware Responses[AE 9]: 43573 |
+------------------------------------------------+
But I used multi-threaded test to cause the following problems
Message from syslogd@vyos at Nov 10 09:08:13 ...
kernel:[ 1556.647161] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [opennhrp:3573]
Message from syslogd@vyos at Nov 10 09:08:33 ...
kernel:[ 1576.643161] watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [iperf3:4771]
Message from syslogd@vyos at Nov 10 09:08:41 ...
kernel:[ 1584.627161] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
Message from syslogd@vyos at Nov 10 09:08:41 ...
kernel:[ 1584.647161] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [opennhrp:3573]
Message from syslogd@vyos at Nov 10 09:09:09 ...
kernel:[ 1612.627161] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
Message from syslogd@vyos at Nov 10 09:09:09 ...
kernel:[ 1612.643161] watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [iperf3:4771]
Message from syslogd@vyos at Nov 10 09:09:09 ...
kernel:[ 1612.647160] watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [opennhrp:3573]
Now the system is stuck and can only be restarted，I tested the vyos 1.3 version and the same problem
tail -f /var/log/message
Nov 10 09:00:17 vyos kernel: [ 1080.627169] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
Nov 10 09:00:17 vyos kernel: [ 1080.710338] Modules linked in: drbg ansi_cprng echainiv xfrm4_mode_transport ip_gre ip_tunnel gre af_packet twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_avx_x86_64 cast5_generic cast_common ctr ecb des_generic cbc algif_skcipher camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 xcbc md4 algif_hash af_alg xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo usdm_drv(O) iptable_nat nf_nat_ipv4 ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_filter ip6table_raw ip6_tables iptable_filter xt_tcpudp xt_CT nfnetlink_cthelper nfnetlink iptable_raw ip_tables x_tables nf_nat_pptp nf_conntrack_pptp
Nov 10 09:00:17 vyos kernel: [ 1080.710360] nf_conntrack_proto_gre nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat_proto_gre nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_generic sha512_ssse3 sha512_generic nfit libnvdimm fuse x86_pkg_temp_thermal br_netfilter coretemp bridge crct10dif_pclmul crc32_pclmul stp llc ghash_clmulni_intel pcbc aesni_intel qat_c62x(O) aes_x86_64 crypto_simd cryptd intel_qat(O) glue_helper intel_cstate intel_uncore ipmi_si ipmi_devintf evdev dh_generic uio authenc intel_rapl_perf pcspkr ipmi_msghandler iTCO_wdt lpc_ich iTCO_vendor_support pcc_cpufreq mei_me mei acpi_power_meter button autofs4 usb_storage ohci_hcd ehci_hcd uhci_hcd squashfs zstd_decompress xxhash loop overlay ext4 crc16 mbcache jbd2 nls_ascii
Nov 10 09:00:17 vyos kernel: [ 1080.710386] sd_mod hid_generic usbhid hid ahci libahci libata megaraid_sas nvme mlx5_core xhci_pci nvme_core devlink i40e(O) i2c_i801 crc32c_intel xhci_hcd scsi_mod
Nov 10 09:00:17 vyos kernel: [ 1080.710395] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.19.176-amd64-vyos #1
Nov 10 09:00:17 vyos kernel: [ 1080.710398] RIP: 0010:0xffffffffc06c0f5d
Nov 10 09:00:17 vyos kernel: [ 1080.710399] Code: 89 28 89 6b 40 4c 89 ef e8 40 fb 54 cc 31 c0 5b 5d 41 5c 41 5d c3 85 ed 74 b2 41 0f b6 04 24 88 01 eb a9 48 8b 47 08 f0 ff 08 <b8> f5 ff ff ff c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 53 48
Nov 10 09:00:17 vyos kernel: [ 1080.710400] RSP: 0018:ffff8c183fa036c8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13
Nov 10 09:00:17 vyos kernel: [ 1080.710402] RAX: ffff8c182f2b5358 RBX: 00000000fffffff5 RCX: 00000000000005c4
Nov 10 09:00:17 vyos kernel: [ 1080.710402] RDX: ffff8c182f2b5358 RSI: ffff8c18173730b0 RDI: ffff8c1811198890
Nov 10 09:00:17 vyos kernel: [ 1080.710403] RBP: ffff8c18173730b0 R08: 0000000000000000 R09: 0000000000000000
Nov 10 09:00:17 vyos kernel: [ 1080.710403] R10: ffff8c182e7d2070 R11: 0000000080000000 R12: ffff8c17f9c10808
Nov 10 09:00:17 vyos kernel: [ 1080.710404] R13: ffff8c1817373000 R14: ffff8c17f9c10848 R15: ffff8c1817373010
Nov 10 09:00:17 vyos kernel: [ 1080.710405] FS: 0000000000000000(0000) GS:ffff8c183fa00000(0000) knlGS:0000000000000000
Nov 10 09:00:17 vyos kernel: [ 1080.710406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 10 09:00:17 vyos kernel: [ 1080.710406] CR2: 00007f638137b000 CR3: 00000006ed20a006 CR4: 00000000007606f0
Nov 10 09:00:17 vyos kernel: [ 1080.710407] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 10 09:00:17 vyos kernel: [ 1080.710408] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 10 09:00:17 vyos kernel: [ 1080.710408] PKRU: 55555554
Nov 10 09:00:17 vyos kernel: [ 1080.710408] Call Trace:
Nov 10 09:00:17 vyos kernel: [ 1080.710412] <IRQ>
Nov 10 09:00:17 vyos kernel: [ 1080.710413] 0xffffffffc06cce7b
Nov 10 09:00:17 vyos kernel: [ 1080.710415] 0xffffffffc077c13e
Nov 10 09:00:17 vyos kernel: [ 1080.710417] 0xffffffffc077c323
Nov 10 09:00:17 vyos kernel: [ 1080.710417] 0xffffffff8cb90db4
Nov 10 09:00:17 vyos kernel: [ 1080.710418] ? 0xffffffff8cad6886
Nov 10 09:00:17 vyos kernel: [ 1080.710419] ? 0xffffffff8cac17b8
Nov 10 09:00:17 vyos kernel: [ 1080.710419] 0xffffffff8cb911b1
Nov 10 09:00:17 vyos kernel: [ 1080.710420] 0xffffffff8cb83925
Nov 10 09:00:17 vyos kernel: [ 1080.710421] ? 0xffffffff8cb83690
Nov 10 09:00:17 vyos kernel: [ 1080.710421] 0xffffffff8cb788c0
Nov 10 09:00:17 vyos kernel: [ 1080.710423] 0xffffffffc08dc417
Nov 10 09:00:17 vyos kernel: [ 1080.710425] 0xffffffffc08e240d
Nov 10 09:00:17 vyos kernel: [ 1080.710426] 0xffffffff8cad7229
Nov 10 09:00:17 vyos kernel: [ 1080.710426] 0xffffffff8cad7abd
Nov 10 09:00:17 vyos kernel: [ 1080.710427] ? 0xffffffff8cac17b8
Nov 10 09:00:17 vyos kernel: [ 1080.710428] ? 0xffffffffc08e1ccf
Nov 10 09:00:17 vyos kernel: [ 1080.710429] ? 0xffffffff8cb2a474
Nov 10 09:00:17 vyos kernel: [ 1080.710429] 0xffffffff8cb2a474
Nov 10 09:00:17 vyos kernel: [ 1080.710430] 0xffffffff8cb2c92c
Nov 10 09:00:17 vyos kernel: [ 1080.710431] ? 0xffffffff8cb2b180
Nov 10 09:00:17 vyos kernel: [ 1080.710431] 0xffffffff8cb2c3c1
Nov 10 09:00:17 vyos kernel: [ 1080.710432] 0xffffffff8cb466ef
Nov 10 09:00:17 vyos kernel: [ 1080.710433] 0xffffffff8cb47e09
Nov 10 09:00:17 vyos kernel: [ 1080.710434] 0xffffffff8cb48cc8
Nov 10 09:00:17 vyos kernel: [ 1080.710434] 0xffffffff8cb434bd
Nov 10 09:00:17 vyos kernel: [ 1080.710435] 0xffffffff8cb4f1d1
Nov 10 09:00:17 vyos kernel: [ 1080.710436] 0xffffffff8cb50546
Nov 10 09:00:17 vyos kernel: [ 1080.710437] 0xffffffff8cb26bee
Nov 10 09:00:17 vyos kernel: [ 1080.710438] 0xffffffff8cb27206
Nov 10 09:00:17 vyos kernel: [ 1080.710438] ? 0xffffffff8cb26b50
Nov 10 09:00:17 vyos kernel: [ 1080.710439] 0xffffffff8cb272cd
Nov 10 09:00:17 vyos kernel: [ 1080.710439] ? 0xffffffff8cb26d30
Nov 10 09:00:17 vyos kernel: [ 1080.710440] 0xffffffff8cad8ea5
Nov 10 09:00:17 vyos kernel: [ 1080.710441] 0xffffffff8cad8f6a
Nov 10 09:00:17 vyos kernel: [ 1080.710442] 0xffffffff8cad9d1c
Nov 10 09:00:17 vyos kernel: [ 1080.710442] 0xffffffff8cb083b5
Nov 10 09:00:17 vyos kernel: [ 1080.710443] 0xffffffff8cad94ee
Nov 10 09:00:17 vyos kernel: [ 1080.710444] 0xffffffff8d0000fb
Nov 10 09:00:17 vyos kernel: [ 1080.710445] 0xffffffff8c66aed3
Nov 10 09:00:17 vyos kernel: [ 1080.710446] 0xffffffff8ce019f5
Nov 10 09:00:17 vyos kernel: [ 1080.710446] 0xffffffff8ce00a0f
Nov 10 09:00:17 vyos kernel: [ 1080.710448] </IRQ>
Nov 10 09:00:17 vyos kernel: [ 1080.710448] RIP: 0010:0xffffffff8cc1079e
Nov 10 09:00:17 vyos kernel: [ 1080.710449] Code: 08 0f 84 5f ff ff ff eb aa 90 90 90 e9 07 00 00 00 0f 00 2d 96 6e 5f 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d 86 6e 5f 00 fb f4 <c3> 90 53 e8 5a 9f ab ff 65 8b 05 fb e9 3f 73 fb 66 0f 1f 44 00 00
Nov 10 09:00:17 vyos kernel: [ 1080.710450] RSP: 0018:ffffffff8d403e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffc7
Nov 10 09:00:17 vyos kernel: [ 1080.710451] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8c183fa1a680
Nov 10 09:00:17 vyos kernel: [ 1080.710451] RDX: ffffffff8d4294b8 RSI: ffff8c183fa1a680 RDI: 000000f6f11b57fc
Nov 10 09:00:17 vyos kernel: [ 1080.710452] RBP: ffffffff8d413500 R08: ffffff48e83ec8d6 R09: 0000000000000000
Nov 10 09:00:17 vyos kernel: [ 1080.710452] R10: 0000000000000000 R11: 000000000000001e R12: 0000000000000000
Nov 10 09:00:17 vyos kernel: [ 1080.710453] R13: 0000000000000000 R14: ffffffff8d413500 R15: ffffffff8d413500
Nov 10 09:00:17 vyos kernel: [ 1080.710454] 0xffffffff8cc105ec
Nov 10 09:00:17 vyos kernel: [ 1080.710455] 0xffffffff8c6929b0
Nov 10 09:00:17 vyos kernel: [ 1080.710455] 0xffffffff8c692c3a
Nov 10 09:00:17 vyos kernel: [ 1080.710456] 0xffffffff8d74de94
Nov 10 09:00:17 vyos kernel: [ 1080.710457] 0xffffffff8c6000d4
Nov 10 09:00:19 vyos kernel: [ 1082.219168] rcu: INFO: rcu_sched self-detected stall on CPU
Nov 10 09:00:19 vyos kernel: [ 1082.285731] rcu: 9-....: (1 GPs behind) idle=b52/1/0x4000000000000002 softirq=4757/4757 fqs=2630
Nov 10 09:00:19 vyos kernel: [ 1082.392747] rcu: (t=5293 jiffies g=63873 q=4672)
Nov 10 09:00:19 vyos kernel: [ 1082.449977] NMI backtrace for cpu 9
Try to disable hyper-threading in BIOS
Disable hyper-threading is still useless
root@vyos:/home/vyos# iperf3 -c 172.0.0.1 -P 10
Connecting to host 172.0.0.1, port 5201
[ 4] local 172.0.0.2 port 56396 connected to 172.0.0.1 port 5201
[ 6] local 172.0.0.2 port 56398 connected to 172.0.0.1 port 5201
[ 8] local 172.0.0.2 port 56400 connected to 172.0.0.1 port 5201
[ 10] local 172.0.0.2 port 56402 connected to 172.0.0.1 port 5201
[ 12] local 172.0.0.2 port 56404 connected to 172.0.0.1 port 5201
[ 14] local 172.0.0.2 port 56406 connected to 172.0.0.1 port 5201
[ 16] local 172.0.0.2 port 56408 connected to 172.0.0.1 port 5201
[ 18] local 172.0.0.2 port 56410 connected to 172.0.0.1 port 5201
[ 20] local 172.0.0.2 port 56412 connected to 172.0.0.1 port 5201
[ 22] local 172.0.0.2 port 56414 connected to 172.0.0.1 port 5201
^C
Message from syslogd@vyos at Nov 11 07:08:35 ...
kernel:[76940.627171] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
Message from syslogd@vyos at Nov 11 07:08:35 ...
kernel:[76940.643171] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [swapper/7:0]
Message from syslogd@vyos at Nov 11 07:08:35 ...
kernel:[76940.747172] watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [iperf3:6807]
Message from syslogd@vyos at Nov 11 07:08:51 ...
kernel:[76956.643170] watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [opennhrp:6399]
Message from syslogd@vyos at Nov 11 07:09:03 ...
kernel:[76968.627170] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
Dmitry
November 11, 2021, 9:41am
5
Could you please share your HW? I hope this is not VM with passthrough.
show hardware cpu
show hardware dmi
show hardware pci
show hardware mem
root@vyos:/home/vyos# show hardware cpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 1
Core(s) per socket: 12
Socket(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
Stepping: 4
CPU MHz: 1001.585
CPU max MHz: 3200.0000
CPU min MHz: 1000.0000
BogoMIPS: 4602.21
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 16896K
root@vyos:/home/vyos# show hardware dmi
bios_date: 06/29/2019
bios_vendor: INSYDE Corp.
bios_version: 6.83
board_asset_tag: Huawei
board_name: BC11SPSCB0
board_serial: 024AFQ10K7001508
board_vendor: Huawei
board_version: V100R005
chassis_asset_tag: Huawei
chassis_serial: Huawei
chassis_type: 17
chassis_vendor: Huawei
chassis_version: Huawei
product_family: Purley
product_name: 2288H V5
product_serial: 2102311TYB10K8000878
product_sku: Purley
product_uuid: 5e2b5f30-183d-a0d1-e911-c5ba7212bf19
product_version: Purley
root@vyos:/home/vyos# show hardware mem
MemTotal: 527774604 kB
MemFree: 526671832 kB
MemAvailable: 524086836 kB
Buffers: 50972 kB
Cached: 149700 kB
SwapCached: 0 kB
Active: 176624 kB
Inactive: 93104 kB
Active(anon): 73252 kB
Inactive(anon): 9272 kB
Active(file): 103372 kB
Inactive(file): 83832 kB
Unevictable: 4840 kB
Mlocked: 4840 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 73920 kB
Mapped: 35024 kB
Shmem: 10120 kB
Slab: 160552 kB
SReclaimable: 43744 kB
SUnreclaim: 116808 kB
KernelStack: 5648 kB
PageTables: 5156 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 263887300 kB
Committed_AS: 438356 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 14016 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 140288 kB
DirectMap2M: 4675584 kB
DirectMap1G: 533725184 kB
root@vyos:/home/vyos# show hardware pci
00:00.0 Host bridge: Intel Corporation Device 2020 (rev 04)
00:04.0 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.1 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.2 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.3 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.4 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.5 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.6 System peripheral: Intel Corporation Device 2021 (rev 04)
00:04.7 System peripheral: Intel Corporation Device 2021 (rev 04)
00:05.0 System peripheral: Intel Corporation Device 2024 (rev 04)
00:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
00:05.4 PIC: Intel Corporation Device 2026 (rev 04)
00:08.0 System peripheral: Intel Corporation Device 2014 (rev 04)
00:08.1 Performance counters: Intel Corporation Device 2015 (rev 04)
00:08.2 System peripheral: Intel Corporation Device 2016 (rev 04)
00:11.0 Unassigned class [ff00]: Intel Corporation Device a1ec (rev 09)
00:11.1 Unassigned class [ff00]: Intel Corporation Device a1ed (rev 09)
00:11.5 SATA controller: Intel Corporation Device a1d2 (rev 09)
00:14.0 USB controller: Intel Corporation Device a1af (rev 09)
00:14.2 Signal processing controller: Intel Corporation Device a1b1 (rev 09)
00:16.0 Communication controller: Intel Corporation Device a1ba (rev 09)
00:16.1 Communication controller: Intel Corporation Device a1bb (rev 09)
00:16.4 Communication controller: Intel Corporation Device a1be (rev 09)
00:17.0 SATA controller: Intel Corporation Device a182 (rev 09)
00:1c.0 PCI bridge: Intel Corporation Device a190 (rev f9)
00:1c.4 PCI bridge: Intel Corporation Device a194 (rev f9)
00:1c.5 PCI bridge: Intel Corporation Device a195 (rev f9)
00:1f.0 ISA bridge: Intel Corporation Device a1c2 (rev 09)
00:1f.2 Memory controller: Intel Corporation Device a1a1 (rev 09)
00:1f.4 SMBus: Intel Corporation Device a1a3 (rev 09)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device a1a4 (rev 09)
02:00.0 VGA compatible controller: Device 19e5:1711 (rev 01)
03:00.0 Signal processing controller: Device 19e5:1710 (rev 01)
17:00.0 PCI bridge: Intel Corporation Device 2030 (rev 04)
17:02.0 PCI bridge: Intel Corporation Device 2032 (rev 04)
17:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
17:05.2 System peripheral: Intel Corporation Device 2035 (rev 04)
17:05.4 PIC: Intel Corporation Device 2036 (rev 04)
17:08.0 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.1 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.2 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.3 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.4 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.5 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.6 System peripheral: Intel Corporation Device 208d (rev 04)
17:08.7 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.0 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.1 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.2 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.3 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.4 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.5 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.6 System peripheral: Intel Corporation Device 208d (rev 04)
17:09.7 System peripheral: Intel Corporation Device 208d (rev 04)
17:0a.0 System peripheral: Intel Corporation Device 208d (rev 04)
17:0a.1 System peripheral: Intel Corporation Device 208d (rev 04)
17:0e.0 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.1 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.2 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.3 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.4 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.5 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.6 System peripheral: Intel Corporation Device 208e (rev 04)
17:0e.7 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.0 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.1 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.2 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.3 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.4 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.5 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.6 System peripheral: Intel Corporation Device 208e (rev 04)
17:0f.7 System peripheral: Intel Corporation Device 208e (rev 04)
17:10.0 System peripheral: Intel Corporation Device 208e (rev 04)
17:10.1 System peripheral: Intel Corporation Device 208e (rev 04)
17:1d.0 System peripheral: Intel Corporation Device 2054 (rev 04)
17:1d.1 System peripheral: Intel Corporation Device 2055 (rev 04)
17:1d.2 System peripheral: Intel Corporation Device 2056 (rev 04)
17:1d.3 System peripheral: Intel Corporation Device 2057 (rev 04)
17:1e.0 System peripheral: Intel Corporation Device 2080 (rev 04)
17:1e.1 System peripheral: Intel Corporation Device 2081 (rev 04)
17:1e.2 System peripheral: Intel Corporation Device 2082 (rev 04)
17:1e.3 System peripheral: Intel Corporation Device 2083 (rev 04)
17:1e.4 System peripheral: Intel Corporation Device 2084 (rev 04)
17:1e.5 System peripheral: Intel Corporation Device 2085 (rev 04)
17:1e.6 System peripheral: Intel Corporation Device 2086 (rev 04)
18:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 09)
19:03.0 PCI bridge: Intel Corporation Device 37c5 (rev 09)
1a:00.0 Ethernet controller: Intel Corporation Device 37d0 (rev 09)
1a:00.1 Ethernet controller: Intel Corporation Device 37d0 (rev 09)
1a:00.2 Ethernet controller: Intel Corporation Device 37d1 (rev 09)
1a:00.3 Ethernet controller: Intel Corporation Device 37d1 (rev 09)
1b:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02)
3a:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
3a:05.2 System peripheral: Intel Corporation Device 2035 (rev 04)
3a:05.4 PIC: Intel Corporation Device 2036 (rev 04)
3a:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
3a:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
3a:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
3a:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
3a:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
3a:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
3a:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
3a:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
3a:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
3a:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
3a:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
3a:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
3a:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
3a:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
3a:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
3a:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
3a:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
3a:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
3a:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
3a:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
3a:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
3a:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
3a:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
3a:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
3a:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
3a:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
5d:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
5d:05.2 System peripheral: Intel Corporation Device 2035 (rev 04)
5d:05.4 PIC: Intel Corporation Device 2036 (rev 04)
5d:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
5d:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
5d:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
5d:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
5d:12.0 Performance counters: Intel Corporation Device 204c (rev 04)
5d:12.1 Performance counters: Intel Corporation Device 204d (rev 04)
5d:12.2 System peripheral: Intel Corporation Device 204e (rev 04)
5d:15.0 System peripheral: Intel Corporation Device 2018 (rev 04)
5d:16.0 System peripheral: Intel Corporation Device 2018 (rev 04)
5d:16.4 System peripheral: Intel Corporation Device 2018 (rev 04)
80:04.0 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.1 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.2 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.3 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.4 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.5 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.6 System peripheral: Intel Corporation Device 2021 (rev 04)
80:04.7 System peripheral: Intel Corporation Device 2021 (rev 04)
80:05.0 System peripheral: Intel Corporation Device 2024 (rev 04)
80:05.2 System peripheral: Intel Corporation Device 2025 (rev 04)
80:05.4 PIC: Intel Corporation Device 2026 (rev 04)
80:08.0 System peripheral: Intel Corporation Device 2014 (rev 04)
80:08.1 Performance counters: Intel Corporation Device 2015 (rev 04)
80:08.2 System peripheral: Intel Corporation Device 2016 (rev 04)
85:00.0 PCI bridge: Intel Corporation Device 2030 (rev 04)
85:02.0 PCI bridge: Intel Corporation Device 2032 (rev 04)
85:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
85:05.2 System peripheral: Intel Corporation Device 2035 (rev 04)
85:05.4 PIC: Intel Corporation Device 2036 (rev 04)
85:08.0 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.1 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.2 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.3 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.4 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.5 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.6 System peripheral: Intel Corporation Device 208d (rev 04)
85:08.7 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.0 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.1 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.2 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.3 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.4 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.5 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.6 System peripheral: Intel Corporation Device 208d (rev 04)
85:09.7 System peripheral: Intel Corporation Device 208d (rev 04)
85:0a.0 System peripheral: Intel Corporation Device 208d (rev 04)
85:0a.1 System peripheral: Intel Corporation Device 208d (rev 04)
85:0e.0 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.1 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.2 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.3 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.4 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.5 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.6 System peripheral: Intel Corporation Device 208e (rev 04)
85:0e.7 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.0 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.1 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.2 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.3 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.4 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.5 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.6 System peripheral: Intel Corporation Device 208e (rev 04)
85:0f.7 System peripheral: Intel Corporation Device 208e (rev 04)
85:10.0 System peripheral: Intel Corporation Device 208e (rev 04)
85:10.1 System peripheral: Intel Corporation Device 208e (rev 04)
85:1d.0 System peripheral: Intel Corporation Device 2054 (rev 04)
85:1d.1 System peripheral: Intel Corporation Device 2055 (rev 04)
85:1d.2 System peripheral: Intel Corporation Device 2056 (rev 04)
85:1d.3 System peripheral: Intel Corporation Device 2057 (rev 04)
85:1e.0 System peripheral: Intel Corporation Device 2080 (rev 04)
85:1e.1 System peripheral: Intel Corporation Device 2081 (rev 04)
85:1e.2 System peripheral: Intel Corporation Device 2082 (rev 04)
85:1e.3 System peripheral: Intel Corporation Device 2083 (rev 04)
85:1e.4 System peripheral: Intel Corporation Device 2084 (rev 04)
85:1e.5 System peripheral: Intel Corporation Device 2085 (rev 04)
85:1e.6 System peripheral: Intel Corporation Device 2086 (rev 04)
86:00.0 Ethernet controller: Mellanox Technologies MT27630 Family
86:00.1 Ethernet controller: Mellanox Technologies MT27630 Family
87:00.0 Ethernet controller: Mellanox Technologies Device 101d
87:00.1 Ethernet controller: Mellanox Technologies Device 101d
ae:02.0 PCI bridge: Intel Corporation Device 2032 (rev 04)
ae:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
ae:05.2 System peripheral: Intel Corporation Device 2035 (rev 04)
ae:05.4 PIC: Intel Corporation Device 2036 (rev 04)
ae:08.0 System peripheral: Intel Corporation Device 2066 (rev 04)
ae:09.0 System peripheral: Intel Corporation Device 2066 (rev 04)
ae:0a.0 System peripheral: Intel Corporation Device 2040 (rev 04)
ae:0a.1 System peripheral: Intel Corporation Device 2041 (rev 04)
ae:0a.2 System peripheral: Intel Corporation Device 2042 (rev 04)
ae:0a.3 System peripheral: Intel Corporation Device 2043 (rev 04)
ae:0a.4 System peripheral: Intel Corporation Device 2044 (rev 04)
ae:0a.5 System peripheral: Intel Corporation Device 2045 (rev 04)
ae:0a.6 System peripheral: Intel Corporation Device 2046 (rev 04)
ae:0a.7 System peripheral: Intel Corporation Device 2047 (rev 04)
ae:0b.0 System peripheral: Intel Corporation Device 2048 (rev 04)
ae:0b.1 System peripheral: Intel Corporation Device 2049 (rev 04)
ae:0b.2 System peripheral: Intel Corporation Device 204a (rev 04)
ae:0b.3 System peripheral: Intel Corporation Device 204b (rev 04)
ae:0c.0 System peripheral: Intel Corporation Device 2040 (rev 04)
ae:0c.1 System peripheral: Intel Corporation Device 2041 (rev 04)
ae:0c.2 System peripheral: Intel Corporation Device 2042 (rev 04)
ae:0c.3 System peripheral: Intel Corporation Device 2043 (rev 04)
ae:0c.4 System peripheral: Intel Corporation Device 2044 (rev 04)
ae:0c.5 System peripheral: Intel Corporation Device 2045 (rev 04)
ae:0c.6 System peripheral: Intel Corporation Device 2046 (rev 04)
ae:0c.7 System peripheral: Intel Corporation Device 2047 (rev 04)
ae:0d.0 System peripheral: Intel Corporation Device 2048 (rev 04)
ae:0d.1 System peripheral: Intel Corporation Device 2049 (rev 04)
ae:0d.2 System peripheral: Intel Corporation Device 204a (rev 04)
ae:0d.3 System peripheral: Intel Corporation Device 204b (rev 04)
af:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 04)
b0:00.0 PCI bridge: Intel Corporation Device 37c2 (rev 04)
b0:01.0 PCI bridge: Intel Corporation Device 37c3 (rev 04)
b0:02.0 PCI bridge: Intel Corporation Device 37c4 (rev 04)
b1:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
b2:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
b3:00.0 Co-processor: Intel Corporation Device 37c8 (rev 04)
d7:00.0 PCI bridge: Intel Corporation Device 2030 (rev 04)
d7:01.0 PCI bridge: Intel Corporation Device 2031 (rev 04)
d7:02.0 PCI bridge: Intel Corporation Device 2032 (rev 04)
d7:03.0 PCI bridge: Intel Corporation Device 2033 (rev 04)
d7:05.0 System peripheral: Intel Corporation Device 2034 (rev 04)
d7:05.2 System peripheral: Intel Corporation Device 2035 (rev 04)
d7:05.4 PIC: Intel Corporation Device 2036 (rev 04)
d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 04)
d7:0e.1 System peripheral: Intel Corporation Device 2059 (rev 04)
d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 04)
d7:0f.1 System peripheral: Intel Corporation Device 2059 (rev 04)
d7:12.0 Performance counters: Intel Corporation Device 204c (rev 04)
d7:12.1 Performance counters: Intel Corporation Device 204d (rev 04)
d7:12.2 System peripheral: Intel Corporation Device 204e (rev 04)
d7:15.0 System peripheral: Intel Corporation Device 2018 (rev 04)
d7:16.0 System peripheral: Intel Corporation Device 2018 (rev 04)
d7:16.4 System peripheral: Intel Corporation Device 2018 (rev 04)
d8:00.0 Non-Volatile memory controller: Device 19e5:3714 (rev 20)
d9:00.0 Non-Volatile memory controller: Device 19e5:3714 (rev 20)
I used a physical server to test, not virtual machine
Dmitry
November 17, 2021, 6:55pm
8
Hello @710354153 , it can be not easy to reproduce this in our lab. I guess it might be happening for some power-saving policies. As HT is already disabled, can you try to set scaling governor to performance? On 1.3 it is possible to do this with command:set system option performance throughputidle=poll and intel_idle.max_cstate=0 processor.max_cstate=1