IPsec vti based - very slow performance, max 20.75 MEGABYTES via VPN

rufzor · November 15, 2020, 4:34pm

Hello

I have VPN site-to-site on VyOS 1.2.6 sitting in the same rack two server connected LOCALY to the 10G network, I have created between them VPN s2s.

This is what I got, output from iperf3, let me remember both server are connected via 10G network, and between them I have created IPsec s2s, what is going on ?

No traffic policy.

>     # iperf3 -c 10.0.80.2 -f m -i 10 -t 60 -l 1100 -M 1100 -P 4 -u -b 175M
> Connecting to host 10.0.80.2, port 5201
> [  4] local 10.0.80.1 port 49601 connected to 10.0.80.2 port 5201
> [  6] local 10.0.80.1 port 39300 connected to 10.0.80.2 port 5201
> [  8] local 10.0.80.1 port 56943 connected to 10.0.80.2 port 5201
> [ 10] local 10.0.80.1 port 42506 connected to 10.0.80.2 port 5201
> [ ID] Interval           Transfer     Bandwidth       Total Datagrams
> [  4]   0.00-10.00  sec   196 MBytes   164 Mbits/sec  186759  
> [  6]   0.00-10.00  sec   196 MBytes   164 Mbits/sec  186759  
> [  8]   0.00-10.00  sec   196 MBytes   164 Mbits/sec  186759  
> [ 10]   0.00-10.00  sec   196 MBytes   164 Mbits/sec  186759  
> [SUM]   0.00-10.00  sec   784 MBytes   657 Mbits/sec  747036  
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [  4]  10.00-20.00  sec   197 MBytes   165 Mbits/sec  187895  
> [  6]  10.00-20.00  sec   197 MBytes   165 Mbits/sec  187895  
> [  8]  10.00-20.00  sec   197 MBytes   165 Mbits/sec  187895  
> [ 10]  10.00-20.00  sec   197 MBytes   165 Mbits/sec  187895  
> [SUM]  10.00-20.00  sec   788 MBytes   661 Mbits/sec  751580  
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [  4]  20.00-30.00  sec   197 MBytes   165 Mbits/sec  187534  
> [  6]  20.00-30.00  sec   197 MBytes   165 Mbits/sec  187534  
> [  8]  20.00-30.00  sec   197 MBytes   165 Mbits/sec  187534  
> [ 10]  20.00-30.00  sec   197 MBytes   165 Mbits/sec  187534  
> [SUM]  20.00-30.00  sec   787 MBytes   660 Mbits/sec  750136  
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [  4]  30.00-40.00  sec   197 MBytes   165 Mbits/sec  187967  
> [  6]  30.00-40.00  sec   197 MBytes   165 Mbits/sec  187967  
> [  8]  30.00-40.00  sec   197 MBytes   165 Mbits/sec  187967  
> [ 10]  30.00-40.00  sec   197 MBytes   165 Mbits/sec  187967  
> [SUM]  30.00-40.00  sec   789 MBytes   662 Mbits/sec  751868  
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [  4]  40.00-50.00  sec   200 MBytes   168 Mbits/sec  190431  
> [  6]  40.00-50.00  sec   200 MBytes   168 Mbits/sec  190431  
> [  8]  40.00-50.00  sec   200 MBytes   168 Mbits/sec  190431  
> [ 10]  40.00-50.00  sec   200 MBytes   168 Mbits/sec  190431  
> [SUM]  40.00-50.00  sec   799 MBytes   670 Mbits/sec  761724  
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [  4]  50.00-60.00  sec   198 MBytes   166 Mbits/sec  188660  
> [  6]  50.00-60.00  sec   198 MBytes   166 Mbits/sec  188660  
> [  8]  50.00-60.00  sec   198 MBytes   166 Mbits/sec  188660  
> [ 10]  50.00-60.00  sec   198 MBytes   166 Mbits/sec  188660  
> [SUM]  50.00-60.00  sec   792 MBytes   664 Mbits/sec  754640  
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
> [  4]   0.00-60.00  sec  1.16 GBytes   166 Mbits/sec  0.018 ms  0/1129246 (0%)  
> [  4] Sent 1129246 datagrams
> [  6]   0.00-60.00  sec  1.16 GBytes   166 Mbits/sec  0.015 ms  0/1129246 (0%)  
> [  6] Sent 1129246 datagrams
> [  8]   0.00-60.00  sec  1.16 GBytes   166 Mbits/sec  0.012 ms  0/1129246 (0%)  
> [  8] Sent 1129246 datagrams
> [ 10]   0.00-60.00  sec  1.16 GBytes   166 Mbits/sec  0.019 ms  0/1129246 (0%)  
> [ 10] Sent 1129246 datagrams
> [SUM]   0.00-60.00  sec  4.63 GBytes   662 Mbits/sec  0.016 ms  0/4516984 (0%)

On both sides, bare metal DELL R620 server (non VM!) configuration below

set interfaces vti vti0 address 10.0.80.1/30
set interfaces vti vti0 mtu 1436
!
set vpn ipsec esp-group ESP-GF compression disable
set vpn ipsec esp-group ESP-GF lifetime 3600
set vpn ipsec esp-group ESP-GF mode tunnel
set vpn ipsec esp-group ESP-GF pfs dh-group2
set vpn ipsec esp-group ESP-GF proposal 1 encryption aes128
set vpn ipsec esp-group ESP-GF proposal 1 hash sha1
!
set vpn ipsec ike-group IKE-GF dead-peer-detection action restart
set vpn ipsec ike-group IKE-GF dead-peer-detection interval 15
set vpn ipsec ike-group IKE-GF dead-peer-detection timeout 30
set vpn ipsec ike-group IKE-GF ikev2-reauth yes
set vpn ipsec ike-group IKE-GF key-exchange ikev2
set vpn ipsec ike-group IKE-GF lifetime 28800
set vpn ipsec ike-group IKE-GF proposal 1 dh-group 2
set vpn ipsec ike-group IKE-GF proposal 1 encryption aes128
set vpn ipsec ike-group IKE-GF proposal 1 hash sha1
set vpn ipsec ipsec-interfaces interface bond0

set vpn ipsec site-to-site peer x.x.x.x authentication id x.x.x.x
set vpn ipsec site-to-site peer x.x.x.x authentication mode pre-shared-secret
set vpn ipsec site-to-site peer x.x.x.x  authentication pre-shared-secret  x.x.x.x
set vpn ipsec site-to-site peer x.x.x.x authentication remote-id x.x.x.x
set vpn ipsec site-to-site peer x.x.x.x connection-type initiate
set vpn ipsec site-to-site peer x.x.x.x default-esp-group ESP-GF
set vpn ipsec site-to-site peer x.x.x.x ike-group IKE-GF
set vpn ipsec site-to-site peer x.x.x.x ikev2-reauth inherit
set vpn ipsec site-to-site peer x.x.x.x local-address  x.x.x.x
set vpn ipsec site-to-site peer x.x.x.x vti bind vti0
set vpn ipsec site-to-site peer  x.x.x.x vti esp-group ESP-GF

Dmitry · November 15, 2020, 4:51pm

Hello @rufzor, usually 600 megabits per second this is maximum if cipher calculated via CPU. Try to find an Intel QAT solution that VyOS supported.

rufzor · November 15, 2020, 8:54pm

This is what I have on both sides, so vyos got issues with this CPU?

cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-4617 0 @ 2.90GHz
stepping	: 7
microcode	: 0x718
cpu MHz		: 1200.013
cache size	: 15360 KB
physical id	: 0
siblings	: 6
core id		: 0
cpu cores	: 6
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips	: 5800.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

Dmitry · November 16, 2020, 7:14am

Can you provide command top output and press 1 when iperf3 running?