We are trying to connect two networks using site to site vpn connection by using vyos as vpn gateway. We could not get bandwidth more than 1 Gbps. We are launching vyos as a vm in openstack/Nutanix platform. We tried multiple options like enabling pcrypt, using more vcpus and RAM(upto 8G RAM and 16 vcpus), Using different combination of encryption/hash algo, creating multiple tunnel between two vyos(In this case bandwidhth gets devided) etc. But in all the case we are not able to achieve bandwidhth more than 1 Gbps. Is there a way to get more performance than this ?
without ipsec tunnel we are getting upto 20Gbps so underlying network is not a bottleneck.
which version you use?
In general, IPSec encryption limited to only CPU (and memory speed) resources and little overhead in packets. Can you check system load at moment when you perform test?
It is vyos version 1.1.8
We tested with multiple combination of cpu and memory but the result is not affected much by that. Following is the observation:
1 cpu 1gb ram:
cpu0 utilization: 88%
performance: 635 Mbps
2 cpu 1gb ram:
cpu0 utilization: 85%
cpu1 utilization: 81%
performance: 1Gbps(approx)
4 cpu 2gb ram:
cpu0 utilization:60%
cpu1 utilization:65%
cpu2 utilization:40%
cpu3 utilization:30%
performance: 1Gbps(approx)
8cpu 4g bram:
cpu0 utilization:47%
cpu1 utilization:17%
cpu2 utilization:17%
cpu3 utilization:51%
cpu4 utilization:25%
cpu5 utilization:17%
cpu6 utilization:17%
cpu7 utilization:16%
performance: 1Gbps(approx)
Memory utilisation is not more than 20%
The 1G was limited due to your NIC or?
I would expect that it is possible to get over 1G
Even in that case if I create multiple tunnel using different interface cumulative speed should increase right ?
If you make sure, that noting goes twice over one interface, you should be right.
However, as you have some sort of magic wall at 1G I would say you reach some sort of bandwidth limit on one of the 1G links.
If you want, I can test ipsec s-s later that day on our 10G Lab.
I am getting upto 20Gbps without ipsec on the same setup.
@hiteshhapani, can you share the details of your testing methodology? I’d like to run some tests on AWS EC2 and compare the results to yours.
@jeffbrl I created two ipsec site to site tunnel between vyos1 and vyos2 and vyos2 and vyos3 vms as per following diagram and tested throughput between cli1 and cli2 and cli3 and cli4 using iperf. cli2 and cli3 are attahed to vyos2
cli1----vyos1-----------vyos2--------vyos3-----cli4
I’m having difficultly getting enhanced networking to work in EC2 using the 1.2 version I built. My tests may have to wait until the core developers publish an official 1.2 AMI.
@hiteshhapani I just tested an ipsec tunnel between only two VM hosts, and found the following results:
vyos version 1.1.8 ova
vm host1: dual E5-2680 cpus - 192gb ram dual 10gb interfaces
vm vyos - 10 vcpus 8gb ram dual 10gb interfaces
vm linux - 10 vcpus 8gb ram 10gb interface
vm host2: quad E7540 cpus - 192gb ram dual 10gb interfaces
vm vyos - 10 vcpus 8gb ram dual 10gb interfaces
vm linux - 10 vcpus 8gb ram 10gb interface
I was able to average 5gb via iperf3 with the other traffic on the host - no vpn
With ipsec configured I was only able to average 700Mbps.
I did do some parallel (iperf3 -c 1.1.1.1 -P 10) and was able to average 1gb a few times but was rather inconsistent across the board.
Hope this helps.
@jaym0n Thanks for the response. It seems like no one here has achieved more than 1 Gbps when vyos is used as a vm. Can anyone confirm whether it is a limitation from linux kernel ? Is there a way to overcome this limitation ?
Posted this on a different thread already.
This PDF might be helpful for those chasing more than 1Gbit of IPSec performance with vyos
https://www.strongswan.org/docs/Steffen_Klassert_Parallelizing_IPsec.pdf
@oam we are using pcrypt as mentioned in the pdf. Even they are getting upto 1Gbps as mentioned in the section “Maximum theoretical throughput on Layer 3” of above pdf.