IPSec GRE tunnel seems limited at 1Gbps

General questions
1

Hi

We started using VyOS due to other options not being able to do a 1Gbps IPSec tunnel well without spending a tonne of money.

We’ve been very happy, however we recently upgraded our link between the 2 sites to be 2Gbps.

And doing an iperf through the WAN facing link without the tunnel is able to get 2Gbps no issue, however through the IPSec tunnel we seem to be getting 1Gbps, looking at the CPU it doesn’t appear to be the issue.

Unsure if there is anything I should be doing differently or what I need look at.

Config below

set firewall name OUTSIDE-LOCAL rule 10 action 'accept'
set firewall name OUTSIDE-LOCAL rule 10 state established 'enable'
set firewall name OUTSIDE-LOCAL rule 10 state related 'enable'
set firewall name OUTSIDE-LOCAL rule 20 action 'accept'
set firewall name OUTSIDE-LOCAL rule 20 icmp type-name 'echo-request'
set firewall name OUTSIDE-LOCAL rule 20 protocol 'icmp'
set firewall name OUTSIDE-LOCAL rule 20 state new 'enable'
set firewall name OUTSIDE-LOCAL rule 30 action 'drop'
set firewall name OUTSIDE-LOCAL rule 30 destination port '22'
set firewall name OUTSIDE-LOCAL rule 30 protocol 'tcp'
set firewall name OUTSIDE-LOCAL rule 30 recent count '4'
set firewall name OUTSIDE-LOCAL rule 30 recent time '60'
set firewall name OUTSIDE-LOCAL rule 30 state new 'enable'
set firewall name OUTSIDE-LOCAL rule 31 action 'accept'
set firewall name OUTSIDE-LOCAL rule 31 destination port '22'
set firewall name OUTSIDE-LOCAL rule 31 protocol 'tcp'
set firewall name OUTSIDE-LOCAL rule 31 state new 'enable'
set interfaces ethernet eth0 address 'x.x.x.x/28'
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth0 firewall 'in'
set interfaces ethernet eth0 firewall 'local'
set interfaces ethernet eth0 hw-id '00:0c:29:48:c0:9f'
set interfaces ethernet eth1 address '172.16.75.253/30'
set interfaces ethernet eth1 description 'LAN'
set interfaces ethernet eth1 hw-id '00:0c:29:48:c0:a9'
set interfaces loopback 'lo'
set interfaces tunnel tun0 address 'x.x.x.x'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 local-ip 'x.x.x.x'
set interfaces tunnel tun0 remote-ip 'x.x.x.x'
set protocols static interface-route 10.0.29.0/24 next-hop-interface 'tun0'
set protocols static interface-route 10.1.66.0/24 next-hop-interface 'tun0'
set protocols static interface-route 172.16.29.0/24 next-hop-interface 'tun0'
set protocols static interface-route 192.168.28.0/24 next-hop-interface 'tun0'
set protocols static interface-route 192.168.29.0/24 next-hop-interface 'tun0'
set protocols static interface-route 192.168.248.0/24 next-hop-interface 'tun0'
set protocols static route 10.0.1.0/24 next-hop '172.16.75.254'
set protocols static route 10.0.2.0/24 next-hop '172.16.75.254'
set protocols static route 10.6.66.0/24 next-hop '172.16.75.254'
set protocols static route 172.16.1.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.20.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.60.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.61.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.62.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.63.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.64.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.66.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.68.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.100.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.128.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.167.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.168.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.169.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.170.0/23 next-hop '172.16.75.254'
set protocols static route 192.168.172.0/23 next-hop '172.16.75.254'
set protocols static route 192.168.200.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.220.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.252.0/24 next-hop '172.16.75.254'
set protocols static route 192.168.253.0/24 next-hop '172.16.75.254'
set service snmp community Management authorization 'ro'
set service snmp community Management client '192.168.64.175'
set service ssh listen-address '172.16.75.253'
set service ssh port '22'
set system config-management commit-revisions '20'
set system 'console'
set system gateway-address 'x.x.x.x'
set system host-name 'mtl-munvpn01'
set system login user nigelincognito authentication encrypted-password '$6$IwyMO8AwuCz$F3Sh91UsRxmxZZ2YMw8znDXjYbQAnZtZkWrny2jVKOmRmwYzJ2JEY3Ec0bMRSwCooJheRAbVqBXQMEb9KcgTd.'
set system login user nigelincognito authentication plaintext-password ''
set system package repository community components 'main'
set system package repository community distribution 'helium'
set system syslog global facility all level 'notice'
set system syslog global facility protocols level 'debug'
set vpn ipsec esp-group mtl-to-mun-esp proposal 1 encryption 'aes128'
set vpn ipsec esp-group mtl-to-mun-esp proposal 1 hash 'sha1'
set vpn ipsec ike-group mtl-to-mun-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group mtl-to-mun-ike dead-peer-detection interval '30'
set vpn ipsec ike-group mtl-to-mun-ike dead-peer-detection timeout '30'
set vpn ipsec ike-group mtl-to-mun-ike proposal 1 dh-group '2'
set vpn ipsec ike-group mtl-to-mun-ike proposal 1 encryption 'aes128'
set vpn ipsec ike-group mtl-to-mun-ike proposal 1 hash 'sha1'
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec site-to-site peer x.x.x.x authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer x.x.x.x authentication pre-shared-secret 'secret123'
set vpn ipsec site-to-site peer x.x.x.x default-esp-group 'mtl-to-mun-esp'
set vpn ipsec site-to-site peer x.x.x.x ike-group 'mtl-to-mun-ike'
set vpn ipsec site-to-site peer x.x.x.x local-address 'x.x.x.x'
set vpn ipsec site-to-site peer x.x.x.x tunnel 1 protocol 'gre'
2

Is this virtual deployment or VyOS installed on bare metal server ?

3

Sorry forgot to mention that, it is installed on a hypervisor yes at both ends

4

Which hypervisor? and which nic type you use?

5

VMWare ESXi 6.0

Hardware is

Dell PowerEdge R730
2 x Intel Xeon E5 2643 CPUs
256GB of RAM
Intel 10GB NIC

6

What type of NIC in VM? Suspect that will be e1000

7

Is the virtual NIC using virtio?
Also, which VyOS version? 1.2.0 preview builds have a much newer kernel, and thus better support for AES-NI etc.

8

Make sure your using the VXNET3 NICs in your VM. Otherwise you might have some VM CPU contention maybe?

9

No it is VMXNET3 NICs.

As mentioned we can get the higher speeds for traffic not going through the IPSec tunnel, that gets 2Gbps no problems.

It’s only when sending traffic through the IPSec tunnel where we seem to have an issue of it being 1Gbps

It is using 1.1.8 version, I may try 1.2.0 and see if it makes a difference.

10

I’m still having issues with this, it seems like maybe the IPsec tunnel being single threaded is the bottleneck.
I suspect it is also the way we are using it, because the connection is trans atlantic, East Coast America to Germany, we are using an open source tool from CERN called FDT, it’s basically using UDP to fill a WAN pipe, but it generates a lot of traffic.

Does anyone have examples of people doing 2+ Gbps of IPsec traffic through VyOS ?

My next idea was to make multiple IPSec tunnels, and use policy based routes to send it through separate tunnels, is this possible with GRE + IPSec ?

11

Hi,
can you test it locally maybe ? (see if you getting anything more than 1gb)

12

As previously mentioned the internet traffic outside of the tunnel goes up to the 2Gbps, it’s when it is traversing the tunnel that it is limited.

13

Have you tested performance over the GRE tunnel WITHOUT the IPSEC tunnel?

14

No because I can’t take the tunnel down

15

Drop the IPSEC config specific for each site. Then try with only a pure GRE tunnel.

16

It gets the full speed of the bandwidth, but that doesn’t help solve the problem, legally I can’t send unencrypted data.

17

ok, so it’s about ipsec encryption and this happens on 1.2, correct?

18

Yup, we’re just trying to find out where the bottleneck is. How’s your CPU when you’re pushing a lot of traffic down?

How many vCPU’s have you assigned your vRouter?

19

What’s the CPU load when stuck at 1Gb/s?
Note that a reading like 25% means “enough headroom” for some, but on a 4 core machine, it might mean a single core is stuck at 100%

20

IPSec is single threaded, so yes a single core is maxed at 100%.

We’ve just had to use a physical machine with brand new i9 CPUs to overcome this problem, they handle the higher IPSec tunnel speeds better.

The lower clock speed/higher core Xeons don’t seem to be up to snuff to go over 1Gbps.