Strange speed issue when IPSEC site-to-site is running


#1

Greetings all,

I am running VyOS 1.1.7 and having a strange problem with speeds when an IPSEC site-to-site VPN is active. The system has two interfaces, one a LAN interface and the other a WAN that is running PPPoE. The PPPoE connection is gigabit and as is typical, the provider speeds are limited by their peering agreements. I am not under the belief that I am going to get gigabit on this connection to the real world.

Most are familiar with the Ookla tools for speed testing and when I run the tools, I am getting about 900Mbps download and 400Mbps upload for baseline reference.

To a certain content provider on the internet, it is very repeatable that large TCP transfers across two interfaces run at about 150Mbps when VPN is disabled with set interfaces vti vti0 disabled, but only about 15Mbps when it is enabled. It’s nothing to do with the traffic being sent over the VPN; show interfaces vti detail and show interfaces pppoe pppoe0 show all the traffic going over the pppoe0 in all cases and vti0 is idle. Yet when I run the Ookla tools, it returns the same numbers whether the VPN is active or not (and the tools are supported by statistics on my hardwired client running the test).

What’s also very interesting is if I start a download across the router with the VPN disabled and get the 150Mbps speeds, I can restart the VPN during the download and it is not affected. Conversely, if I start a download with the VPN active and begin a slow download, the download does not speed up if I disable the VPN. In other words, the downloads that are started before transitioning the VPN remain at either their high or low speeds.

I have gone back and forth on enabling and disabling the VPN dozens of times to get my head around it. On the machine I am testing from, the routing tables are not affected when bringing the router up or down. Both this router and the VPN peer are VyOS 1.1.7 and are sharing routes via OSPF. Both are default routing to the proper interface, and as I said, counters indicate that the VPN is not being used when it is on.

As for the router itself, I may have seen it consuming 3% CPU and it is at 25% memory usage at the most. It is virtualized on XenServer 7 with a recent vintage Xeon, so it is not resource constrained.

Any ideas of what could possibly be causing this behavior?

Thanks, Brian

firewall { all-ping enable broadcast-ping disable config-trap disable ipv6-receive-redirects disable ipv6-src-route disable ip-src-route disable log-martians enable name allow-all { default-action accept } name public_to_router { default-action drop rule 10 { action accept protocol all state { established enable related enable } } rule 110 { action accept destination { port 500,1701,4500,10188 } protocol udp } rule 200 { action accept protocol esp } rule 201 { action accept source { address 10.0.0.0/8 } } } name to_private { default-action drop enable-default-log rule 1 { action accept protocol all state { established enable related enable } } } name to_public { default-action accept } receive-redirects disable send-redirects enable source-validation disable syn-cookies enable twa-hazards-protection disable } interfaces { ethernet eth0 { description OUTSIDE duplex auto hw-id 5a:37:5a:06:30:9a smp_affinity auto speed auto vif 201 { pppoe 0 { default-route auto mtu 1484 name-server auto } } } ethernet eth1 { address 192.168.10.1/24 description INSIDE duplex auto hw-id be:76:9e:50:42:5a smp_affinity auto speed auto } loopback lo { } vti vti0 { address 10.9.254.250/30 description "Home VPN" disable ip { ospf { dead-interval 40 hello-interval 10 network point-to-point priority 1 retransmit-interval 5 transmit-delay 1 } } mtu 1394 } } nat { source { rule 50 { destination { address 10.0.0.0/8 } exclude outbound-interface eth0 source { address 192.168.10.0/24 } } rule 100 { outbound-interface pppoe0 protocol all source { address 192.168.10.0/24 } translation { address masquerade } } } } policy { prefix-list RFC1918PREFIXES { rule 1 { action permit le 32 prefix 10.0.0.0/8 } rule 2 { action permit le 32 prefix 172.16.0.0/12 } rule 3 { action permit le 32 prefix 192.168.0.0/16 } } route-map VPNPREFIXES { rule 10 { action permit match { ip { address { prefix-list RFC1918PREFIXES } } } } rule 999 { action deny } } } protocols { ospf { area 0.0.0.0 { network 10.9.254.248/30 } default-information { originate { always metric 10 metric-type 2 } } log-adjacency-changes { } parameters { abr-type cisco router-id 192.168.10.1 } redistribute { connected { metric-type 2 route-map VPNPREFIXES } } } } service { dhcp-server { ... } ssh { port 22 } } system { config-management { commit-revisions 20 } console { device ttyS0 { speed 9600 } } host-name vyos ntp { server 0.pool.ntp.org { } server 1.pool.ntp.org { } server 2.pool.ntp.org { } } package { auto-sync 1 repository community { components main distribution helium password "" url http://packages.vyos.net/vyos username "" } } syslog { global { facility all { level notice } facility protocols { level debug } } } time-zone UTC } vpn { ipsec { esp-group ESP-1W { compression disable lifetime 1800 mode tunnel pfs enable proposal 1 { encryption aes256 hash sha1 } } ike-group IKE-1W { ikev2-reauth no key-exchange ikev1 lifetime 3600 proposal 1 { encryption aes256 hash sha1 } } ipsec-interfaces { interface pppoe0 } site-to-site { peer y.y.y.y { authentication { mode pre-shared-secret pre-shared-secret ssssssssssssssss } connection-type initiate default-esp-group ESP-1W ike-group IKE-1W ikev2-reauth inherit local-address x.x.x.x vti { bind vti0 esp-group ESP-1W } } } } } zone-policy { zone private { default-action reject from public { firewall { name to_private } } from vyatta { firewall { name allow-all } } interface eth1 interface vti+ } zone public { default-action reject from private { firewall { name to_public } } from vyatta { firewall { name to_public } } interface pppoe0 } zone vyatta { default-action reject from private { firewall { name allow-all } } from public { firewall { name public_to_router } } local-zone } }


#2

Double check if traffic really isn’t sent out via VPN: Visit a site like https://whatismyipaddress.com/ to see if your public IP changes

btw: NAT rule 50 seems useless to me


#3

Thanks, definitely no change in outbound IP address. I know what IP address would change if the traffic was emerging from the other side of the site-to-site.

I’ll check on NAT rule 50. Thanks for spying that, not sure what it was originally used for, maybe before I got the pppoe working.