Hello! I’m testing VyOS 1.8.1 as endpoints for GRE tunnels. Currently I have it running in 2 DCs (SJC and LAX), and I am running into an issue where performance over the GRE tunnel is great from LAX to SJC (600+ Mbps) however from SJC to LAX I get just over 1Mbps.
Bandwidth between the two sites without GRE factored in is great bi-directionally >1Gbps. Any thoughts as to why the performance in one direction over GRE would suffer significantly?
Running in Vultr (they use KVM) and running with 1CPU and 1GB RAM.
Thanks for your thoughts!
Data: iperf3 running on the VyOS VMs
SJC -> LAX; No GRE;
vyos@sjc-test-gw1:~$ iperf -c xx.xx.73.130 -m
------------------------------------------------------------
Client connecting to xx.xx.73.130, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local xx.xx.190.126 port 60923 connected with xx.xx.73.130 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 925 MBytes 775 Mbits/sec
[ 3] MSS size 1288 bytes (MTU 1328 bytes, unknown interface)
LAX -> SJC; No GRE;
vyos@lax-test-gw1:~$ iperf -c xx.xx.190.126 -m
------------------------------------------------------------
Client connecting to xx.xx.190.126, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local xx.xx.73.130 port 33591 connected with xx.xx.190.126 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.93 GBytes 1.66 Gbits/sec
[ 3] MSS size 1288 bytes (MTU 1328 bytes, unknown interface)
SJC -> LAX; Over GRE Tunnel;
vyos@bbisp-test-gw1:~$ iperf -c 10.10.10.6 -m
------------------------------------------------------------
Client connecting to 10.10.10.6, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.5 port 51783 connected with 10.10.10.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 1.35 MBytes 1.12 Mbits/sec <<<<<------ Super low.
[ 3] MSS size 1348 bytes (MTU 1388 bytes, unknown interface)
LAX -> SJC; Over GRE Tunnel
vyos@lax-test-gw1:~$ iperf -c 10.10.10.5 -m
------------------------------------------------------------
Client connecting to 10.10.10.5, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.6 port 48211 connected with 10.10.10.5 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 736 MBytes 617 Mbits/sec
[ 3] MSS size 1348 bytes (MTU 1388 bytes, unknown interface)
Configs below - as you see I have tried to mess with MTU & TCP-MSS settings to no avail. Other than that being that this is dev, the config is as bare bones as it gets.
vyos@lax-test-gw1:~$ show configuration commands
set interfaces ethernet eth0 address 'xx.xx.73.130/23'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 mtu '1476'
set interfaces ethernet eth0 policy route 'mss-clamp'
set interfaces ethernet eth0 smp_affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces loopback 'lo'
set interfaces tunnel tun1 address '10.10.10.6/30'
set interfaces tunnel tun1 encapsulation 'gre'
set interfaces tunnel tun1 local-ip 'xx.xx.73.130'
set interfaces tunnel tun1 mtu '1400'
set interfaces tunnel tun1 multicast 'disable'
set interfaces tunnel tun1 remote-ip 'xx.xx.190.126'
set policy route mss-clamp rule 10 protocol 'tcp'
set policy route mss-clamp rule 10 set tcp-mss '1300'
set policy route mss-clamp rule 10 tcp flags 'SYN'
set protocols static route 0.0.0.0/0 next-hop 'xx.xx.72.1'
set service ssh port '22'
set system config-management commit-revisions '20'
set system console device ttyS0 speed '9600'
set system host-name 'lax-test-gw1'
set system login user vyos authentication encrypted-password ''
set system login user vyos authentication plaintext-password ''
set system login user vyos level 'admin'
set system package auto-sync '1'
set system package repository community components 'main'
set system package repository community distribution 'helium'
set system package repository community password ''
set system package repository community username ''
set system syslog global facility all level 'notice'
set system syslog global facility protocols level 'debug'
set system time-zone 'UTC'
vyos@sjc-test-gw1:~$ show configuration commands
set interfaces ethernet eth0 address 'xx.xx.190.126/23'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 mtu '1476'
set interfaces ethernet eth0 policy route 'mss-clamp'
set interfaces ethernet eth0 smp_affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces loopback 'lo'
set interfaces tunnel tun1 address '10.10.10.5/30'
set interfaces tunnel tun1 encapsulation 'gre'
set interfaces tunnel tun1 local-ip 'xx.xx.190.126'
set interfaces tunnel tun1 mtu '1400'
set interfaces tunnel tun1 multicast 'disable'
set interfaces tunnel tun1 remote-ip 'xx.xx.73.130'
set policy route mss-clamp rule 10 protocol 'tcp'
set policy route mss-clamp rule 10 set tcp-mss '1300'
set policy route mss-clamp rule 10 tcp flags 'SYN'
set protocols static route 0.0.0.0/0 next-hop 'xx.xx.190.1'
set service ssh port '22'
set system config-management commit-revisions '20'
set system console device ttyS0 speed '9600'
set system host-name 'sjc-test-gw1'
set system login user vyos authentication encrypted-password ''
set system login user vyos authentication plaintext-password ''
set system login user vyos level 'admin'
set system package auto-sync '1'
set system package repository community components 'main'
set system package repository community distribution 'helium'
set system package repository community password ''
set system package repository community username ''
set system syslog global facility all level 'notice'
set system syslog global facility protocols level 'debug'
set system time-zone 'UTC'