Recommened MTU to wireguard interface

Vamp · February 18, 2020, 2:38pm

Hello there,

i use VyOS a VM (proxmox hypervisor). Use Wireguard to access my network remotelly.

i do this changes in vyos config, it need to the websites working well:

set policy route pppoe-out description 'PPPoE TCPMSS clamping'
set policy route pppoe-out rule 100 protocol 'tcp'
set policy route pppoe-out rule 100 set tcp-mss '1452'
set policy route pppoe-out rule 100 tcp flags 'SYN'

My question is that the best MTU setting to the wg0 interface in this case?

If i change it like 9000 (both server and client side) it 2x faster. But not think it is the best way…

Dmitry · February 18, 2020, 2:51pm

Hi @Vamp, I think best MTU 1500 for ethernet
Did you read this article Firewall — VyOS 1.4.x (sagitta) documentation

Vamp · February 18, 2020, 3:25pm

Hi @Dmitry,

I add this line, but same speed:

set firewall options interface wg0 adjust-mss '1372'

The speed both side is 1000/1000 mbit.

the max VPN speed about 60-70 mbit.

Dmitry · February 18, 2020, 3:29pm

Maybe bottleneck with interrupts and CPU loads? Can you take a screenshot top command and press 1, when you run test.

Vamp · February 18, 2020, 3:48pm

@Dmitry

here the picture

Dmitry · February 18, 2020, 6:58pm

I propose first check connection with iperf/iperf3
Clietnt---wg tunnel---VyOS>
In the screenshot bottleneck does not exist.

Vamp · February 18, 2020, 10:50pm

OK i will Test it Tomorrow

Vamp · February 19, 2020, 7:12am

Hi @Dmitry

Here the result:

Connecting to host 192.168.31.105, port 5201
[  4] local 192.168.32.2 port 2764 connected to 192.168.31.105 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  4.88 MBytes  40.9 Mbits/sec
[  4]   1.00-2.00   sec  4.88 MBytes  40.9 Mbits/sec
[  4]   2.00-3.00   sec  4.88 MBytes  40.9 Mbits/sec
[  4]   3.00-4.00   sec  4.75 MBytes  39.9 Mbits/sec
[  4]   4.00-5.00   sec  4.62 MBytes  38.8 Mbits/sec
[  4]   5.00-6.00   sec  4.62 MBytes  38.8 Mbits/sec
[  4]   6.00-7.00   sec  4.75 MBytes  39.9 Mbits/sec
[  4]   7.00-8.00   sec  4.62 MBytes  38.8 Mbits/sec
[  4]   8.00-9.00   sec  4.75 MBytes  39.8 Mbits/sec
[  4]   9.00-10.00  sec  4.38 MBytes  36.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  47.1 MBytes  39.5 Mbits/sec                  sender
[  4]   0.00-10.00  sec  47.0 MBytes  39.4 Mbits/sec                  receiver

iperf Done.

Dmitry · February 19, 2020, 7:15am

Hi @Vamp, can you run this test again but without WG tunnel?

Vamp · February 19, 2020, 7:58am

@Dmitry

Hmmm same…

Connecting to host v*************, port 60145
[  4] local 100.100.2.240 port 4634 connected to 84.2.69.xxx port 60145
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  4.38 MBytes  36.6 Mbits/sec
[  4]   1.00-2.00   sec  4.25 MBytes  35.6 Mbits/sec
[  4]   2.00-3.01   sec  4.38 MBytes  36.6 Mbits/sec
[  4]   3.01-4.01   sec  4.12 MBytes  34.5 Mbits/sec
[  4]   4.01-5.01   sec  4.38 MBytes  36.7 Mbits/sec
[  4]   5.01-6.01   sec  4.25 MBytes  35.6 Mbits/sec
[  4]   6.01-7.01   sec  4.25 MBytes  35.6 Mbits/sec
[  4]   7.01-8.01   sec  4.25 MBytes  35.6 Mbits/sec
[  4]   8.01-9.00   sec  4.38 MBytes  37.1 Mbits/sec
[  4]   9.00-10.00  sec  4.25 MBytes  35.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  42.9 MBytes  36.0 Mbits/sec                  sender
[  4]   0.00-10.00  sec  42.7 MBytes  35.8 Mbits/sec                  receiver

iperf Done.

Test the speed both side:

Side 1

speedtest --server 28124
Retrieving speedtest.net configuration...
Testing from Magyar Telekom (84.2.69.xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by Vodafone Magyarország Zrt. (Budapest) [59.36 km]: 11.801 ms
Testing download speed................................................................................
Download: 363.60 Mbit/s
Testing upload speed......................................................................................................
Upload: 433.55 Mbit/s

Side 2

>speedtest.exe -s 28124

   Speedtest by Ookla

     Server: Vodafone Magyarország Zrt. - Budapest (id = 28124)
        ISP: Invitech ICT Services Kft.
    Latency:     3.43 ms   (0.09 ms jitter)
   Download:   250.13 Mbps (data used: 267.3 MB)
     Upload:   611.03 Mbps (data used: 850.5 MB)
Packet Loss: Not available.
 Result URL: https://www.speedtest.net/result/c/cb4f6b90-e606-46e3-8fda-78806256c45e

Vamp · February 19, 2020, 10:26am

@Dmitry

I try normal iperf with 10 session (inside VPN):

iperf -s -p 60145
------------------------------------------------------------
Server listening on TCP port 60145
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7139
[  5] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7140
[  6] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7141
[  7] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7142
[  8] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7143
[  9] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7144
[ 10] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7145
[ 11] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7146
[ 12] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7147
[ 13] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 7148
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.2 sec  21.0 MBytes  17.3 Mbits/sec
[  8]  0.0-10.2 sec  22.0 MBytes  18.2 Mbits/sec
[  9]  0.0-10.2 sec  21.4 MBytes  17.7 Mbits/sec
[ 10]  0.0-10.2 sec  21.4 MBytes  17.7 Mbits/sec
[ 12]  0.0-10.1 sec  21.9 MBytes  18.1 Mbits/sec
[ 13]  0.0-10.2 sec  20.9 MBytes  17.3 Mbits/sec
[  4]  0.0-10.2 sec  21.1 MBytes  17.4 Mbits/sec
[  5]  0.0-10.2 sec  21.4 MBytes  17.7 Mbits/sec
[  7]  0.0-10.2 sec  21.0 MBytes  17.3 Mbits/sec
[ 11]  0.0-10.2 sec  21.9 MBytes  18.1 Mbits/sec
[SUM]  0.0-10.2 sec   214 MBytes   177 Mbits/sec

Dmitry · February 19, 2020, 10:27am

Nice, this result through WG tunnel?
upd:// @Vamp, can you try to change sysctl params and test again with and without WG tunnel?

set system sysctl custom net.ipv4.tcp_congestion_control value 'htcp'
set system sysctl custom net.ipv4.tcp_mtu_probing value '1'
set system sysctl custom net.ipv4.tcp_window_scaling value '1'

Vamp · February 19, 2020, 10:50am

@Dmitry

1 session or 10 session?

Dmitry · February 19, 2020, 10:51am

Try both, for best analyse.

Vamp · February 19, 2020, 11:29am

@Dmitry

with VPN:

------------------------------------------------------------
Server listening on TCP port 60145
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8734
[  5] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8735
[  6] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8736
[  7] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8737
[  8] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8738
[  9] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8739
[ 10] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8740
[ 11] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8741
[ 12] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8742
[ 13] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8743
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.1 sec  15.3 MBytes  12.7 Mbits/sec
[  6]  0.0-10.1 sec  15.3 MBytes  12.7 Mbits/sec
[  7]  0.0-10.1 sec  15.4 MBytes  12.7 Mbits/sec
[  8]  0.0-10.2 sec  15.3 MBytes  12.7 Mbits/sec
[  9]  0.0-10.1 sec  15.4 MBytes  12.7 Mbits/sec
[ 11]  0.0-10.1 sec  15.4 MBytes  12.7 Mbits/sec
[ 12]  0.0-10.1 sec  15.4 MBytes  12.8 Mbits/sec
[  4]  0.0-10.2 sec  15.3 MBytes  12.7 Mbits/sec
[ 10]  0.0-10.2 sec  15.3 MBytes  12.7 Mbits/sec
[ 13]  0.0-10.2 sec  15.3 MBytes  12.6 Mbits/sec
[SUM]  0.0-10.2 sec   154 MBytes   127 Mbits/sec

indent preformatted text by 4 spaces


------------------------------------------------------------
Server listening on TCP port 60145
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.31.105 port 60145 connected with 192.168.32.2 port 8779
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  57.4 MBytes  48.2 Mbits/sec

without vpn

------------------------------------------------------------
Server listening on TCP port 60145
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 56032
[  5] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 61682
[  6] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 61440
[  7] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 59485
[  8] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 47612
[  9] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 23915
[ 10] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 27296
[ 11] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 6096
[ 12] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 9240
[ 13] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 45368
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.2 sec  13.2 MBytes  10.9 Mbits/sec
[ 10]  0.0-10.2 sec  13.3 MBytes  11.0 Mbits/sec
[ 11]  0.0-10.2 sec  13.2 MBytes  10.9 Mbits/sec
[  4]  0.0-10.2 sec  14.3 MBytes  11.7 Mbits/sec
[  5]  0.0-10.2 sec  14.3 MBytes  11.8 Mbits/sec
[  6]  0.0-10.2 sec  13.2 MBytes  10.9 Mbits/sec
[  7]  0.0-10.2 sec  13.3 MBytes  10.9 Mbits/sec
[  9]  0.0-10.2 sec  13.2 MBytes  10.8 Mbits/sec
[ 12]  0.0-10.2 sec  12.3 MBytes  10.0 Mbits/sec
[ 13]  0.0-10.2 sec  13.2 MBytes  10.9 Mbits/sec
[SUM]  0.0-10.2 sec   134 MBytes   110 Mbits/sec

------------------------------------------------------------
Server listening on TCP port 60145
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.31.105 port 60145 connected with 62.77.201.114 port 31893
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  23.2 MBytes  19.4 Mbits/sec

Vamp · February 19, 2020, 1:33pm

@Dmitry

A other people test it with a different network (1/1 Gbit network speed) and different machines (not vyos, he use CentOS and FreeBSD) and experience the same. One session is max 40 mbit… So i think it is a WG, not vyos problem.

Vamp · February 19, 2020, 2:51pm

@Dmitry

Well, new info… It working well between linux and linux (about 430Mbit speed, with iperf3 one session)

The problem if i test VyOS and Windows speed (i use official WireGuard on windows)