GRE (non-ipsec) throughput

wesgeorge · January 13, 2020, 3:38am

I am using VyOS 1.3-rolling-202001110217 on bare metal HP DL360G7 with a single E5620 CPU (4 core + HT), 32GB RAM, along with Broadcom BCM5709 4xGE as a dedicated IPv6 tunnel router. There are 3 interfaces active- the first is the uplink, and is IPv4-only. The other two are two dedicated IPv6-only networks. IPv6 Traffic routes to/from those networks over the GRE tunnel, which is carrying IPv6 traffic to a remote router, with BGP running across it. GRE traffic is sourced from the IP of Eth0. No IPSec, just plain vanilla GRE.
This replaces an UBNT ER3 (Cavium dual-core) that was doing the same thing. The ER3 was CPU-limited to ~160mbps on the tunnel because IPv6/GRE was not hardware accelerated.
Testing shows a very similar throughput limit in VyOS on much stouter hardware, but the CPU never exceeds a few percent. I have tried removing the firewall on the GRE interface, tweaking smp affinities, and increasing the buffer sizes of the ethernet interfaces to the maximum supported by the card with ethtool with almost no improvement in performance.
iperf3 between the router and another server just beyond the tunnel endpoint can reach 850+mbps over IPv4 (bypasses the tunnel but uses same network path), but only 150 over IPv6 (through the tunnel). A speed test through the device to a downstream host sees basically the same results, which points at GRE as some sort of bottleneck. Other users on this same setup (with ER10s) are routinely getting closer to linerate throughput, so this isn’t a bottleneck on the remote end as far as we can determine.

What can I check to see where I might be getting a bottleneck? What other optimizations should I be implementing? Is there any difference between IPIP and GRE tunnels in terms of performance such that I should try IPIP instead?

Here’s a sanitized config.

me@vyos:~$ show conf
firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians disable
    name Inet6Inbound {
        default-action drop
        description "Inbound rule set from the world"
        rule 10 {
            action accept
            description "Established connections"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            description "ICMP (all types)"
            protocol icmp
        }
        rule 30 {
            action accept
            description "Post-NAT ssh inbound (all inside)"
            destination {
                port 22
            }
            protocol tcp
        }
    }
    name InternetInbound {
        default-action drop
        description "Inbound rule set from the world"
        rule 1 {
            action accept
            description "Established connections"
            state {
                established enable
                related enable
            }
        }
        rule 10 {
            action accept
            description "ICMP (all types)"
            protocol icmp
        }
        rule 15 {
            action drop
            description fail2ban
            destination {
                port 22
            }
            protocol tcp
            recent {
                count 4
                time 60
            }
            state {
                new enable
            }
        }
        rule 20 {
            action accept
            description "Post-NAT ssh inbound (all inside)"
            destination {
                port 22
            }
            protocol tcp
        }
        rule 30 {
            action accept
            description "ip6ip tunnel"
            protocol 41
        }
        rule 40 {
            action accept
            description "gre tunnel"
            protocol gre
        }
        rule 50 {
            action accept
            description NTP
            destination {
                port ntp
            }
            protocol udp
        }
    }
    options {
        interface tun1 {
            adjust-mss6 1416
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
    twa-hazards-protection disable
}
interfaces {
    ethernet eth0 {
        address 192.168.1.253/24
        duplex auto
        firewall {
            in {
                name InternetInbound
            }
            local {
                name InternetInbound
            }
        }
        hw-id e4:11:5b:d9:96:c4
        smp-affinity auto
        speed auto
    }
    ethernet eth1 {
        address {redacted}:8::1/64
        duplex auto
        hw-id e4:11:5b:d9:96:c6
        smp-affinity auto
        speed auto
    }
    ethernet eth2 {
        address {redacted}8:666::1/64
        duplex auto
        hw-id e4:11:5b:d9:a8:7c
        smp-affinity auto
        speed auto
    }
    ethernet eth3 {
        address dhcp
        hw-id e4:11:5b:d9:a8:7e
        smp-affinity auto
    }
    loopback lo {
    }
    tunnel tun1 {
        address {redacted}C::2/64
        encapsulation gre
        firewall {
            in {
                name Inet6Inbound
            }
        }
        local-ip 192.168.1.253
        multicast disable
        remote-ip {redacted}.252.251
    }
}
protocols {
    bgp 65532 {
        address-family {
            ipv6-unicast {
                aggregate-address {redacted}:8::/48 {
                    summary-only
                }
                network {redacted}:8::/48 {
                }
                redistribute {
                    connected {
                    }
                }
            }
        }
        neighbor {redacted}C::1 {
            address-family {
                ipv6-unicast {
                }
            }
            remote-as {redacted}
        }
    }
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.1.1 {
            }
        }
    }
}
service {
    lldp {
        interface all {
        }
    }
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
}

zsdc · January 14, 2020, 12:40pm

Hello, @wesgeorge!
It is a bit hard to fully understand your topology without the scheme, so it would be great if you will add some topology map to this post.
But, as far as I see the situation, I could advise you to check MTU/MSS on clients and tunnel, and RTT between test points, as packet fragmentation and high or unstable RTT values could be one of the reasons for speed regression.
For example, you may try to run iperf UDP test with different packet sizes (1000, 1100, 1200, 1300, 1400) and check if at some point you will not get decreasing of throughput.

wesgeorge · January 16, 2020, 4:28am

using udp in iperf3 causes a major drop in performance. I suspect that there is a rate-limit on UDP on the remote server I’m using, as throughput drops to ~1-2mbps regardless of packet size.
specifying a packet size via -l for TCP has no effect on throughput. I tested everything from 900 (definitely not fragmented) to 2000 (definitely fragmented) and got the same ~150mbps throughput.
In addition to the firewall config that is imposing a MSS, the GRE tunnel MTU is set properly and the entire path supports a 1500 byte MTU.
RTT is consistent at 3-4ms.
Local transfers using iperf3 over IPv6 between the Vyos router and a local host are able to reach 900+mbps, so the router is capable of forwarding/processing IPv6 packets at the appropriate rate, so I’m really suspecting something is wrong on the GRE encap/decap side.
Are there any commands I can issue to see whether packets are being dropped, fragmented, or otherwise bottlenecked by the vyos system itself, either at IPv6 or once they are encapsulated in GRE?

wesgeorge · January 16, 2020, 4:30am

as for the topology it is very simple. Ethernet 2 and Ethernet 3 are local segments with a switch connected to them for end hosts. Ethernet 1 is connected to my ISP router and is the source for the GRE tunnel packets. IPv6 traffic to/from Ethernet2 and 3 to the Internet is encapsulated in GRE, sent to the tunnel destination, and then decapsulated to be routed. No magic here. Just a tunnel to get IPv6 packets over my stupid IPv4-only FiOS connection.

zsdc · January 16, 2020, 1:48pm

Can you show the exact iperf command, which you use for speed testing (TCP and UDP variants), and the output of " sudo ip -s l " commands from both sides of the GRE tunnel?
GRE protocol is very light and simple and implemented at the kernel level, so I believe that there must be something else, not related to GRE directly.

wesgeorge · January 17, 2020, 3:12am

iperf3 -c {host} -l {length}. Udp just adds -u
here’s the output from the vyos side. the other side is a juniper, so that command is not applicable.

    wes@c-3po:~$ sudo ip -s l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast
    23239      172      0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    23239      172      0       0       0       0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether e4:11:5b:d9:96:c4 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    18114365709 30033693 0       0       0       284982
    TX: bytes  packets  errors  dropped carrier collsns
    9368684265 43618581 0       0       0       0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether e4:11:5b:d9:96:c6 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    4743958594 44050223 0       0       0       303754
    TX: bytes  packets  errors  dropped carrier collsns
    16062459779 13060817 0       0       0       0
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether e4:11:5b:d9:a8:7c brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    24697294   296101   0       0       0       239182
    TX: bytes  packets  errors  dropped carrier collsns
    2280011912 17170701 0       0       0       0
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether e4:11:5b:d9:a8:7e brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    0          0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0
6: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
    RX: bytes  packets  errors  dropped overrun mcast
    0          0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0
7: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    0          0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0
8: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    0          0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0
9: tun1@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1476 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/gre 192.168.1.253 peer <redacted>.251
    RX: bytes  packets  errors  dropped overrun mcast
    16771030032 28528021 0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    5345561121 40957513 253     0       253     0

zsdc · January 17, 2020, 10:55am

And the output from the remote side?
So, 192.168.1.253 is a real address from your config? Is this GRE link through the Internet or local network? What devices are located between GRE endpoints?
To avoid flooding the whole channel, iperf in UDP mode transmit with only 1 Mpbs, you need to define expected channel bandwidth obviously in command. Try to run a speed test with the followed commands (and give us the output):

iperf3 -c SERVERADDRESS -f m -i 10 -t 60 -l 1100 -M 1100 -P 1
iperf3 -c SERVERADDRESS -f m -i 10 -t 60 -l 1100 -M 1100 -P 4
iperf3 -c SERVERADDRESS -f m -i 10 -t 60 -l 1100 -M 1100 -P 1 -u -b 700M
iperf3 -c SERVERADDRESS -f m -i 10 -t 60 -l 1100 -M 1100 -P 4 -u -b 175M

You may tune 700M/175M (700 Mbps, 175 Mbps) values to higher or lower if you will see that GRE is not utilized fully or there will be high values in Lost Datagrams.

wesgeorge · January 18, 2020, 4:10am

as noted the remote side is not vyos. command is not applicable.
The ISP router upstream is natting the traffic outbound, i.e. the GRE is actually being sourced from the external address. NAT works at linerate, as it is possible to push 900mbps over IPv4 between the same two devices using iperf.

doesn’t appear to like tcp mss, but here is the output of the commands I ran, both on the Vyos side and the remote server (not the remote end of the GRE tunnel, but a device downstream of the remote GRE router.

 iperf3: error - unable to set TCP/SCTP MSS: Cannot assign requested address
 wes@c-3po:~$ sudo iperf3 -c {SERVER} -f m -i 10 -t 60 -l 1100 -P 1
 Connecting to host {SERVER}, port 5201
 [  5] local {ip6_prefix}:fff:fffc::2 port 45376 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
 [  5]   0.00-10.00  sec   179 MBytes   150 Mbits/sec   27    289 KBytes
 [  5]  10.00-20.00  sec   170 MBytes   142 Mbits/sec   15    271 KBytes
 [  5]  20.00-30.00  sec   176 MBytes   148 Mbits/sec   24    266 KBytes
 [  5]  30.00-40.00  sec   180 MBytes   151 Mbits/sec   21    261 KBytes
 [  5]  40.00-50.00  sec   179 MBytes   151 Mbits/sec   32    265 KBytes
 [  5]  50.00-60.00  sec   176 MBytes   147 Mbits/sec   24    248 KBytes
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval           Transfer     Bitrate         Retr
 [  5]   0.00-60.00  sec  1.04 GBytes   148 Mbits/sec  143             sender
 [  5]   0.00-60.05  sec  1.03 GBytes   148 Mbits/sec                  receiver
 
 iperf Done.
 wes@c-3po:~$ sudo iperf3 -c {SERVER} -f m -i 10 -t 60 -l 1100 -P 4
 Connecting to host {SERVER}, port 5201
 [  5] local {ip6_prefix}:fff:fffc::2 port 45380 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [  7] local {ip6_prefix}:fff:fffc::2 port 45382 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [  9] local {ip6_prefix}:fff:fffc::2 port 45384 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [ 11] local {ip6_prefix}:fff:fffc::2 port 45386 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
 [  5]   0.00-10.00  sec  52.9 MBytes  44.4 Mbits/sec   68   75.4 KBytes
 [  7]   0.00-10.00  sec  37.7 MBytes  31.7 Mbits/sec   35    126 KBytes
 [  9]   0.00-10.00  sec  35.9 MBytes  30.1 Mbits/sec   39   38.4 KBytes
 [ 11]   0.00-10.00  sec  47.8 MBytes  40.1 Mbits/sec   46   91.9 KBytes
 [SUM]   0.00-10.00  sec   174 MBytes   146 Mbits/sec  188
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  10.00-20.00  sec  47.5 MBytes  39.9 Mbits/sec   40    104 KBytes
 [  7]  10.00-20.00  sec  44.8 MBytes  37.6 Mbits/sec   51   83.6 KBytes
 [  9]  10.00-20.00  sec  38.3 MBytes  32.1 Mbits/sec   57   78.2 KBytes
 [ 11]  10.00-20.00  sec  50.6 MBytes  42.5 Mbits/sec   38   65.8 KBytes
 [SUM]  10.00-20.00  sec   181 MBytes   152 Mbits/sec  186
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  20.00-30.00  sec  41.4 MBytes  34.8 Mbits/sec   61   82.3 KBytes
 [  7]  20.00-30.00  sec  48.9 MBytes  41.0 Mbits/sec   47   82.3 KBytes
 [  9]  20.00-30.00  sec  51.0 MBytes  42.8 Mbits/sec   38   98.7 KBytes
 [ 11]  20.00-30.00  sec  40.8 MBytes  34.2 Mbits/sec   45   45.2 KBytes
 [SUM]  20.00-30.00  sec   182 MBytes   153 Mbits/sec  191
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  30.00-40.00  sec  41.6 MBytes  34.9 Mbits/sec   47   65.8 KBytes
 [  7]  30.00-40.00  sec  52.9 MBytes  44.4 Mbits/sec   40   83.6 KBytes
 [  9]  30.00-40.00  sec  45.9 MBytes  38.5 Mbits/sec   66   61.7 KBytes
 [ 11]  30.00-40.00  sec  39.8 MBytes  33.4 Mbits/sec   39   75.4 KBytes
 [SUM]  30.00-40.00  sec   180 MBytes   151 Mbits/sec  192
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  40.00-50.00  sec  45.2 MBytes  37.9 Mbits/sec   43    108 KBytes
 [  7]  40.00-50.00  sec  51.3 MBytes  43.0 Mbits/sec   48   93.2 KBytes
 [  9]  40.00-50.00  sec  39.2 MBytes  32.8 Mbits/sec   56   72.7 KBytes
 [ 11]  40.00-50.00  sec  44.1 MBytes  37.0 Mbits/sec   40   64.4 KBytes
 [SUM]  40.00-50.00  sec   180 MBytes   151 Mbits/sec  187
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  50.00-60.00  sec  48.3 MBytes  40.5 Mbits/sec   76   78.2 KBytes
 [  7]  50.00-60.00  sec  38.7 MBytes  32.4 Mbits/sec   54   72.7 KBytes
 [  9]  50.00-60.00  sec  51.8 MBytes  43.5 Mbits/sec   49   75.4 KBytes
 [ 11]  50.00-60.00  sec  39.3 MBytes  33.0 Mbits/sec   52    129 KBytes
 [SUM]  50.00-60.00  sec   178 MBytes   149 Mbits/sec  231
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval           Transfer     Bitrate         Retr
 [  5]   0.00-60.00  sec   277 MBytes  38.7 Mbits/sec  335             sender
 [  5]   0.00-60.04  sec   276 MBytes  38.6 Mbits/sec                  receiver
 [  7]   0.00-60.00  sec   274 MBytes  38.3 Mbits/sec  275             sender
 [  7]   0.00-60.04  sec   274 MBytes  38.2 Mbits/sec                  receiver
 [  9]   0.00-60.00  sec   262 MBytes  36.6 Mbits/sec  305             sender
 [  9]   0.00-60.04  sec   262 MBytes  36.5 Mbits/sec                  receiver
 [ 11]   0.00-60.00  sec   262 MBytes  36.7 Mbits/sec  260             sender
 [ 11]   0.00-60.04  sec   262 MBytes  36.6 Mbits/sec                  receiver
 [SUM]   0.00-60.00  sec  1.05 GBytes   150 Mbits/sec  1175             sender
 [SUM]   0.00-60.04  sec  1.05 GBytes   150 Mbits/sec                  receiver
 
 iperf Done.
 wes@c-3po:~$ sudo iperf3 -c {SERVER} -f m -i 10 -t 60 -l 1100 -M 1100 -P 4 -u -b 175M
 Connecting to host {SERVER}, port 5201
 [  5] local {ip6_prefix}:fff:fffc::2 port 50502 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [  7] local {ip6_prefix}:fff:fffc::2 port 50735 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [  9] local {ip6_prefix}:fff:fffc::2 port 49309 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [ 11] local {ip6_prefix}:fff:fffc::2 port 51207 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [ ID] Interval           Transfer     Bitrate         Total Datagrams
 [  5]   0.00-10.00  sec   199 MBytes   167 Mbits/sec  189383
 [  7]   0.00-10.00  sec   199 MBytes   167 Mbits/sec  189383
 [  9]   0.00-10.00  sec   199 MBytes   167 Mbits/sec  189383
 [ 11]   0.00-10.00  sec   199 MBytes   167 Mbits/sec  189383
 [SUM]   0.00-10.00  sec   795 MBytes   667 Mbits/sec  757532
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  10.00-20.00  sec   198 MBytes   166 Mbits/sec  189193
 [  7]  10.00-20.00  sec   198 MBytes   166 Mbits/sec  189193
 [  9]  10.00-20.00  sec   198 MBytes   166 Mbits/sec  189193
 [ 11]  10.00-20.00  sec   198 MBytes   166 Mbits/sec  189193
 [SUM]  10.00-20.00  sec   794 MBytes   666 Mbits/sec  756772
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  20.00-30.00  sec   199 MBytes   167 Mbits/sec  189242
 [  7]  20.00-30.00  sec   199 MBytes   167 Mbits/sec  189242
 [  9]  20.00-30.00  sec   199 MBytes   167 Mbits/sec  189242
 [ 11]  20.00-30.00  sec   199 MBytes   167 Mbits/sec  189242
 [SUM]  20.00-30.00  sec   794 MBytes   666 Mbits/sec  756968
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  30.00-40.00  sec   199 MBytes   167 Mbits/sec  189229
 [  7]  30.00-40.00  sec   199 MBytes   167 Mbits/sec  189229
 [  9]  30.00-40.00  sec   199 MBytes   167 Mbits/sec  189229
 [ 11]  30.00-40.00  sec   199 MBytes   167 Mbits/sec  189229
 [SUM]  30.00-40.00  sec   794 MBytes   666 Mbits/sec  756916
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  40.00-50.00  sec   198 MBytes   166 Mbits/sec  188963
 [  7]  40.00-50.00  sec   198 MBytes   166 Mbits/sec  188963
 [  9]  40.00-50.00  sec   198 MBytes   166 Mbits/sec  188963
 [ 11]  40.00-50.00  sec   198 MBytes   166 Mbits/sec  188963
 [SUM]  40.00-50.00  sec   793 MBytes   665 Mbits/sec  755852
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  50.00-60.00  sec   206 MBytes   173 Mbits/sec  196795
 [  7]  50.00-60.00  sec   206 MBytes   173 Mbits/sec  196795
 [  9]  50.00-60.00  sec   206 MBytes   173 Mbits/sec  196795
 [ 11]  50.00-60.00  sec   206 MBytes   173 Mbits/sec  196795
 [SUM]  50.00-60.00  sec   826 MBytes   693 Mbits/sec  787180
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
 [  5]   0.00-60.00  sec  1.17 GBytes   168 Mbits/sec  0.000 ms  0/1142805 (0%)  sender
 [  5]   0.00-60.24  sec   241 MBytes  33.5 Mbits/sec  0.306 ms  913214/1142746 (80%)  receiver
 [  7]   0.00-60.00  sec  1.17 GBytes   168 Mbits/sec  0.000 ms  0/1142805 (0%)  sender
 [  7]   0.00-60.24  sec   241 MBytes  33.5 Mbits/sec  0.318 ms  913213/1142746 (80%)  receiver
 [  9]   0.00-60.00  sec  1.17 GBytes   168 Mbits/sec  0.000 ms  0/1142805 (0%)  sender
 [  9]   0.00-60.24  sec   241 MBytes  33.5 Mbits/sec  0.361 ms  913199/1142746 (80%)  receiver
 [ 11]   0.00-60.00  sec  1.17 GBytes   168 Mbits/sec  0.000 ms  0/1142805 (0%)  sender
 [ 11]   0.00-60.24  sec   241 MBytes  33.5 Mbits/sec  0.368 ms  913252/1142746 (80%)  receiver
 [SUM]   0.00-60.00  sec  4.68 GBytes   670 Mbits/sec  0.000 ms  0/4571220 (0%)  sender
 [SUM]   0.00-60.24  sec   963 MBytes   134 Mbits/sec  0.338 ms  3652878/4570984 (80%)  receiver
 
 iperf Done.
 wes@c-3po:~$ sudo iperf3 -c {SERVER} -f m -i 10 -t 60 -l 1100 -M 1100 -P 1 -u -b 700M
 Connecting to host {SERVER}, port 5201
 [  5] local {ip6_prefix}:fff:fffc::2 port 42629 connected to {ip6_prefix}:2:1:0:214:66:1 port 5201
 [ ID] Interval           Transfer     Bitrate         Total Datagrams
 [  5]   0.00-10.00  sec   679 MBytes   570 Mbits/sec  647161
 [  5]  10.00-20.00  sec   679 MBytes   570 Mbits/sec  647673
 [  5]  20.00-30.00  sec   677 MBytes   568 Mbits/sec  645263
 [  5]  30.00-40.00  sec   682 MBytes   572 Mbits/sec  650359
 [  5]  40.00-50.00  sec   680 MBytes   570 Mbits/sec  648270
 [  5]  50.00-60.00  sec   680 MBytes   570 Mbits/sec  648032
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
 [  5]   0.00-60.00  sec  3.98 GBytes   570 Mbits/sec  0.000 ms  0/3886758 (0%)  sender
 [  5]   0.00-60.06  sec   964 MBytes   135 Mbits/sec  0.091 ms  2967886/3886753 (76%)  receiver
 
 iperf Done.

remote side will be in the next post due to character limits

wesgeorge · January 18, 2020, 4:13am

and the remote side part 1/2:
 
 wes@{SERVER}:~$ iperf3 -s
 -----------------------------------------------------------
 Server listening on 5201
 -----------------------------------------------------------
 iperf3: error - unable to set TCP/SCTP MSS: Bad file number
 -----------------------------------------------------------
 Server listening on 5201
 -----------------------------------------------------------
 Accepted connection from {ip6_prefix}:fff:fffc::2, port 45374
 [  5] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 45376
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-1.00   sec  15.8 MBytes   133 Mbits/sec
 [  5]   1.00-2.00   sec  17.7 MBytes   149 Mbits/sec
 [  5]   2.00-3.00   sec  18.3 MBytes   153 Mbits/sec
 [  5]   3.00-4.00   sec  18.5 MBytes   155 Mbits/sec
 [  5]   4.00-5.00   sec  18.2 MBytes   152 Mbits/sec
 [  5]   5.00-6.00   sec  17.5 MBytes   147 Mbits/sec
 [  5]   6.00-7.00   sec  17.3 MBytes   145 Mbits/sec
 [  5]   7.00-8.00   sec  18.2 MBytes   153 Mbits/sec
 [  5]   8.00-9.00   sec  17.2 MBytes   144 Mbits/sec
 [  5]   9.00-10.00  sec  18.2 MBytes   153 Mbits/sec
 [  5]  10.00-11.00  sec  18.1 MBytes   152 Mbits/sec
 [  5]  11.00-12.00  sec  18.3 MBytes   153 Mbits/sec
 [  5]  12.00-13.00  sec  18.1 MBytes   152 Mbits/sec
 [  5]  13.00-14.00  sec  11.5 MBytes  96.8 Mbits/sec
 [  5]  14.00-15.00  sec  14.2 MBytes   119 Mbits/sec
 [  5]  15.00-16.00  sec  17.6 MBytes   148 Mbits/sec
 [  5]  16.00-17.00  sec  17.6 MBytes   147 Mbits/sec
 [  5]  17.00-18.00  sec  18.0 MBytes   151 Mbits/sec
 [  5]  18.00-19.00  sec  18.1 MBytes   152 Mbits/sec
 [  5]  19.00-20.00  sec  18.2 MBytes   153 Mbits/sec
 [  5]  20.00-21.00  sec  17.7 MBytes   149 Mbits/sec
 [  5]  21.00-22.00  sec  18.1 MBytes   152 Mbits/sec
 [  5]  22.00-23.00  sec  17.3 MBytes   145 Mbits/sec
 [  5]  23.00-24.00  sec  18.4 MBytes   154 Mbits/sec
 [  5]  24.00-25.00  sec  15.7 MBytes   132 Mbits/sec
 [  5]  25.00-26.00  sec  17.9 MBytes   150 Mbits/sec
 [  5]  26.00-27.00  sec  16.7 MBytes   140 Mbits/sec
 [  5]  27.00-28.00  sec  17.0 MBytes   143 Mbits/sec
 [  5]  28.00-29.00  sec  18.4 MBytes   155 Mbits/sec
 [  5]  29.00-30.00  sec  18.6 MBytes   156 Mbits/sec
 [  5]  30.00-31.00  sec  18.7 MBytes   157 Mbits/sec
 [  5]  31.00-32.00  sec  17.7 MBytes   149 Mbits/sec
 [  5]  32.00-33.00  sec  18.2 MBytes   152 Mbits/sec
 [  5]  33.00-34.00  sec  18.4 MBytes   154 Mbits/sec
 [  5]  34.00-35.00  sec  18.4 MBytes   154 Mbits/sec
 [  5]  35.00-36.00  sec  17.9 MBytes   150 Mbits/sec
 [  5]  36.00-37.00  sec  17.6 MBytes   148 Mbits/sec
 [  5]  37.00-38.00  sec  17.6 MBytes   147 Mbits/sec
 [  5]  38.00-39.00  sec  18.5 MBytes   155 Mbits/sec
 [  5]  39.00-40.00  sec  17.1 MBytes   144 Mbits/sec
 [  5]  40.00-41.00  sec  18.5 MBytes   155 Mbits/sec
 [  5]  41.00-42.00  sec  17.7 MBytes   148 Mbits/sec
 [  5]  42.00-43.00  sec  18.3 MBytes   153 Mbits/sec
 [  5]  43.00-44.00  sec  18.1 MBytes   152 Mbits/sec
 [  5]  44.00-45.00  sec  17.7 MBytes   148 Mbits/sec
 [  5]  45.00-46.00  sec  18.1 MBytes   152 Mbits/sec
 [  5]  46.00-47.00  sec  17.5 MBytes   147 Mbits/sec
 [  5]  47.00-48.00  sec  16.6 MBytes   140 Mbits/sec
 [  5]  48.00-49.00  sec  18.5 MBytes   155 Mbits/sec
 [  5]  49.00-50.00  sec  18.5 MBytes   155 Mbits/sec
 [  5]  50.00-51.00  sec  18.6 MBytes   156 Mbits/sec
 [  5]  51.00-52.00  sec  18.2 MBytes   152 Mbits/sec
 [  5]  52.00-53.00  sec  17.6 MBytes   148 Mbits/sec
 [  5]  53.00-54.00  sec  16.5 MBytes   138 Mbits/sec
 [  5]  54.00-55.00  sec  18.2 MBytes   153 Mbits/sec
 [  5]  55.00-56.00  sec  18.3 MBytes   153 Mbits/sec
 [  5]  56.00-57.00  sec  16.5 MBytes   139 Mbits/sec
 [  5]  57.00-58.00  sec  17.9 MBytes   150 Mbits/sec
 [  5]  58.00-59.00  sec  18.5 MBytes   155 Mbits/sec
 [  5]  59.00-60.00  sec  15.4 MBytes   129 Mbits/sec
 [  5]  60.00-60.05  sec  1.17 MBytes   203 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-60.05  sec  1.03 GBytes   148 Mbits/sec                  receiver
 -----------------------------------------------------------
 Server listening on 5201
 -----------------------------------------------------------
 Accepted connection from {ip6_prefix}:fff:fffc::2, port 45378
 [  5] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 45380
 [  8] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 45382
 [ 10] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 45384
 [ 12] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 45386
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-1.00   sec  8.23 MBytes  69.0 Mbits/sec
 [  8]   0.00-1.00   sec  2.69 MBytes  22.5 Mbits/sec
 [ 10]   0.00-1.00   sec  2.61 MBytes  21.8 Mbits/sec
 [ 12]   0.00-1.00   sec  3.95 MBytes  33.1 Mbits/sec
 [SUM]   0.00-1.00   sec  17.5 MBytes   147 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   1.00-2.00   sec  6.32 MBytes  53.0 Mbits/sec
 [  8]   1.00-2.00   sec  3.01 MBytes  25.3 Mbits/sec
 [ 10]   1.00-2.00   sec  4.15 MBytes  34.8 Mbits/sec
 [ 12]   1.00-2.00   sec  3.88 MBytes  32.5 Mbits/sec
 [SUM]   1.00-2.00   sec  17.4 MBytes   146 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   2.00-3.00   sec  5.88 MBytes  49.3 Mbits/sec
 [  8]   2.00-3.00   sec  3.34 MBytes  28.0 Mbits/sec
 [ 10]   2.00-3.00   sec  4.06 MBytes  34.1 Mbits/sec
 [ 12]   2.00-3.00   sec  3.85 MBytes  32.3 Mbits/sec
 [SUM]   2.00-3.00   sec  17.1 MBytes   144 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   3.00-4.00   sec  3.45 MBytes  29.0 Mbits/sec
 [  8]   3.00-4.00   sec  3.50 MBytes  29.4 Mbits/sec
 [ 10]   3.00-4.00   sec  2.43 MBytes  20.4 Mbits/sec
 [ 12]   3.00-4.00   sec  3.41 MBytes  28.6 Mbits/sec
 [SUM]   3.00-4.00   sec  12.8 MBytes   107 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   4.00-5.00   sec  4.54 MBytes  38.1 Mbits/sec
 [  8]   4.00-5.00   sec  4.27 MBytes  35.8 Mbits/sec
 [ 10]   4.00-5.00   sec  3.59 MBytes  30.1 Mbits/sec
 [ 12]   4.00-5.00   sec  4.64 MBytes  38.9 Mbits/sec
 [SUM]   4.00-5.00   sec  17.0 MBytes   143 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   5.00-6.00   sec  4.08 MBytes  34.2 Mbits/sec
 [  8]   5.00-6.00   sec  3.69 MBytes  31.0 Mbits/sec
 [ 10]   5.00-6.00   sec  5.06 MBytes  42.5 Mbits/sec
 [ 12]   5.00-6.00   sec  5.01 MBytes  42.0 Mbits/sec
 [SUM]   5.00-6.00   sec  17.8 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   6.00-7.00   sec  4.78 MBytes  40.1 Mbits/sec
 [  8]   6.00-7.00   sec  2.99 MBytes  25.1 Mbits/sec
 [ 10]   6.00-7.00   sec  4.31 MBytes  36.2 Mbits/sec
 [ 12]   6.00-7.00   sec  5.84 MBytes  49.0 Mbits/sec
 [SUM]   6.00-7.00   sec  17.9 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   7.00-8.00   sec  4.05 MBytes  34.0 Mbits/sec
 [  8]   7.00-8.00   sec  2.99 MBytes  25.1 Mbits/sec
 [ 10]   7.00-8.00   sec  4.36 MBytes  36.5 Mbits/sec
 [ 12]   7.00-8.00   sec  6.09 MBytes  51.1 Mbits/sec
 [SUM]   7.00-8.00   sec  17.5 MBytes   147 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   8.00-9.00   sec  4.93 MBytes  41.4 Mbits/sec
 [  8]   8.00-9.00   sec  5.01 MBytes  42.0 Mbits/sec
 [ 10]   8.00-9.00   sec  2.63 MBytes  22.0 Mbits/sec
 [ 12]   8.00-9.00   sec  5.94 MBytes  49.8 Mbits/sec
 [SUM]   8.00-9.00   sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]   9.00-10.00  sec  5.66 MBytes  47.5 Mbits/sec
 [  8]   9.00-10.00  sec  5.64 MBytes  47.4 Mbits/sec
 [ 10]   9.00-10.00  sec  2.29 MBytes  19.2 Mbits/sec
 [ 12]   9.00-10.00  sec  4.57 MBytes  38.4 Mbits/sec
 [SUM]   9.00-10.00  sec  18.2 MBytes   152 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  10.00-11.00  sec  3.67 MBytes  30.8 Mbits/sec
 [  8]  10.00-11.00  sec  7.40 MBytes  62.1 Mbits/sec
 [ 10]  10.00-11.00  sec  2.85 MBytes  23.9 Mbits/sec
 [ 12]  10.00-11.00  sec  4.47 MBytes  37.5 Mbits/sec
 [SUM]  10.00-11.00  sec  18.4 MBytes   154 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  11.00-12.00  sec  3.61 MBytes  30.3 Mbits/sec
 [  8]  11.00-12.00  sec  6.06 MBytes  50.8 Mbits/sec
 [ 10]  11.00-12.00  sec  4.28 MBytes  35.9 Mbits/sec
 [ 12]  11.00-12.00  sec  4.57 MBytes  38.3 Mbits/sec
 [SUM]  11.00-12.00  sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  12.00-13.00  sec  4.76 MBytes  39.9 Mbits/sec
 [  8]  12.00-13.00  sec  3.78 MBytes  31.7 Mbits/sec
 [ 10]  12.00-13.00  sec  4.73 MBytes  39.7 Mbits/sec
 [ 12]  12.00-13.00  sec  3.95 MBytes  33.1 Mbits/sec
 [SUM]  12.00-13.00  sec  17.2 MBytes   145 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  13.00-14.00  sec  5.08 MBytes  42.6 Mbits/sec
 [  8]  13.00-14.00  sec  3.66 MBytes  30.7 Mbits/sec
 [ 10]  13.00-14.00  sec  4.65 MBytes  39.0 Mbits/sec
 [ 12]  13.00-14.00  sec  4.65 MBytes  39.0 Mbits/sec
 [SUM]  13.00-14.00  sec  18.0 MBytes   151 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  14.00-15.00  sec  5.37 MBytes  45.0 Mbits/sec
 [  8]  14.00-15.00  sec  3.91 MBytes  32.8 Mbits/sec
 [ 10]  14.00-15.00  sec  3.76 MBytes  31.5 Mbits/sec
 [ 12]  14.00-15.00  sec  5.07 MBytes  42.5 Mbits/sec
 [SUM]  14.00-15.00  sec  18.1 MBytes   152 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  15.00-16.00  sec  5.76 MBytes  48.3 Mbits/sec
 [  8]  15.00-16.00  sec  3.87 MBytes  32.5 Mbits/sec
 [ 10]  15.00-16.00  sec  3.77 MBytes  31.6 Mbits/sec
 [ 12]  15.00-16.00  sec  4.55 MBytes  38.1 Mbits/sec
 [SUM]  15.00-16.00  sec  17.9 MBytes   151 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  16.00-17.00  sec  4.46 MBytes  37.4 Mbits/sec
 [  8]  16.00-17.00  sec  3.92 MBytes  32.9 Mbits/sec
 [ 10]  16.00-17.00  sec  3.59 MBytes  30.1 Mbits/sec
 [ 12]  16.00-17.00  sec  6.52 MBytes  54.7 Mbits/sec
 [SUM]  16.00-17.00  sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  17.00-18.00  sec  5.04 MBytes  42.2 Mbits/sec
 [  8]  17.00-18.00  sec  4.00 MBytes  33.6 Mbits/sec
 [ 10]  17.00-18.00  sec  2.87 MBytes  24.1 Mbits/sec
 [ 12]  17.00-18.00  sec  5.88 MBytes  49.3 Mbits/sec
 [SUM]  17.00-18.00  sec  17.8 MBytes   149 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  18.00-19.00  sec  4.69 MBytes  39.3 Mbits/sec
 [  8]  18.00-19.00  sec  4.36 MBytes  36.5 Mbits/sec
 [ 10]  18.00-19.00  sec  3.05 MBytes  25.5 Mbits/sec
 [ 12]  18.00-19.00  sec  6.01 MBytes  50.4 Mbits/sec
 [SUM]  18.00-19.00  sec  18.1 MBytes   152 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  19.00-20.00  sec  5.04 MBytes  42.3 Mbits/sec
 [  8]  19.00-20.00  sec  3.78 MBytes  31.7 Mbits/sec
 [ 10]  19.00-20.00  sec  4.70 MBytes  39.4 Mbits/sec
 [ 12]  19.00-20.00  sec  4.98 MBytes  41.7 Mbits/sec
 [SUM]  19.00-20.00  sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  20.00-21.00  sec  4.60 MBytes  38.5 Mbits/sec
 [  8]  20.00-21.00  sec  5.46 MBytes  45.8 Mbits/sec
 [ 10]  20.00-21.00  sec  4.62 MBytes  38.7 Mbits/sec
 [ 12]  20.00-21.00  sec  3.51 MBytes  29.4 Mbits/sec
 [SUM]  20.00-21.00  sec  18.2 MBytes   152 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  21.00-22.00  sec  3.94 MBytes  33.0 Mbits/sec
 [  8]  21.00-22.00  sec  5.33 MBytes  44.7 Mbits/sec
 [ 10]  21.00-22.00  sec  4.38 MBytes  36.8 Mbits/sec
 [ 12]  21.00-22.00  sec  4.94 MBytes  41.4 Mbits/sec
 [SUM]  21.00-22.00  sec  18.6 MBytes   156 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  22.00-23.00  sec  3.82 MBytes  32.0 Mbits/sec
 [  8]  22.00-23.00  sec  4.37 MBytes  36.6 Mbits/sec
 [ 10]  22.00-23.00  sec  5.81 MBytes  48.7 Mbits/sec
 [ 12]  22.00-23.00  sec  3.43 MBytes  28.8 Mbits/sec
 [SUM]  22.00-23.00  sec  17.4 MBytes   146 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  23.00-24.00  sec  4.72 MBytes  39.6 Mbits/sec
 [  8]  23.00-24.00  sec  3.62 MBytes  30.3 Mbits/sec
 [ 10]  23.00-24.00  sec  7.39 MBytes  62.0 Mbits/sec
 [ 12]  23.00-24.00  sec  2.85 MBytes  23.9 Mbits/sec
 [SUM]  23.00-24.00  sec  18.6 MBytes   156 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  24.00-25.00  sec  3.88 MBytes  32.5 Mbits/sec
 [  8]  24.00-25.00  sec  5.03 MBytes  42.2 Mbits/sec
 [ 10]  24.00-25.00  sec  5.08 MBytes  42.6 Mbits/sec
 [ 12]  24.00-25.00  sec  4.42 MBytes  37.1 Mbits/sec
 [SUM]  24.00-25.00  sec  18.4 MBytes   154 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  25.00-26.00  sec  3.08 MBytes  25.8 Mbits/sec
 [  8]  25.00-26.00  sec  7.02 MBytes  58.9 Mbits/sec
 [ 10]  25.00-26.00  sec  3.41 MBytes  28.6 Mbits/sec
 [ 12]  25.00-26.00  sec  4.62 MBytes  38.8 Mbits/sec
 [SUM]  25.00-26.00  sec  18.1 MBytes   152 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  26.00-27.00  sec  4.04 MBytes  33.9 Mbits/sec
 [  8]  26.00-27.00  sec  5.08 MBytes  42.6 Mbits/sec
 [ 10]  26.00-27.00  sec  5.36 MBytes  44.9 Mbits/sec
 [ 12]  26.00-27.00  sec  4.17 MBytes  35.0 Mbits/sec
 [SUM]  26.00-27.00  sec  18.6 MBytes   156 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  27.00-28.00  sec  4.71 MBytes  39.5 Mbits/sec
 [  8]  27.00-28.00  sec  3.74 MBytes  31.4 Mbits/sec
 [ 10]  27.00-28.00  sec  4.59 MBytes  38.5 Mbits/sec
 [ 12]  27.00-28.00  sec  4.13 MBytes  34.7 Mbits/sec
 [SUM]  27.00-28.00  sec  17.2 MBytes   144 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  28.00-29.00  sec  3.97 MBytes  33.3 Mbits/sec
 [  8]  28.00-29.00  sec  4.54 MBytes  38.0 Mbits/sec
 [ 10]  28.00-29.00  sec  4.29 MBytes  36.0 Mbits/sec
 [ 12]  28.00-29.00  sec  5.48 MBytes  45.9 Mbits/sec
 [SUM]  28.00-29.00  sec  18.3 MBytes   153 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  29.00-30.00  sec  4.67 MBytes  39.2 Mbits/sec
 [  8]  29.00-30.00  sec  4.70 MBytes  39.4 Mbits/sec
 [ 10]  29.00-30.00  sec  5.89 MBytes  49.4 Mbits/sec
 [ 12]  29.00-30.00  sec  3.22 MBytes  27.1 Mbits/sec
 [SUM]  29.00-30.00  sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  30.00-31.00  sec  4.36 MBytes  36.6 Mbits/sec
 [  8]  30.00-31.00  sec  5.38 MBytes  45.2 Mbits/sec
 [ 10]  30.00-31.00  sec  5.38 MBytes  45.2 Mbits/sec
 [ 12]  30.00-31.00  sec  3.44 MBytes  28.9 Mbits/sec
 [SUM]  30.00-31.00  sec  18.6 MBytes   156 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  31.00-32.00  sec  4.10 MBytes  34.4 Mbits/sec
 [  8]  31.00-32.00  sec  6.00 MBytes  50.3 Mbits/sec
 [ 10]  31.00-32.00  sec  4.56 MBytes  38.2 Mbits/sec
 [ 12]  31.00-32.00  sec  3.98 MBytes  33.4 Mbits/sec
 [SUM]  31.00-32.00  sec  18.6 MBytes   156 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  32.00-33.00  sec  4.44 MBytes  37.3 Mbits/sec
 [  8]  32.00-33.00  sec  5.69 MBytes  47.8 Mbits/sec
 [ 10]  32.00-33.00  sec  3.51 MBytes  29.5 Mbits/sec
 [ 12]  32.00-33.00  sec  4.09 MBytes  34.3 Mbits/sec
 [SUM]  32.00-33.00  sec  17.7 MBytes   149 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  33.00-34.00  sec  4.75 MBytes  39.8 Mbits/sec
 [  8]  33.00-34.00  sec  6.76 MBytes  56.7 Mbits/sec
 [ 10]  33.00-34.00  sec  3.14 MBytes  26.4 Mbits/sec
 [ 12]  33.00-34.00  sec  3.63 MBytes  30.4 Mbits/sec
 [SUM]  33.00-34.00  sec  18.3 MBytes   153 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  34.00-35.00  sec  3.68 MBytes  30.9 Mbits/sec
 [  8]  34.00-35.00  sec  6.34 MBytes  53.2 Mbits/sec
 [ 10]  34.00-35.00  sec  4.39 MBytes  36.8 Mbits/sec
 [ 12]  34.00-35.00  sec  2.99 MBytes  25.1 Mbits/sec
 [SUM]  34.00-35.00  sec  17.4 MBytes   146 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  35.00-36.00  sec  3.97 MBytes  33.3 Mbits/sec
 [  8]  35.00-36.00  sec  4.22 MBytes  35.4 Mbits/sec
 [ 10]  35.00-36.00  sec  5.20 MBytes  43.6 Mbits/sec
 [ 12]  35.00-36.00  sec  4.53 MBytes  38.0 Mbits/sec
 [SUM]  35.00-36.00  sec  17.9 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  36.00-37.00  sec  4.11 MBytes  34.5 Mbits/sec
 [  8]  36.00-37.00  sec  3.91 MBytes  32.8 Mbits/sec
 [ 10]  36.00-37.00  sec  4.90 MBytes  41.1 Mbits/sec
 [ 12]  36.00-37.00  sec  4.96 MBytes  41.6 Mbits/sec
 [SUM]  36.00-37.00  sec  17.9 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  37.00-38.00  sec  3.16 MBytes  26.5 Mbits/sec
 [  8]  37.00-38.00  sec  4.45 MBytes  37.3 Mbits/sec
 [ 10]  37.00-38.00  sec  5.52 MBytes  46.3 Mbits/sec
 [ 12]  37.00-38.00  sec  3.80 MBytes  31.9 Mbits/sec
 [SUM]  37.00-38.00  sec  16.9 MBytes   142 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  38.00-39.00  sec  4.61 MBytes  38.6 Mbits/sec
 [  8]  38.00-39.00  sec  4.97 MBytes  41.7 Mbits/sec
 [ 10]  38.00-39.00  sec  4.94 MBytes  41.4 Mbits/sec
 [ 12]  38.00-39.00  sec  3.90 MBytes  32.7 Mbits/sec
 [SUM]  38.00-39.00  sec  18.4 MBytes   154 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  39.00-40.00  sec  4.40 MBytes  37.0 Mbits/sec
 [  8]  39.00-40.00  sec  5.16 MBytes  43.3 Mbits/sec
 [ 10]  39.00-40.00  sec  4.37 MBytes  36.7 Mbits/sec
 [ 12]  39.00-40.00  sec  4.44 MBytes  37.3 Mbits/sec
 [SUM]  39.00-40.00  sec  18.4 MBytes   154 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  40.00-41.00  sec  4.14 MBytes  34.7 Mbits/sec
 [  8]  40.00-41.00  sec  5.06 MBytes  42.5 Mbits/sec
 [ 10]  40.00-41.00  sec  4.68 MBytes  39.2 Mbits/sec
 [ 12]  40.00-41.00  sec  4.60 MBytes  38.6 Mbits/sec
 [SUM]  40.00-41.00  sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  41.00-42.00  sec  5.07 MBytes  42.5 Mbits/sec
 [  8]  41.00-42.00  sec  3.92 MBytes  32.9 Mbits/sec
 [ 10]  41.00-42.00  sec  3.98 MBytes  33.4 Mbits/sec
 [ 12]  41.00-42.00  sec  5.43 MBytes  45.6 Mbits/sec
 [SUM]  41.00-42.00  sec  18.4 MBytes   154 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  42.00-43.00  sec  4.06 MBytes  34.0 Mbits/sec
 [  8]  42.00-43.00  sec  4.88 MBytes  40.9 Mbits/sec
 [ 10]  42.00-43.00  sec  4.18 MBytes  35.1 Mbits/sec
 [ 12]  42.00-43.00  sec  4.38 MBytes  36.8 Mbits/sec
 [SUM]  42.00-43.00  sec  17.5 MBytes   147 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  43.00-44.00  sec  4.24 MBytes  35.6 Mbits/sec
 [  8]  43.00-44.00  sec  4.10 MBytes  34.4 Mbits/sec
 [ 10]  43.00-44.00  sec  4.10 MBytes  34.4 Mbits/sec
 [ 12]  43.00-44.00  sec  5.17 MBytes  43.4 Mbits/sec
 [SUM]  43.00-44.00  sec  17.6 MBytes   148 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  44.00-45.00  sec  4.77 MBytes  40.1 Mbits/sec
 [  8]  44.00-45.00  sec  4.24 MBytes  35.6 Mbits/sec
 [ 10]  44.00-45.00  sec  3.79 MBytes  31.8 Mbits/sec
 [ 12]  44.00-45.00  sec  5.00 MBytes  41.9 Mbits/sec
 [SUM]  44.00-45.00  sec  17.8 MBytes   149 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  45.00-46.00  sec  5.56 MBytes  46.6 Mbits/sec
 [  8]  45.00-46.00  sec  6.21 MBytes  52.1 Mbits/sec
 [ 10]  45.00-46.00  sec  2.40 MBytes  20.1 Mbits/sec
 [ 12]  45.00-46.00  sec  3.76 MBytes  31.5 Mbits/sec
 [SUM]  45.00-46.00  sec  17.9 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  46.00-47.00  sec  4.54 MBytes  38.1 Mbits/sec
 [  8]  46.00-47.00  sec  6.29 MBytes  52.8 Mbits/sec
 [ 10]  46.00-47.00  sec  3.13 MBytes  26.3 Mbits/sec
 [ 12]  46.00-47.00  sec  3.92 MBytes  32.9 Mbits/sec
 [SUM]  46.00-47.00  sec  17.9 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  47.00-48.00  sec  4.44 MBytes  37.2 Mbits/sec
 [  8]  47.00-48.00  sec  5.36 MBytes  44.9 Mbits/sec
 [ 10]  47.00-48.00  sec  3.82 MBytes  32.0 Mbits/sec
 [ 12]  47.00-48.00  sec  4.38 MBytes  36.7 Mbits/sec
 [SUM]  47.00-48.00  sec  18.0 MBytes   151 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  48.00-49.00  sec  3.76 MBytes  31.5 Mbits/sec
 [  8]  48.00-49.00  sec  5.16 MBytes  43.3 Mbits/sec
 [ 10]  48.00-49.00  sec  5.11 MBytes  42.9 Mbits/sec
 [ 12]  48.00-49.00  sec  3.67 MBytes  30.8 Mbits/sec
 [SUM]  48.00-49.00  sec  17.7 MBytes   148 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  49.00-50.00  sec  4.67 MBytes  39.2 Mbits/sec
 [  8]  49.00-50.00  sec  6.05 MBytes  50.8 Mbits/sec
 [ 10]  49.00-50.00  sec  3.97 MBytes  33.3 Mbits/sec
 [ 12]  49.00-50.00  sec  3.79 MBytes  31.8 Mbits/sec
 [SUM]  49.00-50.00  sec  18.5 MBytes   155 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  50.00-51.00  sec  6.25 MBytes  52.4 Mbits/sec
 [  8]  50.00-51.00  sec  4.69 MBytes  39.3 Mbits/sec
 [ 10]  50.00-51.00  sec  3.87 MBytes  32.5 Mbits/sec
 [ 12]  50.00-51.00  sec  3.15 MBytes  26.5 Mbits/sec
 [SUM]  50.00-51.00  sec  18.0 MBytes   151 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  51.00-52.00  sec  5.28 MBytes  44.3 Mbits/sec
 [  8]  51.00-52.00  sec  5.06 MBytes  42.5 Mbits/sec
 [ 10]  51.00-52.00  sec  4.20 MBytes  35.2 Mbits/sec
 [ 12]  51.00-52.00  sec  3.26 MBytes  27.4 Mbits/sec
 [SUM]  51.00-52.00  sec  17.8 MBytes   149 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  52.00-53.00  sec  4.76 MBytes  39.9 Mbits/sec
 [  8]  52.00-53.00  sec  3.73 MBytes  31.3 Mbits/sec
 [ 10]  52.00-53.00  sec  5.84 MBytes  49.0 Mbits/sec
 [ 12]  52.00-53.00  sec  3.31 MBytes  27.7 Mbits/sec
 [SUM]  52.00-53.00  sec  17.6 MBytes   148 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  53.00-54.00  sec  5.62 MBytes  47.2 Mbits/sec
 [  8]  53.00-54.00  sec  3.93 MBytes  33.0 Mbits/sec
 [ 10]  53.00-54.00  sec  5.56 MBytes  46.6 Mbits/sec
 [ 12]  53.00-54.00  sec  2.86 MBytes  24.0 Mbits/sec
 [SUM]  53.00-54.00  sec  18.0 MBytes   151 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  54.00-55.00  sec  5.16 MBytes  43.3 Mbits/sec
 [  8]  54.00-55.00  sec  3.94 MBytes  33.1 Mbits/sec
 [ 10]  54.00-55.00  sec  6.09 MBytes  51.1 Mbits/sec
 [ 12]  54.00-55.00  sec  2.84 MBytes  23.8 Mbits/sec
 [SUM]  54.00-55.00  sec  18.0 MBytes   151 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  55.00-56.00  sec  6.75 MBytes  56.6 Mbits/sec
 [  8]  55.00-56.00  sec  3.63 MBytes  30.5 Mbits/sec
 [ 10]  55.00-56.00  sec  5.20 MBytes  43.6 Mbits/sec
 [ 12]  55.00-56.00  sec  2.81 MBytes  23.6 Mbits/sec
 [SUM]  55.00-56.00  sec  18.4 MBytes   154 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  56.00-57.00  sec  5.30 MBytes  44.5 Mbits/sec
 [  8]  56.00-57.00  sec  4.14 MBytes  34.7 Mbits/sec
 [ 10]  56.00-57.00  sec  4.63 MBytes  38.8 Mbits/sec
 [ 12]  56.00-57.00  sec  3.74 MBytes  31.4 Mbits/sec
 [SUM]  56.00-57.00  sec  17.8 MBytes   149 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  57.00-58.00  sec  2.97 MBytes  24.9 Mbits/sec
 [  8]  57.00-58.00  sec  2.85 MBytes  23.9 Mbits/sec
 [ 10]  57.00-58.00  sec  6.13 MBytes  51.5 Mbits/sec
 [ 12]  57.00-58.00  sec  5.33 MBytes  44.7 Mbits/sec
 [SUM]  57.00-58.00  sec  17.3 MBytes   145 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  58.00-59.00  sec  2.90 MBytes  24.3 Mbits/sec
 [  8]  58.00-59.00  sec  2.87 MBytes  24.0 Mbits/sec
 [ 10]  58.00-59.00  sec  5.78 MBytes  48.5 Mbits/sec
 [ 12]  58.00-59.00  sec  5.66 MBytes  47.5 Mbits/sec
 [SUM]  58.00-59.00  sec  17.2 MBytes   144 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  59.00-60.00  sec  3.35 MBytes  28.1 Mbits/sec
 [  8]  59.00-60.00  sec  3.83 MBytes  32.2 Mbits/sec
 [ 10]  59.00-60.00  sec  4.48 MBytes  37.6 Mbits/sec
 [ 12]  59.00-60.00  sec  6.26 MBytes  52.5 Mbits/sec
 [SUM]  59.00-60.00  sec  17.9 MBytes   150 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [  5]  60.00-60.04  sec   152 KBytes  32.7 Mbits/sec
 [  8]  60.00-60.04  sec   150 KBytes  32.3 Mbits/sec
 [ 10]  60.00-60.04  sec   158 KBytes  34.0 Mbits/sec
 [ 12]  60.00-60.04  sec   267 KBytes  57.4 Mbits/sec
 [SUM]  60.00-60.04  sec   726 KBytes   156 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-60.04  sec   276 MBytes  38.6 Mbits/sec                  receiver
 [  8]   0.00-60.04  sec   274 MBytes  38.2 Mbits/sec                  receiver
 [ 10]   0.00-60.04  sec   262 MBytes  36.5 Mbits/sec                  receiver
 [ 12]   0.00-60.04  sec   262 MBytes  36.6 Mbits/sec                  receiver
 [SUM]   0.00-60.04  sec  1.05 GBytes   150 Mbits/sec                  receiver
 -----------------------------------------------------------

wesgeorge · January 18, 2020, 4:17am

remote side pt 2/2
Accepted connection from {ip6_prefix}:fff:fffc::2, port 45388
[ 5] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 50502
[ 6] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 50735
[ 9] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 49309
[ 11] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 51207
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 3.90 MBytes 32.7 Mbits/sec 0.422 ms 14402/18123 (79%)
[ 6] 0.00-1.00 sec 3.90 MBytes 32.7 Mbits/sec 0.398 ms 14404/18123 (79%)
[ 9] 0.00-1.00 sec 3.90 MBytes 32.7 Mbits/sec 0.387 ms 14405/18122 (79%)
[ 11] 0.00-1.00 sec 3.89 MBytes 32.6 Mbits/sec 0.387 ms 14413/18122 (80%)
[SUM] 0.00-1.00 sec 15.6 MBytes 131 Mbits/sec 0.398 ms 57624/72490 (79%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 4.03 MBytes 33.8 Mbits/sec 0.452 ms 15193/19038 (80%)
[ 6] 1.00-2.00 sec 4.03 MBytes 33.8 Mbits/sec 0.436 ms 15193/19038 (80%)
[ 9] 1.00-2.00 sec 4.04 MBytes 33.9 Mbits/sec 0.429 ms 15183/19038 (80%)
[ 11] 1.00-2.00 sec 4.03 MBytes 33.8 Mbits/sec 0.430 ms 15192/19038 (80%)
[SUM] 1.00-2.00 sec 16.1 MBytes 135 Mbits/sec 0.437 ms 60761/76152 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 4.07 MBytes 34.1 Mbits/sec 0.417 ms 15108/18990 (80%)
[ 6] 2.00-3.00 sec 4.08 MBytes 34.2 Mbits/sec 0.418 ms 15105/18990 (80%)
[ 9] 2.00-3.00 sec 4.07 MBytes 34.1 Mbits/sec 0.410 ms 15110/18991 (80%)
[ 11] 2.00-3.00 sec 4.07 MBytes 34.1 Mbits/sec 0.408 ms 15110/18990 (80%)
[SUM] 2.00-3.00 sec 16.3 MBytes 137 Mbits/sec 0.413 ms 60433/75961 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 4.04 MBytes 33.9 Mbits/sec 0.463 ms 15098/18946 (80%)
[ 6] 3.00-4.00 sec 4.04 MBytes 33.9 Mbits/sec 0.439 ms 15096/18946 (80%)
[ 9] 3.00-4.00 sec 4.04 MBytes 33.9 Mbits/sec 0.435 ms 15095/18946 (80%)
[ 11] 3.00-4.00 sec 4.04 MBytes 33.9 Mbits/sec 0.431 ms 15095/18947 (80%)
[SUM] 3.00-4.00 sec 16.2 MBytes 136 Mbits/sec 0.442 ms 60384/75785 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 4.01 MBytes 33.6 Mbits/sec 0.355 ms 15096/18920 (80%)
[ 6] 4.00-5.00 sec 4.01 MBytes 33.6 Mbits/sec 0.378 ms 15094/18919 (80%)
[ 9] 4.00-5.00 sec 4.01 MBytes 33.6 Mbits/sec 0.379 ms 15096/18919 (80%)
[ 11] 4.00-5.00 sec 4.01 MBytes 33.6 Mbits/sec 0.376 ms 15096/18919 (80%)
[SUM] 4.00-5.00 sec 16.0 MBytes 135 Mbits/sec 0.372 ms 60382/75677 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 4.05 MBytes 34.0 Mbits/sec 0.400 ms 15013/18878 (80%)
[ 6] 5.00-6.00 sec 4.05 MBytes 34.0 Mbits/sec 0.390 ms 15017/18879 (80%)
[ 9] 5.00-6.00 sec 4.06 MBytes 34.0 Mbits/sec 0.366 ms 15013/18879 (80%)
[ 11] 5.00-6.00 sec 4.06 MBytes 34.0 Mbits/sec 0.387 ms 15012/18878 (80%)
[SUM] 5.00-6.00 sec 16.2 MBytes 136 Mbits/sec 0.386 ms 60055/75514 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 4.07 MBytes 34.1 Mbits/sec 0.340 ms 15063/18938 (80%)
[ 6] 6.00-7.00 sec 4.06 MBytes 34.1 Mbits/sec 0.343 ms 15064/18938 (80%)
[ 9] 6.00-7.00 sec 4.07 MBytes 34.1 Mbits/sec 0.324 ms 15063/18938 (80%)
[ 11] 6.00-7.00 sec 4.07 MBytes 34.1 Mbits/sec 0.331 ms 15064/18939 (80%)
[SUM] 6.00-7.00 sec 16.3 MBytes 136 Mbits/sec 0.335 ms 60254/75753 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 4.00 MBytes 33.5 Mbits/sec 0.424 ms 15174/18987 (80%)
[ 6] 7.00-8.00 sec 4.00 MBytes 33.5 Mbits/sec 0.432 ms 15174/18987 (80%)
[ 9] 7.00-8.00 sec 4.00 MBytes 33.6 Mbits/sec 0.404 ms 15171/18987 (80%)
[ 11] 7.00-8.00 sec 4.00 MBytes 33.5 Mbits/sec 0.408 ms 15173/18986 (80%)
[SUM] 7.00-8.00 sec 16.0 MBytes 134 Mbits/sec 0.417 ms 60692/75947 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 4.06 MBytes 34.0 Mbits/sec 0.362 ms 15022/18889 (80%)
[ 6] 8.00-9.00 sec 4.06 MBytes 34.1 Mbits/sec 0.379 ms 15018/18888 (80%)
[ 9] 8.00-9.00 sec 4.06 MBytes 34.0 Mbits/sec 0.380 ms 15022/18888 (80%)
[ 11] 8.00-9.00 sec 4.06 MBytes 34.1 Mbits/sec 0.376 ms 15021/18889 (80%)
[SUM] 8.00-9.00 sec 16.2 MBytes 136 Mbits/sec 0.374 ms 60083/75554 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 4.04 MBytes 33.8 Mbits/sec 0.435 ms 15069/18916 (80%)
[ 6] 9.00-10.00 sec 4.04 MBytes 33.9 Mbits/sec 0.499 ms 15067/18916 (80%)
[ 9] 9.00-10.00 sec 4.04 MBytes 33.9 Mbits/sec 0.444 ms 15066/18916 (80%)
[ 11] 9.00-10.00 sec 4.04 MBytes 33.8 Mbits/sec 0.439 ms 15069/18916 (80%)
[SUM] 9.00-10.00 sec 16.1 MBytes 135 Mbits/sec 0.454 ms 60271/75664 (80%)
{snip for length}
[ 5] 14.00-15.00 sec 3.82 MBytes 32.0 Mbits/sec 0.316 ms 15271/18909 (81%)
[ 6] 14.00-15.00 sec 3.82 MBytes 32.0 Mbits/sec 0.316 ms 15272/18909 (81%)
[ 9] 14.00-15.00 sec 3.82 MBytes 32.0 Mbits/sec 0.351 ms 15269/18906 (81%)
[ 11] 14.00-15.00 sec 3.81 MBytes 32.0 Mbits/sec 0.363 ms 15205/18841 (81%)
[SUM] 14.00-15.00 sec 15.3 MBytes 128 Mbits/sec 0.337 ms 61017/75565 (81%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 15.00-16.00 sec 4.06 MBytes 34.1 Mbits/sec 0.279 ms 15095/18966 (80%)
[ 6] 15.00-16.00 sec 4.06 MBytes 34.1 Mbits/sec 0.275 ms 15094/18966 (80%)
[ 9] 15.00-16.00 sec 4.06 MBytes 34.1 Mbits/sec 0.318 ms 15097/18967 (80%)
[ 11] 15.00-16.00 sec 4.06 MBytes 34.0 Mbits/sec 0.338 ms 15099/18966 (80%)
[SUM] 15.00-16.00 sec 16.2 MBytes 136 Mbits/sec 0.302 ms 60385/75865 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 16.00-17.00 sec 4.02 MBytes 33.7 Mbits/sec 0.487 ms 15176/19012 (80%)
[ 6] 16.00-17.00 sec 4.02 MBytes 33.7 Mbits/sec 0.299 ms 15120/18955 (80%)
[ 9] 16.00-17.00 sec 4.03 MBytes 33.8 Mbits/sec 0.301 ms 15117/18957 (80%)
[ 11] 16.00-17.00 sec 4.02 MBytes 33.7 Mbits/sec 0.317 ms 15122/18958 (80%)
[SUM] 16.00-17.00 sec 16.1 MBytes 135 Mbits/sec 0.351 ms 60535/75882 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 17.00-18.00 sec 4.07 MBytes 34.1 Mbits/sec 0.483 ms 15042/18920 (80%)
[ 6] 17.00-18.00 sec 4.07 MBytes 34.2 Mbits/sec 0.310 ms 15039/18920 (79%)
[ 9] 17.00-18.00 sec 4.08 MBytes 34.2 Mbits/sec 0.476 ms 15092/18977 (80%)
[ 11] 17.00-18.00 sec 4.07 MBytes 34.2 Mbits/sec 0.456 ms 15094/18976 (80%)
[SUM] 17.00-18.00 sec 16.3 MBytes 137 Mbits/sec 0.431 ms 60267/75793 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 18.00-19.00 sec 4.03 MBytes 33.8 Mbits/sec 0.383 ms 14954/18798 (80%)
[ 6] 18.00-19.00 sec 4.03 MBytes 33.8 Mbits/sec 0.386 ms 15010/18854 (80%)
[ 9] 18.00-19.00 sec 4.03 MBytes 33.8 Mbits/sec 0.362 ms 14955/18797 (80%)
[ 11] 18.00-19.00 sec 4.03 MBytes 33.8 Mbits/sec 0.376 ms 14959/18798 (80%)
[SUM] 18.00-19.00 sec 16.1 MBytes 135 Mbits/sec 0.377 ms 59878/75247 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 19.00-20.00 sec 3.97 MBytes 33.3 Mbits/sec 0.470 ms 15169/18949 (80%)
[ 6] 19.00-20.00 sec 3.97 MBytes 33.3 Mbits/sec 0.508 ms 15163/18950 (80%)
[ 9] 19.00-20.00 sec 3.97 MBytes 33.3 Mbits/sec 0.339 ms 15107/18893 (80%)
[ 11] 19.00-20.00 sec 3.97 MBytes 33.3 Mbits/sec 0.468 ms 15165/18949 (80%)
[SUM] 19.00-20.00 sec 15.9 MBytes 133 Mbits/sec 0.446 ms 60604/75741 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 20.00-21.00 sec 4.04 MBytes 33.9 Mbits/sec 0.367 ms 15057/18907 (80%)
[ 6] 20.00-21.00 sec 4.04 MBytes 33.9 Mbits/sec 0.357 ms 15057/18907 (80%)
[ 9] 20.00-21.00 sec 4.04 MBytes 33.9 Mbits/sec 0.352 ms 15114/18964 (80%)
[ 11] 20.00-21.00 sec 4.04 MBytes 33.9 Mbits/sec 0.329 ms 15057/18908 (80%)
[SUM] 20.00-21.00 sec 16.2 MBytes 136 Mbits/sec 0.351 ms 60285/75686 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 21.00-22.00 sec 4.03 MBytes 33.8 Mbits/sec 0.374 ms 15099/18945 (80%)
[ 6] 21.00-22.00 sec 4.04 MBytes 33.8 Mbits/sec 0.364 ms 15098/18945 (80%)
[ 9] 21.00-22.00 sec 4.04 MBytes 33.9 Mbits/sec 0.366 ms 15096/18945 (80%)
[ 11] 21.00-22.00 sec 4.03 MBytes 33.8 Mbits/sec 0.362 ms 15100/18944 (80%)
[SUM] 21.00-22.00 sec 16.1 MBytes 135 Mbits/sec 0.366 ms 60393/75779 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 22.00-23.00 sec 3.95 MBytes 33.1 Mbits/sec 0.343 ms 15103/18870 (80%)
[ 6] 22.00-23.00 sec 3.95 MBytes 33.2 Mbits/sec 0.338 ms 15101/18870 (80%)
[ 9] 22.00-23.00 sec 3.95 MBytes 33.1 Mbits/sec 0.329 ms 15105/18870 (80%)
[ 11] 22.00-23.00 sec 3.95 MBytes 33.1 Mbits/sec 0.329 ms 15106/18870 (80%)
[SUM] 22.00-23.00 sec 15.8 MBytes 133 Mbits/sec 0.335 ms 60415/75480 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 23.00-24.00 sec 3.99 MBytes 33.5 Mbits/sec 0.358 ms 15064/18871 (80%)
[ 6] 23.00-24.00 sec 3.99 MBytes 33.5 Mbits/sec 0.335 ms 15069/18871 (80%)
[ 9] 23.00-24.00 sec 3.99 MBytes 33.5 Mbits/sec 0.314 ms 15066/18871 (80%)
[ 11] 23.00-24.00 sec 3.99 MBytes 33.5 Mbits/sec 0.363 ms 15066/18871 (80%)
[SUM] 23.00-24.00 sec 16.0 MBytes 134 Mbits/sec 0.342 ms 60265/75484 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 24.00-25.00 sec 4.01 MBytes 33.6 Mbits/sec 0.354 ms 15133/18952 (80%)
[ 6] 24.00-25.00 sec 4.01 MBytes 33.6 Mbits/sec 0.350 ms 15131/18952 (80%)
[ 9] 24.00-25.00 sec 4.01 MBytes 33.6 Mbits/sec 0.333 ms 15128/18952 (80%)
[ 11] 24.00-25.00 sec 4.01 MBytes 33.6 Mbits/sec 0.344 ms 15133/18952 (80%)
[SUM] 24.00-25.00 sec 16.0 MBytes 134 Mbits/sec 0.345 ms 60525/75808 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 25.00-26.00 sec 4.05 MBytes 34.0 Mbits/sec 0.505 ms 15163/19020 (80%)
[ 6] 25.00-26.00 sec 4.05 MBytes 33.9 Mbits/sec 0.363 ms 15106/18962 (80%)
[ 9] 25.00-26.00 sec 4.04 MBytes 33.9 Mbits/sec 0.504 ms 15167/19019 (80%)
[ 11] 25.00-26.00 sec 4.05 MBytes 34.0 Mbits/sec 0.492 ms 15163/19020 (80%)
[SUM] 25.00-26.00 sec 16.2 MBytes 136 Mbits/sec 0.466 ms 60599/76021 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 26.00-27.00 sec 3.99 MBytes 33.4 Mbits/sec 0.324 ms 15060/18859 (80%)
[ 6] 26.00-27.00 sec 3.99 MBytes 33.4 Mbits/sec 0.499 ms 15115/18914 (80%)
[ 9] 26.00-27.00 sec 3.98 MBytes 33.4 Mbits/sec 0.444 ms 15060/18858 (80%)
[ 11] 26.00-27.00 sec 3.98 MBytes 33.4 Mbits/sec 0.423 ms 15060/18858 (80%)
[SUM] 26.00-27.00 sec 15.9 MBytes 134 Mbits/sec 0.422 ms 60295/75489 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 27.00-28.00 sec 3.98 MBytes 33.4 Mbits/sec 0.381 ms 15139/18933 (80%)
[ 6] 27.00-28.00 sec 3.99 MBytes 33.4 Mbits/sec 0.378 ms 15136/18935 (80%)
[ 9] 27.00-28.00 sec 3.98 MBytes 33.4 Mbits/sec 0.390 ms 15135/18933 (80%)
[ 11] 27.00-28.00 sec 3.98 MBytes 33.4 Mbits/sec 0.387 ms 15138/18933 (80%)
[SUM] 27.00-28.00 sec 15.9 MBytes 134 Mbits/sec 0.384 ms 60548/75734 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 28.00-29.00 sec 4.00 MBytes 33.6 Mbits/sec 0.454 ms 15136/18949 (80%)
[ 6] 28.00-29.00 sec 4.00 MBytes 33.6 Mbits/sec 0.449 ms 15133/18949 (80%)
[ 9] 28.00-29.00 sec 4.00 MBytes 33.5 Mbits/sec 0.458 ms 15137/18949 (80%)
[ 11] 28.00-29.00 sec 4.00 MBytes 33.5 Mbits/sec 0.307 ms 15085/18894 (80%)
[SUM] 28.00-29.00 sec 16.0 MBytes 134 Mbits/sec 0.417 ms 60491/75741 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 29.00-30.00 sec 4.03 MBytes 33.8 Mbits/sec 0.419 ms 15080/18921 (80%)
[ 6] 29.00-30.00 sec 4.03 MBytes 33.8 Mbits/sec 0.418 ms 15080/18921 (80%)
[ 9] 29.00-30.00 sec 4.03 MBytes 33.8 Mbits/sec 0.409 ms 15079/18923 (80%)
[ 11] 29.00-30.00 sec 4.03 MBytes 33.8 Mbits/sec 0.410 ms 15137/18977 (80%)
[SUM] 29.00-30.00 sec 16.1 MBytes 135 Mbits/sec 0.414 ms 60376/75742 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 30.00-31.00 sec 4.05 MBytes 34.0 Mbits/sec 0.503 ms 15035/18899 (80%)
[ 6] 30.00-31.00 sec 4.05 MBytes 33.9 Mbits/sec 0.506 ms 15043/18900 (80%)
[ 9] 30.00-31.00 sec 4.05 MBytes 34.0 Mbits/sec 0.477 ms 15038/18899 (80%)
[ 11] 30.00-31.00 sec 4.05 MBytes 34.0 Mbits/sec 0.464 ms 15036/18900 (80%)
[SUM] 30.00-31.00 sec 16.2 MBytes 136 Mbits/sec 0.488 ms 60152/75598 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 31.00-32.00 sec 4.03 MBytes 33.8 Mbits/sec 0.358 ms 15096/18938 (80%)
[ 6] 31.00-32.00 sec 4.03 MBytes 33.8 Mbits/sec 0.334 ms 15093/18938 (80%)
[ 9] 31.00-32.00 sec 4.03 MBytes 33.8 Mbits/sec 0.317 ms 15094/18938 (80%)
[ 11] 31.00-32.00 sec 4.03 MBytes 33.8 Mbits/sec 0.318 ms 15092/18938 (80%)
[SUM] 31.00-32.00 sec 16.1 MBytes 135 Mbits/sec 0.332 ms 60375/75752 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 32.00-33.00 sec 4.06 MBytes 34.1 Mbits/sec 0.339 ms 15089/18961 (80%)
[ 6] 32.00-33.00 sec 4.06 MBytes 34.1 Mbits/sec 0.358 ms 15089/18960 (80%)
[ 9] 32.00-33.00 sec 4.06 MBytes 34.0 Mbits/sec 0.335 ms 15091/18959 (80%)
[ 11] 32.00-33.00 sec 4.06 MBytes 34.0 Mbits/sec 0.371 ms 15090/18958 (80%)
[SUM] 32.00-33.00 sec 16.2 MBytes 136 Mbits/sec 0.351 ms 60359/75838 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 33.00-34.00 sec 4.05 MBytes 33.9 Mbits/sec 0.369 ms 15075/18933 (80%)
[ 6] 33.00-34.00 sec 4.05 MBytes 34.0 Mbits/sec 0.385 ms 15074/18934 (80%)
[ 9] 33.00-34.00 sec 4.05 MBytes 34.0 Mbits/sec 0.346 ms 15076/18935 (80%)
[ 11] 33.00-34.00 sec 4.05 MBytes 34.0 Mbits/sec 0.336 ms 15077/18936 (80%)
[SUM] 33.00-34.00 sec 16.2 MBytes 136 Mbits/sec 0.359 ms 60302/75738 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 34.00-35.00 sec 4.00 MBytes 33.6 Mbits/sec 0.451 ms 15134/18950 (80%)
[ 6] 34.00-35.00 sec 4.01 MBytes 33.6 Mbits/sec 0.458 ms 15132/18950 (80%)
[ 9] 34.00-35.00 sec 4.00 MBytes 33.6 Mbits/sec 0.457 ms 15134/18950 (80%)
[ 11] 34.00-35.00 sec 4.01 MBytes 33.6 Mbits/sec 0.421 ms 15132/18950 (80%)
[SUM] 34.00-35.00 sec 16.0 MBytes 134 Mbits/sec 0.447 ms 60532/75800 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 35.00-36.00 sec 4.06 MBytes 34.0 Mbits/sec 0.487 ms 15076/18943 (80%)
[ 6] 35.00-36.00 sec 4.05 MBytes 34.0 Mbits/sec 0.446 ms 15080/18943 (80%)
[ 9] 35.00-36.00 sec 4.06 MBytes 34.0 Mbits/sec 0.438 ms 15076/18943 (80%)
[ 11] 35.00-36.00 sec 4.05 MBytes 34.0 Mbits/sec 0.443 ms 15081/18943 (80%)
[SUM] 35.00-36.00 sec 16.2 MBytes 136 Mbits/sec 0.453 ms 60313/75772 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 36.00-37.00 sec 4.08 MBytes 34.2 Mbits/sec 0.421 ms 14998/18883 (79%)
[ 6] 36.00-37.00 sec 4.07 MBytes 34.2 Mbits/sec 0.421 ms 15002/18883 (79%)
[ 9] 36.00-37.00 sec 4.07 MBytes 34.2 Mbits/sec 0.423 ms 15000/18883 (79%)
[ 11] 36.00-37.00 sec 4.07 MBytes 34.1 Mbits/sec 0.433 ms 15003/18882 (79%)
[SUM] 36.00-37.00 sec 16.3 MBytes 137 Mbits/sec 0.424 ms 60003/75531 (79%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 37.00-38.00 sec 3.99 MBytes 33.4 Mbits/sec 0.333 ms 15070/18870 (80%)
[ 6] 37.00-38.00 sec 3.99 MBytes 33.4 Mbits/sec 0.281 ms 15069/18870 (80%)
[ 9] 37.00-38.00 sec 3.99 MBytes 33.4 Mbits/sec 0.310 ms 15069/18870 (80%)
[ 11] 37.00-38.00 sec 3.99 MBytes 33.5 Mbits/sec 0.322 ms 15067/18870 (80%)
[SUM] 37.00-38.00 sec 16.0 MBytes 134 Mbits/sec 0.311 ms 60275/75480 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 38.00-39.00 sec 4.06 MBytes 34.0 Mbits/sec 0.315 ms 14994/18863 (79%)
[ 6] 38.00-39.00 sec 4.06 MBytes 34.0 Mbits/sec 0.324 ms 14996/18862 (80%)
[ 9] 38.00-39.00 sec 4.06 MBytes 34.0 Mbits/sec 0.327 ms 14993/18862 (79%)
[ 11] 38.00-39.00 sec 4.06 MBytes 34.0 Mbits/sec 0.321 ms 14994/18863 (79%)
[SUM] 38.00-39.00 sec 16.2 MBytes 136 Mbits/sec 0.321 ms 59977/75450 (79%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 39.00-40.00 sec 4.03 MBytes 33.8 Mbits/sec 0.416 ms 15123/18966 (80%)
[ 6] 39.00-40.00 sec 4.03 MBytes 33.8 Mbits/sec 0.395 ms 15125/18967 (80%)
[ 9] 39.00-40.00 sec 4.03 MBytes 33.8 Mbits/sec 0.382 ms 15125/18967 (80%)
[ 11] 39.00-40.00 sec 4.03 MBytes 33.8 Mbits/sec 0.447 ms 15124/18966 (80%)
[SUM] 39.00-40.00 sec 16.1 MBytes 135 Mbits/sec 0.410 ms 60497/75866 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 40.00-41.00 sec 4.08 MBytes 34.2 Mbits/sec 0.321 ms 14958/18846 (79%)
[ 6] 40.00-41.00 sec 4.08 MBytes 34.2 Mbits/sec 0.328 ms 14955/18846 (79%)
[ 9] 40.00-41.00 sec 4.08 MBytes 34.2 Mbits/sec 0.287 ms 14954/18846 (79%)
[ 11] 40.00-41.00 sec 4.08 MBytes 34.2 Mbits/sec 0.288 ms 14956/18847 (79%)
[SUM] 40.00-41.00 sec 16.3 MBytes 137 Mbits/sec 0.306 ms 59823/75385 (79%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 41.00-42.00 sec 4.08 MBytes 34.2 Mbits/sec 0.531 ms 15125/19011 (80%)
[ 6] 41.00-42.00 sec 4.08 MBytes 34.2 Mbits/sec 0.524 ms 15124/19012 (80%)
[ 9] 41.00-42.00 sec 4.07 MBytes 34.2 Mbits/sec 0.485 ms 15128/19011 (80%)
[ 11] 41.00-42.00 sec 4.08 MBytes 34.2 Mbits/sec 0.466 ms 15126/19011 (80%)
[SUM] 41.00-42.00 sec 16.3 MBytes 137 Mbits/sec 0.501 ms 60503/76045 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 42.00-43.00 sec 4.07 MBytes 34.1 Mbits/sec 0.359 ms 15047/18926 (80%)
[ 6] 42.00-43.00 sec 4.07 MBytes 34.1 Mbits/sec 0.332 ms 15047/18925 (80%)
[ 9] 42.00-43.00 sec 4.07 MBytes 34.1 Mbits/sec 0.311 ms 15050/18926 (80%)
[ 11] 42.00-43.00 sec 4.06 MBytes 34.1 Mbits/sec 0.317 ms 15052/18926 (80%)
[SUM] 42.00-43.00 sec 16.3 MBytes 136 Mbits/sec 0.330 ms 60196/75703 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 43.00-44.00 sec 3.91 MBytes 32.8 Mbits/sec 0.514 ms 15188/18912 (80%)
[ 6] 43.00-44.00 sec 3.90 MBytes 32.7 Mbits/sec 0.483 ms 15193/18912 (80%)
[ 9] 43.00-44.00 sec 3.90 MBytes 32.7 Mbits/sec 0.476 ms 15189/18911 (80%)
[ 11] 43.00-44.00 sec 3.90 MBytes 32.7 Mbits/sec 0.478 ms 15190/18911 (80%)
[SUM] 43.00-44.00 sec 15.6 MBytes 131 Mbits/sec 0.488 ms 60760/75646 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 44.00-45.00 sec 4.02 MBytes 33.7 Mbits/sec 0.401 ms 15008/18837 (80%)
[ 6] 44.00-45.00 sec 4.01 MBytes 33.7 Mbits/sec 0.384 ms 15011/18836 (80%)
[ 9] 44.00-45.00 sec 4.01 MBytes 33.7 Mbits/sec 0.369 ms 15012/18837 (80%)
[ 11] 44.00-45.00 sec 4.01 MBytes 33.7 Mbits/sec 0.337 ms 15011/18837 (80%)
[SUM] 44.00-45.00 sec 16.1 MBytes 135 Mbits/sec 0.373 ms 60042/75347 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 45.00-46.00 sec 4.00 MBytes 33.6 Mbits/sec 0.312 ms 15001/18814 (80%)
[ 6] 45.00-46.00 sec 4.00 MBytes 33.6 Mbits/sec 0.332 ms 14998/18814 (80%)
[ 9] 45.00-46.00 sec 4.00 MBytes 33.5 Mbits/sec 0.320 ms 15002/18814 (80%)
[ 11] 45.00-46.00 sec 4.00 MBytes 33.5 Mbits/sec 0.318 ms 15002/18814 (80%)
[SUM] 45.00-46.00 sec 16.0 MBytes 134 Mbits/sec 0.321 ms 60003/75256 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 46.00-47.00 sec 4.07 MBytes 34.1 Mbits/sec 0.269 ms 15065/18941 (80%)
[ 6] 46.00-47.00 sec 4.07 MBytes 34.2 Mbits/sec 0.334 ms 15058/18942 (79%)
[ 9] 46.00-47.00 sec 4.07 MBytes 34.2 Mbits/sec 0.259 ms 15060/18942 (80%)
[ 11] 46.00-47.00 sec 4.07 MBytes 34.2 Mbits/sec 0.252 ms 15059/18942 (80%)
[SUM] 46.00-47.00 sec 16.3 MBytes 137 Mbits/sec 0.278 ms 60242/75767 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 47.00-48.00 sec 4.07 MBytes 34.2 Mbits/sec 0.391 ms 15068/18949 (80%)
[ 6] 47.00-48.00 sec 4.07 MBytes 34.2 Mbits/sec 0.398 ms 15069/18949 (80%)
[ 9] 47.00-48.00 sec 4.06 MBytes 34.1 Mbits/sec 0.397 ms 15076/18949 (80%)
[ 11] 47.00-48.00 sec 4.07 MBytes 34.2 Mbits/sec 0.337 ms 15067/18949 (80%)
[SUM] 47.00-48.00 sec 16.3 MBytes 137 Mbits/sec 0.381 ms 60280/75796 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 48.00-49.00 sec 4.00 MBytes 33.6 Mbits/sec 0.373 ms 15043/18860 (80%)
[ 6] 48.00-49.00 sec 4.00 MBytes 33.6 Mbits/sec 0.384 ms 15044/18860 (80%)
[ 9] 48.00-49.00 sec 4.00 MBytes 33.6 Mbits/sec 0.380 ms 15045/18860 (80%)
[ 11] 48.00-49.00 sec 4.00 MBytes 33.5 Mbits/sec 0.387 ms 15050/18860 (80%)
[SUM] 48.00-49.00 sec 16.0 MBytes 134 Mbits/sec 0.381 ms 60182/75440 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 49.00-50.00 sec 4.05 MBytes 33.9 Mbits/sec 0.490 ms 15037/18895 (80%)
[ 6] 49.00-50.00 sec 4.05 MBytes 33.9 Mbits/sec 0.447 ms 15036/18894 (80%)
[ 9] 49.00-50.00 sec 4.05 MBytes 34.0 Mbits/sec 0.430 ms 15031/18894 (80%)
[ 11] 49.00-50.00 sec 4.05 MBytes 34.0 Mbits/sec 0.429 ms 15034/18894 (80%)
[SUM] 49.00-50.00 sec 16.2 MBytes 136 Mbits/sec 0.449 ms 60138/75577 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 50.00-51.00 sec 3.99 MBytes 33.4 Mbits/sec 0.359 ms 15165/18967 (80%)
[ 6] 50.00-51.00 sec 3.98 MBytes 33.4 Mbits/sec 0.356 ms 15173/18968 (80%)
[ 9] 50.00-51.00 sec 3.98 MBytes 33.4 Mbits/sec 0.337 ms 15170/18967 (80%)
[ 11] 50.00-51.00 sec 3.98 MBytes 33.4 Mbits/sec 0.338 ms 15169/18967 (80%)
[SUM] 50.00-51.00 sec 15.9 MBytes 134 Mbits/sec 0.348 ms 60677/75869 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 51.00-52.00 sec 3.97 MBytes 33.3 Mbits/sec 0.457 ms 15097/18880 (80%)
[ 6] 51.00-52.00 sec 3.97 MBytes 33.3 Mbits/sec 0.461 ms 15097/18880 (80%)
[ 9] 51.00-52.00 sec 3.96 MBytes 33.3 Mbits/sec 0.458 ms 15102/18880 (80%)
[ 11] 51.00-52.00 sec 3.97 MBytes 33.3 Mbits/sec 0.453 ms 15100/18880 (80%)
[SUM] 51.00-52.00 sec 15.9 MBytes 133 Mbits/sec 0.457 ms 60396/75520 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 52.00-53.00 sec 3.94 MBytes 33.0 Mbits/sec 0.355 ms 15093/18847 (80%)
[ 6] 52.00-53.00 sec 3.93 MBytes 33.0 Mbits/sec 0.367 ms 15098/18846 (80%)
[ 9] 52.00-53.00 sec 3.94 MBytes 33.0 Mbits/sec 0.333 ms 15093/18847 (80%)
[ 11] 52.00-53.00 sec 3.93 MBytes 33.0 Mbits/sec 0.340 ms 15097/18847 (80%)
[SUM] 52.00-53.00 sec 15.7 MBytes 132 Mbits/sec 0.349 ms 60381/75387 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 53.00-54.00 sec 4.02 MBytes 33.7 Mbits/sec 0.340 ms 14981/18810 (80%)
[ 6] 53.00-54.00 sec 4.02 MBytes 33.7 Mbits/sec 0.325 ms 14982/18811 (80%)
[ 9] 53.00-54.00 sec 4.02 MBytes 33.7 Mbits/sec 0.321 ms 14981/18811 (80%)
[ 11] 53.00-54.00 sec 4.02 MBytes 33.7 Mbits/sec 0.357 ms 14982/18810 (80%)
[SUM] 53.00-54.00 sec 16.1 MBytes 135 Mbits/sec 0.336 ms 59926/75242 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 54.00-55.00 sec 3.89 MBytes 32.6 Mbits/sec 0.361 ms 15175/18883 (80%)
[ 6] 54.00-55.00 sec 3.89 MBytes 32.7 Mbits/sec 0.360 ms 15172/18883 (80%)
[ 9] 54.00-55.00 sec 3.89 MBytes 32.7 Mbits/sec 0.353 ms 15173/18883 (80%)
[ 11] 54.00-55.00 sec 3.89 MBytes 32.7 Mbits/sec 0.364 ms 15172/18883 (80%)
[SUM] 54.00-55.00 sec 15.6 MBytes 131 Mbits/sec 0.360 ms 60692/75532 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 55.00-56.00 sec 3.99 MBytes 33.4 Mbits/sec 0.440 ms 15124/18923 (80%)
[ 6] 55.00-56.00 sec 3.99 MBytes 33.5 Mbits/sec 0.431 ms 15123/18924 (80%)
[ 9] 55.00-56.00 sec 3.99 MBytes 33.5 Mbits/sec 0.432 ms 15122/18923 (80%)
[ 11] 55.00-56.00 sec 3.99 MBytes 33.4 Mbits/sec 0.441 ms 15124/18924 (80%)
[SUM] 55.00-56.00 sec 15.9 MBytes 134 Mbits/sec 0.436 ms 60493/75694 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 56.00-57.00 sec 4.02 MBytes 33.7 Mbits/sec 0.334 ms 15757/19588 (80%)
[ 6] 56.00-57.00 sec 4.01 MBytes 33.6 Mbits/sec 0.289 ms 15762/19587 (80%)
[ 9] 56.00-57.00 sec 4.02 MBytes 33.7 Mbits/sec 0.297 ms 15755/19588 (80%)
[ 11] 56.00-57.00 sec 4.02 MBytes 33.7 Mbits/sec 0.333 ms 15753/19587 (80%)
[SUM] 56.00-57.00 sec 16.1 MBytes 135 Mbits/sec 0.313 ms 63027/78350 (80%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 57.00-58.00 sec 4.01 MBytes 33.7 Mbits/sec 0.511 ms 17497/21324 (82%)
[ 6] 57.00-58.00 sec 4.01 MBytes 33.6 Mbits/sec 0.333 ms 17437/21259 (82%)
[ 9] 57.00-58.00 sec 4.01 MBytes 33.6 Mbits/sec 0.480 ms 17500/21323 (82%)
[ 11] 57.00-58.00 sec 4.01 MBytes 33.7 Mbits/sec 0.482 ms 17497/21324 (82%)
[SUM] 57.00-58.00 sec 16.0 MBytes 135 Mbits/sec 0.452 ms 69931/85230 (82%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 58.00-59.00 sec 4.09 MBytes 34.3 Mbits/sec 0.482 ms 17341/21238 (82%)
[ 6] 58.00-59.00 sec 4.09 MBytes 34.3 Mbits/sec 0.476 ms 17405/21303 (82%)
[ 9] 58.00-59.00 sec 4.09 MBytes 34.3 Mbits/sec 0.444 ms 17341/21239 (82%)
[ 11] 58.00-59.00 sec 4.09 MBytes 34.3 Mbits/sec 0.450 ms 17339/21238 (82%)
[SUM] 58.00-59.00 sec 16.4 MBytes 137 Mbits/sec 0.463 ms 69426/85018 (82%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 59.00-60.00 sec 4.01 MBytes 33.6 Mbits/sec 0.492 ms 17394/21214 (82%)
[ 6] 59.00-60.00 sec 4.01 MBytes 33.6 Mbits/sec 0.484 ms 17394/21214 (82%)
[ 9] 59.00-60.00 sec 4.01 MBytes 33.6 Mbits/sec 0.459 ms 17392/21213 (82%)
[ 11] 59.00-60.00 sec 4.01 MBytes 33.6 Mbits/sec 0.477 ms 17393/21214 (82%)
[SUM] 59.00-60.00 sec 16.0 MBytes 135 Mbits/sec 0.478 ms 69573/84855 (82%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 60.00-60.24 sec 175 KBytes 6.06 Mbits/sec 0.306 ms 680/843 (81%)
[ 6] 60.00-60.24 sec 175 KBytes 6.06 Mbits/sec 0.318 ms 680/843 (81%)
[ 9] 60.00-60.24 sec 176 KBytes 6.09 Mbits/sec 0.361 ms 680/844 (81%)
[ 11] 60.00-60.24 sec 176 KBytes 6.09 Mbits/sec 0.368 ms 680/844 (81%)
[SUM] 60.00-60.24 sec 703 KBytes 24.3 Mbits/sec 0.338 ms 2720/3374 (81%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.24 sec 241 MBytes 33.5 Mbits/sec 0.306 ms 913214/1142746 (80%) receiver
[ 6] 0.00-60.24 sec 241 MBytes 33.5 Mbits/sec 0.318 ms 913213/1142746 (80%) receiver
[ 9] 0.00-60.24 sec 241 MBytes 33.5 Mbits/sec 0.361 ms 913199/1142746 (80%) receiver
[ 11] 0.00-60.24 sec 241 MBytes 33.5 Mbits/sec 0.368 ms 913252/1142746 (80%) receiver
[SUM] 0.00-60.24 sec 963 MBytes 134 Mbits/sec 0.338 ms 3652878/4570984 (80%) receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from {ip6_prefix}:fff:fffc::2, port 45390
[ 5] local {ip6_prefix}:2:1:0:214:66:1 port 5201 connected to {ip6_prefix}:fff:fffc::2 port 42629
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 14.9 MBytes 125 Mbits/sec 0.062 ms 46353/60598 (76%)
{snip for length}
[ 5] 42.00-43.00 sec 16.2 MBytes 136 Mbits/sec 0.068 ms 49215/64615 (76%)
[ 5] 43.00-44.00 sec 16.2 MBytes 136 Mbits/sec 0.070 ms 49552/65042 (76%)
[ 5] 44.00-45.00 sec 16.1 MBytes 135 Mbits/sec 0.125 ms 49433/64745 (76%)
[ 5] 45.00-46.00 sec 15.8 MBytes 133 Mbits/sec 0.143 ms 49838/64906 (77%)
[ 5] 46.00-47.00 sec 16.3 MBytes 136 Mbits/sec 0.152 ms 49233/64739 (76%)
[ 5] 47.00-48.00 sec 16.3 MBytes 137 Mbits/sec 0.079 ms 49067/64587 (76%)
[ 5] 48.00-49.00 sec 16.1 MBytes 135 Mbits/sec 0.059 ms 49503/64833 (76%)
[ 5] 49.00-50.00 sec 16.1 MBytes 135 Mbits/sec 0.055 ms 49557/64884 (76%)
[ 5] 50.00-51.00 sec 16.3 MBytes 136 Mbits/sec 0.054 ms 49227/64736 (76%)
[ 5] 51.00-52.00 sec 15.8 MBytes 132 Mbits/sec 0.073 ms 49535/64567 (77%)
[ 5] 52.00-53.00 sec 16.2 MBytes 136 Mbits/sec 0.063 ms 49598/65020 (76%)
[ 5] 53.00-54.00 sec 16.0 MBytes 134 Mbits/sec 0.183 ms 49485/64761 (76%)
[ 5] 54.00-55.00 sec 15.9 MBytes 134 Mbits/sec 0.075 ms 49385/64580 (76%)
[ 5] 55.00-56.00 sec 16.3 MBytes 136 Mbits/sec 0.207 ms 49690/65181 (76%)
[ 5] 56.00-57.00 sec 16.2 MBytes 136 Mbits/sec 0.071 ms 49334/64739 (76%)
[ 5] 57.00-58.00 sec 16.3 MBytes 137 Mbits/sec 0.062 ms 49010/64535 (76%)
[ 5] 58.00-59.00 sec 16.1 MBytes 135 Mbits/sec 0.055 ms 49670/65055 (76%)
[ 5] 59.00-60.00 sec 15.9 MBytes 133 Mbits/sec 0.076 ms 49712/64832 (77%)
[ 5] 60.00-60.06 sec 1024 KBytes 137 Mbits/sec 0.091 ms 3045/3998 (76%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.06 sec 964 MBytes 135 Mbits/sec 0.091 ms 2967886/3886753 (76%) receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

I also see the same behavior if I switch client and server directions.

wesgeorge · January 31, 2020, 2:02am

bump. issue persists.

wesgeorge · February 20, 2020, 10:24pm

I’ve upgraded to the latest rolling image and the same issue persists. Is there any optimizations I should be doing to improve GRE throughput?

Breizh · March 29, 2021, 8:30am

Hi,

I know this is a quite old thread but…
With a similar use case, I’m also facing a ~1 Gbps GRE limit on latest VyOS 1.4 rolling (outside the tunnel, I can get up to ~ 4 - 6 Gbps). The issue does not come from CPU (just a few percents usage despite NAT and firewall).

Did you find a way to improve this ?

I have been told this could come from GRE implementation module. Changing it for another one could help, but is this possible on VyOS ?

Thank you

Viacheslav · March 30, 2021, 3:47am

Which device on the remote site?
Which version do you use?

Breizh · March 30, 2021, 8:14am

Hi,
Thank you for your reply.
AFAIK, this is a Microtik router.
I use VyOS 1.4-rolling-202103251004

My use case is to redirect the whole internet traffic from some machines of my LAN to this tunnel. There is an associative ISP beyond the tunnel, which provides me with better peering than my usual ISP.

I know from a person in this remote organization that with a Microtik router at home, with the same regular ISP at home, he can achieve around 3.5 Gbps with a similar tunnel.

That’s why we thought the issue could come from ip_gre implementation ? or interoperability with Microtik gre stack ? But there may be other reasons… and hopefully optimizations to make ?

rosato · April 27, 2024, 4:22pm

Interesting thread. Any updates? I’m on Vyos 1.4 and also have this kind of “asymmetric” bandwidth with GRE