Internet speed capped, but only in download

General questions
, , , , , ,
1

Hello!

I have two VyOS routers at two different locations, but the network setup is almost identical.
The OS versions are currently 2025.03.15-0018-rolling and 2025.04.29-0019-rolling.

iperf reports a steady 9.90 Gb/s throughput in both directions between the router and devices that are connected through fiber and 10 Gb/s NICs + transceivers.

The Internet connection is provided through the same NIC that is used for the LAN, but on a separate port.
Contract is 2500/1000 (D/U) at one location and 2500/500 at the other. The actual maximum speed that is achievable right now is ~900/~900 due to the provider temporarily relying on a radio bridge in our area (allegedly).

I can reach the limit just fine when performing tests on remote servers directly on VyOS, but the exact same route (verified through traceroute) results in significantly lower download speeds on the same devices that can push 9.90 Gb/s just fine.

I have a WireGuard VPN on the router that can be accessed through a VLAN on the LAN bridge for routed Internet traffic. With that I can saturate the download bandwidth just fine on the same devices.

I immediately thought about a possible MTU issue, which is currently 1500 for the WAN interface and 1420 for the VPN one, but setting the following options has no effect whatsoever:

set interfaces ethernet eth6 ip adjust-mss clamp-mss-to-pmtu
set interfaces ethernet eth6 ipv6 adjust-mss clamp-mss-to-pmtu

The only difference that comes to mind is the presence of masquerade NAT for devices other than the router itself:

 nat {
     source {
         rule 835 {
             outbound-interface {
                 name eth6.835
             }
             translation {
                 address masquerade
             }
         }
    }
}

But I don’t see why or how it would affect the network traffic and only in the download direction.

Please let me know if you need hardware info.

Thanks in advance!

2

I’ve read this a few times, but I can’t “draw” a diagram in my head of what you mean.

This, especially:

Are you saying if you do some sort of test via CLI on VyOS that it’s very performant, but then you move one hop behind it and it’s much slower? Or something else?

And then you mention wireguard and a LAN Bridge?

Without understanding where the issue actually is, I would suggest ensuring you’ve turned off offloads, and then if they’re already off, try turning them on.

It’s unlikely to be NAT/conntrack causing issues, but you could try enabling software flowtables and see if that helps. Again without understanding what/where Wireguard fits I’m not sure if that’s a good suggestion or not.

And bridges on VyOS/Linux in general tend to come with performance impacts.

1 Like
3

Sorry, let me clarify with a test example and the resulting inconsistency.

I select a public iperf server from iPerf - Public iPerf3 servers.

On VyOS:

$ iperf3 -c ping.online.net -p 5200
Connecting to host ping.online.net, port 5200
[  5] local xxx.xxx.xxx.xxx port 35004 connected to 51.158.1.21 port 5200
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  89.5 MBytes   751 Mbits/sec    0   7.63 MBytes       
[  5]   1.00-2.00   sec   104 MBytes   870 Mbits/sec    0   7.63 MBytes       
[  5]   2.00-3.00   sec   106 MBytes   891 Mbits/sec    0   7.66 MBytes       
[  5]   3.00-4.00   sec   105 MBytes   881 Mbits/sec    0   7.85 MBytes       
[  5]   4.00-5.00   sec   105 MBytes   881 Mbits/sec    0   7.85 MBytes       
[  5]   5.00-6.00   sec   105 MBytes   881 Mbits/sec    0   7.85 MBytes       
[  5]   6.00-7.00   sec   106 MBytes   891 Mbits/sec    0   7.85 MBytes       
[  5]   7.00-8.00   sec   105 MBytes   881 Mbits/sec    0   7.91 MBytes       
[  5]   8.00-9.00   sec   102 MBytes   860 Mbits/sec    0   7.91 MBytes       
[  5]   9.00-10.00  sec   106 MBytes   891 Mbits/sec    0   7.91 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.01 GBytes   868 Mbits/sec    0             sender
[  5]   0.00-10.04  sec  1.01 GBytes   865 Mbits/sec                  receiver

$ iperf3 -R -c ping.online.net -p 5200
Connecting to host ping.online.net, port 5200
Reverse mode, remote host ping.online.net is sending
[  5] local xxx.xxx.xxx.xxx port 39490 connected to 51.158.1.21 port 5200
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  53.1 MBytes   445 Mbits/sec                  
[  5]   1.00-2.00   sec  69.8 MBytes   585 Mbits/sec                  
[  5]   2.00-3.00   sec  60.1 MBytes   504 Mbits/sec                  
[  5]   3.00-4.00   sec  56.1 MBytes   470 Mbits/sec                  
[  5]   4.00-5.00   sec  56.3 MBytes   473 Mbits/sec                  
[  5]   5.00-6.00   sec  52.2 MBytes   437 Mbits/sec                  
[  5]   6.00-7.00   sec  57.2 MBytes   480 Mbits/sec                  
[  5]   7.00-8.00   sec  56.2 MBytes   472 Mbits/sec                  
[  5]   8.00-9.00   sec  60.1 MBytes   504 Mbits/sec                  
[  5]   9.00-10.00  sec  53.6 MBytes   450 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec   593 MBytes   496 Mbits/sec  28031             sender
[  5]   0.00-10.00  sec   575 MBytes   482 Mbits/sec                  receiver

On a LAN device:

$  iperf3 -c ping.online.net -p 5200
Connecting to host ping.online.net, port 5200
[  5] local xxx.xxx.xxx.xxx port 46158 connected to 51.158.1.21 port 5200
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  96.9 MBytes   812 Mbits/sec  4562   3.66 MBytes       
[  5]   1.00-2.00   sec  97.8 MBytes   820 Mbits/sec  554   1.86 MBytes       
[  5]   2.00-3.00   sec  74.6 MBytes   626 Mbits/sec    0   1.91 MBytes       
[  5]   3.00-4.00   sec  80.0 MBytes   671 Mbits/sec    0   1.96 MBytes       
[  5]   4.00-5.00   sec  85.4 MBytes   716 Mbits/sec    0   2.02 MBytes       
[  5]   5.00-6.00   sec  80.0 MBytes   671 Mbits/sec    0   2.07 MBytes       
[  5]   6.00-7.00   sec  90.6 MBytes   760 Mbits/sec    0   2.13 MBytes       
[  5]   7.00-8.00   sec  85.5 MBytes   717 Mbits/sec    0   2.18 MBytes       
[  5]   8.00-9.00   sec  90.6 MBytes   760 Mbits/sec    0   2.23 MBytes       
[  5]   9.00-10.00  sec  90.6 MBytes   760 Mbits/sec    0   2.28 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   872 MBytes   731 Mbits/sec  5116            sender
[  5]   0.00-10.04  sec   861 MBytes   719 Mbits/sec                  receiver

$ iperf3 -R -c ping.online.net -p 5200
Connecting to host ping.online.net, port 5200
Reverse mode, remote host ping.online.net is sending
[  5] local xxx.xxx.xxx.xxx port 53064 connected to 51.158.1.21 port 5200
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  19.8 MBytes   166 Mbits/sec                  
[  5]   1.00-2.00   sec  15.5 MBytes   130 Mbits/sec                  
[  5]   2.00-3.00   sec  12.8 MBytes   107 Mbits/sec                  
[  5]   3.00-4.00   sec  9.12 MBytes  76.5 Mbits/sec                  
[  5]   4.00-5.00   sec  6.88 MBytes  57.7 Mbits/sec                  
[  5]   5.00-6.00   sec  5.62 MBytes  47.2 Mbits/sec                  
[  5]   6.00-7.00   sec  5.12 MBytes  43.0 Mbits/sec                  
[  5]   7.00-8.00   sec  4.75 MBytes  39.8 Mbits/sec                  
[  5]   8.00-9.00   sec  4.88 MBytes  40.9 Mbits/sec                  
[  5]   9.00-10.00  sec  4.50 MBytes  37.7 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   102 MBytes  84.9 Mbits/sec  26868            sender
[  5]   0.00-10.00  sec  88.9 MBytes  74.5 Mbits/sec                  receiver

And then you mention wireguard and a LAN Bridge?

All ports for LAN communication are members of the bridge, they are not used individually (i.e. no IP address on any of them).

As for WireGuard: it’s a typical road warrior client setup with a VPN provider. The LAN bridge has a VLAN defined with a different subnet and the default route pointing at the WireGuard interface.

There is still masquerade NAT involved, which I forgot to mention in the initial post.

These are the results on the same client but through that VLAN rather than the trunk:

$ iperf3 -c ping.online.net -p 5200
Connecting to host ping.online.net, port 5200
[  5] local xxx.xxx.xxx.xxx port 34946 connected to 51.158.1.21 port 5200
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  99.4 MBytes   833 Mbits/sec   31   1.25 MBytes       
[  5]   1.00-2.00   sec  53.5 MBytes   449 Mbits/sec  108   91.7 KBytes       
[  5]   2.00-3.00   sec  15.9 MBytes   133 Mbits/sec   12    111 KBytes       
[  5]   3.00-4.00   sec  21.1 MBytes   177 Mbits/sec    0    226 KBytes       
[  5]   4.00-5.00   sec  15.9 MBytes   133 Mbits/sec   14    205 KBytes       
[  5]   5.00-6.00   sec  31.8 MBytes   266 Mbits/sec    0    316 KBytes       
[  5]   6.00-7.00   sec  42.5 MBytes   357 Mbits/sec    0    428 KBytes       
[  5]   7.00-8.00   sec  37.0 MBytes   310 Mbits/sec   87    165 KBytes       
[  5]   8.00-9.00   sec  26.5 MBytes   222 Mbits/sec    0    280 KBytes       
[  5]   9.00-10.00  sec  37.1 MBytes   311 Mbits/sec    0    389 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   381 MBytes   319 Mbits/sec  252            sender
[  5]   0.00-10.03  sec   368 MBytes   307 Mbits/sec                  receiver

iperf Done.

$ iperf3 -R -c ping.online.net -p 5200
Connecting to host ping.online.net, port 5200
Reverse mode, remote host ping.online.net is sending
[  5] local xxx.xxx.xxx.xxx port 33956 connected to 51.158.1.21 port 5200
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  56.0 MBytes   469 Mbits/sec                  
[  5]   1.00-2.00   sec  75.0 MBytes   629 Mbits/sec                  
[  5]   2.00-3.00   sec  69.2 MBytes   581 Mbits/sec                  
[  5]   3.00-4.00   sec  76.1 MBytes   639 Mbits/sec                  
[  5]   4.00-5.00   sec  75.0 MBytes   629 Mbits/sec                  
[  5]   5.00-6.00   sec  72.8 MBytes   610 Mbits/sec                  
[  5]   6.00-7.00   sec  66.0 MBytes   554 Mbits/sec                  
[  5]   7.00-8.00   sec  71.4 MBytes   599 Mbits/sec                  
[  5]   8.00-9.00   sec  74.0 MBytes   621 Mbits/sec                  
[  5]   9.00-10.00  sec  79.9 MBytes   670 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   798 MBytes   667 Mbits/sec  633            sender
[  5]   0.00-10.00  sec   715 MBytes   600 Mbits/sec                  receiver

iperf Done.

And bridges on VyOS/Linux in general tend to come with performance impacts.

I understand, but the CPU is an i5-7500T, which I don’t expect to be causing any bottlenecks whatsoever.

Without understanding where the issue actually is, I would suggest ensuring you’ve turned off offloads, and then if they’re already off, try turning them on.

Thank you, I’m going to try right away.

4

Issue solved by disabling “TCP Segmentation Offloading” (tso) on the WAN interface alone, keeping every other offload enabled.

For reference, one router has a Emulex OCE11102:

01:00.0 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 03)
        Subsystem: Emulex Corporation OneConnect 10Gb NIC (be3) [10df:e72a]
        Kernel driver in use: be2net
        Kernel modules: be2net
01:00.1 Ethernet controller [0200]: Emulex Corporation OneConnect 10Gb NIC (be3) [19a2:0710] (rev 03)
        Subsystem: Emulex Corporation OneConnect 10Gb NIC (be3) [10df:e72a]
        Kernel driver in use: be2net
        Kernel modules: be2net

The other has a Silicom PE310G4SPI9-XR:

03:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
03:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
1 Like