The problem over here is a massive network throughput performance loss. Enabling offload tso
results in nearly zero throughput:
The current setup is quite simple:
- Proxmox (Intel x710, Dual-Port 10GBit Fibre; non-pci-passthrough)
- Switch (offers 2x 10Gbits/s Fibre + 16 Gbit/s copper)
- physical Win10 Client 1GBit Ethernet connected to the switch (VLAN 20)
- kvm1: vyos 1.5 nightly (eth1 connected to vmbr1.10, VLAN 10)
- kvm2: vyos 1.5 nightly (eth1 connected to vmbr1.10, VLAN 10)
- physical Lnx-Server 1GBit Ethernet connected to the switch (VLAN 1001)
Only one of Proxmox’s 10Gbit/s-Fibre-Ports is directly connected to the switch, the other x710-port is unused (no LAG; no Bond, just straight).
Both kvms are setup nearly identical: kvm1 is a plain router, kvm2 uses a small IPv4-firewall ruleset (that imo doesn’t matter here).
iperf3@Win10 <-VLAN20-> eth2|KVM1|eth1 <-VLAN10-> eth1|KVM2|eth6 <-VLAN1001-> iperf3@Lnx-Server
Interface setup of kvm1 and kvm2:
# sho interfaces ethernet eth1
address 192.168...
address 2001:...
description transfer
offload {
gro
gso
sg
tso
}
running iperf from Win10 to Lnx-Server shows:
# iperf -c Lnx-Server
[ 5] local Win10 port 57547 connected to Lnx-Server port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 256 KBytes 2.09 Mbits/sec
[ 5] 1.00-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.01-4.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.01-5.02 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.02-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 256 KBytes 209 Kbits/sec sender
[ 5] 0.00-10.02 sec 4.28 KBytes 3.50 Kbits/sec receiver
After deleting ‘offload tso’ on eth1|kvm1 (leaving kvm2’s completely untouched, aka tso enabled!) everything seems fine:
[ 5] local Win10 port 57574 connected to Lnx-Server port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 931 Mbits/sec
[ 5] 2.00-3.01 sec 111 MBytes 931 Mbits/sec
[ 5] 3.01-4.01 sec 111 MBytes 930 Mbits/sec
[ 5] 4.01-5.01 sec 111 MBytes 931 Mbits/sec
[ 5] 5.01-6.01 sec 111 MBytes 930 Mbits/sec
[ 5] 6.01-7.02 sec 110 MBytes 924 Mbits/sec
[ 5] 7.02-8.00 sec 110 MBytes 931 Mbits/sec
[ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec
[ 5] 9.00-10.01 sec 111 MBytes 931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 1.08 GBytes 930 Mbits/sec sender
[ 5] 0.00-10.02 sec 1.08 GBytes 928 Mbits/sec receiver
I don’t exactly understand what’s going on here in detail. As a first step (just a workaround) I tried to get vyos to respect the physical and virtual bridge interfaces’ tso: off
setting during start up.
These are my Proxmox’s physical interface’s port offload settings (disabled using pre-up ethtool -K ...
via /etc/network/interfaces, see above):
# ethtool -k enp10s0f0np0 | grep -v fixed
Features for enp10s0f0np0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ip-generic: on
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-udp-segmentation: on
tx-nocache-copy: off
loopback: off
l2-fwd-offload: off
hw-tc-offload: off
rx-udp_tunnel-port-offload: on
rx-gro-list: off
rx-udp-gro-forwarding: off
And these are Proxmox’s virtual-bridge interface’s offload settings (disabled using post-up ethtool -K ...
during proxmox boot):
# ethtool -k vmbr1 | grep -v fixed
Features for vmbr1:
tx-checksumming: on
tx-checksum-ip-generic: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
tx-vlan-offload: on
highdma: on
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: on
tx-sctp-segmentation: on
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: on
tx-nocache-copy: off
tx-vlan-stag-hw-insert: on
rx-gro-list: off
rx-udp-gro-forwarding: off
But vyos still seems to ignore these settings: after another reboot eth1’s tso offload was automatically enabled again.
Additionally disabling tso offloading on proxmox’s virtual-bridge’s vlan interface (vmbr1.10)(eth1@kvm1) had no effect, too:
# ethtool -K vmbr1.10 tso off
# ethtool -k vmbr1.10 | grep -v fixed
Features for vmbr1.10:
tx-checksumming: on
tx-checksum-ip-generic: on
tx-checksum-fcoe-crc: off [requested on]
tx-checksum-sctp: off [requested on]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
highdma: on
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-sctp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: on
fcoe-mtu: off [requested on]
tx-nocache-copy: off
rx-gro-list: off
rx-udp-gro-forwarding: off
root@pve01:/usr/local/home/hansi#
Thus my question still persists: How do I get vyos to disable tso offloading during boot?
Leaving tso offloading enabled on all other interfaces doesn’t harm. It seems to me tso enabled on both ends of the link kvm1|eth1<–>eth1|kvm2 some kind of collide? I’m out of ideas. Any ideas are welcome!
Thank you very much.
Best regards,
vyozzy