Hi,
it doesn’t seem to matter, whether I delete offloading completely using
delete interface ether eth1 offload or deleting just the leaves using
delete interface ether eth1 offload gro|gso|...
Although removed, commited and saved, the offload settings automatically appear again after any reboot. Thus my question is:
How can I prevent vyos (1.5 nightly) from automatically activating network interface offloading (gro gso …) during boot?
Thanks a lot for your support!
Best regards,
vyozzy
Oh, I just found out it’s just the tcp segment offloading (tso) which causes problems here.
As my vyos instance runs in a proxmox kvm I think I need to disable tso there:
iface vmbr1 inet manual
bridge-ports enp10s0f0np0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
pre-up ethtool -K enp10s0f0np0 tso off #<- pre disables tso on the physical nic
post-up ethtool -K vmbr1 tso off #<- post disables tso on the virtual bridge
Haven’t tested this yet (waiting for a proxmox reboot-window), but the above settings seem promissing to me.
I too see that offloading gets enabled after each boot.
What even more interesting (at least to me) is with offloading “on”, my CPU load is between 0.10 and and 0.50, where as with offload deleted “off”, my CPU load drops to near 0 (between 0.00 to 0.02). This seems totally counter intuitive - I would have expected just the opposite.
Either way I’m not too worried since the load is small for my 4-core CPU - its more a curiosity.
The problem over here is a massive network throughput performance loss. Enabling offload tso results in nearly zero throughput:
The current setup is quite simple:
Only one of Proxmox’s 10Gbit/s-Fibre-Ports is directly connected to the switch, the other x710-port is unused (no LAG; no Bond, just straight).
Both kvms are setup nearly identical: kvm1 is a plain router, kvm2 uses a small IPv4-firewall ruleset (that imo doesn’t matter here).
iperf3@Win10 <-VLAN20-> eth2|KVM1|eth1 <-VLAN10-> eth1|KVM2|eth6 <-VLAN1001-> iperf3@Lnx-Server
Interface setup of kvm1 and kvm2:
# sho interfaces ethernet eth1
address 192.168...
address 2001:...
description transfer
offload {
gro
gso
sg
tso
}
running iperf from Win10 to Lnx-Server shows:
# iperf -c Lnx-Server
[ 5] local Win10 port 57547 connected to Lnx-Server port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 256 KBytes 2.09 Mbits/sec
[ 5] 1.00-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.01-4.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.01-5.02 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.02-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 256 KBytes 209 Kbits/sec sender
[ 5] 0.00-10.02 sec 4.28 KBytes 3.50 Kbits/sec receiver
After deleting ‘offload tso’ on eth1|kvm1 (leaving kvm2’s completely untouched, aka tso enabled!) everything seems fine:
[ 5] local Win10 port 57574 connected to Lnx-Server port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 931 Mbits/sec
[ 5] 2.00-3.01 sec 111 MBytes 931 Mbits/sec
[ 5] 3.01-4.01 sec 111 MBytes 930 Mbits/sec
[ 5] 4.01-5.01 sec 111 MBytes 931 Mbits/sec
[ 5] 5.01-6.01 sec 111 MBytes 930 Mbits/sec
[ 5] 6.01-7.02 sec 110 MBytes 924 Mbits/sec
[ 5] 7.02-8.00 sec 110 MBytes 931 Mbits/sec
[ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec
[ 5] 9.00-10.01 sec 111 MBytes 931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 1.08 GBytes 930 Mbits/sec sender
[ 5] 0.00-10.02 sec 1.08 GBytes 928 Mbits/sec receiver
I don’t exactly understand what’s going on here in detail. As a first step (just a workaround) I tried to get vyos to respect the physical and virtual bridge interfaces’ tso: off setting during start up.
These are my Proxmox’s physical interface’s port offload settings (disabled using pre-up ethtool -K ... via /etc/network/interfaces, see above):
# ethtool -k enp10s0f0np0 | grep -v fixed
Features for enp10s0f0np0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ip-generic: on
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-udp-segmentation: on
tx-nocache-copy: off
loopback: off
l2-fwd-offload: off
hw-tc-offload: off
rx-udp_tunnel-port-offload: on
rx-gro-list: off
rx-udp-gro-forwarding: off
And these are Proxmox’s virtual-bridge interface’s offload settings (disabled using post-up ethtool -K ... during proxmox boot):
# ethtool -k vmbr1 | grep -v fixed
Features for vmbr1:
tx-checksumming: on
tx-checksum-ip-generic: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
tx-vlan-offload: on
highdma: on
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: on
tx-sctp-segmentation: on
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: on
tx-nocache-copy: off
tx-vlan-stag-hw-insert: on
rx-gro-list: off
rx-udp-gro-forwarding: off
But vyos still seems to ignore these settings: after another reboot eth1’s tso offload was automatically enabled again.
Additionally disabling tso offloading on proxmox’s virtual-bridge’s vlan interface (vmbr1.10)(eth1@kvm1) had no effect, too:
# ethtool -K vmbr1.10 tso off
# ethtool -k vmbr1.10 | grep -v fixed
Features for vmbr1.10:
tx-checksumming: on
tx-checksum-ip-generic: on
tx-checksum-fcoe-crc: off [requested on]
tx-checksum-sctp: off [requested on]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
highdma: on
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-sctp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: on
fcoe-mtu: off [requested on]
tx-nocache-copy: off
rx-gro-list: off
rx-udp-gro-forwarding: off
root@pve01:/usr/local/home/hansi#
Thus my question still persists: How do I get vyos to disable tso offloading during boot?
Leaving tso offloading enabled on all other interfaces doesn’t harm. It seems to me tso enabled on both ends of the link kvm1|eth1<–>eth1|kvm2 some kind of collide? I’m out of ideas. Any ideas are welcome!
Thank you very much.
Best regards,
vyozzy
So you run VyOS as a VM guest in Proxmox?
How is your proxmox settings for this VM?
Inspecting the code, I believe it’s forcing the config to reflect what the kernel has it set to at boot time. I’m not exactly sure why this is being done. I believe the fix would be to remove these lines - Maybe file a bug (https://vyos.dev) and see what the more knowledgeable folks say.
Thank you very much for this hint!
Deleting both lines works fine over here: After reboot tso isn’t activated anymore.
I filed a bug (⚓ T6716 Offload settings are being automatically updated to reflect kernel settings) - lets see what the developers say…
Looks like a similar (or the same) issue being discussed here: