Disable interface offloading permanently

Hi,

it doesn’t seem to matter, whether I delete offloading completely using
delete interface ether eth1 offload or deleting just the leaves using
delete interface ether eth1 offload gro|gso|...
Although removed, commited and saved, the offload settings automatically appear again after any reboot. Thus my question is:

How can I prevent vyos (1.5 nightly) from automatically activating network interface offloading (gro gso …) during boot?

Thanks a lot for your support!
Best regards,
vyozzy

Oh, I just found out it’s just the tcp segment offloading (tso) which causes problems here.

As my vyos instance runs in a proxmox kvm I think I need to disable tso there:

iface vmbr1 inet manual
        bridge-ports enp10s0f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        pre-up  ethtool -K enp10s0f0np0 tso off #<- pre disables tso on the physical nic
        post-up ethtool -K vmbr1 tso off #<- post disables tso on the virtual bridge

Haven’t tested this yet (waiting for a proxmox reboot-window), but the above settings seem promissing to me.

I too see that offloading gets enabled after each boot.

What even more interesting (at least to me) is with offloading “on”, my CPU load is between 0.10 and and 0.50, where as with offload deleted “off”, my CPU load drops to near 0 (between 0.00 to 0.02). This seems totally counter intuitive - I would have expected just the opposite.

Either way I’m not too worried since the load is small for my 4-core CPU - its more a curiosity.

The problem over here is a massive network throughput performance loss. Enabling offload tso results in nearly zero throughput:

The current setup is quite simple:

  • Proxmox (Intel x710, Dual-Port 10GBit Fibre; non-pci-passthrough)
  • Switch (offers 2x 10Gbits/s Fibre + 16 Gbit/s copper)
  • physical Win10 Client 1GBit Ethernet connected to the switch (VLAN 20)
  • kvm1: vyos 1.5 nightly (eth1 connected to vmbr1.10, VLAN 10)
  • kvm2: vyos 1.5 nightly (eth1 connected to vmbr1.10, VLAN 10)
  • physical Lnx-Server 1GBit Ethernet connected to the switch (VLAN 1001)

Only one of Proxmox’s 10Gbit/s-Fibre-Ports is directly connected to the switch, the other x710-port is unused (no LAG; no Bond, just straight).
Both kvms are setup nearly identical: kvm1 is a plain router, kvm2 uses a small IPv4-firewall ruleset (that imo doesn’t matter here).

iperf3@Win10 <-VLAN20-> eth2|KVM1|eth1 <-VLAN10-> eth1|KVM2|eth6 <-VLAN1001-> iperf3@Lnx-Server

Interface setup of kvm1 and kvm2:

# sho interfaces ethernet eth1
 address 192.168...
 address 2001:...
 description transfer
 offload {
     gro
     gso
     sg
     tso
 }

running iperf from Win10 to Lnx-Server shows:

# iperf -c Lnx-Server
[  5] local Win10 port 57547 connected to Lnx-Server port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   256 KBytes  2.09 Mbits/sec
[  5]   1.00-2.01   sec  0.00 Bytes  0.00 bits/sec
[  5]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec
[  5]   3.01-4.01   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.01-5.02   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.02-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.01   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   256 KBytes   209 Kbits/sec                  sender
[  5]   0.00-10.02  sec  4.28 KBytes  3.50 Kbits/sec                  receiver

After deleting ‘offload tso’ on eth1|kvm1 (leaving kvm2’s completely untouched, aka tso enabled!) everything seems fine:

[  5] local Win10 port 57574 connected to Lnx-Server port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   933 Mbits/sec
[  5]   1.00-2.00   sec   111 MBytes   931 Mbits/sec
[  5]   2.00-3.01   sec   111 MBytes   931 Mbits/sec
[  5]   3.01-4.01   sec   111 MBytes   930 Mbits/sec
[  5]   4.01-5.01   sec   111 MBytes   931 Mbits/sec
[  5]   5.01-6.01   sec   111 MBytes   930 Mbits/sec
[  5]   6.01-7.02   sec   110 MBytes   924 Mbits/sec
[  5]   7.02-8.00   sec   110 MBytes   931 Mbits/sec
[  5]   8.00-9.00   sec   111 MBytes   928 Mbits/sec
[  5]   9.00-10.01  sec   111 MBytes   931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.08 GBytes   930 Mbits/sec                  sender
[  5]   0.00-10.02  sec  1.08 GBytes   928 Mbits/sec                  receiver

I don’t exactly understand what’s going on here in detail. As a first step (just a workaround) I tried to get vyos to respect the physical and virtual bridge interfaces’ tso: off setting during start up.

These are my Proxmox’s physical interface’s port offload settings (disabled using pre-up ethtool -K ... via /etc/network/interfaces, see above):

# ethtool -k enp10s0f0np0 | grep -v fixed
Features for enp10s0f0np0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-udp-segmentation: on
tx-nocache-copy: off
loopback: off
l2-fwd-offload: off
hw-tc-offload: off
rx-udp_tunnel-port-offload: on
rx-gro-list: off
rx-udp-gro-forwarding: off

And these are Proxmox’s virtual-bridge interface’s offload settings (disabled using post-up ethtool -K ... during proxmox boot):

# ethtool -k vmbr1 | grep -v fixed
Features for vmbr1:
tx-checksumming: on
        tx-checksum-ip-generic: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
tx-vlan-offload: on
highdma: on
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: on
tx-sctp-segmentation: on
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: on
tx-nocache-copy: off
tx-vlan-stag-hw-insert: on
rx-gro-list: off
rx-udp-gro-forwarding: off

But vyos still seems to ignore these settings: after another reboot eth1’s tso offload was automatically enabled again.
Additionally disabling tso offloading on proxmox’s virtual-bridge’s vlan interface (vmbr1.10)(eth1@kvm1) had no effect, too:

# ethtool -K vmbr1.10 tso off
# ethtool -k vmbr1.10 | grep -v fixed
Features for vmbr1.10:
tx-checksumming: on
        tx-checksum-ip-generic: on
        tx-checksum-fcoe-crc: off [requested on]
        tx-checksum-sctp: off [requested on]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: on
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
highdma: on
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-sctp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: on
fcoe-mtu: off [requested on]
tx-nocache-copy: off
rx-gro-list: off
rx-udp-gro-forwarding: off
root@pve01:/usr/local/home/hansi#

Thus my question still persists: How do I get vyos to disable tso offloading during boot?
Leaving tso offloading enabled on all other interfaces doesn’t harm. It seems to me tso enabled on both ends of the link kvm1|eth1<–>eth1|kvm2 some kind of collide? I’m out of ideas. Any ideas are welcome!
Thank you very much.

Best regards,
vyozzy

So you run VyOS as a VM guest in Proxmox?

How is your proxmox settings for this VM?

Inspecting the code, I believe it’s forcing the config to reflect what the kernel has it set to at boot time. I’m not exactly sure why this is being done. I believe the fix would be to remove these lines - Maybe file a bug (https://vyos.dev) and see what the more knowledgeable folks say.

1 Like

Thank you very much for this hint!

Deleting both lines works fine over here: After reboot tso isn’t activated anymore.

I filed a bug (⚓ T6716 Offload settings are being automatically updated to reflect kernel settings) - lets see what the developers say…

Looks like a similar (or the same) issue being discussed here:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.