Something keeps adding offloads back to my interface, breaking my WAN

Bugs
1

This is a major crisis. Offloads are broken on the most common NICs out there, the i21x and i22x devices. What keeps re-enabling offloads on VyOS? My configuration never added them, and something keeps adding them back in. I keep deleting them, and they keep coming back.

A lot of people have reported this issue.

Version:          VyOS 1.5-rolling-202408060021
Release train:    current
Release flavor:   generic

Built by:         autobuild@vyos.net
Built on:         Tue 06 Aug 2024 02:50 UTC
Build UUID:       804c50de-8b8e-46d3-9c21-05fcb46600c6
Build commit ID:  4cec5e2d52ca57

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  HP
Hardware model:   HP EliteDesk 800 G5 Desktop Mini
...

Copyright:        VyOS maintainers and contributors
2

Agree that lot of people had reported this issue.
Feel free to test newer image, since some patches were introduced recently.
Task: ⚓ T6716 Offload settings are being automatically updated to reflect kernel settings

3

Just a heads up - I went ahead and updated, rebooted, and I see that the offloads came back:

$ ssh vyos@10.2.0.1
Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202409250007
   └ ──┘  current

 * Support portal: https://support.vyos.io
 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
vyos@vyos:~$ show configuration commands | grep offload
set interfaces ethernet eth2 offload gro
4

This is just after update?
Can you remove offloads, commit, save, and reboot in the latest version?

5

That is what I did - I removed offloads, committed, saved, added the new system image, said yes to copying the config, and rebooted into the latest version.

Here is the complete bash history showing exactly what I did:

(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202408060021
   └ ──┘  current

 * Support portal: https://support.vyos.io
 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Wed Sep 25 00:42:44 2024 from appmana-001
vyos@vyos:~$ show configuration commands | grep gso
set interfaces ethernet eth0 offload gso
vyos@vyos:~$ configure
[edit]
vyos@vyos# delete set interfaces ethernet eth0 offload

  Configuration path: [set] is not valid
  Delete failed

[edit]
vyos@vyos# delete interfaces ethernet eth0 offload
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# show configuration commands | grep gso

  Configuration path: [configuration] is not valid
  Show failed

[edit]
vyos@vyos# save
[edit]
vyos@vyos# exit
exit
vyos@vyos:~$ show version
Version:          VyOS 1.5-rolling-202408060021
Release train:    current
Release flavor:   generic

Built by:         autobuild@vyos.net
Built on:         Tue 06 Aug 2024 02:50 UTC
Build UUID:       804c50de-8b8e-46d3-9c21-05fcb46600c6
Build commit ID:  4cec5e2d52ca57

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  HP
Hardware model:   HP EliteDesk 800 G5 Desktop Mini
Hardware S/N:     MXL95025NY
Hardware UUID:    800b5dc3-e6c8-ba65-0bcb-dc6bfdfbccb2

Copyright:        VyOS maintainers and contributors
vyos@vyos:~$ add system image https://github.com/vyos/vyos-nightly-build/releases/download/1.5-rolling-202409250007/vyos-1.5-rolling-202409250007-generic-amd64.iso
Redirecting to https://objects.githubusercontent.com/github-production-release-asset-2e65be/674742659/4f26d8ed-24b6-4108-9de2-6579f1b5ad77?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240926%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240926T184041Z&X-Amz-Expires=300&X-Amz-Signature=00721b70881b2c3981aebf2090d189509a9806f32bb20299f7aee6999d5315fd&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvyos-1.5-rolling-202409250007-generic-amd64.iso&response-content-type=application%2Foctet-stream
The file is 466.000 MiB.
[#########################################################################################################] 100%
Validating signature
Signature is valid
Validating image compatibility
Validating image checksums
What would you like to name this image? (Default: 1.5-rolling-202409250007)
Would you like to set the new image as the default one for boot? [Y/n] Y
An active configuration was found. Would you like to copy it to the new image? [Y/n] Y
Copying configuration directory
Would you like to copy SSH host keys? [Y/n] Y
Copying SSH host keys
Copying system image files
Cleaning up
Unmounting target filesystems
Removing temporary files
vyos@vyos:~$ reboot
Are you sure you want to reboot this system (vyos)? [y/N] y

Broadcast message from root@debian on pts/1 (Thu 2024-09-26 18:42:22 UTC):

The system will reboot now!


Connection to 10.2.0.1 closed by remote host.
Connection to 10.2.0.1 closed.
sh: C:\Windows\System32\OpenSSH\ssh.exe: Error 0xffffffff
(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
ssh: connect to host 10.2.0.1 port 22: Connection refused
(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
ssh: connect to host 10.2.0.1 port 22: Connection refused
(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202409250007
   └ ──┘  current

 * Support portal: https://support.vyos.io
 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
\vyos@vyos:~$ show configuration commands | grep offload
set interfaces ethernet eth2 offload gro
vyos@vyos:~$ configure
[edit]
vyos@vyos# delete interfaces ethernet eth2 offload
...

Observe that the interface that had offload enabled changed. So I am convinced there is still something that is adding offloads back in.

6

Understood.
But once the upgrade has finished and you are running the latest image, please:

  • Remove offloads (yes, once again, I know)
  • Commit
  • Save
  • Reboot

I tested this on bare metal installation on Monday, and worked. Let me know

7

Looks good!

$ ssh vyos@10.2.0.1
Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202409250007
   └ ──┘  current

 * Support portal: https://support.vyos.io
 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Thu Sep 26 18:50:43 2024 from appmana-001
vyos@vyos:~$ reboot
Are you sure you want to reboot this system (vyos)? [y/N] y

Broadcast message from root@debian on pts/1 (Thu 2024-09-26 19:07:05 UTC):

The system will reboot now!


Connection to 10.2.0.1 closed by remote host.
Connection to 10.2.0.1 closed.
sh: C:\Windows\System32\OpenSSH\ssh.exe: Error 0xffffffff
(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
ssh: connect to host 10.2.0.1 port 22: Connection refused
(.venv) ~/Documents/appmana $ ssh vyos@10.2.0.1
Welcome to VyOS!

   ┌── ┐
   . VyOS 1.5-rolling-202409250007
   └ ──┘  current

 * Support portal: https://support.vyos.io
 * Documentation:  https://docs.vyos.io/en/latest
 * Project news:   https://blog.vyos.io
 * Bug reports:    https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Thu Sep 26 19:06:59 2024 from appmana-001
vyos@vyos:~$ show configuration commands | grep offload
2 Likes