Running VyOs in Proxmox, issues about high CPU Usage

General questions
1

Good evening all,

Foremost, sorry if this topic is a clone of another topic here. The one I had read had good workarounds, but sadly, I haven’t been able to make it work.

To the main issue, then. I’ve been trying to route 10G traffic through a Proxmox Virtual Machine. I’ve tried VyOS baremetal on the machine, and it works flawlessly using XDP (getting multi gig speeds even with 68Bytes packets). In the VM though, not so much. I’ve tried PCIe Passthrough, virtio, DPDK with OVS, but I still can’t make it work properly (ie : 10G speeds, no loss). Right now with OVS & DPDK, I’m only getting around 5600Mbit/s with 1470 Bytes packets.

Context : I’m running Proxmox on a Dell XR12, with 256 Gigs of ram, an Intel(R) Xeon(R) Gold 6338N CPU, and two network cards : one Intel E810 and the integrated Broadcomm Net Extreme 10/25G.

Testing wise, I’m using EXFOs, which can generate any traffic up to 10G.

I’m mainly getting IRQ issues, where the host will see 100% CPU ksoftirq processes, or in the case of OVS with DPDK, 100% ksoftirq processes within the router.

I’m running the latest VyOs nightly release.

Any help is very much appreciated.

Cheers

1 Like
2

Try to disable hyper threading
Only real cores should to process traffic
Only local node socket and local socket memory banks should process packets. How many sockets are you having?
Could you share screenshot “sudo top” and press “1”?

3

First & foremost, thanks for responding.

I’ve disabled hyperthreading on the host. Memory wise, we only have one socket, so there should be no issue with NICs not on the good CPU.

I’ve done a clean installation of the Proxmox Host, and a clean installation of a VyOS VM. I’ve abandoned DPDK with OVS as it’s very cumbersome to set up and does not resist a reboot of the host.

I’ve setup IOMMU and passed through the whole broadcomm card to the VM. I’ve only got one virtio for management purposes. Here is the VM Config : (Sorry, apparently I can’t post more than one embedded, I can post it later if needed)

Now, without XDP, I’m getting, as expected, bad results on my throughput test, only getting around 400kpps, with lots and lots of frame Loss. (Test is 9900Mbit/s continuous, random packet size)

The issue is, I now cannot turn XDP on the interfaces, as “XDP requires additional TX queues, too few available!”. I’ve seen this before on virtio interfaces, but not on pcie cards. I’m now at a complete loss as to what to do next.

For reference, here’s the interface physical detail :

vyos@whyWontYouWork:~$ show interfaces ethernet eth0 physical 
Settings for eth0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
                                25000baseCR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: RS  BASER
        Advertised link modes:  10000baseT/Full
                                25000baseCR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: RS         BASER
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x00002081 (8321)
                               drv tx_err hw
        Link detected: yes
Ring parameters for eth0:
Pre-set maximums:
RX:             2047
RX Mini:        n/a
RX Jumbo:       8191
TX:             2047
Current hardware settings:
RX:             511
RX Mini:        n/a
RX Jumbo:       0
TX:             511
driver: bnxt_en
version: 5.10.133-amd64-vyos
firmware-version: 220.0.57.0/pkg 22.00.07.60
expansion-rom-version: 
bus-info: 0000:06:10.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
vyos@whyWontYouWork:~$

Here’s the sudo top → 1 as requested.

Screenshot_20220728_124208
Screenshot_20220728_124208845×846 116 KB

Cheers

4

An update to this.

I’ve tried setting the queue manually on the interfaces with ethtool : (it core dumped the first time I tried 70 combined… works now)

vyos@whyWontYouWork# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:             37
TX:             37
Other:          n/a
Combined:       74
Current hardware settings:
RX:             0
TX:             0
Other:          n/a
Combined:       70
[edit]

However, it seems VyOs wasn’t that happy about the whole thing when enabling xdp :

vyos@whyWontYouWork# set interfaces ethernet eth0 xdp 
[edit]
vyos@whyWontYouWork# commit
[ interfaces ethernet eth0 ]
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Contact us using the online help desk if you have a subscription:
  https://support.vyos.io/
- Make sure you are running the latest version of VyOS available at:
  https://vyos.net/get/
- Consult the community forum to see how to handle this issue:
  https://forum.vyos.io
- Join us on Slack where our users exchange help and advice:
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your 
  business policy requires it)
- and include all the information presented below

Report time:      2022-07-28 13:01:32
Image version:    VyOS 1.4-rolling-202207270217
Release train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Wed 27 Jul 2022 02:17 UTC
Build UUID:       db3bf91b-152e-4cae-9faf-f7176be287d4
Build commit ID:  3327957356be99

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (Q35 + ICH9, 2009)
Hardware S/N:     
Hardware UUID:    88f32e10-cde2-4e63-bb89-f80e2d396e05

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 215, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 204, in apply
    e.update(ethernet)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 363, in update
    super().update(config)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1590, in update
    self.set_xdp('xdp' in config)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1382, in set_xdp
    return self._cmd(cmd)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
    return cmd(command, self.debug)
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
OSError: [Errno 30] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
returned: 
exit code: 30

noteworthy:
cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth0" }'
returned (out):

returned (err):
Error: No such file or directory
delete element inet vrf_zones ct_iface_map { eth0 }
                    ^^^^^^^^^
cmd 'xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0'
returned (out):

returned (err):
ERR: ifindex(2) link set xdp fd failed (12): Cannot allocate memory

[[interfaces ethernet eth0]] failed
Commit failed
[edit]
vyos@whyWontYouWork#

Here’s the current config for it (vanilla, except for the interfaces) :

vyos@whyWontYouWork# run show configuration 
interfaces {
    ethernet eth0 {
        address 10.1.1.1/24
    }
    ethernet eth1 {
        address 10.2.1.1/24
    }
    ethernet eth2 {
        address 10.178.128.52/25
    }
    ethernet eth3 {
        hw-id 90:8d:6e:f9:d7:94
    }
    ethernet eth4 {
        hw-id 90:8d:6e:f9:d7:95
    }
}
service {
    ssh {
        port 22
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name whyWontYouWork
    login {
        user vyos {
            authentication {
                encrypted-password ****************
                plaintext-password ****************
            }
        }
    }
    ntp {
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    option {
        performance throughput
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
}
[edit]
vyos@whyWontYouWork#

Hope this may help.

5

All interrupts are binded to one core
Try download and execute next sctips

sudo wget -O /config/scripts/irq-affinity.pl https://raw.githubusercontent.com/vyos/vyatta-cfg-system/crux/scripts/system/irq-affinity.pl
sudo perl /config/scripts/irq-affinity.pl eth0 auto
sudo perl /config/scripts/irq-affinity.pl eth1 auto
sudo perl /config/scripts/irq-affinity.pl eth2 auto

Also

sudo ethtool -G  eth0 rx 2047 
sudo ethtool -G  eth0 tx 2047
6

Alright, XDP is working now with your commands. However, it doesn’t survive post reboot, and as such the configuration fails to load.

What would be the prefered way on vyos to make changes via ethtools permanent?

Thanks

7

There is one bug with starting xdp after reboot T3939
Regarding ethtool and other post-boot commands, you can use the "vyos-postconfig-bootup.script " script

/config/scripts/vyos-postconfig-bootup.script

It will be executed after reboot

8

Hello,

I’ve taken a look at the post boot script you mentioned, but I still cannot load my configuration after boot for XDP.

Here’s what I’m doing :

vyos@whyWontYouWork:~$ cat /config/scripts/vyos-postconfig-bootup.script 
#!/bin/vbash
# This script is executed at boot time after VyOS configuration is fully applied.
# Any modifications required to work around unfixed bugs
# or use services not available through the VyOS CLI system can be placed here.

source /opt/vyatta/etc/functions/script-template

loadkeys fr

sudo perl /config/scripts/irq-affinity.pl eth0 auto
sudo perl /config/scripts/irq-affinity.pl eth1 auto
sudo perl /config/scripts/irq-affinity.pl eth2 auto

sudo ethtool -L eth0 combined 4
sudo ethtool -L eth1 combined 4

sudo ethtool -G  eth0 rx 2047 
sudo ethtool -G  eth0 tx 2047 

sudo ethtool -G  eth1 rx 2047 
sudo ethtool -G  eth1 tx 2047 


echo "STAND BY FOR EXPLOSIONS, APPLYING XDP"
configure
set interfaces ethernet eth0 xdp
set interfaces ethernet eth1 xdp
commit
exit

vyos@whyWontYouWork:~$

This causes the error 40 mentioned in T3939 :

Jul 29 10:49:24 systemd[1]: systemd-hostnamed.service: Succeeded.
Jul 29 10:49:29 vyos-router[1908]: STAND BY FOR EXPLOSIONS, APPLYING XDP
Jul 29 10:49:29 sudo[2031]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='eth0' /usr/libexec/vyos/conf_mode/interfaces-ethernet.py
Jul 29 10:49:29 sudo[2031]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jul 29 10:49:29 vyos-configd[712]: Received message: {"type": "init"}
Jul 29 10:49:29 vyos-configd[712]: config session pid is 1908
Jul 29 10:49:29 vyos-configd[712]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=eth0/usr/libexec/vyos/conf_mode/interfaces-ethernet.py"}
Jul 29 10:49:29 vyos-configd[712]: [Errno 40] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
Jul 29 10:49:29 vyos-configd[712]: returned:
Jul 29 10:49:29 vyos-configd[712]: exit code: 40
Jul 29 10:49:29 vyos-configd[712]: Sending response 4
Jul 29 10:49:29 systemd[1]: serial-getty@ttyS0.service: Succeeded.
Jul 29 10:49:29 systemd[1]: serial-getty@ttyS0.service: Scheduled restart job, restart counter is at 3.
Jul 29 10:49:29 systemd[1]: Stopped Serial Getty on ttyS0.
Jul 29 10:49:29 systemd[1]: Started Serial Getty on ttyS0.
Jul 29 10:49:29 agetty[2084]: /dev/ttyS0: not a tty
Jul 29 10:49:29 python3[2063]: Report time:      2022-07-29 10:49:29
Jul 29 10:49:29 python3[2063]: Image version:    VyOS 1.4-rolling-202207270217
Jul 29 10:49:29 python3[2063]: Release train:    sagitta
Jul 29 10:49:29 python3[2063]: Built by:         autobuild@vyos.net
Jul 29 10:49:29 python3[2063]: Built on:         Wed 27 Jul 2022 02:17 UTC
Jul 29 10:49:29 python3[2063]: Build UUID:       db3bf91b-152e-4cae-9faf-f7176be287d4
Jul 29 10:49:29 python3[2063]: Build commit ID:  3327957356be99
Jul 29 10:49:29 Architecture[2063]:     x86_64
Jul 29 10:49:29 python3[2063]: Boot via:         installed image
Jul 29 10:49:29 python3[2063]: System type:      KVM guest
Jul 29 10:49:29 python3[2063]: Hardware vendor:  QEMU
Jul 29 10:49:29 python3[2063]: Hardware model:   Standard PC (Q35 + ICH9, 2009)
Jul 29 10:49:29 python3[2063]: Hardware S/N:
Jul 29 10:49:29 python3[2063]: Hardware UUID:    88f32e10-cde2-4e63-bb89-f80e2d396e05
Jul 29 10:49:29 python3[2063]: Traceback (most recent call last):
Jul 29 10:49:29 python3[2063]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 215, in <module>
Jul 29 10:49:29 python3[2063]:     apply(c)
Jul 29 10:49:29 python3[2063]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 204, in apply
Jul 29 10:49:29 python3[2063]:     e.update(ethernet)
Jul 29 10:49:29 python3[2063]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 363, in update
Jul 29 10:49:29 python3[2063]:     super().update(config)
Jul 29 10:49:29 python3[2063]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1590, in update
Jul 29 10:49:29 python3[2063]:     self.set_xdp('xdp' in config)
Jul 29 10:49:29 python3[2063]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1382, in set_xdp
Jul 29 10:49:29 python3[2063]:     return self._cmd(cmd)
Jul 29 10:49:29 python3[2063]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
Jul 29 10:49:29 python3[2063]:     return cmd(command, self.debug)
Jul 29 10:49:29 python3[2063]:   File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
Jul 29 10:49:29 python3[2063]:     raise OSError(code, feedback)
Jul 29 10:49:29 OSError[2063]: [Errno 40] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
Jul 29 10:49:29 returned[2063]: 
Jul 29 10:49:29 python3[2063]: exit code: 40
Jul 29 10:49:29 noteworthy[2063]: 
Jul 29 10:49:29 python3[2063]: cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth0" }'
Jul 29 10:49:29 python3[2063]: returned (out):
Jul 29 10:49:29 python3[2063]: returned (err):
Jul 29 10:49:29 Error[2063]: No such file or directory
Jul 29 10:49:29 python3[2063]: delete element inet vrf_zones ct_iface_map { eth0 }
Jul 29 10:49:29 python3[2063]:                     ^^^^^^^^^
Jul 29 10:49:29 python3[2063]: cmd 'xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0'
Jul 29 10:49:29 python3[2063]: returned (out):
Jul 29 10:49:29 python3[2063]: returned (err):
Jul 29 10:49:29 libbpf[2063]: failed to create map (name: 'redirect_params'): Operation not permitted(-1)
Jul 29 10:49:29 libbpf[2063]: failed to load object '/usr/share/vyos/xdp/xdp_prog_kern.o'
Jul 29 10:49:29 ERR[2063]: loading BPF-OBJ file(/usr/share/vyos/xdp/xdp_prog_kern.o) (-22): Invalid argument
Jul 29 10:49:29 ERR[2063]: loading file: /usr/share/vyos/xdp/xdp_prog_kern.o
Jul 29 10:49:29 vyos-router[2027]: [ interfaces ethernet eth0 ]
Jul 29 10:49:29 vyos-router[2027]: VyOS had an issue completing a command.
Jul 29 10:49:29 vyos-router[2027]: We are sorry that you encountered a problem while using VyOS.
Jul 29 10:49:29 vyos-router[2027]: There are a few things you can do to help us (and yourself):
Jul 29 10:49:29 vyos-router[2027]: - Contact us using the online help desk if you have a subscription:
Jul 29 10:49:29 vyos-router[2027]:   https://support.vyos.io/
Jul 29 10:49:29 vyos-router[2027]: - Make sure you are running the latest version of VyOS available at:
Jul 29 10:49:29 vyos-router[2027]:   https://vyos.net/get/
Jul 29 10:49:29 vyos-router[2027]: - Consult the community forum to see how to handle this issue:
Jul 29 10:49:29 vyos-router[2027]:   https://forum.vyos.io
Jul 29 10:49:29 vyos-router[2027]: - Join us on Slack where our users exchange help and advice:
Jul 29 10:49:29 vyos-router[2027]:   https://vyos.slack.com
Jul 29 10:49:29 vyos-router[2027]: When reporting problems, please include as much information as possible:
Jul 29 10:49:29 vyos-router[2027]: - do not obfuscate any data (feel free to contact us privately if your
Jul 29 10:49:29 vyos-router[2027]:   business policy requires it)
Jul 29 10:49:29 vyos-router[2027]: - and include all the information presented below
Jul 29 10:49:29 vyos-router[2027]: Report time:      2022-07-29 10:49:29
Jul 29 10:49:29 vyos-router[2027]: Image version:    VyOS 1.4-rolling-202207270217
Jul 29 10:49:29 vyos-router[2027]: Release train:    sagitta
Jul 29 10:49:29 vyos-router[2027]: Built by:         autobuild@vyos.net
Jul 29 10:49:29 vyos-router[2027]: Built on:         Wed 27 Jul 2022 02:17 UTC
Jul 29 10:49:29 vyos-router[2027]: Build UUID:       db3bf91b-152e-4cae-9faf-f7176be287d4
Jul 29 10:49:29 vyos-router[2027]: Build commit ID:  3327957356be99
Jul 29 10:49:29 vyos-router[2027]: Architecture:     x86_64
Jul 29 10:49:29 vyos-router[2027]: Boot via:         installed image
Jul 29 10:49:29 vyos-router[2027]: System type:      KVM guest
Jul 29 10:49:29 vyos-router[2027]: Hardware vendor:  QEMU
Jul 29 10:49:29 vyos-router[2027]: Hardware model:   Standard PC (Q35 + ICH9, 2009)
Jul 29 10:49:29 vyos-router[2027]: Hardware S/N:
Jul 29 10:49:29 vyos-router[2027]: Hardware UUID:    88f32e10-cde2-4e63-bb89-f80e2d396e05
Jul 29 10:49:29 vyos-router[2027]: Traceback (most recent call last):
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 215, in <module>
Jul 29 10:49:29 vyos-router[2027]:     apply(c)
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 204, in apply
Jul 29 10:49:29 vyos-router[2027]:     e.update(ethernet)
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 363, in update
Jul 29 10:49:29 vyos-router[2027]:     super().update(config)
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1590, in update
Jul 29 10:49:29 vyos-router[2027]:     self.set_xdp('xdp' in config)
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1382, in set_xdp
Jul 29 10:49:29 vyos-router[2027]:     return self._cmd(cmd)
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
Jul 29 10:49:29 vyos-router[2027]:     return cmd(command, self.debug)
Jul 29 10:49:29 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
Jul 29 10:49:29 vyos-router[2027]:     raise OSError(code, feedback)
Jul 29 10:49:29 vyos-router[2027]: OSError: [Errno 40] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
Jul 29 10:49:29 vyos-router[2027]: returned:
Jul 29 10:49:29 vyos-router[2027]: exit code: 40
Jul 29 10:49:29 vyos-router[2027]: noteworthy:
Jul 29 10:49:29 vyos-router[2027]: cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth0" }'
Jul 29 10:49:29 vyos-router[2027]: returned (out):
Jul 29 10:49:29 vyos-router[2027]: returned (err):
Jul 29 10:49:29 vyos-router[2027]: Error: No such file or directory
Jul 29 10:49:29 vyos-router[2027]: delete element inet vrf_zones ct_iface_map { eth0 }
Jul 29 10:49:29 vyos-router[2027]:                     ^^^^^^^^^
Jul 29 10:49:29 vyos-router[2027]: cmd 'xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0'
Jul 29 10:49:29 vyos-router[2027]: returned (out):
Jul 29 10:49:29 vyos-router[2027]: returned (err):
Jul 29 10:49:29 vyos-router[2027]: libbpf: failed to create map (name: 'redirect_params'): Operation not permitted(-1)
Jul 29 10:49:29 vyos-router[2027]: libbpf: failed to load object '/usr/share/vyos/xdp/xdp_prog_kern.o'
Jul 29 10:49:29 vyos-router[2027]: ERR: loading BPF-OBJ file(/usr/share/vyos/xdp/xdp_prog_kern.o) (-22): Invalid argument
Jul 29 10:49:29 vyos-router[2027]: ERR: loading file: /usr/share/vyos/xdp/xdp_prog_kern.o
Jul 29 10:49:29 sudo[2031]: pam_unix(sudo:session): session closed for user root
Jul 29 10:49:29 sudo[2102]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='eth1' /usr/libexec/vyos/conf_mode/interfaces-ethernet.py
Jul 29 10:49:29 sudo[2102]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Jul 29 10:49:29 vyos-configd[712]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=eth1/usr/libexec/vyos/conf_mode/interfaces-ethernet.py"}
Jul 29 10:49:29 vyos-configd[712]: [Errno 40] failed to run command: xdp_loader -d eth1 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth1
Jul 29 10:49:29 vyos-configd[712]: returned:
Jul 29 10:49:29 vyos-configd[712]: exit code: 40
Jul 29 10:49:29 vyos-configd[712]: Sending response 4
Jul 29 10:49:30 python3[2131]: Report time:      2022-07-29 10:49:30
Jul 29 10:49:30 python3[2131]: Image version:    VyOS 1.4-rolling-202207270217
Jul 29 10:49:30 python3[2131]: Release train:    sagitta
Jul 29 10:49:30 python3[2131]: Built by:         autobuild@vyos.net
Jul 29 10:49:30 python3[2131]: Built on:         Wed 27 Jul 2022 02:17 UTC
Jul 29 10:49:30 python3[2131]: Build UUID:       db3bf91b-152e-4cae-9faf-f7176be287d4
Jul 29 10:49:30 python3[2131]: Build commit ID:  3327957356be99
Jul 29 10:49:30 Architecture[2131]:     x86_64
Jul 29 10:49:30 python3[2131]: Boot via:         installed image
Jul 29 10:49:30 python3[2131]: System type:      KVM guest
Jul 29 10:49:30 python3[2131]: Hardware vendor:  QEMU
Jul 29 10:49:30 python3[2131]: Hardware model:   Standard PC (Q35 + ICH9, 2009)
Jul 29 10:49:30 python3[2131]: Hardware S/N:
Jul 29 10:49:30 python3[2131]: Hardware UUID:    88f32e10-cde2-4e63-bb89-f80e2d396e05
Jul 29 10:49:30 python3[2131]: Traceback (most recent call last):
Jul 29 10:49:30 python3[2131]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 215, in <module>
Jul 29 10:49:30 python3[2131]:     apply(c)
Jul 29 10:49:30 python3[2131]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 204, in apply
Jul 29 10:49:30 python3[2131]:     e.update(ethernet)
Jul 29 10:49:30 python3[2131]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 363, in update
Jul 29 10:49:30 python3[2131]:     super().update(config)
Jul 29 10:49:30 python3[2131]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1590, in update
Jul 29 10:49:30 python3[2131]:     self.set_xdp('xdp' in config)
Jul 29 10:49:30 python3[2131]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1382, in set_xdp
Jul 29 10:49:30 python3[2131]:     return self._cmd(cmd)
Jul 29 10:49:30 python3[2131]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
Jul 29 10:49:30 python3[2131]:     return cmd(command, self.debug)
Jul 29 10:49:30 python3[2131]:   File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
Jul 29 10:49:30 python3[2131]:     raise OSError(code, feedback)
Jul 29 10:49:30 OSError[2131]: [Errno 40] failed to run command: xdp_loader -d eth1 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth1
Jul 29 10:49:30 returned[2131]: 
Jul 29 10:49:30 python3[2131]: exit code: 40
Jul 29 10:49:30 noteworthy[2131]: 
Jul 29 10:49:30 python3[2131]: cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth1" }'
Jul 29 10:49:30 python3[2131]: returned (out):
Jul 29 10:49:30 python3[2131]: returned (err):
Jul 29 10:49:30 Error[2131]: No such file or directory
Jul 29 10:49:30 python3[2131]: delete element inet vrf_zones ct_iface_map { eth1 }
Jul 29 10:49:30 python3[2131]:                     ^^^^^^^^^
Jul 29 10:49:30 python3[2131]: cmd 'xdp_loader -d eth1 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth1'
Jul 29 10:49:30 python3[2131]: returned (out):
Jul 29 10:49:30 python3[2131]: returned (err):
Jul 29 10:49:30 libbpf[2131]: failed to create map (name: 'redirect_params'): Operation not permitted(-1)
Jul 29 10:49:30 libbpf[2131]: failed to load object '/usr/share/vyos/xdp/xdp_prog_kern.o'
Jul 29 10:49:30 ERR[2131]: loading BPF-OBJ file(/usr/share/vyos/xdp/xdp_prog_kern.o) (-22): Invalid argument
Jul 29 10:49:30 ERR[2131]: loading file: /usr/share/vyos/xdp/xdp_prog_kern.o
Jul 29 10:49:30 vyos-router[2027]: [[interfaces ethernet eth0]] failed
Jul 29 10:49:30 vyos-router[2027]: [ interfaces ethernet eth1 ]
Jul 29 10:49:30 vyos-router[2027]: VyOS had an issue completing a command.
Jul 29 10:49:30 vyos-router[2027]: We are sorry that you encountered a problem while using VyOS.
Jul 29 10:49:30 vyos-router[2027]: There are a few things you can do to help us (and yourself):
Jul 29 10:49:30 vyos-router[2027]: - Contact us using the online help desk if you have a subscription:
Jul 29 10:49:30 vyos-router[2027]:   https://support.vyos.io/
Jul 29 10:49:30 vyos-router[2027]: - Make sure you are running the latest version of VyOS available at:
Jul 29 10:49:30 vyos-router[2027]:   https://vyos.net/get/
Jul 29 10:49:30 vyos-router[2027]: - Consult the community forum to see how to handle this issue:
Jul 29 10:49:30 vyos-router[2027]:   https://forum.vyos.io
Jul 29 10:49:30 vyos-router[2027]: - Join us on Slack where our users exchange help and advice:
Jul 29 10:49:30 vyos-router[2027]:   https://vyos.slack.com
Jul 29 10:49:30 vyos-router[2027]: When reporting problems, please include as much information as possible:
Jul 29 10:49:30 vyos-router[2027]: - do not obfuscate any data (feel free to contact us privately if your
Jul 29 10:49:30 vyos-router[2027]:   business policy requires it)
Jul 29 10:49:30 vyos-router[2027]: - and include all the information presented below
Jul 29 10:49:30 vyos-router[2027]: Report time:      2022-07-29 10:49:30
Jul 29 10:49:30 vyos-router[2027]: Image version:    VyOS 1.4-rolling-202207270217
Jul 29 10:49:30 vyos-router[2027]: Release train:    sagitta
Jul 29 10:49:30 vyos-router[2027]: Built by:         autobuild@vyos.net
Jul 29 10:49:30 vyos-router[2027]: Built on:         Wed 27 Jul 2022 02:17 UTC
Jul 29 10:49:30 vyos-router[2027]: Build UUID:       db3bf91b-152e-4cae-9faf-f7176be287d4
Jul 29 10:49:30 vyos-router[2027]: Build commit ID:  3327957356be99
Jul 29 10:49:30 vyos-router[2027]: Architecture:     x86_64
Jul 29 10:49:30 vyos-router[2027]: Boot via:         installed image
Jul 29 10:49:30 vyos-router[2027]: System type:      KVM guest
Jul 29 10:49:30 vyos-router[2027]: Hardware vendor:  QEMU
Jul 29 10:49:30 vyos-router[2027]: Hardware model:   Standard PC (Q35 + ICH9, 2009)
Jul 29 10:49:30 vyos-router[2027]: Hardware S/N:
Jul 29 10:49:30 vyos-router[2027]: Hardware UUID:    88f32e10-cde2-4e63-bb89-f80e2d396e05
Jul 29 10:49:30 vyos-router[2027]: Traceback (most recent call last):
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 215, in <module>
Jul 29 10:49:30 vyos-router[2027]:     apply(c)
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 204, in apply
Jul 29 10:49:30 vyos-router[2027]:     e.update(ethernet)
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 363, in update
Jul 29 10:49:30 vyos-router[2027]:     super().update(config)
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1590, in update
Jul 29 10:49:30 vyos-router[2027]:     self.set_xdp('xdp' in config)
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1382, in set_xdp
Jul 29 10:49:30 vyos-router[2027]:     return self._cmd(cmd)
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
Jul 29 10:49:30 vyos-router[2027]:     return cmd(command, self.debug)
Jul 29 10:49:30 vyos-router[2027]:   File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
Jul 29 10:49:30 vyos-router[2027]:     raise OSError(code, feedback)
Jul 29 10:49:30 vyos-router[2027]: OSError: [Errno 40] failed to run command: xdp_loader -d eth1 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth1
Jul 29 10:49:30 vyos-router[2027]: returned:
Jul 29 10:49:30 vyos-router[2027]: exit code: 40
Jul 29 10:49:30 vyos-router[2027]: noteworthy:
Jul 29 10:49:30 vyos-router[2027]: cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth1" }'
Jul 29 10:49:30 vyos-router[2027]: returned (out):
Jul 29 10:49:30 vyos-router[2027]: returned (err):
Jul 29 10:49:30 vyos-router[2027]: Error: No such file or directory
Jul 29 10:49:30 vyos-router[2027]: delete element inet vrf_zones ct_iface_map { eth1 }
Jul 29 10:49:30 vyos-router[2027]:                     ^^^^^^^^^
Jul 29 10:49:30 vyos-router[2027]: cmd 'xdp_loader -d eth1 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth1'
Jul 29 10:49:30 vyos-router[2027]: returned (out):
Jul 29 10:49:30 vyos-router[2027]: returned (err):
Jul 29 10:49:30 vyos-router[2027]: libbpf: failed to create map (name: 'redirect_params'): Operation not permitted(-1)
Jul 29 10:49:30 vyos-router[2027]: libbpf: failed to load object '/usr/share/vyos/xdp/xdp_prog_kern.o'
Jul 29 10:49:30 vyos-router[2027]: ERR: loading BPF-OBJ file(/usr/share/vyos/xdp/xdp_prog_kern.o) (-22): Invalid argument
Jul 29 10:49:30 vyos-router[2027]: ERR: loading file: /usr/share/vyos/xdp/xdp_prog_kern.o
Jul 29 10:49:30 sudo[2102]: pam_unix(sudo:session): session closed for user root
Jul 29 10:49:30 systemd[1]: opt-vyatta-config-tmp-new_config_1908.mount: Succeeded.
Jul 29 10:49:30 vyos-router[2027]: [[interfaces ethernet eth1]] failed
Jul 29 10:49:30 vyos-router[2027]: Commit failed
Jul 29 10:49:30 systemd[1]: opt-vyatta-config-tmp-new_config_1908.mount: Succeeded.
Jul 29 10:49:39 systemd[1]: serial-getty@ttyS0.service: Succeeded.
Jul 29 10:49:39 systemd[1]: serial-getty@ttyS0.service: Scheduled restart job, restart counter is at 4.
Jul 29 10:49:39 systemd[1]: Stopped Serial Getty on ttyS0.
Jul 29 10:49:39 systemd[1]: Started Serial Getty on ttyS0.
Jul 29 10:49:39 agetty[2197]: /dev/ttyS0: not a tty
Jul 29 10:49:49 systemd[1]: serial-getty@ttyS0.service: Succeeded.
Jul 29 10:49:50 systemd[1]: serial-getty@ttyS0.service: Scheduled restart job, restart counter is at 5.
Jul 29 10:49:50 systemd[1]: Stopped Serial Getty on ttyS0.
Jul 29 10:49:50 systemd[1]: Started Serial Getty on ttyS0.
Jul 29 10:49:50 agetty[2201]: /dev/ttyS0: not a tty
Jul 29 10:50:00 systemd[1]: serial-getty@ttyS0.service: Succeeded.
Jul 29 10:50:00 systemd[1]: serial-getty@ttyS0.service: Scheduled restart job, restart counter is at 6.
Jul 29 10:50:00 systemd[1]: Stopped Serial Getty on ttyS0.
Jul 29 10:50:00 systemd[1]: Started Serial Getty on ttyS0.
Jul 29 10:50:00 agetty[2206]: /dev/ttyS0: not a tty
Jul 29 10:50:00 sshd[2202]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=5giss-chr-1  user=vyos
Jul 29 10:50:02 sshd[2202]: Failed password for vyos from 10.178.128.1 port 46364 ssh2
Jul 29 10:50:05 sshd[2202]: Accepted password for vyos from 10.178.128.1 port 46364 ssh2
Jul 29 10:50:05 sshd[2202]: pam_unix(sshd:session): session opened for user vyos(uid=1002) by (uid=0)
Jul 29 10:50:05 systemd[1]: Created slice User Slice of UID 1002.
Jul 29 10:50:05 systemd[1]: Starting User Runtime Directory /run/user/1002...
Jul 29 10:50:05 systemd-logind[868]: New session 1 of user vyos.
Jul 29 10:50:05 systemd[1]: Finished User Runtime Directory /run/user/1002.
Jul 29 10:50:05 systemd[1]: Starting User Manager for UID 1002...
Jul 29 10:50:05 systemd[2208]: pam_unix(systemd-user:session): session opened for user vyos(uid=1002) by (uid=0)
Jul 29 10:50:05 systemd[2208]: Queued start job for default target Main User Target.
Jul 29 10:50:05 systemd[2208]: Created slice User Application Slice.
Jul 29 10:50:05 systemd[2208]: Reached target Paths.
Jul 29 10:50:05 systemd[2208]: Reached target Timers.
Jul 29 10:50:05 systemd[2208]: Listening on GnuPG network certificate management daemon.
Jul 29 10:50:05 systemd[2208]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jul 29 10:50:05 systemd[2208]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jul 29 10:50:05 systemd[2208]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jul 29 10:50:05 systemd[2208]: Listening on GnuPG cryptographic agent and passphrase cache.
Jul 29 10:50:05 systemd[2208]: Listening on Podman API Socket.
Jul 29 10:50:05 systemd[2208]: Reached target Sockets.
Jul 29 10:50:05 systemd[2208]: Reached target Basic System.
Jul 29 10:50:05 systemd[2208]: Reached target Main User Target.
Jul 29 10:50:05 systemd[2208]: Startup finished in 204ms.
Jul 29 10:50:05 systemd[1]: Started User Manager for UID 1002.
Jul 29 10:50:05 systemd[1]: Started Session 1 of user vyos.
Jul 29 10:50:05 systemd[2208]: opt-vyatta-config-tmp-new_config_2224.mount: Succeeded.
Jul 29 10:50:05 systemd[1]: opt-vyatta-config-tmp-new_config_2224.mount: Succeeded.

What I don’t understand is that running the commands myself, or the script, will work.

As far as I’m aware, this seems to be a permission problem :

libbpf[2063]: failed to create map (name: 'redirect_params'): Operation not permitted(-1)

Seems the environment variables aren’t the same when in shell & when the post boot script runs :

1,2c1,3
< VYATTA_ACTIVE_CONFIGURATION_DIR=/opt/vyatta/config/active
< VYATTA_CONFIG_TMP=/opt/vyatta/config/tmp/tmp_1910
---
> CAML_LD_LIBRARY_PATH=/opt/opam/4.07.0/lib/stublibs:/opt/opam/4.07.0/lib/ocaml/stublibs:/opt/opam/4.07.0/lib/ocaml
> MANPATH=:/opt/opam/4.07.0/man
> XDG_SESSION_ID=1
4,5c5
< "'@><=;|&(
< TERM=linux
---
> "'><=;|&(
7a8,9
> TERM=xterm-256color
> SHELL=/bin/vbash
9,11c11,12
< VYATTA_COMP_LINE_EMPTY=VYATTA_COMP_LINE_EMPTY
< vyatta_htmldir=/opt/vyatta/share/man
< VYATTA_TEMPLATE_LEVEL=/
---
> vyatta_htmldir=/opt/vyatta/share/html
> SSH_CLIENT=10.178.128.1 47464 22
12a14
> OPAM_SWITCH_PREFIX=/opt/opam/4.07.0
13a16,19
> SSH_TTY=/dev/pts/0
> OPAMROOT=/opt/opam
> XDG_SESSION_CLASS=user
> OCAML_TOPLEVEL_PATH=/opt/opam/4.07.0/lib/toplevel
15c21,22
< INVOCATION_ID=60bccfe27dac40668708e72716ef9b79
---
> USER=vyos
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
18d24
< vyos_prefix=/opt/vyatta
19a26
> vyos_prefix=/opt/vyatta
21a29
> MOTD_SHOWN=pam
27,28c35,36
< PWD=/
< VYATTA_COMP_LINE=VYATTA_COMP_LINE_EMPTY
---
> PWD=/home/vyos
> XDG_SESSION_TYPE=tty
30,31d37
< VYATTA_CHANGES_ONLY_DIR=/opt/vyatta/config/tmp/changes_only_1910
< vyshim=/usr/sbin/vyshim
34d39
< PS1=[edit]\n\u@\H# 
38,39d42
< BOOTFILE=/opt/vyatta/etc/config/config.boot
< HISTCONTROL=
40a44
> HISTCONTROL=
42c46,47
< SHLVL=2
---
> SHLVL=1
> HOME=/home/vyos
44d48
< vyatta_libdir=/opt/vyatta/lib
46,48c50,51
< VYATTA_CONFIG_TEMPLATE=/opt/vyatta/share/vyatta-cfg/templates
< VYATTA_EDIT_LEVEL=/
< vyatta_localstatedir=/opt/vyatta/var
---
> vyatta_libdir=/opt/vyatta/lib
> LOGNAME=vyos
49a53,55
> vyatta_localstatedir=/opt/vyatta/var
> VYATTA_PAGER=less       --buffers=64    --auto-buffers  --no-lessopen           --QUIT-AT-EOF   --quit-if-one-screen    --RAW-CONTROL-CHARS     --squeeze-blank-lines   --no-init
> SSH_CONNECTION=10.178.128.1 47464 10.178.128.52 22
52,53d57
< PROMPT_COMMAND=history -a
< JOURNAL_STREAM=7:1258
54a59
> PROMPT_COMMAND=history -a
55a61
> XDG_RUNTIME_DIR=/run/user/1002
59d64
< VYATTA_TEMP_CONFIG_DIR=/opt/vyatta/config/tmp/new_config_1910

I think it may have to do with how the

Jul 29 10:49:29 sudo[2031]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c /usr/sbin/vyshim VYOS_TAGNODE_VALUE='eth0' /usr/libexec/vyos/conf_mode/interfaces-ethernet.py

is ran and with which context.