sFlow/flow accounting broken in 1.4 rolling

runbgp · March 7, 2023, 4:56am

Sflow appears to be completely broken in 1.4 rolling.

Similar posts:

Version:          VyOS 1.4-rolling-202303040317
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Sat 04 Mar 2023 03:17 UTC
Build UUID:       642a0f97-d078-40b8-915f-6d988a851ce9
Build commit ID:  afdaac1012d7b1

Architecture:     x86_64
Boot via:         installed image
System type:       guest

Hardware vendor:  Supermicro
Hardware model:   Super Server
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-ac1f6b79a20e

Copyright:        VyOS maintainers and contributors

Mar  6 22:50:20 edge1 kernel: [256902.519785] uacctd[43221]: segfault at 6 ip 00007f056345b6c0 sp 00007ffc2ef5baa8 error 4 in libc.so.6[7f056332f000+155000] likely on CPU 15 (core 28, socket 0)
Mar  6 22:50:20 edge1 kernel: [256902.519803] Code: 4c 16 fc 8b 36 89 4c 17 fc 89 37 c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 66 0f 1f 84 00 00 00 00 00 <48> 8b 4c 16 f8 48 8b 36 48 89 37 48 89 4c 17 f8 c3 c5 fe 6f 54 16
Mar  6 22:50:20 edge1 systemd[1]: uacctd.service: Main process exited, code=killed, status=11/SEGV
Mar  6 22:50:20 edge1 systemd[1]: uacctd.service: Failed with result 'signal'.
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Scheduled restart job, restart counter is at 38.
Mar  6 22:50:30 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:50:30 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:50:30 edge1 systemd[1]: Started uacctd.service - ulog accounting daemon.
Mar  6 22:50:30 edge1 kernel: [256912.762117] uacctd[43225]: segfault at 6 ip 00007f44557106c0 sp 00007ffdf280cf58 error 4 in libc.so.6[7f44555e4000+155000] likely on CPU 15 (core 28, socket 0)
Mar  6 22:50:30 edge1 kernel: [256912.762136] Code: 4c 16 fc 8b 36 89 4c 17 fc 89 37 c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 66 0f 1f 84 00 00 00 00 00 <48> 8b 4c 16 f8 48 8b 36 48 89 37 48 89 4c 17 f8 c3 c5 fe 6f 54 16
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Main process exited, code=killed, status=11/SEGV
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Failed with result 'signal'.
Mar  6 22:50:40 edge1 systemd[1]: uacctd.service: Scheduled restart job, restart counter is at 39.
Mar  6 22:50:40 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:50:40 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:50:40 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:52:11 edge1 systemd[1]: uacctd.service: start operation timed out. Terminating.
Mar  6 22:52:11 edge1 systemd[1]: uacctd.service: Failed with result 'timeout'.
Mar  6 22:52:11 edge1 systemd[1]: Failed to start uacctd.service - ulog accounting daemon.
Mar  6 22:52:21 edge1 systemd[1]: uacctd.service: Scheduled restart job, restart counter is at 40.
Mar  6 22:52:21 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:52:21 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:52:21 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:53:08 edge1 vyos-configd[921]: Received message: {"type": "init"}
Mar  6 22:53:09 edge1 vyos-configd[921]: config session pid is 40097
Mar  6 22:53:09 edge1 vyos-configd[921]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/flow_accounting_conf.py"}
Mar  6 22:53:09 edge1 systemd[1]: Reloading.
Mar  6 22:53:10 edge1 systemd[1]: uacctd.service: Deactivated successfully.
Mar  6 22:53:10 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:53:10 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:53:10 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:53:10 edge1 systemd[1]: Started uacctd.service - ulog accounting daemon.
Mar  6 22:53:10 edge1 vyos-configd[921]: Sending response 1
Mar  6 22:53:11 edge1 systemd[1]: opt-vyatta-config-tmp-new_config_40097.mount: Deactivated successfully.
Mar  6 22:53:14 edge1 commit: Successful change to active configuration by user vyos on /dev/pts/0

Setting set system flow-accounting disable-imt gets us a little further, but no dice.

vyos@edge1t# run show flow-accounting interface eth2
Traceback (most recent call last):
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 249, in <module>
    flows_list = _get_flows_list()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 99, in _get_flows_list
    out = cmd(f'/usr/bin/pmacct -s -O json -T flows -p {uacctd_pipefile}',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
PermissionError: [Errno 1] Failed to get flows list
failed to run command: /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
returned: INFO: Connection refused while trying to connect to '/tmp/uacctd.pipe'
exit code: 1
[edit]

kyle · March 7, 2023, 5:12am

Can you please provide the flow-accounting section of your config?

fernando · March 7, 2023, 12:12pm

I’ve checked it with the flowing configuration , I didn’t see any problem:

vyos@vyos# run show configuration commands  | match flow
set system flow-accounting interface 'eth0'
set system flow-accounting sflow agent-address '172.16.50.44'
set system flow-accounting sflow sampling-rate '36'
set system flow-accounting sflow server 172.16.50.1 port '20255'

``
cloud you share your current configuration ?

runbgp · March 7, 2023, 3:25pm

I am testing this on a brand new lab VM now on VyOS 1.4-rolling-202303071039 and am seeing mixed results - interestingly, it seems to work briefly, then stops.

vyos@vyos# run show configuration commands  | match flow
set system flow-accounting interface 'eth0'
set system flow-accounting sflow agent-address '10.1.1.59'
set system flow-accounting sflow sampling-rate '5000'
set system flow-accounting sflow server 10.0.10.18
set system flow-accounting sflow source-address '10.1.1.59'

vyos@vyos# run show flow-accounting interface eth0
IN_IFACE    SRC_MAC            DST_MAC            SRC_IP                     DST_IP             SRC_PORT    DST_PORT  PROTOCOL      TOS    PACKETS    FLOWS    BYTES
----------  -----------------  -----------------  -------------------------  ---------------  ----------  ----------  ----------  -----  ---------  -------  -------
eth0        f4:d4:88:9a:4b:c7  01:00:5e:7f:ff:fa  10.1.1.51                  239.255.255.250       55864        1900  udp             0          4        1      816
eth0        f4:d4:88:9a:4b:c7  33:33:00:00:00:fb  fe80::490:25d5:8fd7:f3ca   ff02::fb               5353        5353  udp             0          7        1     3372
eth0        4c:c9:5e:77:3b:da  01:00:5e:7f:ff:fa  10.1.1.240                 239.255.255.250       38661       15600  udp             0          1        1       63
[edit]
vyos@vyos# run show flow-accounting interface eth0
flow-accounting is not active
[edit]

/var/log/messages

Mar  7 15:22:18 vyos kernel: [  211.623521] uacctd[3084]: segfault at 6 ip 00007fcba1ba66c0 sp 00007ffc8c4cfe78 error 4 in libc.so.6[7fcba1a7a000+155000] likely on CPU 1 (core 1, socket 0)
Mar  7 15:22:18 vyos kernel: [  211.623537] Code: 4c 16 fc 8b 36 89 4c 17 fc 89 37 c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 66 0f 1f 84 00 00 00 00 00 <48> 8b 4c 16 f8 48 8b 36 48 89 37 48 89 4c 17 f8 c3 c5 fe 6f 54 16
Mar  7 15:22:18 vyos systemd[1]: uacctd.service: Main process exited, code=killed, status=11/SEGV
Mar  7 15:22:18 vyos systemd[1]: uacctd.service: Failed with result 'signal'.

Running sudo /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe manually, similar behavior - works momentarily.

vyos@vyos# sudo /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
{"mac_src": "70:b3:06:08:39:f3", "mac_dst": "33:33:00:00:00:16", "vlan": 0, "iface_in": 2, "ip_src": "fe80::14b5:e8de:51e2:e588", "ip_dst": "ff02::16", "port_src": 0, "port_dst": 0, "ip_proto": "ipv6-icmp", "tos": 0, "packets": 2, "flows": 1, "bytes": 152}
{"mac_src": "d8:eb:46:b7:dc:c9", "mac_dst": "33:33:00:00:00:fb", "vlan": 0, "iface_in": 2, "ip_src": "fe80::daeb:46ff:feb7:dcc9", "ip_dst": "ff02::fb", "port_src": 5353, "port_dst": 5353, "ip_proto": "udp", "tos": 0, "packets": 1, "flows": 1, "bytes": 102}
[edit]
vyos@vyos# sudo /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
INFO: Connection refused while trying to connect to '/tmp/uacctd.pipe'

sflow · March 11, 2023, 9:38pm

This question prompted me to take a look at VyOS sFlow, see VyOS blog post. Installing the Host sFlow agent works well in a VirtualBox test network.

Viacheslav · March 14, 2023, 11:02am

@sflow I created a feature request T5086
Thanks

Scarecrow777 · March 19, 2023, 8:24am

So was the error actually fixed or still ongoing? I’m having the same issue on 1.4 rolling.

Viacheslav · March 19, 2023, 9:06am

We implemented new sflow
Try it

set system sflow agent-address ‘192.168.122.14’
set system sflow interface ‘eth0’
set system sflow interface ‘eth1’
set system sflow polling ‘30’
set system sflow sampling-rate ‘1000’
set system sflow server 192.168.122.1 port ‘6343’
set system sflow server 192.168.122.11 port ‘6343’

Scarecrow777 · March 19, 2023, 9:40am

Hey,

So i upgraded to the latest nightly then tried it. If i have IMT enabled i get

show flow-accounting 
flow-accounting is not active

but if i disable it, i get

show flow-accounting 
Traceback (most recent call last):
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 249, in <module>
    flows_list = _get_flows_list()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 99, in _get_flows_list
    out = cmd(f'/usr/bin/pmacct -s -O json -T flows -p {uacctd_pipefile}',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
PermissionError: [Errno 1] Failed to get flows list
failed to run command: /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
returned: INFO: Connection refused while trying to connect to '/tmp/uacctd.pipe'
exit code: 1

Viacheslav · March 19, 2023, 9:49am

There are no op-mode commands for this feature yet
It is a different daemon “hsflowd”
The old implementation uses “system flow-accounting sflow” (pmacct) and op mode commands for old implementation, as it is different daemon you get error that service is not configured
Delete completely “system flow-accounting” and use “system sflow” only

You can check that flows are exported with “sudo tcpdump -ni ethX port 6343”
Or check flows on the remote server on which you receive the flows

WebToor · May 29, 2023, 9:02am

for my part flow-accounting is configured, but any traffic on the interface out.
Which configured on the agent-adress+sources-address options configuration.

Viacheslav · May 29, 2023, 9:23am

You can use another sFlow, instead of old uacctd
set system sflow
there is an example sFlow — VyOS 1.4.x (sagitta) documentation

WebToor · May 29, 2023, 9:25am

Not available on my system =

So need to upgrade OS Version,…