sFlow/flow accounting broken in 1.4 rolling

Sflow appears to be completely broken in 1.4 rolling.

Similar posts:

https://vyos.dev/T4098

Version:          VyOS 1.4-rolling-202303040317
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Sat 04 Mar 2023 03:17 UTC
Build UUID:       642a0f97-d078-40b8-915f-6d988a851ce9
Build commit ID:  afdaac1012d7b1

Architecture:     x86_64
Boot via:         installed image
System type:       guest

Hardware vendor:  Supermicro
Hardware model:   Super Server
Hardware S/N:     0123456789
Hardware UUID:    00000000-0000-0000-0000-ac1f6b79a20e

Copyright:        VyOS maintainers and contributors
Mar  6 22:50:20 edge1 kernel: [256902.519785] uacctd[43221]: segfault at 6 ip 00007f056345b6c0 sp 00007ffc2ef5baa8 error 4 in libc.so.6[7f056332f000+155000] likely on CPU 15 (core 28, socket 0)
Mar  6 22:50:20 edge1 kernel: [256902.519803] Code: 4c 16 fc 8b 36 89 4c 17 fc 89 37 c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 66 0f 1f 84 00 00 00 00 00 <48> 8b 4c 16 f8 48 8b 36 48 89 37 48 89 4c 17 f8 c3 c5 fe 6f 54 16
Mar  6 22:50:20 edge1 systemd[1]: uacctd.service: Main process exited, code=killed, status=11/SEGV
Mar  6 22:50:20 edge1 systemd[1]: uacctd.service: Failed with result 'signal'.
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Scheduled restart job, restart counter is at 38.
Mar  6 22:50:30 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:50:30 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:50:30 edge1 systemd[1]: Started uacctd.service - ulog accounting daemon.
Mar  6 22:50:30 edge1 kernel: [256912.762117] uacctd[43225]: segfault at 6 ip 00007f44557106c0 sp 00007ffdf280cf58 error 4 in libc.so.6[7f44555e4000+155000] likely on CPU 15 (core 28, socket 0)
Mar  6 22:50:30 edge1 kernel: [256912.762136] Code: 4c 16 fc 8b 36 89 4c 17 fc 89 37 c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 66 0f 1f 84 00 00 00 00 00 <48> 8b 4c 16 f8 48 8b 36 48 89 37 48 89 4c 17 f8 c3 c5 fe 6f 54 16
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Main process exited, code=killed, status=11/SEGV
Mar  6 22:50:30 edge1 systemd[1]: uacctd.service: Failed with result 'signal'.
Mar  6 22:50:40 edge1 systemd[1]: uacctd.service: Scheduled restart job, restart counter is at 39.
Mar  6 22:50:40 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:50:40 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:50:40 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:52:11 edge1 systemd[1]: uacctd.service: start operation timed out. Terminating.
Mar  6 22:52:11 edge1 systemd[1]: uacctd.service: Failed with result 'timeout'.
Mar  6 22:52:11 edge1 systemd[1]: Failed to start uacctd.service - ulog accounting daemon.
Mar  6 22:52:21 edge1 systemd[1]: uacctd.service: Scheduled restart job, restart counter is at 40.
Mar  6 22:52:21 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:52:21 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:52:21 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:53:08 edge1 vyos-configd[921]: Received message: {"type": "init"}
Mar  6 22:53:09 edge1 vyos-configd[921]: config session pid is 40097
Mar  6 22:53:09 edge1 vyos-configd[921]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/flow_accounting_conf.py"}
Mar  6 22:53:09 edge1 systemd[1]: Reloading.
Mar  6 22:53:10 edge1 systemd[1]: uacctd.service: Deactivated successfully.
Mar  6 22:53:10 edge1 systemd[1]: Stopped uacctd.service - ulog accounting daemon.
Mar  6 22:53:10 edge1 systemd[1]: Starting uacctd.service - ulog accounting daemon...
Mar  6 22:53:10 edge1 systemd[1]: uacctd.service: Can't open PID file /run/pmacct/uacctd.pid (yet?) after start: Operation not permitted
Mar  6 22:53:10 edge1 systemd[1]: Started uacctd.service - ulog accounting daemon.
Mar  6 22:53:10 edge1 vyos-configd[921]: Sending response 1
Mar  6 22:53:11 edge1 systemd[1]: opt-vyatta-config-tmp-new_config_40097.mount: Deactivated successfully.
Mar  6 22:53:14 edge1 commit: Successful change to active configuration by user vyos on /dev/pts/0

Setting set system flow-accounting disable-imt gets us a little further, but no dice.

vyos@edge1t# run show flow-accounting interface eth2
Traceback (most recent call last):
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 249, in <module>
    flows_list = _get_flows_list()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 99, in _get_flows_list
    out = cmd(f'/usr/bin/pmacct -s -O json -T flows -p {uacctd_pipefile}',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
PermissionError: [Errno 1] Failed to get flows list
failed to run command: /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
returned: INFO: Connection refused while trying to connect to '/tmp/uacctd.pipe'
exit code: 1
[edit]

Can you please provide the flow-accounting section of your config?

1 Like

I’ve checked it with the flowing configuration , I didn’t see any problem:

vyos@vyos# run show configuration commands  | match flow
set system flow-accounting interface 'eth0'
set system flow-accounting sflow agent-address '172.16.50.44'
set system flow-accounting sflow sampling-rate '36'
set system flow-accounting sflow server 172.16.50.1 port '20255'

``
cloud you share your current configuration ?

I am testing this on a brand new lab VM now on VyOS 1.4-rolling-202303071039 and am seeing mixed results - interestingly, it seems to work briefly, then stops.

vyos@vyos# run show configuration commands  | match flow
set system flow-accounting interface 'eth0'
set system flow-accounting sflow agent-address '10.1.1.59'
set system flow-accounting sflow sampling-rate '5000'
set system flow-accounting sflow server 10.0.10.18
set system flow-accounting sflow source-address '10.1.1.59'
vyos@vyos# run show flow-accounting interface eth0
IN_IFACE    SRC_MAC            DST_MAC            SRC_IP                     DST_IP             SRC_PORT    DST_PORT  PROTOCOL      TOS    PACKETS    FLOWS    BYTES
----------  -----------------  -----------------  -------------------------  ---------------  ----------  ----------  ----------  -----  ---------  -------  -------
eth0        f4:d4:88:9a:4b:c7  01:00:5e:7f:ff:fa  10.1.1.51                  239.255.255.250       55864        1900  udp             0          4        1      816
eth0        f4:d4:88:9a:4b:c7  33:33:00:00:00:fb  fe80::490:25d5:8fd7:f3ca   ff02::fb               5353        5353  udp             0          7        1     3372
eth0        4c:c9:5e:77:3b:da  01:00:5e:7f:ff:fa  10.1.1.240                 239.255.255.250       38661       15600  udp             0          1        1       63
[edit]
vyos@vyos# run show flow-accounting interface eth0
flow-accounting is not active
[edit]

/var/log/messages

Mar  7 15:22:18 vyos kernel: [  211.623521] uacctd[3084]: segfault at 6 ip 00007fcba1ba66c0 sp 00007ffc8c4cfe78 error 4 in libc.so.6[7fcba1a7a000+155000] likely on CPU 1 (core 1, socket 0)
Mar  7 15:22:18 vyos kernel: [  211.623537] Code: 4c 16 fc 8b 36 89 4c 17 fc 89 37 c3 c5 fa 6f 06 c5 fa 6f 4c 16 f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 66 0f 1f 84 00 00 00 00 00 <48> 8b 4c 16 f8 48 8b 36 48 89 37 48 89 4c 17 f8 c3 c5 fe 6f 54 16
Mar  7 15:22:18 vyos systemd[1]: uacctd.service: Main process exited, code=killed, status=11/SEGV
Mar  7 15:22:18 vyos systemd[1]: uacctd.service: Failed with result 'signal'.

Running sudo /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe manually, similar behavior - works momentarily.

vyos@vyos# sudo /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
{"mac_src": "70:b3:06:08:39:f3", "mac_dst": "33:33:00:00:00:16", "vlan": 0, "iface_in": 2, "ip_src": "fe80::14b5:e8de:51e2:e588", "ip_dst": "ff02::16", "port_src": 0, "port_dst": 0, "ip_proto": "ipv6-icmp", "tos": 0, "packets": 2, "flows": 1, "bytes": 152}
{"mac_src": "d8:eb:46:b7:dc:c9", "mac_dst": "33:33:00:00:00:fb", "vlan": 0, "iface_in": 2, "ip_src": "fe80::daeb:46ff:feb7:dcc9", "ip_dst": "ff02::fb", "port_src": 5353, "port_dst": 5353, "ip_proto": "udp", "tos": 0, "packets": 1, "flows": 1, "bytes": 102}
[edit]
vyos@vyos# sudo /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
INFO: Connection refused while trying to connect to '/tmp/uacctd.pipe'

This question prompted me to take a look at VyOS sFlow, see VyOS blog post. Installing the Host sFlow agent works well in a VirtualBox test network.

1 Like

@sflow I created a feature request T5086
Thanks

So was the error actually fixed or still ongoing? I’m having the same issue on 1.4 rolling.

We implemented new sflow
Try it

set system sflow agent-address ‘192.168.122.14’
set system sflow interface ‘eth0’
set system sflow interface ‘eth1’
set system sflow polling ‘30’
set system sflow sampling-rate ‘1000’
set system sflow server 192.168.122.1 port ‘6343’
set system sflow server 192.168.122.11 port ‘6343’

Hey,

So i upgraded to the latest nightly then tried it. If i have IMT enabled i get

show flow-accounting 
flow-accounting is not active

but if i disable it, i get

show flow-accounting 
Traceback (most recent call last):
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 249, in <module>
    flows_list = _get_flows_list()
                 ^^^^^^^^^^^^^^^^^
  File "/usr/libexec/vyos/op_mode/flow_accounting_op.py", line 99, in _get_flows_list
    out = cmd(f'/usr/bin/pmacct -s -O json -T flows -p {uacctd_pipefile}',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
PermissionError: [Errno 1] Failed to get flows list
failed to run command: /usr/bin/pmacct -s -O json -T flows -p /tmp/uacctd.pipe
returned: INFO: Connection refused while trying to connect to '/tmp/uacctd.pipe'
exit code: 1

There are no op-mode commands for this feature yet
It is a different daemon “hsflowd”
The old implementation uses “system flow-accounting sflow” (pmacct) and op mode commands for old implementation, as it is different daemon you get error that service is not configured
Delete completely “system flow-accounting” and use “system sflow” only

You can check that flows are exported with “sudo tcpdump -ni ethX port 6343”
Or check flows on the remote server on which you receive the flows

for my part flow-accounting is configured, but any traffic on the interface out.
Which configured on the agent-adress+sources-address options configuration.


You can use another sFlow, instead of old uacctd
set system sflow
there is an example sFlow — VyOS 1.4.x (sagitta) documentation

Not available on my system =

So need to upgrade OS Version,…

ON 1.4 it’s seems be a bug when you configure sampling-rating & max options

Now it works !

Tell us more about bug you mentioned

about :

Version:          VyOS 1.4-rolling-202307271350
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Thu 27 Jul 2023 13:50 UTC
Build UUID:       2c0718fb-8ddb-4a28-950a-06c6fb3818af
Build commit ID:  b298605464b759

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  To be filled by O.E.M.
Hardware model:   To be filled by O.E.M.
Hardware S/N:     To be filled by O.E.M.
Hardware UUID:    03000200-0400-0500-0006-000700080009

Copyright:        VyOS maintainers and contributors

When i disconfigured
image

I get back data to netflow server nfcapd, but here it 's concerning netflow not sflow !

I prefered use netflow instead sflow.

Provide he set of commands to reproduce

image