Proper way to count traffic per interface per IP family for prometheus metric export

General questions
1

Hello,

we’re running a custom build of VyOS including nft-exporter to get (prometheus) metrics for nft firewall rules. As we were interested in leveraging that exporter to peek at total amounts of traffic per interface per address family, we introduced a custom nft ruleset with counters for each ethernet and wireguard interface for the input, output and forward chains. Unfortunately those counters do not appear to properly count all traffic flowing over an interface. We’re aware there are also prerouting chains and flowtables having the potential to hide traffic, but we neither have flowtables configured and at least for pure IPv6 traffic we don’t have any prerouting rules. As we’re running out of ideas what could cause nftables to miss large portions of the traffic, we’d like to ask for input here in the forums.

I can provide config details if needed, I was just unsure which might be relevant and didn’t want to clutter the initial post right away.

Best regards

2

Cant you just enable the builtin snmp-server (make it v2c or v3) and just pick up the traffic per interface through that?

3

From what I understand of the question, that only counts bytes and is not granular enough,

If you want to have separate counters for IPv4 and IPv6 traffic, you need to setup something that counts them seperately (for example nftables rules) and report them seperately (won’t know how, don’t use prometheus).

You could for example do something with nftables counters: Counters - nftables wiki

4

That’s not granular enough, as WanWizard pointed out. We explicitly want to count traffic by address family to get a grasp on the actual distribution (and prove some doubters in the company going IPv6 was actually a good choice).

That’s exactly what we are doing right now, I might have been too vague in my initial post. However, that way we’re apparently missing out some traffic and are unable to figure out why. The counters are set up for input/output/forward chains, but at least for IPv6 forwarded traffic some large portions of traffic do not appear/are not counted there. We know that, as e.g. a frequently occurring backup task streaming data to an external server does consistently cause the traffic graph for the host it originates from to show 1Gbit/s of traffic, while the nft counters on the router do not bodge. We are out of ideas what could cause this, as our pre-routing rules do not affect that traffic and we don’t have flowtables configured.

Could you try to reproduce on a test machine? A simple routed file transfer should suffice to verify the counter not going up.

5

The best place to add nftables rules for hit counters is in a netdev hook. This is hit before all other hooks like raw/prerouting/input/output/forwarding. You can use something like this for the rules, just modify the interfaces for your needs. Don’t forget to update the interfaces in both the sets and the hook declarations:

table netdev iface_stats {
    set ingress {
        typeof meta protocol . meta iifname
        counter
        elements = {
            ip  . "eth0",
            ip6 . "eth0",
            ip  . "eth1",
            ip6 . "eth1"
        }
    }

    set egress {
        typeof meta protocol . meta oifname
        counter
        elements = {
            ip  . "eth0",
            ip6 . "eth0",
            ip  . "eth1",
            ip6 . "eth1"
        }
    }

    chain ingress {
        type filter hook ingress devices = { "eth0", "eth1" } priority filter; policy accept;

        meta protocol . meta iifname @ingress
    }

    chain egress {
        type filter hook egress devices = { "eth0", "eth1" } priority filter; policy accept;

        meta protocol . meta oifname @egress
    }
}

You can view the stats using this:

nft list sets netdev

You can also use a python script like this to check stats locally:

#!/usr/bin/env python3

import json

from tabulate import tabulate
from vyos.utils.process import cmd

def get_iface_stats() -> str:
    stats = {}

    for direction in ("ingress", "egress"):
        data = json.loads(
            cmd(f"nft -j list set netdev iface_stats {direction}")
        )

        for item in data["nftables"]:
            if "set" not in item:
                continue

            for entry in item["set"].get("elem", []):
                elem = entry["elem"]
                af, interface = elem["val"]["concat"]
                counter = elem["counter"]

                stats.setdefault(interface, {}).setdefault(af, {})[direction] = counter

    rows = []

    for interface, families in stats.items():
        for af in ("ip", "ip6"):
            if af not in families:
                continue

            ingress = families[af].get("ingress", {})
            egress = families[af].get("egress", {})

            rows.append([
                interface,
                "IPv4" if af == "ip" else "IPv6",
                ingress.get("packets", 0),
                ingress.get("bytes", 0),
                egress.get("packets", 0),
                egress.get("bytes", 0),
            ])

    return tabulate(
        rows,
        headers=[
            "Interface",
            "Address Family",
            "Pkts In",
            "Bytes In",
            "Pkts Out",
            "Bytes Out",
        ],
        tablefmt="simple",
        numalign="right",
    )


if __name__ == "__main__":
    print(get_iface_stats())

Output should look something like this:

python3 test.py 
Interface    Address Family      Pkts In    Bytes In    Pkts Out    Bytes Out
-----------  ----------------  ---------  ----------  ----------  -----------
eth0         IPv4                  45260    12054809       52423     17216053
eth0         IPv6                      2         128           0            0
eth1         IPv4                  58346    17385847       47285     12886587
eth1         IPv6                   2619      556135           0            0
6

That is an interesting approach I will have a look at later this week or early next week. The important bits for me are that nft exporter should be able to pick this up and export it accordingly. I’d really like to avoid having to implement yet another Prometheus exporter just for these stats. Will report back when I got to test it!

7

Somewhere I have some configexample of how to create a custom OID and use SNMP to fetch the data a script will output if that can be of interest?

8

Here is what I was thinking of:

And it seems that VyOS already have this covered :slight_smile: