We are using VYOS 1.0.5 on our edge router with a pppoe (fiber) uplink. In order to do volume monitoring I have added flow-accounting
set system flow-accounting interface eth0.2
set system flow-accounting netflow server 10.10.50.101 port 2055
set system flow-accounting syslog-facility daemon
I have set up nfsen as the collector on 10.10.50.101 and I am able to see flows. But only the flows that go from our network to the outside, I am not seeing the return flows.
I have also checked with the flow accounting tools on Vyos and running
show flow-accounting interface eth0.2 host 10.10.50.6
Src Addr Dst Addr Sport Dport Proto Packets Bytes Flows
10.10.50.6 1.2.3.4 42121 443 tcp 109 25187 13
10.10.50.6 1.2.3.5 53700 443 tcp 52 19436 4
10.10.50.6 1.2.3.6 42124 443 tcp 86 16442 13
10.10.50.6 1.2.3.7 44172 443 tcp 60 15067 8
...
SNIP
...
Total entries: 24
Total flows : 81
Total pkts : 719
Total bytes : 138,907
gives only Src Addr FROM the host, never the return flow.
I have added the external pppoe interface and I then get all the incoming flows, but only with Dst Addr of the router and they are not “connected” to the internal host as we are doing masquerading
Following the flow in nfsen
Yes, Vyos only does flow accounting on the packets inbound to an interface, and in the case of NAT, it shows the pre-nat destination, which is an address on your vyos box. You need to correlate the packets inbound on your internal interface (pre-nat they show your internal addresses) with the packets inbound on your external interface (they at least show the external source and port). Matching the external source ip and port number in those two flows, you can re-create the internal destination address.
The old vyatta forums had a patch to allow post-nat flow accounting, but I cannot find that now.
Ubiquity has both ingress and egress flow accounting, which is probably a better way to go. If someone can get their version of /opt/vyatta/sbin/vyatta-netflow.pl (I presume that is still GPL code), I could try to integrate that in Vyos.
thank you for the fast response. If we don’t want to script something up that connects the two flows, we will need egress netflows from our internal interface.
I think this is the cleaner way to do it and I will have a look at pmacct myself, but might need help to integrate it to vyos. Will setup a test environment and keep the forum posted. If you do succeed to acquire/integrate the vyatta-netflow.pl, do let me know.
UniFi - Rethinking IT - Ubiquiti - the “gpl archive” link. It looks like the changes are in the vyatta-netflow package, but ubiquiti has an older version of pmacct than vyos, combined with a lot more “set system flow-accounting …” config options. We cannot just take the current ubiquiti config stuff, since there are vyatta/vyos changes since that split that we probably want to preserve.
Thank you for the link. I downloaded the gpl-archive of the EdgeMax. (The download is 530MB not 70MB).
In the vyatta-netflow_0.34.4.tgz package the commands were in the scripts/vyatta-netflow.pl . Combining the tables from “sub acct_get_table_chain” with the commands in “sub acct_add_ulog_target” I was able to extract:
iptables -t filter -I VYATTA_POST_FW_IN_HOOK 1 -o eth0.2 -j ULOG --ulog-nlgroup 2 --ulog-cprange 64 --ulog-qthreshold 10
iptables -t filter -I VYATTA_POST_FW_FWD_HOOK 1 -o eth0.2 -j ULOG --ulog-nlgroup 2 --ulog-cprange 64 --ulog-qthreshold 10
iptables -t filter -I VYATTA_POST_FW_OUT_HOOK 1 -o eth0.2 -j ULOG --ulog-nlgroup 2 --ulog-cprange 64 --ulog-qthreshold 10
In my case eth0.2 is the inward facing interface. Success!! Now egress netflows are displayed in my netflow collector/analyzer.
How can this be integrated into VyOS so that it is possible to do it via the standard configuration procedure?
I am ready to help, just don’t know where to start.
You might look at the ubiquity templates as a starting point - they should be in the vyatta-netflow package in the templates directory. Those all get installed into /opt/vyatta/share/vyatta* directories. And that perl script gets installed at /opt/vyatta/sbin/vyatta-netflow.pl.
The ubiquity vyatta-netflow package needs to be merged with the vyos vyatta-netflow package, such that our package has modified scripts and template files. It would be nice to use the same commands as ubiquity.
Thanks for publishing this in the forums. I’ve successfully used this for egress netflow on two internal interfaces. Here is what I did (although getting built-in support for this would be good).
Add to /opt/vyatta/etc/config/scripts/vyatta-postconfig-bootup.script :
_NF_EGRESS_IFACES="eth0 eth2"
_NF_EGRESS_CHAINS="VYATTA_POST_FW_IN_HOOK VYATTA_POST_FW_FWD_HOOK VYATTA_POST_FW_OUT_HOOK"
for IFACE in $_NF_EGRESS_IFACES; do
for CHAIN in $_NF_EGRESS_CHAINS; do
/sbin/iptables -t filter -I $CHAIN 1 -o $IFACE -j ULOG --ulog-nlgroup 2 --ulog-cprange 64 --ulog-qthreshold 10
done
done
That’s quite a big statement How do you come to the conclusion that it is “almost unusable” ? A little detail helps in working out what is missing so developers can improve things…
First i must say i like vyos very much and am gratefull to developers.
Netflow in its current form is only thing preventing me from switching to vyos.
Now, the answer.
“Unusable” because there is no way to do it with configure commands, it reverts to original state after every system update and state is not reflected in config.boot.
“Almost” because workaround exists.
If you configure netflow export like on any other router you will not get what you expect.
There is no egress accounting and ingres capture defaults to pre-dnat which is little problematic on routers that do NAT.
Ubiquity fixed this with
set system flow-accounting ingress-capture [pre-dnat | post-dnat]
set system flow-accounting netflow enable-egress
and i hoped it would trickle down to vyos.
Fair enough, I agree the workaround isn’t ideal. But the workaround should persist between system updates. /etc/rc.local specifically states:
# Do not remove the following call to vyatta-postconfig-bootup.script.
# Any boot time workarounds should be put in script below so that they
# get preserved for the new image during image upgrade.
POSTCONFIG=/opt/vyatta/etc/config/scripts/vyatta-postconfig-bootup.script
So if you place the workaround in vyatta-postconfig-bootup.script, it will persist between updates. It won’t persist between hardware or virtual installations, but it’s only a few line hack to reinstate if you change server.
The ingress/egress on inside interface(s) would capture any flows post-dnat though right? In Cisco land, I don’t ever recall a way of recording netflow on the outside interface with post-dnat (for example). The workaround was always to capture on inside interfaces.
This would be nice to have. Somebody in the VyOS community pointed me to the Ubiquiti source tarball. If I get a chance, I’ll see what is required.
How do you come to the conclusion that it is “almost unusable” ? A little detail helps in working out what is missing so developers can improve things…