Incorrect flow-accounting on subinterfaces

sergy.silk · May 24, 2019, 6:36am

I have problem with flow-accounting data on subinterface: all information except of destination address is wrong.

$ sh flow-accounting interface eth1.334
flow-accounting for [eth1.334]
Src Addr        Dst Addr        Sport Dport Proto    Packets      Bytes   Flows
64.6.70.15      172.16.0.100    0     0       194          1         63       1
64.6.70.21      172.16.0.100    0     0       188          1         63       1
64.6.70.13      172.16.0.100    0     0       196          1         63       1
64.6.70.9       172.16.0.100    0     0       200          1         63       1
64.6.70.19      172.16.0.100    0     0       190          1         63       1
64.6.70.11      172.16.0.100    0     0       198          1         63       1
64.6.70.17      172.16.0.100    0     0       192          1         63       1
64.6.70.23      172.16.0.100    0     0       186          1         63       1
64.6.70.9       172.16.0.100    0     0       202          1         61       1
64.6.70.37      172.16.0.100    0     0       195          1         40       1
64.6.70.43      172.16.0.100    0     0       189          1         40       1
64.6.70.29      172.16.0.100    0     0       203          1         40       1
64.6.70.35      172.16.0.100    0     0       197          1         40       1
64.6.70.41      172.16.0.100    0     0       191          1         40       1
64.6.70.33      172.16.0.100    0     0       199          1         40       1
64.6.70.39      172.16.0.100    0     0       193          1         40       1
64.6.70.31      172.16.0.100    0     0       201          1         40       1
64.6.70.45      172.16.0.100    0     0       187          1         40       1

$ sh nat source translations address 172.16.0.100
Pre-NAT src          Pre-NAT dst        Post-NAT src         Post-NAT dst
172.16.0.100:5896    192.70.196.45:2061 xxx.yyy.223.194:5896 192.70.196.45:2061
  tcp: 172.16.0.100 ==> xxx.yyy.223.194  timeout: 299 use: 1

$ monitor traffic interface eth1.334 filter "host 172.16.0.100"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1.334, link-type EN10MB (Ethernet), capture size 262144 bytes
09:31:11.749977 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [P.], seq 157720286:157720313, ack 2053498746, win 32120, length 27
09:31:11.753586 IP 192.70.196.45.2061 > 172.16.0.100.5896: Flags [P.], seq 1:23, ack 27, win 64737, length 22
09:31:12.420381 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [.], ack 23, win 32120, length 0
09:31:13.019616 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [P.], seq 27:54, ack 23, win 32120, length 27
09:31:13.061320 IP 192.70.196.45.2061 > 172.16.0.100.5896: Flags [.], ack 54, win 64710, length 0
09:31:13.068682 IP 192.70.196.45.2061 > 172.16.0.100.5896: Flags [P.], seq 23:45, ack 54, win 64710, length 22
09:31:13.724740 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [.], ack 45, win 32120, length 0
09:31:14.339969 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [P.], seq 54:82, ack 45, win 32120, length 28
09:31:14.344559 IP 192.70.196.45.2061 > 172.16.0.100.5896: Flags [P.], seq 45:67, ack 82, win 64682, length 22
09:31:14.959218 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [.], ack 67, win 32120, length 0
09:31:15.566137 IP 172.16.0.100.5896 > 192.70.196.45.2061: Flags [P.], seq 82:109, ack 67, win 32120, length 27
09:31:15.569549 IP 192.70.196.45.2061 > 172.16.0.100.5896: Flags [P.], seq 67:89, ack 109, win 64655, length 22

Version: VyOS 1.2.0-rolling+201904190439
Hardware model: KVM

rob · May 27, 2019, 8:00pm

there is one open task in phabricator:
https://phabricator.vyos.net/T446

sadly the forum link in the task don’t work anymore. Maybe this is the problem.
Or does it work in other releases?

EDIT:

i see the src address is wrong so the task in phabricator is maybe not your issue… mh
can you provide your flow-accounting configuration?

sergy.silk · May 28, 2019, 6:07am

system {
<skipped...>
    flow-accounting {
        interface eth2
        interface eth1.334
        netflow {
            engine-id 70
            sampling-rate 1
            server 10.99.0.123 {
                port 2055
            }
            source-ip 10.99.0.70
            version 9
        }
    }
<skipped...>
}

or “command style”:

set system flow-accounting interface 'eth2'
set system flow-accounting interface 'eth1.334'
set system flow-accounting netflow engine-id '70'
set system flow-accounting netflow sampling-rate '1'
set system flow-accounting netflow server 10.99.0.123 port '2055'
set system flow-accounting netflow source-ip '10.99.0.70'
set system flow-accounting netflow version '9'

rob · May 28, 2019, 7:01pm

Hey,

i tryed it with the official 1.2.1 release and the latest rolling vyos-1.2.0-rolling+201905280337-amd64.iso

both worked for me, in this setup:

ISP  <--- 192.168.122.0/24 eth0 vyos eth1 10.1.1.0/24 ---> client

set nat source rule 1 outbound-interface 'eth0'
set nat source rule 1 translation address 'masquerade'


vyos@vyos:~$ show flow-accounting
flow-accounting for [eth0]
Src Addr        Dst Addr        Sport Dport Proto    Packets      Bytes   Flows
185.144.208.249 192.168.122.182 443   39768   tcp     155628  436672883       1
1.1.1.1         192.168.122.182 0     0      icmp         11        924       1
216.58.207.67   192.168.122.182 80    57442   tcp          4        742       1
145.239.0.197   192.168.122.182 123   123     udp          6        456       6
193.141.27.6    192.168.122.182 123   123     udp          6        456       6
192.168.122.1   192.168.122.182 53    55669   udp          2        202       1
1.1.1.1         192.168.122.182 53    47405   udp          2        154       1
192.168.122.1   192.168.122.182 53    46800   udp          2        142       1
78.46.53.2      192.168.122.182 123   123     udp          1         76       1
185.144.208.249 192.168.122.182 443   39770   tcp          1         60       1

Total entries: 10
Total flows  : 20
Total pkts   : 155,663
Total bytes  : 436,676,095

flow-accounting for [eth1]
Src Addr        Dst Addr        Sport Dport Proto    Packets      Bytes   Flows
10.1.1.10       1.1.1.1         0     0      icmp         11        924       1
10.1.1.10       216.58.207.67   57442 80      tcp          6        393       1
10.1.1.10       1.1.1.1         47405 53      udp          2        110       1

i download the rolling release on the vyos machine and do icmp and a “curl google.com”

is this setup like yours?

please, can you test the latest rolling?

sergy.silk · May 29, 2019, 6:37am

Hi, Rob.

If I get flow-accounting from whole ethernet interface there is no problems.
But I need flow-accounting from subinterfaces (vif) as I wrote in first message.

rob · May 31, 2019, 12:10pm

oh i see sorry, will test it later again with vif

rob · May 31, 2019, 8:43pm

Hi @sergy.silk,

i tested it again with a vif interface and it worked

vyos@vyos# run show flow-accounting interface eth1.222
flow-accounting for [eth1.222]
Src Addr        Dst Addr        Sport Dport Proto    Packets      Bytes   Flows
10.1.1.13       1.1.1.1         0     0      icmp         92       7728       1
10.1.1.14       172.217.18.163  35018 80      tcp          6        393       1
10.1.1.14       10.1.1.1        0     0      icmp          2        168       1
10.1.1.14       193.99.144.80   47718 80      tcp          2        112       1
10.1.1.14       1.1.1.1         36845 53      udp          2        110       1
10.1.1.14       1.1.1.1         33733 53      udp          2        108       1

can you test the latest rolling release?

sergy.silk · October 16, 2019, 7:19am

I tested it on latest rolling release:

$ sh ver
Version:          VyOS 1.2-rolling-201910150117
Built by:         [email protected]
Built on:         Tue 15 Oct 2019 01:17 UTC
Build UUID:       ece9a491-935e-4e1a-9be3-4a9e2cfe45e1
Build Commit ID:  25bb74bc51f7ee

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  Red Hat
Hardware model:   KVM
Hardware S/N:
Hardware UUID:    981110f7-8aaf-4d3c-8f3b-2e9c54121e62

Copyright:        VyOS maintainers and contributors

The result was incorrect:

$ sh flow-accounting interface eth1.17
flow-accounting for [eth1.17]
Src Addr        Dst Addr        Sport Dport Proto    Packets      Bytes   Flows
128.6.126.87    172.16.1.17     0     0       106          1       1400       1
128.6.228.16    172.16.1.17     0     0       239          1       1248       1
128.6.52.195    172.16.1.17     0     0     ipx-in-ip          1        617       1
128.6.52.105    172.16.1.17     0     0       201          1        617       1
128.6.50.52     172.16.1.17     0     0       254          1        617       1
128.6.51.6      172.16.1.17     0     0     ipv6-frag          1        617       1
128.6.52.35     172.16.1.17     0     0        15          1        617       1
128.6.51.110    172.16.1.17     0     0       196          1        617       1
128.6.51.86     172.16.1.17     0     0       220          1        617       1
128.6.53.17     172.16.1.17     0     0        33          1        617       1
128.6.51.77     172.16.1.17     0     0       229          1        617       1
128.6.52.54     172.16.1.17     0     0       252          1        617       1
128.6.52.125    172.16.1.17     0     0       181          1        617       1
128.6.51.45     172.16.1.17     0     0         5          1        617       1

sergy.silk · October 16, 2019, 9:22am

Also I checked flow-accounting on VyOS VM deployed on VMware Hypervisor.
I got correct result:

$ sh flow-accounting interface eth1.800
flow-accounting for [eth1.800]
Src Addr        Dst Addr        Sport Dport Proto    Packets      Bytes   Flows
10.99.0.123     10.99.252.20    0     0      icmp         41       3444       1
10.99.252.17    10.99.252.20    0     0      icmp          3        252       1

I conclude that the problem is observed on virtual machines deployed on KVM hypervisors.

sergy.silk · October 16, 2019, 12:59pm

Problem is fixed by changing network interface driver from virtio to rtl8139 or e1000.

system · October 18, 2019, 4:01pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.