Question about implement Flowtable

Hi there

I’m new to the VyOS, and what frustrates me is the firewall configuration.
Thus, I need your kind help to get it done.
Thanks in advance.

Context:

  • VyOS 1.4
  • Intel X710-DA2
  • i5-8400T
  • 32G RAM

Question is:

  1. How can I implement a correct Flowtable implementation for the below-running firewall configuration?
  2. Is a zone-based firewall supported flowable also?
  1. What criteria can support/implement the hardware offload?

Here is the firewall configuration:

#
# Global Configuration Items
#
set firewall global-options all-ping 'enable'
set firewall global-options broadcast-ping 'disable'
set firewall global-options ip-src-route 'disable'
set firewall global-options ipv6-src-route 'disable'
set firewall global-options log-martians 'enable'
set firewall global-options receive-redirects 'disable'
set firewall global-options send-redirects 'disable'
set firewall global-options source-validation 'strict'
set firewall global-options state-policy established action 'accept'
set firewall global-options state-policy related action 'accept'
set firewall global-options state-policy invalid action 'drop'
set firewall global-options state-policy invalid log
set firewall global-options state-policy invalid log-level 'warn'
set firewall global-options syn-cookies 'enable'
set firewall global-options resolver-cache
commit

#
# IPV4
#

# Group: Network, Interfaces, Ports
set firewall group network-group HomeLAN-v4 network '192.168.1.0/24'
set firewall group ipv6-network-group HomeLAN-v6 network 'fd12:3456:789a:1::/64'
set firewall group interface-group WAN interface eth2
set firewall group interface-group WAN interface pppoe0
set firewall group interface-group LAN interface eth3
set firewall group interface-group LAN interface eth1
set firewall group interface-group Mgmt interface eth0
set firewall group port-group PORT-TEAMS-UDP port '3478-3481'
set firewall group port-group Web-TCP port '80'
set firewall group port-group Web-TCP port '443'
set firewall group port-group Service_TCP-UDP port '53'
set firewall group port-group Service_TCP-UDP port '7777'
set firewall group port-group Service_TCP-UDP port '67'
set firewall group port-group Service_TCP-UDP port '68'
set firewall group port-group Service_TCP-UDP port '3000'
set firewall group port-group Service_TCP-UDP port '3001'

commit

# Rules
set firewall ipv4 name CONN_FILTER default-action 'return'
set firewall ipv4 name CONN_FILTER rule 10 action 'accept'
set firewall ipv4 name CONN_FILTER rule 10 state established
set firewall ipv4 name CONN_FILTER rule 10 state related

set firewall ipv4 name WAN-IN default-action 'drop'

set firewall ipv4 name VyOS_Mgmt default-action 'return'
set firewall ipv4 name VyOS_Mgmt rule 10 action 'accept'
set firewall ipv4 name VyOS_Mgmt rule 10 state new
set firewall ipv4 name VyOS_Mgmt rule 10 state established
set firewall ipv4 name VyOS_Mgmt rule 10 inbound-interface group 'Mgmt'
set firewall ipv4 name VyOS_Mgmt rule 20 action 'accept'
set firewall ipv4 name VyOS_Mgmt rule 20 state related
set firewall ipv4 name VyOS_Mgmt rule 20 inbound-interface group 'Mgmt'

commit

# Chains
## Forward
set firewall ipv4 forward filter rule 10 action 'jump'
set firewall ipv4 forward filter rule 10 jump-target CONN_FILTER

set firewall ipv4 forward filter rule 100 action jump
set firewall ipv4 forward filter rule 100 jump-target WAN-IN
set firewall ipv4 forward filter rule 100 inbound-interface group WAN
set firewall ipv4 forward filter rule 100 destination group network-group HomeLAN-v4

## Input
### Allow internet access.
set firewall ipv4 input filter default-action 'drop'
set firewall ipv4 input filter rule 10 action 'jump'
set firewall ipv4 input filter rule 10 jump-target CONN_FILTER

### Allow Management access. 
set firewall ipv4 input filter rule 20 action jump
set firewall ipv4 input filter rule 20 jump-target VyOS_Mgmt
set firewall ipv4 input filter rule 20 destination group port-group Service_TCP-UDP
set firewall ipv4 input filter rule 20 protocol tcp

### Allow services access
set firewall ipv4 input filter rule 30 action 'accept'
set firewall ipv4 input filter rule 30 icmp type-name 'echo-request'
set firewall ipv4 input filter rule 30 protocol 'icmp'
set firewall ipv4 input filter rule 30 state new
set firewall ipv4 input filter rule 30 inbound-interface group '!WAN'

set firewall ipv4 input filter rule 40 action 'accept'
set firewall ipv4 input filter rule 40 destination group port-group Service_TCP-UDP
set firewall ipv4 input filter rule 40 protocol 'tcp_udp'
set firewall ipv4 input filter rule 40 source group network-group HomeLAN-v4

### Allowing all connections coming from localhost
set firewall ipv4 input filter rule 50 action 'accept'
set firewall ipv4 input filter rule 50 source address 127.0.0.0/8

### Allowing ntopNG access
set firewall ipv4 input filter rule 60 action 'accept'
set firewall ipv4 input filter rule 60 description 'Allow ntopNG access'
set firewall ipv4 input filter rule 60 inbound-interface name 'lo'

commit
save

Here is the firewall summary.

ouzy@vyos:~$ show firewall
Rulesets Information

---------------------------------
ipv4 Firewall "forward filter"

Rule     Action    Protocol      Packets    Bytes  Conditions
-------  --------  ----------  ---------  -------  -------------------------------------------------------
10       jump      all             18030  1550241  jump NAME_CONN_FILTER
100      jump      all                 0        0  ip daddr @N_HomeLAN-v4 iifname @I_WAN  jump NAME_WAN-IN
default  accept    all             18030  1550241

---------------------------------
ipv4 Firewall "input filter"

Rule     Action    Protocol      Packets    Bytes  Conditions
-------  --------  ----------  ---------  -------  ------------------------------------------------------------------------------------
10       jump      all             16803  2357884  jump NAME_CONN_FILTER
20       jump      tcp                94     4852  tcp dport @P_Service_TCP-UDP  jump NAME_VyOS_Mgmt
30       accept    icmp                4      112  ct state new iifname != @I_WAN icmp type echo-request  accept
40       accept    tcp_udp           154    15852  meta l4proto { tcp, udp } th dport @P_Service_TCP-UDP ip saddr @N_HomeLAN-v4  accept
50       accept    all                 3      180  ip saddr 127.0.0.0/8  accept
60       accept    all                 4      224  iifname "lo"  accept
default  drop      all             16547  2336784

---------------------------------
ipv4 Firewall "name CONN_FILTER"

Rule     Action    Protocol      Packets    Bytes  Conditions
-------  --------  ----------  ---------  -------  -----------------------------------------
10       accept    all                 0        0  ct state { established, related }  accept
default  return    all             34833  3908125

---------------------------------
ipv4 Firewall "name VyOS_Mgmt"

Rule     Action    Protocol      Packets    Bytes  Conditions
-------  --------  ----------  ---------  -------  -----------------------------------------------------
10       accept    all                91     4732  ct state { established, new } iifname @I_Mgmt  accept
20       accept    all                 0        0  ct state related iifname @I_Mgmt  accept
default  return    all                 3      120

---------------------------------
ipv4 Firewall "name WAN-IN"

Rule     Action    Protocol      Packets    Bytes
-------  --------  ----------  ---------  -------
default  drop      all                 0        0

The first try but no luck.
This one took the Flowtables Firewall Configuration — VyOS 1.4.x (sagitta) documentation as a reference.

### change Forward chain rule
set firewall ipv4 name CONN_FILTER rule 15 action 'offload'
set firewall ipv4 name CONN_FILTER rule 15 offload-target 'FT-WAN-LAN'
set firewall ipv4 name CONN_FILTER rule 15 state 'established'
set firewall ipv4 name CONN_FILTER rule 15 state 'related'

The 2nd try, but no luck too.

## official example as ref. 
delete firewall ipv4 forward filter rule 10 action 'jump'
delete firewall ipv4 forward filter rule 10 jump-target CONN_FILTER

delete firewall ipv4 forward filter rule 100 action jump
delete firewall ipv4 forward filter rule 100 jump-target WAN-IN
delete firewall ipv4 forward filter rule 100 inbound-interface group WAN
delete firewall ipv4 forward filter rule 100 destination group network-group HomeLAN-v4

set firewall ipv4 forward filter default-action 'drop'      // I have to change to delete given it blocked internet access after implementation. 

set firewall ipv4 forward filter rule 10 action 'offload'
set firewall ipv4 forward filter rule 10 offload-target 'FT-WAN-LAN'
set firewall ipv4 forward filter rule 10 state 'established'
set firewall ipv4 forward filter rule 10 state 'related'

set firewall ipv4 forward filter rule 20 action 'accept'
set firewall ipv4 forward filter rule 20 state 'established'
set firewall ipv4 forward filter rule 20 state 'related'

set firewall ipv4 forward filter rule 100 action 'accept'
set firewall ipv4 forward filter rule 100 inbound-interface group WAN
set firewall ipv4 forward filter rule 100 destination group network-group HomeLAN-v4

Addtional info

show interfaces ethernet eth2 physical

Settings for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseSR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseSR/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
Ring parameters for eth2:
Pre-set maximums:
RX:                     4096
RX Mini:                n/a
RX Jumbo:               n/a
TX:                     4096
TX push buff len:       n/a
Current hardware settings:
RX:                     4096
RX Mini:                n/a
RX Jumbo:               n/a
TX:                     4096
RX Buf Len:             n/a
CQE Size:               n/a
TX Push:                off
RX Push:                off
TX push buff len:       n/a
TCP data split:         n/a
driver: i40e
version: 6.6.49-amd64-vyos
firmware-version: 8.30 0x8000a9c1 1.2766.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

show interfaces ethernet eth2 physical offload

rx-checksumming               on
tx-checksumming               on
tx-checksum-ip-generic        on
tx-checksum-sctp              on
scatter-gather                on
tx-scatter-gather             on
tcp-segmentation-offload      on
tx-tcp-segmentation           on
tx-tcp-ecn-segmentation       on
tx-tcp-mangleid-segmentation  off
tx-tcp6-segmentation          on
generic-segmentation-offload  on
generic-receive-offload       on
rx-vlan-offload               on
tx-vlan-offload               on
ntuple-filters                on
receive-hashing               on
highdma                       on
tx-gre-segmentation           on
tx-gre-csum-segmentation      on
tx-ipxip4-segmentation        on
tx-ipxip6-segmentation        on
tx-udp_tnl-segmentation       on
tx-udp_tnl-csum-segmentation  on
tx-gso-partial                on
tx-udp-segmentation           on
tx-nocache-copy               off
loopback                      off
l2-fwd-offload                off
hw-tc-offload                 on
rx-udp_tunnel-port-offload    on
rx-gro-list                   off
rx-udp-gro-forwarding         off

@Apachez @GurliGebis
I read several firewall-related threads, and you have deep knowledge of this. I am not quite sure ping you whether it is impolite. Having over 40 read but have not replied yet, please allow me to ping you to have a look and share your comments.

As far as I remember, you should not use global state policies while using flowtables.
Can you try removing it?

Thanks for the tips.

Do you mean to remove the below all or just first two lines?

set firewall global-options state-policy established action 'accept'
set firewall global-options state-policy related action 'accept'
set firewall global-options state-policy invalid action 'drop'
set firewall global-options state-policy invalid log
set firewall global-options state-policy invalid log-level 'warn'

Ohh, seems I forgot to add flowtable config. Here it is.

### Create flowtable
set firewall flowtable FT-WAN-LAN description 'Flowtable between WAN and LAN interfaces'
set firewall flowtable FT-WAN-LAN interface 'pppoe0'					// Can't turn on flowtable hardware for PPPoE 
set firewall flowtable FT-WAN-LAN interface 'eth2'
set firewall flowtable FT-WAN-LAN interface 'eth3'
set firewall flowtable FT-WAN-LAN offload 'software'					// Offloading method (default: software)

Thus, One more question here is: Should I included pppoe0 in the flowable?

Sigh ~~~ This user forum isn’t as active as I imagined.

Were you able to test without global state policies?

delete firewall global-options state-policy

Also, first you have:

set firewall ipv4 name CONN_FILTER default-action 'return'
set firewall ipv4 name CONN_FILTER rule 10 action 'accept'
set firewall ipv4 name CONN_FILTER rule 10 state established
set firewall ipv4 name CONN_FILTER rule 10 state related

And in your first attempt of adding flowtables:

set firewall ipv4 name CONN_FILTER rule 15 action 'offload'
set firewall ipv4 name CONN_FILTER rule 15 offload-target 'FT-WAN-LAN'
set firewall ipv4 name CONN_FILTER rule 15 state 'established'
set firewall ipv4 name CONN_FILTER rule 15 state 'related'

As you can see, matching criteria for both rules 10 and 15 are the same. This means that packet will first hit rule 10, and then it won’t match rule 15.

As in the example written in the doc:

  • Don’t use global-state policies
  • In base chain, define a rule that will offload to the flowtables, all connections established and related
  • Next rules: go on as needed, accepting connections that you need.

Thanks for the reply.

Don’t use global-state policies

I go back to read the Flowtable configuration document again. I think there isn’t such a highlight, only the bold one below before the example config, but I am not sure it’s relevant.

As described, first packet will be evaluated by all the firewall path, so desired connection should be explicitely accepted. Same thing should be taken into account for traffic in reverse order. In most cases state policies are used in order to accept connection in reverse patch.
Flowtables Firewall Configuration — VyOS 1.4.x (sagitta) documentation

set firewall flowtable FT01 interface 'eth0'
set firewall flowtable FT01 interface 'eth1'

set firewall ipv4 forward filter default-action 'drop'

set firewall ipv4 forward filter rule 10 action 'offload'
set firewall ipv4 forward filter rule 10 offload-target 'FT01'
set firewall ipv4 forward filter rule 10 state 'established'
set firewall ipv4 forward filter rule 10 state 'related'

set firewall ipv4 forward filter rule 20 action 'accept'
set firewall ipv4 forward filter rule 20 state 'established'
set firewall ipv4 forward filter rule 20 state 'related'

set firewall ipv4 forward filter rule 110 action 'accept'
set firewall ipv4 forward filter rule 110 destination address '192.0.2.100'
set firewall ipv4 forward filter rule 110 destination port '1122'
set firewall ipv4 forward filter rule 110 inbound-interface name 'eth0'
set firewall ipv4 forward filter rule 110 protocol 'tcp'
  • In base chain, define a rule that will offload to the flowtables, all connections established and related
  • Next rules: go on as needed, accepting connections that you need.

As you said, like the example, do the offload before the accept.

Here is [TEST #4]

delete firewall global-options state-policy established action 'accept'
delete firewall global-options state-policy related action 'accept'
delete firewall ipv4 name CONN_FILTER default-action 'return'
delete firewall ipv4 name CONN_FILTER rule 10 action 'accept'
delete firewall ipv4 name CONN_FILTER rule 10 state established
delete firewall ipv4 name CONN_FILTER rule 10 state related

set firewall flowtable FT-WAN-LAN description 'Flowtable between WAN and LAN interfaces'
set firewall flowtable FT-WAN-LAN interface 'pppoe0'
set firewall flowtable FT-WAN-LAN interface 'eth2'
set firewall flowtable FT-WAN-LAN interface 'eth3'
set firewall flowtable FT-WAN-LAN offload 'software'

set firewall ipv4 name CONN_FILTER rule 10 action 'offload'
set firewall ipv4 name CONN_FILTER rule 10 offload-target 'FT-WAN-LAN'
set firewall ipv4 name CONN_FILTER rule 10 state 'established'
set firewall ipv4 name CONN_FILTER rule 10 state 'related'
set firewall ipv4 name CONN_FILTER rule 15 action 'accept'
set firewall ipv4 name CONN_FILTER rule 15 state 'established'
set firewall ipv4 name CONN_FILTER rule 15 state 'related'


Failed when commit

ouzy@vyos# commit

Failed to apply firewall: /run/nftables.conf:59:9-82: Error: Could not
process rule: Operation not supported         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_FT-WAN-LAN
comment "ipv4-NAM-CONN_FILTER-10"         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:60:18-38: Error: Could not process rule: Operation
not supported         ct state {established,related} counter accept
comment "ipv4-NAM-CONN_FILTER-15"                  ^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:60:9-53: Error: Could not process rule: Operation not
supported         ct state {established,related} counter accept comment
"ipv4-NAM-CONN_FILTER-15"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:61:9-20: Error: Could not process rule: Operation not
supported         counter drop comment "CONN_FILTER default-action drop"
^^^^^^^^^^^^ /run/nftables.conf:64:18-34: Error: Could not process rule:
Operation not supported         ct state {new,established} iifname
@I_Mgmt counter accept comment "ipv4-NAM-VyOS_Mgmt-10"
^^^^^^^^^^^^^^^^^ /run/nftables.conf:64:9-66: Error: Could not process
rule: Operation not supported         ct state {new,established} iifname
@I_Mgmt counter accept comment "ipv4-NAM-VyOS_Mgmt-10"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:65:9-58: Error: Could not process rule: Operation not
supported         ct state {related} iifname  @I_Mgmt counter accept
comment "ipv4-NAM-VyOS_Mgmt-20"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:66:9-22: Error: Could not process rule: Operation not
supported         counter return comment "VyOS_Mgmt default-action
return"         ^^^^^^^^^^^^^^ /run/nftables.conf:69:9-20: Error: Could
not process rule: Operation not supported         counter drop comment
"WAN-IN default-action drop"         ^^^^^^^^^^^^
/run/nftables.conf:119:9-36: Error: Could not process rule: Operation
not supported         ct state established counter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:120:9-82: Error: Could
not process rule: Operation not supported         ct state invalid log
prefix "[STATE-POLICY-INV-D]" level warn counter drop         ^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:121:9-32: Error: Could not process rule: Operation
not supported         ct state related counter
^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:122:9-14: Error: Could not
process rule: Operation not supported         return         ^^^^^^

[[firewall]] failed
Commit failed


Yes, our documentation always needs to be improved. Sorry for that. All contributions to documentation are always welcome.

And back on technical details, please, try offloading on base chain, as said before:

This means that you should define offload rules in base chain, such as forward, and not in custom rule-set, as you’ve done in CONN_FILTER.

Also, the logic described in [explanation] (Flowtables Firewall Configuration — VyOS 1.4.x (sagitta) documentation) section might be useful to understand the procedure, rather than copy and pasting.

Thanks for actively responding.
It could help me to understand the logic of configuring the firewall properly and running it efficiently.

Meanwhile, to make a good document. Here is my thought.
The “Quick Start” chapter is good, but, If I may, I’d recommend the examples in the following related chapter to have an “evolutionary”. Such as the example in the Quick Start chapter is simple, then gets more complex cause more features are involved.

Quick Start — VyOS 1.4.x (sagitta) documentation
Option 1: Global State Policies
and will be evaluated before any other rule defined in the firewall.

Maybe we should

  1. What specific means about evaluated before any other rule Flowtable offload could be a good example?
  2. Note here that some global state configurations do not apply to Flowtable.

[Test #5] offloading on base chain but not implement to custom rule-sets

delete firewall global-options state-policy established action 'accept'
delete firewall global-options state-policy related action 'accept'
delete firewall ipv4 forward filter rule 10 action 'jump'
delete firewall ipv4 forward filter rule 10 jump-target CONN_FILTER

set firewall flowtable FT-WAN-LAN description 'Flowtable between WAN and LAN interfaces'
set firewall flowtable FT-WAN-LAN interface 'pppoe0'
set firewall flowtable FT-WAN-LAN interface 'eth2'
set firewall flowtable FT-WAN-LAN interface 'eth3'
set firewall flowtable FT-WAN-LAN offload 'software'

set firewall ipv4 forward filter rule 10 action 'offload'
set firewall ipv4 forward filter rule 10 offload-target 'FT-WAN-LAN'
set firewall ipv4 forward filter rule 10 state 'established'
set firewall ipv4 forward filter rule 10 state 'related'

set firewall ipv4 forward filter rule 15 action 'accept'
set firewall ipv4 forward filter rule 15 state 'established'
set firewall ipv4 forward filter rule 15 state 'related'

It work~~~

ouzy@vyos# run show firewall ipv4 forward filter
Ruleset Information

---------------------------------
ipv4 Firewall "forward filter"

Rule     Action    Protocol      Packets     Bytes  Conditions
-------  --------  ----------  ---------  --------  ----------------------------------------------------------------------
10       offload   all             31333  10095715  ct state { established, related }  flow add @VYOS_FLOWTABLE_FT-WAN-LAN
15       accept    all             31333  10095715  ct state { established, related }  accept
100      jump      all                 0         0  ip daddr @N_HomeLAN-v4 iifname @I_WAN  jump NAME_WAN-IN
default  accept    all              1122     83103



Quick follow-up question: Why does applying the below commit break the internet?

set firewall ipv4 forward filter default-action 'drop'

Then, the rest of the questions were initially asked, looking for answers.

  1. Is a zone-based firewall also supported and flowable? How can I modify it based on my firewall configuration?
  2. What criteria can support/implement the hardware offload? Maybe this one is more realistic than question No. 2
ouzy@vyos# delete firewall flowtable FT-WAN-LAN offload 'software'
[edit]
ouzy@vyos# delete firewall flowtable FT-WAN-LAN interface 'pppoe0'
[edit]
ouzy@vyos# set firewall flowtable FT-WAN-LAN offload 'hardware'
[edit]
ouzy@vyos# commit

Failed to apply firewall: /run/nftables.conf:22:15-39: Error: Could not
process rule: Operation not supported     flowtable VYOS_FLOWTABLE_FT-
WAN-LAN {               ^^^^^^^^^^^^^^^^^^^^^^^^^
/run/nftables.conf:32:48-82: Error: Could not process rule: No such file
or directory         ct state {established,related} counter flow add
@VYOS_FLOWTABLE_FT-WAN-LAN comment "ipv4-FWD-filter-10"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:129:15-39: Error:
Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_FT-WAN-LAN {               ^^^^^^^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.