Next hop traffic getting blocked by invalid state rule

Well after allowing invalid traffic on bridge interfaces this morning, my Tailscale container on VyOS is connecting however, some traffic destined to the remote network from the local VyOS network is getting blocked by the invalid state rule now.

[STATE-POLICY-INV-D]IN=bond0.450 OUT=pod-tailscale MAC=a6:28:21:91:59:26:00:50:56:8d:25:12:08:00 SRC=10.3.7.2 DST=10.4.1.66 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=4069 DF PROTO=TCP SPT=39952 DPT=2233 WINDOW=65535 RES=0x00 FIN URGP=0

I’ve added the next hop rule, the source subnet is “trusted” and has a rule to allow all forwarded traffic. What am I missing?

Here’s my current routes:

S>* 0.0.0.0/0 [210/0] via [my isp], eth0, weight 1, 00:26:32
C>* 10.3.3.0/24 is directly connected, bond0.300, 00:26:35
C>* 10.3.4.0/27 is directly connected, bond0, 00:21:57
C>* 10.3.4.32/28 is directly connected, pod-tailscale, 00:26:34
C>* 10.3.5.0/24 is directly connected, bond0.500, 00:26:35
C>* 10.3.6.0/24 is directly connected, bond0.600, 00:26:35
C>* 10.3.7.0/28 is directly connected, bond0.450, 00:26:35
C>* 10.3.7.16/28 is directly connected, bond0.700, 00:26:35
C>* 10.3.7.64/27 is directly connected, bond0.550, 00:26:35
S>* 10.4.0.0/16 [1/0] via 10.3.4.36, pod-tailscale, weight 1, 00:26:33
C>* [my isp]/24 is directly connected, eth0, 00:26:32
S>* 100.64.0.0/10 [1/0] via 10.3.4.36, pod-tailscale, weight 1, 00:26:33
C>* 192.168.1.0/24 is directly connected, eth1, 00:26:38

Here’s my full firewall config:

 global-options {
     apply-to-bridged-traffic {
         invalid-connections
     }
     state-policy {
         established {
             action accept
         }
         invalid {
             action drop
             log
         }
         related {
             action accept
         }
     }
 }
 group {
     address-group hass {
         address 10.3.3.15
         description "Home assistant exceptions"
     }
     address-group sonos {
         address 10.3.5.92
         address 10.3.5.96
         address 10.3.5.124
         description "Home assistant exceptions"
     }
     network-group internal {
         network 100.64.0.0/10
         network 10.0.0.0/8
         network 192.168.0.0/16
         network 172.16.0.0/12
     }
     network-group nat {
         network 192.168.1.0/24
         network 10.3.4.32/28
         network 10.3.3.0/24
         network 10.3.7.0/28
         network 10.3.5.0/24
         network 10.3.7.64/27
         network 10.3.6.0/24
         network 10.3.7.16/28
     }
     network-group services {
         network 10.3.3.0/24
         network 10.4.3.0/24
     }
     network-group trusted {
         network 100.64.0.0/10
         network 10.4.1.0/27
         network 10.4.1.48/28
         network 10.4.1.32/28
         network 10.4.1.64/28
         network 10.3.4.0/27
         network 10.3.7.0/28
         network 10.3.7.16/28
         network 10.3.3.0/24
         network 10.4.3.0/24
         network 10.3.4.32/28
     }
     port-group hass {
         description "Home assistant exceptions for sonos"
         port 1443
         port 1400
         port 8123
     }
     port-group public-services {
         port 53
         port 123
         port 80
         port 443
     }
 }
 ipv4 {
     forward {
         filter {
             default-action drop
             rule 100 {
                 action accept
                 description "Allow trusted networks to all destinations"
                 source {
                     group {
                         network-group trusted
                     }
                 }
             }
             rule 101 {
                 action accept
                 description "Allow internet access"
                 destination {
                     group {
                         network-group !internal
                     }
                 }
                 protocol all
             }
             rule 200 {
                 action accept
                 description "Allow services internally"
                 destination {
                     group {
                         network-group services
                         port-group public-services
                     }
                 }
                 protocol tcp_udp
                 source {
                     group {
                         network-group internal
                     }
                 }
             }
             rule 300 {
                 action accept
                 description "Allow ICMP to service networks"
                 destination {
                     group {
                         network-group services
                     }
                 }
                 protocol icmp
             }
             rule 400 {
                 action accept
                 description "Home assistant exceptions for sonos"
                 destination {
                     group {
                         address-group hass
                         port-group hass
                     }
                 }
                 protocol tcp_udp
                 source {
                     group {
                         address-group sonos
                     }
                 }
             }
         }
     }
     input {
         filter {
             default-action drop
             rule 10 {
                 action accept
                 destination {
                     port ssh
                 }
                 inbound-interface {
                     name eth1
                 }
                 protocol tcp_udp
                 source {
                     address 192.168.1.0/24
                 }
             }
             rule 20 {
                 action accept
                 description "Allow tailscale container in"
                 source {
                     address 10.3.4.32/28
                 }
             }
         }
     }
 }

It looks like there’s been some changes to how the firewall works very recently. I’m certainly used to attaching rulesets to interfaces, not what appears to be global rules in the rolling version. I took a look at this post and a few other example configs and looking at the cli tree in the latest rolling, it looks like you can still configure the invalid state rule under forward filter instead of in global option. Would that make a difference? Clearly, I’m really struggling to understand how the invalid state rule gets applied.

Podman uses bridges for networks and bridges for firewall use own table and own state rules

But isnt a bridge a bridge?

Sure you can have two different bridges but a physical interface can only belong to one at a time?

Each container network is a bridge between netnsXXX and “default” netns which connected via “veth” pair.

Thanks for the replies everyone! I’ve discarded the global options and added a new jump rule as shown in the quick start guide. I attached that rule to just the WAN interface for now. This has cleared up the log messages however, traffic still isn’t making it to the other site. I can load things on the VyOS site from the remote site but it looks like traffic initiated out from the VyOS site is still getting dropped somewhere.

Right now I’m especially focusing on missing ESXi heartbeats, which are UDP port 902 sent from the host to the vCenter server at the other site. I can capture these going in my management interface and out the pod-tailscale interface on VyOS so they are getting lost somewhere in the container stack I think. The first thing I can think of is when setting this up on standard linux, there’s some commands you need to run to enable IP forwarding.

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

Is there a way to run those commands inside the container or is that unnecessary with containers?

Looks like I solved that one. The problem was the --stateful-filtering=false' flag from the original guide. Even though I had set it to false, I had it true at one point and I had to delete the docker volume to get it to fully disable. Maybe a bug I need to investigate more and report to Tailscale.

Fingers crossed I don’t find any more problems. I’m leaving the VyOS router installed for now!

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.