Routing issue from LAN to LAN with publiic IP block with load-balance wan

0lzi · August 15, 2025, 12:37pm

I have an odd routing issue I can’t seem to work out.

setup:

VLAN4000 - WAN xxx.xxx.131.224/29

  FW1 - xxx.xxx.131.228/29
  FW2 - xxx.xxx.131.229/29
  VRRP - xxx.xxx.131.230/29
  UPSTREAM VRRP - xxx.xxx.131.225/29

VLAN33 - INTERNAL 10.10.33.0/24 - SNAT rules for this vlan

  FW1 - 10.10.33.252/24
  FW2 - 10.10.33.253/24
  VRRP - 10.10.33.254/24

VLAN69 - OPENSTACK xxx.xxx.160.96/27 - routed from upstream to WAN-VRRP no NAT for this vlan

  FW1 - 10.10.69.1/24
  FW2 - 10.10.69.2/24
  VRRP xxx.xxx.160.99/27

I set this up to use the least amount of IP's in the /27.

Visual

Above the dotted line is our upstream provider.

Config Snippets

# Firewall Groups
set firewall group address-group OPENSTACK address 'xxx.xxx.160.100-xxx.xxx.160.127'
set firewall group network-group NET-VLAN33 network '10.10.33.0/24'
set firewall group network-group NET-VLAN69 network 'xxx.xxx.160.96/27'
# Firewall Forward filter
set firewall ipv4 forward filter rule 43 action 'jump'
set firewall ipv4 forward filter rule 43 inbound-interface name 'bond0.33'
set firewall ipv4 forward filter rule 43 jump-target 'VLAN33-FORWARD'
set firewall ipv4 forward filter rule 79 action 'jump'
set firewall ipv4 forward filter rule 79 inbound-interface name 'bond0.69'
set firewall ipv4 forward filter rule 79 jump-target 'VLAN69-FORWARD'

# VLAN33 Rules
set firewall ipv4 name VLAN33-FORWARD rule 3500 action 'accept'
set firewall ipv4 name VLAN33-FORWARD rule 3500 description 'Allow packets from VLAN33 to VLAN35'
set firewall ipv4 name VLAN33-FORWARD rule 3500 destination group network-group 'NET-VLAN35'
set firewall ipv4 name VLAN33-FORWARD rule 3500 source group network-group 'NET-VLAN33'
set firewall ipv4 name VLAN33-FORWARD rule 3500 state 'new'
set firewall ipv4 name VLAN33-FORWARD rule 999999 action 'drop'
set firewall ipv4 name VLAN33-FORWARD rule 999999 description 'Drop packets from VLAN33 to rfc1918 networks'
set firewall ipv4 name VLAN33-FORWARD rule 999999 destination group network-group 'NET-RFC1918'
set firewall ipv4 name VLAN33-FORWARD rule 999999 inbound-interface name 'bond0.33'
set firewall ipv4 name VLAN33-FORWARD rule 999999 state 'new'

# VLAN69 Rules
set firewall ipv4 name VLAN69-FORWARD default-action 'accept'
set firewall ipv4 name VLAN69-FORWARD description 'VLAN69 (public)'
set firewall ipv4 name VLAN69-FORWARD rule 999999 action 'drop'
set firewall ipv4 name VLAN69-FORWARD rule 999999 description 'Drop packets from VLAN69 to rfc1918 networks'
set firewall ipv4 name VLAN69-FORWARD rule 999999 destination group network-group 'NET-RFC1918'
set firewall ipv4 name VLAN69-FORWARD rule 999999 inbound-interface name 'bond0.69'
set firewall ipv4 name VLAN69-FORWARD rule 999999 state 'new'

# NAT
set nat source rule 201033 description 'VLAN33 (os-ext) via WAN Connection'
set nat source rule 201033 destination address '!xxx.xxx.160.96/27'
set nat source rule 201033 outbound-interface name 'bond0.4000'
set nat source rule 201033 source address '10.10.33.0/24'
set nat source rule 201033 translation address 'xxx.xxx.131.230'

# Static routes
set protocols static route 0.0.0.0/0 next-hop xxx.xxx.131.225 distance '201'

# Wan-load-balance
set load-balancing wan rule 13 destination address 'xxx.xxx.160.96/27'
set load-balancing wan rule 13 exclude
set load-balancing wan rule 13 inbound-interface 'any'

VLAN33 can reach internet and other local vlans apart from the puplic ip block in VLAN69, when monitoring traffic doing mtr to an IP in VLAN69 it goes out of bond0.400 not being natted or translated as far as I can see.

VLAN69 can also route out to the internet but can’t get to VLAN33 , which is expected in firewall name VLAN69-FORWARD rule 999999 to dtop RFC1918

Not sure what I am missing here, as VLAN69 is directly connected to FW1 & 2, so there should be no reason for VLAN33 not to be able to get to VLAN69.

Apachez · August 15, 2025, 1:39pm

Towards your WAN-provider - are all devices connected to the same switch?

Since direct connection as the drawing shows will most likely not work unless you got some L2 going on at your or their end since you got VRRP at play here.

As troubleshooting set the physical addresses at FW1 and FW2 to the public /27 range on VLAN69. Like x.x.x.97/27 and x.x.x.98/27 or whatever.

Since you use rfc1918 address on the physical interfaces the box wont pick up that you also have a /27 that belongs on VLAN69.

So if changing to public IP at FW1 and FW2 and it now works but you still want to use RFC1918 on the physical interfaces you will need to setup a static route to tell the box that this public /27 range exists on VLAN69.

Not sure how the config for that would be if its possible to do something like “ip route x.x.169.96/27 nexthop vlan69”.

0lzi · August 15, 2025, 2:03pm

Towards your WAN-provider - are all devices connected to the same switch?

Its more like

where Upstream is VLAN400 and is an access port on a switch , firewalls are connected to leaf via a trunk port.

As troubleshooting set the physical addresses at FW1 and FW2 to the public /27 range on VLAN69. Like x.x.x.97/27 and x.x.x.98/27 or whatever.

This would then use 3 IP’s from that range which I was trying to avoid, but I fear this may be the cause, but if i do show ip route xxx.xxx.160.96/27 I get

user@firewall-1:~$ show ip route xxx.xxx.160.96/27
Routing entry for xxx.xxx.160.96/27
  Known via "ospf", distance 110, metric 1
  Last update 01w3d21h ago
    directly connected, bond0.69, weight 1

Routing entry for xxx.xxx.160.96/27
  Known via "connected", distance 0, metric 0, best
  Last update 01w3d21h ago
  * directly connected, bond0.69

user@firewall-1:~$ show interfaces
...
bond0.69  10.10.69.1/24        1a:3c:a7:4b:7d:54  default   1500  u/u    VLAN69 (public)
          xxx.xxx.160.99/27

0lzi · August 20, 2025, 7:58am

So when running:

delete load-balancing

routing to that network works fine from VLAN33 - VLAN69,

Here are the other load-balancing rules, I cant see what would make it route out bond0.4000 apart from possibly rule 1033

load-balancing wan rules

set load-balancing wan disable-source-nat
set load-balancing wan flush-connections
set load-balancing wan hook ‘/config/scripts/wan-failover.script’
set load-balancing wan interface-health bond0.4000 failure-count ‘4’
set load-balancing wan interface-health bond0.4000 nexthop ‘xxx.xxx.131.225’
set load-balancing wan interface-health bond0.4000 success-count ‘10’
set load-balancing wan interface-health bond0.4000 test 0 resp-time ‘3’
set load-balancing wan interface-health bond0.4000 test 0 target ‘xxx.xxx.131.225’
set load-balancing wan interface-health bond0.4000 test 0 type ‘ping’
set load-balancing wan rule 1 description ‘Dont load-balance RFC1918 traffic on eth - 10.0.0.0/8’
set load-balancing wan rule 1 destination address ‘10.0.0.0/8’
set load-balancing wan rule 1 exclude
set load-balancing wan rule 1 inbound-interface ‘eth+’
set load-balancing wan rule 2 description ‘Dont load-balance RFC1918 traffic on eth - 172.16.0.0/12’
set load-balancing wan rule 2 destination address ‘172.16.0.0/12’
set load-balancing wan rule 2 exclude
set load-balancing wan rule 2 inbound-interface ‘eth+’
set load-balancing wan rule 3 description ‘Dont load-balance RFC1918 traffic on eth - 192.168.0.0/16’
set load-balancing wan rule 3 destination address ‘192.168.0.0/16’
set load-balancing wan rule 3 exclude
set load-balancing wan rule 3 inbound-interface ‘eth+’
set load-balancing wan rule 4 description ‘Dont load-balance RFC1918 traffic on bond - 10.0.0.0/8’
set load-balancing wan rule 4 destination address ‘10.0.0.0/8’
set load-balancing wan rule 4 exclude
set load-balancing wan rule 4 inbound-interface ‘bond+’
set load-balancing wan rule 5 description ‘Dont load-balance RFC1918 traffic on bond - 172.16.0.0/12’
set load-balancing wan rule 5 destination address ‘172.16.0.0/12’
set load-balancing wan rule 5 exclude
set load-balancing wan rule 5 inbound-interface ‘bond+’
set load-balancing wan rule 6 description ‘Dont load-balance RFC1918 traffic on bond - 192.168.0.0/16’
set load-balancing wan rule 6 destination address ‘192.168.0.0/16’
set load-balancing wan rule 6 exclude
set load-balancing wan rule 6 inbound-interface ‘bond+’
set load-balancing wan rule 7 description ‘Dont load-balance RFC1918 traffic on tun - 10.0.0.0/8’
set load-balancing wan rule 7 destination address ‘10.0.0.0/8’
set load-balancing wan rule 7 exclude
set load-balancing wan rule 7 inbound-interface ‘vtun+’
set load-balancing wan rule 8 description ‘Dont load-balance RFC1918 traffic on tun - 172.16.0.0/12’
set load-balancing wan rule 8 destination address ‘172.16.0.0/12’
set load-balancing wan rule 8 exclude
set load-balancing wan rule 8 inbound-interface ‘vtun+’
set load-balancing wan rule 9 description ‘Dont load-balance RFC1918 traffic on tun - 192.168.0.0/16’
set load-balancing wan rule 9 destination address ‘192.168.0.0/16’
set load-balancing wan rule 9 exclude
set load-balancing wan rule 9 inbound-interface ‘vtun+’
set load-balancing wan rule 10 description ‘Dont load-balance RFC1918 traffic on vti - 10.0.0.0/8’
set load-balancing wan rule 10 destination address ‘10.0.0.0/8’
set load-balancing wan rule 10 exclude
set load-balancing wan rule 10 inbound-interface ‘vti+’
set load-balancing wan rule 11 description ‘Dont load-balance RFC1918 traffic on vti - 172.16.0.0/12’
set load-balancing wan rule 11 destination address ‘172.16.0.0/12’
set load-balancing wan rule 11 exclude
set load-balancing wan rule 11 inbound-interface ‘vti+’
set load-balancing wan rule 12 description ‘Dont load-balance RFC1918 traffic on vti - 192.168.0.0/16’
set load-balancing wan rule 12 destination address ‘192.168.0.0/16’
set load-balancing wan rule 12 exclude
set load-balancing wan rule 12 inbound-interface ‘vti+’
set load-balancing wan rule 13 destination address ‘xxx.xxx.160.96/27’
set load-balancing wan rule 13 exclude
set load-balancing wan rule 13 inbound-interface ‘any’
set load-balancing wan rule 1033 failover
set load-balancing wan rule 1033 inbound-interface ‘bond0.33’
set load-balancing wan rule 1033 interface bond0.4000 weight ‘100’
set load-balancing wan rule 1033 protocol ‘all’
set load-balancing wan sticky-connections inbound

as when loadbalance is on, get two routing tables:

local-oliverkelly@firewall-1.m3:~$ show ip route table allCodes: K - kernel route, C - connected, S - static, R - RIP,O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,f - OpenFabric,> - selected route, * - FIB route, q - queued, r - rejected, b - backupt - trapped, o - offload failure 

VRF default table 201:
K>* 0.0.0.0/0 [0/0] via xxx.xxx.131.225, bond0.4000, 00:08:40

VRF default table 254:
S>* 0.0.0.0/0 [201/0] via xxx.xxx.131.225, bond0.4000, weight 1, 04w5d19h…
C>* 10.85.33.0/24 is directly connected, bond0.33, 08w0d16h
C>* 10.85.69.0/29 is directly connected, bond0.69, 08w0d16h
C>* xxx.xxx.160.96/27 is directly connected, bond0.69, 02w1d15h

0lzi · August 20, 2025, 8:34am

although I have just rolled back to confim that was the issue, and then did the same delete load-balancing and the issue is still there

0lzi · August 20, 2025, 12:44pm

Update2:

SO.. It was the wan load-balance that was the issue, but seems to be a bug, when i deleted the config, then rolled back, then tried to delete the config again, it kept the second VRF Default table 201 .

A reboot, so it applied the config cleanly fixed it. But would like to know how I would fix this with wan load-balance enabled, as we will probably be adding a secondary WAN connection and will need to use the wan load-balance feature.