Q: Zone-based firewall and 2xISP links strange artifacts

General questions
1

Hi!

I’m configuring 2nd router, setup very similar to previous, 2 x ISP uplinks, DMZ 192.168.1.x and local net zone 192.168.0.x, this time without load-balancing. Copied most of the VyOS commands from 1st, yet encountered some strange artifacts.

Static routing for 2 ISP uplinks:
set protocols static route 0.0.0.0/0 next-hop ‘isp1.gw1.xx.xx’ distance 10
set protocols static route 0.0.0.0/0 next-hop ‘isp2.gw2.yy.yy’ distance 100

Zones defined as following:
set zone-policy zone ZONE-WAN interface eth0
set zone-policy zone ZONE-WAN interface eth1
set zone-policy zone ZONE-DMZ interface eth2
set zone-policy zone ZONE-INT interface eth3

DNAT from WAN to DMZ zone working OK (WAN and DMZ ports are the same).

  1. I have temporary server in INTERNAL zone which should be accessible from outside for same time.

    … other rules FW-WAN-INT …
    set firewall name FW-WAN-INT rule 510 action accept
    set firewall name FW-WAN-INT rule 510 protocol tcp
    set firewall name FW-WAN-INT rule 510 destination port 99
    …
    set zone-policy zone ZONE-INT from ZONE-WAN firewall name FW-WAN-INT

    set nat destination rule 510 description ‘WAN-eth1-TO-INT’
    set nat destination rule 510 protocol ‘tcp’
    set nat destination rule 510 destination port 99
    set nat destination rule 510 destination address isp1.ip1.xx.xx
    set nat destination rule 510 inbound-interface ‘eth0’
    set nat destination rule 510 translation address 192.168.0.xx
    set nat destination rule 510 translation port 9999

    set nat destination rule 511 description ‘WAN-eth1-TO-INT’
    set nat destination rule 511 protocol ‘tcp’
    set nat destination rule 511 destination port 99
    set nat destination rule 511 destination address isp1.ip1.xx.xx
    set nat destination rule 511 inbound-interface ‘eth1’
    set nat destination rule 511 translation address 192.168.0.xx
    set nat destination rule 511 translation port 9999

In previous router with load-balancing that worked. Yet in current, without load-balancing, it doesn’t.
If I change
set firewall name FW-WAN-INT rule 510 destination port 99
to
set firewall name FW-WAN-INT rule 510 destination port 99, 9999
it works.
What is a cause of such weird behavior ?

  1. Access from outside to DMZ server via eth0 works OK. From eth1 it starts to function only after I ping from router to gateway isp2.gw2.yy.yy via eth1. Of course, I can set timed cron script which ping gateway, yet is kinda awkward. Any idea why this happens?

Thanks in advance for any suggestion(s).

2

Firewall rules come after DNAT rules, therefore when you apply any filters on DNAT’ed targets you always have to use the internal IP addresses and ports.

Therefore you should use only the port 9999 (and the private IP address of the server).

(I seriously doubt that with the WAN-load-balancing feature the rule worked with the “public” port. Perhaps there was no firewall specified for the LAN.)

This sounds strange… I guess you have configured SNAT correctly?

Are you using any kind of “fault-detection” feature that might perhaps consider the eth1 link as down?

3

Something really strange. I restarted KVM appliance with VyOS, and now
set firewall name FW-WAN-INT rule 510 destination port 9999
works as expected.

If there were no firewall specified there will be no traffic in INT zone, since it is zone-based (not interface based) firewall setup. I copied most of the commands from my notepad, possibly I forgot to update it after made final working configuration on 1st router.

Seems so, everything works.

Yes, I wrote Bash shell script which pings both gateways each 30 sec and disables/enables routes when necessary with one of these

set protocols static route 0.0.0.0/0 next-hop ‘isp1.gw1.xx.xx’ disable
set protocols static route 0.0.0.0/0 next-hop ‘isp2.gw2.yy.yy’ disable

and logs these events with

logger $LOGPREFIX": Gateway #1 is down at "$NOW

I suppose its OK doing so, please correct if I’m wrong here.

System needs some more testing yet.
Unfortunately. as usually happens with my VyOS setups, when almost everything is done, something bad happens. Debian KVM host (with 2 VyOS instances) crashed twice badly, requiring hard reset. Common pattern seems to be juggling Ethernet links in order to check failover capabilities. After first freeze I changed KVM Ethernet devices from virtio to rtl, didn’t remove problem. Debian KVM host uses separate Ethernet interface for itself, +8 bridged devices.

auto br2_LTC1
iface br2_LTC1 inet manual
   bridge_ports enp16s0
   bridge_stp off
   bridge_waitpot 0
   bridge_fd 0.0
auto br2_BTC2
iface br2_BTC2 inet manual
   bridge_ports enp17s0
   bridge_stp off
   bridge_waitpot 0
   bridge_fd 0.0
auto br2_DMZ
iface br2_DMZ inet manual
   bridge_ports enp18s0
   bridge_stp off
   bridge_waitpot 0
   bridge_fd 0.0
auto br2_INT
iface br2_INT inet manual
   bridge_ports enp19s0
   bridge_stp off
   bridge_waitpot 0
   bridge_fd 0.0

Possibly some options need to be adjusted to keep bridge on with no regard of presence of Ethernet cable link, so KVM won’t shutdown/disconnect it? Anyone can suggest? Thanks.