WAN LB and MSS clamping issues

[harguna]

[solved] WAN load-balancing with `policy route` rules [previously: WAN load-balancing with 2 PPPoE connections with TCP-MSS clamping]

Update It seems I was able to “hack” my way through this problem and solve it as described in my last comment. Original It seems that WAN load-balancing and TCP-MSS clamping policies don’t mix well. Has anyone managed to correctly configure such a setup? Is there an alternative setup? (Perhaps with different routing tables?) From what I gather the WAN load-balancing feature introduces -j ACCEPT rules in the -t mangle table, which come before the TCP-MSS clamping rules generated by the…

MSS Clamping / Possible Issue or I'm doing something incorrect

I think the VYOS team should come out with a robust TCP MSS clamping recommendation. After reading the above comments and also the VYOS knowledgebase on TCP MSS clamping for PPPOE connections: https://support.vyos.io/en/kb/articles/pppoe-sub-interfaces-2#comments I see that both of these point the user to setting the TCP MSS clamping on the LAN interface, whereas the actual clamping should take place on the WAN interface. There is another option referred to by a Ubiquity thread: https://commu…

These posts seem similar to mine when traffic going to tun0 going to Cloudflare via GRE can connect but fails when upgrading to HTTP. Without WAN LB this works fine, just wondering why is this still a thing?

iptables -t mangle -S

-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N VYATTA_FW_OUT_HOOK
-N VYATTA_FW_IN_HOOK
-N VYATTA_FW_LOCALOUT_HOOK
-N magic-wan
-N VYATTA_PBR_100
-N VYOS_FW_OPTIONS
-N WANLOADBALANCE_PRE
-N ISP_eth0
-N ISP_eth0_IN
-N ISP_eth1
-N ISP_eth1_IN
-A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN
-A PREROUTING -i eth0 -m state --state NEW -j ISP_eth0_IN
-A PREROUTING -j WANLOADBALANCE_PRE
-A PREROUTING -j VYATTA_FW_IN_HOOK
-A FORWARD -j VYOS_FW_OPTIONS
-A OUTPUT -j VYATTA_FW_LOCALOUT_HOOK
-A POSTROUTING -j VYATTA_FW_OUT_HOOK
-A VYATTA_FW_IN_HOOK -i eth2.200 -j magic-wan
-A magic-wan -s xxx.xxx.71.3/32 -p tcp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.3/32 -p tcp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -s xxx.xxx.71.3/32 -p udp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.3/32 -p udp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -s xxx.xxx.71.3/32 -p tcp -m tcp --tcp-flags SYN SYN -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.3/32 -p tcp -m tcp --tcp-flags SYN SYN -m comment --comment magic-wan-100 -j TCPMSS --set-mss 1436
-A magic-wan -s xxx.xxx.71.3/32 -p udp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.3/32 -p udp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -s xxx.xxx.69.6/32 -p udp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.69.6/32 -p udp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -s xxx.xxx.71.4/32 -p udp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.4/32 -p udp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -s xxx.xxx.71.3/32 -p udp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.3/32 -p udp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -s xxx.xxx.71.5/32 -p udp -m comment --comment magic-wan-100 -j LOG --log-prefix "[magic-wan-100-] "
-A magic-wan -s xxx.xxx.71.5/32 -p udp -m comment --comment magic-wan-100 -j VYATTA_PBR_100
-A magic-wan -m comment --comment "magic-wan-1000000 default-action accept" -j LOG --log-prefix "[magic-wan-default-A]"
-A magic-wan -m comment --comment "magic-wan-1000000 default-action accept" -j RETURN
-A VYATTA_PBR_100 -j MARK --set-xmark 0x80000063/0xffffffff
-A VYATTA_PBR_100 -j ACCEPT
-A VYOS_FW_OPTIONS -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436
-A WANLOADBALANCE_PRE -s xxx.xxx.69.0/24 -d xxx.xxx.70.0/24 -i eth2 -j ACCEPT
-A WANLOADBALANCE_PRE -s xxx.xxx.69.0/24 -d xxx.xxx.71.0/24 -i eth2 -j ACCEPT
-A WANLOADBALANCE_PRE -d xxx.xxx.72.20/31 -i eth2.100 -j ACCEPT
-A WANLOADBALANCE_PRE -d xxx.xxx.72.20/31 -i eth2.200 -j ACCEPT
-A WANLOADBALANCE_PRE -d xxx.xxx.72.20/31 -i eth2 -j ACCEPT
-A WANLOADBALANCE_PRE -i tun0 -m state --state NEW -j ISP_eth0
-A WANLOADBALANCE_PRE -i tun0 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A WANLOADBALANCE_PRE -i eth2.100 -m state --state NEW -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth2.100 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A WANLOADBALANCE_PRE -i eth2.200 -m state --state NEW -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth2.200 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A WANLOADBALANCE_PRE -i eth2 -m state --state NEW -j ISP_eth0
-A WANLOADBALANCE_PRE -i eth2 -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A ISP_eth0 -j CONNMARK --set-xmark 0xc9/0xffffffff
-A ISP_eth0 -j MARK --set-xmark 0xc9/0xffffffff
-A ISP_eth0 -j ACCEPT
-A ISP_eth0_IN -j CONNMARK --set-xmark 0xc9/0xffffffff
-A ISP_eth1 -j CONNMARK --set-xmark 0xca/0xffffffff
-A ISP_eth1 -j MARK --set-xmark 0xca/0xffffffff
-A ISP_eth1 -j ACCEPT
-A ISP_eth1_IN -j CONNMARK --set-xmark 0xca/0xffffffff

[harguna]

Config:

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/29'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group network-group cf-ipv4 network 'xxx.xxx.48.0/20'
set firewall group network-group cf-ipv4 network 'xxx.xxx.244.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.200.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.4.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.64.0/18'
set firewall group network-group cf-ipv4 network 'xxx.xxx.192.0/18'
set firewall group network-group cf-ipv4 network 'xxx.xxx.240.0/20'
set firewall group network-group cf-ipv4 network 'xxx.xxx.96.0/20'
set firewall group network-group cf-ipv4 network 'xxx.xxx.240.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.128.0/17'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/15'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/13'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/14'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/13'
set firewall group network-group cf-ipv4 network 'xxx.xxx.72.0/22'
set firewall ipv6-name EXTERNAL-IN-v6 default-action 'drop'
set firewall ipv6-name EXTERNAL-IN-v6 enable-default-log
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 action 'accept'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 log 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 state established 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 state related 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 action 'accept'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 destination port '80,443'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 log 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 protocol 'tcp_udp'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 source group network-group 'cf-ipv6'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 default-action 'drop'
set firewall ipv6-name EXTERNAL-LOCAL-v6 enable-default-log
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 state established 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 state related 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 icmpv6 type 'echo-request'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 protocol 'icmpv6'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 action 'drop'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 description 'ssh'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 destination port '22'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 protocol 'tcp'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 recent count '15'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 recent time '60'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 destination port '22'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 protocol 'tcp'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 state new 'enable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name EXTERNAL-IN default-action 'drop'
set firewall name EXTERNAL-IN enable-default-log
set firewall name EXTERNAL-IN rule 10 action 'accept'
set firewall name EXTERNAL-IN rule 10 log 'enable'
set firewall name EXTERNAL-IN rule 10 state established 'enable'
set firewall name EXTERNAL-IN rule 10 state related 'enable'
set firewall name EXTERNAL-IN rule 20 action 'accept'
set firewall name EXTERNAL-IN rule 20 description 'servarr-vlan200'
set firewall name EXTERNAL-IN rule 20 destination address 'xxx.xxx.71.2'
set firewall name EXTERNAL-IN rule 20 destination port '80,443'
set firewall name EXTERNAL-IN rule 20 log 'enable'
set firewall name EXTERNAL-IN rule 20 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 20 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-IN rule 20 state new 'enable'
set firewall name EXTERNAL-IN rule 21 action 'drop'
set firewall name EXTERNAL-IN rule 21 description 'bind-vlan200'
set firewall name EXTERNAL-IN rule 21 destination address 'xxx.xxx.71.2'
set firewall name EXTERNAL-IN rule 21 destination port '5053'
set firewall name EXTERNAL-IN rule 21 log 'enable'
set firewall name EXTERNAL-IN rule 21 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 21 recent count '100'
set firewall name EXTERNAL-IN rule 21 recent time '60'
set firewall name EXTERNAL-IN rule 21 state new 'enable'
set firewall name EXTERNAL-IN rule 22 action 'accept'
set firewall name EXTERNAL-IN rule 22 description 'bind-vlan200'
set firewall name EXTERNAL-IN rule 22 destination address 'xxx.xxx.71.2'
set firewall name EXTERNAL-IN rule 22 destination port '5053'
set firewall name EXTERNAL-IN rule 22 log 'enable'
set firewall name EXTERNAL-IN rule 22 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 22 state new 'enable'
set firewall name EXTERNAL-IN rule 30 action 'accept'
set firewall name EXTERNAL-IN rule 30 description 'kvm'
set firewall name EXTERNAL-IN rule 30 destination address 'xxx.xxx.69.6'
set firewall name EXTERNAL-IN rule 30 destination port '80,443'
set firewall name EXTERNAL-IN rule 30 log 'enable'
set firewall name EXTERNAL-IN rule 30 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 30 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-IN rule 30 state new 'enable'
set firewall name EXTERNAL-LOCAL default-action 'drop'
set firewall name EXTERNAL-LOCAL enable-default-log
set firewall name EXTERNAL-LOCAL rule 10 action 'accept'
set firewall name EXTERNAL-LOCAL rule 10 log 'enable'
set firewall name EXTERNAL-LOCAL rule 10 state established 'enable'
set firewall name EXTERNAL-LOCAL rule 10 state related 'enable'
set firewall name EXTERNAL-LOCAL rule 20 action 'accept'
set firewall name EXTERNAL-LOCAL rule 20 icmp type-name 'echo-request'
set firewall name EXTERNAL-LOCAL rule 20 log 'enable'
set firewall name EXTERNAL-LOCAL rule 20 protocol 'icmp'
set firewall name EXTERNAL-LOCAL rule 20 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 30 action 'drop'
set firewall name EXTERNAL-LOCAL rule 30 description 'ssh'
set firewall name EXTERNAL-LOCAL rule 30 destination port '22'
set firewall name EXTERNAL-LOCAL rule 30 log 'enable'
set firewall name EXTERNAL-LOCAL rule 30 protocol 'tcp'
set firewall name EXTERNAL-LOCAL rule 30 recent count '15'
set firewall name EXTERNAL-LOCAL rule 30 recent time '60'
set firewall name EXTERNAL-LOCAL rule 30 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 31 action 'accept'
set firewall name EXTERNAL-LOCAL rule 31 destination port '22'
set firewall name EXTERNAL-LOCAL rule 31 log 'enable'
set firewall name EXTERNAL-LOCAL rule 31 protocol 'tcp'
set firewall name EXTERNAL-LOCAL rule 31 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 40 action 'accept'
set firewall name EXTERNAL-LOCAL rule 40 description 'magic-wan'
set firewall name EXTERNAL-LOCAL rule 40 log 'enable'
set firewall name EXTERNAL-LOCAL rule 40 protocol 'gre'
set firewall name EXTERNAL-LOCAL rule 40 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-LOCAL rule 50 action 'accept'
set firewall name EXTERNAL-LOCAL rule 50 icmp type-name 'echo-reply'
set firewall name EXTERNAL-LOCAL rule 50 log 'enable'
set firewall name EXTERNAL-LOCAL rule 50 protocol 'icmp'
set firewall name VLAN-100 default-action 'accept'
set firewall name VLAN-100 enable-default-log
set firewall name VLAN-100 rule 10 action 'accept'
set firewall name VLAN-100 rule 10 log 'enable'
set firewall name VLAN-100 rule 10 state established 'enable'
set firewall name VLAN-100 rule 10 state related 'enable'
set firewall name VLAN-100 rule 30 action 'accept'
set firewall name VLAN-100 rule 30 description 'Printer access'
set firewall name VLAN-100 rule 30 destination address 'xxx.xxx.69.12'
set firewall name VLAN-100 rule 31 action 'accept'
set firewall name VLAN-100 rule 31 description 'Pihole DNS'
set firewall name VLAN-100 rule 31 destination address 'xxx.xxx.69.7'
set firewall name VLAN-100 rule 31 destination port '53'
set firewall name VLAN-100 rule 31 protocol 'tcp_udp'
set firewall name VLAN-100 rule 40 action 'drop'
set firewall name VLAN-100 rule 40 description 'Restrict Access to INTERNAL1 network'
set firewall name VLAN-100 rule 40 destination address 'xxx.xxx.69.0/24'
set firewall name VLAN-100 rule 41 action 'drop'
set firewall name VLAN-100 rule 41 description 'Restrict Access to VLAN200 network'
set firewall name VLAN-100 rule 41 destination address 'xxx.xxx.71.0/24'
set firewall name VLAN-200 default-action 'accept'
set firewall name VLAN-200 enable-default-log
set firewall name VLAN-200 rule 10 action 'accept'
set firewall name VLAN-200 rule 10 log 'enable'
set firewall name VLAN-200 rule 10 state established 'enable'
set firewall name VLAN-200 rule 10 state related 'enable'
set firewall name VLAN-200 rule 30 action 'accept'
set firewall name VLAN-200 rule 30 description 'Printer access'
set firewall name VLAN-200 rule 30 destination address 'xxx.xxx.69.12'
set firewall name VLAN-200 rule 31 action 'accept'
set firewall name VLAN-200 rule 31 description 'Pihole DNS'
set firewall name VLAN-200 rule 31 destination address 'xxx.xxx.69.7'
set firewall name VLAN-200 rule 31 destination port '53'
set firewall name VLAN-200 rule 31 protocol 'tcp_udp'
set firewall name VLAN-200 rule 32 action 'accept'
set firewall name VLAN-200 rule 32 description 'ERFI1 Access'
set firewall name VLAN-200 rule 32 destination address 'xxx.xxx.69.3'
set firewall name VLAN-200 rule 32 protocol 'tcp_udp'
set firewall name VLAN-200 rule 40 action 'drop'
set firewall name VLAN-200 rule 40 description 'Restrict Access to INTERNAL1 network'
set firewall name VLAN-200 rule 40 destination address 'xxx.xxx.69.0/24'
set firewall name VLAN-200 rule 41 action 'drop'
set firewall name VLAN-200 rule 41 description 'Restrict Access to VLAN100 network'
set firewall name VLAN-200 rule 41 destination address 'xxx.xxx.70.0/24'
set firewall options interface tun0 adjust-mss '1436'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'EXTERNAL1'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces ethernet eth0 firewall in name 'EXTERNAL-IN'
set interfaces ethernet eth0 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces ethernet eth0 firewall local name 'EXTERNAL-LOCAL'
set interfaces ethernet eth0 hw-id 'xx:xx:xx:xx:xx:de'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address 'dhcp'
set interfaces ethernet eth1 description 'EXTERNAL2'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces ethernet eth1 firewall in name 'EXTERNAL-IN'
set interfaces ethernet eth1 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces ethernet eth1 firewall local name 'EXTERNAL-LOCAL'
set interfaces ethernet eth1 hw-id 'xx:xx:xx:xx:xx:df'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 address 'xxx.xxx.69.1/24'
set interfaces ethernet eth2 description 'INTERNAL1'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 hw-id 'xx:xx:xx:xx:xx:e0'
set interfaces ethernet eth2 ip arp-cache-timeout '30'
set interfaces ethernet eth2 speed 'auto'
set interfaces ethernet eth2 vif 100 address 'xxx.xxx.70.1/24'
set interfaces ethernet eth2 vif 100 description 'asus'
set interfaces ethernet eth2 vif 100 firewall in name 'VLAN-100'
set interfaces ethernet eth2 vif 200 address 'xxx.xxx.71.1/24'
set interfaces ethernet eth2 vif 200 description 'servarr'
set interfaces ethernet eth2 vif 200 firewall in name 'VLAN-200'
set interfaces ethernet eth2 vif 200 policy route 'magic-wan'
set interfaces ethernet eth3 address 'xxx.xxx.73.1/24'
set interfaces ethernet eth3 description 'INTERNAL2'
set interfaces ethernet eth3 duplex 'auto'
set interfaces ethernet eth3 hw-id 'xx:xx:xx:xx:xx:e1'
set interfaces ethernet eth3 speed 'auto'
set interfaces loopback lo
set interfaces tunnel tun0 address 'xxx.xxx.72.20/31'
set interfaces tunnel tun0 description 'magic-wan'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 mtu '1476'
set interfaces tunnel tun0 remote 'xxx.xxx.66.5'
set interfaces tunnel tun0 source-address 'xxx.xxx.189.102'
set load-balancing wan flush-connections
set load-balancing wan interface-health eth0 failure-count '2'
set load-balancing wan interface-health eth0 nexthop 'dhcp'
set load-balancing wan interface-health eth0 success-count '1'
set load-balancing wan interface-health eth0 test 10 resp-time '5'
set load-balancing wan interface-health eth0 test 10 target 'xxx.xxx.8.8'
set load-balancing wan interface-health eth0 test 10 ttl-limit '1'
set load-balancing wan interface-health eth0 test 10 type 'ping'
set load-balancing wan interface-health eth0 test 20 resp-time '5'
set load-balancing wan interface-health eth0 test 20 target 'xxx.xxx.1.1'
set load-balancing wan interface-health eth0 test 20 ttl-limit '1'
set load-balancing wan interface-health eth0 test 20 type 'ping'
set load-balancing wan interface-health eth1 failure-count '2'
set load-balancing wan interface-health eth1 nexthop 'dhcp'
set load-balancing wan interface-health eth1 success-count '1'
set load-balancing wan interface-health eth1 test 10 resp-time '5'
set load-balancing wan interface-health eth1 test 10 target 'xxx.xxx.8.8'
set load-balancing wan interface-health eth1 test 10 ttl-limit '1'
set load-balancing wan interface-health eth1 test 10 type 'ping'
set load-balancing wan interface-health eth1 test 20 resp-time '5'
set load-balancing wan interface-health eth1 test 20 target 'xxx.xxx.1.1'
set load-balancing wan interface-health eth1 test 20 ttl-limit '1'
set load-balancing wan interface-health eth1 test 20 type 'ping'
set load-balancing wan rule 10 description 'vlan-exclusion-100'
set load-balancing wan rule 10 destination address 'xxx.xxx.70.1/24'
set load-balancing wan rule 10 exclude
set load-balancing wan rule 10 inbound-interface 'eth2'
set load-balancing wan rule 10 protocol 'all'
set load-balancing wan rule 10 source address 'xxx.xxx.69.1/24'
set load-balancing wan rule 20 description 'vlan-exclusion-200'
set load-balancing wan rule 20 destination address 'xxx.xxx.71.1/24'
set load-balancing wan rule 20 exclude
set load-balancing wan rule 20 inbound-interface 'eth2'
set load-balancing wan rule 20 protocol 'all'
set load-balancing wan rule 20 source address 'xxx.xxx.69.1/24'
set load-balancing wan rule 30 description 'tun0-exclusion-vlan100'
set load-balancing wan rule 30 destination address 'xxx.xxx.72.20/31'
set load-balancing wan rule 30 exclude
set load-balancing wan rule 30 inbound-interface 'eth2.100'
set load-balancing wan rule 30 protocol 'all'
set load-balancing wan rule 31 description 'tun0-exclusion-vlan200'
set load-balancing wan rule 31 destination address 'xxx.xxx.72.20/31'
set load-balancing wan rule 31 exclude
set load-balancing wan rule 31 inbound-interface 'eth2.200'
set load-balancing wan rule 31 protocol 'all'
set load-balancing wan rule 32 description 'tun0-exclusion-eth2'
set load-balancing wan rule 32 destination address 'xxx.xxx.72.20/31'
set load-balancing wan rule 32 exclude
set load-balancing wan rule 32 inbound-interface 'eth2'
set load-balancing wan rule 32 protocol 'all'
set load-balancing wan rule 40 failover
set load-balancing wan rule 40 inbound-interface 'tun0'
set load-balancing wan rule 40 interface eth0 weight '10'
set load-balancing wan rule 40 interface eth1 weight '1'
set load-balancing wan rule 40 protocol 'all'
set load-balancing wan rule 50 failover
set load-balancing wan rule 50 inbound-interface 'eth2.100'
set load-balancing wan rule 50 interface eth0 weight '10'
set load-balancing wan rule 50 interface eth1 weight '1'
set load-balancing wan rule 50 protocol 'all'
set load-balancing wan rule 51 failover
set load-balancing wan rule 51 inbound-interface 'eth2.200'
set load-balancing wan rule 51 interface eth0 weight '10'
set load-balancing wan rule 51 interface eth1 weight '1'
set load-balancing wan rule 51 protocol 'all'
set load-balancing wan rule 52 failover
set load-balancing wan rule 52 inbound-interface 'eth2'
set load-balancing wan rule 52 interface eth0 weight '10'
set load-balancing wan rule 52 interface eth1 weight '1'
set load-balancing wan rule 52 protocol 'all'
set load-balancing wan sticky-connections inbound
set nat destination rule 10 description 'servarr-vlan200-eth0'
set nat destination rule 10 destination port '80,443'
set nat destination rule 10 inbound-interface 'eth0'
set nat destination rule 10 log 'enable'
set nat destination rule 10 protocol 'tcp_udp'
set nat destination rule 10 translation address 'xxx.xxx.71.2'
set nat destination rule 11 description 'bind-vlan200-eth0'
set nat destination rule 11 destination port '5053'
set nat destination rule 11 inbound-interface 'eth0'
set nat destination rule 11 protocol 'tcp_udp'
set nat destination rule 11 translation address 'xxx.xxx.71.2'
set nat destination rule 11 translation port '5053'
set nat destination rule 20 description 'servarr-vlan200-eth1'
set nat destination rule 20 destination port '80,443'
set nat destination rule 20 inbound-interface 'eth1'
set nat destination rule 20 log 'enable'
set nat destination rule 20 protocol 'tcp_udp'
set nat destination rule 20 translation address 'xxx.xxx.71.2'
set nat destination rule 21 description 'bind-vlan200-eth1'
set nat destination rule 21 destination port '5053'
set nat destination rule 21 inbound-interface 'eth1'
set nat destination rule 21 protocol 'tcp_udp'
set nat destination rule 21 translation address 'xxx.xxx.71.2'
set nat destination rule 21 translation port '5053'
set nat destination rule 30 description 'kvm-eth0'
set nat destination rule 30 destination port '2053'
set nat destination rule 30 inbound-interface 'eth0'
set nat destination rule 30 log 'enable'
set nat destination rule 30 protocol 'tcp_udp'
set nat destination rule 30 translation address 'xxx.xxx.69.6'
set nat destination rule 30 translation port '443'
set nat destination rule 40 description 'kvm-eth1'
set nat destination rule 40 destination port '2053'
set nat destination rule 40 inbound-interface 'eth1'
set nat destination rule 40 log 'enable'
set nat destination rule 40 protocol 'tcp_udp'
set nat destination rule 40 translation address 'xxx.xxx.69.6'
set nat destination rule 40 translation port '443'
set nat source rule 100 description 'eth0'
set nat source rule 100 log 'enable'
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 source address 'xxx.xxx.0.0/16'
set nat source rule 100 translation address 'masquerade'
set nat source rule 200 description 'eth1'
set nat source rule 200 log 'enable'
set nat source rule 200 outbound-interface 'eth1'
set nat source rule 200 source address 'xxx.xxx.0.0/16'
set nat source rule 200 translation address 'masquerade'
set policy route magic-wan enable-default-log
set policy route magic-wan rule 100 description 'magic-wan'
set policy route magic-wan rule 100 log 'enable'
set policy route magic-wan rule 100 protocol 'tcp_udp'
set policy route magic-wan rule 100 set table '100'
set policy route magic-wan rule 100 source address 'xxx.xxx.71.3'
set protocols static table 100 route xxx.xxx.0.0/0 next-hop xxx.xxx.72.21
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 default-router 'xxx.xxx.69.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 lease '300'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 name-server 'xxx.xxx.69.7'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 name-server 'xxx.xxx.69.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 range 0 start 'xxx.xxx.69.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 range 0 stop 'xxx.xxx.69.254'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.3'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:b6'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.6'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:33'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.7'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:64'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.4'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:28'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 default-router 'xxx.xxx.73.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 lease '300'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 name-server 'xxx.xxx.69.7'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 name-server 'xxx.xxx.73.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 range 0 start 'xxx.xxx.73.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.73.0/24 range 0 stop 'xxx.xxx.73.254'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 default-router 'xxx.xxx.70.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 lease '300'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 name-server 'xxx.xxx.69.7'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 name-server 'xxx.xxx.70.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 range 0 start 'xxx.xxx.70.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 range 0 stop 'xxx.xxx.70.254'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.70.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:d8'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 default-router 'xxx.xxx.71.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 lease '300'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 name-server 'xxx.xxx.69.7'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 name-server 'xxx.xxx.71.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 range 0 start 'xxx.xxx.71.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 range 0 stop 'xxx.xxx.71.254'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.71.4'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:53'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.71.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:07'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.71.3'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:c9'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.71.5'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.71.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:1c'
set service dns forwarding allow-from 'xxx.xxx.0.0/16'
set service dns forwarding cache-size '0'
set service dns forwarding listen-address 'xxx.xxx.69.1'
set service dns forwarding listen-address 'xxx.xxx.70.1'
set service dns forwarding listen-address 'xxx.xxx.71.1'
set service dns forwarding listen-address 'xxx.xxx.73.1'
set service ssh disable-password-authentication
set service ssh loglevel 'verbose'
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication public-keys [email protected] key xxxxxx
set system login user xxxxxx authentication public-keys [email protected] type 'ecdsa-sha2-nistp256'
set system name-server 'xxx.xxx.69.1'
set system name-server 'xxx.xxx.1.1'
set system name-server 'xxx.xxx.8.8'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.70.2'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.3'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.6'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.7'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.1'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.71.4'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.71.2'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.4'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.71.3'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.71.5'
set system sysctl custom net.ipv4.conf.all.accept_local value '1'
set system syslog global facility all level 'all'
set system syslog global facility protocols level 'all'
set system time-zone 'Asia/Singapore'

[zsdc]

Can you explain the situation a bit more?
I may guess that magic-wan PBR does not work for traffic excluded from wan load-balancing. Or is this something else?

[harguna]

I may guess that magic-wan PBR does not work for traffic excluded from wan load-balancing. Or is this something else?

It still routes the traffic (meaning that I could initate a connection but it doesn’t upgrade to HTTP) but I think the WAN LB routing might not work with MSS clamping on the tun0 interface. The exclude traffic is only between the local interfaces because if not, they’ll go via WAN LB and it won’t be able to reach each other, hence why there are so many exclude rules.

[harguna]

Is there more information you need?

[zsdc]

Actually, yes. I still cannot get what is wrong here. I mean that it is clear that some HTTP connection does not work. But which one, from and to where, etc.?..

So, I think that you going to need to provide step-by-step info on how to reproduce this on a newly installed VyOS instance.

Like:

1. I have a network with the next topology: [topology map here]
2. You need to use VyOS 1.X.X on a router.
3. Load the next configuration (without masking details).
4. Try to fetch URL XXX via HTTP from host X connected to interface X.
5. I expect X but see X instead.

With this info, theoretically, it should be possible to reproduce the problem and see why it appears.

[harguna]

I mean that it is clear that some HTTP connection does not work. But which one, from and to where, etc.?..

From a VM on eth2.200 xxx.xxx.71.3 with is routed to tun0 via the PBR magic-wan

    ethernet eth2 {
        address xxx.xxx.69.1/24
        description INTERNAL1
        duplex auto
        hw-id xxx:xxx:xxx:2d:eb:e0
        ip {
            arp-cache-timeout 30
        }
        speed auto
        vif 100 {
            address xxx.xxx.70.1/24
            description asus
            firewall {
                in {
                    name VLAN-100
                }
            }
        }
        vif 200 {
            address xxx.xxx.71.1/24
            description servarr
            firewall {
                in {
                    name VLAN-200
                }
            }
            policy {
                route magic-wan
            }
        }
    }

policy {
    route magic-wan {
        enable-default-log
        rule 100 {
            description magic-wan
            destination {
                port 80,443
            }
            log enable
            protocol tcp_udp
            set {
                table 100
            }
            source {
                address xxx.xxx.71.3
            }
        }
    }
}
protocols {
    static {
        table 100 {
            route 0.0.0.0/0 {
                next-hop xxx.xxx.72.21 {
                }
            }
        }
    }
}

    options {
        interface tun0 {
            adjust-mss 1436
        }
    }

tunnel tun0 {
        address xxx.xxx.72.20/31 - .21 is Cloudflare's side of the tunnel
        description magic-wan
        encapsulation gre
        mtu 1476
        remote xxx.xxx.66.5 - CF endpoint
        source-address xxx.xxx.189.102 - my static public IP address
    }
}

Release version: 1.3-rolling-202202261129

I can’t provide the rest cause you’d need access to Cloudflare’s endpoints. The GRE tunnel via tun0 works when there’s no WAN LB essentially and the only thing that stands out is that Cloudflare needs an MTU size of 1476 and MSS of 1436

I think all you need to reproduce this on a box is to have WAN LB (even if you only have one eth on WAN LB, it still won’t work, it will only work when there’s no LB config at all) and route any traffic to another gateway via GRE and apply MSS clamping like I did, make sure the other interface that you’re accepts only that MSS so that you can simulate this, then try accessing websites via the client/device that you’re routing to via the PBR, you can also just try routing all devices on that interface without setting a source IP.

[zsdc]

I think that firewall options are not the reason for the problem. As I mentioned in the first message - LB can break PBR. I created a task to track the problem: ⚓ T4452 WAN load-balancing exclude rules break PBR

For testing purposes, you can try to configure load-balancing in a way when it will not contains any rules (match or exclude) that will catch traffic to www.cloudflare.com and see if this helps.

[harguna]

Same thing, I can connect on the TCP layer but HTTP doesn’t work, and WAN LB can’t be used unless I have one rule with an outbound-interface so I just left the failover rules in and left the rest out.

Also without the exclude rules the clients on the network can’t talk to each other cause they get routed over the WAN LB.

enable-local-traffic messes with the routing too

[zsdc]

From everything that we can see here, there is one real problem mentioned above. The rest is most likely a misconfiguration. Load-balancing is like a hammer - a very powerful tool, but we must be careful and know how it works very well to use it without impact on other features. Would be great to have protection from every mistake, however, it is not as easy as with simpler features.

Back to workarounds. I still have no idea how TCP can work but not HTTP, since VyOS as a router does not care about anything that is above the transport layer. So, if HTTP does not work - TCP does not work as well, but it can be not so obvious or easy to detect.
Currently, you must use only one - PBR or LB. They will not work together. You can replace PBR with:

set load-balancing wan interface-health tun0 nexthop 'xxx.xxx.72.21'
set load-balancing wan interface-health tun0 test 10 target 'xxx.xxx.72.21'
set load-balancing wan rule 50 failover
set load-balancing wan rule 50 inbound-interface 'eth2.200'
set load-balancing wan rule 50 interface tun0
set load-balancing wan rule 50 source address 'xxx.xxx.71.3/32'

[harguna]

@zsdc

Back to workarounds. I still have no idea how TCP can work but not HTTP

Yeah you’re right.

@zsdc

set load-balancing wan interface-health tun0 nexthop 'xxx.xxx.72.21'
set load-balancing wan interface-health tun0 test 10 target 'xxx.xxx.72.21'
set load-balancing wan rule 50 failover
set load-balancing wan rule 50 inbound-interface 'eth2.200'
set load-balancing wan rule 50 interface tun0
set load-balancing wan rule 50 source address 'xxx.xxx.71.3/32'

So I did this, I could access the internet via the VM, but it was not being routed through the tunnel. I couldn’t ping xxx.xxx.72.21 via the VM either. So normally when it works I get a different egress IP, as I’d be proxied via Cloudflare through the GRE tunnel as the on-ramp.

0     0 ISP_tun0   all  --  eth2.200 *       10.68.71.3           0.0.0.0/0            state NEW
8109  702K CONNMARK   all  --  eth2.200 *       10.68.71.3           0.0.0.0/0            CONNMARK restore

[matthewr]

This is rather weird, and not helped by the OP not having provided a full config. Indeed the partial config does not make it clear how the MSS is being clamped. Also, there appears to be no WAN Load Balancing but only policy routing in the OP’s config extract.

@zsdc

I still have no idea how TCP can work but not HTTP

My guess (which may be very wrong) is what the OP is trying to say is that the TCP connection forms correctly but that subsequent traffic does not flow. This sounds like the classic symptom of a “Path MTU Blackhole”: it works for small packets and breaks with larger packets.

Correct me if I am mistaken, but the MSS clamping should be done on the INGRESS interface, which in this case is eth1.100 and NOT on tun0.

Could this be related to the problem? It would be easy to confirm with tcpdump whether the MSS is being set on outbound TCP connections.

[zsdc]

Yes, true. Without a full unmasked config, this is just guessing, not debugging… I am happy that we were able to identify at least one problem.

In VyOS 1.3 MSS clamping is done in the forward hook, so we can use the output interface as a classifier for packets. It looks like this (displayed only related to MSS part):

table ip mangle {
	chain FORWARD {
		type filter hook forward priority mangle; policy accept;
		counter packets 0 bytes 0 jump VYOS_FW_OPTIONS
	}
	chain VYOS_FW_OPTIONS {
		oifname "tun0" meta l4proto tcp tcp flags & (syn|rst) == syn counter packets 0 bytes 0 tcp option maxseg size set 1436
	}
}

[harguna]

The full config, I did still strip away some like my actual IP and the CF remote IP:

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group ipv6-network-group cf-ipv6 network '2400:cb00::/32'
set firewall group ipv6-network-group cf-ipv6 network '2606:4700::/32'
set firewall group ipv6-network-group cf-ipv6 network '2803:f800::/32'
set firewall group ipv6-network-group cf-ipv6 network '2405:b500::/32'
set firewall group ipv6-network-group cf-ipv6 network '2405:8100::/32'
set firewall group ipv6-network-group cf-ipv6 network '2a06:98c0::/29'
set firewall group ipv6-network-group cf-ipv6 network '2c0f:f248::/32'
set firewall group network-group cf-ipv4 network '173.245.48.0/20'
set firewall group network-group cf-ipv4 network '103.21.244.0/22'
set firewall group network-group cf-ipv4 network '103.22.200.0/22'
set firewall group network-group cf-ipv4 network '103.31.4.0/22'
set firewall group network-group cf-ipv4 network '141.101.64.0/18'
set firewall group network-group cf-ipv4 network '108.162.192.0/18'
set firewall group network-group cf-ipv4 network '190.93.240.0/20'
set firewall group network-group cf-ipv4 network '188.114.96.0/20'
set firewall group network-group cf-ipv4 network '197.234.240.0/22'
set firewall group network-group cf-ipv4 network '198.41.128.0/17'
set firewall group network-group cf-ipv4 network '162.158.0.0/15'
set firewall group network-group cf-ipv4 network '104.16.0.0/13'
set firewall group network-group cf-ipv4 network '104.24.0.0/14'
set firewall group network-group cf-ipv4 network '172.64.0.0/13'
set firewall group network-group cf-ipv4 network '131.0.72.0/22'
set firewall ipv6-name EXTERNAL-IN-v6 default-action 'drop'
set firewall ipv6-name EXTERNAL-IN-v6 enable-default-log
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 action 'accept'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 log 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 state established 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 state related 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 action 'accept'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 destination port '80,443'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 log 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 protocol 'tcp_udp'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 source group network-group 'cf-ipv6'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 default-action 'drop'
set firewall ipv6-name EXTERNAL-LOCAL-v6 enable-default-log
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 state established 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 state related 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 icmpv6 type 'echo-request'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 protocol 'icmpv6'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 action 'drop'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 description 'ssh'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 destination port '22'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 protocol 'tcp'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 recent count '15'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 recent time '60'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 destination port '22'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 protocol 'tcp'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 state new 'enable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name EXTERNAL-IN default-action 'drop'
set firewall name EXTERNAL-IN enable-default-log
set firewall name EXTERNAL-IN rule 10 action 'accept'
set firewall name EXTERNAL-IN rule 10 log 'enable'
set firewall name EXTERNAL-IN rule 10 state established 'enable'
set firewall name EXTERNAL-IN rule 10 state related 'enable'
set firewall name EXTERNAL-IN rule 20 action 'accept'
set firewall name EXTERNAL-IN rule 20 description 'servarr-vlan200'
set firewall name EXTERNAL-IN rule 20 destination address '10.68.71.2'
set firewall name EXTERNAL-IN rule 20 destination port '80,443'
set firewall name EXTERNAL-IN rule 20 log 'enable'
set firewall name EXTERNAL-IN rule 20 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 20 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-IN rule 20 state new 'enable'
set firewall name EXTERNAL-IN rule 21 action 'drop'
set firewall name EXTERNAL-IN rule 21 description 'bind-vlan200'
set firewall name EXTERNAL-IN rule 21 destination address '10.68.71.2'
set firewall name EXTERNAL-IN rule 21 destination port '5053'
set firewall name EXTERNAL-IN rule 21 log 'enable'
set firewall name EXTERNAL-IN rule 21 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 21 recent count '100'
set firewall name EXTERNAL-IN rule 21 recent time '60'
set firewall name EXTERNAL-IN rule 21 state new 'enable'
set firewall name EXTERNAL-IN rule 22 action 'accept'
set firewall name EXTERNAL-IN rule 22 description 'bind-vlan200'
set firewall name EXTERNAL-IN rule 22 destination address '10.68.71.2'
set firewall name EXTERNAL-IN rule 22 destination port '5053'
set firewall name EXTERNAL-IN rule 22 log 'enable'
set firewall name EXTERNAL-IN rule 22 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 22 state new 'enable'
set firewall name EXTERNAL-IN rule 30 action 'accept'
set firewall name EXTERNAL-IN rule 30 description 'kvm'
set firewall name EXTERNAL-IN rule 30 destination address '10.68.69.6'
set firewall name EXTERNAL-IN rule 30 destination port '80,443'
set firewall name EXTERNAL-IN rule 30 log 'enable'
set firewall name EXTERNAL-IN rule 30 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 30 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-IN rule 30 state new 'enable'
set firewall name EXTERNAL-LOCAL default-action 'drop'
set firewall name EXTERNAL-LOCAL enable-default-log
set firewall name EXTERNAL-LOCAL rule 10 action 'accept'
set firewall name EXTERNAL-LOCAL rule 10 log 'enable'
set firewall name EXTERNAL-LOCAL rule 10 state established 'enable'
set firewall name EXTERNAL-LOCAL rule 10 state related 'enable'
set firewall name EXTERNAL-LOCAL rule 20 action 'accept'
set firewall name EXTERNAL-LOCAL rule 20 icmp type-name 'echo-request'
set firewall name EXTERNAL-LOCAL rule 20 log 'enable'
set firewall name EXTERNAL-LOCAL rule 20 protocol 'icmp'
set firewall name EXTERNAL-LOCAL rule 20 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 30 action 'drop'
set firewall name EXTERNAL-LOCAL rule 30 description 'ssh'
set firewall name EXTERNAL-LOCAL rule 30 destination port '22'
set firewall name EXTERNAL-LOCAL rule 30 log 'enable'
set firewall name EXTERNAL-LOCAL rule 30 protocol 'tcp'
set firewall name EXTERNAL-LOCAL rule 30 recent count '15'
set firewall name EXTERNAL-LOCAL rule 30 recent time '60'
set firewall name EXTERNAL-LOCAL rule 30 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 31 action 'accept'
set firewall name EXTERNAL-LOCAL rule 31 destination port '22'
set firewall name EXTERNAL-LOCAL rule 31 log 'enable'
set firewall name EXTERNAL-LOCAL rule 31 protocol 'tcp'
set firewall name EXTERNAL-LOCAL rule 31 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 40 action 'accept'
set firewall name EXTERNAL-LOCAL rule 40 description 'magic-wan'
set firewall name EXTERNAL-LOCAL rule 40 log 'enable'
set firewall name EXTERNAL-LOCAL rule 40 protocol 'gre'
set firewall name EXTERNAL-LOCAL rule 40 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-LOCAL rule 50 action 'accept'
set firewall name EXTERNAL-LOCAL rule 50 icmp type-name 'echo-reply'
set firewall name EXTERNAL-LOCAL rule 50 log 'enable'
set firewall name EXTERNAL-LOCAL rule 50 protocol 'icmp'
set firewall name VLAN-100 default-action 'accept'
set firewall name VLAN-100 enable-default-log
set firewall name VLAN-100 rule 10 action 'accept'
set firewall name VLAN-100 rule 10 log 'enable'
set firewall name VLAN-100 rule 10 state established 'enable'
set firewall name VLAN-100 rule 10 state related 'enable'
set firewall name VLAN-100 rule 20 action 'accept'
set firewall name VLAN-100 rule 20 description 'Printer access'
set firewall name VLAN-100 rule 20 destination address '10.68.69.12'
set firewall name VLAN-100 rule 30 action 'accept'
set firewall name VLAN-100 rule 30 description 'Pihole DNS'
set firewall name VLAN-100 rule 30 destination address '10.68.69.7'
set firewall name VLAN-100 rule 30 destination port '53'
set firewall name VLAN-100 rule 30 protocol 'tcp_udp'
set firewall name VLAN-100 rule 40 action 'drop'
set firewall name VLAN-100 rule 40 description 'Restrict Access to INTERNAL1 network'
set firewall name VLAN-100 rule 40 destination address '10.68.69.0/24'
set firewall name VLAN-100 rule 41 action 'drop'
set firewall name VLAN-100 rule 41 description 'Restrict Access to VLAN200 network'
set firewall name VLAN-100 rule 41 destination address '10.68.71.0/24'
set firewall name VLAN-200 default-action 'accept'
set firewall name VLAN-200 enable-default-log
set firewall name VLAN-200 rule 10 action 'accept'
set firewall name VLAN-200 rule 10 log 'enable'
set firewall name VLAN-200 rule 10 state established 'enable'
set firewall name VLAN-200 rule 10 state related 'enable'
set firewall name VLAN-200 rule 20 action 'accept'
set firewall name VLAN-200 rule 20 description 'Printer access'
set firewall name VLAN-200 rule 20 destination address '10.68.69.12'
set firewall name VLAN-200 rule 30 action 'accept'
set firewall name VLAN-200 rule 30 description 'Pihole DNS'
set firewall name VLAN-200 rule 30 destination address '10.68.69.7'
set firewall name VLAN-200 rule 30 destination port '53,9100'
set firewall name VLAN-200 rule 30 protocol 'tcp_udp'
set firewall name VLAN-200 rule 40 action 'accept'
set firewall name VLAN-200 rule 40 description 'ERFI1 Access'
set firewall name VLAN-200 rule 40 destination address '10.68.69.3'
set firewall name VLAN-200 rule 50 action 'drop'
set firewall name VLAN-200 rule 50 description 'Restrict Access to INTERNAL1 network'
set firewall name VLAN-200 rule 50 destination address '10.68.69.0/24'
set firewall name VLAN-200 rule 51 action 'drop'
set firewall name VLAN-200 rule 51 description 'Restrict Access to VLAN100 network'
set firewall name VLAN-200 rule 51 destination address '10.68.70.0/24'
set firewall options interface tun0 adjust-mss '1436'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'EXTERNAL1'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces ethernet eth0 firewall in name 'EXTERNAL-IN'
set interfaces ethernet eth0 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces ethernet eth0 firewall local name 'EXTERNAL-LOCAL'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address 'dhcp'
set interfaces ethernet eth1 description 'EXTERNAL2'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces ethernet eth1 firewall in name 'EXTERNAL-IN'
set interfaces ethernet eth1 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces ethernet eth1 firewall local name 'EXTERNAL-LOCAL'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 address '10.68.69.1/24'
set interfaces ethernet eth2 description 'INTERNAL1'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 hw-id '00:e0:67:2d:eb:e0'
set interfaces ethernet eth2 ip arp-cache-timeout '30'
set interfaces ethernet eth2 speed 'auto'
set interfaces ethernet eth2 vif 100 address '10.68.70.1/24'
set interfaces ethernet eth2 vif 100 description 'asus'
set interfaces ethernet eth2 vif 100 firewall in name 'VLAN-100'
set interfaces ethernet eth2 vif 200 address '10.68.71.1/24'
set interfaces ethernet eth2 vif 200 description 'servarr'
set interfaces ethernet eth2 vif 200 firewall in name 'VLAN-200'
set interfaces ethernet eth3 address '10.68.73.1/24'
set interfaces ethernet eth3 description 'INTERNAL2'
set interfaces ethernet eth3 duplex 'auto'
set interfaces ethernet eth3 speed 'auto'
set interfaces loopback lo
set interfaces tunnel tun0 address '10.68.72.20/31'
set interfaces tunnel tun0 description 'magic-wan'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 mtu '1476'
set interfaces tunnel tun0 remote 'xxx.xxx.66.5'
set interfaces tunnel tun0 source-address 'xxx.xxx.189.102'
set load-balancing wan flush-connections
set load-balancing wan interface-health eth0 failure-count '2'
set load-balancing wan interface-health eth0 nexthop 'dhcp'
set load-balancing wan interface-health eth0 success-count '1'
set load-balancing wan interface-health eth0 test 10 resp-time '5'
set load-balancing wan interface-health eth0 test 10 target '8.8.8.8'
set load-balancing wan interface-health eth0 test 10 ttl-limit '1'
set load-balancing wan interface-health eth0 test 10 type 'ping'
set load-balancing wan interface-health eth0 test 20 resp-time '5'
set load-balancing wan interface-health eth0 test 20 target '1.1.1.1'
set load-balancing wan interface-health eth0 test 20 ttl-limit '1'
set load-balancing wan interface-health eth0 test 20 type 'ping'
set load-balancing wan interface-health eth1 failure-count '2'
set load-balancing wan interface-health eth1 nexthop 'dhcp'
set load-balancing wan interface-health eth1 success-count '1'
set load-balancing wan interface-health eth1 test 10 resp-time '5'
set load-balancing wan interface-health eth1 test 10 target '8.8.8.8'
set load-balancing wan interface-health eth1 test 10 ttl-limit '1'
set load-balancing wan interface-health eth1 test 10 type 'ping'
set load-balancing wan interface-health eth1 test 20 resp-time '5'
set load-balancing wan interface-health eth1 test 20 target '1.1.1.1'
set load-balancing wan interface-health eth1 test 20 ttl-limit '1'
set load-balancing wan interface-health eth1 test 20 type 'ping'
set load-balancing wan interface-health tun0 nexthop '10.68.72.21'
set load-balancing wan interface-health tun0 test 10 target '10.68.72.21'
set load-balancing wan rule 10 description 'vlan100-exclusion-eth2'
set load-balancing wan rule 10 destination address '10.68.69.1/24'
set load-balancing wan rule 10 exclude
set load-balancing wan rule 10 inbound-interface 'eth2.100'
set load-balancing wan rule 10 protocol 'all'
set load-balancing wan rule 11 description 'vlan200-exclusion-eth2'
set load-balancing wan rule 11 destination address '10.68.69.1/24'
set load-balancing wan rule 11 exclude
set load-balancing wan rule 11 inbound-interface 'eth2.200'
set load-balancing wan rule 11 protocol 'all'
set load-balancing wan rule 12 description 'eth2-exclusion-vlan100'
set load-balancing wan rule 12 destination address '10.68.70.1/24'
set load-balancing wan rule 12 exclude
set load-balancing wan rule 12 inbound-interface 'eth2'
set load-balancing wan rule 12 protocol 'all'
set load-balancing wan rule 13 description 'eth2-exclusion-vlan200'
set load-balancing wan rule 13 destination address '10.68.71.1/24'
set load-balancing wan rule 13 exclude
set load-balancing wan rule 13 inbound-interface 'eth2'
set load-balancing wan rule 13 protocol 'all'
set load-balancing wan rule 20 description 'tun0-exclusion-vlan100'
set load-balancing wan rule 20 destination address '10.68.70.1/24'
set load-balancing wan rule 20 exclude
set load-balancing wan rule 20 inbound-interface 'tun0'
set load-balancing wan rule 20 protocol 'all'
set load-balancing wan rule 21 description 'tun0-exclusion-vlan200'
set load-balancing wan rule 21 destination address '10.68.71.1/24'
set load-balancing wan rule 21 exclude
set load-balancing wan rule 21 inbound-interface 'tun0'
set load-balancing wan rule 21 protocol 'all'
set load-balancing wan rule 22 description 'tun0-exclusion-eth2'
set load-balancing wan rule 22 destination address '10.68.69.1/24'
set load-balancing wan rule 22 exclude
set load-balancing wan rule 22 inbound-interface 'tun0'
set load-balancing wan rule 22 protocol 'all'
set load-balancing wan rule 23 description 'vlan100-exclusion-tun0'
set load-balancing wan rule 23 destination address '10.68.72.20/31'
set load-balancing wan rule 23 exclude
set load-balancing wan rule 23 inbound-interface 'eth2.100'
set load-balancing wan rule 23 protocol 'all'
set load-balancing wan rule 24 description 'vlan200-exclusion-tun0'
set load-balancing wan rule 24 destination address '10.68.72.20/31'
set load-balancing wan rule 24 exclude
set load-balancing wan rule 24 inbound-interface 'eth2.200'
set load-balancing wan rule 24 protocol 'all'
set load-balancing wan rule 25 description 'eth2-exclusion-tun0'
set load-balancing wan rule 25 destination address '10.68.72.20/31'
set load-balancing wan rule 25 exclude
set load-balancing wan rule 25 inbound-interface 'eth2'
set load-balancing wan rule 25 protocol 'all'
set load-balancing wan rule 30 failover
set load-balancing wan rule 30 inbound-interface 'tun0'
set load-balancing wan rule 30 interface eth0 weight '10'
set load-balancing wan rule 30 interface eth1 weight '1'
set load-balancing wan rule 30 protocol 'all'
set load-balancing wan rule 31 failover
set load-balancing wan rule 31 inbound-interface 'eth2.100'
set load-balancing wan rule 31 interface eth0 weight '10'
set load-balancing wan rule 31 interface eth1 weight '1'
set load-balancing wan rule 31 protocol 'all'
set load-balancing wan rule 32 failover
set load-balancing wan rule 32 inbound-interface 'eth2.200'
set load-balancing wan rule 32 interface eth0 weight '10'
set load-balancing wan rule 32 interface eth1 weight '1'
set load-balancing wan rule 32 protocol 'all'
set load-balancing wan rule 33 failover
set load-balancing wan rule 33 inbound-interface 'eth2'
set load-balancing wan rule 33 interface eth0 weight '10'
set load-balancing wan rule 33 interface eth1 weight '1'
set load-balancing wan rule 33 protocol 'all'
set load-balancing wan rule 50 failover
set load-balancing wan rule 50 inbound-interface 'eth2.200'
set load-balancing wan rule 50 interface tun0
set load-balancing wan rule 50 source address '10.68.71.3/32'
set load-balancing wan sticky-connections inbound
set nat destination rule 10 description 'servarr-vlan200-eth0'
set nat destination rule 10 destination port '80,443'
set nat destination rule 10 inbound-interface 'eth0'
set nat destination rule 10 log 'enable'
set nat destination rule 10 protocol 'tcp_udp'
set nat destination rule 10 translation address '10.68.71.2'
set nat destination rule 11 description 'bind-vlan200-eth0'
set nat destination rule 11 destination port '5053'
set nat destination rule 11 inbound-interface 'eth0'
set nat destination rule 11 protocol 'tcp_udp'
set nat destination rule 11 translation address '10.68.71.2'
set nat destination rule 11 translation port '5053'
set nat destination rule 20 description 'servarr-vlan200-eth1'
set nat destination rule 20 destination port '80,443'
set nat destination rule 20 inbound-interface 'eth1'
set nat destination rule 20 log 'enable'
set nat destination rule 20 protocol 'tcp_udp'
set nat destination rule 20 translation address '10.68.71.2'
set nat destination rule 21 description 'bind-vlan200-eth1'
set nat destination rule 21 destination port '5053'
set nat destination rule 21 inbound-interface 'eth1'
set nat destination rule 21 protocol 'tcp_udp'
set nat destination rule 21 translation address '10.68.71.2'
set nat destination rule 21 translation port '5053'
set nat destination rule 30 description 'kvm-eth0'
set nat destination rule 30 destination port '2053'
set nat destination rule 30 inbound-interface 'eth0'
set nat destination rule 30 log 'enable'
set nat destination rule 30 protocol 'tcp_udp'
set nat destination rule 30 translation address '10.68.69.6'
set nat destination rule 30 translation port '443'
set nat destination rule 40 description 'kvm-eth1'
set nat destination rule 40 destination port '2053'
set nat destination rule 40 inbound-interface 'eth1'
set nat destination rule 40 log 'enable'
set nat destination rule 40 protocol 'tcp_udp'
set nat destination rule 40 translation address '10.68.69.6'
set nat destination rule 40 translation port '443'
set nat source rule 100 description 'eth0'
set nat source rule 100 log 'enable'
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 source address '10.68.0.0/16'
set nat source rule 100 translation address 'masquerade'
set nat source rule 200 description 'eth1'
set nat source rule 200 log 'enable'
set nat source rule 200 outbound-interface 'eth1'
set nat source rule 200 source address '10.68.0.0/16'
set nat source rule 200 translation address 'masquerade'
set policy route magic-wan enable-default-log
set policy route magic-wan rule 100 description 'magic-wan'
set policy route magic-wan rule 100 destination port '80,443'
set policy route magic-wan rule 100 log 'enable'
set policy route magic-wan rule 100 protocol 'tcp_udp'
set policy route magic-wan rule 100 set table '100'
set policy route magic-wan rule 100 source address '10.68.71.3'
set protocols static table 100 route 0.0.0.0/0 next-hop 10.68.72.21
set service dhcp-server shared-network-name INTERNAL1 subnet 10.68.69.0/24 default-router '10.68.69.1'
set service dhcp-server shared-network-name INTERNAL1 subnet 10.68.69.0/24 domain-name 'vyos1.lan'
set service dhcp-server shared-network-name INTERNAL1 subnet 10.68.69.0/24 lease '300'
set service dhcp-server shared-network-name INTERNAL1 subnet 10.68.69.0/24 name-server '10.68.69.1'
set service dhcp-server shared-network-name INTERNAL1 subnet 10.68.69.0/24 range 0 start '10.68.69.2'
set service dhcp-server shared-network-name INTERNAL1 subnet 10.68.69.0/24 range 0 stop '10.68.69.254'
set service dhcp-server shared-network-name INTERNAL2 subnet 10.68.73.0/24 default-router '10.68.73.1'
set service dhcp-server shared-network-name INTERNAL2 subnet 10.68.73.0/24 domain-name 'vyos2.lan'
set service dhcp-server shared-network-name INTERNAL2 subnet 10.68.73.0/24 lease '300'
set service dhcp-server shared-network-name INTERNAL2 subnet 10.68.73.0/24 name-server '10.68.73.1'
set service dhcp-server shared-network-name INTERNAL2 subnet 10.68.73.0/24 range 0 start '10.68.73.2'
set service dhcp-server shared-network-name INTERNAL2 subnet 10.68.73.0/24 range 0 stop '10.68.73.254'
set service dhcp-server shared-network-name VLAN100 subnet 10.68.70.0/24 default-router '10.68.70.1'
set service dhcp-server shared-network-name VLAN100 subnet 10.68.70.0/24 domain-name 'vyos100.vlan'
set service dhcp-server shared-network-name VLAN100 subnet 10.68.70.0/24 lease '300'
set service dhcp-server shared-network-name VLAN100 subnet 10.68.70.0/24 name-server '10.68.70.1'
set service dhcp-server shared-network-name VLAN100 subnet 10.68.70.0/24 range 0 start '10.68.70.2'
set service dhcp-server shared-network-name VLAN100 subnet 10.68.70.0/24 range 0 stop '10.68.70.254'
set service dhcp-server shared-network-name VLAN200 subnet 10.68.71.0/24 default-router '10.68.71.1'
set service dhcp-server shared-network-name VLAN200 subnet 10.68.71.0/24 domain-name 'vyos200.vlan'
set service dhcp-server shared-network-name VLAN200 subnet 10.68.71.0/24 lease '300'
set service dhcp-server shared-network-name VLAN200 subnet 10.68.71.0/24 name-server '10.68.71.1'
set service dhcp-server shared-network-name VLAN200 subnet 10.68.71.0/24 range 0 start '10.68.71.2'
set service dhcp-server shared-network-name VLAN200 subnet 10.68.71.0/24 range 0 stop '10.68.71.254'
set service dns forwarding allow-from '10.68.0.0/16'
set service dns forwarding cache-size '0'
set service dns forwarding listen-address '10.68.69.1'
set service dns forwarding listen-address '10.68.70.1'
set service dns forwarding listen-address '10.68.71.1'
set service dns forwarding listen-address '10.68.73.1'
set service dns forwarding name-server '10.68.69.7'
set service ssh disable-password-authentication
set service ssh loglevel 'verbose'
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system name-server '10.68.69.7'
set system name-server '1.1.1.1'
set system name-server '8.8.8.8'
set system ntp server 0.sg.pool.ntp.org
set system ntp server 1.sg.pool.ntp.org
set system ntp server 2.sg.pool.ntp.org
set system ntp server 3.sg.pool.ntp.org
set system static-host-mapping host-name asus inet '10.68.70.2'
set system static-host-mapping host-name erfi1 inet '10.68.69.3'
set system static-host-mapping host-name erfikvm inet '10.68.69.6'
set system static-host-mapping host-name erfipie inet '10.68.69.7'
set system static-host-mapping host-name erfiyos inet '10.68.69.1'
set system static-host-mapping host-name fedora inet '10.68.71.4'
set system static-host-mapping host-name servarr inet '10.68.71.2'
set system static-host-mapping host-name switcheroo inet '10.68.69.4'
set system static-host-mapping host-name ubuntu inet '10.68.71.3'
set system static-host-mapping host-name windows-vm inet '10.68.71.5'
set system sysctl custom net.ipv4.conf.all.accept_local value '1'
set system syslog global facility all level 'all'
set system syslog global facility protocols level 'all'
set system time-zone 'Asia/Singapore'

[harguna]

I think Taras found it, MSS being set on tun0 which is the outgoing tunnel connection should be right, as this has worked without WAN LB, so it could have been the incompatibility betweeen PBR and WAN LB.

@matthewr

This sounds like the classic symptom of a “Path MTU Blackhole”: it works for small packets and breaks with larger packets.

This actually happens IF I didn’t set the mss clamping on the tun0 interface