Hi,
Looking for ideas to troubleshoot a very weird issue. VLANs seem to work for the common cases (such as web browsing) but a problem seems to foul up App Store handshaking and Blink IP camera handshaking.
I have a home network with UniFi access points and switches, with VLAN setups to segregate guest/kids/IP cameras LAN from the normal (root) lan which contains trusted devices and servers. The VLANs were set up under an EdgeRouter Lite as the DHCP server/network bridge/NAT agent, and worked marvelously for 8+ years.
However recently I have become concerned with hardware failure and opted to explore VyOS (1.5-rolling-202502190007) on more replaceable hardware (in my case an Asus PN42 running ProxMox 8.4.0). The vyos config is a port from the EdgeRouter config, with some rule cleanup and removal of QoS stuff I thought was not really useful, and setting the LAN subnet to 192.168.210.x instead of 192.168.1.x which tended to clash with local WiFi subnets when VPNning from a remote location.
Things are up and running, with two issues that I have been unable to resolve. I feel they are probably the same underlying issue.
- An Amazon Blink camera Sync Module (on a VLAN) is unable to connect with the Amazon mothership, and keeps rebooting in an effort to sort itself out.
- Kids’ iPads on the Kids VLAN are unable to connect to the App Store.
When the devices are connected to the non-VLAN (root) LAN, they work normally. And what is interesting is that on the iPads, problems with the App Store are the only symptom. Web browsing/Minecraft works just fine.
What have I done thus far?
- Checked firewall rules. Nothing from these devices are hitting a deny rule.
- Ruled out DHCP: manually configured IP and DNS.
- Played with MTU settings
- tcpdump to look at traffic, both on VLAN and WAN sides.
tcpdump: I captured packets of the Blink camera when it is on the root LAN (passing case) vs when it is on a VLAN (failing case). In both cases, I see NTP requests and responses, and HTTPS requests and responses. Since I cannot see the contents of the HTTP packets, I don’t know what is in them, but in both cases I see the same servers being contacted (not exactly, as there appears to be some distribution amongst endpoint hosts). The only difference is that there seems to be more traffic in the passing case, as would be expected. Somewhere in the handshaking process, someone (either the client device or the server) must get stuck or determine something is fouled up and aborts.
I’m kind of stumped. Because the config is in the same spirit as the working config from Vyatta days on the Edgerouter, I don’t think I have introduced a misconfiguration. I may have left out something important, but obviously it’s harder to spot the thing you’re missing.
Unless someone has an idea, my solution will probably be to move the trusted stuff (at least the servers) to a VLAN, to segregate them from “everyone else”. The idea of the Kids VLAN is much less workable now that iOS randomizes MAC addresses (I can’t prevent them from joining the wide-open LAN if the kids want to get to unfiltered internet).
I appreciate any feedback!
Brian
Subset of config (I am hitting message length limits)
set firewall group interface-group LAN interface 'eth1'
set firewall group interface-group WAN interface 'eth0'
set firewall group network-group ALLOW_WAN_NET include 'ROOT_NET'
set firewall group network-group ALLOW_WAN_NET include 'GUEST_NET'
set firewall group network-group ALLOW_WAN_NET include 'STREAMING_NET'
set firewall group network-group ALLOW_WAN_NET include 'KIDS_NET'
set firewall group network-group ALLOW_WAN_NET include 'NETAPPS'
set firewall group network-group GUEST_NET description 'Guest Network'
set firewall group network-group GUEST_NET network 'xxx.xxx.212.0/24'
set firewall group network-group ISOLATED_LAN include 'KIDS_NET'
set firewall group network-group ISOLATED_LAN include 'GUEST_NET'
set firewall group network-group ISOLATED_LAN include 'STREAMING_NET'
set firewall group network-group KIDS_NET description 'Kids Network'
set firewall group network-group KIDS_NET network 'xxx.xxx.18.0/24'
set firewall group network-group NET-INSIDE-v4 network 'xxx.xxx.0.0/16'
set firewall group network-group NOWAN_NET description 'No Internet Access'
set firewall group network-group NOWAN_NET network 'xxx.xxx.9.0/24'
set firewall group network-group ROOT_NET description 'Root Network (no VLAN tag)'
set firewall group network-group ROOT_NET network 'xxx.xxx.210.0/24'
set firewall group network-group STREAMING_NET description 'Streaming Devices'
set firewall group network-group STREAMING_NET network 'xxx.xxx.5.0/24'
set firewall ipv4 forward filter default-action 'drop'
set firewall ipv4 forward filter rule 201 action 'drop'
set firewall ipv4 forward filter rule 201 description 'Prevent Isolated Networks from Reaching Root'
set firewall ipv4 forward filter rule 201 destination group network-group 'ROOT_NET'
set firewall ipv4 forward filter rule 201 source group network-group 'ISOLATED_LAN'
set firewall ipv4 forward filter rule 500 action 'accept'
set firewall ipv4 forward filter rule 500 description 'Allow Internet'
set firewall ipv4 forward filter rule 500 destination group network-group '!NET-INSIDE-v4'
set firewall ipv4 forward filter rule 500 source group network-group 'ALLOW_WAN_NET'
set firewall ipv4 forward filter rule 510 action 'accept'
set firewall ipv4 forward filter rule 600 action 'accept'
set firewall ipv4 forward filter rule 600 connection-status nat 'destination'
set firewall ipv4 forward filter rule 600 description 'Allow all port forwards from WAN'
set firewall ipv4 forward filter rule 600 state 'new'
set firewall ipv4 input filter default-action 'drop'
set firewall ipv4 input filter rule 10 action 'accept'
set firewall ipv4 input filter rule 10 state 'established'
set firewall ipv4 input filter rule 10 state 'related'
set firewall ipv4 input filter rule 15 action 'drop'
set firewall ipv4 input filter rule 15 state 'invalid'
set firewall ipv4 input filter rule 90 action 'drop'
set firewall ipv4 input filter rule 90 description 'Drop anything initiating from WAN'
set firewall ipv4 input filter rule 90 inbound-interface group 'WAN'
set firewall ipv4 input filter rule 120 action 'accept'
set firewall ipv4 input filter rule 120 description 'Allow DNS'
set firewall ipv4 input filter rule 120 destination port '53'
set firewall ipv4 input filter rule 120 protocol 'tcp_udp'
set firewall ipv4 input filter rule 120 source group network-group 'NET-INSIDE-v4'
set firewall ipv4 input filter rule 130 action 'accept'
set firewall ipv4 input filter rule 130 description 'Allow DHCP'
set firewall ipv4 input filter rule 130 destination port '68'
set firewall ipv4 input filter rule 130 protocol 'tcp_udp'
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'WAN Interface'
set interfaces ethernet eth0 hw-id 'xx:xx:xx:xx:xx:a3'
set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth1 address 'xxx.xxx.210.1/24'
set interfaces ethernet eth1 description 'LAN Trunk'
set interfaces ethernet eth1 hw-id 'xx:xx:xx:xx:xx:91'
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces ethernet eth1 vif 102 address 'xxx.xxx.212.1/24'
set interfaces ethernet eth1 vif 102 description 'Guest Network'
set interfaces ethernet eth1 vif 102 mtu '1496'
set interfaces ethernet eth1 vif 105 address 'xxx.xxx.5.1/24'
set interfaces ethernet eth1 vif 105 description 'StreamingTV'
set interfaces ethernet eth1 vif 105 mtu '1496'
set interfaces ethernet eth1 vif 109 address 'xxx.xxx.9.1/24'
set interfaces ethernet eth1 vif 109 description 'Non_WAN'
set interfaces ethernet eth1 vif 109 mtu '1496'
set interfaces ethernet eth1 vif 118 address 'xxx.xxx.18.1/24'
set interfaces ethernet eth1 vif 118 description 'Kids'
set interfaces ethernet eth1 vif 118 mtu '1496'
set interfaces ethernet eth1 vif 151 address 'xxx.xxx.214.1/24'
set interfaces ethernet eth1 vif 151 description 'NetApps'
set interfaces ethernet eth1 vif 151 mtu '1496'
set interfaces loopback lo
set nat destination rule 110 description 'Allow HTTP/HTTPS'
set nat destination rule 110 destination port '80,443'
set nat destination rule 110 inbound-interface group 'WAN'
set nat destination rule 110 protocol 'tcp'
set nat destination rule 110 translation address 'xxx.xxx.210.41'
set nat destination rule 200 description 'Hairpin NAT for HTTPS'
set nat destination rule 200 destination address 'xxx.xxx.214.151'
set nat destination rule 200 destination port '443'
set nat destination rule 200 inbound-interface name 'eth1'
set nat destination rule 200 protocol 'tcp'
set nat destination rule 200 translation address 'xxx.xxx.210.41'
set nat source rule 99 outbound-interface group 'WAN'
set nat source rule 99 source group network-group 'ALLOW_WAN_NET'
set nat source rule 99 translation address 'masquerade'
set nat source rule 200 description 'Hairpin NAT for HTTPS'
set nat source rule 200 destination address 'xxx.xxx.210.41'
set nat source rule 200 destination port '443'
set nat source rule 200 outbound-interface name 'eth1'
set nat source rule 200 protocol 'tcp'
set nat source rule 200 source address 'xxx.xxx.0.0/16'
set nat source rule 200 translation address 'masquerade'
set service dns forwarding allow-from 'xxx.xxx.212.0/24'
set service dns forwarding allow-from 'xxx.xxx.210.0/24'
set service dns forwarding allow-from 'xxx.xxx.18.0/24'
set service dns forwarding dnssec 'process-no-validate'
set service dns forwarding listen-address 'xxx.xxx.210.1'
set service dns forwarding listen-address 'xxx.xxx.212.1'
set service dns forwarding listen-address 'xxx.xxx.18.1'
set service dns forwarding name-server xxx.xxx.4.4
set service dns forwarding name-server xxx.xxx.8.8