Hello Everyone,
I have a weird issue with my downstream vyos router and am unable to find any similar threads after days of scouring around. My setup is as follows:
On my edge router (opnsense) I am routing 172.16.0.0/12 to my downstream VyOS router. VyOS router has following config:
eth0 (WAN): 192.168.60.10/24 (transit address)
eth1 (LAN): 172.16.5.1/24 (DHCP server handing out leases from this address)
Routes:
S>* 0.0.0.0/0 [1/0] via 192.168.60.1, eth0, 00:14:04
S 172.16.5.0/24 [1/0] is directly connected, eth0, 00:14:04
C>* 172.16.5.0/24 is directly connected, eth1, 00:14:07
C>* 192.168.60.0/24 is directly connected, eth0, 00:14:06
I can ping hosts in other subnets and upstream in my network and out to the internet. But if I do ping 8.8.8.8 interface eth1 I get destination unreachable.
As for hosts on LAN, they cannot reach any other subnets or the internet unless I manually define the upstream gateway (192.168.60.1). Once I specify the gateway manually they have complete connectivity, everything is reachable. But using the default router (172.16.5.1) nothing is reachable. Same goes for DNS.
I have no firewall rules configured and no NAT.
I think I am missing something obvious from staring at this for so long. Any help would be greatly appreciated.
What does a traceroute from a client show? Is it stopping at the vyos, or edge?
Can your edge reach a client on this subnet?
So your network is:
Internet ↔ OPNsense ↔ VyOS ↔ Clients ?
Where OPNsense:
WAN: DHCP?
LAN: 192.168.60.1/24
ip route:
0.0.0.0/0 nexthop ISP
172.16.0.0/12 nexthop 192.168.60.10
SNAT:
Traffic egressing WAN interface will SNAT to the IP provided by ISP configured at WAN interface.
And VyOS:
WAN: 192.168.60.10/24
LAN: 172.16.5.1/24
ip route:
0.0.0.0/0 nexthop 192.168.60.1
SNAT:
No SNAT rules are active (since SNATing occurs at WAN interface of OPNsense).
You have also have DHCP-server running on VyOS sharing dynamic IP-addresses to clients who then get:
IP: 172.16.5.2 - 172.16.5.254
MASK: /24
GW: 172.16.5.1
DNS: 8.8.8.8 and 8.8.4.4 ?
Is the above correct?
Can client (172.16.5.x) ping 192.168.60.1 ?
Do you see a log about this at OPNsense?
What about when the same client pings 8.8.8.8 - what does the logs of OPNsense then tell you?
Thanks for the reply!
I actually made a slight amount of progress. I tried changing the dhcp server’s default router to the eth0/wan iface’s IP (192.168.60.10). This fixed the connectivity issue for all the clients on the LAN subnet. I then changed the default router back to 172.16.5.1 and connected a ‘new’ host. The new host can fully reach out now. I have no idea why that worked as I verified the config multiple times.
I still cannot ping anything out-bound from the LAN/eth1 iface through.
Hi Apachez, thanks for taking the time to reply,
The config you listed out looks correct except for DNS is 192.168.60.1
Weirdly by changing the DHCP server’s default router to the eth0/wan address (192.168.60.10) commiting, then reverting it back to (172.16.5.1) and commiting again, has resolved all the client host connectivity issues. I verified by connecting a ‘new’ host.
However, I still get ‘destination unreachable’ when I try to ping from eth1 to 8.8.8.8. Opnsense logs are empty so I’m assuming that the icmp packets aren’t making it past the eth1 interface.
Just to confirm, these subnets are on separate physical networks, or separate VLANs, yes?
I would still confirm reachability internally first. You mentioned they can’t get to other internal subnets either.
Can clients get to the edge? Can the edge reach clients? If they can’t, where is it dying? I would consider PCAPs on both sides of that device to see what is happening with the traffic if you believe the configuration to be correct.
I would repeat that for subnet to subnet traffic to see where that traffic appears to be stopped.
Next step would be to verify stuff using tcpdump on the VyOS box to see what the client is actually sending you when it tries to ping 8.8.8.8 and what is actually egressing on the VyOS WAN side.
Remember since you do SNAT at the OPNsense you dont have to do that on VyOS aswell - let VyOS just route the traffic without NATing.
1 Like