I’m liking missing something very basic here but I’m faced with the following issue.
I have a network external to VyOS (172.31.0.0/24) that routes through a connection on VyOS to the rest of my network. The interface on VyOS that is the next hop is 192.168.210.3 (coming from 192.168.210.1 on the external network). From there VyOs routes to several other networks (192.168.110.0/24 being the one I’m most concerned with right now).
The 192.168.210.3 interface is on eth1 on VyOs which also has a few other IP addresses assigned. eth0 on VyOs has an address of 192.168.0.2 and is for external traffic.
My original configuration had the 192.168.210.0/24 network on VLAN 210 and all other networks were untagged. I had no issues with this configuration as you can see from a traceroute output (from a system on the 172.31.0.0/24 network to a system on the 192.168.110.0/24 network):
traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
1 172.31.0.1 (172.31.0.1) 1.306 ms 13.016 ms 12.695 ms
2 100.64.128.0 (100.64.128.0) 21.670 ms 21.673 ms 21.662 ms
3 192.168.210.3 (192.168.210.3) 47.869 ms 47.885 ms 48.443 ms
4 nsxmanager.corp.tanzu (192.168.110.49) 49.027 ms 49.026 ms 49.017 ms
However, if I modify the 192.168.210.0/24 network such that it is also untagged, traffic is sent through the external interface, 192.168.0.2:
traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
1 172.31.0.1 (172.31.0.1) 1.732 ms 10.964 ms 10.965 ms
2 100.64.128.0 (100.64.128.0) 20.518 ms 20.513 ms 20.495 ms
3 192.168.0.2 (192.168.0.2) 46.947 ms 46.953 ms 46.930 ms
4 nsxmanager.corp.tanzu (192.168.110.49) 47.878 ms 47.868 ms 47.908 ms
While ICMP traffic is working in this scenario, no other traffic flows to the destination address. There are no firewall rules enabled to account for this behavior.
I have found that if I take down eth0, traffic at least flows through eth1 but never via the 192.168.210.3 interface I have configured, as seen below:
traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
1 172.31.0.1 (172.31.0.1) 1.306 ms 13.016 ms 12.695 ms
2 100.64.128.0 (100.64.128.0) 21.670 ms 21.673 ms 21.662 ms
3 192.168.100.1 (192.168.100.1) 47.869 ms 47.885 ms 48.443 ms
4 nsxmanager.corp.tanzu (192.168.110.49) 49.027 ms 49.026 ms 49.017 ms
In this scenario, all traffic will make it to the destination.
If I bring eth0 back up, traffic keeps flowing through eth1 for at least a few hours until it “magically” ends up going back through eth0.
With all of this in mind, I would be happy if it even just stayed on eth1 as that results in a semi-functional network but I’d really like to understand why I’m not seeing the desired address (192.168.210.3) replying.
Routing with VLANs:
S>* 0.0.0.0/0 [1/0] via 192.168.0.1, eth0, 3d16h35m
C>* 10.10.20.0/24 is directly connected, eth1, 3d16h35m
C>* 10.10.30.0/24 is directly connected, eth1, 3d16h35m
B>* 10.40.14.0/24 [20/0] via 192.168.210.3, eth1.210, 00:01:36
C>* 192.168.0.0/24 is directly connected, eth0, 3d16h35m
C>* 192.168.100.0/24 is directly connected, eth1, 3d16h35m
C>* 192.168.110.0/24 is directly connected, eth1, 3d16h35m
C>* 192.168.120.0/24 is directly connected, eth1.120, 3d16h35m
C>* 192.168.130.0/24 is directly connected, eth1.130, 3d16h35m
C>* 192.168.200.0/24 is directly connected, eth1, 3d16h35m
C>* 192.168.210.0/24 is directly connected, eth1.210, 3d16h35m
C>* 192.168.220.0/23 is directly connected, eth1.220, 3d16h35m
Routing without VLANs:
S>* 0.0.0.0/0 [1/0] via 192.168.0.1, eth0, 5d00h05m
C>* 10.10.20.0/24 is directly connected, eth1, 5d00h06m
C>* 10.10.30.0/24 is directly connected, eth1, 5d00h06m
B>* 10.40.14.0/24 [20/0] via 192.168.210.3, eth1, 00:04:01
C>* 192.168.0.0/24 is directly connected, eth0, 5d00h05m
C>* 192.168.100.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.110.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.120.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.130.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.200.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.210.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.220.0/23 is directly connected, eth1, 5d00h06m
Interfaces with VLANs:
show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 192.168.0.2/24 u/u External Network
eth1 192.168.100.1/24 u/u Internal Network
192.168.110.1/24
192.168.200.1/24
10.10.30.1/24
10.10.20.1/24
eth1.120 192.168.120.1/24 u/u VLAN 120 gateway
eth1.130 192.168.130.1/24 u/u VLAN 130 gateway
eth1.210 192.168.210.1/24 u/u VLAN 210 gateway
eth1.220 192.168.220.1/23 u/u VLAN 220 gateway
lo 127.0.0.1/8 u/u
::1/128
Interfaces without VLANs:
show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 192.168.0.2/24 u/u External Network
eth1 192.168.100.1/24 u/u Internal Network
192.168.110.1/24
192.168.200.1/24
10.10.30.1/24
10.10.20.1/24
192.168.120.1/24
192.168.130.1/24
192.168.210.1/24
192.168.220.1/23
lo 127.0.0.1/8 u/u
::1/128
Firewall information:
show firewall
-----------------------------
Rulesets Information
-----------------------------
Any guidance that could be provided would be greatly appreciated. Thanks.