VyOS responding on different interface depending on VLAN usage

I’m liking missing something very basic here but I’m faced with the following issue.

I have a network external to VyOS (172.31.0.0/24) that routes through a connection on VyOS to the rest of my network. The interface on VyOS that is the next hop is 192.168.210.3 (coming from 192.168.210.1 on the external network). From there VyOs routes to several other networks (192.168.110.0/24 being the one I’m most concerned with right now).

The 192.168.210.3 interface is on eth1 on VyOs which also has a few other IP addresses assigned. eth0 on VyOs has an address of 192.168.0.2 and is for external traffic.

My original configuration had the 192.168.210.0/24 network on VLAN 210 and all other networks were untagged. I had no issues with this configuration as you can see from a traceroute output (from a system on the 172.31.0.0/24 network to a system on the 192.168.110.0/24 network):

traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
 1  172.31.0.1 (172.31.0.1)  1.306 ms  13.016 ms  12.695 ms
 2  100.64.128.0 (100.64.128.0)  21.670 ms  21.673 ms  21.662 ms
 3  192.168.210.3 (192.168.210.3)  47.869 ms  47.885 ms  48.443 ms
 4  nsxmanager.corp.tanzu (192.168.110.49)  49.027 ms  49.026 ms  49.017 ms

However, if I modify the 192.168.210.0/24 network such that it is also untagged, traffic is sent through the external interface, 192.168.0.2:

traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
 1  172.31.0.1 (172.31.0.1)  1.732 ms  10.964 ms  10.965 ms
 2  100.64.128.0 (100.64.128.0)  20.518 ms  20.513 ms  20.495 ms
 3  192.168.0.2 (192.168.0.2)  46.947 ms  46.953 ms  46.930 ms
 4  nsxmanager.corp.tanzu (192.168.110.49)  47.878 ms  47.868 ms  47.908 ms

While ICMP traffic is working in this scenario, no other traffic flows to the destination address. There are no firewall rules enabled to account for this behavior.

I have found that if I take down eth0, traffic at least flows through eth1 but never via the 192.168.210.3 interface I have configured, as seen below:

traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
 1  172.31.0.1 (172.31.0.1)  1.306 ms  13.016 ms  12.695 ms
 2  100.64.128.0 (100.64.128.0)  21.670 ms  21.673 ms  21.662 ms
 3  192.168.100.1 (192.168.100.1)  47.869 ms  47.885 ms  48.443 ms
 4  nsxmanager.corp.tanzu (192.168.110.49)  49.027 ms  49.026 ms  49.017 ms

In this scenario, all traffic will make it to the destination.

If I bring eth0 back up, traffic keeps flowing through eth1 for at least a few hours until it “magically” ends up going back through eth0.

With all of this in mind, I would be happy if it even just stayed on eth1 as that results in a semi-functional network but I’d really like to understand why I’m not seeing the desired address (192.168.210.3) replying.

Routing with VLANs:

S>* 0.0.0.0/0 [1/0] via 192.168.0.1, eth0, 3d16h35m
C>* 10.10.20.0/24 is directly connected, eth1, 3d16h35m
C>* 10.10.30.0/24 is directly connected, eth1, 3d16h35m
B>* 10.40.14.0/24 [20/0] via 192.168.210.3, eth1.210, 00:01:36
C>* 192.168.0.0/24 is directly connected, eth0, 3d16h35m
C>* 192.168.100.0/24 is directly connected, eth1, 3d16h35m
C>* 192.168.110.0/24 is directly connected, eth1, 3d16h35m
C>* 192.168.120.0/24 is directly connected, eth1.120, 3d16h35m
C>* 192.168.130.0/24 is directly connected, eth1.130, 3d16h35m
C>* 192.168.200.0/24 is directly connected, eth1, 3d16h35m
C>* 192.168.210.0/24 is directly connected, eth1.210, 3d16h35m
C>* 192.168.220.0/23 is directly connected, eth1.220, 3d16h35m

Routing without VLANs:

S>* 0.0.0.0/0 [1/0] via 192.168.0.1, eth0, 5d00h05m
C>* 10.10.20.0/24 is directly connected, eth1, 5d00h06m
C>* 10.10.30.0/24 is directly connected, eth1, 5d00h06m
B>* 10.40.14.0/24 [20/0] via 192.168.210.3, eth1, 00:04:01
C>* 192.168.0.0/24 is directly connected, eth0, 5d00h05m
C>* 192.168.100.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.110.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.120.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.130.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.200.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.210.0/24 is directly connected, eth1, 5d00h06m
C>* 192.168.220.0/23 is directly connected, eth1, 5d00h06m

Interfaces with VLANs:

show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.0.2/24                    u/u  External Network
eth1             192.168.100.1/24                  u/u  Internal Network
                 192.168.110.1/24
                 192.168.200.1/24
                 10.10.30.1/24
                 10.10.20.1/24
eth1.120         192.168.120.1/24                  u/u  VLAN 120 gateway
eth1.130         192.168.130.1/24                  u/u  VLAN 130 gateway
eth1.210         192.168.210.1/24                  u/u  VLAN 210 gateway
eth1.220         192.168.220.1/23                  u/u  VLAN 220 gateway
lo               127.0.0.1/8                       u/u
                 ::1/128

Interfaces without VLANs:

show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.0.2/24                    u/u  External Network
eth1             192.168.100.1/24                  u/u  Internal Network
                 192.168.110.1/24
                 192.168.200.1/24
                 10.10.30.1/24
                 10.10.20.1/24
                 192.168.120.1/24
                 192.168.130.1/24
                 192.168.210.1/24
                 192.168.220.1/23
lo               127.0.0.1/8                       u/u
                 ::1/128

Firewall information:

show firewall

-----------------------------
Rulesets Information
-----------------------------

Any guidance that could be provided would be greatly appreciated. Thanks.

Hi clittle,

I understand that you want to use vlan to segment the traffic and the eth0 interfaces is where you receive external traffic ( 172.31.0.0/24) to internal eth1( 192.168.110.0/24), so if you have setting on vlan 210 , it works fine but you have been getting external traffic for eth1( i think it should be receiving 192.168.0.2 -eth0) , i didn’t know if it was correct .

Could you share with your vyos version ? and the following command :

show configuration commands | strip-private

thanks

Thanks for the quick reply Fernandao. I’m actually trying to flatten my network and remove all of the VLANs but keep separate subnets. This is a nested lab environment and getting functional VLANs in it is proving to be overly burdensome. The routing works fine with VLANs configured but the noted issue is present without them.

The VyOS version is 1.2.6-S1. Please find the output from the requested command below:

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 address 'xxx.xxx.0.2/24'
set interfaces ethernet eth0 description 'External Network'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 hw-id 'XX:XX:XX:XX:XX:c6'
set interfaces ethernet eth0 mtu '8940'
set interfaces ethernet eth0 smp-affinity 'auto'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 address 'xxx.xxx.100.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.110.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.200.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.30.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.20.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.120.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.130.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.210.1/24'
set interfaces ethernet eth1 address 'xxx.xxx.220.1/23'
set interfaces ethernet eth1 description 'Internal Network'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 hw-id 'XX:XX:XX:XX:XX:c7'
set interfaces ethernet eth1 mtu '8940'
set interfaces ethernet eth1 smp-affinity 'auto'
set interfaces ethernet eth1 speed 'auto'
set interfaces loopback lo
set nat destination rule 100 description 'RDP to xxx.xxx.110.10:3389'
set nat destination rule 100 destination port '3389'
set nat destination rule 100 inbound-interface 'eth0'
set nat destination rule 100 protocol 'tcp'
set nat destination rule 100 translation address 'xxx.xxx.110.10'
set nat destination rule 100 translation port '3389'
set nat destination rule 200 description 'vSphere Client to xxx.xxx.110.22'
set nat destination rule 200 destination port '443'
set nat destination rule 200 inbound-interface 'eth0'
set nat destination rule 200 protocol 'tcp'
set nat destination rule 200 translation address 'xxx.xxx.110.22'
set nat destination rule 200 translation port '443'
set nat destination rule 300 description 'VNC connection to xxx.xxx.110.100'
set nat destination rule 300 destination port '5901'
set nat destination rule 300 inbound-interface 'eth0'
set nat destination rule 300 protocol 'tcp'
set nat destination rule 300 translation address 'xxx.xxx.110.100'
set nat destination rule 300 translation port '5901'
set nat destination rule 400 description 'SSH to xxx.xxx.110.100'
set nat destination rule 400 destination port '22'
set nat destination rule 400 inbound-interface 'eth0'
set nat destination rule 400 protocol 'tcp'
set nat destination rule 400 translation address 'xxx.xxx.110.100'
set nat destination rule 400 translation port '22'
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 translation address 'masquerade'
set protocols bgp XXXXXX address-family ipv4-unicast redistribute connected
set protocols bgp XXXXXX maximum-paths ebgp '4'
set protocols bgp XXXXXX maximum-paths ibgp '4'
set protocols bgp XXXXXX neighbor xxx.xxx.100.3 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.3 remote-as '65001'
set protocols bgp XXXXXX neighbor xxx.xxx.100.4 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.4 remote-as '65001'
set protocols bgp XXXXXX neighbor xxx.xxx.100.100 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.100 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.101 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.101 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.102 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.102 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.103 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.103 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.104 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.104 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.105 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.105 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.106 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.106 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.107 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.107 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.108 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.108 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.109 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.109 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.110 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.110 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.111 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.111 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.100.112 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.100.112 remote-as '65022'
set protocols bgp XXXXXX neighbor xxx.xxx.110.32 remote-as '65032'
set protocols bgp XXXXXX neighbor xxx.xxx.110.120 remote-as '65032'
set protocols bgp XXXXXX neighbor xxx.xxx.110.121 remote-as '65032'
set protocols bgp XXXXXX neighbor xxx.xxx.200.3 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.200.3 remote-as '65011'
set protocols bgp XXXXXX neighbor xxx.xxx.200.4 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.200.4 remote-as '65011'
set protocols bgp XXXXXX neighbor xxx.xxx.210.3 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.210.3 remote-as '65012'
set protocols bgp XXXXXX neighbor xxx.xxx.210.4 address-family ipv4-unicast default-originate
set protocols bgp XXXXXX neighbor xxx.xxx.210.4 remote-as '65012'
set protocols bgp XXXXXX parameters router-id 'xxx.xxx.100.1'
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.0.1
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.0.2
set service dhcp-server shared-network-name xxxxxx authoritative
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.130.0/24 default-router 'xxx.xxx.130.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.130.0/24 dns-server 'xxx.xxx.110.10'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.130.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.130.0/24 lease '864000'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.130.0/24 range 0 start 'xxx.xxx.130.155'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.130.0/24 range 0 stop 'xxx.xxx.130.254'
set service dhcp-server shared-network-name xxxxxx authoritative
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.100.0/24 default-router 'xxx.xxx.100.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.100.0/24 dns-server 'xxx.xxx.110.10'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.100.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.100.0/24 lease '864000'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.100.0/24 range 0 start 'xxx.xxx.100.100'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.100.0/24 range 0 stop 'xxx.xxx.100.250'
set service ssh access-control allow user xxxxxx
set service ssh port '22'
set system config-management commit-revisions '100'
set system console device ttyS0 speed '9600'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx level 'admin'
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx key xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx type ssh-xxx
set system login user xxxxxx level 'admin'
set system name-server 'xxx.xxx.110.10'
set system ntp allow-clients address 'xxx.xxx.0.0/0'
set system ntp listen-address 'xxx.xxx.100.1'
set system ntp server xxxxx.tld
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'UTC'

I’d start looking into routing table on host 100.64.128.0

i think that behavior is caused for share many networks without Vlan segmentation . First I try to segment the networks into different vlans (if it is allowed on your network ) something like this for example :

set interfaces ethernet eth1 description 'untag - INT'
set interfaces ethernet eth1 address 192.168.100.1/24
set interfaces ethernet eth1 vif 110 description 'VLAN 100 - INT'
set interfaces ethernet eth1 vif 110 address '192.168.110.1/24'
set interfaces ethernet eth1 vif 200 description 'VLAN 200 - INT'
set interfaces ethernet eth1 vif 200 address '192.168.200.1/24'

although, there is something strange in your first output (where it works fine) :

traceroute nsxmanager.corp.tanzu
traceroute to nsxmanager.corp.tanzu (192.168.110.49), 30 hops max, 60 byte packets
1 172.31.0.1 (172.31.0.1) 1.306 ms 13.016 ms 12.695 ms ------ external -eth0 ?
2 100.64.128.0 (100.64.128.0) 21.670 ms 21.673 ms 21.662 ms ------ external -eth0 ?
3 192.168.210.3 (192.168.210.3) 47.869 ms 47.885 ms 48.443 ms ----- internal -eth1
4 nsxmanager.corp.tanzu (192.168.110.49) 49.027 ms 49.026 ms 49.017 ms ----Internal -eth1

this part is when I see external traffic incoming for eth1 (internal) , it may be a good point to check .