How to configure inter-vlan routing

General questions
, ,
1

I’m new to vyos and coming from pfsense and I’ve setup a simple router with ETH0 being a WAN interface and ETH1 being a LAN interface with multiple VLANs This is running on ESX and the LAN interface is on a trunked port group (vlan 4095). If i set the vlan id of a VM to 10 i cant ping the vlan 10 gateway, i can ping the WAN gateway but i cant reach any other vlan gateways. From what i’ve read this should ‘just work’ but I must be missing something in the config.

interfaces {
     ethernet eth0 {
         address dhcp
         description "WAN Interface"
         hw-id 00:50:56:ac:61:a1
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth1 {
         address 10.0.0.1/24
         hw-id 00:50:56:ac:84:1f
         mtu 9000
         offload {
             gro
             gso
             sg
             tso
         }
         vif 10 {
             address 10.0.10.1/24
             description VLAN10-Mgmt
             mtu 9000
         }
         vif 11 {
             address 10.0.11.1/24
             description VLAN11-vMotion
             mtu 9000
         }
         vif 12 {
             address 10.0.12.1/24
             description VLAN12-vSAN
             mtu 9000
         }
         vif 13 {
             address 10.0.13.1/24
             description VLAN13-HostOverlay
             mtu 9000
         }
         vif 14 {
             address 10.0.14.1/24
             description VLAN14-EdgeOverlay
             mtu 9000
         }
         vif 15 {
             address 10.0.15.1/24
             description VLAN15-EdgeUplink1
             mtu 9000
         }
         }
         vif 16 {
             address 10.0.16.1/24
             description VLAN16-EdgeUplink2
             mtu 9000
         }
         vif 17 {
             address 10.0.17.1/24
             description VLAN17-Reserved
             mtu 9000
         }
     }
     loopback lo {
     }
 }
 protocols {
     static {
         route 0.0.0.0/0 {
             next-hop 182.168.1.1 {
             }
         }
     }
 }
2

Do you have the firewall configured at all?

3

Do you have the firewall configured at all?

no, i wasn’t sure if i had to bother with it since this is all LAN traffic, even the WAN just connects to my non lab network and picks up a dhcp from my home network server

4

Gotcha! Do you maybe have your Pfsense still active with the same IPs?

Assuming you have everything setup correctly in ESXi, it should work just fine. You can add a dummy interface and try to ping that from a host as a test.

5

no pfsense is gone, home network uses a fios gateway with IP 192.168.1.1, this hands off dhcp to the home network (for now) which my vyos router picks up on ETH0 for a 192.168.1.x ‘wan address’ This portgroup in ESX is not trunked and does not tag anything. On a separate vswitch with a trunked portgroup i have ETH1 configured with vlans as a ‘LAB’ network, these are all 10.0.0.x which im looking to have vyos act as a router

6

You’ll probably need to check ARP on each host and make sure they’re at least communicating on layer 2. You may want to disable the offloads on the interfaces because they can cause issues sometimes. Lastly, you may want to play with NIC types and see if they are maybe eating frames.

7

so i’m not sure what you mean by “disable the offloads” I feel like it probably something simple… my windows VM sitting on the same porgroup as ETH1 has a vlan ID of 10 and a 10.0.10.200 address with 10.0.10.1 as its gateway… i can ping the gateway just fine and access the internet via the WAN interface. But I cant ping another vlan gateway like 10.0.11.1 which sits on the vyos router… I feel like that should work… do i need static routes from vlan 10 to 11?

arp shows 10.0.0.x/24 and 192.168.1.x/24 reachable

if I recall in pfsense, you had to do something with the firewall to allow different networks to talk, im just not sure on what that would look like with vyos

8

You can delete the offloads with this in conf mode:

delete interfaces ethernet eth0 offload
delete interfaces ethernet eth1 offload

You don’t need to add any static routes, the connected routes are enough to route between subnets when everything is done correctly. Since you say you’re not using the firewall, there won’t be anything there to configure to allow the traffic.

Paste the output of these commands from op-mode into here:

show ip route
show arp
show firewall
9 
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [1/0] via 192.168.1.1, eth0, weight 1, 00:00:14
S   0.0.0.0/0 [210/0] via 192.168.1.1, eth0, weight 1, 00:11:23
C>* 10.0.0.0/24 is directly connected, eth1, weight 1, 01:16:38
L>* 10.0.0.1/32 is directly connected, eth1, weight 1, 01:16:38
C>* 10.0.10.0/24 is directly connected, eth1.10, weight 1, 01:53:01
L>* 10.0.10.1/32 is directly connected, eth1.10, weight 1, 01:53:01
C>* 10.0.11.0/24 is directly connected, eth1.11, weight 1, 01:47:58
L>* 10.0.11.1/32 is directly connected, eth1.11, weight 1, 01:47:58
C>* 10.0.12.0/24 is directly connected, eth1.12, weight 1, 01:47:58
L>* 10.0.12.1/32 is directly connected, eth1.12, weight 1, 01:47:58
C>* 10.0.13.0/24 is directly connected, eth1.13, weight 1, 01:41:48
L>* 10.0.13.1/32 is directly connected, eth1.13, weight 1, 01:41:48
C>* 10.0.14.0/24 is directly connected, eth1.14, weight 1, 01:41:48
L>* 10.0.14.1/32 is directly connected, eth1.14, weight 1, 01:41:48
C>* 10.0.15.0/24 is directly connected, eth1.15, weight 1, 01:41:47
L>* 10.0.15.1/32 is directly connected, eth1.15, weight 1, 01:41:47
C>* 10.0.16.0/24 is directly connected, eth1.16, weight 1, 01:41:47
L>* 10.0.16.1/32 is directly connected, eth1.16, weight 1, 01:41:47
C>* 10.0.17.0/24 is directly connected, eth1.17, weight 1, 01:41:47
L>* 10.0.17.1/32 is directly connected, eth1.17, weight 1, 01:41:47
C>* 192.168.1.0/24 is directly connected, eth0, weight 1, 00:11:23
K * 192.168.1.0/24 [0/0] is directly connected, eth0, weight 1, 00:11:23
L>* 192.168.1.106/32 is directly connected, eth0, weight 1, 00:11:23


vyos@vyos:~$ show arp
Address      Interface    Link layer address    State
-----------  -----------  --------------------  ---------
10.0.10.200  eth1.10      00:0c:29:5f:44:1d     STALE
10.0.0.6     eth1         00:0c:29:ce:f4:b3     REACHABLE
10.0.0.30    eth1         00:50:56:ac:e1:dd     STALE
10.0.0.11    eth1         00:0c:29:ef:85:32     STALE
192.168.1.1  eth0         ac:91:9b:40:83:47     REACHABLE
10.0.0.200   eth1         00:0c:29:5f:44:1d     REACHABLE
10.0.11.10   eth1.11      00:50:56:65:97:b7     REACHABLE
10.0.0.185   eth1         00:50:0a:00:00:b9     STALE

vyos@vyos:~$ show firewall
Rulesets Information
vyos@vyos:~$

10.0.0.6 is my DNS server

10.0.11.10 is a vmkernel port on an ESX server sitting on the correct vlan

10.0.0.200 is my windows VM that i assumed i would be able to reach 10,0.11.10 from

10

Everything looks correct in the RIB and ARP.
Try to do this from VyOS:

ping 10.0.10.200 source-address 10.0.11.1
ping 10.0.11.10 source-address 10.0.10.1

11

so 10.0.10.200 was my workstation that i changed IPs during testing, it is currently 10.0.0.200 with no vlan ID… this pings, interesting though i cant reach the vkernel 10.0.11.10 from 10.0.10.1 even on the router

vyos@vyos:~$ ping 10.0.0.200 source-address 10.0.11.1
PING 10.0.0.200 (10.0.0.200) from 10.0.11.1 : 56(84) bytes of data.
64 bytes from 10.0.0.200: icmp_seq=1 ttl=128 time=0.233 ms
64 bytes from 10.0.0.200: icmp_seq=2 ttl=128 time=0.293 ms
64 bytes from 10.0.0.200: icmp_seq=3 ttl=128 time=0.285 ms
^C
--- 10.0.0.200 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 0.233/0.270/0.293/0.026 ms
vyos@vyos:~$ ping 10.0.11.10 source-address 10.0.10.1
PING 10.0.11.10 (10.0.11.10) from 10.0.10.1 : 56(84) bytes of data.
^X^C
--- 10.0.11.10 ping statistics ---
14 packets transmitted, 0 received, 100% packet loss, time 13292ms
12

Sounds like you may have an issue with your ESXi setup on VLAN 11 then.

Everything looks good on VyOS. You can tell for sure where packets are and are not reaching with tcpdump on both sides. That’ll be your best bet for following the bouncing ball.

13

hey thanks for your help, as long as i know its not some kind of config problem ill look elsewhere

14

thought I’d follow up and let you know that i figured out my problem. I was using a multi-homed jump box and a simple traceroute showed that for some reason it was using my 192.168.1.1 gateway to access 10.x.x.x networks…. once i put in a static roue I was able to access all of my vlans without issue… rookie mistake

Ethernet adapter Ethernet0:

   Connection-specific DNS Suffix  . : home.lab.biz
   IPv4 Address. . . . . . . . . . . : 192.168.1.110
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1

Ethernet adapter Ethernet1:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 10.0.0.200
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.0.0.1

tracert 10.0.10.180

Tracing route to 10.0.10.180 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    37 ms    37 ms    41 ms  lo0-100.blah.blah
  3     *        *        *     Request timed out.

route -p add 10.0.0.0 mask 255.0.0.0 10.0.0.1

tracert 10.0.10.180

Tracing route to 10.0.10.180 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  10.0.0.1
  2     1 ms     1 ms    <1 ms  10.0.10.180

Trace complete.

ping 10.0.10.180

Pinging 10.0.10.180 with 32 bytes of data:
Reply from 10.0.10.180: bytes=32 time<1ms TTL=63
Reply from 10.0.10.180: bytes=32 time<1ms TTL=63
Reply from 10.0.10.180: bytes=32 time<1ms TTL=63
Reply from 10.0.10.180: bytes=32 time<1ms TTL=63

Ping statistics for 10.0.10.180:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms