Routing configuration between two VyOS Routers

General questions
#1

Hello, could you please help me to identify routing issue between two VYOS routers.

I’ve been running a laptop lab leveraging VYOS v1.1.7 for a few years now with no issues. I’ve now added a second separate laptop lab leveraging VYOS v1.3.0 and trying to configure routing between both labs to unify them into the same network but so far no luck.

In particular:
Home router 192.168.1.1
eth0 WAN on VYOS1 192.168.1.254
Subnets/Interfaces: 192.168.2.0/24, 192.168.3.0/24, etc

eth0 WAN on VYOS2 192.168.1.253
Subnets/Interfaces: 192.168.5.0/24, 192.168.6.0/24, etc

I am trying to ping from 192.168.2.10 to 192.168.5.10 and fail. Windows firewalls disabled

Configurations are below. I have seen this earlier topic but it doesn’t seem to help

Any pointers are much appreciated!

Version: VyOS 1.1.7
vyos@vyos# show interfaces
ethernet eth0 {
address 192.168.1.254/24
duplex auto
hw-id 00:15:5d:01:6a:2a
smp_affinity auto
speed auto
}
ethernet eth1 {
address 192.168.10.254/24
duplex auto
hw-id 00:15:5d:01:6a:3d
smp_affinity auto
speed auto
}
ethernet eth2 {
address 192.168.2.254/24
description East
duplex auto
hw-id 00:15:5d:01:6a:40
smp_affinity auto
speed auto
}
ethernet eth3 {
address 192.168.3.254/24
description West
duplex auto
hw-id 00:15:5d:01:6a:41
smp_affinity auto
speed auto
}
ethernet eth4 {
address 192.168.4.254/24
duplex auto
hw-id 00:15:5d:01:6a:42
smp_affinity auto
speed auto
}
loopback lo {
}
[edit]
vyos@vyos# show nat
source {
rule 9 {
destination {
address 192.168.5.0/24
}
exclude
outbound-interface eth0
source {
address 192.168.2.0/24
}
}
rule 10 {
outbound-interface eth0
source {
address 192.168.2.0/24
}
translation {
address masquerade
}
}
rule 11 {
outbound-interface eth0
source {
address 192.168.3.0/24
}
translation {
address masquerade
}
}
rule 12 {
outbound-interface eth0
source {
address 192.168.4.0/24
}
translation {
address masquerade
}
}
}
[edit]
vyos@vyos# show protocols
static {
route 192.168.5.0/24 {
next-hop 192.168.1.253 {
}
}
}

Version: VyOS 1.3.0-rc6
vyos@vyos# show int
ethernet eth0 {
address 192.168.1.253/24
hw-id 00:15:5d:01:a0:04
}
ethernet eth1 {
address 192.168.5.253/24
hw-id 00:15:5d:01:a0:05
}
ethernet eth2 {
address 192.168.6.253/24
hw-id 00:15:5d:01:a0:0a
}
loopback lo {
}
[edit]
vyos@vyos# show nat
destination {
}
source {
rule 10 {
outbound-interface eth0
source {
address 192.168.5.0/24
}
translation {
address masquerade
}
}
rule 11 {
outbound-interface eth0
source {
address 192.168.6.0/24
}
translation {
address masquerade
}
}
}
[edit]
vyos@vyos# show prot
static {
route 0.0.0.0/0 {
next-hop 192.168.1.1 {
}
}
route 192.168.2.0/24 {
next-hop 192.168.1.254 {
}
}
}

#2

Hi ddv

Please consider the following:

  1. Firewall setting in VyOS, if you follow VyOS initial setup, there is a firewall rule to block traffic initial from outside network into inside networks. (no firewall information from your message)
  2. Your configuration perform Source NAT (masquerade), so networks behind eth0 should not be visible by outside networks.

Please try to connect these two routers with inside interfaces and make routing between these interfaces. This is assumed the two routers are trusted, if security must be concerned, then it may need another configuration scene…

#3

sorry, my explanation in before on VyOS firewall is something wrong.

My meaning is VyOS firewall rule only allow traffic initial from inside networks to outside networks, and block all other traffics from outside by default…

#4

Thanks, ccng. I don’t have any firewall rules configured on VYOS.
How do I configure internet access for subnets without NATing? The guy in the linked tread posted working configuration for him, and it appears to have NAT in place as well.

#5

First you need the NAT exceptions for the traffic between LANs:

yos@vyos# show nat
source {
rule 5 {
destination {
address 192.168.5.0/24
}
exclude
outbound-interface eth0
}

Next, static routes between VyOS are required:

vyos@vyos# show protocols
static {
route 192.168.5.0/24 {
next-hop 192.168.1.253 {
}
}
}

This needs to be done for every VyOS. If I understand your topology correctly, you will need 2 exceptions and 2 routes for each VyOS

#6

To remove SNAT,

delete nat source

commit

save

Nikolay suggestion should work, but two exclude statement should be added on VyOS1, one for 192.168.5.0/24 as Nikolay stated, another one entry for 192.168.6.0/24.

On VyOS2, two similar statements shall be added, one for 192.168.2.0/24 and one for 192.168.3.0/24.

However, if you don’t need SNAT, just delete the function as above on both routers… And make sure you don’t have firewall rule on each interfaces…

#7

Thank you very much, appreciate the prompt responses! The configuration appears to be working now, but only when deployed on a single PC, i.e.

  • when both VYOS routers deployed on first PC, endpoints from different LANs are able to ping each other
  • when both VYOS routers deployed on second PC, endpoints from different LANs are able to ping each other
  • when one VYOS and one endpoint is on first PC, second VYOS and second endpoint on the second PC - pings between endpoints are not going through, however the opposite WAN interfaces are pingable.

In all cases VYOS configuration is the same, for WAN they are connecting to external Linksys router over WiFi.

It doesn’t make sense, cause in all cases everything is the same connecting over same WiFi router, however when on same PC it could be smart enough not to go to router and route locally.

External router Linksys
Virtual Platform on both PCs is Hyper V

Any suggestions or pointers where to dig further would be much appreciated! as my head is starting to spin a little :slight_smile:

#8

Configuration for data exchange between networks: 192.168.5.0/24 - 192.68.2.0/24:

VyOS 1.1.7:

delete nat source rule 9
delete nat source rule 10

set nat source rule 10 destination address !192.168.5.0/24
set nat source rule 10 outbound-interface eth0
set nat source rule 10 source address 192.168.2.0/24
set nat source rule 10 translation address 192.168.1.254

Other rules change masquerade to ip 192.168.1.254

VyOS 1.3.0-rc6:

set nat source rule 10 destination address !192.168.2.0/24
set nat source rule 10 outbound-interface eth0
set nat source rule 10 source address 192.168.5.0/24
set nat source rule 10 translation address 192.168.1.253

Other rules change masquerade to ip 192.168.1.253

#9

Thank you, RyVolodya, however still doesn’t seem to work when deployed on two different computers. Posting below current configuration again in case something catches the eye.
The fact that opposite WAN interfaces are pingable tells that packets go between computers through WiFi router fine, but pings between LANs are still timing out.
Tried removing static routes config for the sake of test as well but didn’t help.

VYOS1
vyos@vyos# show interfaces
ethernet eth0 {
address 192.168.1.254/24
hw-id 00:15:5d:01:6a:66
}
ethernet eth1 {
address 192.168.2.254/24
hw-id 00:15:5d:01:6a:67
}
loopback lo {
}
[edit]
vyos@vyos# show nat
source {
rule 10 {
destination {
address !192.168.5.0/24
}
outbound-interface eth0
source {
address 192.168.2.0/24
}
translation {
address 192.168.1.254
}
}
}
[edit]
vyos@vyos# show protocols
static {
route 0.0.0.0/0 {
next-hop 192.168.1.1 {
}
}
route 192.168.5.0/24 {
next-hop 192.168.1.253 {
}
}
}
[edit]
vyos@vyos#

VYOS2
vyos@vyos# show interfaces
ethernet eth0 {
address 192.168.1.253/24
hw-id 00:15:5d:01:a0:0e
}
ethernet eth1 {
address 192.168.5.253/24
hw-id 00:15:5d:01:a0:0f
}
loopback lo {
}
[edit]
vyos@vyos# show nat
source {
rule 10 {
destination {
address !192.168.2.0/24
}
outbound-interface eth0
source {
address 192.168.5.0/24
}
translation {
address 192.168.1.253
}
}
}
[edit]
vyos@vyos# show protocols
static {
route 0.0.0.0/0 {
next-hop 192.168.1.1 {
}
}
route 192.168.2.0/24 {
next-hop 192.168.1.254 {
}
}
}
[edit]
vyos@vyos#

#10

Hi. Regardless configuration, while pinging from LAN-1 to LAN-2, you may analyze what’s going on using tcpdump:
If I’m not wrong you have: LAN_01 → eth1 VYOS-1 → eth0 VYOS-1 → eth0 VYOS-2 → eth1 VyOS-2 → LAN_02

So, if pinging from host located on LAN_01, I would analyze traffic using tcpdump, from left to right

# 1 Lan interface on VyOS-1
sudo tcpdump -i eth1

#2 WAN interface on VyOS-1
sudo tcpdump -i eth0

#3 WAN interface on VyOS-2
sudo tcpdump -i eth0

#4 LAN interface on VyOS-1
sudo tcpdump -i eth1

This will let you find where packet is missing and if nat is applying as you expect

#11

Hi ddv

Here is just my opinion, may be in-correct, but you may consider and try.

  1. If you enable NAT (no matter in any form of NAT), endpoints behind a router seems cannot direct communicate with other endpoints behind another router.

  2. You mentioned that if you installed two routers in two PCs via Hyper V, two outside interfaces of two routers can ping each other as well as the External router. I guess all they are seat in the same subnet, right? So they can direct communicate with each other. But traffic between endpoints from two routers should go through VyOS1 → Hyper V1 → PC1 → PC2 → Hyper V2 → VyOS2. As your PCs may never know about your endpoints subnets, so the traffic dropped.

You can add static routes on each PCs and try.

In PC holding192.168.1.254 router, execute the following commands in PowerShell (with ADMIN rights):
route add 192.168.5.0 mask 255.255.255.0 192.168.1.253
route add 192.168.6.0 mask 255.255.255.0 192.168.1.253

In PC holding 192168.1.253 router, execute the following:
route add 192.168.2.0 mask 255.255.255.0 192.168.1.254
route add 192.168.3.0 mask 255.255.255.0 192.168.1.254

Hope my thinking is correct, good luck…

#12

Ar…

Even the PC holding 192.168.1.254, it may never know 192.168.2.0/24 and 192.168.3.0/24 too, this also true in PC holding 1921.68.1.253 as well. So all the four static routes should be added to both PCs, please try and good luck…

#13

sorry… my think should be wrong… don’t waste time to try… :disappointed_relieved:

#14

Thank you very much for your responses, All! However I still couldn’t make it work. Did some reading on Hyper V networking and came to a conclusion that it might not be possible altogether. For routing traffic between guest VMs on different Hyper V hosts some special ‘lookup records’ have to be created, something like static routes for Hyper V, and for that you need a ‘Windows Network Virtualization’ module, which is supported on Windows Servers only, so perhaps routing between two Hyper V hosts on Windows Clients will not be possible (see reference link for details).

If somebody happen to figure out how to configure routing between guest VMs running on Hyper V Windows Clients, I would be so happy to hear it!

#15

Hey DDV!

Quick question, can you describe in more detail your underlay network and where the VyOS imgs are hosted?

#16

Hey Lean,
I have two window client hosts running Hyper V. There are VYOS router on each, with WAN interface connecting to WiFi router over virtual external switch. Internal VYOS interfaces connect to VMs over private virtual switches. Trying to connect VMs on different hosts into the same domain network via two VYOS routers but so far no luck.
Hope this makes sense. There are more details with particular configs earlier in this thread.