Strange Routing Issue

General questions
1

Trying to setup a lab that emulates our live network(3 sites, point to point connections between 2 sites and a data center). This is all setup in an ESX server with multiple vSwitches to emulate the separation of the networks. 3 VyOS routers “connected” as follows:

vyos-oh ------ vyos-datacenter ------ vyos-fl

Configs for each below:

vyos-OH

[code]vyos@vyos-oh# show interfaces
ethernet eth0 {
address 172.16.30.100/23
description OUTSIDE
duplex auto
hw-id 00:0c:29:c3:5d:ef
smp_affinity auto
speed auto
}
ethernet eth1 {
address 10.0.100.1/24
description INSIDE
duplex auto
hw-id 00:0c:29:c3:5d:f9
smp_affinity auto
speed auto
}
ethernet eth2 {
address 10.250.0.1/30
description DATACENTER
duplex auto
hw-id 00:0c:29:c3:5d:03
smp_affinity auto
speed auto
}
loopback lo {
}

[edit]
vyos@vyos-oh# show protocols static
route 0.0.0.0/0 {
next-hop 172.16.30.1 {
distance 1
}
}
route 10.0.105.0/24 {
next-hop 10.250.0.2 {
distance 1
}
}
route 10.0.110.0/24 {
next-hop 10.250.0.2 {
distance 1
}
}

[edit]
vyos@vyos-oh# show nat source
rule 100 {
outbound-interface eth0
source {
address 10.0.100.0/24
}
translation {
address masquerade
}
}

[edit]
vyos@vyos-oh# [/code]

vyos-datacenter

[code]
vyos@vyos-datacenter# show interfaces
ethernet eth0 {
address 172.16.30.102/23
description OUTSIDE
duplex auto

[edit]
vyos@vyos-datacenter# show interfaces
ethernet eth0 {
address 172.16.30.102/23
description OUTSIDE
duplex auto
hw-id 00:0c:29:57:5d:ac
smp_affinity auto
speed auto
}
ethernet eth1 {
address 10.0.110.1/24
description INSIDE
duplex auto
hw-id 00:0c:29:57:5d:b6
smp_affinity auto
speed auto
}
ethernet eth2 {
address 10.250.0.2/30
description OH
duplex auto
hw-id 00:0c:29:57:5d:c0
smp_affinity auto
speed auto
}
ethernet eth3 {
address 10.250.10.2/30
description FL
duplex auto
hw-id 00:0c:29:57:5d:ca
smp_affinity auto
speed auto
}
loopback lo {
}

[edit]
vyos@vyos-datacenter# show protocols static
route 0.0.0.0/0 {
next-hop 172.16.30.1 {
distance 1
}
}
route 10.0.100.0/24 {
next-hop 10.250.0.1 {
distance 1
}
}
route 10.0.105.0/24 {
next-hop 10.250.10.1 {
distance 1
}
}

[edit]
vyos@vyos-datacenter# show nat source
rule 100 {
outbound-interface eth0
source {
address 10.0.110.0/24
}
translation {
address masquerade
}
}

[edit]
vyos@vyos-datacenter#[/code]

vyos-FL

[code]
vyos@vyos-fl# show interfaces
ethernet eth0 {
address 172.16.30.101/23
description OUTSIDE
duplex auto
hw-id 00:0c:29:70:84:f5
smp_affinity auto
speed auto
}
ethernet eth1 {
address 10.0.105.1/24
description INSIDE
duplex auto
hw-id 00:0c:29:70:84:ff
smp_affinity auto
speed auto
}
ethernet eth2 {
address 10.250.10.1/30
description DATACENTER
duplex auto
hw-id 00:0c:29:70:84:09
smp_affinity auto
speed auto
}
loopback lo {
}

[edit]
vyos@vyos-fl# show protocols static
route 0.0.0.0/0 {
next-hop 172.16.30.1 {
distance 1
}
}
route 10.0.100.0/24 {
next-hop 10.250.10.2 {
distance 2
}
}
route 10.0.110.0/24 {
next-hop 10.250.10.2 {
distance 1
}
}

[edit]
vyos@vyos-fl# show nat source
rule 100 {
outbound-interface eth0
source {
address 10.0.105.0/24
}
translation {
address masquerade
}
}

[edit]
vyos@vyos-fl#[/code]

I can ping from vyos-oh to vyos-datacenter. I can ping from vyos-fl to vyos-datacenter. I can ping oh and fl from datacenter but I cannot ping fl from oh and vice versa.

There are currently no firewall rules in place at the moment. Is there something obvious that I am missing?

Trace routes from OH to FL die at the datacenter and vice versa.

2

Since you have multiple directly connected subnets, I would start by making sure that ping is choosing the correct interface to send icmp packets on. Then I would tcpdump on each vyos router on each interface to see the flow and where you expect packets to leave and where you expect packets to arrive. Chances are when you send the ping it is being sent out the wrong interface on one of the routers, either on the send direction or the return trip. NAT looks like it’s not a problem, but you could disable to rule it out and get routing working.

3

So when you mentioned about checking to make sure it was going out the correct interface, I tried to do a tracert to 10.0.105.1 from a test server I have setup on the 10.0.100.0/24 network and it worked. Why would the vyos-oh router not be able to do it? Seems rather strange.

Here is from the server. Server address is 10.0.100.10 if that matters.[/code]

Tracing route to 10.0.105.1 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.0.100.1
2 <1 ms <1 ms <1 ms 10.250.0.2
3 <1 ms <1 ms <1 ms 10.0.105.1

Trace complete.