VYOS asymmetric routing issue with 3 interface setup

General questions
, ,
1

Hello Team,

I am currently deploying a VyOS instance in a PoC environment and looking for the most logical and robust approach to resolve an asymmetric routing issue regarding management traffic.

The Scenario & Architecture:
I have a 3-interface setup designed for strict traffic isolation:

eth0: Dedicated for Management (SSH) and DNS traffic. It also maintains active BGP sessions with local DNS servers.

eth1 : Used for establishing GRE tunnels to get the traffic.

eth2 : Internet breakout. The system’s default route (0.0.0.0/0) points to the gateway on this interface.

The Issue:
The customer requires SSH access strictly via eth0. When an SSH connection is initiated from an external management subnet to eth0, the inbound packets arrive correctly. However, because the main routing table dictates the default route via eth2, VyOS sends the return traffic (SYN-ACK) out through eth2. This asymmetric routing breaks the connection. Currently, SSH is only successful when initiated from the exact same local subnet as eth0.

My Question:
What is the officially recommended VyOS “best practice” to resolve this asymmetric routing for locally generated return traffic?

Thank you in advance for your insights.

2

To me the best practice is to NEVER mix production and management traffic.

That is if you use eth0 for mgmt then ONLY use management here as in remote access + logging and ntp (so the VyOS box have correct time) etc.

Other than that Im using vrfs (virtual routing and forwarding) to have separate routingtables between MGMT and PROD.

In VyOS there is also netns (network namespaces) that can be configured but they currently dont seem to have any usage yet.

Compared to others (lets say Arista) when you create a vrf there what happens in the backend is that they will create a netns for full isolation of the interfaces along with a vrf as in dedicated routingtable.

So what VyOS currently supports is whats often refered to as “vrf-lite” (just separate routingtables and not fully separated interfaces).

Here is some good reading on this topic:

And here is a config-example:

set interfaces ethernet eth0 address '10.99.0.100/24'
set interfaces ethernet eth0 hw-id '<REMOVED>'
set interfaces ethernet eth0 ip arp-cache-timeout '240'
set interfaces ethernet eth0 ip source-validation 'strict'
set interfaces ethernet eth0 ipv6 disable-forwarding
set interfaces ethernet eth0 ipv6 source-validation 'strict'
set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload rfs
set interfaces ethernet eth0 offload rps
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth0 ring-buffer rx '1024'
set interfaces ethernet eth0 ring-buffer tx '256'
set interfaces ethernet eth0 vrf 'MGMT'
set interfaces ethernet eth1 address '192.168.1.20/24'
set interfaces ethernet eth1 hw-id '<REMOVED>'
set interfaces ethernet eth1 ip arp-cache-timeout '240'
set interfaces ethernet eth1 ip source-validation 'strict'
set interfaces ethernet eth1 ipv6 disable-forwarding
set interfaces ethernet eth1 ipv6 source-validation 'strict'
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload rfs
set interfaces ethernet eth1 offload rps
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces ethernet eth1 ring-buffer rx '1024'
set interfaces ethernet eth1 ring-buffer tx '256'
set interfaces ethernet eth1 vrf 'PROD'
set interfaces loopback lo ip source-validation 'strict'
set netns name MGMT
set netns name PROD
set vrf bind-to-all
set vrf name MGMT ipv6 disable-forwarding
set vrf name MGMT protocols static route 0.0.0.0/0 next-hop 10.99.0.254 interface 'eth0'
set vrf name MGMT table '100'
set vrf name MGMT vni '10000100'
set vrf name PROD ipv6 disable-forwarding
set vrf name PROD protocols static route 0.0.0.0/0 next-hop 192.168.1.254 interface 'eth1'
set vrf name PROD table '101'
set vrf name PROD vni '10000101'

In above case I have disabled IPv6 and as you can see defined which interface belongs to which vrf.

And then each vrf can have its own default routes and whatelse.

Note that any service not setup/configured for vrf will use the default vrf “local” so with above setup these services will not function any longer.

You can see how the kernel currently see things (routingwise) by doing something like:

cat /proc/1/task/1/net/fib_trie

To fix this you need to define vrf for each service aswell something like this:

set service ntp server 10.99.0.10 prefer
set service ntp vrf 'MGMT'
set service ssh access-control allow user 'vyos'
set service ssh cipher 'aes256-gcm@openssh.com'
set service ssh disable-host-validation
set service ssh dynamic-protection allow-from '10.99.0.0/24'
set service ssh dynamic-protection block-time '60'
set service ssh dynamic-protection detect-time '3600'
set service ssh dynamic-protection threshold '10'
set service ssh listen-address '10.99.0.100'
set service ssh mac 'hmac-sha2-512'
set service ssh rekey data '1024'
set service ssh rekey time '60'
set service ssh vrf 'MGMT'

In above example the local NTP service will sync VyOS against 10.99.0.10 located on vrf MGMT.

And similar for SSH where I define both listenaddress and which vrf to be used along with an “allow-from” to filter who will be able to handshake with the service.

Also dont forget if possible to also setup proper filtering in the firewall section (aka layered security). But I leave that exercise to the reader :wink:

3

Hello @Apachez,

Thank you for the detailed reply and the configuration examples!

This is my first time working with VyOS. I have previously worked with pfSense and also had the opportunity to examine a TWAG setup. While I fully understand how VRFs work from my past experience, I am trying to avoid making the configuration too complicated at this stage of the PoC.

Since I designed this entire VyOS configuration myself, I am starting to wonder if I created a fundamental design flaw. If I didn’t dedicate a separate port (eth2) for the internet breakout and instead routed that traffic out through eth1 (the same interface where the client traffic comes from), wouldn’t this asymmetric routing problem be inherently resolved?

I would appreciate your thoughts on whether consolidating those interfaces is a more logical design choice here to keep things simple.

4

I would still recommend to do things properly from the beginning because suddently your PoC code goes into production and if you ignored best practices because “its just PoC” that will come and bite you later on.

One of the issues Mikrotik have had that admins using Mikrotik just connect it to internet and dont segment management from production so within a few hours their gear have been taken over by botnets and used in DDoS-attacks against others.

Regarding asymetric routing or not if you got 2x ISPs connected to two different interfaces that will be “asymetric” since its the sending device who will look in its routingtable and select a nexthop. If your box only have a single nexthop then sure you wont have any asymetric routing issues since there is no asymetric routing to begin with - you need at least 2x nexthops before “asymetric routing” comes into play.