Multiple external interfaces with 1to1 DNAT

nat

#1

First of all I would like to say Hi to everyone since I new here.

I would like to configure VyOS as a Virtual Router on VirtualBox or VMWare Player in configuration showed below :

[attachment=145]

And now I would like to set 1to1 DNAT for each VM. I configure VyOS like this :

 interfaces {
     ethernet eth0 {
         address 192.168.109.1/24
         description LAN
         duplex auto
         hw-id 08:00:27:be:63:bc
         smp_affinity auto
         speed auto
     }
     ethernet eth1 {
         address dhcp
         description WAN
         duplex auto
         hw-id 08:00:27:ca:91:84
         smp_affinity auto
         speed auto
     }
     ethernet eth2 {
         address dhcp
         description EXT1
         duplex auto
         hw-id 08:00:27:6d:68:4f
         smp_affinity auto
         speed auto
     }
     ethernet eth3 {
         address dhcp
         description EXT2
         duplex auto
         hw-id 08:00:27:e1:7d:d9
         smp_affinity auto
         speed auto
     }
     ethernet eth4 {
         address dhcp
         description EXT3
         duplex auto
         hw-id 08:00:27:4e:73:0f
         smp_affinity auto
         speed auto
     }
     ethernet eth5 {
         address dhcp
         description EXT4
         duplex auto
         hw-id 18:00:27:56:a2:32
         smp_affinity auto
         speed auto
     }
     loopback lo {
     }
 }
 nat {
     destination {
         rule 20 {
             inbound-interface eth2
             translation {
                 address 192.168.109.20
             }
         }
         rule 30 {
             inbound-interface eth3
             translation {
                 address 192.168.109.21
             }
         }
         rule 40 {
             inbound-interface eth4
             translation {
                 address 192.168.109.22
             }
         }
         rule 50 {
             inbound-interface eth5
             translation {
                 address 192.168.109.23
             }
         }
     }
     source {
         rule 1 {
             outbound-interface eth1
             source {
                 address 192.168.109.0/24
             }
             translation {
                 address masquerade
             }
         }
         rule 20 {
             description "EXT1 to VM1"
             outbound-interface eth2
             source {
                 address 192.168.109.20
             }
             translation {
                 address masquerade
             }
         }
		 rule 30 {
             description "EXT2 to VM2"
             outbound-interface eth3
             source {
                 address 192.168.109.21
             }
             translation {
                 address masquerade
             }
         }
		 rule 40 {
             description "EXT3 to VM3"
             outbound-interface eth4
             source {
                 address 192.168.109.22
             }
             translation {
                 address masquerade
             }
         }
		 rule 50 {
             description "EXT4 to VM4"
             outbound-interface eth5
             source {
                 address 192.168.109.23
             }
             translation {
                 address masquerade
             }
         }
     }
 }
 service {
     dhcp-server {
         disabled false
         shared-network-name LAN {
             authoritative enable
             subnet 192.168.109.0/24 {
                 default-router 192.168.109.1
                 dns-server 192.168.109.1
                 dns-server 10.3.3.33
                 lease 86400
                 start 192.168.109.100 {
                     stop 192.168.109.254
                 }
                 static-mapping vm1 {
                     ip-address 192.168.109.20
                     mac-address 08:00:27:9A:BD:84
                 }
                 static-mapping vm2 {
                     ip-address 192.168.109.21
                     mac-address 08:00:27:7F:04:3F
                 }
                 static-mapping vm3 {
                     ip-address 192.168.109.22
                     mac-address 08:00:27:98:28:89
                 }
                 static-mapping vm4 {
                     ip-address 192.168.109.23
                     mac-address 08:00:27:D5:BC:FE
                 }
             }
         }
     }
     dns {
         forwarding {
             cache-size 0
             listen-on eth0
             name-server 10.3.3.33
         }
     }
     ssh {
         port 22
     }
 }
 system {
     config-management {
         commit-revisions 20
     }
     console {
         device ttyS0 {
             speed 9600
         }
     }
     host-name vyos
     login {
         user vyos {
             authentication {
                 encrypted-password $1$HR42KG7n$Ynpv5D8LEnJiOZPX85Wt.1
                 plaintext-password ""
             }
             level admin
         }
     }
     ntp {
         server 0.pool.ntp.org {
         }
         server 1.pool.ntp.org {
         }
         server 2.pool.ntp.org {
         }
     }
     package {
         auto-sync 1
         repository community {
             components main
             distribution helium
             password ""
             url http://packages.vyos.net/vyos
             username ""
         }
     }
     syslog {
         global {
             facility all {
                 level notice
             }
             facility protocols {
                 level debug
             }
         }
     }
     time-zone UTC
 }

the problem is that I can reach only one external interface of my router from Workstations and Server which is strange.
I take care of MAC addresses to be different, my VyOS Router gets addresses from DHCP and everything looks fine in configuration.
I found that the Workstation can’t find other external interfaces of my VyOS in ARP when I add it manually ping and other services works.

I tried on both VirtualBox and VMWare and even on different VM Servers the result was the same every time.

I also found out that even without any nat configuration I can reach only one external interface.

After several tries I tried to run Live Fedora to check if I will be able to reach any of external interfaces connected to VM and after fedora got 5 external addresses from DHCP I was able to reach all of them from Workstation machine.

After that try I reboot VM with VyOS and result was as before only one interface respond ;/
I can ping Workstations through every external interface (ping 10.3.53.150 -i ethX) but in other hand I can ping only one VyOS external interface from Workstation (typically this interface who got the address from dhcp first)

May by I missed something in my configuration. I can’t get it works well it look like VyOS register only one external interface.

To repeat most basic example just create VM with VyOS atach to it more then one bridged interface configure it as external and got adress from dhcp and try to ping from other machine in your physical network.


#2

1st step I’d take is try to get rid of DHCP and use fixed IP addresses instead.
That paves the way for next step: Put all WAN IPs on a single interface. Now NAT-ting becomes easier, since all your rules can be configured on single interface.

The problem with current set-up is routing: You probably have 5 equi-distant routes on 5 interfaces. For each packet to be sent out, Vyos first picks one out of 5 interfaces to send it on. After this routing step source NAT takes places, but only NAT rule(s) configured for the randomly chosen interface is applied.
To get this working , you need PBR, which is way more difficult.


#3

I can’t get rid of DHCP for external interfaces of my VyOS Router because I do not have administration right for 10.0.53.0/22 network.
I can just plug another computer to the network to that network by plug it by cable into the wall :slight_smile: and get adress from DHCP.
In my case set up static adresses for “my virtual environment” will require some paper work, approvel and so on (international corporation).
And in 10.0.53.0/22 network is lots more then 3 workstation that I draw on schema.

And what about when all NAT rules are disabled and only one interface of VyOS respond when Fedora replays on everyone?


#4

As an alternative , just run 4 Vyos instances, so VM1…4 all have their own router. Vyos runs fine in 512MB RAM , you could try using even less.


#5

It is kind of solution but It will be waste of resources especially RAM and configuration of 192.168.109.0/24 network will complicate to get machines in this network see each other.

I’m open for more complicate configuration for one router even on Policy Based.
And I still don’t understand why without nat configuration VyOS responds only on one external interface.


#6

No complication on 192.168.109.0/24 network, all interfaces just plug into a single virtual switch (8 interfaces for 4 vyos instances , for a single vyos instance you would have 5).
Only thing to set-up properly is default gateway, it’s different for each VM


#7

Why would you like to have the router? Could you not just bridge all the VMs?


#8

Short answer is no I can’t because there are some services on Windows and Linux machines which should be separate from 10.3.53.0/22 network with have domain and Active Directory.
They have their own domain witch name, Active Directory, Kerberos and other and they are configured to use 192.168.109.0/24 network.
Those machines are frequently exchanged by semi-automated deployment system.
Besides we got similar configuration on vCloud on other spot, so I wouldn’t like to change machine definition and I don’t have license for vCloud.

I know it is little strange requirement but lets treat that as a challenge.
I’m looking for one virtual router solution because of limited resources, and I don’t care about firewall and other “security” stuff since it will be test environment.