Not able to get anything working without a masquerade

General questions
,
1

2 networks,. each with their own subnet configuration. each with their own internet connection (my house and neighbors house im trying to let on to my Plex server without using our internet connection since we both have Starlink and the upstream isnt great.)

network 1, 192.168.50.0/24
network 2, 192.168.1.0/24
wireless ethernet bridge on the 192.168.1.0/24 network (engenius enh500v3).
the bridge terminates at an esxi host with 2 nics (vswitches).
vyos is running as virtual machine on esxi host and can see both networks.

vyos virtual machine-
eth0: 192.168.1.254/24
eth1: 192.168.50.254/24

network 1, static route - 192.168.1.0/24 - next-hop - 192.168.50.254
network 2, static route - 192.168.50.0/24 - next-hop - 192.168.1.254

i can get this all working if i do the following on vyos –

nat {
source {
rule 100 {
outbound-interface {
name eth0
}
source {
address 192.168.0.0/16
}
translation {
address masquerade
}
}
}
}

but i dont feel like this is right. im not trying to provide access to an internet connection… it seems like traffic should flow both ways, not all designated to eth0 as an outbound. if i remove the nat entirely, i cant ping anything on the other network. if i leave it there, then everything show’s as source the .254 for each network, where as, with my logging, id like to see individual class c ip addresses for each subnet.

2

nightly build –

vyos@vyos# run show version
Version: VyOS 1.5-rolling-202401030023
Release train: current

Built by: autobuild@vyos.net
Built on: Wed 03 Jan 2024 01:47 UTC
Build UUID: ddf469e6-3c50-4114-abb9-d1e93d6e9579
Build commit ID: db11c4bcefba6c

Architecture: x86_64
Boot via: installed image
System type: VMware guest

Hardware vendor: VMware, Inc.
Hardware model: VMware Virtual Platform
Hardware S/N: VMware-56 4d ab 90 6a b0 13 cc-27 7e 65 0e 2c d3 ae f8
Hardware UUID: 90ab4d56-b06a-cc13-277e-650e2cd3aef8

Copyright: VyOS maintainers and contributors

3

Are these configured on the default gateways/routers of each network? Normally if it doesn’t work without NAT it’s because of a lack of correct routes in my experience.

4

yes.
network 1 has a ubiquiti dream station se with that static route defined
network 2 has a netgear nighhawk x6 with that static route defined

5

My other thought is that maybe firewall on a gateway is blocking traffic from the other subnet. If there are traffic logs available on either, I would suggest checking that. You could also use the monitor traffic interface operational command in VyOS to see if return packets are even hitting VyOS.

6

so, just to make sure i understand this, you are saying it should be working without the nat configuration in place?

7

If the routes are correct as you said, yes it should work without NAT. This is a very simple networking configuration.

Could you provide output of show ip route on VyOS? Also, is VyOS itself able to ping individual hosts of each subnet without the NAT rule?

8

Yes, I dont have any problems pinging individual hosts from vyos. however, unable to ping across the the subnet’s from client machines without the nat rule in place.

vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>* 192.168.1.0/24 is directly connected, eth1, 22:50:31
C>* 192.168.50.0/24 is directly connected, eth0, 22:50:31
vyos@vyos:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.50.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
vyos@vyos:~$
9

Everything beyond this is default configuration values, added this along with a service ssh/22.

vyos@vyos:~$ sh conf
interfaces {
    ethernet eth0 {
        address 192.168.50.254/24
        hw-id 52:54:00:7a:34:ca
    }
    ethernet eth1 {
        address 192.168.1.254/24
        hw-id 52:54:00:5e:cb:78
    }
    loopback lo {
    }
}
nat {
    source {
        rule 100 {
            outbound-interface {
                name eth1
            }
            source {
                address 192.168.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
10

i rebuilt this again yesterday, its now a VM on my qnap ts-673a (no longer vm on esxi running on an old mac pro darth vader trashcan).

vyos@vyos:~$ sh version
Version:          VyOS 1.5-rolling-202401030023
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Wed 03 Jan 2024 01:47 UTC
Build UUID:       ddf469e6-3c50-4114-abb9-d1e93d6e9579
Build commit ID:  db11c4bcefba6c

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:
Hardware UUID:    4359f940-edcf-446b-a030-5f4ed55c9855

Copyright:        VyOS maintainers and contributors

vyos@vyos:~$ sudo su -
root@vyos:~# cat /proc/sys/net/ipv4/ip_forward
1
root@vyos:~# uname -a
Linux vyos 6.1.70-amd64-vyos #1 SMP PREEMPT_DYNAMIC Mon Jan  1 23:47:35 UTC 2024 x86_64 GNU/Linux
root@vyos:~# dmesg|grep eth
[    2.122064] e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:7a:34:ca
[    2.122639] e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection
[    2.497885] e1000 0000:00:04.0 eth1: (PCI:33MHz:32-bit) 52:54:00:5e:cb:78
[    2.498334] e1000 0000:00:04.0 eth1: Intel(R) PRO/1000 Network Connection
[    2.502517] e1000 0000:00:03.0 e2: renamed from eth0
[    2.509609] e1000 0000:00:04.0 e3: renamed from eth1
[   22.488634] e1000 0000:00:03.0 eth0: renamed from e2
[   22.512029] e1000 0000:00:04.0 eth1: renamed from e3
[   39.368526] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   39.369085] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   39.775837] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[   39.776443] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
11 
root@vyos:~# scp user@192.168.50.21:/home/user/neofetch .
user@192.168.50.21's password:
neofetch                                                                              100%  334KB  19.9MB/s   00:00
root@vyos:~# ls
neofetch
root@vyos:~# chmod +x neofetch
root@vyos:~# ./neofetch
        #####           root@vyos
       #######          ---------
       ##O#O##          OS: VyOS 1.5-rolling-202401030023 (current) x86_64
       #######          Host: KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i440fx-4.1)
     ###########        Kernel: 6.1.70-amd64-vyos
    #############       Uptime: 1 day, 2 mins
   ###############      Packages: 780 (dpkg)
   ################     Shell: bash 5.2.15
  #################     CPU: AMD Opteron 22xx (Gen 2 Class Opteron) (2) @ 2.195GHz
#####################   GPU: 00:02.0 Cirrus Logic GD 5446
#####################   Memory: 268MiB / 1989MiB
  #################
12

i deleted the nat, and now, all of a sudden, its now working. i did this 5 times in the last few days… not sure what has changed. thanks all !

13

Is that with it on the Qnap? I wonder if ESXi was interfering somehow if that is the case.

14

yes, this is with it on the qnap. same nic configuration,etc. not sure what would be different which is now causing it to perform like it should. very strange.