Nested vSphere with one vyos issue on nested ESXi host connection

General questions
1

Background:

  • Nested 3 x ESXi on 2 x Physical vSphere host environment. The intention is to have 1 x vyos migrated between two physical hosts to balance the load.

  • Phy vSphere hosts with 1 x vDS with 2 x uplink (physical nic - trunk port on phy switches), 1 x PG N-ESX (VGT 4094), Promiscuous, Mac address changes, Forged Transmit as accept

  • 1 x vyos (eth0 with dummy 192.168.100.1/24, 192.168.10.1/24. eth1 with vif 10, 192.168.10.1/24, vif 20 for nested vm @192.16.20.1/24, vif 30 for nested vmotion, vif 40 for nested iscsi storage), isolated environment, no internet connection hence no static route configured.

  • nested ESX1 and 2 installed, vswitch mgmt 192.168.10.11, 192.168.10.12

  • 1 x dc nested vm (192.168.20.200), 1 x test vm (192.68.10.100)

  • dc and test vm can ping each other and all gateway. However dc and test can’t ping ESX1 & 2 (192.168.10.11, 192.168.10.12)

vyos configuration
set interfaces ethernet eth0 address 192.168.100.1/24
set interfaces ethernet eth1 vif 10 address 192.168.10.1/24
set interfaces ethernet eth1 vif 20 address 192.168.20.1/24
set interfaces ethernet eth1 vif 30 address 192.168.30.1/24
set interfaces ethernet eth1 vif 40 address 192.168.40.1/24

Need to fix this issue before moving to Nested vCenter installation. Any suggestion how to make it work. Thanks,

2

Hi @OhCanada,

Can you please elaborate a bit further on what your portgroup configuration looks like? Perhaps you mean VGT 4095?

Typically for nested hosts I would normally just configure your portgroup(s) that your nested hosts are connected to as having a VLAN trunk of 1-4094. Regarding your portgroup security settings, it would be worth investigating whether you can enable Mac learning in your environment (I think vDS 6.7 introduced this capability but requires API calls to VC to enable it on a per-pg basis, William Lam has some good blog posts on this, it’ll significantly improve performance and reduce CPU overheads as well as give you efficient data plane forwarding).

Perhaps you can draw a diagram to better explain your setup, I am not able to derive things like:

  • Are your vyos vm(s) running in a HA pair?
  • Are the vyos vm(s) running on the physical hosts or on the nested hosts?
  • Are the dc and test vm running on the physical hosts or the nested hosts?
  • If the dc and test vm are running on nested hosts, are both of those nested hosts on the same physical host? I’ve not seen many (if any) environments where promiscuous mode works across multiple physical ESXi hosts.
  • Where is L3 terminated for the VLANs on your physical host(s)?

Are you running vyos 1.3 or 1.4? I have recently hit a terrible bug with the 1.4 release whereby having multiple NICs (eth0/Network Interface 1 and eth1/Network Interface 2 in your case) doesn’t work because the mapping of vNICs to ethX interfaces is broken.

Cheers,
Kane.