Recently I build a complex nested lab in vCloud Air that would allow me to quickly jump in to a VMware environment from anywhere in the world. This is great for grabbing a few screenshots, testing something quickly or installing the latest and greatest features.
I really wanted this environment to be long lived and something that I can upgrade and mature over time. So I wanted to make sure that it’s not simply an isolated pod in a nested world with no access to the outside world. Tomas Fojta from my team created a fantastic blog around the definition of a complex nested lab in vCloud Air (I highly recommend you read this) So I followed this as a starting point: Complex Nested Lab in vCloud Air
The first thing I needed to do was plan the deployment; I’m an architect, so a little design work went in to this initially – I’m not talking 100 pages of architecture blueprints, but a small diagram to highlight my proposed architecture:
The proposed architecture had 3 vApp’s within vCloud Air, all with specific org vDC networks routed via the vCloud Edge Services Gateway (spine if you will). The Resource cluster has 3 virtual ESXi hosts and the Edge cluster has the same. Each vmnic presented to the vESXi hosts has it’s own dvSwitch as they are connected to separate vCloud Director networks. I was planning on leveraging vSAN for the shared storage solution, which I had to create emulated SSD’s. For that I leveraged William Lam’s post: How to Trick ESXi 5 in seeing an SSD Datastore.
The next challenge I had was with the deployment of the NSX controllers as they have to be nested as the NSX manager deploys them directly in to the nested infrastructure. The challenge is how can you communicate externally to the management ecosystem from a nested virtual machine on a flat port-group with mac addresses that the vCloud Air infrastructure doesnt know about? In real environments this is easy, you can enable promiscuous mode on the dvSwitch which would enable the virtual MAC’s to be presented through the virtual hosts and in to vCloud Air. We cannot enable this in vCloud Air as we don’t have access to the Virtual Infrastructure layer.
To solve this challenge you need to create a nested edge services gateway and map virtual MAC address to the same mac address assigned to the virtual ESXi hosts NIC. Verify the NIC’s MAC address in vCloud Air (vCloud Director) and when you deploy the ESG you can specify the MAC address for the uplink interface. Now the traffic could get out of the nested router upstream to the vCloud Air Environment, but how does the traffic know to come back in to the nested environment? You need to create a static route on the vCloud Air Edge Service Gateway for the network specified back to the uplink interface of the nested ESG. Now when you deploy your nested NSX controllers, they have upstream connectivity with the NSX manager and the NSX manager can apply the correct configuration for the controller. The following diagram highlights this in my nested environment:
This solution allows me to be able leverage all the NSX services in my nested lab for testing and demo purposes. I can also access this from anywhere in the world.