Running nested ESXi –the real inception

Nested ESXi servers are essentially virtualised ESXi servers.  They can be quiet fun to play with.  Usually the only reason you’d do this is for a test lab.  You don’t need a physical ESXi server to nest virtual ESXi hypervisors.  You can choose your product of choice.

In my case I’m running VMware Workstation 9 and virtualised ESXi 5.1 as the second level and then running virtual guests as a third level within the virtual ESXi.  In Workstation 9 you have the ability to connect to ESXi hosts and view guest VMs --as below.

You don’t need to do anything special to virtualize ESXi.  In fact there’s an option to select ESX as the guest OS in VMware Workstation 9.

When you turn on a virtual ESXi guest (or is it a host???) a message appears in the bottom right corner of the console.  It’s pretty self-explanatory.  You can create a 32-bit guest VM only on this ESXi server.

The cool cats that VMware are, allow you to run 64-bit VM guests too with a small config change.  The feature is only available in ESXi 5 and above.

On the virtualised ESXi guest enable console access (or SSH) and login as root.  The file ‘config’ in /etc/vmware needs to be modified and an additional line add, ‘vhv.allow = “TRUE”’.

If you’re a VI fan go ahead and use that.  If you’re lazy like me you can just append it to the file with echo.


echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Once the change is made no reboot is necassary.  You will now be able to run 64-bit guest VMs within your already virtualised ESXi host.  In an extreme example I ran another ESXi guest at the third level to prove a 64-bit OS would run.  On a second virtualised ESXi guest I’m just running a 32-bit Linux VM at that same third level.

For my test lab three levels is as deep as I need to go.  I’ve read that it’s also as deep as you can go as guest VMs will fail to run at a fourth level.  I don’t have the resources to test this out in my lab.  I heard that there was a VMworld presentation on multiple level nesting ESXi servers.  Would certainly be worth finding.

 

Appendix

As with many things VMware, nested ESXi is not officially supported.
VMware Link:  Support for running ESX/ESXi as a nested virtualization solution

I had trouble with the vmxnet3 network drivers with the nested ESXi.  Instead I chose to go with E1000 particular for that third level.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>