Tag Archives: esx

EqualLogic Multipathing Extension Module – Installing

Last year I wrote a post on an issue attempting to install the DELL EqualLogic Multipathing Extension Module using VMware Update Manager.  I discussed an alternative method to VUM using the CLI to install the MEM.  The post has turned out to be fairly popular.  I’m guessing though that most people are more interested in how to install the EqualLogic MEM using VUM rather than my original workaround.  So I thought I would run through the steps using a version of MEM that now works.  The whole process of importing, attaching, and remediating came out a little longer than expected but I managed to capture all the steps in what I think is fairly easy to follow.

The version of MEM I am using is 1.1.2 (released Dec 2012).  You can obtain it from the EqualLogic support site (sign-in required).  The release notes state that the only change from version 1.1.1 is that it’s now compatible with Update Manager 5.1.  EqualLogic also state that if version 1.1.1 is installed 1.1.2 is not required.  At least this now explains why I had trouble with VUM and version 1.1.1

Using the vSphere Client under Solutions and Applications select Update Manager and click on the Patch Repository tab.

mem01a

Click on Import Patches.  Browse to the location of the patch.  Select the version you want.  In my case for ESX5

*Note: The ZIP file from the EqualLogic support site needs to be extracted prior to importing.  Once extracted there will be two zip versions.  An ESX4 and an ESX5 version.

mem01b

If the Upload is successful you’ll then be asked to confirm the Import.

mem01c

Once imported scroll to the bottom of the repository list and you should see the new Host Extension.

mem01d

With the extension imported into Update Manager we now create a new Baseline.  Click on the Baselines and Groups tab.

mem02

Click on Create to create a  new baseline.  Assign a name to the baseline and a description.  For the Baseline Type select Host Extension and click Next.

mem02a

Scroll to the bottom of the list and select the recently imported MEM patch.  Click the down arrow to Add the Extension and click Next.

mem02b

Confirm that the correct extension was selected and click Finish to create the baseline.

mem02c

With the patch imported and a new Baseline created for the Extension we now have to Attach the baseline.  This can be done at the top of the vCenter level or right down to the Host level.  In this case I just want to do a single host.  So I’m going to select the host and then select the Update Manager tab.  I’m then going to click Attach.

mem03a

Select the newly created baseline and click Attach.

mem03b

The baseline will now appear with a Question Mark beside it until a new scan is performed.  Click Scan, make sure Patches and Extension are selected and click Scan again.

mem03c

Once the scan is complete the Extension will now show up with a red cross signifying that it’s missing and needs to be Remediated.

mem03d

Click the Remediate button to start the process.  Select Extension Baselines on the left and the recently created Baseline on the right.  Then click Next.

mem03e

Omitted is a number of steps from the Remediate Wizard.  The options revolve around how the host and cluster will behave in Maintenance Mode.  The options are fairly straight-forward and the default options usually suffice. The last screen will summarise the options selected.  Make note what options have been selected and that the correct Baseline is selected.  Click Finish to start the Remediation.

mem03f

The host will now enter Maintenance Mode using the options you selected above.  Once complete we can select a datastore and select pathing where we can see a new pathing option and it’s selected by default. We will also see that all paths to the LUN are Active.

mem04

The whole importing and creating a baseline can seem a little tedious at first, but once done, all that’s needed is a scan and remediate on new hosts.

References

Link to original article EqualLogic MultiPathing Extension Module -- Alternative Install

Download the latest Extension module from EqualLogic Support Site

Downgrade VMware Hardware version -the unsupported way!

I tend to run a pretty tight shop with my VMware infrastructure. I always make sure that everything is up to date and running the latest versions. That usually includes the VM Hardware versions. Until recently this never posed an issue. I’m currently in the progress of migrating a vSphere 5.1 environment to a vCloud Director 1.5 environment (with an ESXi 4.x back end).  The maximum H/W version supported is 7 while my environment is version 8 / 9.

The OVF upload process in vCD 1.5 does a few sanity checks before starting an upload.  If it finds your H/W versin isn’t supported it won’t let you proceed.

vcd_hw01

VMware have three supported ways of downgrading.

1. Revert to a previous snapshot while on H/W version 7.

2. Use VMware Converter and perform a V2V to H/W version 7.

3. Create a new VM on H/W version 7 and attach the disks from the original VM.

All very safe ways but there is a fourth, and totally unsupported, quicker way to downgrade by hacking the VMX config file.

The process is relatively simple.   Inside the VMX file for the VM there are a few critical lines that tell vSphere / ESXi what hardware version the VM is.  By editing this file we can make it think it’s a Version 7 VM rather than 8.  In essence fooling it into think it’s an older version and basically making it just skip the new config values.

To start  Power Down the VM.  Then obtain shell access to the ESXi server that has visibility to the VM’s Datastore.  Then use VI to edit the VMX file of the VM.  If you haven’t used VI it’s pretty straight forward.  I to Insert, X to Delete (you’ll figure it out).

Below is the first 19 lines of the VMX file of my VM running on hardware version 8 on a ESXi 5.1 host.  The critical line to change is the virtualHW.version = “8”.  By changing the 8 to a 7 that’s technically all that’s needed.  I’ve taken it one step further and replaced all the 8’s with 7’s with the exception of the first line.

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "8"
pciBridge0.present = "true"
pciBridge4.present = "true"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "true"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "true"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "true"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "true"
nvram = "MYSERVERNAME.nvram"
virtualHW.productCompatibility = "hosted"

My updated VMX config below.

.encoding = "UTF-8"
config.version = "7"
virtualHW.version = "7"
pciBridge0.present = "true"
pciBridge4.present = "true"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "7"
pciBridge5.present = "true"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "7"
pciBridge6.present = "true"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "7"
pciBridge7.present = "true"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "7"
vmci0.present = "true"
nvram = "MYSERVERNAME.nvram"
virtualHW.productCompatibility = "hosted"

Now exit VI and save the changes.  Back in vSphere select the VM and select the option “Remove from  Inventory” (don’t Delete from Disk).  Then open the Datastore that has the VM and select the VMX file of the VM and add back into Inventory.  Basically the same way you normally add a VM into the Inventory from a datastore.  If all has gone well when the VM is added back in inside vSphere it will now say VM Version: 7.

You can now happily migrate the VM between ESXi 5 and 4.  In my case create an OVF of the VM and upload it into vCD 1.5. I’ve performed this quite a lot of times without issue. Probably not recommended for production but in a TEST / DEV environment less of an issue.

Appendix
Supported Vmware KB article to Downgrade
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1028019

The Hung VM on reboot

I recently had a crashed VM running on ESXi4 that hung during reboot on the black VMware boot screen.

After many frustrating hours I realised that it was due to one of its many RDMs.  Initially the only way to get the VM to successfully reboot was to remove all the RDM disks.  I then proceeded to add the RDMs back in one by one with successive reboots to make sure there where no issues.

I identified the faulty RDM when I went to add it back in.  I received the following error message.

File [datastorename] foldername/filename.vmdk is larger than the maximum size supported by datastore

A few online docs pointed to a few areas to investigate but none seemed to be my issue.  The datastore was formatted on VMFS3 with an 8 MB block size.  More than sufficient to handle a 10 GB RDM.

I checked the vmware.log file of the VM.  There was one error that repeated itself multiple times.

A one-of constraint has been violated (-19)

I tried adding the disk in as both a Virtual and Physical RDM without success.  I tried added it to a different VM without success.  As a final resort i mounted the volume from the SAN to a physical Windows server.  While I was expecting a corrupt volume I was surprised to find it in a clean state with accessible data.

As much as i would have like to keep troubleshooting the issue time was a factor.  I created a new 10 GB volume.  Mounted this new volume to the physical server.  Then performed a simple file copy from the original volume at issue to the newly created volume.

I un-presented the volumes to the physical server.  Then back in ESX, went through the process of adding the newly created volume as an RDM and ignoring the original faulty volume.

In conclusion, it’s time to get the hell away from these RDMs.  There’s just little justifiable reason to keep using them nowadays.

Worse yet, as I’m told, I may have resolved the issue but haven’t found the root cause.

 

Appendix

VMware KB article to RDM error

 

Disable Time Synchronization of ESX guest

One thing that I hate is time syncing of VMs.  It’s caused me grief for years.  For the most part, true, it’s not an issue.  Though one issue is too many for me nowadays.

When you install VMware Tools into a guest VM there is an option we are all familiar with, “Time synchronization between the virtual machine and the ESX Server”.  Even with this setting unchecked what you probably don’t realise is that there are “special” events that will still initiate a time sync between the guest and host.

Time will resync with a vMotion, snapshotting, disk shrink, or a restart of the tools.  I can understand the logic of why they are doing this.  All these events are touching the guest in some way and you want logs and timestamps in sync, but it’s still not cool.

I had seen this behavior in the past and though it was more a bug rather than a design feature.  There are certainly situations where you might have time sensitive VMs that rely on external time sources.  Telephony is an example.  There are also other side effects like time jumping over scheduled tasks.

Turns out all these special events can be individually disabled by entering them into the Configuration Parameters section of vCenter for the VM or editing the VMX file.

Configuration Parameters can only be edited while the VM is off.  The settings can be accessed by access the Edit Settings of a VM.  Clicking on VM Options and then Advanced.

The VMX file can be edited anytime but you will need to do it on the ESX server that is hosting the guest VM or you will get a Resource Busy error message.

All VMs will have the below line in their config.  0=Disable, 1=Enable.

tools.syncTime = “0”

The other parameters to disable “special” event time syncing are below.

time.synchronize.continue = “0”

time.synchronize.restore = “0”

time.synchronize.resume.disk = “0”

time.synchronize.shrink = “0”

time.synchronize.tools.startup = “0”

time.synchronize.tools.enable = “0”

time.synchronize.resume.host = “0”

.

Appendix:

Vmware KB article to disabling time synchronization

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1189

Running nested ESXi –the real inception

Nested ESXi servers are essentially virtualised ESXi servers.  They can be quiet fun to play with.  Usually the only reason you’d do this is for a test lab.  You don’t need a physical ESXi server to nest virtual ESXi hypervisors.  You can choose your product of choice.

In my case I’m running VMware Workstation 9 and virtualised ESXi 5.1 as the second level and then running virtual guests as a third level within the virtual ESXi.  In Workstation 9 you have the ability to connect to ESXi hosts and view guest VMs --as below.

You don’t need to do anything special to virtualize ESXi.  In fact there’s an option to select ESX as the guest OS in VMware Workstation 9.

When you turn on a virtual ESXi guest (or is it a host???) a message appears in the bottom right corner of the console.  It’s pretty self-explanatory.  You can create a 32-bit guest VM only on this ESXi server.

The cool cats that VMware are, allow you to run 64-bit VM guests too with a small config change.  The feature is only available in ESXi 5 and above.

On the virtualised ESXi guest enable console access (or SSH) and login as root.  The file ‘config’ in /etc/vmware needs to be modified and an additional line add, ‘vhv.allow = “TRUE”’.

If you’re a VI fan go ahead and use that.  If you’re lazy like me you can just append it to the file with echo.


echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Once the change is made no reboot is necassary.  You will now be able to run 64-bit guest VMs within your already virtualised ESXi host.  In an extreme example I ran another ESXi guest at the third level to prove a 64-bit OS would run.  On a second virtualised ESXi guest I’m just running a 32-bit Linux VM at that same third level.

For my test lab three levels is as deep as I need to go.  I’ve read that it’s also as deep as you can go as guest VMs will fail to run at a fourth level.  I don’t have the resources to test this out in my lab.  I heard that there was a VMworld presentation on multiple level nesting ESXi servers.  Would certainly be worth finding.

 

Appendix

As with many things VMware, nested ESXi is not officially supported.
VMware Link:  Support for running ESX/ESXi as a nested virtualization solution

I had trouble with the vmxnet3 network drivers with the nested ESXi.  Instead I chose to go with E1000 particular for that third level.

EqualLogic MultiPathing Extension Module – Alternative Install

UPDATE: If you’re looking for step-by-step instructions to install the EqualLogic Multipathing Extension Module click HERE to go to my most recent post on the topic.  

UPDATE: Note that the ESXCLI command below  —depot has a double dash.

I recently ran into an issue where vSphere 5.1 Update Manager scans and detects the latest Dell EqualLogic Multipathing Extension Module 1.1.1 as Not Applicable for ESXi 5.1 and will not select the patch during a Remediate.  Below I show how to install this extension via the ESXCLI.

If you’re running a DELL EqualLogic SAN with ESX you should be running DELL’s Multipathing Extension Module (MEM).  MEM is a Path Selection Plugin (PSP) driver for ESX.  In fact no matter what SAN you have you should investigate if they have a PSP driver for ESX.

ESX have three built-in pathing options, (Most Recently Used, Round Robin, and Fixed).  By installing EqualLogic’s MEM you get a fourth option called DELL_PSP_EQL_ROUTED.  EqualLogic PS Series SANs can run Active / Active pathing.  By installing the MEM, ESX can be made aware of this and can load balance appropriately.  This can all lead to increase in bandwidth utilisation and lower network latency.

Since ESX4 I’ve been installing the EqualLogic MEM using VMware Update Manager through the vCenter C# client.  I’ve never had any issues right up until and including ESXi 5 Update1.  The process is quite simple.

  1. Import the MEM as a patch.
  2. Create a new Baseline with a Host Extension and add the MEM extension that was imported. (EqualLogic have different versions for each ESX version so be mindful).
  3. Attach the baseline to the Host
  4. Perform a Scan and Remediate the ESX host.
  5. Reboot.

As mentioned in the beginning, Update Manager 5.1 saw the patch at Not Applicable.  I took this as an opportunity to try and install the patch through the ESXCLI.  To do this I used VMA.  If you’re not running or haven’t tried the VMware Management Assistant (VMA) it’s worth looking into.  It’s a nice convenient way to get CLI access to all your ESX hosts.

To install a PSP driver the host needs to be in Maintenance Mode.  So do this first or you’ll get a similar error to below.

Next transfer the EqualLogic Multipathing files to a location on the ESX host you want to install to.  In my case below I installed them to a folder on datastore1.

Back on the VMA use the ESXCLI and enter the following command, substituting the file location for your own, to install.

Esxcli —server=my_esx_host.domain software vib install —depot /vmfs/volumes/datastore1/EqualLogic-ESX-Multipathing-Module/dell-eql-mem-esx5-1.1.1.270268.zip

This is a kernel driver and a reboot is required for the PSP driver to successfully apply.  Once a reboot is performed the new PSP becomes the default selection.

Now selecting a datastore and selecting pathing you can see a new pathing option and it’s selected by default.  You will also see that all paths to the LUN are Active.

Appendix

Most recent step-by-step MEM installation article EqualLogic Multipathing Extension Module – Installing

Login failed, reason Success

Hmmm, you suck!

I’m still learning the ins and outs of vSphere Management Assistant (vMA).  From what I have seen so far I’m liking it.  vMA is a Vmware appliance that’s been around since vSphere 4.  Since then it’s gone through a couple revisions.

I originally installed vMA 5 while looking for a way to consolidate all my ESX host log files to an easy to access repository.  Researching vMA I came across vilogger.  I thought this was great and installed the appliance.  When I went to configure vilogger in vMA I found that the command was no longer available, vilogger had been deprecated in vMA 5.  Turns out that Syslog is the new ‘in’ way of logging.

All was not lost.  It turned out to be a nice convenient way to manage all my ESXi hosts.  It contains the vSphere command-line interface and the SDK for Perl.  It can store your credentials for all your hosts making it easy to script with.

The above screenshot I’m trying to run Remote ESXTOP on the vMA.  I know VMware has some of the sorriest error messages but this was too good not to post up.  Turns out it’s better to run Remote ESXTOP specifing the FQDN of the host and not just the hostname.  The actual root cause was because the vMA was using DHCP and had a different domain name then the ESX host.

Appendix

vMA Documentation and latest versions

vSphere Management Assistant Documentation (VMware Link)

VM Windows Cluster Volumes Offline in ESX

Windows Clustering on physical hardware is a pain at the best of times.  Just getting it to work can sometimes be a little try and effort… with a whole lot of luck.  Getting clustering to work in VMware is just cruel.

So when tasked to create a VM of a physical Windows Cluster for a test environment, boy was I excited! {Sarcasm sign}.

Actually creating the VM within ESX wasn’t that difficult.  Using Converter I created a VM of the OS.  Then using our DELL EqualLogic SAN I made clone copies of the cluster volumes.  I presented those volumes with the newly created VM as RDMs.  The process seemed to work really well until.  The OS booted up.  I could see all my presented volumes.  Issues began when I tried to start the Clustering Service and take it out of manual mode.  Out of the 6 volumes I had only one would ever become Online while all the others would (after some time) fail.

I spent days working through the issue (I’m pretty sure this is why I’m balding).  Articles seemed to lead me to DISKPART and trying to change the SAN Online Policy, manually online the disk, changing the READONLY attribute.  None of these seemed to work.  I’m assuming because there was an attribute that said the disk was Clustered and would prevent me making any changes.  Still, I thought I was on the wrong ‘path’ and began looking into a lower level issue at the ESX level.

The crux of my issue turned out to be a iSCSI multipathing problem.  DELL EqualLogic SANs run in an Active / Active pathing method where I/O is sent over all paths.  DELL has a third party Storage API plugin for ESXi that change the default behaviour of how mutlipathing works.  This is normally a good thing but for Windows Clustering in ESX… this is bad.

The solution is fairly simple to resolve.  The steps below is a rough outline of how to identify and change the multipathing policy.

Using vSphere vCenter, the changes are made within the Storage Adaptor.  In this case it’s the iSCSI Software Adaptor under the Configuration tab.

In the bottom pane select the paths view.  Expand the Target column and identify one of the cluster volumes with issues.  In this example I have a Dead path due to a recently removed SAN volume which is safe to ignore.  The one below is of interest as it’s one of the clustered volumes.  Remember the Runtime Name in the left column.

Change to the Devices view and locate the Runtime Name.  Right click on this device and select Manage Paths.  In this example DELL_PSP_EQL_ROUTED was selected as default.  Changing this to Most Recently Used (VMware) sends I/O only ever down one path.  The change is immediate.  As my volumes are offline I can safely make the changes.  On a working production volume I wouldn’t be making path selection changes during business hours.

Back over on the Windows Cluster VM I can now restarted the Clustering Service and have it correctly Online all the volumes.

MSCS is quite in depth and not for the faint hearted or something configured before you end home for the night.  Virtualising MSCS requires additional planning and thought in addition to regular planning.

Appendix

VMware -- Setup for Failover Clustering and Microsoft Cluster Service