Monthly Archives: April 2013

vCenter Server 5.1 Inventory Service

Posted by on April 28, 2013 No comments

Way back in October 2012 when I was upgrading a vCenter Server to 5.1 I wrote up a few Installation guides for the various services of vCenter.  In one of the guides, Installing vSphere 5.1 -- vCenter Inventory Server.  I joked about how I had little or no idea what the inventory service is.

Recently I was listening to a podcast with Justin King (@vCenterGuy) doing a Tech Deep Dive on vCenter Server 5.1.  In it he briefly discusses what the Inventory Service is.  He did an excellent job of filling in a lot of the missing gaps I had about what the Inventory Service does.

Below is the summary from Justin on the Inventory Service.

  • Used since vCenter 5.0.
  • Used by web clients only.
  • Cache for vCenter inventory
  • Used to offload read-only requests from the VPDX
  • Allows for better performance, more connections, to vCenter.
  • Enables use of Tags (only in web client).  Can lead to better organisational structure.
  • Inventory Service uses an XML xDB file to store data.

I can’t believe it’s taken this long to get some of these answers on the Inventory Service, but I’m glad I finally have them.  Full credit to Justin and his Tech Deep Dive on vCenter Server 5.1, well worth a watch.

Reference

#vBrownBag Follow-up vCenter 5.1 Technical Deep Dive with Justin King (@vCenterGuy)

EqualLogic HTTP vs HTTPS vs Encrypt Communication

Posted by on April 21, 2013 No comments

This week I realised that I had the option to log into an EqualLogic Web Management Portal with either HTTP or HTTPS.  It got me thinking what effect that has on the Encrypt Communication checkbox during login.

encrypt00

EqualLogic login prompt running Firmware 5.x

Under default configuration of an Equallogic array you have the option to use/select Encrypt Communication during login.  This can be changed and you can force the use of this option.

encrypt06

Under Group configuration select the Administration tab.  You will see that Web access is enabled and under GUI access that the checkbox for Allow only secure communication is unticked.  Ticking this box will force the use of Encrypt Communication during login.   You will then notice that Web access will change to Secure Only.

encrypt07

The above screenshot is running on Firmware 6.x.  On Firmware 5.x the checkbox is called Allow only secure SSL communication.  Oddly enough once enabled on either firmwares this won’t prevent the use of HTTP access to the Web Management Portal.

Now when attempting to login you will have to use Encrypt Communication.  Under Equallogic PS Series 5.x Firmware you have to select the checkbox.  If you don’t you will receive an error when attempting to login.

encrypt03

Under PS Series Firmware 6.x the checkbox will be selected by default and greyed out.  So you won’t get the above message.

encrypt04

As mentioned above, HTTP web access is still possible along with HTTPS.  So what’s going on here?!?!  Hence the reason for this post…

So I fired up Wireshark to watch  communication between my PC and the EqualLogic Array.  I first tried accessing the Web Management Portal with HTTPS and logging in  using the Encrypted Communication checkbox.  I then tried again but this time using Encrypt Communication.  No surprise here, both times all traffic was encrypted right from sign-in.

Next I accessed the Web Management Portal using HTTP,  not using Encrypt Communication, and signed in.  Looking through the Wireshark logs I could see my username and password in plaintext (certainly not recommended).  Again using HTTP to access the Portal I enabled Encrypted Communication and signed in.  This time looking through the Wireshark logs I could see my sign-in details were encrypted and all subsequent information as well.

From what I can see going on here is that the EqualLogic Web Management Portal is a Java Applet.  When loaded a connection is established over port 3002 on Firmware 5.x and Port 3003 on Firmware 6.x.  When Encrypt Communication is selected during sign-in, SSL encryption is handled by the Java Applet.  When not selected during sign-in SSL encryption is determined by whether you use HTTP or HTTPS and relies on the browser securing communication.

So if using HTTPS to access the Management Portal you’ve relatively sure your communication is secure but you can’t guarantee other admins are doing the same.  The safest thing to do is always enable the checkbox in the Administration tab Allow only secure communication.  By enabling this option you can be sure that whether administrators use HTTP or HTTPS all communication to the EqualLogic Array will be secure.

EqualLogic predictive disk failure

Posted by on April 17, 2013 No comments

For users of EqualLogic PS Series SANs.  If you have recently upgraded member firmware to version 6.x you might have noticed a change in behaviour for failed disks.

Prior to firmware V6.x there was no advanced warning for a disk failure.  Once you had a failed disk your array would move into a rebuilding phase with your hot spare disks.  You would then receive a collection of alarms that you have a failed disk, fewer spare drives, and array is being rebuilt.

New in firmware version 6.x is an early disk failure detection algorithm.  This is a great improvement over previous firmwares.   The EqualLogic member will communicate with the HDD firmware and when it detects a pending failure the EqualLogic member will initiate a block level copy from the source disk to the hot spare disk.  When the block level copy is complete the hot spare becomes the active disk and the source disk becomes failed.  The benefit in this approach is that no RAID rebuild phase needs to happen, which can take a significant amount of time with lots of disks.  A RAID rebuild can also cause a performance impact to the SAN which is mitigated with the predictive failure copy.

The only downside I’ve seen of this is the potential for a higher number of disk failures.  In some of the early PS Series SANs certainly types of types of disks, namely SATA, had very aggressive disk failure algorithms built into the HDD firmware.  It meant that there was a higher number, than normal, of false positive disk failures.  This was later corrected with a HDD firmware update that was recommended to all customers, which can still be found of the EqualLogic support site.  While I haven’t seen as many disk failures of years past I have noticed a slight increase in failed disks running Firmware version 6.x since the HDD firmware upgrades.

Predictive disk failure starting a copy to a Hot-Spare disk.

equallogic_repair02

Completion of copy results in a failed source disk.

failed_disk_equallogic

Why I don’t virtualize vCenter

Posted by on April 14, 2013 No comments

because I can…

Traditionally, historically, we use to keep our vCenter physical.  Nowadays it’s fairly well accepted, and there’s even VMware best practices, to virtualize vCenter, which is the direction I see most taking.  I almost feel dirty telling people I still choose not to virtualize vCenter.  I can feel my street cred dropping.  I feel that I’m losing a battle here and my times of running vCenter on separate physical hardware are coming to an end.

I choose to run my current vCenter on physical servers.  It has local RAID disks separate from a SAN.  A local SQL Server not reliant on a shared SQL server.  It also runs a local DNS accepting zone transfers from an internal DNS.  Physical servers generally tend to be way gruntier than a vCenter will usually need too.

This gives the vCenter real high independence from the rest of the infrastructure.  During infrastructure maintenance I can power down an entire ESXi datacenter farms and still have vCenter connectivity.  I can performance SAN maintenance with complete independence from vCenter storage and without fear that a datastore that could have been hosting a vCenter VM might be effected.

I’ve had a lot of great success using this method for vCenter.  Infrastructure maintenance windows seems to go much more smoothly when I have full vCenter connectivity.  The physical server actually turns out to be a good jump box during these periods as well.  I’ve even had unplanned outages that have been recovered from much quicker had I not had an independent vCenter.

In normal operation running vCenter virtualized isn’t an issue.  It just works, right!  There’s little tricks you do, of course, to make life easier for yourself though.  You usually disable DRS on the vCenter VM to make it easier to identify which ESXi host it’s running on.  You put a High restart priority on the VM itself to cater for HA situations.

In my eyes though, that’s just thinking small events.  We’re not foreseeing how we’ll manage the VMware infrastructure during large scale failures.  We’re running vCenter on the same SAN storage as our production VMs.  A SAN failure of some sort can instantly affect our vCenter and ability to manage our Vmware environment.  I’m sure many of us install the vCenter DB onto a shared SQL server too.  What happens when that SQL server goes through maintenance, has an outage, or is just under high load?  vCenter becomes effected immediately.

Perhaps I’m placing way too much importance on vCenter?  Maybe I’m just stubborn and old school?  I really don’t know.  The latest version of the vSphere 5.1 virtual appliance is really good.  So good that I’m considering moving away from the Windows version permanently.  I have a feeling this is going to start leading me down the virtualized vCenter path forever.

Using PowerCLI to speed up Question time

Posted by on April 7, 2013 No comments

It seems like all my posts start of the same way, “I recently had an issue…”, but it’s true 🙂   I recently had an issue with snapshots consuming all of the space in a datastore.  The culprit was CommVault preforming end of month full backups of VMs.  During the backup process it takes a snapshot of the VM prior to that full VMDK backup.

It was first identified with a VM having a pending question in the vCenter Client.

get-vmquestion01

It was also quickly identified as having a snapshot by the xxx-000001.vmdk virtual disk name.  A check of the working snapshots of the VM showed that it was created by CommVault.  A check of the VMs datastores the VM used showed that one of them was full.  It was also a good bet that any other VMs on this datastore with snapshots or thin provisioned disks would be having issues too.

The snapshots couldn’t be deleted as there was no free space on the datastore to commit the transactions from the snapshot VMDK.  I could have grown the size of the datastore but that would have involved Change Requests with changes at the SAN and vSphere level.  Without panicking too much I identified a rather small working VM and migrated it’s storage to another datastore.  The goal here is to free up space on the datastore in preperation to answer the pending questions and get the VM running ASAP.

While the VM was being migrated to a different datastore it gave me a few minutes to identify which VMs had a pending question and would have been in a paused state.  Now there’s probably 17 different ways to approach this but I just went with a direct method.

Using PowerCLI and connecting up the vCenter Server.  I then ran Get-VMQuestion.

Connect-VIServer my_vcenter_server.domain.local

Get-VMQuestion

This will by default return all VMs in the vCenter instance that have pending questions.  If you have a large vCenter with 1000+ VMs this will most likely take a while and you might want to target the command to specific datastores or ESXi servers.  In my situation I wanted to make sure I identified all pending questions and didn’t miss anything.

get-vmquestion02

By this point my VM migration just completed so I could now answer all the pending questions and resume the VMs in the paused state.  This can be done by using the Set-VMQuestion.

Get-VMQuestion | Set-VMQuestion –Option “Retry” -Confirm:$false

It doesn’t really get simpler then the above command.  Identify pending questions with Get-VMQuestion.  Pipe it to Set-VMQuestion, answer all with ‘Retry’, and use the parameter ‘Confirm’ so not to get prompted to confirm action.  Again probably smarter ways to go about this.  You could use the Get-Datastore cmdlet to identify datastores with zero free space.  Then target those datastores with Get-VMQuestion.

Get-Datastore my_datastore | Get-VM | Get-VMQuestion

The Get-VMQuestion / Set-VMQuestion is a great PowerCLI way of working smarter not harder.  Whether answering 1 or 100 questions it’s all the same via PowerCLI.  I don’t really know if there’s an easy way to identify and answer pending questions through the vCenter Web or C# client???

On a side note.  Set-VMQuestion isn’t overly intuitive as a cmdlet name for answering a question.  So there is an alias to it called Answer-VMQuestion.  I guess it sticks with PowerShell tradition with the Get / Set verbs.

Connecting to vCloud Director via PowerCLI

Posted by on April 1, 2013 No comments

I’m currently stuck administrating a vCloud Directory 1.5 environment.  I don’t have any major concerns with vCD 1.5 but sometimes I do find the web portal a little awkward to navigate around.  VMware have done a great job in creating PowerCLI cmdlets that open up access into the vCD APIs.

You can obtain access to the cmdlets via two methods.  You can download PowerCLI from VMware.  You’ll need at least version 5.0.1.  Or you can download PowerCLI for Tenants.  This version contains only the vCD cmdlets and removes all the other vSphere cmdlets not relevant to vCD.

If you’re connecting to vCD over the internet the great thing is you don’t need any extra or special ports opened to use PowerCLI.  Connection is done over HTTPS (Port 443) using the domain name of your Cloud Service Provider’s vCD portal.

You’ll also need your ORG name within vCD.  To find out your ORG name connect up to the vCD web portal.  Navigate to the Administration tab and select General under Settings in the left pane.

vcd_connect01

Open up PowerCLI.  Use the cmdlet Connect-CIServer to initiate a connection.

Connect-CIServer -Server portal.vpdc.domain.com -org MYORG

You should then be prompted for your vCD login credentials.

vcd_connect02

Once connect you you can start playing around with some of the more basic get commands.

To view your usage of CPU, Memory and Storage allocation you can use get-orgvdc.

get-orgvdc | ft -autosize

vcd_connect03

To list all your VMs it’s just a matter of get-civm

get-civm | ft  name, status, cpucount, memorygb -autosize

vcd_connect04

To list details of a specific VM you can try the follow

get-civm -name MYSERVER | fl

vcd_connect05

The cmdlets won’t give you the full feature set as the web portal.  Never the less you’ll find that you can speed up a lot of the daily administrative tasks you do.  It’s also a great way of extracting out information for reporting.

References

Vmware Connect-CIServer 

vCloud Director PowerCLI Community

vCD cmdlet reference