Category Archives: vCenter

Could not establish trust relationship for the SSL/TLS Secure Channel – Invoke-WebRequest

I’ve recently been playing around with VMware’s REST APIs in VCSA 6.5 using PowerShell. I’ve been using a lot of Invoke-WebRequest and Invoke-RestMethod to do my work. Chris Wahl has a great primer on how to get started here.

One issue that I ran into very quickly working again my VCSA was a certificate trust relationship error. I’ve run into this error numerous times in the past.

PS F:\Code> Invoke-WebRequest -Uri https://10.0.0.201/rest/com/vmware/cis/session -Method Post -Headers $head
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At line:1 char:1
+ Invoke-WebRequest -Uri https://10.0.0.201/rest/com/vmware/cis/session ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

The first time I ran into this error I was stumped for while finding a solution. Ultimately it comes down to using Self-Signed Certificates in vCenter, as most of us do.  In general using Invoke-WebRequest or Invoke-RestMethod against a server using a Self-Signed Certificate will cause this error, it’s not just related to vCenter.

The solution is quite simple.  I found a snippet of code some time back that I keep on hand in this situation.  It basically ignores certificate validate in PowerShell allowing you to make a connection with Invoke-WebRequest.  All you have to do it paste this code into your PowerShell session before you run Invoke-WebRequest against a server with a Self-Signed Certificate.

if (-not ([System.Management.Automation.PSTypeName]'ServerCertificateValidationCallback').Type)
{
$certCallback = @"
    using System;
    using System.Net;
    using System.Net.Security;
    using System.Security.Cryptography.X509Certificates;
    public class ServerCertificateValidationCallback
    {
        public static void Ignore()
        {
            if(ServicePointManager.ServerCertificateValidationCallback ==null)
            {
                ServicePointManager.ServerCertificateValidationCallback += 
                    delegate
                    (
                        Object obj, 
                        X509Certificate certificate, 
                        X509Chain chain, 
                        SslPolicyErrors errors
                    )
                    {
                        return true;
                    };
            }
        }
    }
"@
    Add-Type $certCallback
 }
[ServerCertificateValidationCallback]::Ignore()

Once you run the code you will be able to now successfully make a connection.

I’ve seen some simple one liner solutions for Self-Signed Certificates but none of them seemed to work for me.  Whereas the above snippet of code has always worked.  Obviously bypassing certificate validate is not something you want to run on a global scale in PowerShell but this code works great for your current session only.

If there is a simpler way to bypass certificate validation I’d love to hear it.

Streaming Datasets – PowerShell | PowerCLI | Power BI

A large part of my day is spent scripting in PowerShell, specifically with PowerCLI.  One of the strongest areas of PowerCLI, obviously, is being able to retrieve information.  It’s one of the key use cases, in my opinion, for using PowerCLI in a VMware environment, it’s ability to retrieve information for Capacity planning and reporting.

Recently I’ve been looking at how to consume all that information.  You can obviously export it to a CSV, push it into a database, or something that I’ve been playing around with recently, stream it into Power BI.  Now if you haven’t tried it out yet, PowerBI is an analytics service from Microsoft.  At its core it’s a data warehouse for business intelligence.  But putting all those fancy words aside, I use it to create fancy reports.

Exporting information out of a vCenter environment with PowerCLI is dead simple.  I have dozens of scheduled tasks running all the time doing this.  Where I’ve fallen down, is taking that information and trending it over time.  This is where the Streaming Datasets functionality of Power BI comes in.  Using PowerCLI I can get an Object and Value from vCenter and then Post that directly into Power BI, using their API, and have it instantly graphed in a report.  I can then share that report out to anyone I want.  Power BI lets me do this over and over, almost as fast as I can pull the information out of vCenter.

In the example below I show how to create a trend report over time that displays Total and Available Storage of a vCenter Cluster.  Rather simple, I know, but can easily be adapted to show things like number of running VMs running, reserved resources used, etc, etc.  The skies the limit really.

Before we do any scripting the first thing we do is log into Power BI.  If you don’t have an account, don’t worry, the basic version is free.  Hit the Sign Up link and make sure you select Power BI and not Power BI Desktop for Windows, we want the cloud version.

Once logged in we click on Streaming Datasets in the bottom left under the Datasets category.  This is where we create our initial dataset schema so that it can accept streaming input.  We click on ‘Add streaminig dataset’ in the top right.

Then select the source of data, which will be API and click next.

We give our New Streaming Dataset a name and define a few values.  In this example we will define a Date, Total Storage, and Available Storage value, turn on Historic Data Analysis and click Create.  Make note of your data type to the right of the value.  Date is DateTime and the other two are Numbers.

We’ve now created our schema and are provided with a Push URL address and sample code in a few different formats (we want PowerShell).  If you look carefully we are using an Invoke-RestMethod to Post to Power BI.  This sample code has just provided us the template and hardest part of our PowerShell / PowerCLI script.  Click over the code and copy / pasta it out to use in our script (Paste it at the bottom of the script as it will be the last thing that runs).

Now we actually start on the PowerShell / PowerCLI script.  To keep it as concise as possible.  I’ve skip the process I use to actually connect to the vCenter and retrieve the information out using PowerCLI in the code below.  The real goal here is just to retrieve some values and get that into Power BI.  Line 6 is basically retrieving all shared VMFS datastores in Cluster1.  The important lines to note, though, are 4, 8, and 9 where I store my key values in three variables.  One for Date, one for TotalStorage, and one for AvailableStorage.

Import-Module VMware.VimAutomation.Core
Connect-VIServer -Server host.mydomain.local

$date = Get-Date

$datastore = Get-Cluster -Name Cluster1 | Get-Datastore | Where-Object {$_.Type -eq 'VMFS' -and $_.Extensiondata.Summary.MultipleHostAccess}

$TotalStorage = ($datastore | Measure-Object -Property CapacityMB -Sum).Sum / 1024
$AvailableStorage = ($datastore | Measure-Object -Property FreeSpaceMB -Sum).Sum / 1024 

The additional lines below from 11 onward is the important code.  This is our pasted sample code from Power BI that we will slightly modify to push our values up to Power BI.  Don’t copy mine, as your URL and key will be different.  On lines 13, 14, and 15 we will remove the example values and replace it with our three variables, $Date, $TotalStorage, and $AvailableStorage.

Import-Module VMware.VimAutomation.Core
Connect-VIServer -Server 10.1.1.201 -user "mydomain\username"

$date = Get-Date

$datastore = Get-Cluster -Name Cluster1 | Get-Datastore | Where-Object {$_.Type -eq 'VMFS' -and $_.Extensiondata.Summary.MultipleHostAccess}

$TotalStorage = ($datastore | Measure-Object -Property CapacityMB -Sum).Sum / 1024
$AvailableStorage = ($datastore | Measure-Object -Property FreeSpaceMB -Sum).Sum / 1024 

$endpoint = "https://api.powerbi.com/beta/83fe1fa2-fa52-4376-b7f0-cb645a5fcfced/datasets/d57970bc-60b3-46e6-b23b-d782431a72be/rows?key=2zEhgN9mu%2BEH%2FI2Cbk9hd2Kw4b5c84YaO6W8gzFcZbBnO6rti3N631Gjw%2FveNXSBxwR84VcWPGOSrheNwQnCbw%3D%3D"
$payload = @{
"Date" = $Date
"Total Storage" = $TotalStorage
"Available Storage" = $AvailableStorage
}
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json @($payload))

Disconnect-VIServer * -Confirm:$false

On the last line I disconnect  from my vCenter and close any sessions.  This helps if running as a scheduled task.  Finally save the script.

And that’s it for the scripting part.  Assuming everything is correct, no connection issues, correct values being retrieved.  All we have to do is run the script and it will send a POST request using Invoke-RestMethod with our three values.  We can now run this script as many times as we want and it will continue to post the current date and time along with Total Storage and Available Storage.  At this point, if we wish, we can turn the script into a scheduled task or just continue to run manually to suit our needs.

We now go back to Power BI and report on what we have.  Back on our Streaming Datasets browser window we click the Create Report icon under actions.  Now this part is going to be very subjective to the end user who wants the report.  But the key data we want is under RealTimeData on the far right.  Select all three values and we get presented with a default view of our data.  Under Visualizations select line chart and now we start to see a more visual representation of our capacity over time.  Under the Analytics section add a trend line and see a basic view of available capacity over time.  Finally hit save and you have a self updating report from streaming data.

For the report to start to look anything like below it will take time and a few sample datasets.  In the below image I’ve mocked up some numbers over time as an example.

Once you have a working script and it’s streaming data to PowerBI it’s really up to you on how to report on it.  The above example, as simple as it is, lays the ground work to more customized and complex reporting that you might not be able to get out of traditional monitoring and reporting software.  The ability is there to even share out the report.

Streaming datasets, as you might have noticed in the UR, is still in beta.  As great as I have found it to be it does have some quirks.  For one you can’t easily modify data you have already streamed up to Power BI.  So if you send incorrect data / values up to Power BI in a streaming dataset it will remain their.  At which point you will have to consider Filters to exclude it in reports.

In summary I think Power BI is a very underrated free tool from Microsoft.  I’ve only just started to scratch the surface of what’s possible with it.  The simplicity of capturing data with PowerShell and sending it to Power BI is well worth the time and effort to try at least once.  So what are you waiting for?

Modify HTML5 vSphere Client Idle Timeout

Before I go any further, just to make it clear, we’re talking about the new HTML5 client in vSphere 6.5 (GA Build 4602587).  Not the older Flash based vSphere Web Client in vCenter 5 and 6.  So lets call it the vSphere vCenter HTML5 UI Web Client.  Clear now?  Ok, just refer to the pic below.

Below are the steps I used on the vCenter Server Appliance.

Just like the old Web Client I know of no way to change the idle timeout from within the UI today.  So we have to revert to connecting to the console and making the changes through the shell.  We do this by opening up a console window to the VM or using SSH and login with root (remember to enable SSH first).

At the Command prompt of the VCSA type the following to enable Shell access.  You may received a Shell is disabled message.  If you do, enable with shell.set.

Command> shell
Shell is disabled.
Command> shell.set --enabled true
Command> shell
vc01:~ #

Now at the Shell type the following below and locate session.timeout.

cat /etc/vmware/vsphere-ui/webclient.properties

You should find something similar to session.timeout = 120 as this is the default value in minutes.

Make a backup copy of webclient.properties.

cp /etc/vmware/vsphere-ui/webclient.properties /etc/vmware/vsphere-ui/webclient.properties.bak

If you’re comfortable using an editor like VI go ahead and use that to increase or decrease the value in minutes.  Probably for the best, it doesn’t appear that you can set this value to never timeout.  I tried 0 and -1 and both caused the vSphere Client to timeout instantly on login.  The timeout value, though, can quickly and easily be modified using the sed command.

The sed command below locates the specific string session.timeout = 120 and replaces it with session.timeout = 720, which is 12 hours (or in other words my standard work day).  Change 720 to however many idle minutes you want.  If sed doesn’t find the specific string, don’t worry, it won’t modify anything.

sed -i “s/session.timeout = 120/session.timeout = 720/g” /etc/vmware/vsphere-ui/webclient.properties

Run the cat command again and check that the session.timeout value has changed.

cat /etc/vmware/vsphere-ui/webclient.properties

If the session.timeout value has been modified correctly we now have to stop and restart the vsphere-ui service by running the following commands below.  I covered stopping and starting all services on a VCSA in a previous post HERE.

service-control --stop vsphere-ui
service-control --start vsphere-ui

Wait a few minutes for the service to start up fully and open a new browser windows to the vSphere Client.  It should now be running with a new idle timeout.

 

vCenter In VR (Is This VCSA 7?)

The last few months have been extremely fun for me.  I purchased a HTC Vive and have been enjoying every minute with it.  I’m not a huge gamer but I absolutely love the immersion factor.  I’ve lost count of the times I have got lost in games like Onward for hours on end.  The realism and social aspect of coordinating with your team mates on how to take the objective.  The absolute fear of crouching behind a wall while the enemy next to you discusses where you are.  An experience that’s hard to convey.

Games aside though, VR also has the ability to mirror your desktop and applications too.  Nothing like Minority Report or that awesomely realistic movie Hackers.  But think more a VR room with a computer screen in front of you that you can enlarge or shrink to suit your view.

So that got me thinking.  There are a few different VR apps that let you mirror your desktop in VR.  I decided to try out Bigscreen, mainly because it’s free!  And hell, because this is a virtualization blog, I obviously had to try out the vSphere Client to see if I could practically manage my vCenter Homelab environment.

It took a few attempts to find the best viewing mode and way to manage vCenter with the vSphere Client.  I first tried the large projector view on the wall in the VR room.  This turned out to be an absolute joke.  Imagine the worst, lowest, quality projector, and then try reading small text from the other side of a room.  Then think of something worse.  Okay… it wasn’t that bad but still.


Failing to use vCenter in the large projector mode view

The best mode I found was literally just sitting down in a chair.  Switching to the floating screen mode and enlarging the screen to encompass my field of view.  Then placing a small curve to the screen to rap a little around me.


S
omething’s red in my vCenter environment

I first tried managing vCenter with the HTC Vive controllers.  The controllers basically act as laser pointers.  You can pull up a virtual keyboard and laser zap the keys with the controllers as well as move the laser point around on the screen like a mouse cursor.  Using projector mode this was okay but up close it was really awkward.  Ultimately using the physical mouse and keyboard was most practical.  And it was practical.  As long as you can position your hands in the right spot and touch type there was no issues.  You just have to adjust to what feels like a 100 inch screen in your face.

Bigscreen has what they call Mutliplayer rooms.  This is where you can join and create a new room where people can share you screen experience.  I did jump into some of these rooms where movies were playing and had a little chat to the other guests.  I wasn’t game enough to create a room and share my vCenter screen though.  I just felt that the VR community wouldn’t have the same appreciation for my vSphere Homelab environment 😛


J
umping into someones VR cinema room

You can imagine how this multi-user room experience could be interesting though.  Inviting a friend / service desk into your private VR room to help you out on an issue in your environment.  Actually being able to point on the screen and talk through resolving an issue.  Waving your hands in frustration when the service desk can’t fix your issue.  It reminds me of the book Ready Player One.  A dystopian future where lives are lived out in a VR world and virtual chat rooms.

So alright, all of this was a big gimmick.  An excuse to talk about my HTC Vive and somehow justify it on my virtualization blog with vCenter.  It was fun, though, I’m not holding my breath for vCenter 7 VR.  But maybe a fling 🙂

 

SCP to a vCenter Server Appliance (VCSA)

For some this may be a rare situation but from time to time I find that I’m needing to copy files to and from a vCenter Server Appliance (VCSA).  I had one of these situations recently on vCenter 6.  I needed to move some log files off a VCSA box.

I’ve found the easiest way to do this is via SCP -- Secure Copy, which uses the SSH protocol.  It’s a relatively simple process to enable the VCSA to accept SCP connections.  It’s a two step process which first requires enabling SSH on the VCSA and then switching the default Shell.

Step 1, involves enabling SSH  

I’ve written a previous post on how to enable SSH on a VCSA here.  Since that post VMware have re-released the VAMI on vCenter Server Appliance V6 U2.  So I thought I might show this new method to enable SSH.  Only if using VCSA 6 U2 or greater else use my previous post steps.

Connect to the VAMI URL of your vCenter on port 5480 using HTTPS.  In my case it was https://vc.ukoticland.local:5480/login.html

vami-000298

Login with your VCSA root account and password.  Then navigate to Access and click Edit on the far right.  Select Enable ssh login and to make life a little easier also Enable bash shell and click OK.  The timeout refers to how long the Bash shell will stay enabled.  The default is fine.

vami-000299

Step 2, changing the default shell

Even though we enabled the bash shell above the default shell is still the VMware appliance shell which prevents us from connecting to the VCSA via SCP.  So we need to SSH to the VCSA and change the default Shell from the Appliance Shell to Bash.

In my case I used Putty.  Logged in with my root account and type shell.

putty-000300

Now i can change the default shell for the root user to bash using the below command.

chsh -s /bin/bash root

putty-000301

We’re now ready to SCP to our VCSA with the ability to transfer files to and from the VCSA.  I use the simple Windows app, WinSCP.  I change the File Protocol to SCP.  I enter in my vCenter as my host and my root credentials.

winscp-000302

When you’re complete just reverse the changes you made.   In the SSH Putty session type the below to permanently switch the Bash shell back to the default Appliance Shell.  Then log back into the VAMI as above.  In Access deselect SSH and Bash.

chsh -s /bin/appliancesh root

References

Toggling the vCenter Server Appliance 6.x default shell (2100508)

Cannot validate host customizations for host ‘fqdn’. null

I’ve been doing a lot of work recently with Auto Deploy and Host Profiles in vCenter.  I feel both of these technologies are very underrated in vCenter.  With Host Profile, prior to using Auto Deploy, I always felt they were just a little more trouble then it was worth.  Trying to get every host compliant at the same time seemed like this loosing battle I was facing.  But Auto Deploy and Host Profiles go hand in hand so it was an opportunity to get it right.

While deploying Host Profiles to our Cloud environment I ran into a few interesting errors.  One of these were the below.

Cannot validate host customizations for host x.x.x.x. null

host_profile_null

The error was generated in the vSphere Web Client while trying to validate the Host Profile and apply.  This error was very non-descriptive and provided little in the was of help.  As much as I tried not to I defaulted back to the C# Client and tried to apply the Host Profile once again to the host.  This time I received a much more informative error message.

Host Profile execution failed: ‘Balanced’ CPU policy not supported by system

Now this was now much clearer.  What had happened was that I generated the Host Profile on a physical host with slightly different hardware.  I was able to quickly fix the issue by deselecting Power System in the Host Profiles configuration.

The interesting take-away here was not what caused the error but having to revert to the C# Client to get some meaningful information on the error.  So it’s worth trying both the vSphere Web Client and C# Client when facing similar errors.

References

Applying the Host Profile settings on an ESXi host using vSphere Auto Deploy fails with the error: Host Profile execution failed: ‘Balanced’ CPU policy not supported by system

ESXi Host Client Officially Released

A few days ago ESXi 6.0 Update 2 was released.  Quietly added in was version 1 of the ESXi Embedded Host Client.  I’ve spoken a few times about the Host Client.  It started out as a VMware Fling by VMware engineers Etienne Le Sueur and George Estebe.  Since then it has gained a hugely positive response from the community that it has finally found its way into ESXi.

If you’ve recently upgraded or installed ESXi 6.0 Update 2 you can access the host client via a browser connecting over standard SSL (https:/myesxi-host/ui/).   You can login with the host’s root account.  If you’ve never seen the Embedded Host Client before you’re in for a huge surprise.  You’ll be amazed at how similar it looks to the vSphere Web Client.  Not only that but it’s extremely snappy and fast built upon HTML5.

I recently upgraded my NUC home lab hosts to Update 2 to check out the production build.  It looks and feels just like the Tech Preview.  It’s going to be a great replacement to the C# Client.  If you’re running a previous Tech Preview release of the fling there’s a few things to note before you upgrade to Update 2.  Initially I did an upgrade of a host with an old Tech Preview 5 fling installed.  Update 2 left that version of the fling in place.  So on my subsequent hosts I removed the Tech Preview fling before upgrading the host.  That resolved the issue and installed the v1 production release.

Below are the steps to remove the Tech Preview fling before upgrading a host.  The -f represents a force removal just in case you have any third party vibs that may conflict with the uninstall as I did.

[[email protected]:~] esxcli software vib remove -f -n esx-ui
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_esx-ui_0.0.2-0.1.3357452
VIBs Skipped:
[[email protected]:~]

If, like me, you upgraded a host before removing the Tech Preview version of the fling. You can download the official Host Client from the VMware download portal.  List with the ESXi 6.0 U2 Zip and ISO images is the Host Client VIB and Offline Bundle.  Then just run through the steps to remove and install the VIB.

There is a newer build also available up on the flings page --Tech Preview v6.   I chose to upgrade to this build as it’s just my home lab.  The process is simple, I outlined the steps to update the Embedded Host Client to a new build in a previous post.

Latest v1 Production Build

host-client_v1

Latest Tech Preview Build

host_client_tp6

References

Embedded Host Client Fling Page

VMware Host Client Release Notes

Auditing and Alerting on vCenter Permissions – Part 2

In my previous post I discussed auditing permissions in vCenter via PowerCLI.  Going through the process of identifying account and group permissions and what objects they have been set on.  In this second of a two part post, I look at creating a vCenter Alarm in PowerCLI that will trigger when a permission is Added / Removed / or Modified which will then show up in vCenter Alarms.

Part 2 -- Generating a custom alarm on a vCenter permission change

The objective here is not to catch someone out (though you could see how it could help).  The Alarm is by no means meant to be as a deterrent or even used for auditing.  Where it’s useful is as an identifier that a permission add / remove / modification as taken place.  Once triggered a person can review the new permissions and acknowledge the alarm.

Working with vCenter Alarms in PowerCLI hasn’t reached the maturity as many other Cmdlets.  So creating new alarms with PowerCLI is not as straight forward as using a Cmdlet.  To create a new alarm ww have to use the powerful Get-View Cmdlet which exposes us to all the richness of the SDK APIs of vCenter.

Creating an alarm requires a few pieces of information.  First we create the alarm object, give it a name, description, and set its state.

Next we create our Triggers or expressions.  Creating the expression is where the magic lays.  The key piece is the EventTypeID.  This is the specific vCenter event that will trigger the alarm.  I found finding this information through VMware was extremely difficult.    With a little searching though, and playing around, I found the information I was after with Get-View eventManager.

$event = get-view eventManager
$eventman.description.eventinfo | Where-Object {$_.key -like "*permission*"}

Running the above without any filters returned a huge amount of information with possible events to trigger on.  Using the Where-Object Cmdlet I was able to filter the results to anything with the word ‘permission’.  This narrowed down the results to just four.  The key three being  PermissionAddEvent,  PermissionRemoveEvent, and PermissionUpdatedEvent.  Of course if you want to see everything just remove the pipe and everything after it.

With these events and a little more research I had enough information to now create a script for an alarm.  Below I created three expressions.  One for each Permission event and set it’s status to Yellow if triggered.  The alarm is created in the root of vCenter and will propagate down to all objects.

$alarmMgr = Get-View AlarmManager

# Create AlarmSpec object
$alarm = New-Object VMware.Vim.AlarmSpec
$alarm.Name = "Permission Modification"
$alarm.Description = "Track permission changes"
$alarm.Enabled = $TRUE

# Event expression 1 - Permission Added
$expression1 = New-Object VMware.Vim.EventAlarmExpression
$expression1.EventType = "EventEx"
$expression1.EventTypeId = "vim.event.PermissionAddedEvent"
$expression1.ObjectType = "VirtualMachine"
$expression1.status = "Yellow"

# Event expression 2 - Permission Removed
$expression2 = New-Object VMware.Vim.EventAlarmExpression
$expression2.EventType = "EventEx"
$expression2.EventTypeId = "vim.event.PermissionRemovedEvent"
$expression2.ObjectType = "VirtualMachine"
$expression2.status = "Yellow"

# Event expression 3 - Permission Modified
$expression3 = New-Object VMware.Vim.EventAlarmExpression
$expression3.EventType = "EventEx"
$expression3.EventTypeId = "vim.event.PermissionUpdatedEvent"
$expression3.ObjectType = "VirtualMachine"
$expression3.status = "Yellow"

# Add event expressions to alarm
$alarm.expression = New-Object VMware.Vim.OrAlarmExpression
$alarm.expression.expression += $expression1
$alarm.expression.expression += $expression2
$alarm.expression.expression += $expression3

# Create alarm in vCenter root
$alarmMgr.CreateAlarm("Folder-group-d1",$alarm)

# Add action to email alarm
#Get-AlarmDefinition $alarm.Name | New-AlarmAction -Email -Subject "vCenter Permission Modification Occurred" -To [email protected]
#Get-AlarmDefinition $alarm.Name | Get-AlarmAction | New-AlarmActionTrigger -StartStatus 'Green' -EndStatus 'Yellow'

Commented out is also the ability to email a notification out when an event occurs.  This just feels a little overkill but never the less is a notification option.

The beauty of this script is that it can also easily be modified to alarm on any trigger-able event in vCenter.  Whether you need one expression or multiple expressions it’s easily modifiable.

Virten.net has a good article on how to create custom alarms and find events that I used for a reference.  If you don’t want to search for your own events to alarm on.  Virten maintains a database of all events that can be alarmed on.  I’m not sure how current it is but it’s a good starting point.

References

How to create custom vCenter Alarms from Events | Virten.net

Vmware Data Object -- EventAlarmExpression

Auditing and Alerting on vCenter Permissions – Part 1

In this two part post I discuss and cover how to audit vCenter permission using PowerCLI.  How to enumerate basic permissions and then expand it out and send it to a CSV file for further auditing.  In Part 2, I’ll discuss how to create custom alerts using PowerCLI to trigger an event on a vCenter permission modification.

Part 1 -- Auditing vCenter Permissions

Reviewing and auditing vCenter permissions has always been awkward at the best of times.  Unless you’re on top of permissions from day one, over time you can be left with a mess of vCenter permissions.  If you’re not explicitly looking for it you won’t know it either.  This is certainly what I found recently in one of my vCenter environments.

There are a few different ways to view permissions in the C# and Web Client.  You can click on an object and view the Permissions tab or you can go to Roles.  The Roles section is certainly much more useful but still far from ideal.  You have to click on each Role and there is no way to export that information.  Using PowerCLI on the other hand you can consolidate the above two methods into one view and have it exportable.  The quickest and easier way to get started is with Get-VIPermission once connected to a vCenter in PowerCLI.

Get-VIPermission | Format-Table -AutoSize

Get-VIPermission in its default form gives you a quick overview of all Principals / Users defined with explicit permissions along with their Role, whether it’s a group, and if it’s set to propagate down the tree .  Ideally you’ll only have a few groups listed here and nothing else.  Unlike myself, where I had a large amount of individual users explicitly listed on many different objects.

When using Format-Table we loss a little information on the screen. Namely, we can’t see on what objects these permissions are set on.  So we can work with the Get-VIPermission cmdlet to expand it out a little further.

 Get-VIPermission | sort | Format-Table Entity, EntityID, Role, Principal -AutoSize

The two extra values we are looking for here are Entity and EntityID.  In most cases Entity will aid in identifying the vCenter object that has the explicit permissions set on it so you can review.  The Entity could be a Virtual Machine, a Cluster, or a Datastore.  In some cases, though, you will find the Entity is blank.

Again, in my case I had a lot of permissions with a blank Entity.  Using EntityID I could see that the objects appeared to be networking related.  So in this case we need to get a little more creative with the querying of the SDK APIs using Get-View.

Get-VIPermission | sort | ft @{N=”ObjectName”;E={(get-view -Id $_.EntityID).name}}, Entity, EntityID, Role, Principal -AutoSize

This now gets us 95% of the way there.  We use Get-View using the EntityID as the Get-View ID to find the actual name of the vCenter object for that missing Entity value.  In the case of the networking objects we will now get the exact names of the networking switches.

get-vipermission01

Putting it all together.  In this last part I turn the one liner into a little script, make sure the correct snap-in is loaded, make it a little easier to read, and have it output to a CSV file that can be used to audit and review at a later stage.

# Import Snap-in
if ((Get-PSSnapin | where {$_.Name -ilike "Vmware*Core"}).Name -ine "VMware.VimAutomation.Core")

{
Write-Host "Loading PS Snap-in: VMware VimAutomation Core"
Add-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue
}

$Date = (get-date).tostring("yyyyMMddHHmm")
$result = Get-VIPermission | sort-object -Property Principal | Select @{N="ObjectName";E={(get-view -Id $_.EntityID).name}}, EntityID, Role, IsGroup, Principal

$result | Export-CSV -Path "$pwd\vCenter_PermissionAudit-$date.csv" -NoTypeInformation -UseCulture

In the above script the results are outputted to a CSV file appended with the date.  The benefit of doing this is you can schedule it to run and now have a history of the state of permissions over time.  Simple as it is, I think it’s really handy.

In the next post I will look at creating a basic Alarm in vCenter that will trigger when a permission is modified on an object.

Patch Release ESXi550-201601501-BG breaks connectivity to vCenter

A few days back I upgraded an ESXi host from ESXi 5.1 to 5.5.  I had downloaded the latest image my vCenter and hardware would support.  The upgrade went well.  Once the host rebooted and was connected back in vCenter I performed a patch scan to see missing patches.

I had about six missing patches.  One being ESXi550-201601501-BG, released only a few days beforehand on 4 Jan.  It was marked as a Critical Bugfix.  As I have always done with Critical Bugfixes I Remediated.  After all, it was Critical and a Bugfix.  Once the host remediated and rebooted I found that I could no longer connect to the host via vCenter.  Receiving the below error when attempting to connect.

Cannot contact the specific host (hostname). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

I opened a C# Client and was able to successfully connect to the ESXi host directly.  So all looked good there.  I decided to look up the KB article of this latest patch (KB 2141207) to see what it fixed.  Nothing came up.  No page could be found!

With a lack of information on this patch the best thing I could do was just to revert the patches.  The fix was simple.  I rebooted the host.  During the Loading VMware Hypervisor I pressed Shift R for Recovery mode.

recovery01

The VMware Hypervisor Recovery screen then appears.  It shows the current build and the previous build that it will Roll Back to.  Pressing Y starts the process.  It was quick and simple and I was back up and going on the previous 5.5 build I had just upgraded to without the addition patches I installed.

recovery02
(The above screenshot was not the host that I Rolled Back but rather is a base 5.1 image)

The KB article page now works.  It is quite clear you shouldn’t update to version 5.5 Update 3b before you upgrade your vCenter to the same version of 5.5 Update 3b.  The interesting note here is that I didn’t realise that I was updating the host to ESXi 5.5 Update 3b.  My upgrade image was ESXi 5.5 Update 3a and I thought I was just installing a critical bugfix patch.  Where I can undone was this was an esx-base patch that brought the effective build number of the ESXi host up to the same level of a full ESXi 5.5 Update 3b build.

So what was actually going on with this patch…

Reading the KB article isn’t entirely obvious what’s going on.  Yes, we now know not to update unless vCenter is on Update 3b but why?  The article states that the patch updates the esx-base vib to resolve an issue when upgrading from ESXi 5.5 to 6.x, where some VMs and objects may become inaccessible or report non-compliant.  That’s clearly, though, not what we’re doing here.

The KB article now has a new link to a second KB article (KB 2140304) which describes the issue you will face if you don’t upgrade vCenter first.  What it describes is that SSLv3 is now disabled by default in this esx-base vib.  So I can only presume that vCenter wants to connect to the ESXi host using SSLv3.

Fortunately the resolution wasn’t that complex.  I was able to resolve the issue and get the host up and going relatively quickly and all within my outage window.  This is the first time I have experienced an ESXi patch break a host.  I’m surprised that a patch is actually changing the default behaviour of a host.  I can understand, and even support, VMware making the change in the latest ISO image build of ESXi 5.5 but not in a patch.

The Lesson…

So the lesson here (and there’s probably a few but I’m going with), stay away from any esx-base patches that have been released post 4 Jan 2016 until vCenter is upgraded to 5.5 Update 3b.

As a side note I would assume SSLv3 could be re-enabled on the host but I haven’t looked into this yet.

References

After upgrading an ESXi host to 5.5 Update 3b and later, the host is no longer manageable by vCenter Server (2140304)

VMware ESXi 5.5, Patch ESXi550-201601501-BG: Updates esx-base (2141207)