Monthly Archives: January 2016

Auditing and Alerting on vCenter Permissions – Part 2

In my previous post I discussed auditing permissions in vCenter via PowerCLI.  Going through the process of identifying account and group permissions and what objects they have been set on.  In this second of a two part post, I look at creating a vCenter Alarm in PowerCLI that will trigger when a permission is Added / Removed / or Modified which will then show up in vCenter Alarms.

Part 2 -- Generating a custom alarm on a vCenter permission change

The objective here is not to catch someone out (though you could see how it could help).  The Alarm is by no means meant to be as a deterrent or even used for auditing.  Where it’s useful is as an identifier that a permission add / remove / modification as taken place.  Once triggered a person can review the new permissions and acknowledge the alarm.

Working with vCenter Alarms in PowerCLI hasn’t reached the maturity as many other Cmdlets.  So creating new alarms with PowerCLI is not as straight forward as using a Cmdlet.  To create a new alarm ww have to use the powerful Get-View Cmdlet which exposes us to all the richness of the SDK APIs of vCenter.

Creating an alarm requires a few pieces of information.  First we create the alarm object, give it a name, description, and set its state.

Next we create our Triggers or expressions.  Creating the expression is where the magic lays.  The key piece is the EventTypeID.  This is the specific vCenter event that will trigger the alarm.  I found finding this information through VMware was extremely difficult.    With a little searching though, and playing around, I found the information I was after with Get-View eventManager.

$event = get-view eventManager
$eventman.description.eventinfo | Where-Object {$_.key -like "*permission*"}

Running the above without any filters returned a huge amount of information with possible events to trigger on.  Using the Where-Object Cmdlet I was able to filter the results to anything with the word ‘permission’.  This narrowed down the results to just four.  The key three being  PermissionAddEvent,  PermissionRemoveEvent, and PermissionUpdatedEvent.  Of course if you want to see everything just remove the pipe and everything after it.

With these events and a little more research I had enough information to now create a script for an alarm.  Below I created three expressions.  One for each Permission event and set it’s status to Yellow if triggered.  The alarm is created in the root of vCenter and will propagate down to all objects.

$alarmMgr = Get-View AlarmManager

# Create AlarmSpec object
$alarm = New-Object VMware.Vim.AlarmSpec
$alarm.Name = "Permission Modification"
$alarm.Description = "Track permission changes"
$alarm.Enabled = $TRUE

# Event expression 1 - Permission Added
$expression1 = New-Object VMware.Vim.EventAlarmExpression
$expression1.EventType = "EventEx"
$expression1.EventTypeId = "vim.event.PermissionAddedEvent"
$expression1.ObjectType = "VirtualMachine"
$expression1.status = "Yellow"

# Event expression 2 - Permission Removed
$expression2 = New-Object VMware.Vim.EventAlarmExpression
$expression2.EventType = "EventEx"
$expression2.EventTypeId = "vim.event.PermissionRemovedEvent"
$expression2.ObjectType = "VirtualMachine"
$expression2.status = "Yellow"

# Event expression 3 - Permission Modified
$expression3 = New-Object VMware.Vim.EventAlarmExpression
$expression3.EventType = "EventEx"
$expression3.EventTypeId = "vim.event.PermissionUpdatedEvent"
$expression3.ObjectType = "VirtualMachine"
$expression3.status = "Yellow"

# Add event expressions to alarm
$alarm.expression = New-Object VMware.Vim.OrAlarmExpression
$alarm.expression.expression += $expression1
$alarm.expression.expression += $expression2
$alarm.expression.expression += $expression3

# Create alarm in vCenter root
$alarmMgr.CreateAlarm("Folder-group-d1",$alarm)

# Add action to email alarm
#Get-AlarmDefinition $alarm.Name | New-AlarmAction -Email -Subject "vCenter Permission Modification Occurred" -To [email protected]
#Get-AlarmDefinition $alarm.Name | Get-AlarmAction | New-AlarmActionTrigger -StartStatus 'Green' -EndStatus 'Yellow'

Commented out is also the ability to email a notification out when an event occurs.  This just feels a little overkill but never the less is a notification option.

The beauty of this script is that it can also easily be modified to alarm on any trigger-able event in vCenter.  Whether you need one expression or multiple expressions it’s easily modifiable.

Virten.net has a good article on how to create custom alarms and find events that I used for a reference.  If you don’t want to search for your own events to alarm on.  Virten maintains a database of all events that can be alarmed on.  I’m not sure how current it is but it’s a good starting point.

References

How to create custom vCenter Alarms from Events | Virten.net

Vmware Data Object -- EventAlarmExpression

Auditing and Alerting on vCenter Permissions – Part 1

In this two part post I discuss and cover how to audit vCenter permission using PowerCLI.  How to enumerate basic permissions and then expand it out and send it to a CSV file for further auditing.  In Part 2, I’ll discuss how to create custom alerts using PowerCLI to trigger an event on a vCenter permission modification.

Part 1 -- Auditing vCenter Permissions

Reviewing and auditing vCenter permissions has always been awkward at the best of times.  Unless you’re on top of permissions from day one, over time you can be left with a mess of vCenter permissions.  If you’re not explicitly looking for it you won’t know it either.  This is certainly what I found recently in one of my vCenter environments.

There are a few different ways to view permissions in the C# and Web Client.  You can click on an object and view the Permissions tab or you can go to Roles.  The Roles section is certainly much more useful but still far from ideal.  You have to click on each Role and there is no way to export that information.  Using PowerCLI on the other hand you can consolidate the above two methods into one view and have it exportable.  The quickest and easier way to get started is with Get-VIPermission once connected to a vCenter in PowerCLI.

Get-VIPermission | Format-Table -AutoSize

Get-VIPermission in its default form gives you a quick overview of all Principals / Users defined with explicit permissions along with their Role, whether it’s a group, and if it’s set to propagate down the tree .  Ideally you’ll only have a few groups listed here and nothing else.  Unlike myself, where I had a large amount of individual users explicitly listed on many different objects.

When using Format-Table we loss a little information on the screen. Namely, we can’t see on what objects these permissions are set on.  So we can work with the Get-VIPermission cmdlet to expand it out a little further.

 Get-VIPermission | sort | Format-Table Entity, EntityID, Role, Principal -AutoSize

The two extra values we are looking for here are Entity and EntityID.  In most cases Entity will aid in identifying the vCenter object that has the explicit permissions set on it so you can review.  The Entity could be a Virtual Machine, a Cluster, or a Datastore.  In some cases, though, you will find the Entity is blank.

Again, in my case I had a lot of permissions with a blank Entity.  Using EntityID I could see that the objects appeared to be networking related.  So in this case we need to get a little more creative with the querying of the SDK APIs using Get-View.

Get-VIPermission | sort | ft @{N=”ObjectName”;E={(get-view -Id $_.EntityID).name}}, Entity, EntityID, Role, Principal -AutoSize

This now gets us 95% of the way there.  We use Get-View using the EntityID as the Get-View ID to find the actual name of the vCenter object for that missing Entity value.  In the case of the networking objects we will now get the exact names of the networking switches.

get-vipermission01

Putting it all together.  In this last part I turn the one liner into a little script, make sure the correct snap-in is loaded, make it a little easier to read, and have it output to a CSV file that can be used to audit and review at a later stage.

# Import Snap-in
if ((Get-PSSnapin | where {$_.Name -ilike "Vmware*Core"}).Name -ine "VMware.VimAutomation.Core")

{
Write-Host "Loading PS Snap-in: VMware VimAutomation Core"
Add-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue
}

$Date = (get-date).tostring("yyyyMMddHHmm")
$result = Get-VIPermission | sort-object -Property Principal | Select @{N="ObjectName";E={(get-view -Id $_.EntityID).name}}, EntityID, Role, IsGroup, Principal

$result | Export-CSV -Path "$pwd\vCenter_PermissionAudit-$date.csv" -NoTypeInformation -UseCulture

In the above script the results are outputted to a CSV file appended with the date.  The benefit of doing this is you can schedule it to run and now have a history of the state of permissions over time.  Simple as it is, I think it’s really handy.

In the next post I will look at creating a basic Alarm in vCenter that will trigger when a permission is modified on an object.

Patch Release ESXi550-201601501-BG breaks connectivity to vCenter

A few days back I upgraded an ESXi host from ESXi 5.1 to 5.5.  I had downloaded the latest image my vCenter and hardware would support.  The upgrade went well.  Once the host rebooted and was connected back in vCenter I performed a patch scan to see missing patches.

I had about six missing patches.  One being ESXi550-201601501-BG, released only a few days beforehand on 4 Jan.  It was marked as a Critical Bugfix.  As I have always done with Critical Bugfixes I Remediated.  After all, it was Critical and a Bugfix.  Once the host remediated and rebooted I found that I could no longer connect to the host via vCenter.  Receiving the below error when attempting to connect.

Cannot contact the specific host (hostname). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

I opened a C# Client and was able to successfully connect to the ESXi host directly.  So all looked good there.  I decided to look up the KB article of this latest patch (KB 2141207) to see what it fixed.  Nothing came up.  No page could be found!

With a lack of information on this patch the best thing I could do was just to revert the patches.  The fix was simple.  I rebooted the host.  During the Loading VMware Hypervisor I pressed Shift R for Recovery mode.

recovery01

The VMware Hypervisor Recovery screen then appears.  It shows the current build and the previous build that it will Roll Back to.  Pressing Y starts the process.  It was quick and simple and I was back up and going on the previous 5.5 build I had just upgraded to without the addition patches I installed.

recovery02
(The above screenshot was not the host that I Rolled Back but rather is a base 5.1 image)

The KB article page now works.  It is quite clear you shouldn’t update to version 5.5 Update 3b before you upgrade your vCenter to the same version of 5.5 Update 3b.  The interesting note here is that I didn’t realise that I was updating the host to ESXi 5.5 Update 3b.  My upgrade image was ESXi 5.5 Update 3a and I thought I was just installing a critical bugfix patch.  Where I can undone was this was an esx-base patch that brought the effective build number of the ESXi host up to the same level of a full ESXi 5.5 Update 3b build.

So what was actually going on with this patch…

Reading the KB article isn’t entirely obvious what’s going on.  Yes, we now know not to update unless vCenter is on Update 3b but why?  The article states that the patch updates the esx-base vib to resolve an issue when upgrading from ESXi 5.5 to 6.x, where some VMs and objects may become inaccessible or report non-compliant.  That’s clearly, though, not what we’re doing here.

The KB article now has a new link to a second KB article (KB 2140304) which describes the issue you will face if you don’t upgrade vCenter first.  What it describes is that SSLv3 is now disabled by default in this esx-base vib.  So I can only presume that vCenter wants to connect to the ESXi host using SSLv3.

Fortunately the resolution wasn’t that complex.  I was able to resolve the issue and get the host up and going relatively quickly and all within my outage window.  This is the first time I have experienced an ESXi patch break a host.  I’m surprised that a patch is actually changing the default behaviour of a host.  I can understand, and even support, VMware making the change in the latest ISO image build of ESXi 5.5 but not in a patch.

The Lesson…

So the lesson here (and there’s probably a few but I’m going with), stay away from any esx-base patches that have been released post 4 Jan 2016 until vCenter is upgraded to 5.5 Update 3b.

As a side note I would assume SSLv3 could be re-enabled on the host but I haven’t looked into this yet.

References

After upgrading an ESXi host to 5.5 Update 3b and later, the host is no longer manageable by vCenter Server (2140304)

VMware ESXi 5.5, Patch ESXi550-201601501-BG: Updates esx-base (2141207)