Category Archives: Windows 2008 R2

VM Windows Cluster Volumes Offline in ESX

Windows Clustering on physical hardware is a pain at the best of times.  Just getting it to work can sometimes be a little try and effort… with a whole lot of luck.  Getting clustering to work in VMware is just cruel.

So when tasked to create a VM of a physical Windows Cluster for a test environment, boy was I excited! {Sarcasm sign}.

Actually creating the VM within ESX wasn’t that difficult.  Using Converter I created a VM of the OS.  Then using our DELL EqualLogic SAN I made clone copies of the cluster volumes.  I presented those volumes with the newly created VM as RDMs.  The process seemed to work really well until.  The OS booted up.  I could see all my presented volumes.  Issues began when I tried to start the Clustering Service and take it out of manual mode.  Out of the 6 volumes I had only one would ever become Online while all the others would (after some time) fail.

I spent days working through the issue (I’m pretty sure this is why I’m balding).  Articles seemed to lead me to DISKPART and trying to change the SAN Online Policy, manually online the disk, changing the READONLY attribute.  None of these seemed to work.  I’m assuming because there was an attribute that said the disk was Clustered and would prevent me making any changes.  Still, I thought I was on the wrong ‘path’ and began looking into a lower level issue at the ESX level.

The crux of my issue turned out to be a iSCSI multipathing problem.  DELL EqualLogic SANs run in an Active / Active pathing method where I/O is sent over all paths.  DELL has a third party Storage API plugin for ESXi that change the default behaviour of how mutlipathing works.  This is normally a good thing but for Windows Clustering in ESX… this is bad.

The solution is fairly simple to resolve.  The steps below is a rough outline of how to identify and change the multipathing policy.

Using vSphere vCenter, the changes are made within the Storage Adaptor.  In this case it’s the iSCSI Software Adaptor under the Configuration tab.

In the bottom pane select the paths view.  Expand the Target column and identify one of the cluster volumes with issues.  In this example I have a Dead path due to a recently removed SAN volume which is safe to ignore.  The one below is of interest as it’s one of the clustered volumes.  Remember the Runtime Name in the left column.

Change to the Devices view and locate the Runtime Name.  Right click on this device and select Manage Paths.  In this example DELL_PSP_EQL_ROUTED was selected as default.  Changing this to Most Recently Used (VMware) sends I/O only ever down one path.  The change is immediate.  As my volumes are offline I can safely make the changes.  On a working production volume I wouldn’t be making path selection changes during business hours.

Back over on the Windows Cluster VM I can now restarted the Clustering Service and have it correctly Online all the volumes.

MSCS is quite in depth and not for the faint hearted or something configured before you end home for the night.  Virtualising MSCS requires additional planning and thought in addition to regular planning.


VMware -- Setup for Failover Clustering and Microsoft Cluster Service

Forcing removal of tombstoned Domain Controller

I recently faced a issue scenario where a Domain controller at a remote site became tombstoned after not having replicated with Active Directory for 60 days.  Putting aside how I never noticed this, there was little I could do in this situation.  It’s time had skewed out soo far that it stopped participating with replication.  Whatever the issue, if a domain controller doesn’t communicate / replicate with AD within AD’s tombstone lifetime it will eventually become permanently tombstoned.

The default tombstone lifetime in Windows Server 2000 – 2003 is 60 days.  In Windows Server 2003 SP1 and above it’s 180 days.  Despite being Windows 2003 R2, the forest came from SBS 2003.  The originally tombstone lifetime doesn’t change when you upgrade so it stayed 60 days.

The first part to fixing the issue was demoting the domain controller back to a standalone server.  Once performed I could fix whatever issues the network had and re-promote at a later stage.

Even though the network was up and the domain controller in question could connect to other domain controllers.  Being tombstoned meant that it wouldn’t talk with the DCs.  Running the command dcpromo on the DC in question would fail when it attempted to communicate with the domain.

To work around the issue the command needed to be run with the /forceremoval switch.

Dcpromo /forceremoval

Below are the steps to perform a force removal.

1. Run dcpromo /forceremoval from the run box.

2. Click next to start the wizard.

3. Confirm the removal.

4. Sent a new administrator password for when the server becomes a standalone server.

5. Confirm the removal of AD without cleaning up the metadata.  This is an important step to note.  Because we are forcing the removal of AD without cleanup up the metadata this is a manual step we will have to perform in our AD environment on a functioning DC.

6. Demotion will now start and removal the server from being a Domain Controller.

7. Click finish and reboot the server to complete the process.

With the server now successfully demoted it can be promoted back to a domain controller using the standard dcpromo command.  Before this can happen, though,  we have to go back to step 5 above and perform a manual metadata cleanup of Active Directory to removal any references to this tombstoned DC.  I’ll be covering this more indepth step in a later post.  Microsoft has a very thorough article on how to perform this process

With the server demoted and a metadata cleanup performed I could happily promote this server back to a DC.   Preventing the issue happening again would mean fixing my monitoring and sorting out any time sync issues… also a post for a later stage.



Metadata cleanup
How to remove data in Active Directory after an unsuccessful domain controller demotion 

Redirecting name resolution of a single host for an external DNS zone

Redirecting, changing, forwarding, adding… all valid ways of trying to achieve the same thing.  You need to create an A host record for an external DNS zone without becoming  authoritative for the whole zone.

A perfect scenario is you have a limited VPN connection between yourself and a partner company.  You need to access their Intranet server.  The partner company only publishes the Intranet address ( to their Internal DNS server and not to the Internet.

One of the most obvious things you might try and do is create a new zone within DNS called  Then to create an A host record called intranet.  This will work, of course, but with a nasty side effect.  It will make you the authoritative name server for the zone.  Any other records that this zone has published on the internet won’t work.  For example if you tried it wouldn’t resolve to the partner company’s website.  Sure, you could create one to one mappings of each A host record the partner company uses, assuming you know what they are, which you probably don’t.

A more appropriate way is to still create a zone but include the host within the full zone name.  In our above scenario the zone now becomes  We then create an A host record with a blank name that points to the Intranet’s IP.  In essence we are stay we are the authoritative name server but only for this one specific host.

Below I run through the basic steps to create a zone assume some level of knowledge of zone creation.

Step 1

Within DNS create a New Zone

Step 2.

Proceed through the wizard till you get to Zone Name.  Type the full DNS name of the record you want to create.  In this example we are creating

Step 3.

Continue through the wizard and complete the creatation of the zone.  Select the zone and select New Host (A or AAAA).  Leave the Name blank and enter in the IP address of the host you wanting to create / redirect and Add Host.  In this case we are using a private IP of

The end result is a zone that looks similar to below within DNS (click for enlargment). will now resolve to while all other published records for will continue to correctly resolve to there respective external addresses.

Blank Page in OWA 2010

If you’re anything like me there will be times when you’re so busy (or maybe a little lazy) you might just count hope on the fact that a Microsoft install will check installation prerequisites and warn / install them for you.

Unfortunately Exchange 2010 isn’t one of those installations.  Using a default install of Windows Server 2008 R2.  I performed an installation of Exchange Server 2010.  Using only the pre-installation checker during the installation process I installed only what was requested to get Exchange 2010 to install.  For the most part this was fine.  It gave me a functioning Exchange 2010 test server.

It wasn’t till I tested OWA that I found it didn’t install or notify me of all the prerequisite web components.

When I browsed to OWA I was redirected to https://exchangeserver.local/owa/auth/logon.aspx?url=https://exchangeserver.local/owa&reason=0

Fortunately the fix was rather simple.  It required two PowerShell commands and no Exchange service restarts.

Import-Module ServerManager

Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,RPC-Over-HTTP-Proxy -Restart

Update Rollup 3 for Exchange Server 2007 SP3 fails

Over this weekend I had to install Update Rollup 3v2 for Exchange 2007 SP3.  Installation went fine on our Windows 2003 Operating Systems.  When I came to applying the Rollup on our Windows 2008 R2 servers I hit a snag.

When I went to install the patch, the installation wizard would get to the point of stopping services and then roll-back.  It would then display the following message.

Setup Wizard for Update Rollup 3 for Exchange Server 2007 (KB 935999) ended prematurely because of an error. Your system has not been modified. To install this program at a later time, please run the installation again.

Update Rollup 3v2 has some known issues (  The first issue relates to OWA –not my problem.  The other relates to the installation which produces the same error message I received.  If the %SYSTEMDRIVE%ExchangeSetupLogs folder has been deleted, renamed, or moved from the original Exchange installation it can produce this error.  The fix is to restore this installation folder or to create a new one.  A quick check and the folder did exist in my scenario.  Never the less I tried to rename the folder and create a new once with the same name.  Unfortunately the installation once again failed produced the same error.

My next step was to enable Windows Installer Logging. This can be done via the registry or by going to Microsoft at the following URL and selecting the link to enable logging.  This will run a small msi installer that will make the necessary registry changes.  It can also be switched off using this same method.

Once logging was enabled I re-ran Update Rollup 3v2.  When the installation failed again I went to the command prompt and typed the following, ‘cd %temp%’. This took me to the location of the Windows Installer log file.  After some investigating through the log I noticed a line indicating that I might have insufficient permissions.

Now I was logged in as an Enterprise Administrator so that couldn’t have been my issue.  One thing that I knew could cause administrator permission issues was UAC -which was on.  So I turned it to ‘Never Notify’ which is effectively off.  This required a reboot which I performed.

Once the server was restarted with UAC off I tried the installation again and Voilà.  The installation wizard was able to stop services and perform a successful patch.  At the completion of the wizard I turned UAC back on to its default setting and performed another reboot.


Server Specs
Windows 2008 R2 Enterprise SP1
Exchange Server 2007 SP3

Update Rollup 3-v2 for Exchange Server 2007 Service Pack 3 (KB2530488)

How to enable Windows Installer logging

Known Issues

Registry Changes to enable logging
Reg_SZ: Logging
Value: voicewarmupx