Category Archives: SAN

Installing NetApp NFS Plug-in for VMware VAAI

For some time now NetApp have supported VAAI for NFS on vSphere.  If you’re using NFS on your NetApp with vSphere you might want to investigate installing the NFS Plug-in for VMware VAAI.

The plug-in helps vSphere communicate with NetApp basically allowing it to offload certain tasks from vSphere to NetApp.  By passing certain tasks off to the NetApp, tasks can be processed faster and communicated back to vSphere when complete.  An example is provisioning a VMDK file or performing a vMotion task.  Rather than vSphere attempting to perform these tasks over the wire they can be performed directly on the NetApp by the array itself.

There are three different ways to get the NFS Plug-in installed onto an ESXi host.  I’ve detailed the three different options below.  Before you start you’ll obviously need the NFS Plug-in which can be downloaded from the NetApp support site.  You’ll need a login ID and a valid support contract to do this.

Option 1. ESXCLI

This is my preferred option when only needed for a few hosts.  It can be done on the actual ESXi host (e.g. ssh) or via the vMA.

Step 1. Copy the NFS plug-in zip file to a location that the ESXi host has access to.  Below I copied the file to a folder called ‘vib’ on a test datastore.

1_nfs_vib00

Step 2. On the ESXCLI run the following command.

esxcli --server HOST_IP_ADDRESS software vib install -d /PATH_TO_VIB/vib_filename_.zip

nfs_vib01

Step 3. Reboot the host

Step 4. Check that the NFS plugin was installed with the following command.  Scroll till you find NetAppNasPlugin under Name.

nfs_vib03

Option 2. VMware Update Manager

Step 1. Install the Plug-in into the Patch Repository.  Click Import Patches and Browse to the location of the Plug-in zip file.

2_nfs_vib01

Click Next and ignore any certificate warning you may get to Import the patch.

2_nfs_vib02

Click Finish to finish the Import.

You should now see the NetAppNasPlugin in the Patch Repository list.

2_nfs_vib03

Step 2. Create a new baseline for the NFS Plug-in.

Click on Baseline and Groups. Right click to create a New Baseline.  Fill in the Name and Description and select Host Patch.

2_nfs_vib04

Click next and select Fixed.

2_nfs_vib05

Scroll through the list of patches and locate the NetAppNassPlugin.  Add the patch using the down arrow and click next.

2_nfs_vib06

Click Finish to install the patch.

2_nfs_vib07

Step 3. Attach the newly created Baseline to your hosts.  Where you choose to do this is up to you.  I choose to do it at the Cluster level.

2_nfs_vib08

Step 4. Once attached Scan and Remediate your host.

2_nfs_vib09

Option 3. NetApp Virtual Storage Console

This option is obviously dependant on you having already installed the Virtual Storage Console on a server and having the vSphere Plugin enabled.

If correctly installed the NetApp VSC can be found under Solutions and Applications called NetApp.

Navigate to Monitoring and Host Configuration and click on Tools.  Under NFS plug-in for VMware VAAI it will say Unable to location plug-in.

netapp_nfs_plugin00

Step 1. Extract the NFS zip file and locate the vib inside it.  The vib will be denoted with a version number at the end.  Make a copy of the file and call it NetAppNasPlugin.vib

This specific filename is required for the VSC to detect the vib correctly.

netapp_nfs_plugin01

Step 2.

On the server where the Virtual Storage Console was installed.  Copy the renamed file to C:Program FilesNetAppVirtual Storage Consoleetcvscweb

netapp_nfs_plugin02

Step 3. Exit vCenter and log back in.  Open the VSC back up.  If the vib was renamed correctly and copied to the correct location.  The VIB should now be detected under Tools of Monitoring and Host Configuration.

netapp_nfs_plugin00

Step 4. Click on Install on Host to install the VIB plug-in.  Any incompatible hosts will show up greyed out with a null besides there name.

In the below screenshot I have three incompatible hosts.

netapp_nfs_plugin04

So there you have it.  Three different ways to install the NetApp NFS Plug-in onto an ESXi host and three different pain in the ass ways.

Good Luck.

The Virtual Storage Appliance experiment – Part 3

FreeNAS

The Virtual Storage Appliance experiment – Part 1
The Virtual Storage Appliance experiment – Part 2

The second appliance I’m trialling for my Virtual Storage Appliance experiment is FreeNAS.  The history of FreeNAS goes back to 2005 but trying to understand that history and it’s version numbering really just hurts my head.  The current version of FreeNAS I’ll be trialling is FreeNAS 9.1.1.

The original FreeNAS project started back in 2005 under a 0.x version numbering scheme.  It continued to be developed until 2011.  At which point the story differs depending on what you read.  The nearest I can ascertain, FreeNAS was aquired by iXsystems in 2011.  The FreeNAS version numbering changed to fall in line with FreeBSD.  With version 8 being the current FreeBSD release so the next FreeNAS version was also 8.  In 2012 the NAS4Free project was created.  NAS4Free make a point that it is not a fork of FreeNAS but the direct continuation of FreeNAS prior to it’s sale to iXsystems.  The actual fork is FreeNAS after its purchase by iXsystems when it went through a complete code rewrite.  But, that’s enough of the background.

I ended up documenting my complete FreeNAS install process similar to OpenFiler.  I’ve decided not to go through the entire installation process (that may come at a later entry).  Instead I’ll focus on what FreeNAS is capable of offering, and it does offer a lot.  I was a little worried starting with OpenFiler that it would be downhill from there.  It certainly wasn’t the case with FreeNAS.

After installation, the initial configuration of FreeNAS is web driven, just like OpenFiler in Part 2.  The admin portal is split into two panes.  Much like something you would see on a Windows box.  A drop-down menu system on the left with the configuration and settings on the right.  It’s actually really nice, fluid, and fast, in some ways better than some of the enterprise products on the market.  The menu system has almost been ordered in the sequential steps that you want to configure it in.  What I did find slick was you could open up a GUI CLI console window from the web browser.

I found the doco on FreeNAS excellent.  It’s well written and flows well.  I say it flows well because to the uninitiated you can basically start at the top of the doco and work your way down to get yourself up and running quickly, rather than having to read through 100’s of pages of documentation on the ins and outs of all the features.  On the features side FreeNAS is your Swiss Army knife of NAS appliances.  You have your stock iSCSI, NFS, CIFS support.  You also have AFP (Apple Filing Protocol) with Apple Time Machine support, but who cares about Apple 😉   FTP and TFTP servers.  Rsync replication and File system snapshotting.  It also has a plugins architecture which is really just a simplified way of installing add-ons and software via the GUI.

Where Openfiler was a little more intuitive to create and present a LUN over iSCSI.  FreeNAS did require a little more thought and some reading.  They bring a few more concepts to the table than most which just creates a whole lot more steps to get to where you’re going.  For example, you will first create a volume from a pool of disks then a zVol under that.  In the iSCSI section you will create an Extent which you’ll associate with that zVol as the device.   You will then create an Initiator which will allow you to create a Target and then associate the two together… Still with me.  You then take that Target (with the associated Initiator) and associate that with the Extent.  If all goes to plan and you created your Portal correctly it should have a LUN that’s presented out and able to be connected to from another device, e.g. ESXi.

Fortunately NFS was sooooo much easier to setup.  I’m not a big fan of NFS but as much as I don’t like it I found myself willing to use it more as it was orders of magnitude simpler to setup.

Finally, one thing that stood out was that FreeNAS appears to support a few vSphere VAAI primitives, ATS and Zeroing File Blocks.

Where ESXi has no idea what’s going on with OpenFiler.  With FreeNAS it’s able to take advantage of some real storage performance benefits.

freenas14

I feel I’ve given FreeNAS a good workout.  Where I was happy to move on from OpenFiler after a few weeks.  I’ve been using FreeNAS for a few months now.  I’m glad I held off on writing this entry until now.  If I had written it up after the first two weeks I would have said it was too convoluted to use and to stick with something else like Openfiler.  Having past that learning curve it’s quite the opposite now.  The granularity of what it can do, it’s feature set, and community support is a great selling point.

The Virtual Storage Appliance experiment – Part 2

Openfiler

-Follow the start of my Virtual Storage Appliance experiment with Part 1.

First up in my VSA experiment is Openfiler.   An Open Source SAN/NAS solution.  Openfiler comes with a raft of features.  For a home test lab it’s more than you could possibly ask for.  It will do both Block and file back based storage.  CIFS, NFS, RAID, snapshotting, the list goes on.  For my test lab purposes I focused on the Block based iSCSI features.  Openfiler is avaliable as an Open Source edition or with commercial support.  The latter providing advanced features, such as, High Availability, replication, and Fibre Channel target.

The first objective was to install it.  The download comes as an ISO file.  Installation was very simple with straight forward installation instructions supplied to perform either a text based or GUI install.

openfiler01

After a few simple clicks the installation was complete.  It is about as close as an Appliance install as you can get without being an Appliance.

openfiler04 openfiler05

Once installed and booted you are taking to a console prompt.  All administration is done via web GUI.   Both the admin GUI and user GUI are accessed through the same URL.  The admin GUI is accessed using the root account while the user configuration is accessed via the Openfiler account.

openfiler06

Instructions were non-existant on the non-commercial side with access to just the community forums.  For the SAN uninitiated, configuration would no doubt be a challenge.

I installed Openfiler on a 50 GB VM drive within ESXi.  I was immediately faced with an issue trying to create a usable volume.  At midnight on a work day and with no instructions I might have been asking for a bit much of myself.  When the words started floating on the screen  I decided to called it a night and came back to it the next day.  The following day faced with the same issues, unable to create a volume, I started trawling through the forums.

A few posts pointed to the partition type of the disk being msdos and suggested trying to the modify the disk to gpt.  Instead I added a new virtual disk which immediately detected as gpt.   That allowed me to edit the disk and partition out the disk.

openfiler07

In typical Linux fashion it was based around start and ending cylinder numbers.  Again not that intuitive to non-Linux admins.  Once a partition was created, volumes needed to be created.  The process became a little easier at this point.  Volumes could be created in MB with a dropdown menu to select the filesystem type.  As I wanted iSCSI I selected block.

While exploring the tab menus after installation in the GUI interface I noticed that most services were disabled, including iSCSI.  So I changed it to Enabled and click start.  Back over to the Volume tab I went to iSCSI targets.  I could see no iSCSI targets but a button to add one, so I clicked it.  Next under LUN mappings I could see the volumes that were previously created.  On each volume I click Map.

openfiler09

At this point using my previous knowledge of SANs I felt I had done everything I needed to now present that storage to a device.  In my case an ESXi host.  I had already preconfigured iSCSI on my ESXI host.  I had a software adapter added and bindings all setup.   I added the Openfile IP as an Target and then performed a Rescan.

openfiler011

Once the scan completed my disk appeared.  While still in ESXi I went over to Storage and attempted to add in a new Datastore.  The disks I created weren’t appearing and now after midnight again, I was ready to throw in the towel.  After a short think about what was going on.  I knew that a VMFS datastore did require a small amount of storage overhead.   So I created a new volume in Openfiler which was a bit more realistic in size at 2 GB (rather than piddly 256 MB I originally did).  This time the disk appeared and I was able to create a Datastore.  As expected, close to 800 MB was gone after formatting and VMDK overhead.

Putting aside the final process of adding a disk into ESXi.  The whole process of installing and configuring Openfiler went relatively smooth.  It did require a little bit of troubleshooting.  Not having access to official documentation and only community forums didn’t help.  The community is great but as with any community they are quick to lose interest when a solution isn’t straight forward.  Having a good working knowledge of SANs and iSCSI went a long way.   There are some features worth further investigation, namely, Snapshots, LDAP authentication, and NFS.  Some initial testing of snapshots haven’t work so no doubt it will require more time on the forums.

I did later come across a site that specialises in Virtualization solutions with some a installation and configuration documentation for Openfiler and ESXi.  I’ve provided a link the the PDF below.  It’s actually quite a good step by step document.

References

Xtravirt Openfiler install and ESXi iSCSI configuration

The Virtual Storage Appliance experiment – Part 1

The Virtual Storage Appliance experiment – Part 1

I have wanted to play around with Virtual Storage Appliances (VSA) for a while now.  I have never really had a need to use them though.  I have colleagues tell me about the VSAs they use in their home labs all the time.  I guess I have been fortunate enough to have had expose to enterprise SANs in test and lab environments for many years.

In the current role I’m in I don’t have that luxury.  I manage an overprovisioned vSphere and SAN environment that I just don’t want to mess with.  I guess we tend to take for granted the toys we have in some roles.

I’ve recently been considering pursuing my VCAP certification too.  Storage, iSCSI, multipathing is all a large part of the blueprint guide.  Having a VSA in my home lab where I can creating and test a storage network in vSphere will no doubt help in my study.

With this in mind, I’ve decided to spend the next few weeks researching what is out there in the VSA market.  I don’t have a budget behind me so really just from a free and open source point of view.  My goal is not to find an enterprise ready VSA but rather find a virtual SAN that I can have in my test lab.  An appliance that that is easy to install & setup and able to present data via protocols such as iSCSI and NFS, anything else is a bonus.

The products I have initially planned on testing are Openfiler, FreeNAS, and HP P4000.  The latter being a commercial product but with a trial period.   I’m certainly interested in testing more.  So feel free to comment and leave recommendations on other products worth looking into.

EqualLogic Multipathing Extension Module – Installing

Last year I wrote a post on an issue attempting to install the DELL EqualLogic Multipathing Extension Module using VMware Update Manager.  I discussed an alternative method to VUM using the CLI to install the MEM.  The post has turned out to be fairly popular.  I’m guessing though that most people are more interested in how to install the EqualLogic MEM using VUM rather than my original workaround.  So I thought I would run through the steps using a version of MEM that now works.  The whole process of importing, attaching, and remediating came out a little longer than expected but I managed to capture all the steps in what I think is fairly easy to follow.

The version of MEM I am using is 1.1.2 (released Dec 2012).  You can obtain it from the EqualLogic support site (sign-in required).  The release notes state that the only change from version 1.1.1 is that it’s now compatible with Update Manager 5.1.  EqualLogic also state that if version 1.1.1 is installed 1.1.2 is not required.  At least this now explains why I had trouble with VUM and version 1.1.1

Using the vSphere Client under Solutions and Applications select Update Manager and click on the Patch Repository tab.

mem01a

Click on Import Patches.  Browse to the location of the patch.  Select the version you want.  In my case for ESX5

*Note: The ZIP file from the EqualLogic support site needs to be extracted prior to importing.  Once extracted there will be two zip versions.  An ESX4 and an ESX5 version.

mem01b

If the Upload is successful you’ll then be asked to confirm the Import.

mem01c

Once imported scroll to the bottom of the repository list and you should see the new Host Extension.

mem01d

With the extension imported into Update Manager we now create a new Baseline.  Click on the Baselines and Groups tab.

mem02

Click on Create to create a  new baseline.  Assign a name to the baseline and a description.  For the Baseline Type select Host Extension and click Next.

mem02a

Scroll to the bottom of the list and select the recently imported MEM patch.  Click the down arrow to Add the Extension and click Next.

mem02b

Confirm that the correct extension was selected and click Finish to create the baseline.

mem02c

With the patch imported and a new Baseline created for the Extension we now have to Attach the baseline.  This can be done at the top of the vCenter level or right down to the Host level.  In this case I just want to do a single host.  So I’m going to select the host and then select the Update Manager tab.  I’m then going to click Attach.

mem03a

Select the newly created baseline and click Attach.

mem03b

The baseline will now appear with a Question Mark beside it until a new scan is performed.  Click Scan, make sure Patches and Extension are selected and click Scan again.

mem03c

Once the scan is complete the Extension will now show up with a red cross signifying that it’s missing and needs to be Remediated.

mem03d

Click the Remediate button to start the process.  Select Extension Baselines on the left and the recently created Baseline on the right.  Then click Next.

mem03e

Omitted is a number of steps from the Remediate Wizard.  The options revolve around how the host and cluster will behave in Maintenance Mode.  The options are fairly straight-forward and the default options usually suffice. The last screen will summarise the options selected.  Make note what options have been selected and that the correct Baseline is selected.  Click Finish to start the Remediation.

mem03f

The host will now enter Maintenance Mode using the options you selected above.  Once complete we can select a datastore and select pathing where we can see a new pathing option and it’s selected by default. We will also see that all paths to the LUN are Active.

mem04

The whole importing and creating a baseline can seem a little tedious at first, but once done, all that’s needed is a scan and remediate on new hosts.

References

Link to original article EqualLogic MultiPathing Extension Module -- Alternative Install

Download the latest Extension module from EqualLogic Support Site

EqualLogic HTTP vs HTTPS vs Encrypt Communication

This week I realised that I had the option to log into an EqualLogic Web Management Portal with either HTTP or HTTPS.  It got me thinking what effect that has on the Encrypt Communication checkbox during login.

encrypt00

EqualLogic login prompt running Firmware 5.x

Under default configuration of an Equallogic array you have the option to use/select Encrypt Communication during login.  This can be changed and you can force the use of this option.

encrypt06

Under Group configuration select the Administration tab.  You will see that Web access is enabled and under GUI access that the checkbox for Allow only secure communication is unticked.  Ticking this box will force the use of Encrypt Communication during login.   You will then notice that Web access will change to Secure Only.

encrypt07

The above screenshot is running on Firmware 6.x.  On Firmware 5.x the checkbox is called Allow only secure SSL communication.  Oddly enough once enabled on either firmwares this won’t prevent the use of HTTP access to the Web Management Portal.

Now when attempting to login you will have to use Encrypt Communication.  Under Equallogic PS Series 5.x Firmware you have to select the checkbox.  If you don’t you will receive an error when attempting to login.

encrypt03

Under PS Series Firmware 6.x the checkbox will be selected by default and greyed out.  So you won’t get the above message.

encrypt04

As mentioned above, HTTP web access is still possible along with HTTPS.  So what’s going on here?!?!  Hence the reason for this post…

So I fired up Wireshark to watch  communication between my PC and the EqualLogic Array.  I first tried accessing the Web Management Portal with HTTPS and logging in  using the Encrypted Communication checkbox.  I then tried again but this time using Encrypt Communication.  No surprise here, both times all traffic was encrypted right from sign-in.

Next I accessed the Web Management Portal using HTTP,  not using Encrypt Communication, and signed in.  Looking through the Wireshark logs I could see my username and password in plaintext (certainly not recommended).  Again using HTTP to access the Portal I enabled Encrypted Communication and signed in.  This time looking through the Wireshark logs I could see my sign-in details were encrypted and all subsequent information as well.

From what I can see going on here is that the EqualLogic Web Management Portal is a Java Applet.  When loaded a connection is established over port 3002 on Firmware 5.x and Port 3003 on Firmware 6.x.  When Encrypt Communication is selected during sign-in, SSL encryption is handled by the Java Applet.  When not selected during sign-in SSL encryption is determined by whether you use HTTP or HTTPS and relies on the browser securing communication.

So if using HTTPS to access the Management Portal you’ve relatively sure your communication is secure but you can’t guarantee other admins are doing the same.  The safest thing to do is always enable the checkbox in the Administration tab Allow only secure communication.  By enabling this option you can be sure that whether administrators use HTTP or HTTPS all communication to the EqualLogic Array will be secure.

EqualLogic predictive disk failure

For users of EqualLogic PS Series SANs.  If you have recently upgraded member firmware to version 6.x you might have noticed a change in behaviour for failed disks.

Prior to firmware V6.x there was no advanced warning for a disk failure.  Once you had a failed disk your array would move into a rebuilding phase with your hot spare disks.  You would then receive a collection of alarms that you have a failed disk, fewer spare drives, and array is being rebuilt.

New in firmware version 6.x is an early disk failure detection algorithm.  This is a great improvement over previous firmwares.   The EqualLogic member will communicate with the HDD firmware and when it detects a pending failure the EqualLogic member will initiate a block level copy from the source disk to the hot spare disk.  When the block level copy is complete the hot spare becomes the active disk and the source disk becomes failed.  The benefit in this approach is that no RAID rebuild phase needs to happen, which can take a significant amount of time with lots of disks.  A RAID rebuild can also cause a performance impact to the SAN which is mitigated with the predictive failure copy.

The only downside I’ve seen of this is the potential for a higher number of disk failures.  In some of the early PS Series SANs certainly types of types of disks, namely SATA, had very aggressive disk failure algorithms built into the HDD firmware.  It meant that there was a higher number, than normal, of false positive disk failures.  This was later corrected with a HDD firmware update that was recommended to all customers, which can still be found of the EqualLogic support site.  While I haven’t seen as many disk failures of years past I have noticed a slight increase in failed disks running Firmware version 6.x since the HDD firmware upgrades.

Predictive disk failure starting a copy to a Hot-Spare disk.

equallogic_repair02

Completion of copy results in a failed source disk.

failed_disk_equallogic

Enable EqualLogic Active Directory Authentication

If you’re using a DELL EqualLogic SAN you have the ability to turn on Active Directory authentication.  The benefit once setup is that you can control access to the SAN via AD groups rather than giving out the Group Admin account or maintaining local accounts on the EqualLogic group.

The process to turn on Active Directory authentication is quite simple.  Whether AD authentication is on or off you can still use Local authentication and locally created accounts.  So if you do lose connectivity to AD you will still be able to local on with the default grpadmin account or any other local accounts that you have made.

To begin, login to the EqualLogic Group Admin webpage with local Group Administrator permissions.  On the left select Group Configuration then navigate to the Administration Tab.  It should look similar to below with ‘Local Only’ set as the authentication type.

equal_ldap010

Select the Active Directory radio button.  A new window will appear similar to below.

equal_ldap02

Click Add and type in at least one IP address of an AD Server.  If you have more than one (which you should) you can click Add again and input multiple servers.  The EqualLogic san will connected against the first AD server in the list and if unable to connect will work its way down the list.

On the right you can leave the Secure protocol as none and Use Default Port.  If you’ve successfully put in the correct AD server IP addresses you should be able to click on Get Default and the Base DN should be automatically populated with you the root DN of your AD domain name.

For the User you will need an AD account.  Open up Active Directory Users and Computers.  Create a basic user with no special rights.  Set the account not to expire.  Make sure it has read access into AD (by default all user accounts will have this).  Back in the EqualLogic Group Administrator use this new account created.  Use the full domainusername format.

equal_ldap04

Click the Test AD Settings.  A new window will appear and make a connection to each AD Server you added and perform a test search using the User you just created.

Hopefully all green ticks will be returned and you can click Ok and return to the AD Settings window.  If you receive a red cross and a fail double check the IPs of the AD servers.

equal_ldap05

Click OK and one final window will open asking to join the EqualLogic group to the Windows domain.  You can choose to Cancel this step if you don’t wish to use Single Sign On.  If you proceed, for the username enter in an Administrator account that would have permissions to add workstations to the domain and click ok.

equal_ldap06

If successful you will see the Group name added as a computer in AD Users and Computers under the Computers OU.

The Administration tab should now look similar to below.  You will have a new Active Directory Status section with a couple green ticks indicating that you have successfully added in an AD server and the Group was successfully created as a computer object in AD for SSO.

equal_ldap03

That’s all there is to it.  You can now click on Add to add a new users or group from AD.  A window will appear giving the option to create a standard local user account but now the radio buttons to create an AD user or group are available.

The process is the same for a user or group.  Select the Add a new Active Directory user radio button.  Under General settings type in the username of the AD User omitting the domain name.  You can click on Check name which will verify with a green tick that the user does indeed exist in AD.

equal_ldap07

Click next and specify the permissions you want the account to have.

equal_ldap09

Click next again, verify the details and permissions you have set for the account then click finish.

equal_ldap08

If you choose to use an AD group.  I recommend first creating a Domain Local group first in AD.  Populate this group with the users you want to have access.  Then run through the steps above but select Add a new Active Directory group.