vIDM Upgrade Failed: Disk space check on /opt/vmware/horizon

Continuing on my journey to upgrade VMware Identity Manager from 3.3.4 to 3.3.5. I ran into a failed Precheck validation in VMware Lifecycle Manager during an attempted vIDM upgrade related to disk space.

The Error

The Precheck failed on the check, Disk space check on /opt/vmware/horizon. With the Result Description that at least 7 GB of free space is required. The failed check proceeds to provides how much free space you currently have on this mount point.

Checking Free Space

Even though the Precheck provides how much space is available we can SSH to the vIDM appliance and from the CLI run a df -h to provide a little more information.

[email protected] [ ~ ]# df -h
 Filesystem                     Size  Used Avail Use% Mounted on
 devtmpfs                       2.9G     0  2.9G   0% /dev
 tmpfs                          3.0G   12K  3.0G   1% /dev/shm
 tmpfs                          3.0G  768K  3.0G   1% /run
 tmpfs                          3.0G     0  3.0G   0% /sys/fs/cgroup
 /dev/sda4                      8.8G  7.5G  882M  90% /
 tmpfs                          3.0G  180K  3.0G   1% /tmp
 /dev/sda1                      124M   30M   88M  26% /boot
 /dev/mapper/tomcat_vg-horizon  9.8G  5.4G  3.9G  59% /opt/vmware/horizon
 /dev/mapper/db_vg-db           9.8G  880M  8.4G  10% /db
 tmpfs                          597M     0  597M   0% /run/user/1000
 tmpfs                          597M     0  597M   0% /run/user/107

We can see that /opt/vmware/horizon only has 3.9 GB of available space on a 9.8GB volume.

Solution -- Part 1

The failed Precheck provides a recommendation to VMware KB article How to zip and remove old vIDM logs in vIDM appliance (81220). This article provides a script that can be download and explains that the script, when run, will compress the vIDM logs under the /opt/vmware/horizon mount,

The steps are clearly detailed in the article. Download the script to the vIDM appliance. Change permissions on the script so it can be executed, and run.

[email protected] [ ~ ]# ./81220_checkFreeSpace.sh
Current /opt/vmware/horizon volume free space is 3.9G
Trying to zip old log files


Zipped old logs successfully
Removing old Log files
Removed old Log files
echo checking space on volume /opt/vmware/horizon post zipping
/opt/vmware/horizon has 4.6G free space post old log compression and deletion.
SUCCESS

When the script completes, it will state the current free space. In my case above, 4.6 GB.

Solution -- Part 2

As you can see from above, the script did not free the required 7 GB required to upgrade vIDM to 3.3.5. So a further cleanup was required on /opt/vmware/horizon on the vIDM appliance.

Running the command du -a /opt/vmware/horizon/ | sort -n -r | head -n 10

I found that a large portion of space was taken up by RPM packages under /opt/vmware/horizon/26160/package-pool.

Disclaimer: I have not been able to find internally to VMware or externally that the next steps are supported or recommended to free up space on vIDM. So perform at your own risk.

I changed to the /opt/vmware/horizon/26160/package-pool directory. Many of the RPM packages in this directory appeared to reference version 3.3.4. Working on the assumption that these are the extracted packages from the previous upgrade of vIDM. I proceeded to snapshot vIDM in vCenter and then remove all these packages with the command rm *rpm. This now gave me 7.8 GB of available storage, which is enough to meet the 7 GB required from LCM to perform an upgrade.

Conclusion

If sufficient storage is not available for an vIDM upgrade a manual clean up is required on the appliance. VMware provides a script to clean up log files though this might not produce enough free space. A further clean up can be performed by deleted old RPM binaries used from previous upgrades. Performing both of these clean up steps should produce sufficient free space on the appliance.

At this point you should be able to initiate another upgrade and run a Precheck. This time it should detect sufficient free space and pass the check allowing the upgrade to proceed.

Leave a Reply

Your email address will not be published.