Continuing on my journey to upgrade VMware Identity Manager from 3.3.4 to 3.3.5. I ran into a failed Precheck validation in VMware Lifecycle Manager during an attempted vIDM upgrade related to disk space.
The Error
The Precheck failed on the check, Disk space check on /opt/vmware/horizon. With the Result Description that at least 7 GB of free space is required. The failed check proceeds to provides how much free space you currently have on this mount point.
Checking Free Space
Even though the Precheck provides how much space is available we can SSH to the vIDM appliance and from the CLI run a df -h to provide a little more information.
root@vidm [ ~ ]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.9G 0 2.9G 0% /dev tmpfs 3.0G 12K 3.0G 1% /dev/shm tmpfs 3.0G 768K 3.0G 1% /run tmpfs 3.0G 0 3.0G 0% /sys/fs/cgroup /dev/sda4 8.8G 7.5G 882M 90% / tmpfs 3.0G 180K 3.0G 1% /tmp /dev/sda1 124M 30M 88M 26% /boot /dev/mapper/tomcat_vg-horizon 9.8G 5.4G 3.9G 59% /opt/vmware/horizon /dev/mapper/db_vg-db 9.8G 880M 8.4G 10% /db tmpfs 597M 0 597M 0% /run/user/1000 tmpfs 597M 0 597M 0% /run/user/107
We can see that /opt/vmware/horizon only has 3.9 GB of available space on a 9.8GB volume.
Solution -- Part 1
The failed Precheck provides a recommendation to VMware KB article How to zip and remove old vIDM logs in vIDM appliance (81220). This article provides a script that can be download and explains that the script, when run, will compress the vIDM logs under the /opt/vmware/horizon mount,
The steps are clearly detailed in the article. Download the script to the vIDM appliance. Change permissions on the script so it can be executed, and run.
root@vidm [ ~ ]# ./81220_checkFreeSpace.sh
Current /opt/vmware/horizon volume free space is 3.9G
Trying to zip old log files
…
…
Zipped old logs successfully
Removing old Log files
Removed old Log files
echo checking space on volume /opt/vmware/horizon post zipping
/opt/vmware/horizon has 4.6G free space post old log compression and deletion.
SUCCESS
When the script completes, it will state the current free space. In my case above, 4.6 GB.
Solution -- Part 2
As you can see from above, the script did not free the required 7 GB required to upgrade vIDM to 3.3.5. So a further cleanup was required on /opt/vmware/horizon on the vIDM appliance.
Running the command du -a /opt/vmware/horizon/ | sort -n -r | head -n 10
I found that a large portion of space was taken up by RPM packages under /opt/vmware/horizon/26160/package-pool.
Disclaimer: I have not been able to find internally to VMware or externally that the next steps are supported or recommended to free up space on vIDM. So perform at your own risk.
I changed to the /opt/vmware/horizon/26160/package-pool directory. Many of the RPM packages in this directory appeared to reference version 3.3.4. Working on the assumption that these are the extracted packages from the previous upgrade of vIDM. I proceeded to snapshot vIDM in vCenter and then remove all these packages with the command rm *rpm. This now gave me 7.8 GB of available storage, which is enough to meet the 7 GB required from LCM to perform an upgrade.
Conclusion
If sufficient storage is not available for an vIDM upgrade a manual clean up is required on the appliance. VMware provides a script to clean up log files though this might not produce enough free space. A further clean up can be performed by deleted old RPM binaries used from previous upgrades. Performing both of these clean up steps should produce sufficient free space on the appliance.
At this point you should be able to initiate another upgrade and run a Precheck. This time it should detect sufficient free space and pass the check allowing the upgrade to proceed.