A security vulnerability against Hewlett Packard iLOs was announced last year (link – https://www.rapid7.com/db/modules/auxiliary/admin/hp/hp_ilo_create_admin_account), along with the required firmware to patch. No doubt there are some amazing tools by HP that will allow firmware to be patched across an estate, but as I don’t have access to them I need to find another way to get firmware out to 20+ servers.
There is an API (documentation at https://hewlettpackard.github.io/ilo-rest-api-docs/ilo4/) which is quite straight forward to use, so we’ll have a look at using Postman to update an iLO and then use vRealize Orchestrator to implement a workflow.
Throughout this I will be using the headers below, though the second one is only required when updating the firmware.
To start, we’ll ask the iLO for a Manager interface, which will provide links to the services it offers. Authentication is handled by the headers on a per-request basis. We won’t need to manage session or tokens 🙂
The initial request is a GET to https://<FQDN/IP>/redfish/v1/managers/ as below:
The data returned contains an array with the name “Members” that will point to the URL for the manager for the iLO. We’re going to use that to get the details for the manager – there is a lot of data here so I’m not going to post it all. There are two points of interest for us – firstly the current firmware version:
Secondly the links to services – we are interested in the UpdateService.
We will use the location provided to access the service – which will allow the current firmware status to be retrieved and an update to be triggered.
The documentation has some reference on what to do – https://hewlettpackard.github.io/ilo-rest-api-docs/ilo4/#links-updateuri-extref-and-links-updateuri-extref – which is pictured below.
However, other documentation at http://h22208.www2.hpe.com/eginfolib/servers/docs/HPRestfultool/iLo4/data_model_reference.html#HpiLOFirmwareUpdate details the following:
This explains we’ll need to include an “Action” parameter in our request. To update the iLO we will need to host the firmware on a webserver that is accessible from the iLO address – this blog entry isn’t going to cover how to do this (I’m assuming that you’ve got a webserver to hand and you’re allowing the extension/content-type to be downloaded. Remember to test it manually before trying to automate, especially if your environment is subject to network controls)
Our POST request body will look as follows:
The iLO will restart when complete – there is no need to manually trigger this, though it is generally a good idea to ensure it has completed. To do this, we will call the GET request to the UpdateService/ URL and check the “State” value – it will change to the following values:
- (The request will fail when the iLO automatically restarts)
So if we get a value of COMPLETED or IDLE then we can check the iLO version. However, given that a value of COMPLETED will follow with a period of silence it is more prudent to wait for the IDLE – which I believe we will also receive in the result of failure.
Using vRealize Orchestrator
Now to find a way to put the above into Orchestrator – the workflow to upgrade an iLO is:
The workflow has the following inputs:
The check version value will, if set to true, retrieve the version from the iLO and compare it to the values in the new_major and new_minor fields. If the firmware version on the iLO is newer then the firmware installation will not proceed. This check is dependent on human accuracy as there is no way of reliably retrieving the firmware version from the file, thus if the incorrect version is entered then installation will fail.
The heart of using vRO to drive this process is dynamically creating REST hosts in code, sending the requests and dealing with the response. Taking a look in the vRO workflow “Invoke a dynamic REST operation” we can see how VMware have done it:
The important things to note are:
- We need to create a REST Host object, and then use it to create a transient host.
- We need to specify the base url and authentication on the transient host object.
However if we invoke a REST call over HTTPS without importing the certificate it will fail, so we will need to handle this failure by importing the certificate and re-running the REST invocation.
This is handled by the Import Cert ?/Import a certificate/Log Cert Import elements. This is done by catching the exception that is thrown.
The main part of this workflow is upgrading the firmware – we need to POST the update:
And then loop until we reach the IDLE state. When we exit the loop we can check if the version is different from before we started the process. This will give us our success/failure result to return to the user. The loop is as below:
Wrapping it Up !
Updating multiple iLOs
This is quite straight forward – there are two approaches.
The first one will duplicate the workflow and then adjust the input to take an array of addresses (This does however mean that the credentials to access the iLO must be the same). The schema will be changed to loop through the addresses.
The second approach is the one I favour – we create a new workflow which has duplicate inputs, with the exception of an addition for an array of addresses. The workflow loops through the addresses and calls our original workflow with the other inputs. This method means that we only have one workflow which contains the code that upgrades the iLO, so there is less to manage/maintain.
The workflow in this case will look like this:
with the following inputs:
Updating an ESXi Cluster using a custom attribute
To take this a step further, each ESXi host has a custom attribute with the address of its iLO (work Host so address is blanked):
A workflow is created that will get all the hosts in the cluster, and for each host retrieve the custom attribute. If a value exists, then the original workflow is called to upgrade the firmware. This workflow will only upgrade one physical server at a time, even though there is (or should not be) any impact on the running workloads. It’s just less risky, and isn’t that what our job is really about ? 😉
The challenge here will be ensuring that hosts have a completed (and valid) address for the iLO.
vSphere Client Integration
Still using the Flash Client because it allows you to call vRealize Orchestrator workflows ? Yep, me too. We can export the workflows we’ve created here to make them available on the context menu for hosts and clusters – thus making the upgrade even easier !
Further Automation / Improvements
Taking this further would be to wrap this in an interface (custom HTTP, ansible, etc) that allows end users to provide the firmware file and the host(s) or cluster(s) to upgrade.
This is beyond the scope of this post, but could be provisioned through your DevOps platform of choice. Let me know your thoughts on this one !
The workflow(s) and associated actions/etc are all available to download via the button below.
I hope you’ve found this useful, please let me know your feedback and whether it helped. Thanks for reading 🙂