Orchestrator on Neale's Corner

Enabling SSH on ESXi with vRO

Mon, 19 Aug 2019 00:00:00 +0000

I think it is a good rule of thumb that the security maturity of the organisation can be measured by whether SSH is left enabled (with warnings muted) or disabled on ESXi Hosts. As there is a warning on the host if the service is enabled, and the hardening guides recommend that it is disabled, there is little doubt that this disabled state is VMware’s preference and a security best practice.

So how do we enable SSH on ESXi ? I am assuming that the only accessible interface is the Web Client, as access to the console is likely restricted too - perhaps the hosts are in lockdown mode ? We have a few options..

Web Client

This is a little bit convoluted. We need to follow the steps:

Select the host on which to enable/disable SSH.
Go to the Configure tab.
Select the Security Profile menu option.
*Optional: Minimise one or both Firewall sections (Incoming Connections and Outgoing Connections).
*Optional: Scroll down to the Services section.
Press the Edit button in the Services section.
Select the SSH entry in the list.
Press Start or Stop.

The * noted items are for vCenter 6.5 or earlier. In vCenter 6.7 the Services panel is moved to the top of the Security Profile section.

The menu to manipulate the service looks as below:

In summary, this is quite a bit of work to enable the service on only one host. Next we’ll look at how to do the same on the HTML5 Client.

HTML5 Client

The steps required to enable/disable SSH on a host using the HTML5 client are as follows:

Select the host on which to enable/disable SSH.
Go to the Configure tab.
Select the Services menu option.
Select the SSH entry in the list.
Press the Start button

There are fewer steps in the list compared to the Flex client, however this is still onerous when having to work with multiple hosts. The section for changing service state in this client looks as follows:

vRealize Orchestrator

Ah, so we’ve got to the real reason of why I’m writing this post ! With past experience I’ve needed to use SSH to:

troubleshoot on a single host
perform maintenance/change on every host in the cluster.
disable SSH on host as someone else left it enabled and the timeout value hasn’t been set properly.

This identifies that we need workflows to enable and disable the service for a host, and for every host in a cluster.

However, thinking about this, if a host typically has SSH disabled then the shell/ssh enabled warning will not be suppressed - so you will know whether SSH is enabled or not. So, we will also have a workflow that toggles the service between enabled/disabled on a host - purely as a way of presenting fewer options to the end user so the vRealize Orchestrator submenu is kept as trim as necessary :)

Our approach to the cluster can re-use some of this - in a best case scenario the service is in the opposite of the desired state for every host in the cluster - i.e. if we want to enable, then the state of every host’s service is disabled. However we must also account for the possibility that the service is enabled on some hosts and disabled on others. Conversely, if the SSH service has been left enabled on a few hosts in a big cluster and we want to disable them, it could be easier to do in a single cluster workflow rather than running a workflow per host. A toggle workflow will not work here - we want to explicitly set the state of the service for all hosts in the cluster - whether enabled or disabled - otherwise we could end up reversing the state. From the point of the user - they know what end state they want, let’s get them there !

This leaves us with one last architecture decision between:

A workflow for enable and a workflow for disable.
A single workflow with a boolean parameter that defines enable/disable.

Honestly, I don’t think there is much between the two, both are good and reasonable choices. I decided to go with the top option for the reasons:

The workflow can be invoked remotely with less parameters.
Flex WebClient integration will allow the workflow to be immediately run with less clicks.

Building the workflow

The following code block will enumerate the services available on the host, and if the SSH service is not running, start it.

var services = host.configManager.serviceSystem.serviceInfo.service;

for each (var svc in services)
{
	if (svc.key == "TSM-SSH")
	{
		var current_state = (svc.running == true) ? true : false;	
		
		if (current_state == false)
		{
			host.configManager.serviceSystem.startService("TSM-SSH");
		}	
		
		break;
	}
}

Conversely, to stop the service, most the above is identical, except for the innermost conditional block:

		if (current_state == true)
		{
			host.configManager.serviceSystem.stopService("TSM-SSH");
		}

Our toggle code can be written as:

		if (current_state == false)
		{
			host.configManager.serviceSystem.startService("TSM-SSH");
		}
		else if (current_state == true)
		{
			host.configManager.serviceSystem.stopService("TSM-SSH");
		}		
		
		break;

The workflows for the cluster follow a typical pattern:

Get all the hosts in the cluster
Call the appropriate workflow for each host

The workflow schema looks as follows:

As can be seen, we use a built-in action to get all the hosts in a cluster, and the builtin workflow loop object to run a workflow for every host in the array.

In the workflows I also check the connection state of the host - there is no point trying to change the service state on a host that is disconnected !

Download

If your browser renames the file, you are free to change the file extension to “.package” (the import may still work if you do not). The file is an exported vRO package (from version 7.5) and is ready for import onto your server.

Click this link to download the package.

Managing a DRS group with vRealize Orchestrator

Mon, 25 Mar 2019 00:00:00 +0000

The use case for this challenge was a small (8-10 ESXi hosts) cluster, that only a subset of servers were ‘assigned’ with Windows datacenter licenses - so in order to maintain compliance all the Windows VMs would need to run on those hosts. The natural choice for ensuring this happens is a “must run” DRS rule between a group of VMs and a group of hosts and the automation step for this is ensuring that the VM group is accurate by only containing VMs that are still in the cluster and contains all Windows VMs in the cluster. Thus, the plan is use vRealize Orchestrator to manage the DRS group.

This technique (and workflow) will be left open enough for it to be used with any type of Operating System, after all I believe Red Hat also license on a per-hypervisor basis.

Core

The schema of the workflow is pictured below.

The flow is one of the simplest we can make ! The steps are:

Retrieve the configuration variables from the ConfigurationElement.
Get the VMs from the cluster (exit the workflow if it has none).
Get the DRS Group from the name (exit if it does not exist).
Loop through the VMs and add/remove them to the DRS group as necessary.
Send an email report if changes are made.

It has a single input of a ConfigurationElement, with the following variables. The description should make it apparent what each variable is used for.

The magic of the workflow is the updating of the DRS rule, which the below script covers:

Where next

As it stands now, we can schedule the workflow to run at the end of the day and allow DRS to place the VMs correctly.

However, to really ensure that the VM is running on the correct host we would invoke this workflow on every creation and addition to the cluster. This is possible when using LogInsight - configured as a destination for the ESXi hosts’ syslogs - to call Orchestrator via a Webhook when the appropriate events are received.

Now that we can dynamically change a DRS rule, this is starting to open up possibilities. Stay tuned :)

Summary

You can download the workflow package using the button below. Please let me know if you found this useful !

Download package file

Updating a HP iLO using Rest API and vRealize Orchestrator

Mon, 18 Mar 2019 00:00:00 +0000

A security vulnerability against Hewlett Packard iLOs was announced last year (link - https://www.rapid7.com/db/modules/auxiliary/admin/hp/hp_ilo_create_admin_account), along with the required firmware to patch. No doubt there are some amazing tools by HP that will allow firmware to be patched across an estate, but as I don’t have access to them I need to find another way to get firmware out to 20+ servers.

There is an API (documentation at https://hewlettpackard.github.io/ilo-rest-api-docs/ilo4/) which is quite straight forward to use, so we’ll have a look at using Postman to update an iLO and then use vRealize Orchestrator to implement a workflow.

Using Postman

Throughout this I will be using the headers below, though the second one is only required when updating the firmware.

To start, we’ll ask the iLO for a Manager interface, which will provide links to the services it offers. Authentication is handled by the headers on a per-request basis. We won’t need to manage session or tokens :)

The initial request is a GET to https://<FQDN/IP>/redfish/v1/managers/ as below:

The data returned contains an array with the name “Members” that will point to the URL for the manager for the iLO. We’re going to use that to get the details for the manager - there is a lot of data here so I’m not going to post it all. There are two points of interest for us - firstly the current firmware version:

Secondly the links to services - we are interested in the UpdateService.

We will use the location provided to access the service - which will allow the current firmware status to be retrieved and an update to be triggered.

The documentation has some reference on what to do - https://hewlettpackard.github.io/ilo-rest-api-docs/ilo4/#links-updateuri-extref-and-links-updateuri-extref - which is pictured below.

However, other documentation at http://h22208.www2.hpe.com/eginfolib/servers/docs/HPRestfultool/iLo4/data_model_reference.html#HpiLOFirmwareUpdate details the following:

This explains we’ll need to include an “Action” parameter in our request. To update the iLO we will need to host the firmware on a webserver that is accessible from the iLO address - this blog entry isn’t going to cover how to do this (I’m assuming that you’ve got a webserver to hand and you’re allowing the extension/content-type to be downloaded. Remember to test it manually before trying to automate, especially if your environment is subject to network controls)

Our POST request body will look as follows:

The iLO will restart when complete - there is no need to manually trigger this, though it is generally a good idea to ensure it has completed. To do this, we will call the GET request to the UpdateService/ URL and check the “State” value - it will change to the following values:

UPDATING
COMPLETED
(The request will fail when the iLO automatically restarts)
IDLE

So if we get a value of COMPLETED or IDLE then we can check the iLO version. However, given that a value of COMPLETED will follow with a period of silence it is more prudent to wait for the IDLE - which I believe we will also receive in the result of failure.

Using vRealize Orchestrator

Now to find a way to put the above into Orchestrator - the workflow to upgrade an iLO is:

The workflow has the following inputs:

The check version value will, if set to true, retrieve the version from the iLO and compare it to the values in the new_major and new_minor fields. If the firmware version on the iLO is newer then the firmware installation will not proceed. This check is dependent on human accuracy as there is no way of reliably retrieving the firmware version from the file, thus if the incorrect version is entered then installation will fail.

The heart of using vRO to drive this process is dynamically creating REST hosts in code, sending the requests and dealing with the response. Taking a look in the vRO workflow “Invoke a dynamic REST operation” we can see how VMware have done it:

The important things to note are:

We need to create a REST Host object, and then use it to create a transient host.
We need to specify the base url and authentication on the transient host object.

However if we invoke a REST call over HTTPS without importing the certificate it will fail, so we will need to handle this failure by importing the certificate and re-running the REST invocation.

This is handled by the Import Cert ?/Import a certificate/Log Cert Import elements. This is done by catching the exception that is thrown.

We now need to can connect to the manager address and find out the location of the UpdateService service - which we did earlier in Postman. Doing this in some javascript looks like the following:

The main part of this workflow is upgrading the firmware - we need to POST the update:

And then loop until we reach the IDLE state. When we exit the loop we can check if the version is different from before we started the process. This will give us our success/failure result to return to the user. The loop is as below:

Wrapping it Up !

Updating multiple iLOs

This is quite straight forward - there are two approaches.

The first one will duplicate the workflow and then adjust the input to take an array of addresses (This does however mean that the credentials to access the iLO must be the same). The schema will be changed to loop through the addresses.

The second approach is the one I favour - we create a new workflow which has duplicate inputs, with the exception of an addition for an array of addresses. The workflow loops through the addresses and calls our original workflow with the other inputs. This method means that we only have one workflow which contains the code that upgrades the iLO, so there is less to manage/maintain.

The workflow in this case will look like this:

with the following inputs:

Updating an ESXi Cluster using a custom attribute

To take this a step further, each ESXi host has a custom attribute with the address of its iLO (work Host so address is blanked):

A workflow is created that will get all the hosts in the cluster, and for each host retrieve the custom attribute. If a value exists, then the original workflow is called to upgrade the firmware. This workflow will only upgrade one physical server at a time, even though there is (or should not be) any impact on the running workloads. It’s just less risky, and isn’t that what our job is really about ? ;)

The challenge here will be ensuring that hosts have a completed (and valid) address for the iLO.

vSphere Client Integration

Still using the Flash Client because it allows you to call vRealize Orchestrator workflows ? Yep, me too. We can export the workflows we’ve created here to make them available on the context menu for hosts and clusters - thus making the upgrade even easier !

Further Automation / Improvements

Taking this further would be to wrap this in an interface (custom HTTP, ansible, etc) that allows end users to provide the firmware file and the host(s) or cluster(s) to upgrade.

This is beyond the scope of this post, but could be provisioned through your DevOps platform of choice. Let me know your thoughts on this one !

Summary

The workflow(s) and associated actions/etc are all available to download via the button below.

Download package file

I hope you’ve found this useful, please let me know your feedback and whether it helped. Thanks for reading :)

Managing Snapshots with vRO

Mon, 25 Feb 2019 00:00:00 +0000

One of the main issues that face departments on their virtualisation journey is how they are managing snapshots:

Who can take them
Who can delete them
How long do they stick around for
Name/Description formatting policy

I consider them ’low hanging fruit’ on the management tree - how an organisation approaches this is defined by policy and there shouldn’t be anything too contentious involved. There are also options to have the snapshots deleted automatically, with most management software providing the function. If you don’t have this, then a vSphere alarm can be created to warn on snapshots sizes.

There was no option to automatically remove snapshots that was in accordance with the policy that was agreed at a previous role, so it was time to automate something :). One of the items revolved around communication to the snapshot owner - it is very important to inform them that the snapshot will be automatically removed and when it has been. Another item was where some snapshots that might be automatically cleared up would need to be held for longer - perhaps as a part of a Root Cause Analysis investigation.

Requirements

From the above, the requirements are this:

Inform the user who took the snapshot that it will be deleted in X days, daily until deleted. (X should be configurable).
Inform the user that the snapshot has been deleted.
Report to a team/email address details on all snapshots present and estimate the space use.
Exclude Virtual Machines from having their snapshots automatically deleted.

These will mean that:

The workflow will always produce a report. Therefore our input values should include an email address.
A means of identifying a Virtual Machine as being excluded. We could supply a list, or a naming format to match, a tag or a custom attribute.
Given that we can retrieve the username of the person who created the snapshot (it is contained within the event), we still need a means of identifying the email address of the user.

My preference for the second item above is to use a tag as we can retrieve VMs easier from a management perspective.

The identity source for the vCenter is assumed to be Active Directory, thus we would have the further assumption that no local accounts are allowed (or only in exceptional circumstances) and will be taking snapshots. We can therefore use the Active Directory plugin to query for the user and retrieve the email address. However we should also handle the edge case that the user does not have an email address set, or it is not valid - therefore we will issue warning emails to an appropriate team that the end user was not/cannot be warned of the snapshot expiry.

In summary, our input values are:

# Days when start to warn end users of deletion.
# Days when the snapshot should be deleted.
Email address in case of errors.
Email address for the report.
Tag name for VMs to be excluded from snapshot management.
A ‘WhatIf’ to control whether action is taken.

Our vRO server will also need to have the Active Directory plugin active and configured.

We also need to deal with another assumption - there can be more than one snapshot on a VM, and multiple snapshots may need to be deleted on each day.

\[Hindsight\]

After creating the workflows I added another couple of inputs:

Minimum age of VM snapshot to appear on the report.
Subject of the report.

Workflows

I write vRO workflows in a decentralised manner - for example if I am to do something with a Virtual Machine I have a workflow for:

The Virtual Machine
The Cluster
(Sometimes) The Datacenter
vCenter Server

This can add up to a lot of workflows, but it allows for testing and iterating changes easily. I will often put a “WhatIf” input on the Virtual Machine workflow so that I can test the workflows that function at a higher level - this ties in with the identified inputs earlier. So in this case, the workflow design mirrors the above (but without a datacenter script).

Additionally, when communicating to people the details of a VM with a snapshot I include Custom Attributes and Tags. So the workflows below will contain some tagging actions that I really need to release ;)

Virtual Machine

This workflow will need to do the following:

Get the snapshot information for a VM.
Calculate snapshot size (if possible).
Get the tags for a VM.
Add information to the report.
Check the tag to see if the VM will have it’s snapshots automatically managed.

If the VM is to be automatically managed, it will need to do:

If the snapshot is older than the warning age, send a warning to the snapshot creator, or the errors address.
If the snapshot is older than the deletion age, send a notification to the snapshot creator, or the errors address.

I’ve separate the above into two lists as I think the logic flows naturally in this way - the first list will apply to every VM that we examine whereas the second will only apply to those VMs with snapshots that we will manage. Thus we can break the logic up into two workflows.

Disclaimer: I’m assuming some familiarity with Orchestrator here, and there are plenty of other awesome blogs introducing the tool, so I’m going to dive in to some completed workflows.

The schema for the first flow is below:

Some notable steps:

Excluded from snaps - This is where the tag is checked.
Automatic Management - This is where the WhatIf is handled.
Manage Snapshots - We call the workflow to manage them. This sits inside a loop that will examine all the snapshots on a VM.

Some scripts are included to help with generating the report :)

The schema for Managing Snapshots is here:

The notable steps for this workflow are:

getEmailAddressForUser - this will query Active Directory for the mail attribute for the user identified earlier.
Remove Snapshot - deletes the snapshot and sets a task object so the workflows/entire process operates synchronously.
Customise Email - these write the warning and notification emails to be sent to the end user.

Cluster Workflow

This workflow retrieves all the VMs in the cluster and runs the main VM workflow against it. Nothing special here :)

Clusters / vCenter Servers Workflow

This is fairly standard workflow that goes through all the clusters on a vCenter Server and runs the Cluster workflow. It follows a standard pattern of retrieving a set of items and iterating through them. The end of the schema is devoted to sending the report email.

Configuration Element

As I wrote this a while ago, I put all the configuration inside a Configuration Element so that configuration variables could be updated once a workflow was scheduled. Please excuse my naming format, it does the job but the names made sense at the time I promise !

These names match the inputs that the workflows take:

Emails

Here is an example of the report email running on a live environment:

An example of the email notification a user receives about their snapshot (some field values have been removed):

Download

The workflows, actions and configuration element is available to download here. The package includes the following workflows

Once the package is imported, run one of the workflows and let me know what you think ! You’ll be able to run the cluster and VM workflows without having to configure a Configuration Element.

Download package file

Improvements

The most obvious improvement to this workflow would be to made the notifications optional and controlled via an input. Maybe something for a rainy day ;)

Let me know if this was useful or interesting to you !

vRealize Orchestrator: Finding a REST Host from a vCenter SDK connection

Tue, 13 Mar 2018 00:00:00 +0000

vRealize Orchestrator can communicate directly with the vCenter Server(s) that it is connected to, but uses the old SOAP API. However, with the advent of the REST API for vSphere, some things are easier (for example, tagging) and only possible to handle outside of the builtin abilities. In this post I will look at how to find a REST Host from a vCenter SDK connection in vRealize Orchestrator - this can be used to reduce the number of inputs on a workflow or make it easier to create multiple-vCenter workflows.

Each vSphere object has an “sdkConnection” property that refers to the vCenter Server. This is useful, but if you wish to use the REST API, we’ll need to provide a RestHost object type. We could provide that as an input to the workflow, but this is not dynamic and we shouldn’t force unnecessary parameters on users.

There must be an easier way, right ?

We should assume that every vCenter Server that is connected to vRealize Orchestrator is also configured as a Rest Host. This configuration only needs to be done once, though we will look at automating this at the end of the article.

Thus, we can enumerate the REST Hosts and find the one we’re after by comparing the URL of the vCenter API endpoint to the vCenter REST API endpoint. There is a catch in that the URLs are different so we will need to do a little string manipulation to make it work.

For example:

The vCenter Server is https://auto-vc.auto.internal:443/sdk
The REST Host is https://auto-vc.auto.internal/rest

I believe it is a fair assumption that the protocol will always be https for both, so we can ignore that - to match up we would need to ensure that the vCenter Server has been added with either the FQDN or hostname identically in both locations - if not then the lookup will fail. Personally I don’t see anything wrong with always using the FQDN :)

Adding a REST Host

I have no doubt that this has been done many times before on many other blogs, but for the sake of completeness I will document how I’ve added the REST host below.

This next step is me being lazy - I’m going to pass in the SSO administrator credentials. This is not best or even good practice - you should pass in user credentials or a role-limited vco-specific account.

I haven’t included screenshots of the (3a) and (4a) wizard pages - I have accepted the defaults. Change if you need to !

Comparing Values

Our code will need to strip the protocol and the port (if either of them exist) from an URL, returning the hostname. We’ll use this on both the vCenter URL and every REST Host URL to see if they match. Since this is a straight forward bit of a code, we’ll put it in an action.

The main configuration of the action is below:

As seen above, the action assumes an input named “vcenter” of type VC:SdkConnection and returns an output of type REST:RESTHost.

// Trim out protocol and match url up to the : or "/".
function gethostname(url) {
	var hostname = url.substring(url.indexOf("://") + 3);
	var port = hostname.indexOf(":");
	var slash = hostname.indexOf("/");
	
	var end = -1;
	if (port != -1 && slash != -1) {
		end = (port < slash) ? port : slash;
	
	} else if (port == -1 && slash != -1) {
		end = slash;
	
	} else if (port != -1 && slash == -1) {
		end = port;
		
	}
	
	hostname = hostname.substring(0, end);
	return hostname;
}

var vcenter_hostname = gethostname(vcenter.name);

var hosts = RESTHostManager.getHosts();
for each (var current_host in hosts)
{
	var rest_host = RESTHostManager.getHost(current_host);	
	var rest_hostname = gethostname(rest_host.url);
	if (vcenter_hostname == rest_hostname) {
		host = rest_host;
		break;
	}
}

The “gethostname” function is responsible for stripping the URL down to the hostname. We use the RESTHostManager object to loop through configuration hosts and return the host if there is a match.

Now when creating workflows that would require a REST Host object, we can retrieve the correct object from the sdkConnection property of the managed object, thereby removing the need to add another input to the workflow, and if you have a workflow against multiple vCenters this should really make life easier !

Next Steps

We will take this on to do some vSphere tagging using the REST API :)

Summary

I’ve created an action for this and saved it in my github - the unencrypted action file can be found here.

Let me know your thoughts in the comments :)

vRealise Log Insight and Orchestrator

Fri, 15 Sep 2017 00:00:00 +0000

I’m a big fan of the vRealize suite as it really adds value to a vSphere deployment. In relation to this post, LogInsight is a great tool for providing log analysis and Orchestrator manages workflows to automate and orchestrate most/all of the infrastructure.

A series of postsÂ on vmware.com - “Self Healing Datacenter” - have examined how to connect Operations Manager to Orchestrator using a shim - the first part of the series is here.Â The shim is also capable of taking input from LogInsight, though this isn’t covered by the articles.

That webhooks are offered for LogInsight is great, but it is sad that there is no official tool to send events/data from one product in the vRealize suite to another. I’m interested in how and why things work (and also because it doesn’t appear to work quite right), so instead of using the shim presented in the series above I’m going to write my own in node.js so we can look at how this all works.

Environment

In this environment we’ll be using vRealize LogInsight 4.3 and Orchestrator 7.2, both which have a straight forward and simple deployment - both are single servers (no clusters !) and both are configured to a vSphere 6.5 deployment.

In vSphere we create a nested host to test against, and have put it in its own cluster. I’ve done this because some of the actions I want to take involve shutting the host down (not always gracefully) and I don’t have the spare physical hosts to do this (e.g. the auto-esxi VMs below are the hosts in the Nested cluster).

LogInsight has been set up with vSphere integration - all hosts (including the nested host) have been configured to send syslogs to the LogInsight address. A look at the “Hosts” page in the LogInsight Administration section shows that logs are being successfully received:

Integration

It isn’t actually possible for LogInsight to start Orchestrator workflows, the closest we can come is by using the “webhook” functionality when setting up an alert - this will make an HTTP request to an URL with some information when something happens (below is an example of our LogInsight query for a host going into maintenance mode).

It is because the vRealize Orchestrator API requires some additional parameters that we can’t start Workflows, and when we look at the schema on the swagger API reference page (https://

:8281/vco/api/docs/index.html - where

is the FQDN/IP of your Orchestrator server), we can see what is required:

WebHook Shims

The shim referenced in the “Self Healing Datacenter” series can be found as the “log insight webhook-shims” on githubÂ (this also works with vROPs). To test this I deployed a Photon OS instance and installed it via docker:

systemctl enable docker
systemctl start docker
docker pull vmware/webhook-shims
docker run -it -p 5001:5001 –name lishims vmware/webhook-shims bash

The public page for this docker container shows how to set it up -Â https://hub.docker.com/r/vmware/webhook-shims/ - and after this we need to configure the vRealize Orchestrator connection in the container. At the command line, enter the following:

vi loginsightwebhookdemo/vrealizeorchestrator.py:

When this file is saved we can start the server within the container:

./runserver.py

As per the INFO message above, if we browse to http://

:5001 (where

is the FQDN/IP of the server running the shim) we get a guidance screen on how to use the software - scroll down to view the advice on how to integrate with vRealize Orchestrator.

Now we can return to our LogInsight query and set up an alert based upon it. Select the Alert button on the UI and the following window will appear:

Select the Webhook notify option and put in the formatted address of the webhook server:

http://address : port / endpoint / vro / workflow-id

Aha, we don’t have a workflow set up so we don’t know what worfklow-id to type in here - let’s do this now. Open up the vRealize Orchestrator client and create a new workflow - we’re going to keep this one simple and just write something in the log, the schema for which is below:

When we look at the details of the workflow we see the ID to use the alert:

So now we have a workflow id we can create our LogInsight alert. Go back to the LogInsight alert configuration page, update the webhook and URL and save.

Workflow problems

Time to put a host (that is configured to send syslog to our LogInsight server) into Maintenance Mode - which I did I 0753 in the screenshot below:

We see the following workflow implementations:

Woah - we only put the host into maintenance mode once and the workflow was started 10 times - what could have happened ? The sequence of what happens is:

LogInsight matches the query and notifies shim server via webhook.
Shim server receives request and formats a vRealize Orchestrator API call. Sends to vRealize Orchestrator
vRealize Orchestrator starts the workflow.

If you were watching the console of the Photon OS container then you would have seen that for every workflow invocation a new request was sent by the webhook shim code. So this excludes vRealize Orchestrator from causing this, but is the shim code receiving a new notification for every workflow ?

I did toy with the idea of setting up port mirroring via a distributed switch and using Wireshark to check only one request was sent, but this seems unnecessarily complex. Â The is another way :)

Node.js

We can spin up a very simple node.js server that logs out every request that is made to it. Our server code looks as follows:

// include the http module
var http = require('http');

// create a webserver
http.createServer(function (req, res) {

    // respond to any incoming http request
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end();
    console.log("Request Received.\n");

}).listen(5001, '127.0.0.1');

// log what that we started listening 
console.log('Server running at 127.0.0.1:5001');

When we put a host into maintenance mode now we can see that we only receive one email and that our notification monitor only receives one request from Log Insight. So the problem is somewhere in the shim server. But in the spirit of creating stuff and learning something new, let’s write our own shim server !

The main body of our server function is:

var responseCode = 500;

// Get Body	
var request_body_array = new Array();
req.on('data', function(chunk) {
	request_body_array.push(chunk);
});
	
req.on('end', function() {
	var request_body = "";
	if (request_body_array.length > 0) {
		request_body = request_body_array.join('');
	}

	responseCode = handle_request(req.url, request_body);
	res.writeHead(responseCode, {'Content-Type': 'text/plain'});
	res.end();
});

This is really straight forward and will receive the full request and call handle_request (which I’m not going to detail Â here), which will then invoke the call_vro function with appropriate parameters.

In our call_vro function we will need to set up the parameters for the Orchestrator call:

var creds_base64 = new Buffer(vro_username + ":" + vro_password).toString('base64');
var messages_value = new Buffer(request_body).toString('base64');

var request_body_json = JSON.parse(request_body);

var vro_execution_context = {
	parameters: [
		{
			name: 'messages',
			description: 'Base64 encoded JSON of LogInsight messages',
			scope: 'local',
			type: 'string',
			value: {
				string: {
					value: messages_value
				}
			}
		},
		{
			name: 'alertName',
			scope: 'local',
			type: 'string',
			value: {
				string: {
					value: request_body_json.AlertName
				}
			}
		},
		{
			name: 'hitCount',
			scope: 'local',
			type: 'number',
			value: {
				number: {
					value: request_body_json.HitCount
				}
			}
		}
	]
};

var workflow_request_body = JSON.stringify(vro_execution_context);

The code above sets up the basic authentication string and defines the parameter arrayÂ for the request. We are going to provide three parameters for the workflow:

messages - the messages passed by LogInsight. These will be Base64 encoded so Orchestrator will need to decode them.
alertName - the name of the alert that was configured in LogInsight.
hitCount - the number of times this alert was triggered.

As our alert should be triggered on every invocation we won’t worry about hitCount for now - but we should make a note to test this later.

After the parameters have been set up, they are convered to a JSON string for the request. We can see below in the request header set up that we are declaring our content is JSON.

var vro_auth_options = {
	host: vro_server,
	port: vro_port,
	path: '/vco/api/workflows/' + workflow_id + '/executions',
	method: 'POST',
	rejectUnauthorized: false,
	headers:  {
		'Accept': 'application/json',
		'Content-Type': 'application/json;v=5.1.1',
		'Content-Length': workflow_request_body.length,
		'Authorization': 'Basic ' + creds_base64
	}
};

var auth_data_array = new Array();
var callback = function(response) {
	response.on('data', function(chunk) {
		auth_data_array.push(chunk);
	});

	response.on('error', function(chunk) {
		console.log("ERROR = " + err.message);
	});

	response.on('end', function() {
		var auth_data = auth_data_array.join('');

		if (response.statusCode == 202) {
			if (auth_data.length > 0) {				
				var res = JSON.parse(auth_data);
			}

			console.log("Workflow " + workflow_id + " successfully started");

		} else {
			console.log("Workflow execution response: " + response.statusCode);
		}
	});
}

const req = https.request(vro_auth_options, callback);
req.write(workflow_request_body);
req.end();

This code sets up the connection, defines the functions for what happens when the request errors or is complete and then makes the connection (https.request at the bottom). If we receive an error code of 202 then all is good !

Mental note: current status codes for starting a workflow:

Trying it out

When we put a host into maintenance mode now we can see that we only get one Orchestrator workflow generated for each alert. But if we look at Log Insight, the alert is examined every 5 minutes:

So what happens if we put two hosts into maintenance mode before the alert is examined ?

This is the console of the server:

Looking at the request that has been passed we can see that there are messages for both hosts, but we only have one workflow started:

We need to look at how we pull out the information contained in the ‘messages’ input variable. First, we need to download and install a plugin that will allow us to decode base 64 strings. This can be obtained from Â https://github.com/vmware/o11n-plugin-crypto and installation is straight forward from the Control Center. After installation, the vCO service needs to be restarted (not the server) - the message stating the service needs to be restarted doesn’t disappear immediately, but it does go :)

Then we need to modify our workflow to accept these parameters:

I’ve also added an attribute to the workflow with the name “hosts” and type “Array/string” - but you can omit this and comment out the lines that reference this attribute. We can now add a scriptable task with the following:

// Decode the string and parse into a JSON objecct.
var decoded_str = CryptoEncoding.base64Decode(messages);
var li_obj = JSON.parse(decoded_str);

// Create the array to hold our hosts detail.
hosts = new Array();
for each (var item in li_obj.messages)
{
	// Get the Host name
	var hostname = "";
	
	for each (var field in item.fields)
	{
		if (field.name == "__li_source_path")
		{
			hostname = field.content;
			break;
		}
	}
	
	// Debugging: Write out the host name.
	System.log(hostname + " has entered maintenance mode.");
	
	// Add to the array.
	hosts.push(hostname);
}

This is what we see now in the Logs section for the workflow when we put two hosts into maintenance mode for the same alert interval - we can now handle more than one host, and identify which host has gone into maintenance mode.

Summary

So we can now decode the messages we receive from LogInsight and work with multiple messages - I’ve left the node.js console printing out the contents of the request (line 34) in order to help identify useful fields of information. This needs some tidying up to be useful (and a solution for storing username/passwords in the node.js file should be found) but hopefully is a good proof of concept of what is capable.

As usual, the code is available on github - https://github.com/nelons/loginsight_webhook_shim. Feel free to use/abuse in any way you like, let me know if I’ve made a mistake or if you found this useful.

Using Orchestrator to automatically upgrade Tools

Wed, 08 Mar 2017 00:00:00 +0000

One of the main challenges in every (non-small) place I’ve worked is keeping VMware Tools updated, and with Change Management processes this becomes a little bit more complicated. There is a setting on each VM to upgrade Tools when the VM is booted and this is a good way to keep Tools updated on those VMs that aren’t mission critical and require manual intervention.

In this post we’ll look at creating an action and a workflow to configure this setting, and pushing it as a context setting in the Flex Web Client.

This leads to the issue of how this setting is enabled - firstly as part of the initial push (you don’t want to enable this manually for 100’s (or more) VMs) and with newly created VMs. Although this can be set on a VM template or “Gold Build” (or equivalent terminology), this can still lead to human error so we’ll also look creating a workflow to set this for a cluster and scheduling this using the Flex Web Client.

Upgrade Tools Action

To configure a VM to upgrade Tools we need to create a new configuration specification and set the tools configuration appropriately. Our actionÂ will return the Task for the configuration (note: this can be null) and will take two inputs - the VM to configure and whether we are automatically upgrading Tools. Here is a screenshot of the action configuration:

The script of the action is as follows:

task = null;

if (vm != null)
{
	var toolsconfig = new VcToolsConfigInfo();
	
	if (autoupgrade == true) {
		toolsconfig.toolsUpgradePolicy = "upgradeAtPowerCycle";
	} else {
		toolsconfig.toolsUpgradePolicy = "manual";
	}
	
	var vmconfig = new VcVirtualMachineConfigSpec();
	vmconfig.tools = toolsconfig;

	task = vm.reconfigVM_Task(vmconfig);
}

return task;

In the code snippet above, we ensure that the “vm” variable contains a valid VM. We create a new ToolsConfigInfo object which will contain the tools settings that we wish to apply and then create a new VirtualMachineConfigSpec object and assign the ToolsConfigInfo object to it. Thus when we call the reconfigVM_Task() function with the configuration object, the VM is only reconfigured with the settings we have provided.

Now we have an action, we can create a workflow that will invoke it and then wait for the task to complete. This is the schema diagram:

The workflow (and action) can be downloadedÂ from the file at the end of this post.

Cluster Workflow

Now we have an action to make the changes, we can put together a workflow that will look at all VMs in the cluster and configure them. The schema is as follows:

The workflow will get all VMs in the cluster and for each one:

Only upgrade Windows VMs (“Upgrade VM ?” decision)
Check if the configuration setting currently matches what we want it to. (“Decide setting” script).
Change the setting.

The meat of the process is in the “Decide setting” script item - we will look at what the setting should be, and if the current setting does not match then we will change it. This also means that if we wish the VM to update Tools manually and it is set to automatically update at power on, the setting will be changed to update manually.

Setting as a context action

To set this as a context action on the cluster within the Flex Web Client, we need to look at the Orchestrator pages in the Web Client. The server page will have a Manage screen and a Context Actions page:

Next we need to add a new one - select the workflow to add and the type of object to associate it with - in this case we’re adding the workflow to set the upgrade for Tools for a single virtual machine, so we select the Virtual Machine object:

Press OK and all is done. You’ll need to reload the Flex Web Client in order to see it - it appears at the bottom of the actions menu for the object - under the “vRealize Orchestrator context actions” title.

That’s it !Â This workflow a prime candidate for being scheduled to ensure all your VMs are kept up to date.

The workflows (and action) for this are available at my github repo - https://github.com/nelons/vco-content.