Automation on Neale's Corner

Managing a DRS group with vRealize Orchestrator

Mon, 25 Mar 2019 00:00:00 +0000

The use case for this challenge was a small (8-10 ESXi hosts) cluster, that only a subset of servers were ‘assigned’ with Windows datacenter licenses - so in order to maintain compliance all the Windows VMs would need to run on those hosts. The natural choice for ensuring this happens is a “must run” DRS rule between a group of VMs and a group of hosts and the automation step for this is ensuring that the VM group is accurate by only containing VMs that are still in the cluster and contains all Windows VMs in the cluster. Thus, the plan is use vRealize Orchestrator to manage the DRS group.

This technique (and workflow) will be left open enough for it to be used with any type of Operating System, after all I believe Red Hat also license on a per-hypervisor basis.

Core

The schema of the workflow is pictured below.

The flow is one of the simplest we can make ! The steps are:

Retrieve the configuration variables from the ConfigurationElement.
Get the VMs from the cluster (exit the workflow if it has none).
Get the DRS Group from the name (exit if it does not exist).
Loop through the VMs and add/remove them to the DRS group as necessary.
Send an email report if changes are made.

It has a single input of a ConfigurationElement, with the following variables. The description should make it apparent what each variable is used for.

The magic of the workflow is the updating of the DRS rule, which the below script covers:

Where next

As it stands now, we can schedule the workflow to run at the end of the day and allow DRS to place the VMs correctly.

However, to really ensure that the VM is running on the correct host we would invoke this workflow on every creation and addition to the cluster. This is possible when using LogInsight - configured as a destination for the ESXi hosts’ syslogs - to call Orchestrator via a Webhook when the appropriate events are received.

Now that we can dynamically change a DRS rule, this is starting to open up possibilities. Stay tuned :)

Summary

You can download the workflow package using the button below. Please let me know if you found this useful !

Download package file

Updating a HP iLO using Rest API and vRealize Orchestrator

Mon, 18 Mar 2019 00:00:00 +0000

A security vulnerability against Hewlett Packard iLOs was announced last year (link - https://www.rapid7.com/db/modules/auxiliary/admin/hp/hp_ilo_create_admin_account), along with the required firmware to patch. No doubt there are some amazing tools by HP that will allow firmware to be patched across an estate, but as I don’t have access to them I need to find another way to get firmware out to 20+ servers.

There is an API (documentation at https://hewlettpackard.github.io/ilo-rest-api-docs/ilo4/) which is quite straight forward to use, so we’ll have a look at using Postman to update an iLO and then use vRealize Orchestrator to implement a workflow.

Using Postman

Throughout this I will be using the headers below, though the second one is only required when updating the firmware.

To start, we’ll ask the iLO for a Manager interface, which will provide links to the services it offers. Authentication is handled by the headers on a per-request basis. We won’t need to manage session or tokens :)

The initial request is a GET to https://<FQDN/IP>/redfish/v1/managers/ as below:

The data returned contains an array with the name “Members” that will point to the URL for the manager for the iLO. We’re going to use that to get the details for the manager - there is a lot of data here so I’m not going to post it all. There are two points of interest for us - firstly the current firmware version:

Secondly the links to services - we are interested in the UpdateService.

We will use the location provided to access the service - which will allow the current firmware status to be retrieved and an update to be triggered.

The documentation has some reference on what to do - https://hewlettpackard.github.io/ilo-rest-api-docs/ilo4/#links-updateuri-extref-and-links-updateuri-extref - which is pictured below.

However, other documentation at http://h22208.www2.hpe.com/eginfolib/servers/docs/HPRestfultool/iLo4/data_model_reference.html#HpiLOFirmwareUpdate details the following:

This explains we’ll need to include an “Action” parameter in our request. To update the iLO we will need to host the firmware on a webserver that is accessible from the iLO address - this blog entry isn’t going to cover how to do this (I’m assuming that you’ve got a webserver to hand and you’re allowing the extension/content-type to be downloaded. Remember to test it manually before trying to automate, especially if your environment is subject to network controls)

Our POST request body will look as follows:

The iLO will restart when complete - there is no need to manually trigger this, though it is generally a good idea to ensure it has completed. To do this, we will call the GET request to the UpdateService/ URL and check the “State” value - it will change to the following values:

UPDATING
COMPLETED
(The request will fail when the iLO automatically restarts)
IDLE

So if we get a value of COMPLETED or IDLE then we can check the iLO version. However, given that a value of COMPLETED will follow with a period of silence it is more prudent to wait for the IDLE - which I believe we will also receive in the result of failure.

Using vRealize Orchestrator

Now to find a way to put the above into Orchestrator - the workflow to upgrade an iLO is:

The workflow has the following inputs:

The check version value will, if set to true, retrieve the version from the iLO and compare it to the values in the new_major and new_minor fields. If the firmware version on the iLO is newer then the firmware installation will not proceed. This check is dependent on human accuracy as there is no way of reliably retrieving the firmware version from the file, thus if the incorrect version is entered then installation will fail.

The heart of using vRO to drive this process is dynamically creating REST hosts in code, sending the requests and dealing with the response. Taking a look in the vRO workflow “Invoke a dynamic REST operation” we can see how VMware have done it:

The important things to note are:

We need to create a REST Host object, and then use it to create a transient host.
We need to specify the base url and authentication on the transient host object.

However if we invoke a REST call over HTTPS without importing the certificate it will fail, so we will need to handle this failure by importing the certificate and re-running the REST invocation.

This is handled by the Import Cert ?/Import a certificate/Log Cert Import elements. This is done by catching the exception that is thrown.

We now need to can connect to the manager address and find out the location of the UpdateService service - which we did earlier in Postman. Doing this in some javascript looks like the following:

The main part of this workflow is upgrading the firmware - we need to POST the update:

And then loop until we reach the IDLE state. When we exit the loop we can check if the version is different from before we started the process. This will give us our success/failure result to return to the user. The loop is as below:

Wrapping it Up !

Updating multiple iLOs

This is quite straight forward - there are two approaches.

The first one will duplicate the workflow and then adjust the input to take an array of addresses (This does however mean that the credentials to access the iLO must be the same). The schema will be changed to loop through the addresses.

The second approach is the one I favour - we create a new workflow which has duplicate inputs, with the exception of an addition for an array of addresses. The workflow loops through the addresses and calls our original workflow with the other inputs. This method means that we only have one workflow which contains the code that upgrades the iLO, so there is less to manage/maintain.

The workflow in this case will look like this:

with the following inputs:

Updating an ESXi Cluster using a custom attribute

To take this a step further, each ESXi host has a custom attribute with the address of its iLO (work Host so address is blanked):

A workflow is created that will get all the hosts in the cluster, and for each host retrieve the custom attribute. If a value exists, then the original workflow is called to upgrade the firmware. This workflow will only upgrade one physical server at a time, even though there is (or should not be) any impact on the running workloads. It’s just less risky, and isn’t that what our job is really about ? ;)

The challenge here will be ensuring that hosts have a completed (and valid) address for the iLO.

vSphere Client Integration

Still using the Flash Client because it allows you to call vRealize Orchestrator workflows ? Yep, me too. We can export the workflows we’ve created here to make them available on the context menu for hosts and clusters - thus making the upgrade even easier !

Further Automation / Improvements

Taking this further would be to wrap this in an interface (custom HTTP, ansible, etc) that allows end users to provide the firmware file and the host(s) or cluster(s) to upgrade.

This is beyond the scope of this post, but could be provisioned through your DevOps platform of choice. Let me know your thoughts on this one !

Summary

The workflow(s) and associated actions/etc are all available to download via the button below.

Download package file

I hope you’ve found this useful, please let me know your feedback and whether it helped. Thanks for reading :)

Managing Snapshots with vRO

Mon, 25 Feb 2019 00:00:00 +0000

One of the main issues that face departments on their virtualisation journey is how they are managing snapshots:

Who can take them
Who can delete them
How long do they stick around for
Name/Description formatting policy

I consider them ’low hanging fruit’ on the management tree - how an organisation approaches this is defined by policy and there shouldn’t be anything too contentious involved. There are also options to have the snapshots deleted automatically, with most management software providing the function. If you don’t have this, then a vSphere alarm can be created to warn on snapshots sizes.

There was no option to automatically remove snapshots that was in accordance with the policy that was agreed at a previous role, so it was time to automate something :). One of the items revolved around communication to the snapshot owner - it is very important to inform them that the snapshot will be automatically removed and when it has been. Another item was where some snapshots that might be automatically cleared up would need to be held for longer - perhaps as a part of a Root Cause Analysis investigation.

Requirements

From the above, the requirements are this:

Inform the user who took the snapshot that it will be deleted in X days, daily until deleted. (X should be configurable).
Inform the user that the snapshot has been deleted.
Report to a team/email address details on all snapshots present and estimate the space use.
Exclude Virtual Machines from having their snapshots automatically deleted.

These will mean that:

The workflow will always produce a report. Therefore our input values should include an email address.
A means of identifying a Virtual Machine as being excluded. We could supply a list, or a naming format to match, a tag or a custom attribute.
Given that we can retrieve the username of the person who created the snapshot (it is contained within the event), we still need a means of identifying the email address of the user.

My preference for the second item above is to use a tag as we can retrieve VMs easier from a management perspective.

The identity source for the vCenter is assumed to be Active Directory, thus we would have the further assumption that no local accounts are allowed (or only in exceptional circumstances) and will be taking snapshots. We can therefore use the Active Directory plugin to query for the user and retrieve the email address. However we should also handle the edge case that the user does not have an email address set, or it is not valid - therefore we will issue warning emails to an appropriate team that the end user was not/cannot be warned of the snapshot expiry.

In summary, our input values are:

# Days when start to warn end users of deletion.
# Days when the snapshot should be deleted.
Email address in case of errors.
Email address for the report.
Tag name for VMs to be excluded from snapshot management.
A ‘WhatIf’ to control whether action is taken.

Our vRO server will also need to have the Active Directory plugin active and configured.

We also need to deal with another assumption - there can be more than one snapshot on a VM, and multiple snapshots may need to be deleted on each day.

\[Hindsight\]

After creating the workflows I added another couple of inputs:

Minimum age of VM snapshot to appear on the report.
Subject of the report.

Workflows

I write vRO workflows in a decentralised manner - for example if I am to do something with a Virtual Machine I have a workflow for:

The Virtual Machine
The Cluster
(Sometimes) The Datacenter
vCenter Server

This can add up to a lot of workflows, but it allows for testing and iterating changes easily. I will often put a “WhatIf” input on the Virtual Machine workflow so that I can test the workflows that function at a higher level - this ties in with the identified inputs earlier. So in this case, the workflow design mirrors the above (but without a datacenter script).

Additionally, when communicating to people the details of a VM with a snapshot I include Custom Attributes and Tags. So the workflows below will contain some tagging actions that I really need to release ;)

Virtual Machine

This workflow will need to do the following:

Get the snapshot information for a VM.
Calculate snapshot size (if possible).
Get the tags for a VM.
Add information to the report.
Check the tag to see if the VM will have it’s snapshots automatically managed.

If the VM is to be automatically managed, it will need to do:

If the snapshot is older than the warning age, send a warning to the snapshot creator, or the errors address.
If the snapshot is older than the deletion age, send a notification to the snapshot creator, or the errors address.

I’ve separate the above into two lists as I think the logic flows naturally in this way - the first list will apply to every VM that we examine whereas the second will only apply to those VMs with snapshots that we will manage. Thus we can break the logic up into two workflows.

Disclaimer: I’m assuming some familiarity with Orchestrator here, and there are plenty of other awesome blogs introducing the tool, so I’m going to dive in to some completed workflows.

The schema for the first flow is below:

Some notable steps:

Excluded from snaps - This is where the tag is checked.
Automatic Management - This is where the WhatIf is handled.
Manage Snapshots - We call the workflow to manage them. This sits inside a loop that will examine all the snapshots on a VM.

Some scripts are included to help with generating the report :)

The schema for Managing Snapshots is here:

The notable steps for this workflow are:

getEmailAddressForUser - this will query Active Directory for the mail attribute for the user identified earlier.
Remove Snapshot - deletes the snapshot and sets a task object so the workflows/entire process operates synchronously.
Customise Email - these write the warning and notification emails to be sent to the end user.

Cluster Workflow

This workflow retrieves all the VMs in the cluster and runs the main VM workflow against it. Nothing special here :)

Clusters / vCenter Servers Workflow

This is fairly standard workflow that goes through all the clusters on a vCenter Server and runs the Cluster workflow. It follows a standard pattern of retrieving a set of items and iterating through them. The end of the schema is devoted to sending the report email.

Configuration Element

As I wrote this a while ago, I put all the configuration inside a Configuration Element so that configuration variables could be updated once a workflow was scheduled. Please excuse my naming format, it does the job but the names made sense at the time I promise !

These names match the inputs that the workflows take:

Emails

Here is an example of the report email running on a live environment:

An example of the email notification a user receives about their snapshot (some field values have been removed):

Download

The workflows, actions and configuration element is available to download here. The package includes the following workflows

Once the package is imported, run one of the workflows and let me know what you think ! You’ll be able to run the cluster and VM workflows without having to configure a Configuration Element.

Download package file

Improvements

The most obvious improvement to this workflow would be to made the notifications optional and controlled via an input. Maybe something for a rainy day ;)

Let me know if this was useful or interesting to you !

Detaching Devices from a Cluster

Mon, 14 Aug 2017 00:00:00 +0000

When working with block storage we occasionally need to remove a storage device from the hosts it is connected to. When using the Web Client we can detach devices on a per-host basis, but there is nothing to detach from all its attached hosts.

What we do have is a helpful KB article on how to remove/detach a datastore -Â https://kb.vmware.com/kb/2004605Â and a PowerCLI file that contains some functions. However the functions revolve around managing actions on datastores as opposed to devices that are connected and could be used for RDMs.

Most LUNs are masked to all the hosts in a cluster so that, whether the device is an RDM or has a VMFS datastore on it, VMs can run on any host in the cluster. To deal with this use case, I use a script that will detach a device from every host in a cluster. If the device does not exist on Â a host then an error is displayed and the script moves on to the next device/host.

Before we can detach a device we need to ensure the following:

It is not used for a datastore.
It is not in use as a RDM.
The perennially reserved flag is not set for the device on the host.

The last point comes from the KB article above - if the flag is set (and it should be for any RDMs) then it could prevent the device from being unmounted successfully. So although clearing the flag is not required, clearing it ensures the best chance of success.

So, our script will start off with the following - we will accept a cluster to unmount/detach the device(s) from, an array of device identifiers, and a switch that will allow us to do checks without actually detaching anything.

param(
	[Parameter(Mandatory=$true,ValueFromPipeline)][VMware.VimAutomation.ViCore.Impl.V1.Inventory.ClusterImpl] $Cluster,
	[Parameter(Mandatory=$true)][String[]] $LUNIdentifiers,
	[switch]$CheckUsage = $false
)

For the sake of time, we will make the assumption that all devices are masked to all hosts in the cluster - so when checking what the device is used for we will only check on the first host in the cluster. The following code will get the VMs in the cluster (for checking RDM usage in the next step) and the host that we will use for checking.

$vms = $Cluster | Get-VM;
$cluster_host = $Cluster | Get-VMHost | sort Name | select -First 1

$LUNs_To_Detach = @();
$LUNIdentifiers | % {
	$lunid = $_;
$lun = get-scsilun -VMHost $cluster_host | ? { $_.CanonicalName -eq $lunid }

Finally we will use the Get-ESXCLI cmdlet to check if the perennially reserved flag is set.

`1`	`$myesxcli = get-esxcli -VMHost ESXhost`

When all our checks are complete, we can loop through the hosts in the cluster and attempt to detach the LUN.

	$Cluster | get-vmhost | sort Name | % {
		write-host "Detaching LUNs from $($_.Name)" -Foreground Yellow
		$esx = $_;
		
		$storSys = Get-View $_.Extensiondata.ConfigManager.StorageSystem
		
		foreach ($lunid in $LUNs_To_Detach){				
			$lun = Get-ScsiLun -VmHost $esx | ? { $_.CanonicalName -eq $lunid }
			if ($lun -ne $null) {
				# Ensure the LUN is attached 
				if ($lun.ExtensionData.OperationalState -eq "ok") {
					write-host "Detaching LUN $lun" # from $($esx.Name)"
					$storSys.DetachScsiLun($lun.ExtensionData.Uuid)
					write-host "Detach Complete" -Foreground Green
					
				} elseif ($lun.ExtensionData.OperationalState -eq "off") {
					Write-Host "LUN is already unmounted on this host." -Foreground Green;
					
				} else {
					Write-Host "OperationalState is $($lun.ExtensionData.OperationalState)";
					
				}
				
			} else {
				Write-Host "Could not find LUN $lun on host $esx.Name";
			}
		}
}

The entire script is available from my github here. As always, feel free to fork and make pull requests if you find a mistake.

Using Orchestrator to automatically upgrade Tools

Wed, 08 Mar 2017 00:00:00 +0000

One of the main challenges in every (non-small) place I’ve worked is keeping VMware Tools updated, and with Change Management processes this becomes a little bit more complicated. There is a setting on each VM to upgrade Tools when the VM is booted and this is a good way to keep Tools updated on those VMs that aren’t mission critical and require manual intervention.

In this post we’ll look at creating an action and a workflow to configure this setting, and pushing it as a context setting in the Flex Web Client.

This leads to the issue of how this setting is enabled - firstly as part of the initial push (you don’t want to enable this manually for 100’s (or more) VMs) and with newly created VMs. Although this can be set on a VM template or “Gold Build” (or equivalent terminology), this can still lead to human error so we’ll also look creating a workflow to set this for a cluster and scheduling this using the Flex Web Client.

Upgrade Tools Action

To configure a VM to upgrade Tools we need to create a new configuration specification and set the tools configuration appropriately. Our actionÂ will return the Task for the configuration (note: this can be null) and will take two inputs - the VM to configure and whether we are automatically upgrading Tools. Here is a screenshot of the action configuration:

The script of the action is as follows:

task = null;

if (vm != null)
{
	var toolsconfig = new VcToolsConfigInfo();
	
	if (autoupgrade == true) {
		toolsconfig.toolsUpgradePolicy = "upgradeAtPowerCycle";
	} else {
		toolsconfig.toolsUpgradePolicy = "manual";
	}
	
	var vmconfig = new VcVirtualMachineConfigSpec();
	vmconfig.tools = toolsconfig;

	task = vm.reconfigVM_Task(vmconfig);
}

return task;

In the code snippet above, we ensure that the “vm” variable contains a valid VM. We create a new ToolsConfigInfo object which will contain the tools settings that we wish to apply and then create a new VirtualMachineConfigSpec object and assign the ToolsConfigInfo object to it. Thus when we call the reconfigVM_Task() function with the configuration object, the VM is only reconfigured with the settings we have provided.

Now we have an action, we can create a workflow that will invoke it and then wait for the task to complete. This is the schema diagram:

The workflow (and action) can be downloadedÂ from the file at the end of this post.

Cluster Workflow

Now we have an action to make the changes, we can put together a workflow that will look at all VMs in the cluster and configure them. The schema is as follows:

The workflow will get all VMs in the cluster and for each one:

Only upgrade Windows VMs (“Upgrade VM ?” decision)
Check if the configuration setting currently matches what we want it to. (“Decide setting” script).
Change the setting.

The meat of the process is in the “Decide setting” script item - we will look at what the setting should be, and if the current setting does not match then we will change it. This also means that if we wish the VM to update Tools manually and it is set to automatically update at power on, the setting will be changed to update manually.

Setting as a context action

To set this as a context action on the cluster within the Flex Web Client, we need to look at the Orchestrator pages in the Web Client. The server page will have a Manage screen and a Context Actions page:

Next we need to add a new one - select the workflow to add and the type of object to associate it with - in this case we’re adding the workflow to set the upgrade for Tools for a single virtual machine, so we select the Virtual Machine object:

Press OK and all is done. You’ll need to reload the Flex Web Client in order to see it - it appears at the bottom of the actions menu for the object - under the “vRealize Orchestrator context actions” title.

That’s it !Â This workflow a prime candidate for being scheduled to ensure all your VMs are kept up to date.

The workflows (and action) for this are available at my github repo - https://github.com/nelons/vco-content.

Automating Brocade FC Configuration (Part 2)

Fri, 20 Jan 2017 00:00:00 +0000

From the first article we looked at how to use the SSH Sessions Powershell module to log into a Linux server that will, via a script, log into a Brocade FC switch and get the zoning configuration.

As per the disclaimer in the first part, these articles have been assembled from my notes and memory and may not be 100% correct. I worked with FOS 6.x and so 7.x/later releases might not need this approach - I no longer have access to any Brocade FC switches to test :)

Getting the switch config via a function

As we’re going to need to get the config from two switches, we’ll write a powershell function that will get the config from one switch, so we have one call for each. The function is as follows:

Function Get-BrocadeSwitchZoneConfig {
	[CmdletBinding()]
	Param (
		[Parameter(Mandatory=$true)][string] $Target,
		[Parameter(Mandatory=$true)][string] $Username,
		[Parameter(Mandatory=$true)][string] $Password,
		[switch] $VersionSix
	)
	Process {
		# Import the SSH-Sessions Module.
		Import-Module $PSScriptRoot\..\modules\SSH-Sessions\SSH-Sessions.psd1
		
		# Connect to the target.
		New-SshSession -ComputerName $Target -Username $Username -Password $Password | out-null
		
		# Invoke the command to get the config.
		$config = "";

		if ($VersionSix -eq $False) {
			$config = Invoke-SshCommand -ComputerName $Target -Command "zoneshow" -Quiet
			
		} else {
			$config = Invoke-SshCommand -ComputerName $Target -Command "bash --login -c 'zoneshow'" -Quiet;
		
		}
	
		# Disconnect from the target.
		Remove-SShSession -ComputerName $Target | out-null;
		
		# return the config.
		return $config;
	}
}

So, let’s get the config:

1
2

$config1 = Get-BrocadeSwitchZoneConfig -Target "switch1.fqdn" -Username "automation" -Password "password";
$config2 = Get-BrocadeSwitchZoneConfig -Target "switch2.fqdn" -Username "automation" -Password "password"

Sanity Checking the Zone Configuration

Before changing the configuration we need to make sure anything we create does not already exist. Namely, we’ll be checking:

WWPNs of the server HBAs.
The name of the server zone.

Function Check-FCConfig {
	[CmdletBinding()]
	Param (
        Parameter(Mandatory=$true)][string] $Config,
		[Parameter(Mandatory=$true)][string] $Hostname,
		[Parameter(Mandatory=$true)][string] $HBA
	)
	Process {
		# need to trim the "-" from the hostname as it is an illegal char on the brocades.
		$brocade_hostname = $Hostname.Replace("-", "");
		
		if ($HBA -ne $Null) {			
			# Check to see if the hostname is present
			if ($Config.ToLower().Contains($brocade_hostname.ToLower())) {
				Write-Host "Hostname exists in the current config.";
				return $False;
			}
			
			# Check to see if HBA address is present.
			if ($Config.ToLower().Contains($HBA.ToLower())) {
				Write-Host "HBA exists in the current config.";
				return $False;
			}

                        return $True;
		}

                return $False;
        }
}

So, we’ll then need to call this for each of the switch configs we got earlier. Our new script looks as follows:

$config1_ok = $false;
$config2_ok = $false;

$config1 = Get-BrocadeSwitchZoneConfig -Target "switch1.fqdn" -Username "automation" -Password "password";
if ($config1.Length -gt 0) {
  $config1_ok = Check-FCConfig -Config $config1 -Hostname "server01" -HBA "20:00:00:25:B5:40:02:0F";
}

$config2 = Get-BrocadeSwitchZoneConfig -Target "switch2.fqdn" -Username "automation" -Password "password"
if ($config2.Length -gt 0) {
  $config2_ok = Check-FCConfig -Config $config2 -Hostname "server01" -HBA "20:00:00:25:B5:40:02:1F";
}

if ($config1_ok -eq $true -And $config2_ok -eq $true) {
 // Here is where we create the config.

}

Hopefully the above is quite straight forward - we get the config for the both switches and check that it is ok for the servername and for the HBA address. We store the result in a local variable, and when both checks are complete we will only be making changes to the FC switches if both configs reported good.

The final proxy script

Our final Linux proxy script will change the configuration on a singleÂ switches, and we will call it twice, once for each server. This would allow us to provide different variables for each switch, in case they have not been configured identically. The variables that we will pass, and their position for the script are below:

Hostname of the filer.
Username for the filer.
Password for the filer.
Server hostname
WWPN for the HBA
Name of the active configuration

Our full script will be:

cat « _END_ | sshpass -p ${1} ssh ${3} -tt -o StrictHostKeyChecking=no -l ${2} alicreate ${4}_hba0, ${5} zonecreate ${4}, “${4}_hba0;san_hba0;san_hba1” cfgadd ${6}, ${4} echo y|cfgenable ${6} exit _END_

This is how we would call it:

$brocade_hostname = $Hostname.Replace("-", "");
$brocade_config_name = "config";
$fc_switch1 = "switch1.fqdn";
$fc_switch2 = "switch2.fqdn";
		
# Connect to the brocade scripting host.
New-SshSession -ComputerName $brocade_script_host -Username $brocade_script_host_username -Password $brocade_script_host_password | out-null;
		
# Run the Script.
$results = Invoke-SshCommand -ComputerName $brocade_script_host -Command "./create-fc-zone $fc_switch1 $brocade_username $brocade_password $brocade_hostname $HBA0 $brocade_config_name" -Quiet;

# Run the Script.
$results = Invoke-SshCommand -ComputerName $brocade_script_host -Command "./create-fc-zone $fc_switch2 $brocade_username $brocade_password $brocade_hostname $HBA1 $brocade_config_name" -Quiet;
		
# Disconnect from the host
Remove-SshSession -ComputerName $brocade_script_host | out-null;

Powershell Configuration Script

When we put the above together we will aim to have some powershell that will:

Get the config of each FC switch
Check the config to ensure there will be no clashes or problems.
Call the proxy script to update the config.

This would look like:

Function Create-FC-Zone {
	[CmdletBinding()]
	Param (
		[Parameter(Mandatory=$true)][string] $Hostname,
		[Parameter(Mandatory=$true)][string] $HBA0,
		[Parameter(Mandatory=$true)][string] $HBA1
	)
	Process {
		$brocade_hostname = $Hostname.Replace("-", "");
		$brocade_config_name1 = "config";
		$brocade_config_name2 = "config";
		
		# FC Switch Details.
		$fc_switch1 = "switch1.fqdn";
		$fc_switch2 = "switch2.fqdn";
		$fc_username = "automation";
		$fc_password = "password";
		
		# Proxy Details
		$proxy_host = "proxy.fqdn";
		$proxy_username = "automation";
		$proxy_password = "password";
		
		$config1_ok = $false;
		$config2_ok = $false;

		$config1 = Get-BrocadeSwitchZoneConfig -Target $fc_switch1 -Username $fc_username -Password $fc_password;
		if ($config1.Length -gt 0) {
			$config1_ok = Check-FCConfig -Config $brocade_config_name1 -Hostname $brocade_hostname -HBA $HBA0;
		}

		$config2 = Get-BrocadeSwitchZoneConfig -Target $fc_switch2 -Username $fc_username -Password $fc_password;
		if ($config2.Length -gt 0) {
			$config2_ok = Check-FCConfig -Config $brocade_config_name2 -Hostname $brocade_hostname -HBA $HBA1;
		}

		if ($config1_ok -eq $true -And $config2_ok -eq $true) {			
			# Connect to the brocade scripting host.
			New-SshSession -ComputerName $proxy_host -Username $proxy_username -Password $proxy_password | out-null;
					
			# Run the Script.
			$results = Invoke-SshCommand -ComputerName $proxy_host -Command "./create-fc-zone $fc_switch1 $fc_username $fc_password $brocade_hostname $HBA0 $brocade_config_name" -Quiet;

			# Run the Script.
			$results = Invoke-SshCommand -ComputerName $proxy_host -Command "./create-fc-zone $fc_switch2 $fc_username $fc_password $brocade_hostname $HBA1 $brocade_config_name" -Quiet;
					
			# Disconnect from the host
			Remove-SshSession -ComputerName $proxy_host | out-null;
		}	
    }
}

This uses the function “Get-BrocadeSwitchZoneConfig” at the beginning of this article, and most of the variables are configured in the script. For the sake of security you’ll want to remove these from being hardcoded and pass them in as parameters. I’ve left them as hardcoded to show what is required and where it is used.

This is available in a single file on my psscripts github repo.

Please let me know if anything is wrong or if this helped you !

VMware vCheck NTP and Syslog Plugin Amendments

Tue, 09 Jun 2015 00:00:00 +0000

I’m a big fan of the vCheck script by the very talented Alan Renouf and community, but a couple of the plugins don’t work as well as I’d like with our environment, where we have multiple NTP and Syslog servers and these settings can differ depending on the cluster within a single vCenter. So I’ve made a few changes to the vCheck NTP and Syslog plugins.

NTP Plugin

The file is called “31 NTP Name and Service.ps1” and kept in the “30 Host” folder.

I’m going to cut out the bottom part of the plugin script as there would only be cosmetic changes made there.

# Start of Settings
# The NTP server which should be set on your hosts
$ntpserver = "ntp-server1.cluster1 ntp-server2.cluster1", "ntp-server1.cluster2"
# End of Settings

$Result = @($VMH | Where {$_.Connectionstate -ne "Disconnected"} | Select Name, @{N="NTPServer";E={"$($_ | Get-VMHostNtpServer)"}}, @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ | Where-Object {$_.key -eq "ntpd"}).Running}} | Where {$_.ServiceRunning -eq $false -o $ntpserver -notcontains $_.NTPServer})
$Result

There are two changes here:

The line “$($_ | Get-VMHostNtpServer)” has been enclosed in quotation marks to expand the array into a string.
The check at the end has been changed so we are now checking if the NTP server values are in the provided array.

I’m fairly sure the list of NTP servers are ordered (but have been known to be wrong before), so I’m leaving the list of servers in the order that PowerCLI decides to return them. So when specifying a list of NTP servers, do it in the order they should be used/configured.

Syslog Servers

The syslog plugin file is called “51 Syslog Name.ps1” and it is also in the “30 Host” folder.

I don’t believe that the syslog hosts are ordered (they certainly shouldn’t be), so in the code below we’re looking at the list of syslog hosts sorted alphabetically.

The code is now:

# Start of Settings 
# The Syslog server which should be set on your hosts
# Multiple syslog entries per host must be space separated and in alphabetical order.
# Multiple possible entries per vCenter should be comma separated so an array is created.
$SyslogServer = "udp://syslog-server1 udp://yet-another-syslog-server"
# End of Settings

@($VMH | Where-Object {$_.ExtensionData.Summary.Config.Product.Name -eq 'VMware ESXi'} | Select-Object Name,@{Name='SyslogServer';Expression = {"$(($_ | Get-VMHostSysLogServer).Host | sort Host)"}} | Where-Object {$SyslogServer -notcontains $_.SyslogServer})

There’s a few changes:

The ‘SyslogSettings’ section, supporting ESXi 4.x has been removed.
The line “$(($_ | Get-VMHostSysLogServer).Host | sort Host)” sorts the names and expands the array into a string.
The comparison check at the end now checks for presence in the array.

Summary

This is very much a lazy fix to a problem I had, and there will be a better solution. For instance, it’s not going to check that hosts in a certain cluster have particular settings - just that all hosts have a valid setting permissable for the vCenter. But it’ll do the job, and for everything else there should be host profiles - and there is vCheck plugin that can be used unaltered to check on those ;)

Creating a Cisco UCS Profile using PowerTool

Thu, 26 Mar 2015 00:00:00 +0000

Creating a Cisco UCS Profile using PowerTool

This post is going to walk through creating a powershell function to create a Service Profile on a Cisco UCS domain. Later posts in this series will look at how to assign it to a server, configure shared storage on a NetApp filer, and configure the zoning on Fiber Channel switches - essentially all the work we’d need to do to automate the creation of a physical server.

There are a few ways to create Service Profiles on a Cisco UCS domain, but if you’re looking for consistency, best practice and the possibility to automate then you’ll need to use templates - not just a Service Profile template, but templates within the Service Profile template - for vNICs, vHBAs and for any other policies. I have assumed that templates have been set up and are being used.

We are going to be using Cisco UCS PowerTool for this, so ensure you’ve got this installed on your system. You can get it from , but you’ll need a Cisco.com login.

The File

Open up your favourite IDE or text editor (personally I use Notepad++) and create a new file. Save it with a filename and a location of your choice. My file is going to be called “automate.ps1” and I keep everything in a “scripts” folder in the root of a hard drive. Powershell has a verb system, so the cmdlets are prepended (for example) with “New”, “Get” and “Set”. I’m going to use the verb “Automate” to make our functions stand out from anything else that might exist on your system, but you might wish to use your company name or something else.

Below is a sample function definition for “Automate-NewUCSProfile”.

Function Automate-NewUcsProfile {
	Param()
	Process {
		# Our code will go here
	}
}

The first line declares the function and the curly brackets open its contents. The “Param()” line will be expanded so we can pass parameters into the function, and the main body of code will reside in the curly brackets are the “Process {” section. We then close off every open curly bracket “{” with the opposite curly bracket “}”.

If you’d like to learn more about Powershell functions, your search engine of choice will direct you.

Connections

When we connect to a UCS domain we can be given a handle to it. PowerTool will support connections to multiple UCS domains, though I believe the default is one and it needs to be reconfigured through the Set-UcsPowerToolConfiguration cmdlet (see it and the Connect-UCS cmdlet help for more information). If we are connecting to one UCS domain then we won’t need to use the handle when using other cmdlets, however I believe it is good practice to do so, so I will.

To connect we use:

`1`	`Connect-UCS -Name ucspe01 -Credential (Get-Credential)`

When running the above connect cmdlet (and after entering correct credentials for the target UCS domain) we are given something similar to below (which is a platform emulator output running on VMware Workstation 9):

Proxy                       :
Cookie                      : 1427096220/c399b102-8234-4bad-a94d-4202ed62593a
Domains                     : org-root
LastUpdateTime              : 23/03/2015 07:36:59
Name                        : 192.168.117.135
NoSsl                       : False
NumPendingConfigs           : 0
NumWatchers                 : 0
Port                        : 443
Priv                        : {aaa, admin, ext-lan-config, ext-lan-policy...}
RefreshPeriod               : 600
SessionId                   :
TransactionInProgress       : False
PromptOnCompleteTransaction : False
Ucs                         : UCSPE-192-168-117-135
Uri                         : https://192.168.117.135
UserName                    : config
VirtualIpv4Address          : 192.168.117.135
Version                     : 2.2(2c)
WatchThreadStatus           : None

So, if we want to connect without this information being printed to the screen then we’ll need to accept the handle being returned from the cmdlet, as below:

`1`	`$ucs = Connect-UCS -Name ucspe01 -Credential (Get-Credential)`

Credential can be prompted for by using the “(Get-Credential)” input above, or it can be provided ahead of time, by putting the output of “Get-Credential” into a variable.

1
2

$cred = Get-Credential
$ucs = Connect-Ucs -Name ucspe1 -Credential $cred

To disconnect, we use the cmdlet

`1`	`Disconnect-UCS -Ucs $ucs`

So building our function it will start to look as below. We are passing in the name of the UCS domain and the credentials to connect to it.

Function Automate-NewUcsProfile {
	Param(
		[Parameter(Mandatory=$true)][String] $ucs_domain,
		[Parameter(Mandatory=$true)][System.Management.Automation.PSCredential] $credential = (Get-Credential)
	)
	Process {
		# Our code will go here
		$ucs = Connect-Ucs -Name $ucs_domain -Credential $credential
		if ($ucs -ne $null) {
		
		
		
			Disconnect-Ucs -Ucs $ucs;
		}
	}
}

One thing to note here is that if a Credential is not provided, then the person running the script is prompted to provide one.

Creating a Service Profile from a template

One line of code is required to create a Service Profile based upon a template, and it takes the following parameters:

- ServiceProfile is the name of the template upon which to base the new profile - NewName is the name of the new profile.

It returns a object to represent the service profiles that it creates. An example:

`1`	`$service_profile = Add-UcsServiceProfileFromTemplate -ServiceProfile "template-name" -NewName "new-profile" -Ucs $ucs;`

Note this this leaves the new profile bound to the template. We can change this later.

We should do some sanity checking though - I’m sure you are aware of the two types of templates - “initial” and “updating” (if not, Cisco have a useful page on Service Profiles). These are represented in PowerShell by being a Service Profile object, but with a type of either “initial-template” or “updating-template”. A Service Profile object that is actually a Service Profile has a type of “instance”. So let’s make sure that a profile with this name exists:

`1`	`$template = Get-UCSServiceProfile -Name "template-name" -Ucs $ucs \| ? { $_.type -ne "instance" }`

This code is retrieving all service profiles of the name “template-name”. It then runs a check on them to see if they have type set to “instance”, and if so, the object is returned.

If $template is equal to $null (i.e. $template -eq $null) then no template could be found. Putting our code together, outside of our function, it would look like:

$ucs = Connect-UCS -Name ucspe01 -Credential (Get-Credential)
if ($ucs) {

	$template = Get-UCSServiceProfile -Name "template-name" -Ucs $ucs | ? { $_.type -ne "instance" };
	if ($template -ne $null) {
		$service_profile = Add-UcsServiceProfileFromTemplate -ServiceProfile $template -NewName "new-profile" -Ucs $ucs;
	}

	Disconnect-UCS -UCS $ucs;
}

Optional: Unbinding from the Service Profile template

If you wish to unbind from the template, the line is simple:

`1`	`Set-UcsServiceProfile -ServiceProfile $service_profile -SrcTemplName '' -Force \| out-null;`

Though we could pipe the $service_profile variable to it, as per:

`1`	`$service_profile \| Set-UcsServiceProfile -SrcTemplName '' -Force \| out-null`

The “| out-null” just ensures that nothing is printed on the screen as it is being run.

Assembling our Function

From the above we need to know some things:

- The name of the UCS domain to connect to. - The credentials to connect with. - The name of the Service Profile we want to create. - The name of the Template we are going to bind to.

So when we put the code from above into the function, and configure our parameters, this is what it should look like:

Function Automate-NewUCSProfileFromTemplate {
	Param(
		[Parameter(Mandatory=$true)][String] $Domain,
		[Parameter(Mandatory=$true)][System.Management.Automation.PSCredential] $Credential,
		[Parameter(Mandatory=$true)][String] $new_profile_name,
		[Parameter(Mandatory=$true)][String] $template_name
	)
	Process {
		$ucs = Connect-Ucs -Name $Domain -Credential $Credential
		if ($ucs -ne $null) {			
		
			$template = Get-UCSServiceProfile -Name $template_name -Ucs $ucs | ? { $_.type -ne "instance" };
			if ($template -ne $null) {
				Add-UcsServiceProfileFromTemplate -ServiceProfile $template -NewName $new_profile_name -Ucs $ucs;
			}
		}

		Disconnect-UCS $ucs;	
	}
}

Organisations and checkingÂ Profile existence

We’ve got one last check that we should make - that the service profile doesn’t already exist. This is slightly more complicated due to Organisations - service profile names are unique within each organisation. This is represented in service profile objects by distinguished names which are formed of two parts, the location and the service profile name. Running the code:

`1`	`Get-UcsServiceProfile \| ft Name, Rn, Dn`

will show us the format:

PS C:\> get-ucsserviceprofile | ft Name, Rn, Dn

Name                                    Rn                                      Dn
---- -- --
test                                    ls-test                                 org-root/ls-test
test                                    ls-test                                 org-root/org-Finance/ls-test

As we can see, we have two service profiles with the name “test”, one in the “root” organisation, and one in the “Finance” organisation. Let’s get the details about the organisations:

PS C:\> Get-UcsOrg | ft Name, Dn

Name                                                        Dn
---- --
root                                                        org-root
Acme                                                        org-root/org-Acme
Finance                                                     org-root/org-Finance

So when we create our profile we must make a choice about where we create it - by default it will be in the root organisation. Our code will have to take a string that allows callers to provide an organisation (but have a default of “root”). We will then search for the service profile within that organisation:

`1`	`$existing_profile = Get-UcsServiceProfile -Name $sp_name -Org $org`

It looks simple, but there is a catch - it will still match against any service profiles in sub-organisations. We need to add the “-LimitScope” parameter in order to only match within our desired target organisation. If $existing_profile is equal to $null, then we can continue.

So our function will now look like:

Function Automate-NewUCSProfileFromTemplate {
	Param(
		[Parameter(Mandatory=$true)][String] $Domain,
		[Parameter(Mandatory=$true)][System.Management.Automation.PSCredential] $Credential,
		[Parameter(Mandatory=$true)][String] $NewName,
		[String] $Organisation = "root",
		[Parameter(Mandatory=$true)][String] $Template
	)
	Process {
		$ucs = Connect-Ucs -Name $Domain -Credential $Credential
		if ($ucs -ne $null) {			
			$org = Get-UcsOrg $Organisation;
			if ($org) {
				# Check to see if profile already exists.
				$existing_sp = Get-UcsServiceProfile -Name $NewName -Org $org -LimitScope;
				if ($existing_sp -eq $null) {
		
					# Check to see if template exists.
					$sp_template = Get-UCSServiceProfile -Name $Template -Ucs $ucs | ? { $_.type -ne "instance" };
					if ($sp_template -ne $null) {
						$sp = Add-UcsServiceProfileFromTemplate -ServiceProfile $Template -NewName $NewName -Ucs $ucs;
					}
				}
			}
		}

		Disconnect-UCS $ucs;	
	}
}

Note: Organisation names are not unique throughout the Domain and it is the Dn that will determine uniqueness. This is something that we’ll deal with next.

One Final Step

To make everything simple, we should do all our checks to see if any stage of this will fail at the start of code. For the sake of separation we’ll create a new function for this:

Function Automate-CheckUcsServiceProfileConflict {
	Param(
		[Parameter(Mandatory=$true)][Cisco.Ucs.UcsHandle] $Ucs,
		[Parameter(Mandatory=$true)][String] $NewName,
		[String] $Organisation = "root",
		[Parameter(Mandatory=$true)][String] $Template
	)
	Process {
		$result = $True;
	
		# Check the Organisation exists.
		$org = Get-UcsOrg $Organisation;
		if ($org -eq $null) {
			Write-Host "The Organisation does not exist.";
			$result = $False;
		
		} elseif ($org.Count -gt 1) {
			# We found more than one Organisation
			Write-Host "There is more than one Organisation with the name '$Organisation'. Please use the Org Dn to create Service Profiles.";
			$result = $False;
		
		} else {		
			# Check that a profile with the existing name doesn't already exist.
			$existing_sp = Get-UcsServiceProfile -Name $NewName -Org $org -LimitScope;
			if ($existing_sp -ne $null) {
				Write-Host "A Service Profile with the name '$NewName' already exists in the organisation '$Organisation'.";
				$result = $False;
				
			}			
		}
		
		# Check that the template exists.
		$sp_template = Get-UCSServiceProfile -Name $Template -Ucs $ucs | ? { $_.type -ne "instance" };
		if ($sp_template -eq $null) {
			Write-Host "The template $Template does not exist.";
			$result = $False;
		}
		
		return $result;
	}
}

There are a few differences to our original function - we are passing in the connection handle to the UCS Domain, so we don’t have to deal with any Connect-Ucs/Disconnect-Ucs and credentials. We are then going to run some checks and return whether we can go ahead with creating the service profile.

Let’s go through the code:

- We first get all the Organisations that have the name we supply. - If there are no Organisations then we output this to the user and ensure we will return failure. - We see if there is more than one Organisation with the name. - If there is one Organisation then we check to see if there is a service profile with the same name in it.

- We check to see if the service profile template exists. - We return the result.

This is where we are handling that we could have multiple Organisations could have the same name. If more than one is found, then we will fail and tell the person running our function to supply the Dn if they wish to continue.

When we alter our original function to use this one to do checks, and put everything together, it looks as per the below:

Function Automate-CheckUcsServiceProfileConflict {
	Param(
		[Parameter(Mandatory=$true)][Cisco.Ucs.UcsHandle] $Ucs,
		[Parameter(Mandatory=$true)][String] $NewName,
		[String] $Organisation = "root",
		[Parameter(Mandatory=$true)][String] $Template
	)
	Process {
		$result = $True;
	
		# Check the Organisation exists.
		$org = Get-UcsOrg $Organisation;
		if ($org -eq $null) {
			Write-Host "The Organisation does not exist.";
			$result = $False;
		
		} elseif ($org.Count -gt 1) {
			# We found more than one Organisation
			Write-Host "There is more than one Organisation with the name '$Organisation'. Please use the Org Dn to create Service Profiles.";
			$result = $False;
		
		} else {		
			# Check that a profile with the existing name doesn't already exist.
			$existing_sp = Get-UcsServiceProfile -Name $NewName -Org $org -LimitScope;
			if ($existing_sp -ne $null) {
				Write-Host "A Service Profile with the name '$NewName' already exists in the organisation '$Organisation'.";
				$result = $False;
				
			}			
		}
		
		# Check that the template exists.
		$sp_template = Get-UCSServiceProfile -Name $Template -Ucs $ucs | ? { $_.type -ne "instance" };
		if ($sp_template -eq $null) {
			Write-Host "The template $Template does not exist.";
			$result = $False;
		}
		
		return $result;
	}
}

Function Automate-NewUCSProfileFromTemplate {
	Param(
		[Parameter(Mandatory=$true)][String] $Domain,
		[Parameter(Mandatory=$true)][System.Management.Automation.PSCredential] $Credential = (Get-Credential),
		[Parameter(Mandatory=$true)][String] $NewName,
		[String] $Organisation = "root",
		[Parameter(Mandatory=$true)][String] $Template
	)
	Process {
		$ucs = Connect-Ucs -Name $Domain -Credential $Credential
		if ($ucs -ne $null) {			

			$checks_passed = Automate-CheckUcsServiceProfileConflict -Ucs $ucs -NewName $NewName -Organisation $Organisation -Template $Template;
			if ($checks_passed -eq $True) {
				$sp = Add-UcsServiceProfileFromTemplate -ServiceProfile $Template -NewName $NewName -Ucs $ucs;
				if ($sp) {
					Write-Host "Service Profile Created";
				
					# If you want to unbind from the template then put the code here.
					Set-UcsServiceProfile -ServiceProfile $sp -SrcTemplName '' -Force | out-null;
				}
			}			
		}

		Disconnect-UCS -Ucs $ucs;	
	}
}

And that’s it :). We now have some code that will create a UCS Service Profile from a template in a given organisation and will pass the obvious sanity checks.

We can now create (or at least attempt to) a service profile by doing the following:

1
2

C:\Scripts\>. .\automate.ps1
C:\Scripts\>Automate-NewUCSProfileFromTemplate -Domain ucspe01 -NewName server01 -Template init-windows

Next time, we’re going to get the HBA WWPNs and associate the service profile with a server.