Orchestrator Maintenance Automated


Are you tired of dealing with orphaned runbooks every week, filling up your available slots and causing runbooks to spill over onto the secondary runbook server?  Ideally you should find the root cause of these runbooks orphaning.  In our case it is due to our backup software taking the servers down briefly and causing the runbook server to lose connection to the database.  We cant stop backups for obvious reasons but the next best solution is to let Orchestrator fix itself.  You can download the entire project below.

This will require the

SCORCH Dev – Orchestrator integration pack and the SCOM integration pack

In this scenario we are using the Infront System Center Orchestrator Management Pack to monitor our runbooks in SCOM.  This is a handy tool to use to make sure your monitors are always running.  I highly recommend this MP to ease some of the stress in your larger environments.

My folders are setup as so:

SCORCH Folders.jpg

This is the main runbook that handles all of the tasks.

Scorch MM.jpg

You will notice the several junctions that are in place for this runbook.  This is because we have several monitors in Orchestrator and I don’t want the next runbooks down the line to run multiple times.  Now lets go through each IP to see how they are setup.  There are no variables in the Initialize data IP since this runbook is completely self-contained.  The invoke runbook IPs are named after the runbooks they are calling so that is pretty self explanatory and each one should be setup to wait for completion.

Lets move onto the 1.0 Start MM for Runbook Monitors IP.

1.0 Maintenance Mode for Monitors

MM for RBMon.jpg

This is a simple one that will set all of the Monitor Runbooks into maintenance mode in SCOM.  Again, I am using Infront to monitor these runbooks so you might need to change it to what ever you are using for monitoring or remove it completely if you are not using SCOM to watch over your monitoring runbooks.

Initialize data is blank for this one.  The Get Orchestrator RB Monitor IP is setup as so…Get Orchestrator RB Monitor.jpg

The Start Maintenance Mode IP is as so…you will notice that you need to build the monitor name from what is returned from the Get Orchestrator RB Monitors since it doesn’t return exactly what we need.  Start MM.jpg

On the Return Data IP we are sending back the name we just built for use later down the line…

Return Data.jpg

If you don’t know how to setup return data just right click on the runbook’s tab and choose properties.  Choose the Returned Data tab and add in what you need to return.

Returned Data Setup.jpg

This sends us back to the main runbook where you see I have a junction in place.  Since i have 24 monitoring runbooks in place that means the next process will run 24 times and the next process is stopping all of the monitoring runbooks which is 24 again.  24×24= 576.  We would be hitting the webservice 576 times and we don’t want that.  Junction 3.png

You see I have this setup to return nothing because I don’t need that data quite yet.  My goal with this is to limit the amount of time I hit any external systems for data.  I could just hit SCOM again for that data but why add unnecessary load to the servers when I can just maintain the info in the databus.

Now we move onto the Stop All Monitors…

1.1 Stop All Monitors

Stop all Jobs.png

Again the Initialize Data IP is blank. This is where we will be using the SCORCH Dev – Orchestrator OIP.  If you need some guidance setting this up you can refer to here.

The second IP will retrieve all of the monitors from the SCORCH webservice.

Get Monitors.png

Now we need the job instance details from the runbooks.  This is setup as so…

Job Instance Detail.png

Now lets stop all those monitors.Stop Jobs.png

And the return data IP is blank.  It’s just good practice to have those in place even if you are not returning anything.  Now back to the main runbook.  You will notice again that I have a junction.  This is because I’m about to run a stored procedure against the Orchestrator DB and I don’t want to run it 24 times.  Thats just too much, once is enough.

Now lets clear out all those orphans.  This is using a built in stored procedure to handle this job.

1.2 Clear Orphans

Clear Orphans RB.png

 

Again, blank initialize data.  The Clear Orphans job is as so…Clear Orphans SQL.png

USE [Orchestrator]
DECLARE @return_value int EXEC @return_value = [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].[ClearOrphanedRunbookInstances] SELECT ‘Return Value’ = @return_value

And the return data is blank, again just good practice.  Now our orphans have all gone away so lets start all of our monitors back up.

Screen Shot 2016-03-04 at 20.49.03 PM.png

1.3 Start Monitors

Start Monitors.png

We need to get all of the monitors again.  Screen Shot 2016-03-04 at 20.50.46 PM.png

And start them up…

Screen Shot 2016-03-04 at 20.51.19 PM.png

Now we are back up and running.  Let’s turn off maintenance mode in SCOM so we are keeping a close eye on those monitoring runbooks.  Back to the main runbook.

Screen Shot 2016-03-04 at 20.53.27 PM.png

As you can see I have another junction connecting 1.0 to 1.4.  That is because I want that data I built when I was putting the monitors in maintenance mode the first time.  Screen Shot 2016-03-04 at 20.54.46 PM.png

 

1.4 Remove MM

This takes us to 1.4 Remove MMScreen Shot 2016-03-04 at 20.55.43 PM.png

This time in the initialize data I have the SCOM monitoring names I built in 1.0.

Screen Shot 2016-03-04 at 20.56.19 PM.png

Stopping Maintenance Mode..

Screen Shot 2016-03-04 at 20.57.17 PM.png

and the return data is blank.

Since this is a job that will run every night after backups occur I can’t set this up as a scheduled monitor or will will just kill itself every time it runs and thats not good.  We turn to PowerShell and scheduled tasks to handle this job.  Hopefully you know how to setup a scheduled task on a server but if not there are plenty of articles discussing that.  Here is the PowerShell script I use to call the Orchestrator webservice.

$rbid = “89DBAF5F-BA91-4525-8AA8-9504E2018FB7”
# Create the request object
$request = [System.Net.HttpWebRequest]::Create(“http://ORCHESTRATORWEBSERVER:81/Orchestrator2012/Orchestrator.svc/Jobs”)

# Set the credentials to default or prompt for credentials
$request.UseDefaultCredentials = $true
# $request.Credentials = Get-Credential

# Build the request header
$request.Method = “POST”
$request.UserAgent = “Microsoft ADO.NET Data Services”
$request.Accept = “application/atom+xml,application/xml”
$request.ContentType = “application/atom+xml”
$request.KeepAlive = $true
$request.Headers.Add(“Accept-Encoding”, “identity”)
$request.Headers.Add(“Accept-Language”, “en-US”)
$request.Headers.Add(“DataServiceVersion”, “1.0;NetFx”)
$request.Headers.Add(“MaxDataServiceVersion”, “2.0;NetFx”)
$request.Headers.Add(“Pragma”, “no-cache”)

# If runbook servers are specified, format the string
$rbServerString = “”
if (-not [string]::IsNullOrEmpty($RunbookServers))
{
$rbServerString = -join (“<d:RunbookServers>”, $RunbookServers, “</d:RunbookServers>”)
}

# Format the Runbook parameters, if any
$rbParamString = “”
if ($rbParameters -ne $null)
{

# Format the param string from the Parameters hashtable
$rbParamString = “<d:Parameters><![CDATA[<Data>”
foreach ($p in $rbParameters.GetEnumerator())
{
#$rbParamString = -join ($rbParamString,”&lt;Parameter&gt;&lt;ID&gt;{“,$p.key,”}&lt;/ID&gt;&lt;Value&gt;”,$p.value,”&lt;/Value&gt;&lt;/Parameter&gt;”)
$rbParamString = -join ($rbParamString, “<Parameter><ID>{“, $p.key, “}</ID><Value>”, $p.value, “</Value></Parameter>”)
}
$rbParamString += “</Data>]]></d:Parameters>”
}

# Build the request body
$requestBody = @”
<?xml version=”1.0″ encoding=”utf-8″ standalone=”yes”?>
<entry xmlns:d=”http://schemas.microsoft.com/ado/2007/08/dataservices&#8221; xmlns:m=”http://schemas.microsoft.com/ado/2007/08/dataservices/metadata&#8221; xmlns=”http://www.w3.org/2005/Atom”&gt;
<content type=”application/xml”>
<m:properties>
<d:RunbookId m:type=”Edm.Guid”>$rbid</d:RunbookId>
$rbserverstring
$rbparamstring
</m:properties>
</content>
</entry>
“@

# Create a request stream from the request
$requestStream = new-object System.IO.StreamWriter $Request.GetRequestStream()

# Sends the request to the service
$requestStream.Write($RequestBody)
$requestStream.Flush()
$requestStream.Close()

# Get the response from the request
[System.Net.HttpWebResponse] $response = [System.Net.HttpWebResponse] $Request.GetResponse()

# Write the HttpWebResponse to String
$responseStream = $Response.GetResponseStream()
$readStream = new-object System.IO.StreamReader $responseStream
$responseString = $readStream.ReadToEnd()

# Close the streams
$readStream.Close()
$responseStream.Close()

# Get the ID of the resulting job
if ($response.StatusCode -eq ‘Created’)
{
$xmlDoc = [xml]$responseString
$jobId = $xmlDoc.entry.content.properties.Id.InnerText
Write-Host “Successfully started runbook. Job ID: ” $jobId
}
else
{
Write-Host “Could not start runbook. Status: ” $response.StatusCode
}

This doesn’t quite format properly so you can get more info here.

You will need to replace the $rbaid with your runbook ID.  An easy way to do this is using Power Pivot in Excel.  Here is a good article on how to do that.  Take note of the warning that all of the GUIDs must be lowercase or it will return an error.

There you go, half of your administrative job for Orchestrator is automated.  Stay tuned for automated SCSM data warehouse fixes that can be triggered from the portal so you no longer have to worry about reports not working.

 

If you would like to download this project for yourself and customize it to your environment please feel free and let me know how it works out for you.

I have published the export on Technet for your consumption.

Orchestrator Maintenance.ois_export

 

 

Advertisements

One thought on “Orchestrator Maintenance Automated

  1. Pingback: Orchestrator Health Checker – System Center

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s