SCOM Alerts for Systems that don’t Integrate with SCOM


Everybody has those systems that wont directly integrate into SCOM for some reason or another.  Usually due to no management packs available for that system or the 3rd party management packs are just too expensive for your company ◔_◔.  Well I have a solution for that.

If your system can send emails then we can get them in SCOM and in turn get them in SCSM.  Most systems have a pretty robust reporting system that can send informational emails or critical system emails.  What we did was setup a single use account called svc.alerts with an email account.  In this email account you can set up multiple aliases for every system you have.  We set up 2 aliases per system.  A alerts.system@company.com email and a incidents.system@company.com email.  For example the svc.alerts email would contain two aliases, alerts.oracle@yourcompany.com and incidents.oracle@yourcompany.com.  We liked to capture information alerts in SCOM just for reporting purposes and close them on creation.  I thought that was a little much but hey, I’m just the developer ;).  The incidents.system@company.com emails would create a critical in SCOM and in turn create a ticket in SCSM.  So lets break this runbook down for consumption.

For this project you will need the Exchange User IP that can be downloaded from here and the Data Manipulation IP from here.

This is the basic folder structure I’m using.  Don’t worry about the Auto Close Alerts.  That project for another day.

FolderStructure.JPG

Inside the SCOM Alerts folder we have a monitoring runbook that checks the inbox every 3 minutes.  The reason I have this monitor outside of the main runbook because it allows us to spin up multiple concurrent runbooks that will handle the emails and can dramatically increase the capacity and speed at which emails are handled.  You will understand this a little more when we get to the main engine in this process. This project has handled up to 20,000 emails in one month with only minor issues.  It’s excessive, I know, but again, I’m just the developer here.

EmailMonitor.JPG

 

This is a very basic runbook.  The monitor is set to fire every 3 minutes and the invoke runbook is just calling the secondary main runbook that is handling the process.Invoke.JPG

As you can see I’m on version 4 of this process.  I was not anticipating the load that was coming my way when word got out that we where capable of doing this.

 

Now for the main engine.

MainEngine.JPG

We start off with a Get Item IP from the Exchange pack.

GetItem.JPG

I don’t actually have an Exchange environment in my lab so yours will look a little different when its connected to the service account’s inbox.  You will see an additional property called Search Folder.  That should be set to Inbox.  On the Filters tab you want to set a filter of “Is Read Equals False”.  This will prevent the runbook from grabbing emails it is already processing.  It can take up to 30-45 seconds to create the SCOM alert so you don’t want the runbook duplicating alerts.  No need to cause panic with multiple incidents firing all at once.

The Update Item IP will flip the email to read.  See how that works.

UpdateItem.JPG

Again you will have the ability to add additional properties to this IP.  We will need to add the ID property and plug in the variable ID from Get Item and Is Read should be True.

Now we will run a nifty Powershell script that will parse out any HTML that your fancy system might be including in the email.  We don’t need that in a SCOM alert.

PSScript.JPG

$html = @’
{THIS IS THE VARIABLE  “Body from Get Item”}
‘@

# remove line breaks, replace with spaces
#$html = $html -replace “(`r|`n|`t)”, ” ”

# remove invisible content
@(‘head’, ‘style’, ‘script’, ‘object’, ’embed’, ‘applet’, ‘noframes’, ‘noscript’, ‘noembed’) | % {
$html = $html -replace “<$_[^>]*?>.*?</$_>”, “”
}

# Condense extra whitespace
$html = $html -replace “( )+”, ” ”

# Add line breaks
@(‘div’,’p’,’blockquote’,’h[1-9]’) | % { $html = $html -replace “</?$_[^>]*?>.*?</$_>”, (“`n” + ‘$0’ )}
# Add line breaks for self-closing tags
@(‘div’,’p’,’blockquote’,’h[1-9]’,’br’) | % { $html = $html -replace “<$_[^>]*?/>”, (‘$0’ + “`n”)}

#strip tags
$html = $html -replace “<[^>]*?>”, “”
# write-verbose “removed tags: `n`n$html`n”

# replace common entities
@(
@(“&amp;bull;”, ” * “),
@(“&amp;lsaquo;”, “<“),
@(“&amp;rsaquo;”, “>”),
@(“&amp;(rsquo|lsquo);”, “‘”),
@(“&amp;(quot|ldquo|rdquo);”, ‘”‘),
@(“&amp;trade;”, “(tm)”),
@(“&amp;frasl;”, “/”),
@(“&amp;(quot|#34|#034|#x22);”, ‘”‘),
@(‘&amp;(amp|#38|#038|#x26);’, “&amp;”),
@(“&amp;(lt|#60|#060|#x3c);”, “<“),
@(“&amp;(gt|#62|#062|#x3e);”, “>”),
@(‘&amp;(copy|#169);’, “(c)”),
@(“&amp;(reg|#174);”, “(r)”),
@(“&amp;nbsp;”, ” “),
@(“&amp;(.{2,6});”, “”),
@(“&nbsp;”, ” “)

) | % { $html = $html -replace $_[0], $_[1] }

$plaintext=$html

Remove the bold part and add the appropriate variable there.

And the published data section will look as so…

publisheddata.JPG

Now we plug in a Match Pattern IP so we can see if the email TO address contains an incident or alert word.

match.JPG

This will determine the path that the runbook will follow.  We set this in the link properties.

link one.JPGlink2.JPG

 

We will follow the incident path first.  We want to remove the incident. from the email TO address.  This will make sense in a minute.  We will use a Replace Text IP to do this.

replaceincident.JPG

As you can see we are replacing it with nothing so it will blank it out.  Next we do this again but to remove the @companydomain.com section of the email TO address.

 

removedomain1.jpg

Now we are left with just the application name remaining from the TO address so lets create that critical alert in SCOM.

SCOMAlert.JPG

As you can see, we are placing that application name that we parsed from the TO address and placing it in Custom Field 4 of the SCOM alert.  You can use any custom field in the alert we just like 4.  That will now allow us to route the ticket in SCSM to the appropriate team’s queue.  Is it making more sense now?

Next we delete that email.

delete.JPG

You will need to add an additional property here called ID and pass in the variable ID from Get Item.  Again I don’t have Exchange in my home lab so I cant show you that part.  If you want to donate some money so I can get a bigger server that would be awesome but alas, I don’t have a way for you to do that so moving on.

As you can see in the pic there is a branching path from the Create Alert IP.  This is some error handling I put in because I’ve noticed every once in a while the IP would fail to create the alert or fail but still create the alert.  I’m not a SCOM guy so I’m going to go ahead and blame that on SCOM.  Following that path we use a PowerShell script to get the current UTC time.

get time.JPG

Pass that result to a Format Date/Time IP and subtract 30 minutes.

format.JPG

Next step is to use a Get Alert IP to check if the alert was actually created.

checkalert.JPG

You can see we are looking for that exact description and the time raised must be with in the last 30 minutes.  This isn’t an exact science but it works for us.  If it finds the alert then it just deletes the emaildelete.JPG

Again, add the ID property and pass in the ID of the email so it only deletes that one email.

If it doesn’t find the alert it marks the email unread so it will go back through the process.

unread.JPG

And again, add ID to the properties and add the email ID.

The Alerts path is the same thing but with a few minor exceptions.  on the Create Alert IP instead of Critical we open that as an Informational.

information.JPG

And we use an Update Alert IP to close it out.

Close.JPG

Now set the Job Concurrency on the runbook by right clicking the the tab and bump that up to something like 3 depending on load.  Now you might be wondering why the monitor and the main runbook are separate from each other, well that’s because If we had the monitor in with the main runbook then we would decrease the efficiency of this process.  Say you have 100 emails come in all at once and the monitor runbook was part of the main runbook.  It will grab all of those emails and start processing them which could take quite some time.  During that time you have several more emails come in and they just sit there piling up until the runbook finishes.  You can see how this can start to create a huge backlog.  By separating the monitor from the main runbook and increasing the  job concurrency of that main runbook you increase your capacity by quite a bit.

You might notice the invoke runbook off of the Get Item IP on the main runbook.  This is not a requirement but if you have a particularly frisky system that likes to announce to the world every single thing it does then you can add a link filter based off of the TO address and send all those emails to its own runbook which is just a copy of the main runbook but only processes  emails from that particular system further decreasing wait time and the possibility of emails stacking up.

Now you can revel in the glory of saving your company that 3 grand they didn’t want to spend on that 3rd party management pack.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s