Jason Hand - September 17, 2015
VictorOps has a variety of ways to ingest data used to alert the people and teams that need to know about infrastructure and code issues or events. Traditionally, teams have used the native integration endpoint specific to their monitoring solution. Those who have rolled their own monitoring solution or have yet to request an integration with their favorite monitoring service often utilize the standard REST Endpoint or webhook option to send important information to VictorOps.
A very simple way for teams to send important information to VictorOps is via e-mail endpoint. Each VictorOps implementation is given a unique (randomized) email address that can be used to transmit data in to the VictorOps timeline. Once the email is received an incident is created and the person or team best suited to acknowledge and respond is alerted via push notification, SMS, email, or phone.
Some have taken it upon themselves to create applications or plugins and make them available for anyone to use (ex. RubyGem and NodeJS). Coupled with a chatbot such as Hubot, Lita, or Err, this is a popular way to manually generate incidents as well. However, not all teams have the resources to build their own application or chatbot to generate incidents at any given moment.
Using the popular (API As a) Service Zapier, it’s super easy to manually create alerts via email ingestion and trigger the whole thing from a simple command initiated within a group chat tool like Slack or HipChat.
Behind the scenes, all that really needs to take place is that an email is generated with certain information included within the subject line and body of the message. However, we want to kick this event off directly from within our chat client. After all, that’s where the conversations about our infrastructure and codebase are taking place.
In the below example, we’ll use Zapier and Slack to manually generate an incident within VictorOps that then alerts the primary person on-call for a specific team. We’ll step through the process to manually generate an alert simply by typing a specific, yet simple string of characters in your favorite Group Chat client. By the end of this tutorial, you’ll have taken a very simple step down the path of ChatOps and improved incident management.
(This example uses Slack as the Group Chat client, but the same interaction can be built using HipChat or other chat tools as well.)
We want to be able to enter the text “create new alert” from within our chat client and automagically alert the right person or team that can respond to the alert.
If you don’t have an account already with Zapier, start there and return back to this article to continue with Step 1.
The first step of the Zapier process allows you to select two services (i.e. triggers) that you’d like to interact with each other.
First, we’ll choose Slack and select the ”New Message Posted” option from the dropdown of available actions. This indicates to Zapier that when a New Message is Posted, we (may) want to take an action. I say (may) because we’ll establish specific filters in a later step to determine IF the action should take place.
Next, we’ll select Email by Zapier and the ”Send Outbound Email” option from that trigger’s action list.
In step two, all you need to do is connect or test your integration with the Group Chat client (i.e. Slack). Once Zapier has verified that it can connect to your Slack account, you’re ready to move on to Step Three.
This step is simple. In fact, you don’t have to do anything since Zapier is going to handle the sending of email for you.
Step Four is where things get a bit more interesting. This is where you determine what the secret trigger will be from within Slack that will create the incident for you. This is the IF portion I mentioned above.
In the example, I have configured it so that when a user enters the text “create new alert” in the Slack Channel named “victorops-demo”, Zapier will then move on to Step Five to continue processing the rule.
Now we need to tell Zapier exactly what email address the information should be sent to, as well as what will be contained within the subject line and body of the message. As I mentioned earlier, a unique and randomized email address is available for ingestion of data. This can be found in the Integrations section of VictorOps. We will enter this address in the “To:” field.
[NOTE: I have added the text “+slack_alert” to the rear of the email address (before the @). This is used to create a specific routing key. Routing keys allow for specific routing of alerts to specific teams. For example, if I want the database team to be alerted about something that is clearly related to our databases, I could add “database” as the routing key. From within VictorOps, you can set it so that when something comes in with “database” as the routing key, it automatically alerts the Database Team, rather than Susan (who happens to be on-call at that time). Being able to route alerts to the right person at the right time can dramatically reduce your Mean Time To Resolution (MTTR).]
In our “Subject” field, I’m going to enter “New (Slack) alert”. We can set up rules within VictorOps to key in on the subject line so that very specific actions occur once it hits the timeline. For now, I’m going to keep the subject line fairly basic, but feel free to get a bit more creative here.
In the body field, I’m going to enter text that I want sent within the alert. This will show up in the payload of the message for the on-call person to view when they are alerted.
In addition to that, using the Transmogrifier, I could create awesome rules that look for specific text within the subject or body and automatically provide me the context (i.e. graphs, runbacks, notes, etc.) that I need, right in the alert itself.
Having the answers to how to solve an incident baked right into the alert itself is a huge win for on-call individuals and teams and a major differentiating feature between VictorOps and all other alerting services available today.
In step Six, all we need to do is test out the trigger.
[NOTE: You may need to hop over to Slack and type in a sample alert (text) in order for Zapier to recognize the filter you created.]
For example, I would head over to my “victorops-demo” Slack channel and type “create new alert” before this step. Once that text exists within the channel, Zapier can then recognize it and give you the option to test the trigger.
That’s it! Now just give the trigger a name and turn it on. Once you’ve completed all seven steps you can return to Slack and test out the new trigger by typing (in this case) “create new alert”.
Within a few seconds, you’ll see the alert show up within the timeline of VictorOps and immediately the system will begin alerting the appropriate team or person on-call, depending on your unique VictorOps setup.
Now that you understand the process, you can create multiple alerts for different scenarios. Perhaps you want to manually create an alert that is related to a network issue. All you have to do is make a few changes in steps 4 and 5 in a brand new Zapier trigger.