VictorOps is now Splunk On-Call! Learn More.

Runbook Automation Tools and Examples

Marlo Vernon December 10, 2018

Monitoring & Alerting On-Call
Runbook Automation Tools and Examples Blog Banner

Incident response is the name of the game. With 73% of an average incident’s lifecycle spent in incident response, human collaboration and workflows quickly become one of the most important parts of incident management. Runbooks provide on-call responders with the context and instructions they need to lead rapid incident response and remediation efforts. So clearly, runbook automation can lead to even more efficient workflows.

By combining the speed of DevOps with the power of automation, you can build workflows that actually help people. Automated runbooks surface helpful information to on-call responders immediately, shortening the incident response lifecycle and getting someone on-the-scene to quickly start fixing the incident. When running your runbooks, you’ll want to strike the perfect balance between automatically serving helpful instructions at the right time and providing useful information that actually helps people work.

So, this post will dive into the importance of runbooks in general and automation tools that allow you to quickly serve helpful context to your team.

Why You Need Runbooks

Runbooks allow you to understand situations and take action toward resolving incidents. As long as the team actively maintains and updates their runbooks, everyone is privy to the latest information and can quickly process the state of a situation. Whether runbooks are automated or not, they offer helpful documentation for assessing the health of a situation and taking steps to remediate an incident.

However, due to the increasing demands for the continuous delivery of reliable systems in highly integrated environments, automation is becoming more important than ever. Automation in the development and incident lifecycle allows humans to focus on higher-priority tasks and automate lower-level processes. By working automation into your runbooks and incident response workflows, you can spend more time conducting thorough post-incident reviews, optimizing processes, and building reliable services.

Let’s look at the intersection of automation, people, and runbooks to see how they all come together to make on-call suck less.

Running Into Automated Runbooks

Automation fits into runbooks in two ways: 1) updating runbook documentation automatically and, 2) serving runbooks to the proper person automatically. As an incident management solution, we at VictorOps are typically considered with the latter more than the former. But, runbook automation needs to be addressed on both sides of the workflow. The more you can automate, the more you can quickly serve actionable context and instructions to the people in need.

Let’s dive into a few ways you can automate runbooks and develop more efficient workflows.

  • 1) Centralize Workflows

    Define a centralized platform for consolidating alerts, chat history, and system data. By centralizing this information and creating a single-pane-of-glass, you can better take action before, during, and after an incident occurs. Then, you can combine centralized data with runbook automation, increasing overall visibility into service errors and creating better documentation to drive faster incident remediation. Constant iteration and automation of runbooks leads to more effective post-incident reviews and drives operational improvements.

  • 2) Define Processes and Working Agreements

    Define working agreements for the process of building, updating, and using runbooks. Analyze your processes and use your findings to identify areas where automation can patch up pain points in incident response workflows. By leveraging runbook automation, you can spend less time maintaining runbooks and more time using them.

  • 3) Build Action Plans

    Once you know how you should work, define the specific steps of the action plan. Where will documentation live? Do your runbooks contain the pertinent information they need to make incident remediation easier? Make sure you know exactly how runbooks will be served to on-call responders and make sure this action plan is conducive to efficient workflows.

  • 4) Runbooks In-Line With Alerting and Communication Tools

    Once you’ve built a process and established the purpose of your runbooks, you need to make them actionable. By integrating runbook automation with alerting and communication tools, people can easily work through incidents without switching between applications. Runbooks can automatically be served with incident details and context, allowing teams to more quickly identify an issue and start working on a solution. Automated runbooks should be informative and served directly where people are already working.

Runbook Automation Tools and Examples

Now that you understand how automation can be applied to incident response and remediation, we can look at more specific examples of using automation with your runbooks. To help you get started on your journey, we’ve compiled a list of runbook automation tools to help you quickly realize the benefits of adding runbook automation to your workflows.

Runbook Automation Tools

  • Rundeck - Open source automation tool to assist with automating runbook, DevOps, and other IT operations tasks
  • Confluence - Documentation storage and organization
  • ServiceNow/Jira - Ticket tracking for incident management
  • Slack (or other chat tools) - Real-time communication and collaboration engine
  • VictorOps - Incident response and collaboration solution

A combination of these tools will drive the automation, storage, service, and use of your runbooks. Through strategic process management and the right amount of automation, you can surface incident remediation instructions and ensure runbooks are kept up-to-date. That way, the next time an incident occurs, the proper documentation is updated and it’s surfaced to the right person at the right time.

See what you need to build minimum viable runbooks, automate them, and integrate them into your incident management workflows. Download your free copy of The How and Why of Minimum Viable Runbooks to start making the most of your own runbooks today.

Let us help you make on-call suck less.

Get Started Now