VictorOps is now Splunk On-Call! Learn More.

Incident Response Automation 101

Kelsey Loughman September 09, 2019

DevOps Monitoring & Alerting On-Call Collaboration
The Incident Response Automation 101 Guide Blog Banner Image

While many people in DevOps and IT operations think automation can fix any problem, when it comes to incident response, they’re actually not wrong. Automation won’t solve problems with processes and technology by itself. But, effective use of automation will help people respond to incidents and find solutions faster.

Incident response is not only about identifying and incident and working through the real-time firefight. You need to analyze incidents retroactively, learn from mistakes and prepare better ways to respond in the future. Automation in incident management should be used to address the pain points you discover and implemented as a way to better connect humans with their systems. Our 101 guide to incident response automation will walk you through best practices for using automation throughout the entire on-call incident management process.

Why automate incident management processes?

A lot of teams will skip this question. Why are you automating a certain workflow or process? Automation for the sake of automation just adds confusion to the system and might not actually improve the team’s efficiency. If you simply spend all day automating different tasks, you’re taking time away from developing future value. So, think about automation in terms of people – how automation can improve the day-to-day efficiency of everyone on the team.

Now that you know how to think about automation in the system as a whole, how does automation fit into the incident management lifecycle? From initial on-call notifications to automating fixes for common recurring issues, DevOps and IT teams are using automation to streamline the entire workflow. So, without further ado, let’s dive into the areas of the incident lifecycle that benefit most from automation.

Dev & Ops Incident Management

Areas for incident response automation

Highly observable applications and infrastructure rely on visibility across all aspects of software delivery and incident management. Automation can be used to improve the way DevOps and IT teams act upon the information they get from all of their observability metrics. Aggregate monitoring of logs, metrics and traces alongside on-call schedules and alert automation turns highly observable systems into highly actionable systems.

  • Monitoring and health checks

Automation in the monitoring and health check process can drive faster incident detection. With automated health checks linked up to your alerting system, you can quickly notify on-call responders in case of an emergency. With automation, you don’t need to rely on a person to regularly check on monitoring metrics – distracting from future development and creating a manual process with more room for error.

You can set specific thresholds around all of your monitoring data and automate alerts based on those numbers. For instance, if you have a common ETL issue that normally self-corrects within 10 minutes, you can set an alert to go out after 10 minutes if the issue persists. Effective thresholds and automation can quiet many of a team’s unnecessary alerts without reducing overall visibility or coverage. With better monitoring and optimized health checks, you’re making on-call suck less while ensuring effective 24/7 coverage.

  • Alert routing and escalations

Once an actionable incident is detected, it should be immediately sent to the right person to fix the issue. Automation plays a huge part in alert routing and escalation processes, getting notifications to the right people faster. Instead of sending alerts to a single location where people spend time rerouting alerts based on who they think should handle the issue, the system intelligently does it for them. Automating alert routing and escalation opens up more resources (time and money) for developers and IT operations to focus on feature development and deployment.

As you can see, automated health checks directly tie into automated alerting and escalation. Based on logical automation rules and a comprehensive incident response plan, your engineering and IT teams can drastically reduce the time spent detecting, triaging and investigating incidents. And, the right people receive notifications more often for issues they can actually address. But, how can automation also improve the actual firefight and remediation strategy once the appropriate on-call responder is notified?

  • On-call incident response

Automation in monitoring and alerting is more than simply serving an alert to the right person at the right time. It’s about providing the appropriate context and instructions to make the firefight even easier. Can you automatically attach related logs, graphs, charts or runbooks that show exactly what’s happening with the system? And, many issues in DevOps and IT don’t simply affect one small part of the overall service. How can cross-functional teams collaborate in real-time with the applicable information and share visibility across all applications and infrastructure?

These questions aren’t easy to answer. And, the answer will be different for every team based on the services they build and the way their team’s organized. But, the core of any good on-call incident response system will focus on transparency and collaboration. The better you are at quickly surfacing helpful context and sharing that information with applicable teammates, the better you’ll be at responding to production incidents. So, find ways to automate the creation of conference calls or Slack channels for firefighting, or develop a system for automatically attaching logs to certain alerts.

There are numerous ways to automate the on-call incident response process. But, it requires you and your team to take a hard look at how you operate today, the ideal state of how you’d like to operate and the resources at your disposal.

Integrating automation into DevOps

Automation in incident response reduces downtime, improves MTTA/MTTR and makes customers happier – while also making on-call suck less for employees. But, automation can also be used early in software delivery to improve the quality of releases and create a more proactive system for reliability. A DevOps culture of collaboration and transparency will give developers more exposure to production environments and gives operations teams more exposure to the development pipeline.

Automated testing earlier in the software development lifecycle (SDLC) can help expose vulnerabilities and points of failure. Therefore, on-call teams will be working with more reliable software, leading to fewer customer-affecting production incidents. From planning to production, automation drives faster delivery of reliable features and makes on-call suck less.

Real-time incident response made easier

If you take nothing else from our incident response automation 101 guide, remember that automation in incident response is focused on people. Incident response automation isn’t focused on removing people from the equation, it’s helping people work on the right problems – driving more customer value. Automation can be implemented at any stage of the SDLC or the incident lifecycle to improve the way DevOps and IT teams detect incidents, notify on-call responders and fix issues.

Preparation and automation in real-time response is required for maintaining consistently reliable services. Customers expect constant uptime and depend on your service for their own business. And, in the world of CI/CD, the only surefire way to maintain uptime is through highly-effective incident response. Leverage automation, collaboration and transparency throughout all of software delivery and on-call incident response to drive revenue, keep customers happy and avoid employee burnout.

Learn how a centralized solution for on-call scheduling, automated alerting and communication can lead to rapid incident remediation. Sign up for a 14-day free trial or request a free personalized demo of VictorOps to make the most of your monitoring and intelligently add automation to your on-call and alerting processes.

Let us help you make on-call suck less.

Get Started Now