VictorOps is now Splunk On-Call! Learn More.
Working to prevent downtime is a never-ending battle. But no matter what you do, in today’s era of continuous deployment and integrated services, uptime is not guaranteed. It would be nice to guarantee uptime, but it’s naive to believe you can avoid outages altogether.
Finding ways to better understand your system and prepare yourself for possible downtime is the best way to maintain high levels of uptime for your customers.
Adhering to SLAs and sustaining uptime relies on an obsessive preparation for incident detection, response, and resolution. DevOps in the incident life cycle creates a culture of collaboration and accountability that improves your team’s overall system knowledge and makes incident resolution easier. When engineers are responsible for maintaining the products they create, there’s a larger focus on writing better code and building more reliable features and services.
Collaborative teams will run through a number of scenarios and use previous incident history to determine the best course of action for managing incident workflows. What should the initial on-call incident response be? What’s the best way to monitor for errors, alert on incidents, escalate issues, and communicate with applicable team members? Asking these types of questions will help you determine how you need to prepare for downtime in the future.
Readiness, the fifth stage of the modern incident management life cycle, refers to continuously improving and preparing for future incidents and outages. The first three stages of the incident life cycle–detection, response, and remediation–directly benefit from a focus on downtime readiness.
So, here are a few tips to prepare yourself for incident detection, response, and remediation in order to make on-call suck less:
Setting up system monitoring to detect and alert on high-priority incidents can get convoluted. Constant iteration of your monitoring tools and alert thresholds will optimize notifications and reduce alert fatigue. Try organizing notification methods based on incident severity, update monitoring tools and thresholds to account for frequent unactionable or self-resolving incidents, and centralize monitoring data for improved visibility.
Always be searching for ways to reduce alert noise and improve the quality of life for on-call engineers. The faster you can detect an issue, the faster you can respond to and remediate the problem.
In our State of On-Call Report, we found that incident response, on average, accounts for 73% of an incident’s life cycle. So, prioritizing downtime preparation and incident response workflows is the most efficient way to cut down on the extensive costs of downtime.
Add as much customization and automation into your incident management system as possible. Automated notification policies, on-call scheduling, rotations, alert routing, and escalation policies can all help cut down on time spent manually responding to incidents. You can learn from past incident response to figure out what worked well, optimize workflows, adjust alerting, reorganize on-call scheduling, and better prepare for future issues.
In a collaborative environment, focus on getting the right alerts to the right people at the right times. Not only does this make the process more efficient, but it makes your team happier with the entire experience of being on-call.
Remediation is always about finding the balance between automation and human involvement. By centralizing incident payload data and human communication, you can provide deeper visibility into issues that are currently being worked. By aggregating your monitoring data, communication history, alert payload information, logs, and charts in one place, you can more easily take action–or trigger automation–to speed up incident remediation.
Make runbooks easily accessible to teammates when they receive an alert. When runbooks are in-line with alert context and chat history, on-call engineers immediately get all the information they need to remediate the issue and restore uptime.
After developing a general incident management process, you need to define methods of communication at each stage of the incident life cycle. What’s the best way to communicate internally about incidents–both during the firefight and after the fact? How do you communicate an issue, and its resolution, to external stakeholders? Are there any ways to use ChatOps–manually or automatically–to improve incident collaboration? When you experience downtime, your team needs to prepare for communicating the incident and its effects to both internal and external stakeholders.
If you normally collaborate during an outage via Slack, do you have a backup plan in case Slack goes down? Do you have a backup plan for external stakeholders if your Statuspage goes down? Incident management platforms such as VictorOps should integrate with communication tools like Slack and Statuspage to backup chat history and incident data, and give you an alternative method for communication.
Establish a hierarchy of crisis communication to make sure anyone affected by an outage is made aware of the problem. Also, be honest when setting timeframes and expectations for incident response and remediation. Your customers obviously want you to restore uptime quickly, and they expect you to be forthcoming with information. In fact, creating open lines of communication with customers can sometimes help you resolve an incident faster.
After uptime has been restored, you need to take steps to make the process easier next time around. Conduct thorough post-incident reviews and talk with your team about what went well and the areas of the incident life cycle that need improvement. Centralizing incident data and chat in one location makes for more comprehensive post-incident reporting. By analyzing finite incident details, you can make more educated, granular changes to incident workflows.
Consolidating data and communication into one report will help with the consistency of uptime and better prepare you for the potential of downtime. You can track overall downtime preparedness by tracking metrics such as mean time to acknowledge and mean time to resolve (MTTA/MTTR) over time. As you become faster at acknowledging and resolving incidents, your system maintains more consistent uptime, and ultimately makes customers happier.
VictorOps is purpose-built to centralize incident detection, response, remediation, and analysis. Start maintaining uptime, preparing for downtime, and making on-call suck less with your own 14-day free trial of VictorOps incident management software.