VictorOps is now Splunk On-Call! Learn More.

The Definitive Guide to DevOps Incident Management

Brad Griffith September 17, 2019

DevOps Monitoring & Alerting Collaboration
The Definitive Guide to DevOps Incident Management Blog Banner Image

Software developers and IT professionals alike are spending more time in production environments – detecting anomalies in performance and fixing issues in real-time. Instead of writing code and deploying new updates on a monthly, quarterly or even yearly basis, software companies are now releasing multiple deployments each day. Continuous integration and deployment (CI/CD) of cloud-based services has led to more deployments, faster delivery of customer value and a need for real-time incident management and response in production.

So, engineering and IT teams began to adopt DevOps principles – testing earlier in the SDLC, tightening feedback loops between developers and IT professionals and sharing accountability for service reliability. We’ll walk through a brief history of IT service management, the steps of the incident management lifecycle and the way DevOps is being used to optimize the delivery and upkeep of software.

ITSM, ITIL, and the evolution of IT operations

Throughout the late 1970s and 1980s, computers became more widespread and IT professionals and software developers started building more and more applications and services. Then, the invention of the internet and personal computers further sped up the growth of software development and adoption. But, there was no standardized methodology for creating, releasing and managing IT infrastructure and applications. So, the IT Infrastructure Library (ITIL) was created as a framework of best practices for delivering IT services that are aligned with the business.

IT service management (ITSM) is the actual day-to-day implementation of ITIL and execution of all related IT service development and maintenance. For many years, before the internet, CI/CD, cloud-based technology and Agile software development, ITIL was held up as the gold standard for how IT teams needed to manage their workflows. Over time, ITIL has been updated every few years to keep up with changing technology and development practices.

Today, ITIL still contains a number of great practices for ITSM and software development. But, every business is different and the range of technology is so vast, there can’t be one standardized practice for software development and delivery. So, IT operations teams are evolving – taking the best parts of ITIL and combining them with DevOps principles – learning to work more efficiently with software developers.

The incident management lifecycle

In a world of CI/CD, code ownership and proactive testing, the only real way to maintain uptime is through an organized plan for incident response and management. IT operations aren’t allowed to stop developers from continuously releasing new applications and services. But, software developers are no longer allowed to ship code whenever they’d like without any accountability. Developers, sysadmins and IT security professionals all have a responsibility to improve service resilience, limit vulnerabilities and handle on-call incident management.

So, let’s look at the stages of the incident management lifecycle and learn some best practices for alerting and real-time incident response.

Invincible Incident Management


Without a detection strategy, you won’t be able to expose vulnerabilities or identify production incidents as quickly. Continuous monitoring of the right metrics, logs and traces will help you build observable applications and infrastructure, improving overall incident detection. Constant tweaking of monitoring tools, thresholds and alert rules will ensure you’re consistently tracking the right parts of your system and notifying people appropriately. Detection is about more than just monitoring, it’s about understanding the way services interact within your architecture and adding visibility where there is none.


With an adequate detection strategy, you can start looking at how the team responds when an incident occurs. How do you get alerts in the hands of the right person the first time around? What’s the general process for triaging and investigating the incident? How can you get alert context into the hands of on-call responders faster? Incident response is about taking the information you gained in the detection phase, making it highly transparent and finding ways to collaborate around the information in real-time. Automation, transparency and collaboration are the core elements of any efficient incident response process.


Once the appropriate responders have been notified and the pertinent incident data is shared with the right people, incident remediation is about executing the required steps to fix the issue. Can you find more efficient ways to execute commands or rollbacks to facilitate faster incident resolution? Have you tried leveraging ChatOps tools like Hubot in order to run commands or scripts directly from chat applications? Empowering people with the proper knowledge, autonomy and system access will allow DevOps and IT teams to remediate incidents faster and more thoroughly.


Once the service is restored and end-users are happy, you can start to analyze the incident. Post-incident reviews can expose workflow issues, technical problems and areas for process improvement. A centralized tool for incident detection, response and remediation leads to a single source of truth for incident management – offering holistic analysis of a system’s strengths and weaknesses, including technical infrastructure and human communication.


With the right information at your fingertips, you just need to take action. How can you better prepare the system and the team for real-time incident response? What’s the team missing that can make the process more efficient? A prepared incident management team is the only way to ensure service reliability in a world of CI/CD. Through continuous improvement and constant preparation, DevOps and IT teams are equipped with the tools and expertise needed to handle unknown unknowns and rapidly fix problems in production.

The need for continuous integration and deployment (CI/CD)

In order to maintain more robust software, you may ask, “Why don’t we just make larger deployments less often?” And, depending on the type of service you maintain, this could indeed be a better option. However, the rise in cloud-based applications and infrastructure and a more competitive landscape in software simply won’t allow you to deploy code once per month. In order to stay competitive and serve valuable features to customers faster, CI/CD is the only way forward.

IT operations can’t be a roadblock to CI/CD; they need to find ways of integrating themselves into the development process and allowing for frequent deployments while also minimizing downtime. Developers also can’t put the burden of CI/CD solely on the IT team. If the development team works too efficiently and sends work to IT operations faster than they’re able to deploy it, you’re still unable to deliver services to end-users. Agile software development principles are great but they need to flow efficiently into release management and IT operations teams.

Connecting Agile software development and IT service desks

From IT service desks to backend software engineers, the entire team should be bought into the development and release pipeline. Not only should IT be more involved in the development and testing process but developers should be more involved in the upkeep of services once they’re sent to production. Agile processes can be used to increase the velocity of software developers but a shift-left mindset can allow IT professionals to add reliability to the Agile pipeline. Then, when a release goes sideways, both developers and IT operations have already had more exposure to the systems.

Enter DevOps incident management and response

A DevOps mindset is the only way to make the software delivery lifecycle and incident management process more efficient. By tightening the collaboration between developers and IT operations, and giving both of them more exposure to production and staging, you spread deeper organizational knowledge. With deeper organizational knowledge alongside more transparency and collaborative workflows, software is released more reliably and incidents are detected faster. And, when production incidents pop up, the people who actually build the services are responsible for fixing problems.

DevOps incident management is all about constant communication, continuous improvement and shared accountability for service reliability. Developers take on-call responsibilities and respond to production issues related to the services they build. IT operations will run proactive tests through staging and production environments and identify problems before they can affect customers. DevOps improves the speed and effectiveness of incident response while improving the quality of life for engineering and IT teams.

The business case for DevOps incident management

DevOps lowers the costs of downtime, reduces employee burnout from on-call operations and leads to happier customers. CIOs, CTOs, IT managers and engineering managers everywhere should be looking at DevOps as a way to continuously improve interactions between people, processes and technology. The combination of a faster, Agile development pipeline and more resilient IT operations helps you build better services faster and differentiate yourselves from competitors.

Customers are happier as you achieve more available services while constantly deploying new features. Businesses everywhere should capitalize on the benefits of DevOps incident management and software delivery to drive revenue and reduce costs of IT applications and infrastructure.

See how DevOps teams are adding visibility and collaboration to their incident management process. Sign up for a 14-day free trial of VictorOps or request a free personalized demo to build more reliable services and start making on-call suck less.

Let us help you make on-call suck less.

Get Started Now