When a new on-call process is first introduced at a company, it can be a bit contentious trying to figure out who should fill the on-call role. Many people shy away from the idea that developers and other stakeholders should hold a seat in the on-call rotation, but that is in fact the best and most efficient way. Our customer AlienVault is the perfect example of successfully introducing on-call across an entire company.
Because problems normally stem from an issue in the code, it doesn’t make sense that a support team member should own the entire on-call process. Some companies are successful only having developers on-call; other team structures lean toward support teams as the first line of defense for on-call incidents, using an incident management tool to quickly pull in a developer if the issue is complex.
Regardless, in order to champion both innovation and uptime, today’s developers need to do more than create, toss, and forget their code.
Once you add developers and stakeholders to your on-call group, you can quickly lose control of your escalation policy. And, as Ernest Mueller at AlienVault knows, velocity is crucial—it’s important to use an incident management tool to help manage complex calendars, on-call schedules, and escalation policies.
AlienVault has team members around the globe with different working and seasonal hours. With the help of VictorOps, teams can set complex rules so the correct people on the first-tier product teams receive the alerts when they’re working or on-call. Being able to control communication and escalation policies ensures that AlienVault customers receive a high-quality level of service, and employees can maintain a high quality of life.
With VictorOps in place, stakeholders can receive notifications that matter to them without having to actually be on-call. Communication is easy between developers, management, and other team members with bi-directional integrations between VictorOps and ChatOps tools like Slack and HipChat.
At the end of the day, on-call just doesn’t work if your rules and communication are broken. By using VictorOps, companies like AlienVault are confident in their incident management and enjoy the perks of open and clear collaboration.
The full AlienVault case study describes how the company focuses on collaboration and shared responsibility while relying on the rules engine of the VictorOps Transmogrifier. Read the write-up to see how it happened.