When an outage occurs, it’s all hands on deck until the problem is solved. In the event a problem is defined as high impact, and requires a response that is above and beyond that of a normal incident, it may be time to start a Control Call.
Control Calls can be integral to speedy resolution, so here’s a nifty checklist of control call best practices you can add to your team’s Standard Operating Procedure for the next time you really need it. Enjoy.
Have a Plan
- Before you’re in the thick of it, build a process around who should be on these types of calls. Too many cooks in the kitchen is never a good thing, so be sure you have all instrumental constituents present. After all, you don’t know what you don’t know.
- Create an “Incident Commander” rotation within your VictorOps teams. Alerts will be re-routed to this team when initiating a control call is necessary. This person is responsible for: initiating the call, quickly developing incident objectives, managing all incident operations, determining the application of resources, and documenting actions taken for inclusion in follow-up retrospective and runbooks.
During the Call
- Call Etiquette: Announce yourself when you enter and leave, mute your line when you aren’t talking, add an additional pause to the conversation to allow for mobile and conference bridge delay.
- While members of your team continue to join the call, ensure the primary on-call team has a predefined set of valuable context (data points, graphs, etc.) readily available when the call begins. If you’re using alert annotations (or the VictorOps Transmogrifier), this information will come embedded with your alert. Whew.
- Once all participants are present, ensure all supporting chats are documented for reference during and after the incident. For example, directly within the VictorOps timeline via native chat, or any of VictorOps’ integrated chat platforms.
- A concise communication plan is key. Monopolizing the conversation can be detrimental to reaching resolution. Again, an Incident Commander is crucial to maintaining structure.
- Designate a “scribe”. This participant should be transcribing important highlights for future use and overall transparency. This can be done directly in the VictorOps timeline for ease of reuse.
- All participants are personally responsible for documenting their action plan for resolution during the call.
After the Call
- Once the control call is complete, the Incident Commander should schedule a retrospective that takes place shortly after the incident in order to ensure the team can reflect on the incident and determine necessary actions to take moving forward. If you’re using VictorOps, all documentation (call start/end time, participants, conversation notes, etc.) relevant to the incident will be surfaced in your incident’s post-mortem report.
Interested in learning more about the VictorOps control call feature? Our support team is always available to chat. If you are already a customer, sign up for the beta program today, or get started with your 14-day free trial!