Talon Poole - March 15, 2016
Prior to my current and happy employment at VictorOps, I was not required to be on-call for anything. It was a concerning realization that I would be forfeiting the latter. Being on-call was a mystery to me and mysteries make me anxious.
When a mystery crops up–especially at work–I usually accept it and then attack it. This tactic was not an option. The only way to know about the experience of on-call is to experience on-call. I had to wait.
Fortunately I had a guide into this new world. A friendly coworker was by my side as I learned what it means to be on-call. And I did learn what it means, which is not much. It was inconsequential to me as an individual. This was a profound realization and my motive for sharing this story. Being on-call is okay.
I’m not anxious or afraid to be on-call now. It’s a very passive thing for me. I get a notification from the VictorOps app saying that I am on-call and usually the only other notification is me being off-call. Not mysterious or anxiety-inducing at all.
Occasionally an incident will happen while I’m on-call and it becomes an opportunity for me to learn more about the platform and to let my teammates sleep. When the incident isn’t trivial, it is clear who to ask for help, especially since we’ve done a pretty thorough configuration of alert annotations using the Transmogrifier.
Using our own product to handle on-call rotation was an excellent experience in eating our own dog food. It increases my empathy for customers, which allows me to do my job more effectively.
My tips for those who are not looking forward to their first on-call rotation:
– Use VictorOps.
– In lieu of the above, have a system or process in place that will help you to feel confident when you’re on-call.
– Runbooks are cool. Put them to good use.
– Keep lines of communication open. Don’t be afraid to talk to your team members about your anxieties and whether they have suggestions for you.
– Ask questions. If an alert comes in that you can’t do anything about, ask why. When there’s not a runbook outlining what to do, ask why.
– Embrace the fear and get comfortable being uncomfortable. It’s called growth.
When your name comes up in the on-call scheduling software for the first time, don’t panic. It’s a learning experience and if your team is there to support you, you’ll get through it.