Being on-call during the holidays can suck. You worry about downtime, missed alerts, alerts lacking important details or runbooks, and the added worry of alerts taking over your holiday festivities.
Let’s be honest, an alert during the dreaded in-law dinner might be a welcomed distraction, but push that alert back just a few hours while you’re playing Santa and suddenly you’re not so jolly.
It’s always a juggling act, and some years are easier than others. To help you get through the next few weeks, I rounded up our internal experts to get their take on how to best avoid disaster this holiday season. Here’s what they had to say:
To survive being on-call during the holidays, make sure the following items are full (at all times):
Lose your phone.
Encourage overrides and shift-trading so people can take breaks during their holiday shift. Take on-call for someone else when you’re going to be visiting relatives you don’t like! Remember, triage might be different during the holidays. Maybe an issue can wait a day rather than sounding the “all-hands” alarm on December 25. If you can put some duct tape over the problem for 24-48 hours, you can keep the holidays festive.
Communicate with your team, and help out where you can to make on-call a breeze! Plus, eggnog.
Have a plan for when you’re on-call. You might get paged while doing something important and knowing where to reroute to beforehand can be a lifesaver.
With today’s news on net neutrality, have sufficient tissues on hand (for the crying) and data in your cell plan (in case your home internet provider decides to slash your VPN connection speeds). During one of your not-so-exciting-extended-family-visits, ask a teammate to create a 'manual incident' and send it to you. A perfect excuse to go and browse the social inter webs instead of that brain numbing conversation with Uncle Ray. Keep your devices (laptop & phone) charged as often as you can. That way you have freedom to find a comfortable place to firefight—or easily move from room to room as your environment changes around you.
Aside from making sure you still keep enjoyment in the holidays, the underlying theme from our experts is to plan, plan, plan. Know who is on-call before the holidays start, and be sure you have all your systems and processes in place so everything runs as smoothly as possible.
It won’t always go as planned—that’s the nature of the beast. Issues always happen. So much so that we turned our employee’s stories into awards:
These nightmares are a rite of passage in the on-call world. But right now, during the stress of the holidays, follow the advice in this post so you don’t end up with another disaster on your list.
Cheers to a happy holidays from all of us at VictorOps. We hope your on-call experience is as painless as possible these next few weeks (and always).
Need help tackling your incident management now and in the future? Read our On-Call Firefight Survival Guide for more tips.