VictorOps is now Splunk On-Call! Learn More.
People are expecting streamlined mobile experiences for nearly anything and everything. Everyone’s searching for mobile solutions to all of their daily responsibilities – from social media to productivity and business-oriented apps. According to an article from Perficient Digital, 58% of all website visits in 2018 came from mobile devices. On-call scheduling, alerting and incident response for DevOps and IT is no different. Software engineers and IT professionals are looking for ways to use their smartphones in order to receive contextual alerts in real-time and mobilize on-call teams from any location.
VictorOps was purpose-built for use across all kinds of devices and for DevOps and IT teams of any shape or size. With our mobile-first mindset, we’ve learned a few things about using smartphones to make on-call suck less and surface appropriate alert context faster. But, without proper management of your smartphone’s settings and specific on-call requirements, you could still drop alert notifications or create gaps in on-call coverage.
So, we decided to put a little guide together to help you make sure your phone’s settings are properly set up and share some tips and tricks for using mobile apps to improve alerting and incident response.
Uptime and rapid incident remediation is a necessity for resilient applications and IT service management. Knowing exactly what’s happening with servers, networks and applications as soon as an alert comes through can help on-call teams mobilize faster and begin working toward a resolution. Monitoring and alerting tools are becoming more and more mobile-friendly and helping incident response engineers surface actionable context in real-time.
While many complex incidents aren’t resolvable through a mobile app, the ability to understand a problem at a glance can lead to faster triage and emergency mobilization. At a glance, the initial on-call responder can at least assess who should be involved with a problem, escalate the issue straight from his/her phone and start collaborating around a resolution. If someone gets an alert notification while driving, they can easily reroute the problem or use escalation policies to ensure the next person gets notified in a timely manner.
But, mobile incident response is only effective if an end user’s smartphone is set up and optimized for actionable notifications. So, let’s explore some ways to be mindful of your phone and mobile app when you’re on-call.
Mobile mindfulness simply means you’re ensuring appropriate notifications are coming through when you need them to. And, you have access to the right tools and information when you need it. A number of situations can come up where notifications won’t pop up on your smartphone when you’re expecting them (e.g. Airplane Mode, poor internet connection, improper notification settings, etc.). It’s important to be cognizant of your phone’s settings and some functionality in your on-call mobile app (i.e. VictorOps) that can help you ensure consistent on-call coverage and appropriate notification when alerts come through.
So, here are a few simple things you should check whenever you’re on-call to ensure alerts don’t go unacknowledged and incident response teams start taking action when they need to.
Basically, whenever you’re on-call, your phone should only be in Airplane Mode when you’re, well, on an airplane. And, in this case, you wouldn’t be able to acknowledge a problem anyway unless you’re connected to an airplane’s WiFi network. Notifications need to come through at any time while you’re on-call, so it’s important to not put your phone in Airplane Mode or Do Not Disturb.
While you can often override the silent function on your smartphone, you might not have the ringer volume turned up. We’ve encountered on-call engineers who didn’t notice an alert as soon as it came in simply because their ringer volume was turned all the way down.
The entire point of the on-call app is to notify you of on-call schedules, rotation changes, incoming alerts, messages, etc. So, needless to say, you should ensure that all notifications are enabled when using an incident response app. If you’d rather use the app for communication and firefighting instead of alerting, you can adjust personal paging policies to be notified via SMS, phone call or email instead.
If notification settings aren’t enabled properly, users can get alert notifications silently delivered to their phone. When this happens, you can see in the main search menu on your smartphone that a notification was sent to the phone but was never surfaced on the lock screen or on the home screen. Also, if the phone is simply in silent mode and you haven’t enabled the silent override in the VictorOps app settings, it could only vibrate when you get an alert – potentially leading to more missed alerts.
On-call incident response mobile apps put the power of alerting and incident management in the hands of every engineer. Network operations centers (NOCs) no longer need to handoff a physical phone when on-call shifts change. With purpose-built mobile apps, users across multiple teams and departments have access to real-time monitoring data, historical incident context and collaborative messaging solutions directly from their own smartphones.
With a number of smartwatches, you can enable VictorOps notifications or you can receive phone calls or SMS notifications. This is especially useful when you’re in a situation where you don’t want to bother others near you. We’ve talked with married users who use Apple Watch notifications as a way to get alerted in the middle of the night without having a loud ringer go off that could potentially bother/wake up their partner.
Set up paging policies in a way that makes sense for your lifestyle and preferences. Would you prefer to only get email notifications between 9 AM and 5 PM while you’re in the office? Would you see notifications faster if you received SMS and phone calls in the middle of the night instead of push notifications? You can set up personal paging policies based on time of day and day of the week to help you consistently make on-call suck less.
It’s easy enough to encounter over-alerting from infrastructure and applications, let alone from personal apps on your smartphone. Are there games or other mobile apps that are sending unnecessary notifications that you can silence to improve alert visibility? Taking the time upfront to optimize your smartphone’s notification settings can seem tedious at first but it will save you lots of time in the long run.
Turn off incident preview in the VictorOps mobile app settings. This way, push notifications won’t share too much information to the lock screen if you’re concerned about security or compliance.
When a major incident strikes, it’s likely that you’ll be inundated with alerts from multiple monitoring tools. Acknowledge what you can but focus on the real-time firefight. If the timing is appropriate and you’re already deep in the incident response tool and working toward a resolution, try putting your smartphone into Airplane Mode for the time being to silence numerous alerts. Or, use maintenance mode directly from the mobile app to silence incoming alerts and focus on the major incident at hand.
In the VictorOps mobile app, you can add VictorOps to your contacts list. This way, you can favorite VictorOps as a contact, allowing you to use Night Mode if you’d like to. Even in Night Mode, with VictorOps as a favorite contact, SMS or phone call notifications will come through in the middle of the night while other apps will be silenced. But, make sure you’ve turned up the ringer volume on your smartphone before heading to bed.
More and more functionality is being pushed out through mobile apps for DevOps and IT. Distributed teams and a life-on-the-go mindset is forcing technology to catch up with the people maintaining these services. People want to work seamlessly between mobile interfaces and web interfaces with the appropriate context and ability to collaborate. Across multiple devices and tools, DevOps and IT engineers are finding ways to share visibility into service health and development and incident response workflows.
As mobile operations become more commonplace and technology evolves, on-call incident management will be centered around people’s mobile devices. If on-call users can receive timely notifications, triage incidents, collaborate in real-time and digest the appropriate incident context directly from their smartphone, there becomes less and less of a need for fixing issues from a laptop. Playbook orchestration, ChatOps and other DevOps automation is allowing teams to execute rollbacks and incident remediation strategies directly from chat clients and mobile phones. A streamlined incident response process between mobile and desktop isn’t the future of on-call incident management, it’s the present.
See how the VictorOps mobile app for on-call incident response is already driving faster incident remediation and streamlined workflows across multiple devices and teams. Sign up for a 14-day free trial or check out a personalized demo to see how you can start making on-call suck less.