VictorOps is now Splunk On-Call! Learn More.

Crafting a Comprehensive IT Monitoring Plan

Dan Holloran August 09, 2019

Monitoring & Alerting On-Call
Crafting a Comprehensive DevOps and IT Monitoring Plan Blog Banner Image

Sysadmins, database admins and other IT professionals are constantly tweaking monitoring tools and trying to create more reliable systems. But, IT infrastructure and applications are constantly shifting underneath the people maintaining them – making it hard to maintain robust services. And, to top it all off, microservices, containerized applications, hybrid cloud infrastructure and faster deployment lifecycles are leading to more complex systems. So, how can IT operations build a comprehensive IT monitoring plan that can keep up?

More than ever, IT professionals are pressured into faster software development processes while maintaining uptime for business-critical networks and systems. If the IT team consistently causes delays in deployments, the backlog builds up and business teams can’t provide value to their customers. Then, business teams put more pressure on IT teams to catch up, doubling the anxiety and stress for IT operations.

An IT monitoring plan allows teams to identify production issues faster but also helps IT operations focus on maintaining a more consistent release management and delivery pipeline. Let’s dive into best practices for an IT monitoring plan and see how a comprehensive plan creates more efficient incident response and software delivery processes.

IT monitoring best practices

Eliminating blind spots across applications, infrastructure and software delivery are the first priorities for most DevOps and IT operations teams. Incident detection through well-built IT monitoring practices can lead to greater transparency across all of engineering and IT. Figuring out what’s working and what’s not is the first step in continuous improvement and DevOps adoption.

Let’s look at the five best practices when crafting an IT monitoring plan:

Monitoring & Incident Response

1) Take advantage of internal and external metrics

By monitoring metrics influenced by both internal and external factors, organizations can paint a more complete picture of system health. Internal metrics like throughput, success, error rate and availability combined with external metrics like latency, saturation and traffic can show exactly how customers experience the service. Also, teams can use monitoring metrics to determine the biggest problem areas for the business. Are incidents popping up mostly due to external fluctuations or from the way internal services interact with each other?

2) Establish a “healthy” baseline

What does it mean for a system to be healthy? Is there a certain level of latency that’s acceptable to the end-user? Establishing baselines for monitoring metrics and incident management KPIs is necessary for measuring the performance and success of IT teams, applications and infrastructure.

A “healthy” level should be determined for every metric in order to set up effective alerting and incident response processes. Unfortunately, service health needs to be measured differently from service to service and business to business because different metrics will mean more in different scenarios. Creating visualizations of the way systems, customers and employees interact can help IT and DevOps teams determine which metrics are most important and what a “healthy” baseline should be.

3) Monitor known unknowns and expose unknown unknowns

When creating an IT monitoring plan from scratch, DevOps and IT teams need to map out all known unknowns in the system. Then, determine the types of tools and monitoring solutions needed to identify the health and performance of these unknowns. After implementing these monitoring tools, the team can start tracking known unknowns and measuring health and performance.

And, over time, DevOps and IT teams start to identify unknown unknowns across services and build incident response plans for the unpredictable aspects of their systems. Limiting unknowns and creating transparency across disparate services can help teams get better at incident detection and response.

4) Think about the entire system

Monitoring needs to flow through the entire system. Databases, applications, cloud services, containers, servers and physical hardware at employees’ desks need to be monitored. Errors, transactions, CPU, memory, disk usage, active directory logs and traces should all be monitored. The more granular you become with monitoring and alerting, the easier it is to identify incidents in real-time and take action toward remediation.

5) Intelligent alerting and on-call context

Last but certainly not least, you can start building intelligent alerting processes and improve on-call operations. With monitoring data such as logs, traces, metrics and charts appended to automated alerts, on-call responders have the information they need, right away. And, if you add runbooks alongside the monitoring data, on-call responders can easily follow instructions to fix the problem. Monitoring data is effectively useless without an understanding of how to take action on an alert

Actionable alerting and incident response

Contrary to its name, a comprehensive IT monitoring plan doesn’t only include monitoring tools. It needs to include actionable alerting strategies and a real-time incident response plan for when metrics surpass designated thresholds. How should on-call teams and alert rules be set up to rapidly and effectively respond to production incidents? Who’s the best person or team to respond to certain issues?

Simply mind mapping human workflows can help you build an IT monitoring and alerting plan that works. The interactions between people, processes and technology are where bottlenecks build up and blind spots appear. Don’t only monitor technical applications and infrastructure, monitor the time employees spend on-call and the speed at which you can respond to incidents. And, more importantly, monitor real users and service availability to understand how operations are truly affecting customers.

Proactive resilience and preparation

Resilient systems are a product of continuous improvement and preparation. Nobody simply sets up monitoring tools, gets the right alerts and fixes issues faster. The system changes, the team changes and the process changes. So, system reliability boils down to proactive preparation and constant testing of applications, infrastructure and processes.

Incidents are inevitable in a modern world of IT and DevOps, making preparation the only surefire way to limit downtime and ensure availability. How do you consistently improve the speed at which you identify, respond to and remediate an incident? Thorough post-incident reviews and regular analyses of people, processes and tools can help you drive changes to your monitoring and alerting stack – allowing you to refine and improve your IT monitoring plan over time.

Establishing a culture of DevOps

At its core, DevOps is about continuous improvement in all aspects of collaboration, transparency and automation. How do you shorten feedback loops between developers and IT operations? How do both groups learn from each other in order to build and release reliable services faster? A culture of DevOps isn’t about building a siloed DevOps team, it’s about getting the most from sysadmins, database admins, developers, security analysts, technical support agents and business teams through better communication and visibility.

A powerful IT monitoring plan can expose problems to all of the engineering, IT and business teams. And, it enables cross-departmental work to flow smoothly between people and teams, ensuring streamlined operations and delivery of customer value. Start with an actionable IT monitoring plan, improve the way teams collaborate and you’ll ultimately end up in a place of DevOps continuous improvement.

Putting it all together in one comprehensive IT monitoring plan

The three C’s make up a comprehensive IT monitoring plan – clear, contextual and collaborative. Make it clear across all teams what you’re monitoring and why. Surface the context you’re trying to receive from the systems you’re monitoring. And, find ways to improve the way teams collaborate around the monitoring data.

A dedicated DevOps mindset helps you craft the most comprehensive IT monitoring plan, from initial incident notification to the final post-incident review. Automate what you can, create transparency into technical systems and human workflows, and improve the way people, processes and technology interact. Prepare for the worst but deliver the best. With a comprehensive template for IT monitoring, issues are identified faster and collaborative incident response processes are established to drive business value and keep end-users happy.

Start implementing your IT monitoring plan. Sign up for a 14-day free trial or request a free personalized demo of VictorOps to set up monitoring integrations with automated on-call schedules and intelligent alerting to make incident management suck less.

Let us help you make on-call suck less.

Get Started Now