How Not to Be Wrong About Ops Metrics

Byrce Ambraziunas July 29, 2014

Operations managers have used metrics to gauge the success of their teams, projects, and customer satisfaction since the beginning of time. Depending on the industry and management philosophy, the metrics gathered can vary from the traditional Quality Score, First Touch Resolution, MTTR, Average Handling time to the more current monitoring of application events and deploys per day.

No matter what industry you are in, all of these statistics are just numbers and they aren’t good or bad on their own. The important part is ensuring the methodology used to derive the number is sound and the proper logic is used when they are interpreted.


Let me use an example out of a book I am currently reading, How Not to Be Wrong: The Power of Mathematical Thinking by Jordan Ellenberg. In it, Jordan discusses the Strategic Research Group (SRG) assembled during WWII. This group was “the most extraordinary group of statisticians ever organized in both number and quality” and their mission was to use statistics to help win the war. For instance, they would determine the optimal arc for a dogfight or developing protocols for strategic bombing.

Within this group, one statistician named Abraham Wald was given bullet hole data from the military and asked to advise on the proper placement of additional armor. His answer surprised military personnel because he told them though their data was impressive, they were using the wrong logic to interpret it.

He advised the military not to armor any of the areas where the holes were, but to armor the areas where they weren’t. The logic he used was simple - the planes that made it back were survivors so they were counting non-fatal holes.

With that in mind, let’s take what we learned from Abraham Wald and apply it to some Operations Metrics. One of my favorite long-standing data set is Average Handling Time, which is the measurement of how long it takes a support team to “handle” a customer incident. In the old world, this would be the time a technical representative was on the phone and in today’s world, it is typically measured in the number of electronic exchanges or characters typed, etc.


The traditional thinking goes as follows in regards to Average Handling Time (AHT): the higher the number, the longer it took to resolve the issue; the longer it takes to resolve the issue, the less efficient the support rep is and the more reps you need to have, which costs the company money. Using this logic, many mid-level managers have struggled to keep this number as low as possible. They reprimand support reps for their talk times, inserted advanced phone trees, and fired off auto-responses to FAQ articles.

I don’t know about you, but I’ve never met a phone tree that I liked and I can’t remember the last time a canned auto-response solved the issue I was complaining about.

But isn’t the mid-level manager’s response logical? Sure, if you are looking at it from the company’s perspective. A low AHT seems good because the team is working efficiently and costs in Support seem low. However, if like Abraham Wald, you look at this data through a different lens, it tells a different story.

Let’s take a minute and look at a low AHT through the customer lens. A low AHT means that the manager is incentivized to reduce customer interaction either by adding auto-response technology or pushing Support Reps to reduce their customer interactions. These moves cascade into the customer feeling rushed, not receiving a complete solution to their issue, or being pushed off to a FAQ that may or may not solve their problem. At the end of the day, the desire for a low AHT is actually reducing customer satisfaction, increasing churn, and potentially damaging the company’s reputation.

For this reason, I’ve never used AHT as a quality metric for my support team and have instead used metrics that support the correct behavior like First Engagement Resolution (FER) and Customer Satisfaction.

At the end of the day, the most important thing to remember when designing or evaluating Ops Metrics is to make sure you have a solid methodology for calculating your numbers and are logically interpreting them. From there, make sure you are viewing these metrics through the customer lens to ensure you are incentivizing satisfaction and loyalty.

Our support philosophy at VictorOps: take as long as necessary to get the problem solved. The duration of the support interaction doesn’t matter – the happiness of the customer is all that matters. And of course, be victorious.

Ready to get started?

Let us help you make on-call suck less.