Samantha Haltman - February 09, 2015
In November of 2014, Daniel Zacek, the CTO of 9-1-1 Labs decided he wanted to make on-call suck less. The company promptly participated in the private beta of the VictorOps Transmogrifier. We talked to him about the power of this new feature, and this is what he said.
In our old process, an alarm would fire; we’d head to the company intranet, work our way down the hierarchy of machine naming to get to the correct page, and take 5 or 6 attempts to uncover the right information. With annotations, this lengthy process has been replaced with one click.
As a distributed team with several projects running simultaneously, chaos used to be inevitable. We now leverage Github wikis, and we auto-generate links to each alarm with annotations. When an alert fires and specific documentation has not been created, a link to an incomplete wiki page surfaces. This allows us to proactively update our documentation, which has helped to make our response time substantially more efficient.
We use NodePing for one of our external monitoring tools, and currently, the format of these alerts needs to be altered to reflect correctly within our VictorOps timeline. (VictorOps API hook takes POSTs, NodePing sends GETs). We leverage the Transmogrifier to transform NodePing tags to ensure proper communication between platforms.
Creating rules with the Transmogrifier and checking the results of our rules have become integral to our software building process. When building a new AWS server for example, there are specific steps associated with ensuring that all VictorOps alarms are equipped with the proper documentation and firing correctly.
The VictorOps platform, specifically the Transmogrifier, has become a focal point for our team. It’s baked into our process and it’s making us better.
About 9-1-1 Labs:9-1-1 Labs provides development and technical operations for early-stage and emerging startups. We help entrepreneurs move from idea through scaling to millions of users with 99.999% availability.