Transmogrifier (Alert Annotation and Transformations) has been our code word for the super-secret technology we have building internally for a few months. As Calvin & Hobbes will attest, it’s simple, but really really powerful!
Need to dynamically reroute an infrastructure alert in the middle of night? Transmogrify it!
Need to attach a graphite graph to alerts coming from a group of VM’s and then override one of the rules for one VM specifically? Transmogrify it!
Want to change a CRITICAL alert coming from report server to a WARNING? Transmogrify it!
The Transmogrifier makes the impossible, possible, and now its in BETA for select VictorOps customers.
Let’s Start with the “Why”
In talking with customers, there are several pain points that come up in nearly every conversation related to being on-call:
– Documentation Sucks. It’s write-only, write-once, and it’s honestly easier to re-solve problems than to hunt down the documentation (which is probably out of date anyway).
– I’m Suffering from Alert Fatigue! I want a process for cleaning up alarms, and also ways to quiet alarms I know are not actionable.
– I Have Specific Alerting Needs for X Monitoring Solution. Custom monitoring tools or emails generated by NOCs or humans need to be handled particularly.
– I Need More Sophisticated Routing. Based on specific alert details, I want alerts to be routed differently.
The Story Behind the Feature
Since inception, the concept of helping people meaningfully solve problems faster has been part of our mission here at VictorOps. We proposed that the quickest way to achieve this was to surface ways that the problem was solved last time, along with original alert details. Lack of quick access to documentation is the number one reported issue by our users – we aimed to solve this.
This became our ‘Runbook’ story. We set out to build the ability for our users to attach a runbook to alerts based on key value pairs within the alert details. Simple, really…
It’s Not Just About Runbooks!
Since VictorOps hit the market, we’ve received myriad requests to surface assets based on alert contents. Assets like graphite graphs, links to previous Jira tickets, free form text, etc. We realized we could accomplish all of this with the same technology. Using Annotations, VictorOps users can attach any URL, Image or Text to their alerts.
And you thought we were finished?
Not even close. Users can also ‘Transform’ their alerts using the Transmogrifier. This means that they can overwrite existing alert fields to make alerts behave differently.
It’s no surprise to anyone on-call that one of the most exhausting parts of the job is receiving unactionable alerts. Alert fatigue is real, and not something that is always easily solved.
Do you know that a set of alarms ALWAYS fires at a certain time of night because there’s a job running? Wish you could temporarily quiet those? Now you can. Users may change the STATE of alerts using the Transmogrifier.
Alternatively, some users want to alarm on custom monitoring solutions or emails sent by a NOC or customers. The same feature allows administrators to turn these messages into critical alerts using the Transmogrifier. In addition to state changes, users may also change the route key based on alert fields, ensuring that alarms are always directed to the appropriate person every time.
Interested in checking out the Transmogrifier? Sign up for our Private Beta or stop by our booth at AWS Re:Invent, and we’ll be sure to get you in the queue. We’ve built a powerful ‘rules engine’ and we’re excited to see how our customers use it as we complete work necessary to make it a production-ready feature. We look forward to your feedback!