Reliability and Transparency in the Days of Widespread Internet Outages

Todd Vernon - October 24, 2016

Last Friday the internet as a whole suffered an attack that exposed some of the issues surrounding a connective fabric that has literally come forth in most readers’ lifetimes. I guess, in retrospect, this should have been expected, but for a lot of companies it was the convergence of good ideas and bad timing. VictorOps did not have a problem as we had recently moved to CloudFlare for DDOS prevention to reduce our exposure to these kinds of attacks.

Rapid innovation and shared resources

In recent years, there has been a lot of talk around the amazingly propulsive nature of the cloud as it pertains to the rate of innovation in the technology sector, and the ease of starting new companies based on optimizing scarce resources. But attacks like this do beg the questions: How many shared resources do we all use? Have we lost the perspective of looking for common points of failure?

As most savvy readers will know, Dyn, an outsourced DNS service, was the target of the attack. Technical readers will recognize this as a DNS Denial of Service attack that makes sites under this provider’s directory service unreachable due to a flood of incoming bogus requests to reach their site on the internet.

This will happen again

Was this Dyn’s fault? No, they certainly have discussed this possibility as have many internet sites of scale. Was it a failure of imagination? Perhaps; most problems in this relatively new frontier are in fact just that. Was this the first? No. Was this the last? No, absolutely not.

At VictorOps, like Dyn, we serve the companies that build the platforms that power the incredible resource we all know as the World Wide Web. We work hard everyday to plan and adapt to actions that affect our business in the future. Do we get everything correct? Probably not; no one does. Did we survive Friday’s attack? Yes, we did because we planned for indirect attacks similar to and exactly like this.

We live in a competitive landscape with a competitor, PagerDuty, whose sales team routinely and falsely attacks us on reliability using our public StatusPage as a vector for social attack. Friday was not a good day for PagerDuty.

Our industry must shoot higher than this

VictorOps believes we are all in this together. When a service provider fails us, they must be responsible. But ultimately we are responsible for choosing them. As an industry, no one should weaponize something like StatusPage which was designed for transparency.

Our plan is to be there for you, our customers. That is in our best interest and yours.