As the leader in in-flight connectivity to airlines around the world, it’s no wonder Gogo strives to use best practices across their entire organization. When you’re known for creating an innovative service, you focus on developing and managing the internal process in the same manner. Accordingly, when a cloud DevOps engineer joined Gogo and saw room for improvement, he wasn’t saddled with opposition from upper management to make smart changes—he went straight to work.
“When I started it was a chaotic environment,” explains the cloud DevOps engineer. “People were chasing each other around trying to pinpoint problems within the software. Sometimes it felt like everyone was running around like crazy trying to solve an issue.”
It didn’t take long before this engineer saw the need for an alerting tool. He quickly began investigating incident management tools to use across the entire organization. When deciding on incident management software, Gogo focused on a tool with a bidirectional Slack integration and a commitment to cutting-edge innovation. “VictorOps was an easy decision,” says the cloud DevOps engineer. “If I had to do this process over again, I’d pick VictorOps every time.”
Change is Hard
Convincing an entire organization their on-call rotation was outdated? Well, that wasn’t so easy.
Gogo was still employing a traditional on-call strategy, where Ops people held the responsibility of answering a page when something went wrong. They knew the only way the company could grow was to bring developers into a mature on-call process, so issues could be resolved quickly and confidently.
On-Call Role Changes
When this new process was introduced, there was initial concern about job security for those traditionally on-call, but that quickly faded. “I explained this would allow everyone in the organization to focus on their own high-priority work and would eliminate a lot of their current headaches,” says the cloud DevOps engineer. “By using logs, the Ops team can troubleshoot issues and quickly have visibility into everything they need so they can start calling team A, B, or C to get the problem solved quickly. And if teams don’t answer right away, the logs help keep them from being stuck doing nothing until they get a call back—they have the information at hand to start solving problems that would have normally been too complex to understand.”
Because Gogo uses several tools across their organization—Slack, Status IO, Bugsnag, Apache Spark, Sumo Logic, Hosted Graphite, and ThreatStrike—using a tool that broadly integrates improves collaboration and communication. Additionally, and perhaps more importantly to those on the ground, VictorOps integrates with common tooling to support an on-call experience that doesn’t suck.
Putting Devs On-Call
Another challenge the cloud DevOps engineer faced was convincing developers to play a part they’ve never had before. Change can be hard, so it was important to explain this was a necessary shift within the organization as the company matured. Once developers understood this change as a step towards innovation, it became something they were excited about.
When the shift is explained to developers, the cloud DevOps engineer says, “It’s your code and you’re responsible. You write the code, and that’s it.” “Developers know the code, so if anything goes wrong they’re responsible for their own code because they know it best. There’s no point in making people run around trying to figure out what’s wrong in code when a developer can find it within minutes.”
According to the cloud DevOps engineer, the new on-call process was a struggle in the beginning, but it’s now a huge success across the entire organization. “Now that we’re doing this, developers are more proactive with the code. They jump on-call and instantly start working on problems and interacting with other teams as it’s happening to start fixing things right when they break,” the cloud DevOps engineer says. “It’s really simplified the management process of our issues.”
When onboarding new developers, initial bias against on-call can be quickly thwarted by simply demonstrating the inherent value of the on-call process. By using VictorOps across the entire organization, all developers—including the newly-onboarded business aviation team—have a streamlined on-call process that improves the quality of their jobs.
Success for Gogo Customers
The introduction of VictorOps not only helped Gogo’s engineering culture but also improved the customer experience. Because the developers at Gogo are able to know about—and start tackling—issues right away, customers have a more positive interaction. Gogo can notify users quickly if something interferes with in-flight connectivity, keeping them informed with accurate details so they always know what’s going on and can have a better idea as to when the problem will be fixed.
“This has improved relations with the airlines and our current customers,” says the cloud DevOps engineer. “Having such great visibility into our systems have improved our customer support and relations.”
“I’ve used a lot of tools in my career, and the VictorOps customer service knows right away what I’m talking about when I have an issue. They give solutions or will instantly get on a call to help fix things. It’s amazing.” - Cloud DevOps Engineer, Gogo
More Time for Innovation
With VictorOps up and running, the DevOps team has been able to focus their attention elsewhere and tackle large projects within Gogo. Right now the organization is almost done migrating their data center to Amazon Web Services, a huge process for the team.
Being able to tackle large projects and cultural changes—and see them succeed—is part of what makes Gogo a success. It’s only a matter of time before we hear about the next innovative project at Gogo.