Jason Roth - October 27, 2016
At VictorOps, most of our systems are built upon the Akka framework. It’s an application framework implemented for the JVM (we use Scala), modeled after Erlang, a language designed around the actor model.
We’ve had a lot of success using it to build the kind of distributed, resilient systems our customers depend on to make their on-call experiences suck less.
We make heavy use of Akka clustering, remoting, and persistence libraries throughout our stack. The result is a set of very stateful applications where much of the application state is in memory in the JVM processes. (The pros and cons of this kind of architecture are really interesting, but a whole other topic!)
“What is the state of that actor?”
“How can I correct an actor that has gotten corrupted?”
“How can I fix it?!”
Given that architecture, doing things like inspecting actor state and manipulating it become somewhat tricky. For instance, a lot of our alert processing functionality is entirely in memory using persistent actors. That has many benefits and the very big drawback that we can’t do anything like query a database to determine the current state of things.
To help us manage our running system, we took a page out of Erlang, Clojure, Rails, and other frameworks and languages: we built ourselves a console application.
Ammonite REPL is a reimplementation of the standard Scala console by Li Haoyi, and It. Is. Awesome.
We have built up a console application using a combination of Akka clustering and remoting, and the ammonite REPL, that lets us interact with our running systems using our shared libraries and straightforward Scala code.
I’ll walk through a very simplistic example to demonstrate.
Suppose you have a counting application. Counters increment, and reset. Here’s your CountingActor:
Now, suppose that while that application is running, you want to inspect and possibly manipulate the state of a counter.
That’s all pretty straightforward. We write a trait that has a getCount function, an incrementCount function, and a resetCount function–basically implementing the interface of our CountingActor above.
Left unimplemented are defs for the CountingActor selection we send messages to, and an Akka timeout used by the ask pattern. Next, we’ll need a concrete implementation:
That part is hand-wavy for sure–there’s a hard-coded actor path in there, if you didn’t notice. At VictorOps we leverage our clustering libraries for actor lookup and discovery, but this snippet serves as a good-enough illustration here. The important bit is that we’re using Scala-oriented composition to give us something we can import into the ammonite session – we can instantiate a Console, and use the imports object to load the REPL session. Given the examples above, we have a counter app, and a console app. Here are the mains for those:
That probably seems normal enough if you are familiar with Scala and Akka. We’re just starting an actor system, spinning up a counter actor, and off we go.
For this implementation, we start an actor system in the main method like we normally would, as well as an instance of our own Console from above. Using ammonite’s bind arguments to bind our instance of the console into the repl, we now have that instance in the scope of our repl session. Finally, using the predef capabilities of ammonite we import our task functions into the session’s scope, where they become available.
Here it is in action:
And there we have a console application that can interact with another running Akka application.
Moreover, we’re in an ammonite session so we have all of Scala available to us. Check out some of the examples from that project to learn more. http://www.lihaoyi.com/Ammonite/
At VictorOps, we have implemented many messages in our various Akka applications specifically for use from our console application–simple things like extending trials for new customers and flipping feature flags for customers that request access to some of our beta features, to complex things like rebalancing actors around our cluster and running audit checks of actor state.
We also use our console to troubleshoot customer issues when they arise, and it has been extremely helpful in getting things sorted out quickly and efficiently, allowing us to be very responsive to customer support issues with fast turnaround times.
And it’s all just code, which means we can test the tasks we implement using all the sophisticated testing tools available in the Scala and Akka ecosystems.
It’s proven so incredibly useful for us that we thought others that work with Akka might find it useful too.
Here’s the example from this article up on github: https://github.com/victorops/akka-console-example