Jason Hand - February 19, 2016
We all remember the game from our childhood where one person whispers a phrase to the person directly next to them, who in turn shares the phrase with the following person in line. This continues through a group of people until it makes its way back to the original source. The game went by many names: Telephone, Grapevine, and Operator, among others.
The point of this exercise was primarily to demonstrate how easily information can become corrupted by a lengthy path through which it passes. Minor and major alterations to the information occur naturally, and in some cases intentionally, as details and facts associated with information are diluted by way of indirect communication with the original source.
Rarely was the original phrase provided back to the source and it was clear that the more people the information passed through, lengthening the loop, the more errors were inevitably introduced.
Another more obvious but important observation from this exercise is that the time in which it takes for information to return to the originating source varies greatly and increases with each new point in which the information must pass through.
In short, the volume and frequency of errors in data increase conversely with the path size and time in which it passes.
Feedback loops are used in every industry and social science. A common example used to illustrate complex systems and the way in which we manage them through feedback is the O.O.D.A. Loop first developed by US Air Force Colonel John Boyd.
Adopted and utilized far beyond military operations, the O.O.D.A. Loop helps us to simplify and understand the process in which we make decisions in a recurring cycle. Through observation, orientation, decision, and action, huge advantages are gained regardless of the context in which decisions are being made.
Agile and DevOps principles teach us that removing friction in our processes and communications is a critical component to success in modern software delivery. Shortening feedback loops allow for quicker responses to situations as well as a reduction in opportunities for errors in data.
Shortening the time and steps is the most efficient and thorough way to truly understand if our choices are facilitating and encouraging the desired impact. Companies that have found a competitive advantage know the secret of shortened feedback loops very well. Not only have they adopted the principles of Agile and effective DevOps practice within their IT teams, but throughout the organization. It’s part of an ongoing effort towards continuous improvement.
Today’s best practices quickly become outdated as new processes and tools become available and mature. There are many contributing factors, both positive and negative, that emerge as technology and ideas evolve over time. Much of this is a result of trial and error. Embracing the feedback loop allows us to respond, learn, and improve from those factors, which in turn allows us to innovate our own products and services.
Waterfall planning and delivery methods where software releases take place in long cycles are no longer acceptable. The demands of competition and innovation require much shortened cycles for every phase of the process. The goal of the waterfall approach is to structure everything such that the schedule, scope, and resources can be determined upfront.
Unfortunately, this approach means companies can’t respond quickly. When the needs of customers or the landscape of markets inevitably change, IT teams aren’t equipped to receive that feedback and immediately apply it to new decisions and choices. There is no way to self-correct other than by throwing out an immense amount of planning and work only to start from scratch.
Feedback doesn’t take place only within systems. Verbal and non-verbal communication between co-workers, partners, and customers are other forms of feedback. Looking at that feedback through a systems lens is a far more accurate method of evaluation. By taking a step back and examining the feedback and data we receive, we can understand it much clearer. It has a unique ability to move us away from needless judgement while also enhancing accountability.*
There are three main questions to ask in order to accomplish this:
1. Are differences between the giver and receiver creating friction for the feedback?
2. Is the feedback partly related to the differing roles between giver and receiver as it relates to the common system?
3. Are processes, policies, physical environment, or other factors within the system reinforcing problems with the feedback?
Examining feedback in this manner allows for a deeper understanding of the information flowing to and from the human inputs and outputs. By allowing ourselves to view feedback through a Systems Thinking model, we can begin to look for patterns, understand the feedback loop with more accuracy, and identify contributing factors to both failure and success. Emotions are removed and the data is examined in its truest form, further enhancing our ability to know, understand, and eventually learn.
These ideas are a big reason why so many high-performing IT teams have adopted blameless postmortems. Understanding the events that took place during a disruption from a Systems Thinking model provides a much higher return, not to mention a large sample of possible improvements to be made.
The inevitability of failure has a unique ability to absolve us from the effort of trying to engineer failure out of systems. Because of this, we now design for failure, optimize for a reduction in Time To Repair, and build in feedback loops that prevent us from aimlessly hunting for a root cause of a disruption. From that, we can use divergent thinking to guide our decisions and choices on what to do next to improve the reliability and resilience of our systems. And the by-product of all of that, is a highly available system built, maintained, and continuously improved upon by high-performing IT teams.
Rather than intervention or eradication of problems, repair becomes the aim and the feedback loops taking us from certainty to uncertainty and back again are what guides our trajectory. Instead of explaining what went wrong, we use the opportunity of failure to seek out many possible creative areas of improvement. Prediction is removed completely and replaced with a new idea of “build for failure”, which in turn leads to improvements and advancements in our own processes, tools, and eventually the service we provide.
Continually processing and responding to feedback (including failure) allows us to use information about previous outcomes to guide our actions of the future. Rather than speculating about the future conditions, we can leverage opportunities to learn as a vehicle to form a new trajectory, correcting our path along the way.
As builders and maintainers of complex systems we must take great effort to shorten feedback loops. Output from one system, event, or customer is used to influence input and action to the same and related sources. The quicker we can receive the information to guide our decisions and choices, the better our processes, tools, systems, and services become.