VictorOps is now Splunk On-Call! Learn More.

The Keys for Successful Continuous Testing in DevOps

Dan Holloran January 21, 2020

DevOps Release Collaboration
Keys for a Successful Continuous Testing Framework in DevOps Blog Banner Image

In 2020, businesses continue to take on digital transformation – moving more toward cloud storage and computing as well as adopting more of a DevOps mindset. As more teams practice DevOps and build out resilient CI/CD pipelines, they’re finding out that faster software development and delivery can act as a competitive differentiator in business. In order to quickly serve valuable products and features to customers, without hindering reliability, developers and IT practitioners are finding better ways to implement continuous testing during the development lifecycle.

So, we wanted to define continuous testing for DevOps and lay out a basic framework that nearly any team could use. By testing continuously throughout software development and delivery, you’ll quickly find more incremental bugs and errors, helping you fix small issues before they become large ones. But, in real-world implementations, many teams still place the onus of continuous testing completely on their IT operations. However, this can cause a large backlog of work for sysadmins and IT security teams, actually leading to slower delivery of customer-facing products and services.

So, DevOps-centric organizations are finding ways to distribute testing between developers and IT practitioners throughout development, QA and release. Constant testing at every stage of software development and delivery causes each test to take less time away from development while still uncovering problems. Let’s dive into the strategy behind a DevOps-oriented continuous testing framework and how it drives resilient CI/CD.

The strategy of continuous testing for a CI/CD pipeline

In DevOps, there’s a belief that teams should automate nearly anything that can be automated. More often than not, this is a great approach. But, it’s important to keep automation in the framework that it should always help with human productivity and a better on-call lifestyle. The basic strategic value behind continuous testing for a CI/CD pipeline are based on these three core elements of software delivery:

  • Consistency
  • Speed
  • Scale

Continuous testing, especially when it’s automated, can help engineering and IT teams maintain a higher standard for app health and security requirements. More manual testing requires more people to run those manual tests, whereas automated continuous testing allows developers to simply build and run more apps, builds and commands.

The basic framework for continuous testing

Continuous testing is a compilation of numerous development tasks, processes and tools. Establishing a one-size-fits-all framework for continuous testing for any engineering organization isn’t possible. But, you can define the core elements required for any continuous testing framework and then figure out how they apply to your circumstances:

Why DevOps Matters

Continuous integration

CI is a software development practice where engineers submit code to a shared repository on a daily or multiple-times-a-day basis. However, automated build testing is an integral part of continuous integration because DevOps-minded teams need to detect errors and defects in code before they make it out of a staging environment and into production.

Unit testing

The best overview of unit testing comes straight out of Wikipedia: “Unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.” Basically, unit testing is a great way to ensure that different units of source code connect to each other properly and work.

Performance testing

Performance testing will be more specific to the application or service your team supports. It’s generally used to track the speed, reliability and overall uptime of the product or feature. But, software performance testing is crucial because it’s one of the best ways to determine exactly how customers will experience your service. Additionally, performance testing is often performed in both staging environments and production – helping you detect incidents proactively.

Integration and functional testing

Integration testing is similar to unit testing in the sense that you’re testing individual units of a system. But, integration testing combines these individual elements into a group where they can be tested together. This level of testing will detect potential points of failure between integrated units and expose weaknesses, ideally before they reach production. Oftentimes, this fits in with functional testing, where elements of a system are tested against the requirements, expectations and specifications of the said system. Functional testing is important for businesses because they need to know exactly how a system should respond and whether or not the system is responding properly in a consistent way, especially in production.

End-to-end testing

At a higher level, software engineers will often run end-to-end tests to ensure that data or workflows move through an application or service exactly how they’re expected to. End-to-end testing is imperative for the overall reliability of a product. However, end-to-end testing will often not help you find the specific cause of an error or an incident, it will only tell you that something is wrong.

Acceptance testing

Does the application or service meet the needs of the business and is it compliant with any internal or external regulatory requirements? Acceptance testing will be one of the last tests you run, determining if the feature or service is ready to head to production.

Smoke testing

Smoke testing is an interesting prioritization exercise in that the DevOps organization has to figure out the key elements of a system and then start testing them. Continuously testing the most important functions and processes within an application or service will ensure a high level of effective availability for customers.

Chaos testing

Actively injecting chaos into your staging and production environments in order to see how a system will respond to unknown factors and stress. Chaos testing on a regular basis can help expose problems in infrastructure and applications that you may not have known even existed. Proactively conducting stress tests and implementing chaos engineering as a standard practice will also create engineers who are more prepared for real incidents when they do hit your production environments.


Automating any and all of these types of tests listed above is crucial for continuous testing. Otherwise, you’ll spend all your time hiring new engineers for QA, engineers who will spend time manually testing the entire system instead of building new features for customers. Note that automated continuous testing has to be valuable to all people behind your products – not only your customers, but your employees as well.

Continuous delivery

With continuous testing comes continuous delivery. If you know that code works as soon as it’s continuously integrated into a shared repository, you find yourself shipping to production more often and with fewer bugs. With greater automated testing and a dedication to continuous manual testing throughout all stages of the software development and release lifecycle, you can start delivering value to customers on a daily basis.

A shift-left mindset inherently bolsters CI/CD and continuous testing

As the idea of shift-left DevOps gets adopted more and more, IT practitioners are more often exposed to staging and testing frameworks in development while developers are deepening their involvement with code in production. Then, when incidents strike, developers can help take on-call responsibilities and fix issues faster. But, because IT gets more involved before code even reaches the customer, the team can more often detect vulnerabilities and potential points of failure.

DevOps tightens the collaboration between engineers and IT professionals, builds transparency around development, release and operations, and allows teams to automate menial tasks. It’s somewhat of a chicken or the egg problem because a shift-left mindset can help facilitate better continuous testing, but better continuous testing can also help influence a shift-left mindset. As you continuously test and build out a culture of DevOps, you find yourself inherently building a better CI/CD pipeline – one that appeases everyone on your team and makes customers happier.

Learn how DevOps-minded teams are centralizing monitoring data, testing insights and alerts in VictorOps to automate incident notifications in real-time, improve collaboration and make on-call suck less. Try our 14-day free trial or request a free personalized demo with our sales team today.

Let us help you make on-call suck less.

Get Started Now