The Evolution of DevOps Testing Tools

Engineering teams face the pressure to continuously deliver new features fast. Theoretically developers could pump out new iterations multiple times a day, but parts of the DevOps cycle, including testing and Quality Assurance, aren’t as fully optimized for CICD as other facets of the cycle. The next wave in machine learning testing solutions are emerging to seize this opportunity.

The DevOps ecosystem has seen two evolutions of tools, and we are on the cusp on the third evolution. Within these three evolutions of DevOps tools there are established tools in the various channel types, from coding and code analysis, to build and QA, and overall operations.

The first evolution of tools was the craft, artisan one-off manual tool approach. Within that first, craft level QA section, Selenium and JUnit laid the foundation of testing tools. The next iteration was the adoption of automated, at-scale tools that focus on repeatability. As the need for scalability grew, automated tools like Sauce Labs and Accelerator evolved. Automated testing tools have enabled us to achieve repeatable and scalable testing, but there are still significant hindrances to velocity.

✕

The next evolution of DevOps testing tools is advancing the intelligence of the tools, making them smarter by utilizing AI and machine learning. Maturing their automation capabilities, the next phase in these tools will take the processing of tests to a new level by also prioritizing test suites further speed up test runs.

DevOps Testing Tool Limitations 

Although we’ve come far with testing automation, several factors continue to dampen the velocity of larger testing cycles. Organizations might be working to shift left, but test suites have varying load sizes and the more complex tests require more time. User acceptance testing and integration tests can take time because of the variables included. Truly shifting left and running tests sooner would result in catching risks earlier. 

Along with long test times taxing testing cycles, flaky tests plague development teams and waste time and resources. Typically when flaky tests are suspected, developers have to sift through large volumes of testing data to determine the problem source, and many times, there are simply not enough clear signals. 

Given today’s application design complexity and the dynamic nature of run-time decisions, the only way to solve the problem is through deepening automation functionalities. Progressing testing automation relies on the creation of new, intelligent machine learning testing tools.

The Hurdle to the Next Evolution of DevOps Testing Automation 

Developing a machine learning DevOps testing tool requires a great deal of specialized expertise from individuals trained to be data scientists. Companies need individuals like an AI QA strategist, someone who can weave AI into the DevOps process. The reality is that there is not much of that expertise available in the market. High-end consulting firms, like McKinseys, try to help but the void is large.  

Companies must now develop that expertise themselves, which is impractical. A self-built machine learning stack is extremely complex and expensive to deploy. The individuals who are familiar with it are rare and expensive. Unless a business has the technical depth of the Googles of the world, they lack the infrastructure needed to build their own in-house practice. The task is well beyond the purview of small and medium companies.

Corporations are trying to drive investments in pure math skills into QA teams and embed data scientists into each one. But that process may be a long-term fix -- maybe five to seven years -- so what do they do today?

The Current Landscape of Intelligence in DevOps Testing Solutions

The latest level of DevOps tools is the next generation of automation. It builds on the automation tools and elevates the capabilities by incorporating machine learning and decision-making by harnessing statistics.  This next evolution consists of a number of startups emerging, and their tools can be sorted into two categories: scaling tests and prioritizing tests.

DevOps Solutions for Generating Tests

This category focuses on the issue of not enough tests and is more mature with companies like Mabl. AI is used to generate tests where none exist primarily in the UX testing space. This category addresses specific types of testing, say a bot that generates tests for mobile web pages. The more tests get automatically generated, determining what tests are effective becomes a problem.

DevOps Testing Solutions for Identifying Critical Tests to Run

This category focuses on solving the problem of having too many tests that take too long. A more nascent category, teams like Facebook have been using predictive test selection tools to drive efficiency in the testing process. Companies like Launchable are trying to bring this to the market. Predictive Test Selection that combines coding and quality into one practice. We do this by taking data from your Git changes and past test results to quickly deduce where problems lie, so developers release high-quality code faster. Launchable is a layer on top of your existing CI pipeline and tooling. The Launchable CLI interfaces with your existing build and test tools like Selenium, Maven, Gradle, and Ant.

Enterprises need to push new software releases out at a faster rate. Testing velocity has been a DevOps opportunity to capitalize on. The next evolution of smarter testing automation will harness machine learning, and promises to bridge that gap and speed up development.