Autonomous testing is the ability to create tests autonomously without users’ intervention. Properly testing applications is a time-consuming process, and autonomous testing holds the promise of significantly reducing that work.
I’ve spent the past couple years trying various approaches to autonomous testing. In the next couple of minutes, I’ll summarize my learning and share with you the one practical approach that worked for me and my team after testing a few dozen apps.
I wrote a full chapter in Eran Kinsburner’s book Accelerating Software Quality about autonomous testing and how we at 21 model applications to achieve autonomous testing. I’ll share a few snippets. You can learn more about it on our panel discussion on September 30 or read about it in Kinsburner’s book.
A while ago, I met Oren Ariel, the CTO of the iconic Mercury Interactive. Since Mercury, Oren was the CTO of a few other companies, so the conversation quickly evolved into the ways applications are deployed. “The testing industry hasn’t really evolved much since Mercury days. Too much time is spent on QA. The penetration of automation increased only by a few points, and coverage is still limited,” Oren told me.
He’s right. QA is usually a combination of some unit testing and a lot of manual QA. Decent coverage comes at a high price, and most companies just accept the notion that bugs will slip to production. Five out of six mobile apps we at 21 tested autonomously were buggy. Many times, we found those bugs during a product walkthrough (our demo using their app).
I’ve met hundreds of organizations while working for three testing companies, and, while there are pockets of organizations that do achieve high levels of automation (usually at a high cost and over long periods of time), most organizations struggle due to three main reasons:
There are many challenges associated with deploying autonomous testing. Below are a few things you should take into consideration.
If you give the same scenario to 10 SDETs, it’s likely that each will create this scenario differently and assert different things. If you create scenarios autonomously, which one will you choose, and will it fit what the QA team intended to test?
Sometimes you want to test negative scenarios, such as a user not being able to perform certain actions before certain prerequisites are met. Another common example is testing password validation (e.g., proper error messages for passwords that don’t comply with the policy). How can autonomous testing test such scenarios?
Many apps customize their experience to different user profiles (basic vs. pro, standard vs. administrator). We run quite a few scenarios validating that certain profiles have permissions only to specific functionalities.
While autonomous testing can maybe test the application’s functionality, it doesn’t know the expected outcome and doesn’t know what to assert for. A typical example is the checkout process in a shopping app. An SDET would properly assert that the total amount matches the prices and quantities of the items purchased. How would an autonomous testing tool know to assert for that?
Functional testing is just part of quality assurance. User experience testing includes testing the responsiveness of the app under different network conditions, disruption testing (e.g., app goes in the background), personalization to location, language, and so on.
These are just some of the scenarios our customers at 21 are testing for today. Moreover, we’ve spoken to hundreds of companies about their approach to functional testing and the value of autonomous testing. Organizations that take quality seriously will have a list of scenarios they go over and test for before every release. CTOs, VPs of engineering and QA leaders we’ve spoken to want to know that those scenarios are being tested as opposed to some black box that basically does exploratory testing.
The premise of autonomous testing is clear: immediate coverage with no effort. I’ve come to the realization that creating tests autonomously might not be the right objective. Tests, by nature, look into edge cases and require an understanding of what to expect. Two QA engineers can take the same scenario and test it differently. Moreover, the more organizations I work with and the more tests I see, the less confident I am that I, an intelligent human, would have come up with the scenarios myself.
Let’s redefine autonomous testing as a learning model of the application that can do these three things:
You can think of the way we model applications at 21 as being similar to a sitemap for mobile. Given two screens, the application can autonomously populate the right actions and detect navigational changes (changes to the edges) and functional changes (changes in the screens model) from the baseline. It will then update the baseline, which will trigger a self-maintenance process to edges and screens within your test cases.
Don’t think that will work on your mobile app? Let’s put this to the test.