Most of the companies we speak to fail to achieve full CI/CD. They deploy Jenkins or Bitrise and automate their build process but mobile testing is disconnected from their CI/CD and is a separate process. In the post we’ll discuss the reasons many companies fail to achieve continuous testing, the impact and how to overcome current pitfalls.
The key to achieving continuous testing is a robust test automation infrastructure which includes an easy to maintain framework, high availability testing environment and reporting.
The main reasons enterprises are not connecting their mobile device testing to their CI/CD process and practicing continuous testing are the following:
Let’s dive into some of these challenges, especially as it related to mobile CI/CD.
Testing mobile applications is especially complex: it includes testing sensors, location, network conditions, multiple apps in the background, gestures such as swipe and force touch. None of these aspects are relevant when testing web applications. Moreover, as discussed in what is mobile QA, one needs to test the ability to install and update the app, as well as performance UX testing under real conditions.
Mobile test automation is also more challenging than web test automation: Appium hasn’t been around that long, is not as stable as Selenium, there are many more edge cases such as lack on consistency between the viewport and the object tree, and more scenarios where the cost and complexity of automating them is just too high.
One more ingredient add complexity and barriers to test automation for mobile: multiple platforms. Appium supports iOS and Android differently.
Finally, there are multiple frameworks that one can use to develop a mobile application. Each framework renders the application differently on a mobile device hence your need to customize your test automation to the framework (for example the way text is set different from one framework to another).
The time it takes for the application to render, especially on consumer grade mobile devices, adds delay to your execution time. Moreover, Appium does not work well on cross platform applications, which is why frameworks such as Detox were developed. Pulling the object tree and responses to Appium API calls with cross-platform applications adds additional delay.
A company we recently met with had a React Native application. Their release process was fully automated with the exception of functional and UI testing. Running their regression suite takes a few hours hence they do that nightly. The outcome is two days of feedback cycle.
When we executed our first automated test on a React application the test took over 20 minutes to run so we could relate to their challenges.
Many companies that have automated their tests, transitioned back to manual testing. Changes in the user flow require constant updates to the scripts. Such changes take anywhere from a few hours to a few days, which adds significant delays and diminishes the value of test automation.
Finally, given the nature of mobile devices, frequent crashes, popups and uncleaned devices with historical data leads to inconsistent experience and tests that fail – and not due to defects.
Another difference between web and mobile is the execution environment. While we can execute automated tests and test our web application locally, mobile requires a more complex setup that includes a remote mobile instance, or in some cases a real device. We’ve setup Appium locally as well as integrated with multiple cloud based device labs. Unfortunately each set up is different and such a setup takes 6-8 weeks until it’s stable.
How many times did you decide not to fix a defect since it will delay the sprint ?
Bugs detected off the usual release process tend to be more expensive to fix: the developer needs to switch context from his current feature and branch, trace the right branch, reproduce, fix, potentially address conflicts with newer code base and test forward compatibility. This might also mean rescoping additional capabilities, depending on the number of cycles between when the code base was released and when the defect was detected.
Defects that are device specific tend to be even more challenging, especially when testing is manual. Given limited resources most organizations test various devices sporadically so by the time a bug was detected on a specific configuration, the dev team might be quite a few cycles ahead.
There are ways to reduce Appium execution time, even for cross-platform applications, using predicate strings, batch commands and more. Since working with Appium we’ve been able to shave off over 90% of execution time. We also provide parallel test execution with further reduces overall run time.
To reduce flaky tests we recommend not using xPath but rather build an adaptive locator system. More about it here.
Finally we’re constantly developing our auto-maintenance capabilities to make it as autonomous and fast as possible.
Most of our customers do practice continuous testing and test their mobile app multiple times a day. Sign up and start automating today.
Also we’re holding a webinar to discuss what we learned from 150 mobile application and what you can do tomorrow to achieve continuous testing.