In Part I of the test automation from work we discussed the selection criteria as it related to the app one is testing as well as initial set up, skill-set and authoring time.
This part will center on more generic requirements one needs to consider. At the end of the post are links to all the tools and forums mention in the article.
The purpose of test automation is to automate your release cycle. One wants to make sure that changes made by dev did not break prior functionality. If certain functionality did break one wants to know about it as soon as possible. That is where CI comes in place. There are different CI tools such as Jenkins, Travis, Azure Pipelines, Bitrise etc., so it is important to know that the tool you are picking is supported by CI that is used in your company.
Appium can be integrated with most of the tools using shell commands for Jenkins as an example. If there is no cloud used for test execution then real devices can be connected to the server in case you use this server in-house. If cloud CI tools are used then it would be better idea to go for cloud solutions easiest.
Integrating the Espresso framework is not complex. Espresso is a part of the developer code base that is why usually Grandle is used as a build agent. Gradle commands are used in the CI tool itself with the help of Gradle plugins applicable for most popular CI tools.
XCUITest framework can be integrated with most popular CI tools however it is less reliable than running these tests on xCode locally that provides proper logs to debug.
Automated tests need devices to run on, ideally a variety of them, to test multiple devices, and ideally in parallel to test fast.
Having your automation tests running on cloud is convenient nowadays as it provides 100s of real devices you can use and removes the burden of buying and maintaining mobile devices, not to mention connecting all of them into a single always-on grid. Even if you have devices but your CI is hosted on cloud (which is frequently happening) it will be difficult to connect your devices to that server, unless you set it up inside your organization.
There are various vendors offering cloud solutions such as SauceLab, Browserstack, Perfecto, Kobiton etc. When choosing a framework or a tool you should also check compatibility with your designated cloud solutions. Usually all of them provide various integrations that are well documented. While there should be an issue to integrate Appium , Espresso or XCUITest to any of these clouds the devil is in the details. Different vendors might have a number of commands that are not supported or are partially supported by their platform. Some have altered standard frameworks to offer proprietary libraries extending the framework functionality.
Finally if you are a well known brand with millions of downloads, a variety of devices and zero day availability of devices and OS versions might be top of mind for you. This might impact your infrastructure vendors as well as your framework.
If browser or platform coverage is important on the project – parallel execution can be a time saver as it gives an ability to run a test on multiple devices at the same period of time that also provides better coverage to your project.
There are two ways to run tests in parallel – sequentially and distribute suits across devices in parallel.
You can archive it with popular cloud solutions that provide the ability to run tests in parallel (usually they run it sequentially) and also they are providing good device coverage. The only disadvantage here can be cost however it still depends on how many parallel sessions are needed.
If there are a lot of tests to be run frequently then it worth to think about parallel testing distribution instead of running it sequentially. Firebase Test Lab gives this ability with the help of Flank. It can work with Espresso and XCUITest frameworks.
Another way is to have a device farm in house however it might be more expensive than paying for cloud because the farm needs to be up to date, have well maintained infrastructure and easy access to devices.
Test automation for mobile is not easy to accomplish. It’s relatively new and the variety of devices, OEMs, OS vendors and development frameworks makes it even more challenging. If you are using a commercial tool, most likely you have support that will have your back if something isn’t working or if you’re not sure how to execute one thing or another.
If you are using open source, you will encounter challenges and most likely rely on the assistance of either a consultant of the community for answers. Make sure you have those.
Appium is popular and has a wide and active community with 11,7k Stars and about ~4.7K Forks and frequent releases. There is a Discuss Appium forum where one can find an answers. It is possible to filter topics by category, top categories and latest questions, search for query etc – that makes it easier to find a solution or answer for Appium users. While it is well documented, Appium has a lot of edge cases that are specific to the app or development framework. While you can ask around the overhead of trying all possible suggestions is time consuming and usually there are slight variances. These cases can lead to lengthy implementation cycles and will significantly increase your cost and time.
Espresso is a native framework created by Google and because it is part of developer code documentation is available on android developers website here and also there are forums such as AndroidForums to look for needed information regarding Espresso. There is no dedicated forum for Espresso framework only unfortunately.
XCUITest native framework is developed by Apple so documentation is available on Apple Developer website. Few Apple Developer forums such as Developer Apple or Swift forum might contain some additional information about XCUITest framework. Similar to Espresso there is some information on Stackoverflow in a question/answer format however the community is still growing.
A lot to consider when choosing the framework for your team.Below you can find an excel that will help you to choose the best tool for your team. We summarized the key factors and you can adjust the weights of each criteria based on the importance to your app and drive a fit score to match the framework to your criteria
Many of the challenges mentioned earlier are addressed by commercial tools such as 21. 21 is an AI based test automation tool for mobile that doesn’t require scripting (so skill-set is not an issue), supports all platforms and development frameworks, includes built in integration to devices as well as easy-to-setup integration to 3rd party device providers (including mapping all the relevant APIs), rich reporting, CI integration and mich more. Authoring tests with 21 takes minutes and its self-maintenance capability makes updating tests to user flow changes a breeze.
21 also includes a full analytics layer including integration to production for coverage reporting.