2 1 L A B S


In Part I of the test automation from work we discussed the selection criteria as it related to the app one is testing as well as initial set up, skill-set and authoring time.

Link to part I

This part will center on more generic requirements one needs to consider. At the end of the post are links to all the tools and forums mention in the article.



CI Integration

The purpose of test automation is to automate your release cycle. One wants  to make sure that changes made by dev did not break prior functionality. If certain functionality did break one wants to know about it as soon as possible. That is where CI comes in place. There are different CI tools such as Jenkins, Travis, Azure Pipelines, Bitrise etc., so it is important to know that the tool you are picking is supported by CI that is used in your company.

Appium can be integrated with most of the tools using shell commands for Jenkins as an example. If there is no cloud used for test execution then real devices can be connected to the server in case you use this server in-house. If cloud CI tools are used then it would be better idea to go for cloud solutions easiest. 

Integrating the Espresso framework is not complex. Espresso is a part of the developer code base that is why usually Grandle is used as a build agent. Gradle commands are used in the CI tool itself with the help of Gradle plugins applicable for most popular CI tools. 

XCUITest framework can be integrated with most popular CI tools however it is less reliable than running these tests on xCode locally that provides proper logs to debug. 



Execution Infrastructure

Automated tests need devices to run on, ideally a variety of them, to test multiple devices, and ideally in parallel to test fast.

Having your automation tests running on cloud is convenient nowadays as it provides 100s of real devices you can use and removes the burden of buying and maintaining mobile devices, not to mention connecting all of them into a single always-on grid. Even if you have devices but your CI is hosted on cloud (which is frequently happening) it will be difficult to connect your devices to that server, unless you set it up inside your organization. 

There are various vendors offering cloud solutions such as SauceLab, Browserstack, Perfecto, Kobiton etc. When choosing a framework or a tool you should also check compatibility with your designated cloud solutions. Usually all of them provide various integrations that are well documented. While there should be an issue to integrate Appium , Espresso or XCUITest to any of these clouds the devil is in the details. Different vendors might have a number of commands that are not supported or are partially supported by their platform. Some have altered standard frameworks to offer proprietary libraries extending the framework functionality.

Finally if you are a well known brand with millions of downloads, a variety of devices and zero day availability of devices and OS versions might be top of mind for you. This might impact your infrastructure vendors as well as your framework.


Parallel Execution

If browser or platform coverage is important on the project – parallel execution can be a time saver as it gives an ability to run a test on multiple devices at the same period of time that also provides better coverage to your project.

There are two ways to run tests in parallel – sequentially and distribute suits across devices in parallel.

You can archive it with popular cloud solutions that provide the ability to run tests in parallel (usually they run it sequentially) and also they are providing good device coverage. The only disadvantage here can be cost however it still depends on how many parallel sessions are needed.

If there are a lot of tests to be run frequently then it worth to think about parallel testing distribution instead of running it sequentially. Firebase Test Lab gives this ability with the help of Flank. It can work with Espresso and XCUITest frameworks.

Another way is to have a device farm in house however it might be more expensive than paying for cloud because the farm needs to be up to date, have well maintained infrastructure and easy access to devices.         


Running tests is only part of the job. The main part is to analyze the results and make sure our product under test is stable. This is where reporing comes in place. 
There are usually external tools such as Allure Report, Mochawesome(for js frameworks), JUnit, XCTestHTMLReport (for XCUITest only), etc that can be used with frameworks.   
Apart from a test report with a number of failed or passed tests there is a Code Coverage report that is available in Android Studio IDE using Espresso framework. It shows how many lines of code has been exercised and will give you a per cent after running tests. Logs are always available when running locally as it is running in Android Studio IDE itself. When integrating with CI tools there is always a way to add a reporting tool – example Jenkins with Allure report plugin, Azure Pipelines with Junit etc
Appium itself does not provide any build-in report however depending on the tool and language you choose to author tests it will have options such as Allure Report that is compatible with almost all languages and tools, HTML report that can be used with Cucumber as example and other tools like Rspec, JUnit report, Extend report for Java frameworks mainly etc. Customisation is usually available to add screenshots on failure, send a test results by email/slack or any other communication channels, display a stack trace on failure with proper error message.
Cloud solutions are also providing a nice reporting with video, screenshots, device logs, session logs which can help with analyzing test results. 

API Calls and Database Support

Data intensive apps are dynamic by nature In this case instead of hardcoding the values with indexes best practice is to embed API calls into your tests to make them more stable and data-oriented.
Programmatically it is possible to add API support using libs such as Rest-Assured in Java, Supertest in JavaScript, Client-api on Ruby etc with Appium and reuse it later in our tests. The only challenge might be time consuming as you will need to integrate the API lib into your framework and troubleshoot possible problems that might come on the way. Espresso is having the same approach and external lib can be used in the framework.
To be able to use APIs calls with XCUITest framework it can be done with a help of popular libs as well such as Swifter, SBTUITestTunner or Embassy. 

Community and Support

Test automation for mobile is not easy to accomplish. It’s relatively new and the variety of devices, OEMs, OS vendors and development frameworks makes it even more challenging. If you are using a commercial tool, most likely you have support that will have your back if something isn’t working or if you’re not sure how to execute one thing or another. 

If you are using open source, you will encounter challenges and most likely rely on the assistance of either a consultant of the community for answers. Make sure you have those. 

Appium is popular and has a wide and active community with 11,7k Stars and about ~4.7K Forks and frequent releases. There is a Discuss Appium forum where one can find an answers. It is possible to filter topics by category, top categories and latest questions, search for query etc – that makes it easier to find a solution or answer for Appium users. While it is well documented, Appium has a lot of edge cases that are specific to the app or development framework. While you can ask around the overhead of trying all possible suggestions is time consuming and usually there are slight variances. These cases can lead to lengthy implementation cycles and will significantly increase your cost and time.

Espresso is a native framework created by Google and because it is part of developer code documentation is available on android developers website here and also there are forums such as AndroidForums to look for needed information regarding Espresso. There is no dedicated forum for Espresso framework only unfortunately. 

XCUITest native framework is developed by Apple so documentation is available on Apple Developer website. Few Apple Developer forums such as Developer Apple or Swift forum might contain some additional information about XCUITest framework. Similar to Espresso there is some information on Stackoverflow in a question/answer format however the community is still growing. 


Test Automation Framework Scorecard

Download Now



A lot to consider when choosing the framework for your team.Below you can find an excel that will help you to choose the best tool for your team. We summarized the key factors and you can adjust the weights of each criteria based on the importance to your app and drive a fit score to match the framework to your criteria 


Many of the challenges mentioned earlier are addressed by commercial tools such as 21. 21 is an AI based test automation tool for mobile that doesn’t require scripting (so skill-set is not an issue), supports all platforms and development frameworks, includes built in integration to devices as well as easy-to-setup integration to 3rd party device providers (including mapping all the relevant APIs), rich reporting, CI integration and mich more. Authoring tests with 21 takes minutes and its self-maintenance capability makes updating tests to user flow changes a breeze. 


21 also includes a full analytics layer including integration to production for coverage reporting.


Related Post

Comments are closed.

Subscribe To Our Newsletter


Connecting testing and production.Autonomously.

California, USA

© Copyrights 2020 21Labs All rights reserved. Made with By 21Labs