2 1 L A B S

 

blank

When developing a test automation strategy, choosing a framework is a key consideration. The decision is impacted by answers to questions such as what scripting language to go for, where will the tests be executed, how are we reporting results, what the pipeline looks like and so on. I would like to suggest a skeleton that I used in the past when deciding on a framework and what I learned from years in test automation about decision factors. I’ll share a comparison of 3 popular open source frameworks such as Appium, Espresso and XCUITest.

Link to Part II

 

 

Platform Support

The platform you build the application for is probably the single most important factor influencing your decision. Automation frameworks offer different levels of support for each platform. Common variations in my experience includes support for identifying objects, executing actions (e.g set text), execution performance etc.

For Android only apps  – Espresso would be an option as it is fully integrated with Android Studio and works faster than Appium. For iOS only apps – XCUITest is likely to be the best solution as it is native framework created by Apple. If you need to support both iOS and Android – now or in the future – maintaining two frameworks is expensive and Appium, the cross platform test automation framework for mobile is probably a good choice.

 

 

Development Framework

The decision is not only impacted by the platform but also by the framework used to develop the app. An app can be developed using native Swift and Java or by a cross platform framework such as React, Xamarin, Flutter, Cordova and others which are wrapped inside a native container. The balance between native components and cross platform code is often a tradeoff between native experience and time/cost of development and maintenance. From test automation standpoint, if your app includes cross platform components you want to carefully validate that your test automation framework supports all relevant elements and actions. I’ve seen edge cases around object identification (which can mostly be solved with testIDs), set text actions etc. For example if your application is based on React Native, Detox might be a better, yet relatively proprietary framework, to consider. One known issue is the performance of various automation framework when used to test cross platform applications. Appium, for example, tends to perform 3 to 5 times slower when used to test React Native applications.

There is a way to test hybrid applications with Espresso and XCUITest but it is going to be difficult to adjust it to hybrid app as these frameworks are usually used for Native automation. You can easily use Appium for testing web apps on mobile also. Speaking about Espresso and XCUITest – it won’t be possible as they are native development solutions. Regarding Xamarin apps – usually Xamarin.UITest is used as it is built in the Xamarin framework. Still some people can use Appium however the preference is to use the same ecosystem for development and testing in this case. Native apps are supported by Appium, Espresso (only Android), XCUITest(only iOS).

 

Coverage

Picking up a framework that supports a scripting language that your team is familiar can cut time, ramp up and cost. Appium supports a wide range of languages such as JavaScript, Python, Java, Ruby, PHP, C#. Most of these languages are widely used so most probably QA in your team will know at least one.

Unfortunately Espresso supports only Java or Kotlin, making it a bit limited compared to Appium. Java is pretty popular though, but one needs to take that into consideration. XCUITest, similar to Espresso,  supports only 2 languages – Swift and Objective-C. It is rare to find an automation engineer who knows Swift. The decision might impact the IDE they use, the time that it takes them to debug issues, training etc.        

Sophisticated AI Test Automation for
Mobile

Try Now. No Credit card required
Get Started
 

Framework creation

Apart from the tool itself we need to think how much time needs to be spent on setting the environment up. Setting up the environment includes configuring the execution infrastructure: simulators and/or real devices connected locally, shared on-prem or in the cloud. It might include other tasks such as setting up a central repository of locators, mechanism for ingesting and parsing test data, creating reusable components (e.g login) and reporting.
 
Properly setting these up means 2-3 months before one actually authors his first test. Unfortunately someone still needs to QA the app, which means keeping up with manual testing while setting this up.
 
Appium is a bit difficult to configure for QAs who are new to mobile automation. Appium is a client-server framework so one needs to set up an Appium server and configure devices or emulators to it. It might take a week or more just to make real devices and simulators working on your machine. Integrating a cloud based device lab can also take several weeks. While the initial connection is relatively easy, you might want access to logs, being able to keep track of the execution ID so you can see videos, keeping configuration files of the devices, monitor their availability etc.
 
With XCUITest the main challenge can be the language because Swift or Objective-C is a rare skill at the moment within QAs so it might take time to learn it. However knowing any programming language will help to learn Swift or Objective-C syntax. Once familiar with it – it can be easy to design UI tests, integration is not taking much time either because it is part of xCode project with all needed setup.
 
Espresso framework is embedded into Android Studio IDE so there is no issue to set up an emulator for your tests or test runner. It is using Gradle so you can use its logs for debugging also. Espresso supports Java or Kotlin – Java is one of most used languages within QAs – so it is easy to find skilled professionals to write tests.
 


Authoring Tests

A few weeks went by, your framework is ready and your are about to author your first test. Depending on language you choose and the tech skills of your team members, speed of writing scripts can vary. Appium, being a cross-platform tool, gives you the ability to create tests for both apps at the same time. Having said that authoring tests with appium takes 2-3 days.
 
With XCUITest the main challenge can be the language because Swift or Objective-C is a rare skill at the moment within QAs so it might take time to learn it. However knowing any programming language will help to learn Swift or Objective-C syntax. Once familiar with it – it can be easy to design UI tests, integration is not taking much time either because it is part of xCode project with all needed setup.
 
Espresso framework is embedded into Android Studio IDE so there is no issue to set up an emulator for your tests or test runner. It is using Gradle so you can use its logs for debugging also. Espresso supports Java or Kotlin – Java is one of most used languages within QAs – so it is easy to find skilled professionals to write tests.
 


Cost of Tooling

While all open source frameworks are free, cost of ownership isn’t and the community, hiring for skill-set, ramp up, ease of integration, pre-built libraries and effort to troubleshoot problems should all be taken into account. I have colleagues who spent days troubleshooting a certain Appium function that didn’t work on one application or another. If the community is big enough and you’re lucky you will find someone who faced a similar challenge and can guide you through the resolution. If the community is smaller, such as the case for XCUITest, you might search the web and poke different forms for weeks and might end up learning that it’s a vendor issue. More about communities in sector 12.

Take into account the cost of hiring for the skill-set (e.g XCUITest developers are rare hence they are more expensive) compared to Java or Appium. Take into account the cost of authoring tests. In addition to actually writing a test it is important to take into account a time of debugging a test. Espresso and XCUITest frameworks provide clear logs that can be helpful when debugging. Appium is a black box tool but you can see logs on the appium running server itself. Also IDEs such as Eclipse or RubyMine have a built-in debug functionality that makes debugging easier.

Open source tools are free but If one takes into account the hours spent to make open source tools fit your needs – you are looking at a high cost that the organization bears in a different form.      

The tradeoff between testing on real devices and virtual devices is the tradeoff between cost and risk. In a nutshell real devices are more expensive to test but for some applications they might be the difference between 80% confidence and 90% confidence. For most apps virtual devices can give you decent coverage in terms of testing the functionality and UI. Virtual devices are also cheaper to test since you can set up the virtual device with any flavor and configuration you want.

 

Test Automation Framework Scorecard

Download Now

 

Related Post

Leave a Comment

Subscribe To Our Newsletter

21Labs

Connecting testing and production.Autonomously.

California, USA

© Copyrights 2020 21Labs All rights reserved. Made with By 21Labs