When developing a test automation strategy, choosing a framework is a key consideration. The decision is impacted by answers to questions such as what scripting language to go for, where will the tests be executed, how are we reporting results, what the pipeline looks like and so on. I would like to suggest a skeleton that I used in the past when deciding on a framework and what I learned from years in test automation about decision factors. I’ll share a comparison of 3 popular open source frameworks such as Appium, Espresso and XCUITest.
The platform you build the application for is probably the single most important factor influencing your decision. Automation frameworks offer different levels of support for each platform. Common variations in my experience includes support for identifying objects, executing actions (e.g set text), execution performance etc.
For Android only apps – Espresso would be an option as it is fully integrated with Android Studio and works faster than Appium. For iOS only apps – XCUITest is likely to be the best solution as it is native framework created by Apple. If you need to support both iOS and Android – now or in the future – maintaining two frameworks is expensive and Appium, the cross platform test automation framework for mobile is probably a good choice.
The decision is not only impacted by the platform but also by the framework used to develop the app. An app can be developed using native Swift and Java or by a cross platform framework such as React, Xamarin, Flutter, Cordova and others which are wrapped inside a native container. The balance between native components and cross platform code is often a tradeoff between native experience and time/cost of development and maintenance. From test automation standpoint, if your app includes cross platform components you want to carefully validate that your test automation framework supports all relevant elements and actions. I’ve seen edge cases around object identification (which can mostly be solved with testIDs), set text actions etc. For example if your application is based on React Native, Detox might be a better, yet relatively proprietary framework, to consider. One known issue is the performance of various automation framework when used to test cross platform applications. Appium, for example, tends to perform 3 to 5 times slower when used to test React Native applications.
There is a way to test hybrid applications with Espresso and XCUITest but it is going to be difficult to adjust it to hybrid app as these frameworks are usually used for Native automation. You can easily use Appium for testing web apps on mobile also. Speaking about Espresso and XCUITest – it won’t be possible as they are native development solutions. Regarding Xamarin apps – usually Xamarin.UITest is used as it is built in the Xamarin framework. Still some people can use Appium however the preference is to use the same ecosystem for development and testing in this case. Native apps are supported by Appium, Espresso (only Android), XCUITest(only iOS).
Unfortunately Espresso supports only Java or Kotlin, making it a bit limited compared to Appium. Java is pretty popular though, but one needs to take that into consideration. XCUITest, similar to Espresso, supports only 2 languages – Swift and Objective-C. It is rare to find an automation engineer who knows Swift. The decision might impact the IDE they use, the time that it takes them to debug issues, training etc.
While all open source frameworks are free, cost of ownership isn’t and the community, hiring for skill-set, ramp up, ease of integration, pre-built libraries and effort to troubleshoot problems should all be taken into account. I have colleagues who spent days troubleshooting a certain Appium function that didn’t work on one application or another. If the community is big enough and you’re lucky you will find someone who faced a similar challenge and can guide you through the resolution. If the community is smaller, such as the case for XCUITest, you might search the web and poke different forms for weeks and might end up learning that it’s a vendor issue. More about communities in sector 12.
Take into account the cost of hiring for the skill-set (e.g XCUITest developers are rare hence they are more expensive) compared to Java or Appium. Take into account the cost of authoring tests. In addition to actually writing a test it is important to take into account a time of debugging a test. Espresso and XCUITest frameworks provide clear logs that can be helpful when debugging. Appium is a black box tool but you can see logs on the appium running server itself. Also IDEs such as Eclipse or RubyMine have a built-in debug functionality that makes debugging easier.
Open source tools are free but If one takes into account the hours spent to make open source tools fit your needs – you are looking at a high cost that the organization bears in a different form.
The tradeoff between testing on real devices and virtual devices is the tradeoff between cost and risk. In a nutshell real devices are more expensive to test but for some applications they might be the difference between 80% confidence and 90% confidence. For most apps virtual devices can give you decent coverage in terms of testing the functionality and UI. Virtual devices are also cheaper to test since you can set up the virtual device with any flavor and configuration you want.