2 1 L A B S

blankWith over 60 Android devices launched since the beginning of 2020 and 18 iOS versions released in the first half of 2020, mobile testing is a challenge for many organizations. You  constantly have to catch up with the permutations of new OS versions and test a growing number of devices. Whether you are testing your application manually or through automation you can find two types of devices to test on – virtual devices (emulators/simulators) and real devices.

It is important to simulate the user behavior while testing and also cover as many devices as possible to ensure the end users won’t face any issues using the app.

Is it enough to test on virtual devices or is investing in real devices is must?
What is a virtual device? What’s the difference between emulators and simulators?
A virtual device is a “program” that behaves like a real device and imitates a mobile device’s real software and hardware. There are two types of virtual devices – Emulators and Simulators.

Both emulators and simulators serve the same purpuse – you can run tests using them. Popular development tools such as Android Studio IDE or xCode already have them integrated. Virtual devices are useful for testing basic scenarios and give you immediate availability without the need to wait in line at the Apple store to get access to the latest iPhone or iPad..

Simulators are mimicking software while emulators are mimicking software and hardware. Simulators are common for iOS while emulators are more common for Android.

Virtual devices allow you to debug the code easily with the help of proper logs without a need to buy a real device for this purpose.

It is possible to simulate some edge cases like GPS Location check and low battery however take the results with a grain of salt. They will never produce results that fully align with those you would get on a real device. They do give you a sense

As we are trying to simulate user actions with our automation testing the main disadvantage will be “testing on virtual device” itself. Apps might not work the same way on Real Device and there are zero users who actually use virtual devices to interact with the app.

What are Real Devices?

Real devices are literally physical devices – tablets, mobile phones, laptops etc.  Naturally testing on real devices provides more reliable results and an experience close to that of the user.
Real Devices, as opposed to virtual devices, can test certain scenarios such as:
– Performance
– Lack of network
– GPS locations
– Push notifications
– Messages

When to test on a virtual device and when to test on a real device?

The tradeoff between testing on real devices and virtual devices is the tradeoff between cost and risk. In a nutshell real devices are more expensive to test but for some applications they might be the difference between 80% confidence and 90% confidence. For most apps virtual devices can give you decent coverage in terms of testing the functionality and UI. Virtual devices are also cheaper to test since you can set up the virtual device with any flavor and configuration you want.
You get a fresh instance and don’t have to worry about resetting settings and the initial state. But virtual devices might fail to detect memory leak, high CPU usage, latency and network delays. They will also not support automating certain scenarios such as scenarios involving sensors. Finally some devices include custom OS and OEM “skin” on top of the original OS. Virtual devices typically would not include custom OS and skins. Other differences between virtual and real devices include the lack of SIM card, the type of file being tested (for iOS simulators .APP vs. .IPA for real devices) and app store accessibility.
Real device testing comes at cost: One has to maintain a certain number of devices, procure, maintain and repair these devices. Also automation on real devices requires additional efforts: system messages will appear, the device doesn’t always reset to its initial state etc. Cloud solutions can help in this case as they provide 100s of real devices on the cloud. No need to worry about OS, screen resolution, model etc – you can choose any configuration and change it anytime you need in future.

Best Practices for virtual vs. real device testing

Developers in most agile companies use simulators and emulators to do a basic sanity. Given each device is expensive and speed as well as frequency of testing impacts the decision to test on virtual devices.
After a basic sanity has passed apps will go to QA. Depending on the functionality that needs to be tested, QA will choose virtual or real devices. If you need to test GPS Location, weather, lack of network, push notifications etc  – Real Devices will be the best choice. 
It is also better to use Real Devices for final regression tests before release to make sure that app behaves as expected with the software and hardware similar to that of the end user.
For better device coverage it is useful to use cloud solutions that provide real devices on cloud. We would also recommend in some cases to run your integration tests on real devices. These end-to-end flows can behave differently between virtual and real devices. 21 supports the execution of automated AI based self-maintained mobile tests on both virtual devices and real devices. 21 has seamless integration to most cloud based device labs such as Sauce Labs, Perfecto and others.
With thousands of tests authored on our platform. we’ve seen a number of cases where scenarios behaved differently on emulators vs. real devices. We’ve developed sophisticated locators and root-cause analysis engine to address the challenges of testing on mobile devices.

Related Post

Comments are closed.

Subscribe To Our Newsletter


Connecting testing and production.Autonomously.

California, USA

© Copyrights 2020 21Labs All rights reserved. Made with By 21Labs