2 1 L A B S

blankIn a mobile application driven world every company comes to the point when they need to use a cloud platform. There are many reasons for that – too expensive to buy various devices over and over,  limited budget for device refresh. That is when cloud device labs become a viable solution:. 100s of devices, easy integration, parallel testing – these and more features can be helpful with overall testing coverage and quality for our different from company to company needs. The only challenge is – what tool to choose? Let’s talk about popular tools such as Perfecto, Sauce Labs, Browserstack, Kobiton.
Below are the main considerations to keep in mind when choosing a mobile device lab.
 
Below are the main considerations to keep in mind when choosing a mobile device lab.
 

Variety of devices

Variety is a key reason for transitioning to a cloud based device lab. The effort to extend your coverage and align your lab to your user base is marginal and the ability to debug specific devices without the need to worry about procuring them is a big plus.
 Sauce Labs supports different emulators/simulators starting from Android 5.1 to the latest and iOS 10.3 to the latest. Real device support has different device models – Nokia, Samsung, iPhone, Huawei, Sony Xperia, LG etc – that will help to cover not only OS versions but device models in testing. Sauce Labs provides support for web testing as well with various browsers on Windows, Linux and Mac OS platforms. Browsers are up to date so latest versions can be covered. Sauce Labs are also updating their devices and browsers to the latest version within 2 days once it is released.
Browserstack provides real devices and tablets to run tests on with different device models – Samsung, OnePlus, Motorola, Xiaomi, Google, iPhone etc. Browserstack supports web testing so it is possible to have a good coverage when testing on Safari, Microsoft Edge, Internet Explorer, Chrome, Firefox, Opera, Yandex on Windows and Mac OS. Browserstack is up to date with new versions of devices and web that gives an ability to run tests on newest versions without downtime.
Similarly to Sauce Labs and Browserstack Perfecto also has a wide selection of real devices that can be used for testing apps on Android/iOS and web browsers. Some rare devices – Lava, Meizu, Blackberry – are also supported. Speaking about web testing, most used browsers – Microsoft Edge, Safari, Internet Explorer, Chrome, Firefox – are also covered with latest versions. They recently released support of emulators as well.
Kobiton supports only real devices for iOS and Android. There are also different models from popular devices, such as Samsung, Google, iPhone to rare models such as U, G3 Beat etc. Kobiton has tablet support as well as various iPads, Galaxy Taps can be used for testing web, native and hybrid apps. Same as it’s competitors Kobiton has devices up to date. There is no web browser support yet.
 

Type of devices: Real devices, emulators, web browsers

Sauce Labs and Perfecto provide full support with a range of emulators/simulators to test as well as real devices and web browsers. When there is a need to to test something fast – emulators/simulators can be a good help for it. For integration tests – real devices can be a best solution to ensure the end user won’t face any issues with using the app. Nowadays companies are implementing their products not only on mobile but web – most of the browsers are covered with Sauce Labs and Perfecto. Sauce Labs and Perfecto provides “all in one” solution – sanity tests on virtual devices, integration/system testing on real devices and browser testing that covers most of the browsers for Mac, Windows, Linux platforms
 
Kobiton supports only mobile real devices, there is no support for the web.
Browserstack supports a variety of web browsers and real devices for mobile which can be a good solution for companies that have web based apps and mobile apps as well.
 

Manual testing

Most organizations don’t achieve 100% automation and the ROI to automate everything is low. Some scenarios are better off tested manually. Being able to run automated and manual tests in a single lab, with a reasonable level of user experience is important. Alternatively organizations will need to have two separate solutions.
Perfecto gives an ability to perform manual testing. In addition to standard user actions like swipe, scroll testing on devices in perfecto provides different capabilities to cover more test cases – location, injecting images, authentication user fingerprint, face ID, injecting audio to test siri or similar features, change networks etc. Currently manual testing is limited to real devices only.
Kobiton also provides an ability to test your apps manually. Users can download screenshots, zoom in/out the screen, touch,scroll and swipe. Location can be changed to any using coordinates to perform tests where GPS is involved. Timezone can be changed for Android only as of now however iOS support is coming shortly.
Sauce Labs and Browserstack have similar approaches for web testing and mobile testing. It launches either a web browser or mobile real device and all actions will be the same as if a web browser or physical device was used. Sauce Labs provides a feature to create a screenshot while testing and also another participant from the organisation can be invited to observe the session. Browserstack gives an ability to set a location for your test – you can select it from the location list, and also change the resolution within a session.

Sophisticated AI Test Automation for
Mobile

Try Now. No Credit card required
Get Started
 

Reports

To run tests is one part of the job. To analyze the results from a test session is the second part . A solid and rich reporting can significantly reduce time to resolution.

Browserstack provides recorded video of your tests – it gives a clear picture of what was happening during test execution. Video is available right after the test session. In addition to that there are screenshots taken during test run – it is possible to view screenshot within the video also. Depending on the platform there will be Selenium logs when running on web browser and Appium logs when running on mobile devices. In addition to that there are console logs that are equal to what you can see in your console when running tests locally. Network logs are also available if additional desired capability is added to the config file. By copying a curl command to the local console – logs can be viewed that makes it easier to identify the root cause of a failure.

Sauce Labs also have video recorded after each session as well. It has commands where all API calls are displayed. While test execution screenshots are taken – they can be found next to API calls steps. If  testing is performed on iOS simulator then logs will be embedded within the Appium log. Android emulator  logs can be viewed on the Metadata tab in Logcat.log file. Under Metadata tab screenshots, videos and general information such as start time, end time, build name etc can be found. Network logs are visible to visually analyze how long each process took – it helps to debug slow network issues. Network logs can be available with specifying additional capability in the config file.

Kobiton, similarly to its competitors, also provides HTTP commands for each step in test execution together with screenshots so it is easy to debug the failure if any. In a separate tab there is a full video of test execution that can be shared to anyone in your team. It gives an ability to download file Appium logs, Selenium logs and Device logs for proper debugging.

Perfecto provides videos with the ability to watch it real time, not after test execution. Test flow is available with all the steps that we performed in the test session. Screenshots can be viewed for each step in execution that helps to identify when and what was failing. This particular device can be opened for debugging purposes where you can actually retest any failure or investigate why failure occured. Additionally if a bug is confirmed it is possible to add it to JIRA from the session itself as it is fully integrated. It allows you to download the full report, video, files to your local.
Manual testing
Most organizations don’t achieve 100% automation and the ROI to automate everything is low. Some scenarios are better off tested manually. Being able to run automated and manual tests in a single lab, with a reasonable level of user experience is important. Alternatively organizations will need to have two separate solutions.
 

Perfecto gives an ability to perform manual testing. In addition to standard user actions like swipe, scroll testing on devices in perfecto provides different capabilities to cover more test cases – location, injecting images, authentication user fingerprint, face ID, injecting audio to test siri or similar features, change networks etc. Currently manual testing is limited to real devices only.

Kobiton also provides an ability to test your apps manually. Users can download screenshots, zoom in/out the screen, touch,scroll and swipe. Location can be changed to any using coordinates to perform tests where GPS is involved. Timezone can be changed for Android only as of now however iOS support is coming shortly.

Sauce Labs and Browserstack have similar approaches for web testing and mobile testing. It launches either a web browser or mobile real device and all actions will be the same as if a web browser or physical device was used. Sauce Labs provides a feature to create a screenshot while testing and also another participant from the organisation can be invited to observe the session. Browserstack gives an ability to set a location for your test – you can select it from the location list, and also change the resolution within a session.

Comparing device lab providers

Ranking Perfecto, Sauce Labs, Kobiton and Browserstack
Get Started
 

Related Post

Leave a Comment

Subscribe To Our Newsletter

21Labs

Connecting testing and production.Autonomously.

California, USA

© Copyrights 2020 21Labs All rights reserved. Made with By 21Labs