2 1 L A B S
Measure coverage

Coverage is a metric that measures the risk with your release. It measures what is and isn’t tested today. A VP of Engineering, using the right coverage benchmarks can make an educated decision about whether to release or take additional steps to mitigate the risk, which might impact release date, commitment to customers and next sprint.

Coverage can be measured in the following ways:

Code Coverage

Code coverage measures how many lines of code are covered by test cases. Code coverage looks at the number of lines of code that gets executed during the test run. Code coverage can be divided into a number of segments:

  • Functional coverage – % of functional calls being tested
  • Branch coverage – Assuming your code includes condition statements, % of branches (if…else…) that are being tested
  • Loop coverage – Assuming your code includes loop statements, % of code inside these loops that is being tested

Code Coverage – Pros and Cons

Code coverage is relatively easy to measure and is pretty accurate since the lines of code are definite. The challenge with code coverage is that, while it’s easy to measure, it has little correlation to the user experience and is not a great representation of the risk in your release. You can have 100% code coverage and your app will constantly crash on various environments and mobile devices. You app might also crash depending on network conditions, other apps working in the background, depending on various sensors being on or off etc.

Another cons for code coverage is that most test coverage tools today analyze code coverage only against unit tests and do not support integration tests, UI tests, functional tests etc.

Code Coverage Tools Vs Test Coverage Tools

Python

Coverage.py is a coverage analysis and reporting tool for Python. It includes 3 modules

  • Execution – it runs the code and monitors what lines of code are executed.
  • Analysis – it examines the code to determine what lines could have run.
  • Reporting – producing a coverage number and an indication of missing execution based on the Ana;ysis

Go

Gocov generates function coverage reports. GoCovGUI is a GUI wrapper for Gocov that can also produce XML or HTML reports.

Java

Cobertura is a free jcoverage Java code coverage tool. It supports ant, maven or command line. While instrumenting classes, Cobertura generates cobertura.ser file containing basic information about each class. On running tests, it generates coverage reports for each class within each package.

Test Coverage

Test coverage analyzes the manual and automated tests in relations to either the possible flows or in relations to user behavior. Test coverage can go beyond just actions. To properly measure coverage you want to take into account:

  • Network conditions
  • Type of mobile devices
  • Other apps operating in the background
  • Sensors on/off 
  • Location

 

Test Coverage – Pros and cons

Test coverage is much more difficult to measure since access to the information is limite and the number of potential tests is in the thousands. Having said that tier one brands that take pride in their mobile presence and maybe even drive business through mobile cannot afford not to measure and report coverage and test to optimize their tests to production.

Test coverage can be further broken down into 3 sub-categories:

  • Requirements Coverage – Tests are defined to provide maximum coverage of the product requirements mentioned in various Requirement Specification documents.The requirements can go to the level of what data is valid and what is not, default state etc.
  • Features Coverage – As the name suggests the focus of this index is to assure maximum coverage of product features. Such an example would be to validate that the default language aligns to the location.
  • Risk Coverage – Risk based approach doesn’t take all user flows at equal value. Naturally a checkout flow for a shopping app is much more valuable than changing the phone number. Risk coverage gives a “value’ to each flow.

21 is the only tool connecting testing and production today for coverage reporting at the device and the scenario level. It allows you to easily author automated functional and UI tests, connect to your CDP such as Mixpanel, Segment or Amplitude and correlate test actions to events in production so you can understand your coverage, plan and optimize your tests and use data to release with confidence. 

Signup today and start measuring your test coverage

 

Related Post

Leave a Comment

Subscribe To Our Newsletter

21Labs

Connecting testing and production.Autonomously.

California, USA

© Copyrights 2020 21Labs All rights reserved. Made with By 21Labs