Coverage is a metric that measures the risk with your release. It measures what is and isn’t tested today. A VP of Engineering, using the right coverage benchmarks can make an educated decision about whether to release or take additional steps to mitigate the risk, which might impact release date, commitment to customers and next sprint.
Coverage can be measured in the following ways:
Code coverage measures how many lines of code are covered by test cases. Code coverage looks at the number of lines of code that gets executed during the test run. Code coverage can be divided into a number of segments:
Code coverage is relatively easy to measure and is pretty accurate since the lines of code are definite. The challenge with code coverage is that, while it’s easy to measure, it has little correlation to the user experience and is not a great representation of the risk in your release. You can have 100% code coverage and your app will constantly crash on various environments and mobile devices. You app might also crash depending on network conditions, other apps working in the background, depending on various sensors being on or off etc.
Another cons for code coverage is that most test coverage tools today analyze code coverage only against unit tests and do not support integration tests, UI tests, functional tests etc.
Coverage.py is a coverage analysis and reporting tool for Python. It includes 3 modules
Cobertura is a free jcoverage Java code coverage tool. It supports ant, maven or command line. While instrumenting classes, Cobertura generates cobertura.ser file containing basic information about each class. On running tests, it generates coverage reports for each class within each package.
Test coverage analyzes the manual and automated tests in relations to either the possible flows or in relations to user behavior. Test coverage can go beyond just actions. To properly measure coverage you want to take into account:
Test coverage is much more difficult to measure since access to the information is limite and the number of potential tests is in the thousands. Having said that tier one brands that take pride in their mobile presence and maybe even drive business through mobile cannot afford not to measure and report coverage and test to optimize their tests to production.
Test coverage can be further broken down into 3 sub-categories:
21 is the only tool connecting testing and production today for coverage reporting at the device and the scenario level. It allows you to easily author automated functional and UI tests, connect to your CDP such as Mixpanel, Segment or Amplitude and correlate test actions to events in production so you can understand your coverage, plan and optimize your tests and use data to release with confidence.
Signup today and start measuring your test coverage