Sealights Test Optimization with other test coverage tools

In this article we will discuss the possible affect of the test optimization tools on other coverage tools, such as SonarQube and other coverage tools.

Test optimization and code coverage conflict

Before we discuss the technical details of how to work with Sealights Test Optimization tool and other companies coverage tools, it is important to understand the innate conflict of such tools.

Coverage tools measure how much of the code is being tested when test stages are ran. With this tools it is possible to make statements such as ‘unit tests meet the quality gate and cover 90% of the code when ran’ or ‘our nightly tests test 80% of the code’.

Obviously, if we look at the matric of coverage alone as a measurement of test or code quality, we will be inclined to create and run more tests, as it will raise the coverage percentage. This will drive the amount of tests, and the time we spend testing up.

Test optimization tool are solving the opposite problem - preventing situations of long test stages, and test amount too large to manage. With affective selection of only required tests, who test code that recently been changed, this tool is designed to to drive the amount of test in each test stage execution down.

Test optimization and modified test coverage

Once we understood this basic conflict, it is easy to understand why it is possible at times, that turning the Sealights Test Optimization tool will cause failure in tools managing coverage gates.

when turning on Test Optimization you do not want to cover close to 100% of your entire code. You want to save time and cover only recently changed (and thus prone to failure) code.

Sealights offers code coverage tools. In this tool, it is possible to see overall code coverage, as well as modified code coverage, and set up gates for both.

The Test Optimization tool is almost designed to drive down the overall coverage percentage, but keep the modified coverage percentage, and help your QA management focus on adding tests where it really matters, and not waste time on testing old and already well tested code.

The Sealights code coverage tool will:

• Show clearly the modified coverage VS the overall coverage

• Allow to set up gated for modified coverage

• Show test optimization status in the build information

In addition, the Sealights Test Optimization runs full runs of the entre test stage every X days, according to your chosen policy, and during those runs overall coverage is still valid.

Test Optimization and other coverage tools

When using other coverage tools it is important to first and foremost to be aware of all of the above, and know what to expect.

Your coverage percentage may go down, and subsequently, your coverage gates may fail.

We suggest the following:

1. Where possible, change the coverage gates to be dependent on modified coverage only. This feature is included with the SonarQube tool, for example

2. For overall coverage, only check the coverage during test optimization full runs

3. Inform relevant stake holders in the company about this process, and make sure all parties know how to prepare and adjust.