Coverage Trend Report

The Coverage Trend Report is the first report available under Quality Analytics. It displays app code coverage over time.

Open

First Time Experience

When you first enter the Quality Analytics page, you will see an empty list of saved report (or public reports if those were created previously). Click on the “Create new report” button at the top right corner to create your first report. Follow the instructions in the “Generating A Report” section below , in order to complete the report configuration.

Open

Please notice! Once you create a report you have to click on the save button, to add it to your saved reports list. That way you can return to this report any time to track progress.

By default, the reports you create are private and only available for you. To learn about sharing options see the relevant section below.

Generating A Report

Generating a report is as simple as selecting the app and branch you are interested in.

You have different options to view the data:

Date Range

The popular options available are: Last month, Last 2 months, Last 3 months, Last 6 months, Last year.

If none of the above is what you need, you can always select a custom date range. We recommend a minimum of 4 weeks.

Please note that the start date for the data shown in the charts depends on the selected builds (see section below) and is currently limited to the following Monday or the 1st of the month (for all builds option).

Test Stages

A list of all test stages reported for the selected app and branch during the past year, ordered alphabetically. Each test stage has its own color that is used in the charts.

You can select / deselect any of the test stages to focus on the data you want to see in the charts and asses how different test stages contribute to your overall coverage.

Builds

There are two ways to analyze quality data overtime: either look at all builds of the selected app/branch or only track builds that were manually marked as reference builds (usually these are production/released builds).

• When looking at all builds, you can select in which intervals you would like to aggregate the data. Options are: 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month. When selecting 1 month, the data shown in the chart will start at the 1st of the relevant month. For the rest of the interval options, the data shown in the chart will start at the following Monday.

• When tracking only reference builds, you can select/deselect reference builds from a list (most likely you would like to include all reference builds).

Generating The Report Using API

You can also generate the report using our API. Read more at https://sealights.atlassian.net/wiki/spaces/SUP/pages/2852683826 .

Understanding The Data

The report includes two charts: Modified Coverage and Overall Coverage.

Both charts display an aggregated coverage on top of the specific test stages coverage and include:

• Y-axis: Coverage % (0-100%)

• X-axis: Time, according to the filter selection and intervals

Build Intervals

When selecting any of the interval options (1 week, 2 weeks, 3 weeks, 4 weeks, or 1 month) any point in the X-axis and in the chart line represents an interval. Interval X takes all code changes from the last build in interval X-1, and takes all coverage from all related test stages that were reported in all builds within interval X. (e.g. Interval of 1 month: coverage in July means all code changes since last build on June and their coverage from all test stages executions reported on builds within July).

Reference Builds

When selecting the reference build option any point in the X-axis and in the chart line represents a single reference build. Reference build X takes all code changes from reference build X-1, and takes only the coverage from the test stages that were reported to reference build X (similar to the calculation in the Dashboard).

Modified Coverage Chart

Modified coverage is the coverage of your code changes. Why is it important to track it? Your quality risks are code changes you haven’t tested. The higher the coverage, the lower the chances for escaping defects. You want to see high coverage overtime, to reduce risks. High modified coverage overtime is an indication for a good quality culture. It means that any new or modified code is tested.

Overall Coverage Chart

Overall coverage is the coverage of your entire code. You should track the overall coverage to make sure that overtime your trend is positive, mainly by covering new and modified code. You might see drops in cases you had major code changes with no coverage.

Advanced Report Features

Selecting / Deselecting Test Stages

Using the left filter pane, you can select which test stages will be shown in the charts by clicking on the wanted test stage. Once you change the selection, the report will be updated and the relevant test stage will be added or removed from the chart. As a result, the bold Blue Selected Test Stages aggregated coverage line will be recalculated and updated.

Coverage Of All Test Stages

When your report includes only a subset of your test stages, you might be interested to view the Selected Test Stages aggregated coverage vs. the All Test Stages aggregated coverage (the bold Green line). To add this line to your chart, click on the “Chart elements” and toggle on the “Coverage of all test stages” option.

Number Of Methods

When examining your coverage trend, seeing the data in the right context is important. To that end we have added an option to visualize the scope of the change in the modified chart and the scope of the app that is examined in the overall chart. To add the methods bar to your charts, click on the “Chart elements” and toggle on the “Number of methods” option.

Coverage Quality Gate

When examining your coverage trend, it is important to see how you are meeting the coverage quality gate. To that end we have added an option to visualize the current modified and overall quality gate (if set). To add the coverage quality gate to your charts, click on the “Chart elements” and toggle on the “Coverage quality gate” option. If you wish to set or edit the gate, you can click on the link on the toggle and review the current quality gate setting.

If no modified / overall quality gate is currently set, the following element will be shown on the chart:

Number Of Production Defects

One of the most common KPIs around quality is the number of production defects that were opened against a build or a version. SeaLights now allows you to visualize the number of production defects and compare it to the coverage trend. To add the number of production defects as added bars in your charts, click on the “Chart elements” and toggle on the “Number of production defects” option. You will be required to upload a .CSV file containing the data of the relevant production defects:

• The file should contain relevant production defects to the app and the date range of the report.

• Every line in the file should represent a production defect.

• There should be a column in a UTC format that includes the date of the defect and is named “Date”.

Once the file is uploaded successfully, the data will appear on the charts. Please note:

• The data will be added to both modified and overall coverage charts.

• The data is correlated to the coverage using the defect’s date - the production defects bar will be displayed for the build / date range which is the closest to the defect’s date (the last build/date range before the defect was created).

Automatic Insights

We provide automatic insights in order to help you identify quality risks. Our first insight alerts when large modification took place, but was not covered in testing properly. When these incidents are captured, an “INSIGHTS” button will appear near the chart and will include an indication of the number of found insights. When hovering over the chart, a matching indication will be displayed for the relevant interval / build. Clicking on the “INSIGHTS” button, all insights details will appear above the chart, as can be seen in the image below.

Working with the report

Editing The Report Configuration

At every point you can change the report configuration using the left filter panel.

Duplicating A Report

To create a copy of the existing report, go to the save menu and click on the “Save as new report” option.

Deleting A Report

From the existing report, click on the delete button.

Sharing The Report

After generating a report you can use the share button to copy the report URL to you clipboard or to publish the report as a public report that will be available to all users.

Using the copied URL, other users will be able to view and work with a draft report that includes the same configuration as the report you were sharing.

When publishing a report as a public report, all users will be able to view that report from their reports list view. It’s a great way to share the report with others in your organization.

Got questions? Visit our Quality Analytics FAQ page.