Group Coverage Trend Report
This feature is available only to customers who have provided consent to the usage of SnowFlake database. To learn more please contact Customer Success. |
|---|
The Group Coverage Trend Report is the second report type available under Quality Analytics. It displays multiple apps code coverage over time.
Generating A Report
From the Quality Analytics main page / reports list, click on the “Create new report” button. Generating a report is as simple as selecting the apps and branches you are interested in. Each app has its own color that is used in the charts.
You have different options to view the data:
Date Range
The popular options available are: Last month, Last 2 months, Last 3 months, Last 6 months, Last year.
If none of the above is what you need, you can always select a custom date range. We recommend a minimum of 4 weeks.
Please note that the start date for the data shown in the charts depends on the selected interval (see section below) and is currently limited to the following Monday or the 1st of the month.
Interval
You can select in which intervals you would like to aggregate the data. Options are: 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month. When selecting 1 month, the data shown in the chart will start at the 1st of the relevant month. For the rest of the interval options, the data shown in the chart will start at the following Monday.
Understanding The Data
The report includes two charts: Modified Coverage and Overall Coverage.
Both charts display an aggregated coverage on top of the specific apps coverage and include:
Y-axis: Coverage % (0-100%)
X-axis: Time, according to the filter selection and intervals
Intervals
When selecting any of the interval options (1 week, 2 weeks, 3 weeks, 4 weeks, or 1 month) any point in the X-axis and in the chart line represents an interval. Interval X takes all code changes from the last build in interval X-1, and takes all coverage from all related test stages that were reported in all builds within interval X. (e.g. Interval of 1 month: coverage in July means all code changes since last build on June and their coverage from all test stages executions reported on builds within July).
Modified Coverage Chart
Modified coverage is the coverage of your code changes. Why is it important to track it? Your quality risks are code changes you haven’t tested. The higher the coverage, the lower the chances for escaping defects. You want to see high coverage overtime, to reduce risks. High modified coverage overtime is an indication for a good quality culture. It means that any new or modified code is tested.
Overall Coverage Chart
Overall coverage is the coverage of your entire code. You should track the overall coverage to make sure that overtime your trend is positive, mainly by covering new and modified code. You might see drops in cases you had major code changes with no coverage.
Advanced Report Features
Drill-Down To A Specific App Coverage Trend Report
When reviewing the data included in a group trend report, you might need to drill down to a certain app level in order to examine which test stages contributed to its coverage. In order to quickly view the wanted report click on the “Open report” button from the right legend panel.
Number Of Methods
When examining your group coverage trend, seeing the data in the right context is important. To that end we have added an option to visualize the scope of the change in the modified chart and the scope of the apps that are examined in the overall chart. To add the methods bar to your charts, click on the “Chart elements” and toggle on the “Number of methods” option.
Number Of Production Defects
One of the most common KPIs around quality is the number of production defects that were opened against a build or a version. SeaLights now allows you to visualize the number of production defects and compare it to the coverage trend. To add the number of production defects as added bars in your charts, click on the “Chart elements” and toggle on the “Number of production defects” option. You will be required to upload a .CSV file containing the data of the relevant production defects:
The file should contain relevant production defects to the apps and the date range of the report.
Every line in the file should represent a production defect.
There should be a column in a UTC format that includes the date of the defect and is named “Date”.
Once the file is uploaded successfully, the data will appear on the charts. Please note:
The data will be added to both modified and overall coverage charts, as an aggregated bar (no breakdown according to the apps included in the report).
The data is correlated to the coverage using the defect’s date - the production defects bar will be displayed for the date range which is the closest to the defect’s date (the last date range before the defect was created).
Automatic Insights
We provide automatic insights in order to help you identify quality risks. Our first insight alerts when large modification took place in a certain app, but was not covered in testing properly. When these incidents are captured, an “INSIGHTS” button will appear near the chart and will include an indication of the number of found insights. When hovering over the chart, a matching indication will be displayed for the relevant interval / build. Clicking on the “INSIGHTS” button, all insights details will appear above the chart, as can be seen in the image below.
Working with the report
Editing The Report Configuration / Removing An App From The Report
At every point you can change the report configuration using the left filter panel. For example, you can remove an app from the report by hovering over the app and clicking on the delete button.
Duplicating A Report
To create a copy of the existing report, go to the save menu and click on the “Save as new report” option.
Deleting A Report
From the existing report, click on the delete button.
Sharing The Report
After generating a report you can use the share button to copy the report URL to you clipboard or to publish the report as a public report that will be available to all users.
Using the copied URL, other users will be able to view and work with a draft report that includes the same configuration as the report you were sharing.
When publishing a report as a public report, all users will be able to view that report from their reports list view. It’s a great way to share the report with others in your organization.
Got questions? Visit our Quality Analytics FAQ page.