Test Impact Analysis
- John Vanderzyden (Unlicensed)
- Dime Vakanski (Unlicensed)
Overview of TIA
SeaLights Test Impact Analysis (TIA) is a Smart Test Execution Engine that cuts the testing cycle time by 50% to 90%. It can be tedious and time-consuming to run a full set of unit, functional, and regression tests each time a developer commits new code to the repository. SeaLights TIA eliminates much of this repetitive effort, since it can identify and execute the smallest subset of mandatory tests—without compromising quality.
Test Impact analysis provides support for
All types of tests
All of your apps and programming languages
Support for all types of tests
Through intelligent automation, SeaLights Test Impact Analysis will optimize and expedite the execution of almost any test type. Beside unit tests, you can apply TIA to your Continuous Integration (CI) implementation to automate functional, component, regression, and end-to-end tests. SeaLights TIA also accommodates manual testing and UI testing—including those using playback functionality (such as Selenium).
Support for all your apps and languages
Besides integrating with your CI environment, SeaLights TIA supports various application architectures and programming languages. From monolithic applications to distributed systems, from HTTP methods to microservices, SeaLights TIA can accommodate most any architecture. There’s no need to adapt your development practices to exploit TIA, since it supports Java, Node.js, JavaScript, and .NET/C#.
The SeaLights Test Impact Analysis Process
TIA Outcome
SeaLights Test Impact Analysis produces a list of test recommendations pertaining to these types of tests:
Impacted tests — Tests that correspond to recent code changes.
Failed Tests — Tests that did not pass in the previous run.
Pinned/Flagged Test — Tests that are marked as important to run in every test cycle.
New Tests — Tests that have never been run.
How does it TIA work?
These are the general steps in Test Impact Analysis:
You: Integrate SeaLights with your CI.
You: Install agents, then direct SeaLights to scan the latest build and the tests that are already running.
SeaLights: Correlate each test with the methods that relate to it.
SeaLights: Compare any new build content with the content with which the current test stage was last executed.
SeaLights: Identify any tests that are impacted by modified content and recommend tests.
You: Direct SeaLights to execute only the recommended tests, excluding those that are extraneous.
You: Execute "Full Run" tests on defined scheduled (for example, the first of day/week/month, or N builds) to ensure no tests are ever overlooked.
SeaLights: Analyze and improve using machine-learning algorithms.
Product Walkthrough
Follow these steps to configure TIA:
Click TIA on the top right end of the screen.
The default configuration is that automatic test selection is "Off" (The TIA analysis will provide a list of test recommendations without executing the recommended test list).
Choose the application name and the relevant branch.
Choose the date range of the TIA analysis.
Select the test stage to be analyzed.
Understanding the Data
The high-level report includes 4 main levels of information:
Summary view — Iists all of your applications, sorted by the total time saved (for each app and each test stage).
Application details — lists the app name, branch name, analysis period, and the relevant test stage.
Summary — includes aggregate information:
Average run-time and Average # of tests (with or without tests selection)
Average time saved and Average # of tests reduced (with or without tests selection)
Estimated total time saved for the analysis period
Test Execution details — includes per-build information:
Build number and build date
Recommended tests that should be run
Estimate of the run time for each test
Test Recommendations Report
This report presents a list of the impacted tests for each build.
NOTE: This information is also available through the SeaLights API.