Overview of the TIA Process
- John Vanderzyden (Unlicensed)
The outcome of the TIA is a recommended test list which consists of 4 tests types:
Impacted tests -Tests related to recent code changes
Failed Tests -Tests that didn’t pass in the last run
Pinned/Flagged Test -Tests marked as important to run in every test cycle
New Tests -Tests that have never been run before
How does it work?
Integrate with your CI
Scan the latest build and the tests that are running
Correlate each test with the methods related to it
Compare new build content with content in which the current test stage was last executed
Identify tests impacted by changed content and recommend tests
Execute only the recommended tests, excluding those not “relevant”
Execute "Full Run" tests on defined scheduled (first of day/week/month or N builds) to ensure no tests are ever overlooked
Analyze and improve using machine-learning algorithms
Product Walkthrough
Click on TIA on the top right end of the screen
The default configuration is that automatic test selection is "Off" (The TIA analysis will provide a list of test recommendations without executing the recommended test list).
Pick the application name and the relevant branch
Choose the date range of the TIA analysis
Select the test stage to be analyzed
Understanding the Data
The high-level report includes 4 main levels of information
Summary view
Includes all of your applications sorted by the total time saved (Per app, Per Test Stage)
Application details
includes app name, branch name, analysis period, and the relevant test stage
Summary
aggregated data such as
Average run-time and Average # of tests (with or without tests selection)
Average time saved and Average # of tests reduced (with or without tests selection)
Estimated total time saved for the analysis period
Test Execution details
per build information
Build number & Build date
Tests Recommended
Estimated Run Time
Test Recommendations Report
This report presents a list of impacted tests per build (also available via SeaLights APIs)