Test Optimization

Test Optimization helps you reduce your test cycles and improve test effectiveness.

It is where you can get visibility on your actual test execution time, run a simulator to find where you can reduce execution time and track your savings statistics. 

In the coming year more capabilities will be added, such as TIA for pull request, test prioritization, code to test map, test insights and many more!

What is the TIA (Test Impact Analysis)?

The TIA is currently the main functionality under Test Optimization. It can help you speed up time to market!

By using risk based testing principles and skipping unnecessary tests, you can provide faster feedback for your developers and reduce the test cycle time dramatically (50% to 90%).

The TIA can identify and execute the smallest subset of mandatory tests, without compromising quality. It is good for both automated tests and manual tests.

Who can use the TIA & how?

The TIA can be used by any SeaLights customer, for all types of tests and for any app and programming language supported by SeaLights.

In order to use the TIA, you must meet the following conditions:

• Test events must be reported to SeaLights (either automatically or manually by using SeaLights Chrome Extension)

• Your Test Runner can get a list of tests to be skipped from the coming test cycle.
  Or, you can use SeaLights UI for manual testing recommendations.

There are two ways to automatically report tests and connect to your Test Runner:

• SeaLights Official Integrations
  Use one of the existing integrations with the popular testing frameworks (e.g. TestNG, Junit, Cucumber).

• TIA Public APIs
  Use the public APIs to report tests from any unsupported frameworks and get test recommendations on demand.
  https://sealights.atlassian.net/l/c/tc5enJGS

How it works?

Main Steps

1. you Integrate SeaLights with your CI.

2. you Install agents, then direct SeaLights to scan the latest build and the tests that are already running.

3. SeaLights Correlate each test with the methods that relate to it.

   Open

   

4. SeaLights Compare any new build with the build which the current test stage was last executed.

5. SeaLights Identify all tests that are impacted by modified content, and recommend tests.

6. you Execute only a subset of your tests, based on the recommended tests to skip.

7. you Execute "Full Run" test cycle on defined scheduled to ensure no tests are ever overlooked.

8. SeaLights Analyze and improve the code-to-test correlation by using machine-learning algorithms.

TIA Outcome

The TIA produces a list of test recommendations pertaining to these types of tests:

• Impacted tests — Tests that correspond to recent code changes.

• Failed Tests — Tests that did not pass in the previous run.

• Dependent Tests — Tests that have a dependency with other recommended test.

• New Tests — Tests that have never been run.

• Previously Recommended Tests — Tests that were previously recommended but not yet executed.

Understanding The Data

The page displays the actual test execution time per app, whether TIA is ON or not.
This view should help you easily detect long execution times and understand where the main problems are.

Open

The table is ordered by the total execution time, in descending order.

TIA ON Indication

Each app has indication whether the TIA is ON or not:

• No indication = TIA is OFF for all test stages

• “Full” indication = TIA is ON for all test stages.

• “Partial“ indication = TIA is ON only for some of the test stages.

Total Execution Time

Each app displays the 5 longest test stages in the selected time period.

You can change the selection of the test stages in the left-side filter, if you are interested in analyzing a specific test stage.

App Details

You can clearly see which test stages are currently with the TIA ON. The total time saved or potential time saved is displayed at the bottom.

Each line is clickable and opens a more detailed view for each test stage.

Turning The Simulator ON

You can use the TIA Simulator to understand the impact of the TIA on the test execution time.

Clicking the button switches all TIA pages to simulation mode and a blue bar joins the Test Execution bar, to make it easy to compare the time difference. The blue bar represent the potential total execution time of any app / test stage / build (depends on the view you are looking at).

This information should help you decide where it makes sense to turn the TIA ON and save time.

Turning The TIA ON

The default configuration is that the TIA is OFF for all apps. You can turn it ON anytime on an app level or for specific branch. It is done per test stage.

Where and how you turn the TIA ON (or OFF if needed)?

When you select app and then test stage on Test Optimization page, you go to a page where you can see the Yes/No button.

The button if off by default:

TIA ON for Automated Tests

When TIA is ON, SeaLights agent (or the APIs) will send your Test Runner a list of tests to be skipped from the coming test cycle.

Skipped tests are all the tests that do not exist in any of the following lists:

• Impacted Tests

• Failed Tests

• New Tests

• Previously Recommended and not Executed

• Dependent Tests

TIA ON for Manual Tests

TIA is now available for manual tests reported by the Chrome extension or APIs.

There are two options to consume the TIA recommendations:

• On SeaLights UI, go to the relevant app and test stage page and select the build you are about to test.
  The recommendations list will be available automatically, if the TIA is ON.

• Use SeaLights API to get the list of tests to skip.

Full Run Configuration

TIA builds the code-to-test connection by learning which test hit which area in the code. The more hits the TIA captures per test, the more accurate the connection map. Therefore, it is recommended to have a full run with all the tests every one in a while, to increase the accuracy of the TIA.

How should you know what the frequency of a full run should be?

• For test stage you run a few times a day, it makes sense to set the full run policy to once a day.

• For test stage you run a few times a week, it makes sense to set the full run policy to once a week.

• You can select the option to run it every X times you run the test stage (X is customisable).

Notice! After working with the TIA ON for a while, if you identify the savings are not as you expected, consider changing the full run policy to be less frequent.

When can you set the full run policy?

A Settings link is available next to the TIA ON button, once you turn on the TIA ON button.