Test Impact Analysis with Selection On

Owned by Kobi Malka (Unlicensed)
Dec 08, 2019
3 min read

How to use and how it affects your Dashboard and TGA?

1. TIA and “Selection On”

When turning the “Selection On” functionality, SeaLights agent will send your Test Runner a list of tests to be skipped from the coming test cycle.

Skipped tests list:

  1. Impacted Tests

  2. Failed Tests

  3. New Tests

  4. Previously Recommended and not Executed

2. How to use Coverage metric while using Selective Test Execution (TIA Selection ON Guidelines)

  1. Use TGA (Test Gap Analytics) to track your overall Coverage.

    1. TGA has an aggregation engine that calculates your coverage across all tests and all builds.

  2. Do NOT Use Dashboard to track total coverage 

    1. Overall Coverage will be less/equal than your previous numbers
      (Due to the fact that only a subset of your tests will be running). 

    2. Coverage % fluctuations may occur. 
      (due to periodic full runs)

    3. Use the Dashboard to track your Quality Risks.

  3. Do NOT use Quality Gates and Notifications for “Total Coverage”

    1. Due to Selective test runs, total coverage per each build is expected to drop.

Important! Quality Gates and Notification Should be applied to Modified Code / Failed tests thresholds

3. Best Practice - Step by Step

  1. Review the recommendations for supported frameworks

     

  2. Selection on - Run a subset of tests (Impacted, Failed, etc…)

A typical “Selection On” view - includes all the apps and stages which have activated the smart test execution engine.

TIA Configuration can be easily performed via Settings menu (Settings → TIA Settings)

Step by Step Configuration:

  1. Set default full test Policy

2. Set App run Policy 

  1. Select app

  2. Select Branch

  3. Select the test stage

  4. Full Run 

  5. No code changes

3. Quality Gates - Should be applied only on Modified Code / Failed tests criteria

4. Use QTI/TGA to get the overall coverage data