Overview

Test Impact Analysis allows you to skip irrelevant tests unaffected by a code change.

With Test Optimization, development teams can configure Test Impact Analysis for their test services, set branches to exclude (such as the default branch), and define files to be tracked (which triggers full runs of all tests when any tracked file changes).

Configure and enable Test Impact Analysis for your test services to reduce unnecessary testing time, enhance CI test efficiency, and reduce costs, while maintaining the reliability and performance across your CI environments.

Test Impact Analysis uses code coverage data to determine whether or not tests should be skipped. For more information, see How Test Impact Analysis Works in Datadog.

Set up Test Impact Analysis

To set up Test Impact Analysis, see the following documentation for your programming language:

Enable Test Impact Analysis

To enable Test Impact Analysis:

1. Navigate to Software Delivery > Test Optimization > Settings.
2. On the Test Services tab, click Configure in the Test Impact Analysis column for a service.

You must have the Test Impact Analysis Activation Write permission. For more information, see the Datadog Role Permissions documentation.

Disabling Test Impact Analysis on critical branches (such as your default branch) ensures comprehensive test coverage, whereas enabling it to run on feature or development branches helps maximize testing efficiency.

Configure Test Impact Analysis

You can configure Test Impact Analysis to prevent specific tests from being skipped. These tests are known as unskippable tests, and are run regardless of code coverage data.

To configure Test Impact Analysis:

1. For the test you want to enable it on, click Configure.
2. Click the Status toggle to enable Test Impact Analysis.
3. Specify any branches to exclude (typically the default branch of a repository). Test Impact Analysis does not skip tests for these branches.
4. Specify file directories and files to track (for example, documentation/content/** or domains/shopist/apps/api/BUILD.bazel). Test Impact Analysis runs all CI tests when any of these tracked files change.
5. Click Save Settings.

Once you’ve configured Test Impact Analysis on a test service, execute a test suite run on your default branch. This establishes a baseline for Test Impact Analysis to accurately skip irrelevant tests in future commits.

Use Test Impact Analysis data

Explore the data collected by enabling Test Impact Analysis, such as the time savings achieved by skipping tests, as well as your organization’s usage of Test Impact Analysis, to improve your CI efficiency.

You can create dashboards to visualize your testing metrics, or use an out-of-the-box dashboard containing widgets populated with data collected by Test Impact Analysis to help you identify areas of improvement with usage patterns and trends.

Examine results in the Test Optimization Explorer

The Test Optimization Explorer allows you to create visualizations and filter test spans using the data collected from Test Optimization and Test Impact Analysis. When Test Impact Analysis is active, it displays the amount of time saved for each test session or commit. The duration bars turn purple to indicate active test skipping.

Navigate to Software Delivery > Test Optimization > Test Runs and select Session to start filtering your test session span results.

Navigate to Software Delivery > Test Optimization > Test Runs and select Module to start filtering your test module span results.

Navigate to Software Delivery > Test Optimization > Test Runs and select Suite to start filtering your test suite span results.

Navigate to Software Delivery > Test Optimization > Test Runs and select Test to start filtering your test span results.

Use the following out-of-the-box Test Impact Analysis facets to customize the search query:

Code Coverage Enabled

Indicates whether code coverage tracking was active during the test session.

Skipped by ITR

Number of tests that were skipped during the session by Test Impact Analysis.

Test Skipping Enabled

Indicates if Test Impact Analysis was enabled for the test session.

Test Skipping Type

The method or criteria used by Test Impact Analysis to determine which tests to skip.

Tests Skipped

The total count of tests that were not executed during the test session, which may include tests that were configured to skip, or were set as manual exclusions.

Time Saved

The length of time saved for the session by Test Impact Analysis usage.

For example, to filter test session runs that have Test Skipping Enabled, you can use @test.itr.tests_skipping.enabled:true in the search query.

Then, click on a test session run and see the amount of time saved by Test Impact Analysis in the Test Session Details section on the test session side panel.

Further Reading

Additional helpful documentation, links, and articles:

Streamline your CI testing with Datadog Intelligent Test RunnerBLOG Learn about Test Impact AnalysisDOCUMENTATION Learn about Code CoverageDOCUMENTATION