Overview

Intelligent Test Runner allows you to streamline your organization’s test impact analysis by intelligently excluding irrelevant tests that are not affected by recent code changes.

With Test Visibility, development teams can configure Intelligent Test Runner for their test services, set branches to exclude (such as the default branch), and define files to be tracked (which triggers full runs of all tests when any tracked file changes).

Configure and enable Intelligent Test Runner for your test services to reduce unnecessary testing time, enhance CI test efficiency, and reduce costs, while maintaining the reliability and performance across your CI environments.

Intelligent Test Runner uses code coverage data to determine whether or not tests should be skipped. For more information, see How Intelligent Test Runner Works in Datadog.

Set up the Intelligent Test Runner

To set up the Intelligent Test Runner, see the following documentation for your programming language:

Enable the Intelligent Test Runner

To enable the Intelligent Test Runner:

1. Navigate to Software Delivery > Test Visibility > Settings.
2. On the Test Services tab, click Configure in the Intelligent Test Runner column for a service.

You must have the Intelligent Test Runner Activation Write permission. For more information, see the Datadog Role Permissions documentation.

Disabling the Intelligent Test Runner on critical branches (such as your default branch) ensures comprehensive test coverage, whereas enabling it to run on feature or development branches helps maximize testing efficiency.

Configure the Intelligent Test Runner

You can configure the Intelligent Test Runner to prevent specific tests from being skipped. These tests are known as unskippable tests, and are run regardless of code coverage data.

To configure the Intelligent Test Runner:

1. Click the Status toggle to enable Intelligent Test Runner.
2. Specify any branches to exclude (typically the default branch of a repository). Intelligent Test Runner does not skip tests for these branches.
3. Specify file directories and files to track (for example, documentation/content/** or domains/shopist/apps/api/BUILD.bazel). Intelligent Test Runner runs all CI tests when any of these tracked files change.
4. Click Save Settings.

Once you’ve configured Intelligent Test Runner on a test service, execute a test suite run on your default branch. This establishes a baseline for the Intelligent Test Runner to accurately skip irrelevant tests in future commits.

Use Intelligent Test Runner data

Explore the data collected by enabling Intelligent Test Runner, such as the time savings achieved by skipping tests, as well as your organization’s usage of Intelligent Test Runner, to improve your CI efficiency.

You can create dashboards to visualize your testing metrics, or use an out-of-the-box dashboard containing widgets populated with data collected by the Intelligent Test Runner to help you identify areas of improvement with usage patterns and trends.

Examine results in the Test Visibility Explorer

The Test Visibility Explorer allows you to create visualizations and filter test spans using the data collected from Test Visibility and the Intelligent Test Runner. When the Intelligent Test Runner is active, it displays the amount of time saved for each test session or commit. The duration bars turn purple to indicate active test skipping.

Navigate to Software Delivery > Test Visibility > Test Runs and select Session to start filtering your test session span results.

Navigate to Software Delivery > Test Visibility > Test Runs and select Module to start filtering your test module span results.

Navigate to Software Delivery > Test Visibility > Test Runs and select Suite to start filtering your test suite span results.

Navigate to Software Delivery > Test Visibility > Test Runs and select Test to start filtering your test span results.

Use the following out-of-the-box Intelligent Test Runner facets to customize the search query:

Code Coverage Enabled

Indicates whether code coverage tracking was active during the test session.

Skipped by ITR

Number of tests that were skipped during the session by the Intelligent Test Runner.

Test Skipping Enabled

Indicates if Intelligent Test Runner was enabled for the test session.

Test Skipping Type

The method or criteria used by the Intelligent Test Runner to determine which tests to skip.

Tests Skipped

The total count of tests that were not executed during the test session, which may include tests that were configured to skip, or were set as manual exclusions.

Time Saved

The length of time saved for the session by Intelligent Test Runner usage.

For example, to filter test session runs that have Test Skipping Enabled, you can use @test.itr.tests_skipping.enabled:true in the search query.

Then, click on a test session run and see the amount of time saved by the Intelligent Test Runner in the Test Session Details section on the test session side panel.

Further Reading

Additional helpful documentation, links, and articles:

Streamline your CI testing with Datadog Intelligent Test RunnerBLOG Learn about Intelligent Test RunnerDOCUMENTATION Learn about Code CoverageDOCUMENTATION