Intelligent Test Runner for Java

Docs > Intelligent Test Runner > Configure the Intelligent Test Runner > Intelligent Test Runner for Java

Compatibility

Intelligent Test Runner is supported in dd-java-agent >= 1.27.0.

The following test frameworks are supported:

JUnit >= 4.10 and >= 5.3
TestNG >= 6.4
Spock >= 2.0
Cucumber >= 5.4.0
Karate >= 1.0.0
Scalatest >= 3.0.8

Setup

Test Visibility

Prior to setting up Intelligent Test Runner, set up Test Visibility for Java. If you are reporting data through the Agent, use v6.40 and later or v7.40 and later.

Activate Intelligent Test Runner for the test service

You, or a user in your organization with the Intelligent Test Runner Activation (intelligent_test_runner_activation_write) permission, must activate the Intelligent Test Runner on the Test Service Settings page.

Intelligent test runner enabled in test service settings in the CI section of Datadog.

Run tests with the Intelligent Test Runner enabled

After completing setup, run your tests as you normally do:

./gradlew cleanTest test -Dorg.gradle.jvmargs=\
-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app

MAVEN_OPTS=-javaagent:$DD_TRACER_FOLDER/dd-java-agent.jar=\
dd.civisibility.enabled=true,\
dd.env=ci,\
dd.service=my-java-app \
mvn clean verify

Disabling skipping for specific tests

You can override the Intelligent Test Runner’s behavior and prevent specific tests from being skipped. These tests are referred to as unskippable tests.

Why make tests unskippable?

The Intelligent Test Runner uses code coverage data to determine whether or not tests should be skipped. In some cases, this data may not be sufficient to make this determination.

Examples include:

Tests that read data from text files
Tests that interact with APIs outside of the code being tested (such as remote REST APIs)
Designating tests as unskippable ensures that the Intelligent Test Runner runs them regardless of coverage data.

Compatibility

Unskippable tests are supported in the following versions and testing frameworks:

JUnit >= 4.10 and >= 5.3
TestNG >= 6.4
Spock >= 2.2
Cucumber >= 5.4.0
Scalatest >= 3.0.8

Marking tests as unskippable

Individual test case

Add a JUnit Tag with the value datadog_itr_unskippable to your test case to mark it as unskippable.

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Tags;
import org.junit.jupiter.api.Test;

public class MyTestSuite {

  @Test
  @Tags({@Tag("datadog_itr_unskippable")})
  public void myTest() {
    // ...
  }
}

Test suite

Add a JUnit Tag with the value datadog_itr_unskippable to your test suite to mark it as unskippable.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by ITR.

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Tags;
import org.junit.jupiter.api.Test;

@Tags({@Tag("datadog_itr_unskippable")})
public class MyTestSuite {

  @Test
  public void myTest() {
    // ...
  }
}

Individual test case

Add a JUnit Category with the value datadog_itr_unskippable to your test case to mark it as unskippable. You do not have to create the datadog_itr_unskippable for every test case or test suite, one category is enough for the entire project.

import org.junit.Test;
import org.junit.experimental.categories.Category;

public class MyTestSuite {

  @Category(datadog_itr_unskippable.class)
  @Test
  public void myTest() {
    // ...
  }

  public interface datadog_itr_unskippable {}
}

Test suite

Add a JUnit Tag with the value datadog_itr_unskippable to your test suite to mark it as unskippable. You do not have to create the datadog_itr_unskippable for every test case or test suite, one category is enough for the entire project.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by ITR.

import org.junit.Test;
import org.junit.experimental.categories.Category;

@Category(MyTestSuite.datadog_itr_unskippable.class)
public class MyTestSuite {

  @Test
  public void myTest() {
    // ...
  }

  public interface datadog_itr_unskippable {}
}

Individual test case

Add a group with the value datadog_itr_unskippable to your test case to mark it as unskippable.

import org.testng.annotations.Test;

public class MyTestSuite {

  @Test(groups = "datadog_itr_unskippable")
  public void myTest() {
    // ...
  }
}

Test suite

Add a group with the value datadog_itr_unskippable to your test suite to mark it as unskippable.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by ITR.

import org.testng.annotations.Test;

@Test(groups = "datadog_itr_unskippable")
public class MyTestSuite {

  @Test
  public void myTest() {
    // ...
  }
}

Individual test case

Add a spock.lang.Tag with the value datadog_itr_unskippable to your test case to mark it as unskippable.

import spock.lang.Specification
import spock.lang.Tag

class MyTestSuite extends Specification {

  @Tag("datadog_itr_unskippable")
  def myTest() {
    // ...
  }
}

Test suite

Add a spock.lang.Tag with the value datadog_itr_unskippable to your test suite to mark it as unskippable.

If a suite is marked as unskippable, none of the test cases from that suite can be skipped by ITR.

import spock.lang.Specification
import spock.lang.Tag

@Tag("datadog_itr_unskippable")
class MyTestSuite extends Specification {

  def myTest() {
    // ...
  }
}

Individual scenario

Add datadog_itr_unskippable tag to your gherkin scenario to mark it as unskippable.

Feature: My Feature

  @datadog_itr_unskippable
  Scenario: My Scenario
    # ...

Feature

Add datadog_itr_unskippable tag to your gherkin feature to mark it as unskippable.

If a feature is marked as unskippable, none of the scenarios from that feature can be skipped by ITR.

@datadog_itr_unskippable
Feature: My Feature

  Scenario: My Scenario
    # ...

Create a Tag with the value datadog_itr_unskippable and tag your test case with it:

import org.scalatest.Tag
import org.scalatest.flatspec.AnyFlatSpec

object ItrUnskippableTag extends Tag("datadog_itr_unskippable")

class MyTestSuite extends AnyFlatSpec {
  "myTest" should "assert something" taggedAs ItrUnskippableTag in {
    // ...
  }
}