Continuous Testing and Datadog CI Azure DevOps Extension

Docs > Continuous Testing > Continuous Testing and CI/CD > Continuous Testing and Datadog CI Azure DevOps Extension

Overview

With the Datadog Continuous Testing Azure DevOps Extension, you can run Synthetic tests within your Azure Pipeline configuration and ensure all your teams using Azure DevOps can benefit from Synthetic tests at every stage of the software lifecycle. You can run SyntheticsRunTests as a task.

Authentication

Service Connection

To connect to your Datadog site, Datadog recommends setting up a custom service connection when configuring the Synthetics Run Test task.

You need to provide the following inputs:

Datadog site: Which Datadog site to connect and send data to.
Custom subdomain (default: app): The name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is myorg.datadoghq.com, this value needs to be set to myorg.
API Key: Your Datadog API key. This key is created by your Datadog organization.
Application key: Your Datadog application key. This key is created by your Datadog organization.

API and Application keys

API Key: Your Datadog API key. This key is created by your Datadog organization and is accessed as an environment variable.
Application key: Your Datadog application key. This key is created by your Datadog organization and is accessed as an environment variable.
Datadog site: The Datadog site to connect and send data to.
Custom subdomain (optional): The name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is myorg.datadoghq.com, this value needs to be set to myorg.

Setup

To connect to your Datadog account, create a Datadog CI service connection in your Azure pipelines project. Once created, all you need is the name of the service connection in the tasks.

Install the Datadog Continuous Testing extension from the Visual Studio Marketplace in your Azure Organization.
Add your Datadog API and application keys in the Datadog CI service connection, or as secrets to your Azure Pipelines project.
In your Azure DevOps pipeline, use the SyntheticsRunTests task.

Your task can be simple or complex.

Simple usage

Note: We recently changed the major version of the task from SyntheticsRunTests@0 to SyntheticsRunTests@1.
This is NOT a breaking change, but an alignment between the task version and the extension version.

Example task using public IDs

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    publicIds: |
      abc-d3f-ghi
      jkl-mn0-pqr

Example task using existing `synthetics.json` files

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Example task using pipeline secrets for authentication

- task: SyntheticsRunTests@1
  inputs:
    authenticationType: 'apiAppKeys'
    apiKey: '$(DatadogApiKey)'
    appKey: '$(DatadogAppKey)'
    datadogSite: '$(DatadogSite)'
    subdomain: 'myorg'

Complex usage

Note: We recently changed the major version of the task from SyntheticsRunTests@0 to SyntheticsRunTests@1.
This is NOT a breaking change, but an alignment between the task version and the extension version.

Example task using the `testSearchQuery`

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'

Example task using the `testSearchQuery` and variable overrides

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'
    variables: |
      START_URL=https://staging.website.com
      PASSWORD=$(StagingPassword)

Example task using a global configuration file with `configPath`

By default, the path to the global configuration file is datadog-ci.json. You can override this path with the config_path input.

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    configPath: './global.config.json'
    connectedService: 'my-datadog-ci-connected-service'

For an example of a global configuration file, see this global.config.json file.

Inputs

Name	Description
`apiKey`	Your Datadog API key. This key is created by your Datadog organization and should be stored as a secret. _{Required when authenticationType == apiAppKeys}
`appKey`	Your Datadog application key. This key is created by your Datadog organization and should be stored as a secret. _{Required when authenticationType == apiAppKeys}
`authenticationType`	(Required) How to store and retrieve credentials. _{Must be either apiAppKeys or connectedService}
`batchTimeout`	The duration (in milliseconds) after which the batch fails as timed out. _{Default: 1800000 (30 minutes)}
`connectedService`	The name of the Datadog CI service connection. _{Required when authenticationType == connectedService}
`configPath`	The global JSON configuration used when launching tests. For more information, see the example configuration. _{Default: datadog-ci.json}
`datadogSite`	The Datadog site. _{Default: datadoghq.com}
`failOnCriticalErrors`	Fail the CI job if no tests were triggered, or results could not be fetched from Datadog. _{Default: false}
`failOnMissingTests`	Fail the CI job if at least one specified test with a public ID (using `publicIds` or listed in a test file) is missing in a run (for example, if it has been deleted programmatically or on the Datadog site). _{Default: false}
`failOnTimeout`	Fail the CI job if at least one test exceeds the default test timeout. _{Default: true}
`files`	Glob pattern to detect Synthetic tests’ config files. _{Default: {,!(node_modules)/*/}.synthetics.json}
`jUnitReport`	The filename for a JUnit report if you want to generate one. _{Default: none}
`publicIds`	Public IDs of Synthetic tests to run, separated by newlines or commas. If no value is provided, tests are discovered in `*.synthetics.json` files. _{Default: none}
`subdomain`	The name of the custom subdomain set to access your Datadog application when using the `apiAppKeys` authentication type. If the URL used to access Datadog is `myorg.datadoghq.com`, this value needs to be set to `myorg`. _{Default: app}
`testSearchQuery`	Trigger tests corresponding to a search query. This can be useful if you are tagging your test configurations. For more information, see rules and best practices for naming tags. _{Default: none}
`variables`	Key-value pairs for injecting variables into tests, separated by newlines or commas. For example: `START_URL=https://example.org,MY_VARIABLE=My title`. _{Default: none}

Outputs

Name	Description
`batchUrl`	The URL of the batch.
`criticalErrorsCount`	The number of critical errors.
`failedCount`	The number of failed results.
`failedNonBlockingCount`	The number of failed non-blocking results.
`passedCount`	The number of passed results.
`previouslyPassedCount`	The number of previously passed results.
`testsNotFoundCount`	The number of not found tests.
`testsSkippedCount`	The number of skipped tests.
`timedOutCount`	The number of timed out results.
`rawResults`	The list of results, as a raw JSON string.