Continuous Testing and Datadog CI Azure DevOps Extension
Overview
With the SyntheticsRunTests task, you can run Synthetic tests within your Azure Pipeline configuration and ensure all your teams using Azure DevOps can benefit from Synthetic tests at every stage of the software lifecycle.
For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.
Authentication
Service Connection
To connect to your Datadog site, Datadog recommends setting up a custom service connection when configuring the SyntheticsRunTests task.
You need to provide the following inputs:
API and Application keys
Setup
To connect to your Datadog account, create a Datadog CI service connection in your Azure pipelines project. Once created, all you need is the name of the service connection in the tasks.
- Install the Datadog Continuous Testing extension from the Visual Studio Marketplace in your Azure Organization.
- Add your Datadog API and application keys in the Datadog CI service connection, or as secrets to your Azure Pipelines project.
- In your Azure DevOps pipeline, use the
SyntheticsRunTests task.
Your task can be simple or complex.
Simple usage
Example task using public IDs
- task: SyntheticsRunTests@1
displayName: Run Datadog Synthetic tests
inputs:
authenticationType: 'connectedService'
connectedService: 'my-datadog-ci-connected-service'
publicIds: |
abc-d3f-ghi
jkl-mn0-pqr
Example task using existing synthetics.json files
- task: SyntheticsRunTests@1
displayName: Run Datadog Synthetic tests
inputs:
authenticationType: 'connectedService'
connectedService: 'my-datadog-ci-connected-service'
files: 'e2e-tests/*.synthetics.json'
For an example test file, see this test.synthetics.json file.
Example task using pipeline secrets for authentication
- task: SyntheticsRunTests@1
inputs:
authenticationType: 'apiAppKeys'
apiKey: '$(DatadogApiKey)'
appKey: '$(DatadogAppKey)'
datadogSite: '$(DatadogSite)'
subdomain: 'myorg'
Complex usage
Example task using the testSearchQuery
- task: SyntheticsRunTests@1
displayName: Run Datadog Synthetic tests
inputs:
authenticationType: 'connectedService'
connectedService: 'my-datadog-ci-connected-service'
testSearchQuery: 'tag:e2e-tests'
Example task using the testSearchQuery and variable overrides
- task: SyntheticsRunTests@1
displayName: Run Datadog Synthetic tests
inputs:
authenticationType: 'connectedService'
connectedService: 'my-datadog-ci-connected-service'
testSearchQuery: 'tag:e2e-tests'
variables: |
START_URL=https://staging.website.com
PASSWORD=$(StagingPassword)
Example task using a global configuration file with configPath
By default, the path to the global configuration file is datadog-ci.json. You can override this path with the config_path input.
- task: SyntheticsRunTests@1
displayName: Run Datadog Synthetic tests
inputs:
authenticationType: 'connectedService'
configPath: './global.config.json'
connectedService: 'my-datadog-ci-connected-service'
For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.
| Name | Description |
|---|
apiKey | Your Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
Required when authenticationType == apiAppKeys |
appKey | Your Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
Required when authenticationType == apiAppKeys |
authenticationType | (Required) How to store and retrieve credentials.
Must be either apiAppKeys or connectedService |
batchTimeout | The duration in milliseconds after which the CI batch fails as timed out. This does not affect the outcome of a test run that already started.
Default: 1800000 (30 minutes) |
connectedService | The name of the Datadog CI service connection.
Required when authenticationType == connectedService |
configPath | The path to the global configuration file that configures datadog-ci.
Default: datadog-ci.json |
datadogSite | Your Datadog site. The possible values are listed in this table.
Default: datadoghq.com |
failOnCriticalErrors | Fail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues.
Default: false |
failOnMissingTests | Fail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing.
Default: false |
failOnTimeout | Fail the CI job if the CI batch fails as timed out.
Default: true |
files | Glob patterns to detect Synthetic test configuration files.
Default: {,!(node_modules)/**/}*.synthetics.json |
jUnitReport | The filename for a JUnit report if you want to generate one.
Default: none |
publicIds | Public IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files.
Default: none |
subdomain | The custom subdomain to access your Datadog organization when authenticationType == apiAppKeys. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.
Default: app |
testSearchQuery | Use a search query to select which Synthetic tests to run. Use the Synthetic Tests list page’s search bar to craft your query, then copy and paste it.
Default: none |
variables | Override existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: START_URL=https://example.org,MY_VARIABLE=My title.
Default: none |
Outputs
| Name | Description |
|---|
batchUrl | The URL of the batch. |
criticalErrorsCount | The number of critical errors. |
failedCount | The number of failed results. |
failedNonBlockingCount | The number of failed non-blocking results. |
passedCount | The number of passed results. |
previouslyPassedCount | The number of previously passed results. |
testsNotFoundCount | The number of not found tests. |
testsSkippedCount | The number of skipped tests. |
timedOutCount | The number of timed out results. |
rawResults | The list of results, as a raw JSON string. |
Further reading
Additional helpful documentation, links, and articles:
