This page is about configuring Continuous Testing tests for your Continuous Integration (CI) and Continuous Delivery (CD) pipelines. If you want to bring your CI/CD metrics and data into Datadog dashboards, see the CI Visibility section.
Use the @datadog-ci NPM package to run Continuous Testing tests directly within your CI/CD pipeline. You can automatically halt a build, block a deployment, and roll back a deployment when a Synthetic test detects a regression.
Using a global configuration file (Global Config) is one of the ways to configure datadog-ci. To do so, create a JSON configuration file on your system. Specify the path to the file using the --config flag or configure it through the DATADOG_SYNTHETICS_CONFIG_PATH environment variable when launching your tests or uploading a new application. If you don’t specify a file path, Datadog looks for a file with the default filename of datadog-ci.json.
In addition to the global configuration file, you can configure all properties using environment variables. If a property is defined in both the global configuration file and as an environment variable, the environment variable takes precedence.
The CLI provides another way to set options and configure the behavior of datadog-ci. These options will override the global configuration file and environment variables.
You can also use the datadog-ci package as a library in your Node.js application to trigger tests. To do so, import the package from the Synthetics run-tests command and call the executeWithDetails() function.
You can configure a proxy to be used for outgoing connections to Datadog. To do this, use the proxy key of the global configuration file or the HTTPS_PROXY environment variable.
Note: This is the only exception where the global configuration file takes precedence over the environment variable. There is no option to set this through the CLI.
As the proxy-agent library is used to configure the proxy, the supported protocols include http, https, socks, socks4, socks4a, socks5, socks5h, pac+data, pac+file, pac+ftp, pac+http, and pac+https. The proxy key of the global configuration file is passed to a new proxy-agent instance, which means the same configuration for the library is supported.
To use a proxy, you need to first set the CA certificate so datadog-ci trusts your proxy. You can do this by setting the NODE_EXTRA_CA_CERTS environment variable to the path of your CA certificate. Otherwise, you might get a unable to verify the first certificate error.
The format used for the HTTPS_PROXY environment variable is <protocol>://<username>:<password>@<host>:<port>, as described by the proxy-from-env library that proxy-agent library uses for parsing env variables.
The HTTPS_PROXY variable is used instead of the HTTP_PROXY one, because the Datadog API uses the HTTPS protocol.
Example:
exportHTTPS_PROXY=http://login:pwd@127.0.0.1:3128
If you want to confirm that a proxy is being used, you can set the DEBUG environment variable to proxy-agent like this:
You can decide to have the CLI auto-discover all your **/*.synthetics.json Synthetic tests (see test files) or specify the tests you want to run using the -p,--public-id flag.
Note: If you are launching your tests with a custom filename for the global configuration file, append the command associated to your datadog-ci-synthetics script with --config <CUSTOM_PATH_TO_GLOBAL_CONFIG_FILE>.
Run tests by executing the CLI through Yarn:
The run-tests sub-command accepts the --public-id (or shorthand -p) argument to trigger only the specified test. It can be set multiple times to run multiple tests:
It is also possible to trigger tests corresponding to a search query by using the --search (or shorthand -s) argument. With this option, the overrides defined in your global configuration file apply to all tests discovered with the search query.
Note: If you are launching your tests with a custom filename for the global configuration file, append the command associated to your datadog-ci-synthetics script with --config <CUSTOM_PATH_TO_GLOBAL_CONFIG_FILE>.
The duration (integer in milliseconds) after which datadog-ci stops waiting for test results. The default is 30 minutes. At the CI level, test results completed after this duration are considered failed.
A boolean flag that fails the CI job if at least one specified test with a public ID (a --public-id CLI argument or listed in a test file) is missing in a run (for example, if it has been deleted programmatically or on the Datadog site). The default is set to false.
The proxy to be used for outgoing connections to Datadog. host and port keys are mandatory arguments, the protocol key defaults to http. Supported values for the protocol key are http, https, socks, socks4, socks4a, socks5, socks5h, pac+data, pac+file, pac+ftp, pac+http, and pac+https. The library used to configure the proxy is the proxy-agent library.
A boolean flag to only run the tests which failed in the previous test batches. By default, the organization default setting is used. Use the --no-selectiveRerun CLI flag or selectiveRerun: false to force a full run.
The name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is myorg.datadoghq.com, the subdomain value needs to be set to myorg.
Pass a query to select which Synthetic tests to run.
The syntax for this query is the same as that used in the Synthetics tests list page’s search bar.
You can craft the query in the UI, then copy and paste it in your command line inside single quotes.
Example query with a facet, a value tag, and a <KEY>:<VALUE> tag:
These options can also be set with environment variables starting with DATADOG_SYNTHETICS_OVERRIDE_... or with the --override CLI parameter following this pattern: --override option=value.
Use the provided string as a cookie header in an API or browser test (in addition or as a replacement).
If this is a string, it is used to replace the original cookies.
If this is an object, the format must be {append?: boolean, value: string}, and depending on the value of append, it is appended or replaces the original cookies.
Configuration options
Global/Test Config: "cookies": "name1=value1;name2=value2" (equivalent to "append": false) or "cookies": {"append": true, "value": "name1=value1;name2=value2"}
Use the provided string as a set-cookie header in a browser test only (in addition or as a replacement).
If this is a string, it is used to replace the original set-cookies.
If this is an object, the format must be {append?: boolean, value: string}, and depending on the value of append, it is appended or replaces the original set-cookies.
This object specifies the headers to be replaced in the test. It should have keys representing the names of the headers to be replaced, and values indicating the new header values.
Configuration options
Global/Test Config: "headers": {"NEW_HEADER_1": "NEW VALUE 1", "NEW_HEADER_2": "NEW VALUE 2"}
ENV variable: DATADOG_SYNTHETICS_OVERRIDE_HEADERS='{"NEW_HEADER_1":"NEW VALUE 1", "NEW_HEADER_2":"NEW VALUE 2"}' (note this must be a valid JSON)
An array of regex patterns to modify resource URLs in the test. This can be useful for dynamically changing resource URLs during test execution.
Each regex pattern should be in the format:
your_regex|your_substitution: The pipe-based syntax, to avoid any conflicts with / characters in URLs. For example, https://example.com(.*)|http://subdomain.example.com$1 to transform https://example.com/resource to http://subdomain.example.com/resource.
s/your_regex/your_substitution/modifiers: The slash syntax, which supports modifiers. For example, s/(https://www.)(.*)/$1staging-$2/ to transform https://www.example.com/resource into https://www.staging-example.com/resource.
The regex to modify the starting URL of the test (for browser and HTTP tests only), whether it was given by the original test or the configuration override startUrl.
If the URL contains variables, this regex applies after the interpolation of the variables.
There are two possible formats:
your_regex|your_substitution: The pipe-based syntax, to avoid any conflicts with / characters in URLs. For example, https://example.com(.*)|http://subdomain.example.com$1 to transform https://example.com/test to http://subdomain.example.com/test.
s/your_regex/your_substitution/modifiers: The slash syntax, which supports modifiers. For example, s/(https://www.)(.*)/$1extra-$2/ to transform https://www.example.com into https://www.extra-example.com.
This object defines the variables to be substituted in the test. It should include keys corresponding to the names of the variables to be replaced, and values representing the new values for these variables.
Configuration options
Global/Test Config: "variables": {"NEW_VARIABLE_1": "NEW VARIABLE 1", "NEW_VARIABLE_2": "NEW VARIABLE 2"}
ENV variable: DATADOG_SYNTHETICS_OVERRIDE_VARIABLES='{"NEW_VARIABLE_1":"NEW VARIABLE 1", "NEW_VARIABLE_2":"NEW VARIABLE 2"}' (note this must be a valid JSON)
To configure which URL your test starts on, provide a startUrl to your test object. Build your own starting URL with any part of your test’s original starting URL and include environment variables.
If the organization uses a custom sub-domain to access Datadog, this needs to be set in the DATADOG_SUBDOMAIN environment variable or in the global configuration file under the subdomain key in order to properly display the test results URL.
For example, if the URL used to access Datadog is myorg.datadoghq.com, set the environment variable to myorg:
You can use DATADOG_SYNTHETICS_OVERRIDE_LOCATIONS to override the locations where your tests run. Locations should be separated with a semicolon (;). The configuration in test files takes precedence over other overrides.
Test configuration files (Test Config) let you customize individual tests or set up multiple runs of the same test with different settings, beyond what you can do with other configuration methods.
You can find a list of all these options in the test overrides section.
These files take precedence over global configuration files, environment variables, and CLI parameters. The priority order including test configurations is as follows:
Global Config < Environment variables < CLI parameters < Test Config
To determine which tests to run, one or more of those options may be passed to datadog-ci:
If none of these options is passed, datadog-ci auto-discovers test configuration files with the {,!(node_modules)/**/}*.synthetics.json glob pattern (every file ending with .synthetics.json, except for those in the node_modules folder).
Note: The file search starts from the current working directory, so it may be slow if the command is run from a large directory, like a home folder. If file search command is too slow, consider:
Using the above options to specify the tests (this will disable the file search),
Or refining the glob pattern with the files option.
For example, by using * instead of ** or by adding a specific folder to the pattern.
The <TEST_PUBLIC_ID> can be either the identifier of the test found in the URL of a test details page (for example, for https://app.datadoghq.com/synthetics/details/abc-def-ghi, it would be abc-def-ghi) or the full URL to the details page (for example, directly https://app.datadoghq.com/synthetics/details/abc-def-ghi).
The proxy to be used for outgoing connections to Datadog. host and port keys are mandatory arguments, the protocol key defaults to http. Supported values for the protocol key are http, https, socks, socks4, socks4a, socks5, socks5h, pac+data, pac+file, pac+ftp, pac+http, and pac+https. The library used to configure the proxy is the proxy-agent library.
The default file name for the global configuration file is datadog-ci.json. If you use this name for your global configuration file, you may omit the --config flag.
You can combine variable overrides with Local and Staging Environments to run tests within your development environment. This connection ensures that all test requests sent through the CLI are automatically routed through the datadog-ci client.
This allows you to run tests with end-to-end encryption at every stage of your software development lifecycle, from pre-production environments to your production system.
[abc-def-ghi] Trigger test"Check on testing.website"[abc-def-ghi] Waiting results for"Check on testing.website"===REPORT===Took 11546ms
✓ [abc-def-ghi]| Check on testing.website
✓ location: Frankfurt (AWS) ⎋ total duration: 28.9 ms - result url: https://app.datadoghq.com/synthetics/details/abc-def-ghi?resultId=123456789123456789 ✓ GET - https://testing.website
You can also see the outcome of test executions directly in your CI as your tests are being executed. To identify what caused a test to fail, look at the execution logs and search for causes of the failed assertion.
yarn datadog-ci synthetics run-tests --config global-config.json
yarn run v1.22.4
$ /Users/demo.user/go/src/github.com/Datadog/tmp/test/testDemo/node_modules/.bin/datadog-ci synthetics run-tests --config global-config.json
Finding files matching /Users/demo.user/go/src/github.com/Datadog/tmp/test/testDemo/{,!(node_modules)/**/}*.synthetics.json
Got test files:
- user.synthetics.json
[2cj-h3c-39x] Trigger test"Test CI connection"[2cj-h3c-39x] Waiting results for"Test CI connection"===REPORT=== Took 2242ms
x [2cj-h3c-39x]| Test CI connection
* location: 30019 ⎋ total duration: 32.6 ms - result url: https://app.datadoghq.com/synthetics/details/2cj-h3c-39x?resultId=122140688175981634 x GET - https://www.datadoghq.com
[INCORRECT_ASSUMPTION] - [{"index":1,"operator":"is","property":"content-type","type":"header","target":"text/html","valid":false,"actual":"text/html";charset=utf-8"}]
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
