Continuous Testing and CircleCI Orb
Overview
Run Datadog Synthetic tests in your CircleCI pipelines using the Datadog CircleCI orb.
For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.
Setup
To get started:
- Add your Datadog API and application keys as environment variables to your CircleCI project.
- Ensure the image running the orb is a Linux-x64-based image with
curl installed. - Customize your CircleCI workflow by adding a
synthetics-ci/run-tests step and specifying inputs as listed below.
Your workflow can be simple or complex.
Simple usage
Example orb usage using public IDs
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@4.2.0
jobs:
e2e-tests:
docker:
- image: cimg/base:stable
steps:
- synthetics-ci/run-tests:
public_ids: |
abc-d3f-ghi
jkl-mn0-pqr
workflows:
run-tests:
jobs:
- e2e-tests
Example orb usage using a global configuration override
This orb overrides the path to the pattern for test files.
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@4.2.0
jobs:
e2e-tests:
docker:
- image: cimg/base:stable
steps:
- synthetics-ci/run-tests:
files: e2e-tests/*.synthetics.json
workflows:
run-tests:
jobs:
- e2e-tests
For another example pipeline that triggers Synthetic tests, see the simple-example.yml file.
Complex usage
Example orb usage using the test_search_query
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@4.2.0
jobs:
e2e-tests:
docker:
- image: cimg/base:stable
steps:
- synthetics-ci/run-tests:
test_search_query: 'tag:e2e-tests'
workflows:
run-tests:
jobs:
- e2e-tests
version: 2.1
orbs:
synthetics-ci: datadog/synthetics-ci-orb@4.2.0
jobs:
e2e-tests:
docker:
- image: your-image
steps:
- checkout
- run:
name: Running server in background
command: npm start
background: true
- synthetics-ci/run-tests:
config_path: tests/tunnel-config.json
files: tests/*.synthetics.json
test_search_query: 'tag:e2e-tests'
tunnel: true
workflows:
test-server:
jobs:
- build-image
- integration-tests:
requires:
- build-image
For additional options such as customizing the batchTimeout for your CircleCI pipelines, see CI/CD Integrations Configuration. For another example pipeline that starts a local server and triggers Synthetic tests using the Continuous Testing tunnel, see the advanced-example.yml file.
For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.
| Name | Description |
|---|
api_key | Name of the environment variable containing your Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
Default: DATADOG_API_KEY |
app_key | Name of the environment variable containing your Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
Default: DATADOG_APP_KEY |
background | Whether or not this step should run in the background. See official CircleCI documentation.
Default: false |
batch_timeout | The duration in milliseconds after which the CI batch fails as timed out. This does not affect the outcome of a test run that already started.
Default: 1800000 (30 minutes) |
config_path | The path to the global configuration file that configures datadog-ci.
Default: datadog-ci.json |
fail_on_critical_errors | Fail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues.
Default: false |
fail_on_missing_tests | Fail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing.
Default: false |
fail_on_timeout | Fail the CI job if the CI batch fails as timed out.
Default: true |
files | Glob patterns to detect Synthetic test configuration files, separated by new lines.
Default: {,!(node_modules)/**/}*.synthetics.json |
junit_report | The filename for a JUnit report if you want to generate one.
Default: none |
locations | Override the list of locations to run the test from. The possible values are listed in this API response.
Default: none |
no_output_timeout | Elapsed time the command can run without output. The string is a decimal with unit suffix, such as 20m, 1.25h, 5s. See official CircleCI documentation.
Default: 35m |
public_ids | Public IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files.
Default: none |
site | Your Datadog site. The possible values are listed in this table.
Default: datadoghq.com |
subdomain | The custom subdomain to access your Datadog organization. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.
Default: app |
test_search_query | Use a search query to select which Synthetic tests to run. Use the Synthetic Tests list page’s search bar to craft your query, then copy and paste it.
Default: none |
tunnel | Use the Continuous Testing tunnel to launch tests against internal environments.
Default: false |
variables | Override existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: START_URL=https://example.org,MY_VARIABLE=My title.
Default: none |
Further reading
Additional helpful documentation, links, and articles:
