Continuous Testing and Bitrise

GitHub Release Build Status License

Overview

With the synthetics-test-automation-bitrise-step-run-tests step, you can run Synthetic tests during your Bitrise CI, ensuring that all your teams using Bitrise can benefit from Synthetic tests at every stage of the software lifecycle.

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

Setup

This step is not available on the official Bitrise Step Library. To get started:

  1. Add the following git URL to your workflow. See the official Bitrise documentation on how to do that though the Bitrise app. You can also configure it locally by referencing the git URL in your bitrise.yml file.
- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
  1. Add your API and application keys to your secrets in Bitrise.
  2. Configure your step inputs. You can also configure them in your bitrise.yml file. The only required inputs are the two secrets you configured earlier. For a comprehensive list of inputs, see the Inputs section.

How to use this step locally

You can run this step directly using the Bitrise CLI.

To run this step locally:

  1. Open your terminal or command line.
  2. git clone the Bitrise repository.
  3. cd into the directory of the step (the one you just git cloned).
  4. Create a .bitrise.secrets.yml file in the same directory of bitrise.yml. The .bitrise.secrets.yml file is a Git-ignored file, so you can store your secrets in it.
  5. Check the bitrise.yml file for any secret you should set in .bitrise.secrets.yml.
  6. Once you have the required secret parameters in your .bitrise.secrets.yml file, run this step with the Bitrise CLI: bitrise run test.

An example .bitrise.secrets.yml file:

envs:
- A_SECRET_PARAM_ONE: the value for secret one
- A_SECRET_PARAM_TWO: the value for secret two

Simple usage

Example using public IDs

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - public_ids: |
      abc-d3f-ghi
      jkl-mn0-pqr

Example task using existing synthetics.json files

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Complex usage

Example task using the testSearchQuery

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - test_search_query: 'tag:e2e-tests'

Example task using the testSearchQuery and variable overrides

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - test_search_query: 'tag:e2e-tests'
   - variables: |
      START_URL=https://staging.website.com
      PASSWORD=$STAGING_PASSWORD

Example task using a global configuration override with configPath

This task overrides the path to the global global.config.json file.

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - config_path: './global.config.json'

Example including all possible configurations

For reference, this is an example of a complete configuration:

- git::https://github.com/DataDog/synthetics-test-automation-bitrise-step-run-tests.git@v2.2.0:
   inputs:
   - api_key: <DATADOG_API_KEY>
   - app_key: <DATADOG_APP_KEY>
   - batch_timeout: 4200000
   - config_path: './global.config.json'
   - device_ids: 'apple iphone se (2022),15.4.1, apple iphone 14 pro,16.1'
   - fail_on_critical_errors: true
   - fail_on_missing_tests: true
   - fail_on_timeout: true
   - files: 'e2e-tests/*.synthetics.json'
   - junit_report: 'e2e-test-junit'
   - locations: 'aws:us-west-1'
   - mobile_application_version: '01234567-8888-9999-abcd-efffffffffff'
   - mobile_application_version_file_path: 'path/to/application.apk'
   - public_ids: 'abc-d3f-ghi,jkl-mn0-pqr'
   - selective_rerun: true
   - site: 'datadoghq.com'
   - subdomain: 'myorg'
   - test_search_query: 'tag:e2e-tests'
   - tunnel: true
   - variables: |
      START_URL=https://staging.website.com
      PASSWORD=$STAGING_PASSWORD

Inputs

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

NameDescription
api_key(Required) Your Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
app_key(Required) Your Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
batch_timeoutThe duration in milliseconds after which the CI batch fails as timed out. This does not affect the outcome of a test run that already started.
Default: 1800000 (30 minutes)
config_pathThe path to the global configuration file that configures datadog-ci.
Default: datadog-ci.json
device_idsOverride the list of devices on which to run the Synthetic tests.
Default: none
fail_on_critical_errorsFail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues.
Default: false
fail_on_missing_testsFail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing.
Default: false
fail_on_timeoutFail the CI job if the CI batch fails as timed out.
Default: true
filesGlob patterns to detect Synthetic test configuration files, separated by new lines.
Default: {,!(node_modules)/**/}*.synthetics.json
junit_reportThe filename for a JUnit report if you want to generate one.
Default: none
locationsOverride the list of locations to run the test from. The possible values are listed in this API response.
Default: none
mobile_application_version_file_pathOverride the mobile application version for Synthetic mobile application tests with a local or recently built application. You may use $BITRISE_IPA_PATH or $BITRISE_APK_PATH from your previous build steps.
Default: none
mobile_application_versionOverride the mobile application version for Synthetic mobile application tests. The version must be uploaded and available within Datadog. You can use the Bitrise step to upload an application and use its DATADOG_UPLOADED_APPLICATION_VERSION_ID output here.
Default: none
public_idsPublic IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files.
Default: none
selective_rerunWhether to only rerun failed tests. If a test has already passed for a given commit, it will not be rerun in subsequent CI batches. By default, your organization’s default setting is used. Set it to false to force full runs when your configuration enables it by default.
Default: none
siteYour Datadog site. The possible values are listed in this table.
Default: datadoghq.com
subdomainThe custom subdomain to access your Datadog organization. If the URL used to access Datadog is myorg.datadoghq.com, the custom subdomain is myorg.
Default: app
test_search_queryUse a search query to select which Synthetic tests to run. Use the Synthetic Tests list page’s search bar to craft your query, then copy and paste it.
Default: none
tunnelUse the Continuous Testing tunnel to launch tests against internal environments.
Default: false
variablesOverride existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: START_URL=https://example.org,MY_VARIABLE=My title.
Default: none

Further reading

Additional helpful documentation, links, and articles:

PREVIEWING: guacbot/translation-pipeline