How to Set Up Deployment Data for DORA Metrics

DORA Metrics is not available in the selected site () at this time.

DORA Metrics is in public beta.

Overview

Deployment events are used to compute deployment frequency, change lead time, and change failure rate.

Selecting and configuring a deployment data source

APM Deployment Tracking can be configured as a data source for deployments in DORA Metrics.

To ensure your service deployments tracked by APM contribute to DORA Metrics, the following requirements must be met:

  • Your service has metadata defined in the Service Catalog.
  • Your service has unified service tagging enabled. Deployments are identified using the version tag.

For more information about ensuring service deployments that are tracked by APM contribute to change lead time, see Deployment Data Sources.

To send your own deployment events, use the DORA Metrics API or the datadog-ci dora deployment command.

The following attributes are required:

  • started_at: The time the deployment started.
  • finished_at: The time the deployment finished.
  • service: The service that was deployed. If the provided service is registered in the Service Catalog with metadata set up (see Adding Metadata), the team of the service is automatically retrieved and associated with all metrics.

The repository_url and commit_sha attributes are also required for calculating the Change Lead Time metric. Optionally, you can specify a team attribute to associate a deployment with a different team than is found automatically for the service. You can also specify the env attribute to filter your DORA metrics by environment on the DORA Metrics page.

API (cURL) Example

See the DORA Metrics API reference documentation for the full spec and additional code samples.

For the following example, replace <DD_SITE> in the URL with and ${DD_API_KEY} with your Datadog API Key:

  curl -X POST "https://api.<DD_SITE>/api/v2/dora/deployment" \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -d @- << EOF
  {
    "data": {
      "attributes": {
        "service": "shopist",
        "started_at": 1693491974000000000,
        "finished_at": 1693491984000000000,
        "git": {
          "commit_sha": "66adc9350f2cc9b250b69abddab733dd55e1a588",
          "repository_url": "https://github.com/organization/example-repository"
        },
        "env": "prod",
        "team": "backend"
      }
    }
  }
EOF

CLI Example

The datadog-ci CLI tool provides a shortcut to send deployment events within your Continuous Integration environment.

For the following example, set the DD_SITE environment variable to and set the DD_API_KEY environment variable to your Datadog API Key:

export DD_BETA_COMMANDS_ENABLED=1
export DD_SITE="<DD_SITE>"
export DD_API_KEY="<DD_API_KEY>"

export deploy_start=`date +%s`
./your-deploy-script.sh
datadog-ci dora deployment --service shopist --env prod \
    --started-at $deploy_start --finished-at `date +%s` \
    --git-repository-url "https://github.com/organization/example-repository" \
    --git-commit-sha 66adc9350f2cc9b250b69abddab733dd55e1a588

The deployment finish time is automatically set to now if --finished-at is not provided.

If the deployment CI job is running on the exact same Git revision that is being deployed, git-repository-url and git-commit-sha can be omitted and are automatically inferred from the CI context.

The --skip-git option can be provided to disable sending the repository URL and commit SHA. When this option is added, the Change Lead Time metric becomes unavailable.

Calculating deployment frequency

Deployment frequency is calculated based on the dora.deployments.count metric that is generated and increased with each deployment detected from your selected deployment data source. Frequency is calculated by dividing dora.deployments.count over a specific time frame.

Calculating change lead time

For a single Git commit, change lead time (CLT) is calculated as time from the creation of the commit to when the deployment including that commit was executed.

To calculate change lead time for a deployment, Datadog runs git log between the deployment commit SHA and the previous deployment commit SHA to find all the commits being deployed. Then, it computes the average of the related change lead time values for all these commits. Datadog doesn’t store the actual content of files in your repository, only Git commit and tree objects.

For deployments identified through APM Deployment Tracking, the change lead time of a commit is computed from the time of commit creation to when that commit is first seen in a new version. It means that the dora.deploy_time metric is not available.

For service deployments tracked by APM to contribute to change lead time, ensure the following:

For service deployments tracked by the DORA Metrics API or the datadog-ci dora deployment command to contribute to change lead time, ensure the following:

  • Your repository metadata is synchronized to Datadog through the GitHub integration or by the datadog-ci git-metadata upload command.

For more information about the breakdown of change lead time metrics, see Data Collected.

Synchronize repository metadata to Datadog

GitHub workflows running on pull_request trigger are not currently supported by the GitHub integration. If you are using the pull_request trigger, use the alternative method.

If the GitHub integration is not already installed, install it on the GitHub integration tile.

When configuring the GitHub application:

  1. Select at least Read repository permissions for Contents and Pull Requests.
  2. Subscribe at least to Push, PullRequest and PullRequestReview events.

To confirm that the setup is valid, select your GitHub application in the GitHub integration tile and verify that, under the Features tab, the DORA Metrics: Collect Change Lead Time metric feature is enabled.

You can upload your Git repository metadata with the datadog-ci git-metadata upload command. When this command is executed, Datadog receives the repository URL, the commit SHA of the current branch, and a list of tracked file paths.

Run this command in CI for every new commit. If a deployment is executed for a specific commit SHA, ensure that the datadog-ci git-metadata upload command is run for that commit before the deployment event is sent.

Do not provide the --no-gitsync option to the datadog-ci git-metadata upload command. When that option is included, the commit information is not sent to Datadog and the change lead time metric is not calculated.

You can validate the correct setup of the command by checking the command output. An example of a correct output is:

Reporting commit 007f7f466e035b052415134600ea899693e7bb34 from repository git@github.com:organization/example-repository.git.
180 tracked file paths will be reported.
✅  Handled in 0.077 seconds.

Handling multiple services in the same repository

If the source code of multiple services is present in the same repository, further actions are needed to ensure that the change lead time is calculated by taking into account only the commits affecting the specific service being deployed.

To filter the commits measured to only the ones that affect the service, specify the source code glob file path patterns in the service definition.

If the service definition contains a full GitHub URL to the application folder, a single path pattern is automatically used.

Example (schema version v2.2):

links:
  - name: shopist
    type: repo
    provider: github
    url: https://github.com/organization/example-repository/tree/main/src/apps/shopist

DORA Metrics for the shopist service only consider the Git commits that include changes within src/apps/shopist/**. You can configure more granular control of the filtering with extensions[datadoghq.com/dora-metrics].

Example (schema version v2.2):

extensions:
  datadoghq.com/dora-metrics:
    source_patterns:
      - src/apps/shopist/**
      - src/libs/utils/**

DORA Metrics for the service shopist only consider the Git commits that include changes within src/apps/shopist/** or src/libs/utils/**.

Limitations

  • Change lead time stage breakdown metrics are only available for GitHub.
  • Change lead time is not available for the first deployment of a service that includes Git information.
  • For rebased branches, change lead time calculations consider the new commits created during the rebase, not the original commits.
  • When using “Squash” to merge pull requests:
    • For GitHub: Metrics are emitted for the original commits.
    • For other git providers: Metrics are emitted for the new commit added to the target branch.

Calculating change failure rate

Change failure rate is calculated by dividing dora.incidents.count over dora.deployments.count for the same services and/or teams associated to both an incident and a deployment event.

Further Reading

PREVIEWING: brett0000FF/node-compatibility