Overview

Feature flag tracking in Datadog helps you correlate flag changes with system performance issues, accelerating incident resolution. By tracking when flags are toggled, updated, or rolled out, you can identify whether a feature flag change caused a performance degradation or outage.

With feature flag tracking, you can:

• View flag changes directly in your dashboards, monitors, and service pages
• Automatically identify which services are affected by a flag change
• Roll back problematic flags directly from inside Datadog

Track feature flags

Datadog supports tracking LaunchDarkly flags using the LaunchDarkly integration or tracking feature flags from other feature flag providers using the Events API.

LaunchDarkly flags

To track LaunchDarkly feature flags in your services’ Change Tracking timeline:

1. Enable the LaunchDarkly integration in Datadog.
2. Go to Flags > <your-feature-flag-name> > Settings in LaunchDarkly.
3. In Custom properties, add a tag with key service and value <your-service-name>, matching your Datadog service name exactly.
4. Click Save changes.

For example, to link a flag to the payments_api service used in the examples below, you would set the tag value to payments_api. After you submit the event, you can navigate to the Software Catalog, select the payments_api service, and see the fallback_payments_test feature flag event in the Change Tracking timeline.

Custom feature flags

Send feature flag events from any provider using the Events API. Create a change category event and include a service tag to link the event to your service.

Example request:

{
  "data": {
    "attributes": {
      "aggregation_key": "string",
      "attributes": {
        "author": {
          "name": "datadog@datadog.com",
          "type": "user"
        },
        "change_metadata": {
          "dd": {
            "team": "datadog_team",
            "user_email": "datadog@datadog.com",
            "user_id": "datadog_user_id",
            "user_name": "datadog_username"
          },
          "resource_link": "datadog.com/feature/fallback_payments_test"
        },
        "changed_resource": {
          "name": "fallback_payments_test",
          "type": "feature_flag"
        },
        "impacted_resources": [
          {
            "name": "payments_api",
            "type": "service"
          }
        ],
        "new_value": {
          "enabled": true,
          "percentage": "50%",
          "rule": {
            "datacenter": "devcycle.us1.prod"
          }
        },
        "prev_value": {
          "enabled": true,
          "percentage": "10%",
          "rule": {
            "datacenter": "devcycle.us1.prod"
          }
        }
      },
      "category": "change",
      "message": "payment_processed feature flag has been enabled",
      "tags": [
        "env:test"
      ],
      "timestamp": "string",
      "title": "payment_processed feature flag updated"
    },
    "type": "event"
  }
}

Automatically detect affected services

In addition to tracking when a feature flag’s configuration changes using the LaunchDarkly integration or the Events API, you can also automatically identify every service that evaluates a flag. This auto-enrichment provides deeper context by using APM traces to show the real-time impact of a flag, which is especially useful when a single flag is used by multiple services.

Setup

To automatically detect services using a feature flag, instrument your feature flag evaluation code with the APM tracing library. This allows Datadog to automatically detect all services that evaluate a specific flag, even if they weren’t originally tagged.

1. Instrument your feature flag evaluation code using the Datadog tracing library.
2. Create a custom span with the operation name experiments.IsEnabled to track feature flag evaluations.
3. Tag the span with experiment_id:<flag-id>, where <flag-id> matches the feature flag ID.

For example:

# Trace feature flag evaluation to enable auto-detection
with tracer.trace("experiments.IsEnabled") as span:
    span.set_tag("experiment_id", "fallback_payments_test")
    # Your existing feature flag evaluation code
    flag_value = evaluate_flag("fallback_payments_test")

Remediate feature flag changes with Workflow Automation

When you identify that a feature flag change caused an issue, you can immediately toggle its state without leaving Datadog. This feature uses Workflow Automation to toggle LaunchDarkly flags directly from Change Tracking timelines.

Setup

To set up feature flag toggles using Workflow Automation:

1. Go to Actions > Action Catalog > Connections.
2. Click New Connection.
3. Choose LaunchDarkly.
4. Complete the required information, then click Next, Confirm Access.
5. Set access permissions for the connection.
6. Click Create.

Use feature flag toggles

To toggle feature flags on or off from inside Datadog:

1. Click a LaunchDarkly feature flag change in the Change Tracking timeline.
2. Click the Toggle Feature Flag button.
3. Click Run Action to run the workflow and toggle the feature flag on or off.

Further reading

Additional helpful documentation, links, and articles:

Change TrackingDOCUMENTATION LaunchDarklyDOCUMENTATION