Instrumenting Envoy

Docs > APM > Application Instrumentation > Tracing a Proxy > Instrumenting Envoy

Datadog APM is included in Envoy v1.9.0 and newer.

Enabling Datadog APM

Note: The example configuration below is for Envoy v1.19.

The following settings are required to enable Datadog APM in Envoy:

a cluster for submitting traces to the Datadog Agent
http_connection_manager configuration to activate tracing

Add a cluster for submitting traces to the Datadog Agent:

 clusters:
 ... existing cluster configs ...
 - name: datadog_agent
   connect_timeout: 1s
   type: strict_dns
   lb_policy: round_robin
   load_assignment:
     cluster_name: datadog_agent
     endpoints:
     - lb_endpoints:
       - endpoint:
           address:
             socket_address:
               address: localhost
               port_value: 8126

Change the address value if Envoy is running in a container or orchestrated environment.

Include the following additional configuration in the http_connection_manager sections to enable tracing:

    - name: envoy.filters.network.http_connection_manager
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        generate_request_id: true
        request_id_extension:
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig
            use_request_id_for_trace_sampling: false
        tracing:
          provider:
            name: envoy.tracers.datadog
            typed_config:
              "@type": type.googleapis.com/envoy.config.trace.v3.DatadogConfig
              collector_cluster: datadog_agent
              service_name: envoy-v1.19

The collector_cluster value must match the name provided for the Datadog Agent cluster. The service_name can be changed to a meaningful value for your usage of Envoy.

With this configuration, HTTP requests to Envoy initiate and propagate Datadog traces, and appear in the APM UI.

Example Envoy v1.19 configuration

The following example configuration demonstrates the placement of items required to enable tracing using Datadog APM.

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    traffic_direction: OUTBOUND
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          generate_request_id: true
          request_id_extension:
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig
              use_request_id_for_trace_sampling: false
          tracing:
            provider:
              name: envoy.tracers.datadog
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.DatadogConfig
                collector_cluster: datadog_agent   # matched against the named cluster
                service_name: envoy-v1.19          # user-defined service name
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service1
          # Traces for healthcheck requests should not be sampled.
          http_filters:
          - name: envoy.filters.http.health_check
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
              pass_through_mode: false
              headers:
                - exact_match: /healthcheck
                  name: :path
          - name: envoy.filters.http.router
            typed_config: {}
          use_remote_address: true
  clusters:
  - name: service1
    connect_timeout: 0.250s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: service1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: service1
                port_value: 80
  # Configure this cluster with the address of the datadog Agent
  # for sending traces.
  - name: datadog_agent
    connect_timeout: 1s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: datadog_agent
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: localhost
                port_value: 8126

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Excluding metrics

If you are using Envoy’s dog_statsd configuration to report metrics, you can exclude activity from the datadog_agent cluster with this additional configuration.

stats_config:
  stats_matcher:
    exclusion_list:
      patterns:
      - prefix: "cluster.datadog_agent."

To control the volume of Envoy traces that are sent to Datadog, specify a sampling rate by setting the parameter DD_TRACE_SAMPLING_RULES to a value between 0.0 (0%) and 1.0 (100%). If no value is specified, 100% of traces starting from Envoy are sent.

To use the Datadog Agent calculated sampling rates (10 traces per second per Agent) and ignore the default sampling rule set to 100%, set the parameter DD_TRACE_SAMPLING_RULES to an empty array:

DD_TRACE_SAMPLING_RULES=[]

You can also define an explicit sampling rate between 0.0 (0%) and 1.0 (100%) by service. For example, to set the sample rate to 10% for service envoy-proxy:

DD_TRACE_SAMPLING_RULES=[{"service": "envoy-proxy","sample_rate": 0.1}]

To configure your sampling rate with DD_TRACE_SAMPLING_RULES, use one of the following methods , depending on how you run Envoy:

By shell script: Set the environment variable immediately before executing envoy in the script:
```
#!/bin/sh
export DD_TRACE_SAMPLING_RULES=[]
envoy -c envoy-config.yaml
```

In a Docker Compose setup: Set the environment variable in the environment section of the service definition:

services:
  envoy:
    image: envoyproxy/envoy:v1.19-latest
    entrypoint: []
    command:
        - envoy
        - -c
        - /etc/envoy/envoy.yaml
    volumes:
        - './envoy.yaml:/etc/envoy/envoy.yaml:ro'
    environment:
        - DD_TRACE_SAMPLING_RULES=[]

As a container inside a Kubernetes pod: specify the environment variable in the env section of the corresponding containers entry of the pod’s spec:

apiVersion: v1
kind: Pod
metadata:
  name: envoy
spec:
  containers:
  - name: envoy
    image: envoyproxy/envoy:v1.20-latest
    env:
    - name: DD_TRACE_SAMPLING_RULES
      value: "[]"

Environment variables

Note: The variables DD_AGENT_HOST, DD_TRACE_AGENT_PORT and DD_TRACE_AGENT_URL do not apply to Envoy, as the address of the Datadog Agent is configured using the cluster settings.

The available environment variables depend on the version of the C++ tracer embedded in Envoy. The version of the C++ tracer can be found in the logs, indicated by the line starting with “DATADOG TRACER CONFIGURATION”.

Instrumenting Envoy

Enabling Datadog APM

Example Envoy v1.19 configuration

Excluding metrics

Envoy Sampling

Environment variables

Further Reading