A service can handle a variety of requests, some of which you might not want traced or included in trace metrics. An example of this is, possibly, health checks in a web application.

There are two ways to specify that such an endpoint should be untraced and excluded from trace metrics:

• Trace Agent configuration (in Datadog Agent), or
• Tracer configuration.

If you need assistance, contact Datadog support.

Trace Agent configuration options

The Trace Agent component within the Datadog Agent has two methods to prevent certain traces from coming through: ignoring span tags or ignoring resources. If traces are dropped due to these settings, the trace metrics exclude these requests.

Configuring the Trace Agent to ignore certain spans or resources applies to all services that send traces to this particular Datadog Agent. If you have application-specific requirements, use the Tracer configuration method instead.

Ignoring based on span tags

Starting with Datadog Agent 6.27.0/7.27.0, the filter tags option drops traces with root spans that match specified span tags. This option applies to all services that send traces to this particular Datadog Agent. Traces that are dropped because of filter tags are not included in trace metrics.

If you can programmatically identify a set of traces that you know you don’t want to send to Datadog, and no other option in this guide solves your requirement, you can consider adding a custom span tag so you can drop the traces. Reach out to Support to discuss your use case further so Datadog can continue to expand this functionality.

The filter tags option requires an exact string match. If your use case requires ignoring by regex, see Ignoring based on resources.

You can specify span tags to require or reject by using a list of keys and values separated by spaces in environment variables:

DD_APM_FILTER_TAGS_REQUIRE

Collects only traces that have root spans with an exact match for the specified span tags and values. If it does not match this rule, the trace is dropped. For example, DD_APM_FILTER_TAGS_REQUIRE="key1:value1 key2:value2". In Datadog Agent 7.49+, regular expressions can be provided with DD_APM_FILTER_TAGS_REGEX_REQUIRE.

DD_APM_FILTER_TAGS_REJECT

Rejects traces that have root spans with an exact match for the specified span tags and values. If it matches this rule, the trace is dropped. For example, DD_APM_FILTER_TAGS_REJECT="key1:value1 key2:value2". In Datadog Agent 7.49+, regular expressions can be provided with DD_APM_FILTER_TAGS_REGEX_REJECT.

apm_config:
  filter_tags:
    require: ["db:sql", "db.instance:mysql"]
    reject: ["outcome:success", "key2:value2"]

apm_config:
  filter_tags:
    reject: ["http.url:http://localhost:5050/healthcheck"]

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  override:
    nodeAgent:
      containers:
        trace-agent:
          env:
            - name: DD_APM_FILTER_TAGS_REJECT
              value: tag_key1:tag_val2 tag_key2:tag_val2

kubectl apply -n $DD_NAMESPACE -f datadog-agent.yaml

agents:
  containers:
    traceAgent:
      env:
        - name: DD_APM_FILTER_TAGS_REJECT
          value: tag_key1:tag_val2 tag_key2:tag_val2

helm upgrade -f datadog-values.yaml <RELEASE NAME> datadog/datadog

Filtering traces this way removes these requests from trace metrics. For more information on how to reduce ingestion without affecting the trace metrics, see Ingestion Controls.

On the backend, Datadog creates and adds the following span tags to spans after ingestion. Note, these tags cannot be used to drop traces at the Datadog Agent level, as the agent only filters based on tags available before ingestion.

Network communications

HTTP requests

Database

Message Queue

Remote procedure calls

Errors

Ignoring based on resources

The ignore resources option allows resources to be excluded if the global root span of the trace matches certain criteria. See Exclude resources from being collected. This option applies to all services that send traces to this particular Datadog Agent. Traces that are dropped because of ignore resources are not included in trace metrics.

You can specify resources to ignore either in the Agent configuration file, datadog.yaml, or with the DD_APM_IGNORE_RESOURCES environment variable. See examples below.

apm_config:
## @param ignore_resources - list of strings - optional
## A list of regular expressions can be provided to exclude certain traces based on their resource name.
## All entries must be surrounded by double quotes and separated by commas.

  ignore_resources: ["(GET|POST) /healthcheck","API::NotesController#index"]

Notes:

• The regex syntax that the Trace Agent accepts is evaluated by Go’s regexp.
• Depending on your deployment strategy, you may have to adjust the regex by escaping special characters.
• If you use dedicated containers with Kubernetes, make sure that the environment variable for the ignore resource option is being applied to the trace-agent container.

Example

Consider a trace that contains calls to /api/healthcheck that you don’t want traces from:

Take note of the resource name of the global root span.

• Operation name: rack.request
• Resource name: Api::HealthchecksController#index
• Http.url: /api/healthcheck

To use the ignore resource option correctly, the regex rule written must match with the resource name, Api::HealthchecksController#index. A few regex options are possible, but to filter out traces from this resource exactly as is, a potential regex to use is Api::HealthchecksController#index$.

Depending on how you deploy, the syntax looks a little different:

apm_config:
  ignore_resources: Api::HealthchecksController#index$

apm_config:
  ignore_resources: ["value1","Api::HealthchecksController#index$"]

    environment:
      // other Datadog Agent environment variables
      - DD_APM_IGNORE_RESOURCES=Api::HealthchecksController#index$$

    environment:
      // other Datadog Agent environment variables
      - DD_APM_IGNORE_RESOURCES="value1","Api::HealthchecksController#index$$"

docker run -d --name datadog-agent \
              --cgroupns host \
              --pid host \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -e DD_API_KEY=<> \
              -e DD_APM_IGNORE_RESOURCES="Api::HealthchecksController#index$" \
              -e DD_APM_ENABLED=true \
              -e DD_APM_NON_LOCAL_TRAFFIC=true \
              gcr.io/datadoghq/agent:latest

              -e DD_APM_IGNORE_RESOURCES=["value1","Api::HealthchecksController#index$"] \

    - name: trace-agent
        image: "gcr.io/datadoghq/agent:latest"
        imagePullPolicy: IfNotPresent
        command: ["trace-agent", "-config=/etc/datadog-agent/datadog.yaml"]
        resources: {}
        ports:
        - containerPort: 8126
          hostPort: 8126
          name: traceport
          protocol: TCP
        env:
        - name: DD_API_KEY
          valueFrom:
            secretKeyRef:
              name: "datadog-secret"
              key: api-key
        - name: DD_KUBERNETES_KUBELET_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: KUBERNETES
          value: "yes"
        - name: DOCKER_HOST
          value: unix:///host/var/run/docker.sock
        - name: DD_LOG_LEVEL
          value: "INFO"
        - name: DD_APM_ENABLED
          value: "true"
        - name: DD_APM_NON_LOCAL_TRAFFIC
          value: "true"
        - name: DD_APM_RECEIVER_PORT
          value: "8126"
        - name: DD_KUBELET_TLS_VERIFY
          value: "false"
        - name: DD_APM_IGNORE_RESOURCES
          value: "Api::HealthchecksController#index$"

        - name: DD_APM_IGNORE_RESOURCES
          value: '"value1","Api::HealthchecksController#index$"'

    traceAgent:
      # agents.containers.traceAgent.env -- Additional environment variables for the trace-agent container
      env:
        - name: DD_APM_IGNORE_RESOURCES
          value: Api::HealthchecksController#index$

        - name: DD_APM_IGNORE_RESOURCES
          value: value1, Api::HealthchecksController#index$

helm install dd-agent -f values.yaml \
  --set datadog.apiKeyExistingSecret="datadog-secret" \
  --set agents.containers.traceAgent.env[0].name=DD_APM_IGNORE_RESOURCES, \
    agents.containers.traceAgent.env[0].value="Api::HealthchecksController#index$" \
  datadog/datadog

    "environment": [
	// other environment variables for the Datadog Agent
        {
          "name": "DD_APM_IGNORE_RESOURCES",
          "value": "Api::HealthchecksController#index$"
        }
     ]

Tracer configuration options

Some of the language-specific tracers have an option to modify spans before they are sent to the Datadog Agent. Use this option if you have application-specific requirements and are using a language listed below.

Datadog::Tracing.before_flush(
   Datadog::Tracing::Pipeline::SpanFilter.new { |span| span.resource =~ /Api::HealthchecksController#index/ }
)

from ddtrace import tracer
from ddtrace.filters import FilterRequestsOnUrl
tracer.configure(settings={
    'FILTERS': [
        FilterRequestsOnUrl(r'http://.*/healthcheck$'),
    ],
})

const tracer = require('dd-trace').init();
tracer.use('http', {
  // incoming http requests match on the path
  server: {
    blocklist: ['/healthcheck']
  },
  // outgoing http requests match on a full URL
  client: {
    blocklist: ['https://telemetry.example.org/api/v1/record']
  }
})

//import http

public class GreetingController {
   static {
       // In a class static block to avoid initializing multiple times.
       GlobalTracer.get().addTraceInterceptor(new TraceInterceptor() {
           @Override
           public Collection<? extends MutableSpan> onTraceComplete(Collection<? extends MutableSpan> trace) {
               for (MutableSpan span : trace) {
                   if ("GET /healthcheck".contentEquals(span.getResourceName())) {
                       return Collections.emptyList();
                   }
               }
               return trace;
           }
           @Override
           public int priority() {
               return 200;  // Some unique number
           }
       });
   }
}