Overview

Datadog’s OpenTelemetry protocol (OTLP) intake API endpoint allows you to send traces directly to Datadog. With this feature, you don’t need to run the Datadog Agent or OpenTelemetry Collector + Datadog Exporter.

You might prefer this option if you’re looking for a straightforward setup and want to send traces directly to Datadog without using the Datadog Agent or OpenTelemetry Collector.

Configuration

To export OTLP data to the Datadog OTLP traces intake endpoint:

1. Configure the OTLP HTTP Protobuf exporter.
   • Set the Datadog OTLP traces intake endpoint.
   • Configure the required HTTP headers.
2. (Optional) Set the dd-otel-span-mapping HTTP header to map or filter spans.

Configure the exporter

To send OTLP data to the Datadog OTLP traces intake endpoint, you need to use the OTLP HTTP Protobuf exporter. The process differs depending on whether you are using automatic or manual instrumentation for OpenTelemetry.

Automatic instrumentation

If you are using OpenTelemetry automatic instrumentation, set the following environment variables:

export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=""
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="dd-protocol=otlp,dd-api-key=${DD_API_KEY},dd-otlp-source=${YOUR_SITE}"

Manual instrumentation

If you are using manual instrumentation with OpenTelemetry SDKs, configure the OTLP HTTP Protobuf exporter programmatically.

const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto');  // OTLP http/protobuf exporter

const exporter = new OTLPTraceExporter({
  url: '${YOUR_ENDPOINT}', // Replace this with the correct endpoint
  headers: {
    'dd-protocol': 'otlp',
    'dd-api-key': process.env.DD_API_KEY,
    'dd-otel-span-mapping': '{span_name_as_resource_name: true}',
    'dd-otlp-source': '${YOUR_SITE}', // Replace this with the correct site
  },
});

import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;

OtlpHttpSpanExporter exporter = OtlpHttpSpanExporter.builder()
    .setEndpoint("${YOUR_ENDPOINT}") // Replace this with the correct endpoint
    .addHeader("dd-protocol", "otlp")
    .addHeader("dd-api-key", System.getenv("DD_API_KEY"))
    .addHeader("dd-otel-span-mapping", "{span_name_as_resource_name: true}")
    .addHeader("dd-otlp-source", "${YOUR_SITE}") // Replace this with the correct site
    .build();

import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

traceExporter, err := otlptracehttp.New(
	ctx,
	otlptracehttp.WithEndpoint("${YOUR_ENDPOINT}"), // Replace this with the correct endpoint
	otlptracehttp.WithURLPath("/api/v0.2/traces"),
	otlptracehttp.WithHeaders(
		map[string]string{
			"dd-protocol": "otlp",
			"dd-api-key": os.Getenv("DD_API_KEY"),
			"dd-otel-span-mapping": "{span_name_as_resource_name: true}",
      "dd-otlp-source": "${YOUR_SITE}", // Replace this with the correct site
		}),
)

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="${YOUR_ENDPOINT}", # Replace this with the correct endpoint
    headers={
        "dd-protocol": "otlp",
        "dd-api-key": os.environ.get("DD_API_KEY"),
        "dd-otel-span-mapping": "{span_name_as_resource_name: true}",
        "dd-otlp-source": "${YOUR_SITE}" # Replace this with the correct site
    },
)

(Optional) Map or filter span names

Use the dd-otel-span-mapping header to configure span mapping and filtering. The JSON header contains the following fields:

• ignore_resources: A list of regular expressions to disable traces based on their resource name.
• span_name_remappings: A map of Datadog span names to preferred names.
• span_name_as_resource_name: Specifies whether to use the OpenTelemetry span’s name as the Datadog span’s operation name (default: true). If false, the operation name is derived from a combination of the instrumentation scope name and span kind.

For example:

{
  "span_name_as_resource_name":false,
  "span_name_remappings":{
    "io.opentelemetry.javaagent.spring.client":"spring.client"
  },
  "ignore_resources":[
    "io.opentelemetry.javaagent.spring.internal"
  ]
}

OpenTelemetry Collector

If you are using the OpenTelemetry Collector and don’t want to use the Datadog Exporter, you can configure otlphttpexporter to export traces to the Datadog OTLP traces intake endpoint.

For example, configure your config.yaml like this:

...

exporters:
  otlphttp:
    traces_endpoint: 
    headers:
      dd-protocol: "otlp"
      dd-api-key: ${env:DD_API_KEY}
      dd-otel-span-mapping: "{span_name_as_resource_name: false}"
      dd-otlp-source: "${YOUR_SITE}", # Replace this with the correct site
...

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Troubleshooting

Error: 403 Forbidden

If you receive a 403 Forbidden error when sending traces to the Datadog OTLP traces intake endpoint, it indicates one of the following issues:

• The API key belongs to an organization that is not allowed to access the Datadog OTLP traces intake endpoint. Solution: Verify that you are using an API key from an organization that is allowed to access the Datadog OTLP traces intake endpoint.
• The dd-otlp-source header is missing or has an incorrect value. Solution: Ensure that the dd-otlp-source header is set with the proper value for your site. You should have received an allowlisted value for this header from Datadog if you are a platform partner.
• The endpoint URL is incorrect for your organization. Solution: Use the correct endpoint URL for your organization. Your site is , so you need to use the endpoint.

Error: 413 Request Entity Too Large

If you receive a 413 Request Entity Too Large error when sending traces to the Datadog OTLP traces intake endpoint, it indicates that the payload size sent by the OTLP exporter exceeds the Datadog traces intake endpoint’s limit of 3.2MB.

This error usually occurs when the OpenTelemetry SDK batches too much telemetry data in a single request payload.

Solution: Reduce the export batch size of the SDK’s batch span processor. Here’s an example of how to modify the BatchSpanProcessorBuilder in the OpenTelemetry Java SDK:

CopyBatchSpanProcessor batchSpanProcessor =
    BatchSpanProcessor
        .builder(exporter)
        .setMaxExportBatchSize(10)  // Default is 512
        .build();

Adjust the setMaxExportBatchSize value according to your needs. A smaller value results in more frequent exports with smaller payloads, reducing the likelihood of exceeding the 3.2MB limit.

Warning: “traces export: failed … 202 Accepted” in Go

If you are using the OpenTelemetry Go SDK and see a warning message similar to traces export: failed … 202 Accepted, it is due to a known issue in the OpenTelemetry Go OTLP HTTP exporter.

The OpenTelemetry Go OTLP HTTP exporter treats any HTTP status code other than 200 as an error, even if the export succeeds (Issue 3706). In contrast, other OpenTelemetry SDKs consider any status code in the range [200, 300) as a success. The Datadog OTLP traces intake endpoint returns a 202 Accepted status code for successful exports.

The OpenTelemetry community is still discussing whether other 2xx status codes should be treated as successes (Issue 3203).

Solution: If you are using the Datadog OTLP traces intake endpoint with the OpenTelemetry Go SDK, you can safely ignore this warning message. Your traces are being successfully exported despite the warning.

Issue: Unexpected span operation names

When using the Datadog OTLP trace intake endpoint, you may notice that the span operation names are different from those generated when using the Datadog Agent or OpenTelemetry Collector.

The Datadog OTLP trace intake endpoint has the span_name_as_resource_name option set to true by default. This means that Datadog uses the OpenTelemetry span’s name as the operation name. In contrast, the Datadog Agent and OpenTelemetry Collector have this option set to false by default.

When span_name_as_resource_name is set to false, the operation name is derived from a combination of the instrumentation scope name and the span kind. For example, an operation name might appear as opentelemetry.client.

Solution: If you want to disable the span_name_as_resource_name option in the Datadog OTLP traces intake endpoint to match the behavior of the Datadog Agent or OpenTelemetry Collector, follow these steps:

1. Refer to Map or filter span names in this document.
2. Set the span_name_as_resource_name option to false in the dd-otel-span-mapping header.

For example:

jsonCopy{
  "span_name_as_resource_name": false,
  ...
}

This ensures that the span operation names are consistent across the Datadog OTLP traces intake endpoint, Datadog Agent, and OpenTelemetry Collector.

Further reading

추가 유용한 문서, 링크 및 기사:

General OpenTelemetry SDK ConfigurationEXTERNAL SITE OpenTelemetry Environment Variable SpecEXTERNAL SITE OpenTelemetry Protocol ExporterEXTERNAL SITE