Datadog OTLP Traces Intake Endpoint
The Datadog OTLP traces intake endpoint is in Preview.
Datadog OTLP traces intake endpoint is not supported for your selected
Datadog site (
).
Overview
Datadog’s OpenTelemetry protocol (OTLP) intake API endpoint allows you to send traces directly to Datadog. With this feature, you don’t need to run the Datadog Agent or OpenTelemetry Collector + Datadog Exporter.
You might prefer this option if you’re looking for a straightforward setup and want to send traces directly to Datadog without using the Datadog Agent or OpenTelemetry Collector.
Configuration
To export OTLP data to the Datadog OTLP traces intake endpoint:
- Configure the OTLP HTTP Protobuf exporter.
- Set the Datadog OTLP traces intake endpoint.
- Configure the required HTTP headers.
- (Optional) Set the
dd-otel-span-mapping
HTTP header to map or filter spans.
To send OTLP data to the Datadog OTLP traces intake endpoint, you need to use the OTLP HTTP Protobuf exporter. The process differs depending on whether you are using automatic or manual instrumentation for OpenTelemetry.
Automatic instrumentation
If you are using OpenTelemetry automatic instrumentation, set the following environment variables:
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=""
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="dd-protocol=otlp,dd-api-key=${DD_API_KEY},dd-otlp-source=${YOUR_SITE}"
Manual instrumentation
If you are using manual instrumentation with OpenTelemetry SDKs, configure the OTLP HTTP Protobuf exporter programmatically.
Based on your
Datadog site, which is
:
- Replace
${YOUR_ENDPOINT}
with
. - Replace
${YOUR_SITE}
with the organization name you received from Datadog.
The JavaScript exporter is exporter-trace-otlp-proto
. To configure the exporter, use the following code snippet:
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto'); // OTLP http/protobuf exporter
const exporter = new OTLPTraceExporter({
url: '${YOUR_ENDPOINT}', // Replace this with the correct endpoint
headers: {
'dd-protocol': 'otlp',
'dd-api-key': process.env.DD_API_KEY,
'dd-otel-span-mapping': '{span_name_as_resource_name: true}',
'dd-otlp-source': '${YOUR_SITE}', // Replace this with the correct site
},
});
The Java exporter is OtlpHttpSpanExporter
. To configure the exporter, use the following code snippet:
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
OtlpHttpSpanExporter exporter = OtlpHttpSpanExporter.builder()
.setEndpoint("${YOUR_ENDPOINT}") // Replace this with the correct endpoint
.addHeader("dd-protocol", "otlp")
.addHeader("dd-api-key", System.getenv("DD_API_KEY"))
.addHeader("dd-otel-span-mapping", "{span_name_as_resource_name: true}")
.addHeader("dd-otlp-source", "${YOUR_SITE}") // Replace this with the correct site
.build();
The Go exporter is otlptracehttp
. To configure the exporter, use the following code snippet:
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
traceExporter, err := otlptracehttp.New(
ctx,
otlptracehttp.WithEndpoint("${YOUR_ENDPOINT}"), // Replace this with the correct endpoint
otlptracehttp.WithURLPath("/api/v0.2/traces"),
otlptracehttp.WithHeaders(
map[string]string{
"dd-protocol": "otlp",
"dd-api-key": os.Getenv("DD_API_KEY"),
"dd-otel-span-mapping": "{span_name_as_resource_name: true}",
"dd-otlp-source": "${YOUR_SITE}", // Replace this with the correct site
}),
)
The Python exporter is OTLPSpanExporter
. To configure the exporter, use the following code snippet:
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
exporter = OTLPSpanExporter(
endpoint="${YOUR_ENDPOINT}", # Replace this with the correct endpoint
headers={
"dd-protocol": "otlp",
"dd-api-key": os.environ.get("DD_API_KEY"),
"dd-otel-span-mapping": "{span_name_as_resource_name: true}",
"dd-otlp-source": "${YOUR_SITE}" # Replace this with the correct site
},
)
(Optional) Map or filter span names
Use the dd-otel-span-mapping
header to configure span mapping and filtering. The JSON header contains the following fields:
ignore_resources
: A list of regular expressions to disable traces based on their resource name.span_name_remappings
: A map of Datadog span names to preferred names.span_name_as_resource_name
: Specifies whether to use the OpenTelemetry span’s name as the Datadog span’s operation name (default: true). If false, the operation name is derived from a combination of the instrumentation scope name and span kind.
For example:
{
"span_name_as_resource_name":false,
"span_name_remappings":{
"io.opentelemetry.javaagent.spring.client":"spring.client"
},
"ignore_resources":[
"io.opentelemetry.javaagent.spring.internal"
]
}
OpenTelemetry Collector
If you are using the OpenTelemetry Collector and don’t want to use the Datadog Exporter, you can configure otlphttpexporter
to export traces to the Datadog OTLP traces intake endpoint.
For example, configure your config.yaml
like this:
...
exporters:
otlphttp:
traces_endpoint:
headers:
dd-protocol: "otlp"
dd-api-key: ${env:DD_API_KEY}
dd-otel-span-mapping: "{span_name_as_resource_name: false}"
dd-otlp-source: "${YOUR_SITE}", # Replace this with the correct site
...
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
Troubleshooting
Error: 403 Forbidden
If you receive a 403 Forbidden
error when sending traces to the Datadog OTLP traces intake endpoint, it indicates one of the following issues:
- The API key belongs to an organization that is not allowed to access the Datadog OTLP traces intake endpoint.
Solution: Verify that you are using an API key from an organization that is allowed to access the Datadog OTLP traces intake endpoint.
- The
dd-otlp-source
header is missing or has an incorrect value.
Solution: Ensure that the dd-otlp-source
header is set with the proper value for your site. You should have received an allowlisted value for this header from Datadog if you are a platform partner. - The endpoint URL is incorrect for your organization.
Solution: Use the correct endpoint URL for your organization. Your site is
, so you need to use the
endpoint.
Error: 413 Request Entity Too Large
If you receive a 413 Request Entity Too Large
error when sending traces to the Datadog OTLP traces intake endpoint, it indicates that the payload size sent by the OTLP exporter exceeds the Datadog traces intake endpoint’s limit of 3.2MB.
This error usually occurs when the OpenTelemetry SDK batches too much telemetry data in a single request payload.
Solution: Reduce the export batch size of the SDK’s batch span processor. Here’s an example of how to modify the BatchSpanProcessorBuilder
in the OpenTelemetry Java SDK:
CopyBatchSpanProcessor batchSpanProcessor =
BatchSpanProcessor
.builder(exporter)
.setMaxExportBatchSize(10) // Default is 512
.build();
Adjust the setMaxExportBatchSize
value according to your needs. A smaller value results in more frequent exports with smaller payloads, reducing the likelihood of exceeding the 3.2MB limit.
Warning: “traces export: failed … 202 Accepted” in Go
If you are using the OpenTelemetry Go SDK and see a warning message similar to traces export: failed … 202 Accepted
, it is due to a known issue in the OpenTelemetry Go OTLP HTTP exporter.
The OpenTelemetry Go OTLP HTTP exporter treats any HTTP status code other than 200 as an error, even if the export succeeds (Issue 3706). In contrast, other OpenTelemetry SDKs consider any status code in the range [200, 300) as a success. The Datadog OTLP traces intake endpoint returns a 202 Accepted
status code for successful exports.
The OpenTelemetry community is still discussing whether other 2xx
status codes should be treated as successes (Issue 3203).
Solution: If you are using the Datadog OTLP traces intake endpoint with the OpenTelemetry Go SDK, you can safely ignore this warning message. Your traces are being successfully exported despite the warning.
Issue: Unexpected span operation names
When using the Datadog OTLP trace intake endpoint, you may notice that the span operation names are different from those generated when using the Datadog Agent or OpenTelemetry Collector.
The Datadog OTLP trace intake endpoint has the span_name_as_resource_name
option set to true
by default. This means that Datadog uses the OpenTelemetry span’s name as the operation name. In contrast, the Datadog Agent and OpenTelemetry Collector have this option set to false
by default.
When span_name_as_resource_name
is set to false
, the operation name is derived from a combination of the instrumentation scope name and the span kind. For example, an operation name might appear as opentelemetry.client
.
Solution: If you want to disable the span_name_as_resource_name
option in the Datadog OTLP traces intake endpoint to match the behavior of the Datadog Agent or OpenTelemetry Collector, follow these steps:
- Refer to Map or filter span names in this document.
- Set the
span_name_as_resource_name
option to false
in the dd-otel-span-mapping
header.
For example:
jsonCopy{
"span_name_as_resource_name": false,
...
}
This ensures that the span operation names are consistent across the Datadog OTLP traces intake endpoint, Datadog Agent, and OpenTelemetry Collector.
Further reading
Additional helpful documentation, links, and articles: