Trace Context propagation is the mechanism of passing tracing information like Trace ID, Span ID, and sampling decisions from one part of a distributed application to another. This enables all traces (and additional telemetry) in a request to be correlated. When automatic instrumentation is enabled, trace context propagation is handled automatically by the APM SDK.
By default, the Datadog SDK extracts and injects distributed tracing headers using the following formats:
This default configuration maximizes compatibility with older Datadog SDK versions and products while allowing interoperability with other distributed tracing systems like OpenTelemetry.
Customize trace context propagation
You may need to customize the trace context propagation configuration if your applications:
- Communicate distributed tracing information in a different supported format
- Need to prevent extracting or injecting distributed tracing headers
Use the following environment variables to configure formats for reading and writing distributed tracing headers. Refer to the Language support section for language-specific configuration values.
DD_TRACE_PROPAGATION_STYLE
- Specifies trace context propagation formats for extraction and injection in a comma-separated list. May be overridden by extract-specific or inject-specific configurations.
Default: datadog,tracecontext
Note: With multiple formats, extraction follows the specified order (for example, datadog,tracecontext
checks Datadog headers first). The first valid context continues the trace; additional valid contexts become span links. OTEL_PROPAGATORS
- Specifies trace context propagation formats for both extraction and injection (comma-separated list). Lowest precedence; ignored if any other Datadog trace context propagation environment variable is set.
Note: Only use this configuration when migrating an application from the OpenTelemetry SDK to the Datadog SDK. For more information on this configuration and other OpenTelemetry environment variables, see Using OpenTelemetry Environment Variables with Datadog SDKs.
高度なコンフィギュレーション
Most services send and receive trace context headers using the same format. However, if your service needs to accept trace context headers in one format and send them in another, use these configurations:
DD_TRACE_PROPAGATION_STYLE_EXTRACT
- Specifies trace context propagation formats for extraction only in a comma-separated list. Highest precedence for configuring extraction propagators.
DD_TRACE_PROPAGATION_STYLE_INJECT
- Specifies trace context propagation formats for injection only in comma-separated list. Highest precedence for configuring injection propagators.
The Datadog SDK supports the following trace context formats:
Language support
The Datadog Java SDK supports the following trace context formats, including deprecated configuration values:
追加構成
In addition to the environment variable configuration, you can also update the propagators using System Property configuration:
-Ddd.trace.propagation.style=datadog,b3multi
-Dotel.propagators=datadog,b3multi
-Ddd.trace.propagation.style.inject=datadog,b3multi
-Ddd.trace.propagation.style.extract=datadog,b3multi
The Datadog Python SDK supports the following trace context formats, including deprecated configuration values:
The Datadog Ruby SDK supports the following trace context formats, including deprecated configuration values:
追加構成
In addition to the environment variable configuration, you can also update the propagators in code by using Datadog.configure
:
Datadog.configure do |c|
# List of header formats that should be extracted
c.tracing.propagation_extract_style = [ 'tracecontext', 'datadog', 'b3' ]
# List of header formats that should be injected
c.tracing.propagation_inject_style = [ 'tracecontext', 'datadog' ]
end
The Datadog Go SDK supports the following trace context formats, including deprecated configuration values:
The Datadog Node.js SDK supports the following trace context formats, including deprecated configuration values:
The Datadog PHP SDK supports the following trace context formats, including deprecated configuration values:
さまざまな使用例
The following use cases are specific to the Datadog PHP SDK:
When a new PHP script is launched, the Datadog SDK automatically checks for the presence of Datadog headers for distributed tracing:
x-datadog-trace-id
(環境変数: HTTP_X_DATADOG_TRACE_ID
)x-datadog-parent-id
(環境変数: HTTP_X_DATADOG_PARENT_ID
)x-datadog-origin
(環境変数: HTTP_X_DATADOG_ORIGIN
)x-datadog-tags
(環境変数: HTTP_X_DATADOG_TAGS
)
To manually set tracing information in a CLI script for new or existing traces, use the DDTrace\set_distributed_tracing_context(string $trace_id, string $parent_id, ?string $origin = null, ?array $tags = null)
function.
<?php
function processIncomingQueueMessage($message) {
}
\DDTrace\trace_function(
'processIncomingQueueMessage',
function(\DDTrace\SpanData $span, $args) {
$message = $args[0];
\DDTrace\set_distributed_tracing_context($message->trace_id, $message->parent_id);
}
);
For version 0.87.0 and later, if the raw headers are available, use the DDTrace\consume_distributed_tracing_headers(array|callable $headersOrCallback)
function. Note: The header names must be in lowercase.
$headers = [
"x-datadog-trace-id" => "1234567890",
"x-datadog-parent-id" => "987654321",
];
\DDTrace\consume_distributed_tracing_headers($headers);
To extract the trace context directly as headers, use the DDTrace\generate_distributed_tracing_headers(?array $inject = null): array
function.
$headers = DDTrace\generate_distributed_tracing_headers();
// ヘッダーをどこかに保存し、アウトバウンドリクエストに挿入し、...
// また、これらの $headers は、他のプロセスから \DDTrace\consume_distributed_tracing_headers によって読み返すことができます。
This function’s optional argument accepts an array of injection style names. It defaults to the configured injection style.
The PHP APM SDK supports automatic tracing of the php-amqplib/php-amqplib
library (version 0.87.0+). However, in some cases, your distributed trace may be disconnected. For example, when reading messages from a distributed queue using the basic_get
method outside an existing trace, you need to add a custom trace around the basic_get
call and corresponding message processing:
// 囲むトレースを作成します
$newTrace = \DDTrace\start_trace_span();
$newTrace->name = 'basic_get.process';
$newTrace->service = 'amqp';
// basic_get call(s) + message(s) 処理
$msg = $channel->basic_get($queue);
if ($msg) {
$messageProcessing($msg);
}
// 完了したら、スパンを閉じます
\DDTrace\close_span();
この囲むトレースを消費処理ロジックに作成することで、分散キューの観測可能性を確保することができます。
The Datadog C++ SDK supports the following trace context formats, including deprecated configuration values:
追加構成
In addition to the environment variable configuration, you can also update the propagators in code:
#include <datadog/tracer_config.h>
#include <datadog/propagation_style.h>
namespace dd = datadog::tracing;
int main() {
dd::TracerConfig config;
config.service = "my-service";
// `injection_styles` indicates with which tracing systems trace propagation
// will be compatible when injecting (sending) trace context.
// All styles indicated by `injection_styles` are used for injection.
// `injection_styles` is overridden by the `DD_TRACE_PROPAGATION_STYLE_INJECT`
// and `DD_TRACE_PROPAGATION_STYLE` environment variables.
config.injection_styles = {dd::PropagationStyle::DATADOG, dd::PropagationStyle::B3};
// `extraction_styles` indicates with which tracing systems trace propagation
// will be compatible when extracting (receiving) trace context.
// Extraction styles are applied in the order in which they appear in
// `extraction_styles`. The first style that produces trace context or
// produces an error determines the result of extraction.
// `extraction_styles` is overridden by the
// `DD_TRACE_PROPAGATION_STYLE_EXTRACT` and `DD_TRACE_PROPAGATION_STYLE`
// environment variables.
config.extraction_styles = {dd::PropagationStyle::W3C};
...
}
さまざまな使用例
To extract propagation context, implement a custom DictReader
interface and call Tracer::extract_span
or Tracer::extract_or_create_span
.
Here is an example of extracting propagation context from HTTP Headers:
#include <datadog/dict_reader.h>
#include <datadog/optional.h>
#include <datadog/string_view.h>
#include <unordered_map>
namespace dd = datadog::tracing;
class HTTPHeadersReader : public datadog::tracing::DictReader {
std::unordered_map<dd::StringView, dd::StringView> headers_;
public:
HTTPHeadersReader(std::unordered_map<dd::StringView, dd::StringView> headers)
: headers_(std::move(headers)) {}
~HTTPHeadersReader() override = default;
// Return the value at the specified `key`, or return `nullopt` if there
// is no value at `key`.
dd::Optional<dd::StringView> lookup(dd::StringView key) const override {
auto found = headers_.find(key);
if (found == headers_.cend()) return dd::nullopt;
return found->second;
}
// Invoke the specified `visitor` once for each key/value pair in this object.
void visit(
const std::function<void(dd::StringView key, dd::StringView value)>& visitor)
const override {
for (const auto& [key, value] : headers_) {
visitor(key, value);
}
};
};
// Usage example:
void handle_http_request(const Request& request, datadog::tracing::Tracer& tracer) {
HTTPHeadersReader reader{request.headers};
auto maybe_span = tracer.extract_span(reader);
..
}
To inject propagation context, implement the DictWriter
interface and call Span::inject
on a span instance:
#include <datadog/dict_writer.h>
#include <datadog/string_view.h>
#include <string>
#include <unordered_map>
using namespace dd = datadog::tracing;
class HTTPHeaderWriter : public dd::DictWriter {
std::unordered_map<std::string, std::string>& headers_;
public:
explicit HTTPHeaderWriter(std::unordered_map<std::string, std::string>& headers) : headers_(headers) {}
~HTTPHeaderWriter() override = default;
void set(dd::StringView key, dd::StringView value) override {
headers_.emplace(key, value);
}
};
// Usage example:
void handle_http_request(const Request& request, dd::Tracer& tracer) {
auto span = tracer.create_span();
HTTPHeaderWriter writer(request.headers);
span.inject(writer);
// `request.headers` now populated with the headers needed to propagate the span.
..
}
The Datadog .NET SDK supports the following trace context formats, including deprecated configuration values:
さまざまな使用例
- As of version [2.48.0][6], the default propagation style is
datadog, tracecontext
. This means Datadog headers are used first, followed by W3C Trace Context. - Prior to version 2.48.0, the order was
tracecontext, Datadog
for both extraction and injection propagation. - Prior to version [2.22.0][7], only the
Datadog
injection style was enabled. - As of version [2.42.0][8], when multiple extractors are specified, the
DD_TRACE_PROPAGATION_EXTRACT_FIRST=true
configuration specifies whether context extraction should exit immediately upon detecting the first valid tracecontext
. The default value is false
.
In most cases, header extraction and injection are automatic. However, there are some known cases where your distributed trace can be disconnected. For instance, when reading messages from a distributed queue, some libraries may lose the span context. It also happens if you set DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED
to false
when consuming Kafka messages. In these cases, you can add a custom trace using the following code:
var spanContextExtractor = new SpanContextExtractor();
var parentContext = spanContextExtractor.Extract(headers, (headers, key) => GetHeaderValues(headers, key));
var spanCreationSettings = new SpanCreationSettings() { Parent = parentContext };
using var scope = Tracer.Instance.StartActive("operation", spanCreationSettings);
GetHeaderValues
メソッドを提供します。このメソッドの実装方法は、SpanContext
を保持する構造に依存します。
以下はその例です。
// Confluent.Kafka
IEnumerable<string> GetHeaderValues(Headers headers, string name)
{
if (headers.TryGetLastBytes(name, out var bytes))
{
try
{
return new[] { Encoding.UTF8.GetString(bytes) };
}
catch (Exception)
{
// 無視
}
}
return Enumerable.Empty<string>();
}
// RabbitMQ
IEnumerable<string> GetHeaderValues(IDictionary<string, object> headers, string name)
{
if (headers.TryGetValue(name, out object value) && value is byte[] bytes)
{
return new[] { Encoding.UTF8.GetString(bytes) };
}
return Enumerable.Empty<string>();
}
// SQS
public static IEnumerable<string> GetHeaderValues(IDictionary<string, MessageAttributeValue> headers, string name)
{
// SQS の場合、メッセージ属性ヘッダーは最大 10 個なので、
// Datadog のヘッダーは以下のプロパティを持つ 1 つのヘッダーに統合されます。
// - Key: "_datadog"
// - Value: MessageAttributeValue object
// - DataType: "String"
// - StringValue: <JSON map with key-value headers>
if (headers.TryGetValue("_datadog", out var messageAttributeValue)
&& messageAttributeValue.StringValue is string jsonString)
{
var datadogDictionary = JsonConvert.DeserializeObject<Dictionary<string, string>>(jsonString);
if (datadogDictionary.TryGetValue(name, out string value))
{
return new[] { value };
}
}
return Enumerable.Empty<string>();
}
Kafka コンシューマースパンをトレースするために SpanContextExtractor
API を使用する場合、DD_TRACE_KAFKA_CREATE_CONSUMER_SCOPE_ENABLED
を false
に設定します。これにより、メッセージがトピックから消費された直後にコンシューマースパンが正しく閉じられ、メタデータ (partition
や offset
など) が正しく記録されることが保証されます。SpanContextExtractor
API を使用して Kafka メッセージから作成されたスパンは、プロデューサーのスパンの子であり、コンシューマーのスパンの兄弟になります。
(WCF クライアントのように自動的にインスツルメンテーションされないライブラリに対して) トレースコンテキストを手動で伝播する必要がある場合、SpanContextInjection
API を使用することができます。以下は WCF の例で、this
は WCF クライアントです。
using (OperationContextScope ocs = new OperationContextScope(this.InnerChannel))
{
var spanContextInjector = new SpanContextInjector();
spanContextInjector.Inject(OperationContext.Current.OutgoingMessageHeaders, SetHeaderValues, Tracer.Instance.ActiveScope?.Span?.Context);
}
void SetHeaderValues(MessageHeaders headers, string name, string value)
{
MessageHeader header = MessageHeader.CreateHeader(name, "datadog", value);
headers.Add(header);
}
When the Datadog SDK is configured with the Datadog format for extraction or injection (possibly both), the Datadog SDK interacts with the following request headers:
x-datadog-trace-id
- Specifies the lower 64-bits of the 128-bit trace-id, in decimal format.
x-datadog-parent-id
- Specifies the 64-bits span-id of the current span, in decimal format.
x-datadog-origin
- Specifies the Datadog product that initiated the trace, such as Real User Monitoring or Synthetic Monitoring. If this header is present, the value is expected to be one of:
rum
, synthetics
, synthetics-browser
. x-datadog-sampling-priority
- Specifies the sampling decision made for the represented span as an integer, in decimal format.
x-datadog-tags
- Specifies supplemental Datadog trace state information, including but not limited to the higher 64-bits of the 128-bit trace-id (in hexadecimal format).
When the Datadog SDK is configured with the None format for extraction or injection (possibly both), the Datadog SDK does not interact with request headers, meaning that the corresponding context propagation operation does nothing.
参考資料