If you haven’t set up custom retention filters, this is expected behavior. Here’s why:

The Trace Explorer page allows you to search all ingested or indexed spans using any tag. Here, you can query any of your traces.

By default, after spans have been ingested, they are retained by the Datadog intelligent filter. Datadog also has other retention filters that are enabled by default to give you visibility over your services, endpoints, errors, and high-latency traces.

However, to use these traces in your monitors, you must set custom retention filters.

Custom retention filters allow you to decide which spans are indexed and retained by creating, modifying, and disabling additional filters based on tags. You can also set a percentage of spans matching each filter to be retained. These indexed traces can then be used in your monitors.

Trace metrics and custom span-based metrics can have different values because they are calculated based on different datasets:

• Trace metrics are calculated based on 100% of the application’s traffic, regardless of your trace ingestion sampling configuration. The trace metrics namespace follows this format: trace.<SPAN_NAME>.<METRIC_SUFFIX>.
• Custom span-based metrics are generated based on your ingested spans, which depend on your trace ingestion sampling. For example, if you are ingesting 50% of your traces, your custom span-based metrics are based on the 50% ingested spans.

To ensure that your trace metrics and custom span-based metrics have the same value, configure a 100% ingestion rate for your application or service.

This can happen when the service name is not consistent across all spans.

For example, you might have a single service such as service:test showing multiple services in the Datadog:

• service:test
• service:test-mongodb
• service:test-postgresdb

You can use Inferred Service dependencies (beta). Inferred external APIs use the default naming scheme net.peer.name. For example: api.stripe.com, api.twilio.com, and us6.api.mailchimp.com. Inferred databases use the default naming scheme db.instance.

Or, you can merge the service names using an environment variable such as DD_SERVICE_MAPPING or DD_TRACE_SERVICE_MAPPING, depending on the language.

For more information, see Configure the Datadog Tracing Library or choose your language here:

Spikes in data ingestion and indexing can be caused by various factors. To investigate the cause of an increase, use the APM Traces Estimated Usage metrics:

The APM Traces Usage dashboard contains several widget groups displaying high-level KPIs and additional usage information.

In some traces with an error status, the Errors tab shows Missing error message and stack trace rather than exception details.

A span can show this message for two possible reasons:

• The span contains an unhandled exception.
• An HTTP response within the span returned an HTTP status code between 400 and 599.

When an exception is handled in a try/catch block, error.msg, error.type, and error.stack span tags are not populated. To populate the detailed error span tags, use Custom Instrumentation code.

インスツルメント済みのアプリケーションは、現時点から最大過去18時間および未来2時間までのタイムスタンプのスパンを送信できます。

Datadog accepts the following combinations for a given 40-minute interval:

• 1000 unique environments and service combinations
• 30 unique second primary tag values per environment
• 100 unique operation names per environment and service
• 1000 unique resources per environment, service, and operation name
• 30 unique versions per environment and service

If you need to accommodate larger volumes, contact Datadog support with your use case.

Datadog では、以下の文字列が指定された文字数を超えた場合、切り捨てられます。

また、スパンに存在するスパンタグの数が、1024 以上にならないようにしてください。

サービス数がデータ量ガイドラインで指定されている数を超える場合は、サービスの命名規則について以下のベストプラクティスを試してみてください。

サービス名から環境タグの値を除外する

デフォルトでは、環境 (env) は Datadog APM のプライマリタグになります。

サービスは通常、prod、staging、dev などの複数の環境にデプロイされます。リクエスト数、レイテンシー、エラー率などのパフォーマンスメトリクスは、さまざまな環境間で異なっています。サービスカタログの環境ドロップダウンを使用すると、Performance タブのデータを特定の環境にスコープすることができます。

サービスの数が増えすぎて問題になりがちなのが、サービス名に環境値を含めるパターンです。例えば、prod-web-store と dev-web-store のように 2 つの環境で動作しているため、1 つではなく 2 つのユニークなサービスがある場合です。

Datadog では、サービス名を変更することでインスツルメンテーションを調整することを推奨しています。

トレースメトリクスはアンサンプリングされるため、インスツルメンテーションされたアプリケーションでは、部分的なデータではなく、すべてのデータが表示されます。また、ボリュームガイドラインも適用されます。

メトリクスパーティションを置いたり、変数をサービス名にグループ化する代わりに、第 2 プライマリタグを使用する

第 2 のプライマリタグは、トレースメトリクスのグループ化および集計に使用できる追加タグです。ドロップダウンを使用して、指定されたクラスター名またはデータセンターの値にパフォーマンスデータをスコープすることができます。

第 2 のプライマリタグを適用せず、サービス名にメトリクスパーティションやグループ化変数を含めると、アカウント内のユニークなサービス数が不必要に増加し、遅延やデータ損失の可能性があります。

For example, instead of the service web-store, you might decide to name different instances of a service web-store-us-1, web-store-eu-1, and web-store-eu-2 to see performance metrics for these partitions side-by-side. Datadog recommends implementing the region value (us-1, eu-1, eu-2) as a second primary tag.

Read about how to find and fix these problems in Connection Errors.

トレースコレクションの CPU 使用率の検出と Agent の適切なリソース制限の計算については、Agent のリソース使用量を参照してください。

Datadog Agent ログで、レート制限や 1 秒あたりの最大イベント数に関するエラーメッセージが表示される場合、以下の手順に従い制限を変更します。ご不明な点は、Datadog サポートチームまでお問い合わせください。

There are several configuration options available to scrub sensitive data or discard traces corresponding to health checks or other unwanted traffic that can be configured within the Datadog Agent, or in some languages the tracing client. For details on the options available, see Security and Agent Customization. While this offers representative examples, if you require assistance applying these options to your environment, reach out to Datadog Support.

To capture full details on the Datadog tracer, enable debug mode on your tracer by using the DD_TRACE_DEBUG environment variable. You might enable it for your own investigation or if Datadog support has recommended it for triage purposes. However, be sure to disable debug logging when you are finished testing to avoid the logging overhead it introduces.

これらのログは、インスツルメンテーションエラーやインテグレーション固有のエラーを明らかにすることができます。デバッグログの有効化と取得に関する詳細は、デバッグモードのトラブルシューティングページを参照してください。

During startup, Datadog tracing libraries emit logs that reflect the configurations applied in a JSON object, as well as any errors encountered, including if the Agent can be reached in languages where this is possible. Some languages require these startup logs to be enabled with the environment variable DD_TRACE_STARTUP_LOGS=true. For more information, see the Startup logs.

When you open a support ticket, the Datadog support team may ask for the following types of information:

1. Links to a trace or screenshots of the issue: This helps reproduce your issues for troubleshooting purposes.

2. Tracer startup logs: Startup logs help identify tracer misconfiguration or communication issues between the tracer and the Datadog Agent. By comparing the tracer’s configuration with the application or container settings, support teams can pinpoint improperly applied settings.

3. Tracer debug logs: Tracer debug logs provide deeper insights than startup logs, revealing:

   • Proper integration instrumentation during application traffic flow
   • Contents of spans created by the tracer
   • Connection errors when sending spans to the Agent

4. Datadog Agent flare: Datadog Agent flares enable you to see what is happening within the Datadog Agent, for example, if traces are being rejected or malformed. This does not help if traces are not reaching the Datadog Agent, but does help identify the source of an issue, or any metric discrepancies.

5. A description of your environment: Understanding your application’s deployment configuration helps the Support team identify potential tracer-Agent communication issues and identify misconfigurations. For complex problems, support may request Kubernetes manifests, ECS task definitions, or similar deployment configuration files.

6. Custom tracing code: Custom instrumentation, configuration, and adding span tags can significantly impact trace visualizations in Datadog.

7. Version information: Knowing what language, framework, Datadog Agent, and Datadog tracer versions you are using allows Support to verify Compatiblity Requirements, check for known issues, or recommend a version upgrades. For example: