Frequently Asked APM Questions

Docs > APM > APM Troubleshooting > Frequently Asked APM Questions

Overview

If you experience unexpected behavior while using Datadog APM, refer to the information on this page to help resolve the issue. If you continue to have trouble, reach out to Datadog support.

Trace retention issues

There are more spans in the Trace Explorer than on the Monitors page

If you haven’t set up custom retention filters, this is expected behavior. Here’s why:

The Trace Explorer page allows you to search all ingested or indexed spans using any tag. Here, you can query any of your traces.

By default, after spans have been ingested, they are retained by the Datadog intelligent filter. Datadog also has other retention filters that are enabled by default to give you visibility over your services, endpoints, errors, and high-latency traces.

However, to use these traces in your monitors, you must set custom retention filters.

Custom retention filters allow you to decide which spans are indexed and retained by creating, modifying, and disabling additional filters based on tags. You can also set a percentage of spans matching each filter to be retained. These indexed traces can then be used in your monitors.

PRODUCT	SPAN SOURCE
Monitors	Spans from custom retention filters
Other products (Dashboard, Notebook etc.)	Spans from custom retention filters + Datadog intelligent filter

Trace metric issues

Trace metrics and custom span-based metrics have different values

Trace metrics and custom span-based metrics can have different values because they are calculated based on different data sets:

Trace metrics are calculated based on 100% of the application’s traffic, regardless of your trace ingestion sampling configuration. The trace metrics namespace follows this format: trace.<SPAN_NAME>.<METRIC_SUFFIX>.
Custom span-based metrics are generated based on your ingested spans, which depend on your trace ingestion sampling. For example, if you are ingesting 50% of your traces, your custom span-based metrics are based on the 50% ingested spans.

To ensure that your trace metrics and custom span-based metrics have the same value, configure a 100% ingestion rate for your application or service.

Metric names must follow the metric naming convention. Metric names that start with trace.* are not permitted and are not saved.

Service issues

One service is showing up as multiple services in Datadog

This can happen when the service name is not consistent across all spans.

For example, you might have a single service like service:test showing multiple services in the Datadog:

service:test
service:test-mongodb
service:test-postgresdb

You can merge the service names using an environment variable like DD_SERVICE_MAPPING or DD_TRACE_SERVICE_MAPPING, depending on the language.

For more information, see Configure the Datadog Tracing Library or choose your language here:

dd.service.mapping: Environment Variable: DD_SERVICE_MAPPING
Default: null
Example: mysql:my-mysql-service-name-db, postgresql:my-postgres-service-name-db
Dynamically rename services via configuration. Useful for making databases have distinct names across different services.

DD_SERVICE_MAPPING: Define service name mappings to allow renaming services in traces, for example: postgres:postgresql,defaultdb:postgresql. Available in version 0.47+.

DD_SERVICE_MAPPING: Default: null
Dynamically rename services through configuration. Services can be separated by commas or spaces, for example: mysql:mysql-service-name,postgres:postgres-service-name, mysql:mysql-service-name postgres:postgres-service-name.

DD_SERVICE_MAPPING: Configuration: serviceMapping
Default: N/A
Example: mysql:my-mysql-service-name-db,pg:my-pg-service-name-db
Provide service names for each plugin. Accepts comma separated plugin:service-name pairs, with or without spaces.

DD_TRACE_SERVICE_MAPPING: Rename services using configuration. Accepts a comma-separated list of key-value pairs of service name keys to rename, and the name to use instead, in the format [from-key]:[to-name].
Example: mysql:main-mysql-db, mongodb:offsite-mongodb-service
The from-key value is specific to the integration type, and should exclude the application name prefix. For example, to rename my-application-sql-server to main-db, use sql-server:main-db. Added in version 1.23.0

DD_SERVICE_MAPPING: INI: datadog.service_mapping
Default: null
Change the default name of an APM integration. Rename one or more integrations at a time, for example: DD_SERVICE_MAPPING=pdo:payments-db,mysqli:orders-db (see Integration names).

Ruby does not support DD_SERVICE_MAPPING or DD_TRACE_SERVICE_MAPPING. See Additional Ruby configuration for code options to change the service name.

There is an unexpected increase in ingested/indexed spans on the Plan and Usage page

Spikes in data ingestion and indexing can be caused by various factors. To investigate the cause of an increase, use the APM Traces Estimated Usage metrics:

USAGE TYPE	METRIC	DESCRIPTION
APM Indexed Spans	`datadog.estimated_usage.apm.indexed_spans`	Total number of spans indexed by tag-based retention filters.
APM Ingested Spans	`datadog.estimated_usage.apm.ingested_spans`	Total number of ingested spans.

The APM Traces Usage dashboard contains several widget groups displaying high-level KPIs and additional usage information.

Data volume issues

If you encounter any of the following issues, you may be exceeding Datadog’s volume guidelines:

Your trace metrics are not reporting as you would expect in the Datadog platform.
You are missing some of your resources that you expected to see in the Datadog platform.
You are seeing traces from your service but are not able to find this service on the Service Catalog page.