Overview

With Trace Queries, you can find entire traces based on the properties of multiple spans and the relationships between those spans within the structure of the trace. To create a trace query, you define two or more span queries and then specify the relationship within the searched-for trace structure of the spans that are returned by each span query.

You can search, filter, group, and visualize the traces from the Trace Query explorer.

With structure-based trace querying, you can answer questions such as:

• Which traces include a dependency between two services (service A has a downstream call to service B)?
• What API endpoints are affected by my erroring backend service?

Use Trace Queries to accelerate your investigations and find relevant traces.

Trace query editor

A trace query is composed of two or more span queries, joined by trace query operators.

Span queries

Query for spans from a specific environment, service, or endpoint using the Span query syntax. Use autocomplete suggestions to view facets and recent queries.

Click Add another span query to add a span query and use it in the trace query statement.

Trace query operators

Combine multiple span queries, labeled a, b, c, and so on, into a trace query in the Traces matching field, using operators between the letters that represent each span query:

Trace-level filters

Filter the result set of traces further by applying filters on trace-level attributes like the number of spans or the end-to-end duration of the trace in the Where statement:

Flow Map

The Flow Map helps you understand the request path and service dependencies from the resulting traces that match the Trace Query. Use the map to identify error paths, unusual service dependencies, or abnormally high request rates to a database.

Note: The Flow Map is powered by a sample of the ingested traffic.

Service nodes that match span queries are highlighted to show you which parts of the trace your query conditions are targeting.

To get more information about a single service, hover on the service’s node to see its metrics for request rate and error rate. To see metrics for the request rate and the error rate between two services, hover on an edge connecting the two services.

To filter out traces that do not contain a dependency on a particular service, click on the service’s node on the map.

Trace list

The Trace list shows up to fifty sample traces that match the query and are within the selected time range. Hover on the Latency Breakdown to get a sense of where (in which services) time is spent during the request execution.

Note: Information displayed in the table are attributes from the root span of the trace, including the duration, which does not represent the end-to-end duration of the trace.

Analytics

Select one of the other visualizations, such as Timeseries, Top List, or Table to aggregate results over time, grouped by one or multiple dimensions. Read Span Visualizations for more information on the aggregation options.

In addition to those aggregation options, you must also select which span query (a, b, c, and so on) you want to aggregate the spans from. Select the query that matches the spans from which you’re using the tags and attributes in the aggregation options.

For example, if you query for traces that contain a span from the service web-store (query a) and a span from the service payments-go with some errors (query b), and you visualize a count of spans grouped by @merchant.tier, use spans from query a, because merchant.tier is an attribute from the spans of the service web-store, not from the service payments-go.

How Trace Queries source data

Datadog uses the Intelligent Retention Filter to index data for Trace Queries. It does so by performing:

• Flat sampling: A uniform 1% sample of ingested spans.
• Diversity sampling: A representative, diverse selection of traces to keep visibility over each environment, service, operation, and resource.

These two sampling mechanisms capture complete traces, meaning that all spans of a trace are always indexed to ensure that Trace Queries return accurate results.

Note: Spans indexed by flat sampling and diversity sampling do not count towards the usage of indexed spans, and therefore, do not impact your bill.

1% flat sampling

retained_by:flat_sampled

Flat 1% sampling is applied based on the trace_id, meaning that all spans belonging to the same trace share the same sampling decision. To learn more, read the one percent flat sampling documentation.

Diversity sampling

retained_by:diversity_sampling

Every 15 minutes, diversity sampling retains at least one span and the associated trace for each combination of environment, service, operation, and resource. This occurs for the p75, p90, and p95 percentile of latencies to ensure that you can always find example traces in service and resource pages, even for low traffic endpoints. To learn more, read the diversity sampling documentation.

Further Reading

Additional helpful documentation, links, and articles:

Analyze the root causes and business impact of production issues with Trace QueriesBLOG Trace ExplorerDOCUMENTATION Span Query SyntaxDOCUMENTATION