If you have not read the setup instructions for automatic instrumentation, start with the Python Setup Instructions

If you aren’t using supported library instrumentation (see library compatibility), you may want to manually instrument your code.

You may also want to extend the functionality of the ddtrace library or gain finer control over instrumenting your application. Several techniques are provided by the library to accomplish this.

Creating spans

The ddtrace library creates spans automatically with ddtrace-run for many libraries and frameworks. However, you may want to gain visibility into your own code and this is achieved by using spans.

Within your web request (for example, make_sandwich_request), you may perform several operations, like get_ingredients() and assemble_sandwich(), which are useful to measure.

def make_sandwich_request(request):
    ingredients = get_ingredients()
    sandwich = assemble_sandwich(ingredients)

  from ddtrace import tracer

  @tracer.wrap(service="my-sandwich-making-svc", resource="resource_name")
  def get_ingredients():
      # go to the pantry
      # go to the fridge
      # maybe go to the store
      return

  # You can provide more information to customize the span
  @tracer.wrap("assemble_sandwich", service="my-sandwich-making-svc", resource="resource_name")
  def assemble_sandwich(ingredients):
      return

from ddtrace import tracer

def make_sandwich_request(request):
    # Capture both operations in a span
    with tracer.trace("sandwich.make"):
        ingredients = get_ingredients()
        sandwich = assemble_sandwich(ingredients)

def make_sandwich_request(request):
    # Capture both operations in a span
    with tracer.trace("sandwich.create", resource="resource_name") as outer_span:

        with tracer.trace("get_ingredients", resource="resource_name") as span:
            ingredients = get_ingredients()

        with tracer.trace("assemble_sandwich", resource="resource_name") as span:
            sandwich = assemble_sandwich(ingredients)

def make_sandwich_request(request):
    span = tracer.trace("sandwich.create", resource="resource_name")
    ingredients = get_ingredients()
    sandwich = assemble_sandwich(ingredients)
    span.finish()  # remember to finish the span

Accessing active spans

The built-in instrumentation and your own custom instrumentation create spans around meaningful operations. You can access the active span in order to include meaningful data.

from ddtrace import tracer

def make_sandwich_request(request):
    # Capture both operations in a span
    with tracer.trace("sandwich.make") as my_span:
        ingredients = get_ingredients()
        sandwich = assemble_sandwich(ingredients)

def get_ingredients():
    # Get the active span
    span = tracer.current_span()
    # this span is my_span from make_sandwich_request above

def assemble_sandwich(ingredients):
    with tracer.trace("another.operation") as another_span:
        # Get the active root span
        span = tracer.current_root_span()
        # this span is my_span from make_sandwich_request above

Adding tags

from ddtrace import tracer

def make_sandwich_request(request):
    with tracer.trace("sandwich.make") as span:
        ingredients = get_ingredients()
        span.set_tag("num_ingredients", len(ingredients))

from ddtrace import tracer
from myapp import __version__

# This will be applied to every span
tracer.set_tags({"version": __version__, "<TAG_KEY_2>": "<TAG_VALUE_2>"})

from ddtrace import tracer

with tracer.trace("throws.an.error") as span:
    raise Exception("Oops!")

# `span` will be flagged as erroneous and have
# the stack trace and exception message attached as tags

from ddtrace import tracer

span = tracer.trace("operation")
span.error = 1
span.finish()

import os
from ddtrace import tracer

try:
    raise TypeError
except TypeError as e:
    root_span = tracer.current_root_span()
    (exc_type, exc_val, exc_tb) = sys.exc_info()
    # this sets the error type, marks the span as an error, and adds the traceback
    root_span.set_exc_info(exc_type, exc_val, exc_tb)

Propagating context with headers extraction and injection

You can configure the propagation of context for distributed traces by injecting and extracting headers. Read Trace Context Propagation for information.

Baggage

Manipulating Baggage on a span:

from ddtrace import tracer

# Start a new span and set baggage
with tracer.trace("example") as span:
    # set_baggage_item
    span.context.set_baggage_item("key1", "value1")
    span.context.set_baggage_item("key2", "value2")

    # get_all_baggage_items
    all_baggage = span.context.get_all_baggage_items()
    print(all_baggage) # {'key1': 'value1', 'key2': 'value2'}

    # remove_baggage_item
    span.context.remove_baggage_item("key1")
    print(span.context.get_all_baggage_items()) # {'key2': 'value2'}

    # get_baggage_item
    print(span.context.get_baggage_item("key1")) # None
    print(span.context.get_baggage_item("key2")) # value2

    # remove_all_baggage_items
    span.context.remove_all_baggage_items()
    print(span.context.get_all_baggage_items()) # {}

To see an example in action, see flask-baggage on trace-examples

ddtrace-api

The ddtrace-api package provides a stable public API for Datadog APM’s custom Python instrumentation. This package implements only the API interface, not the underlying functionality that creates and sends spans to Datadog.

This separation between interface (ddtrace-api) and implementation (ddtrace) offers several benefits:

• You can rely on an API that changes less frequently and more predictably for your custom instrumentation
• If you only use automatic instrumentation, you can ignore API changes entirely
• If you implement both single-step and custom instrumentation, you avoid depending on multiple copies of the ddtrace package

To use ddtrace-api:

1. Install both the ddtrace and ddtrace-api libraries:

   pip install 'ddtrace>=3.1' ddtrace-api
   

2. Instrument your Python application using ddtrace-run by prefixing your Python entry-point command:

   ddtrace-run python app.py
   

   Copy

3. After this is set up, you can write custom instrumentation exactly like the examples in the previous sections, but you import from ddtrace_api instead of ddtrace.

   For example:

   from ddtrace_api import tracer
   
   @tracer.wrap(service="my-sandwich-making-svc", resource="resource_name")
   def get_ingredients():
       # go to the pantry
       # go to the fridge
       # maybe go to the store
       return
   

See that package’s API definition for the full list of supported API calls.

Further Reading

Additional helpful documentation, links, and articles:

Connect your Logs and Traces togetherDOCUMENTATION Explore your services, resources, and tracesDOCUMENTATION