Overview

Datadog’s LLM Observability Python SDK provides integrations that automatically trace and annotate calls to LLM frameworks and libraries. Without changing your code, you can get out-of-the-box traces and observability for calls that your LLM application makes to the following frameworks:

You can programmatically enable automatic tracing of LLM calls to a supported LLM model like OpenAI or a framework like LangChain by setting integrations_enabled to true in the LLMOBs.enable() function. In addition to capturing latency and errors, the integrations capture the input parameters, input and output messages, and token usage (when available) of each traced call.

Note: When using the supported LLM Observability frameworks or libraries, no additional manual instrumentation (such as function decorators) is required to capture these calls. For custom or additional calls within your LLM application that are not automatically traced (like API calls, database queries, or internal functions), you can use function decorators to manually trace these operations and capture detailed spans for any part of your application that is not covered by auto-instrumentation.

Enabling and disabling integrations

All integrations are enabled by default.

To disable all integrations, use the in-code SDK setup and specify integrations_enabled=False.

To only enable specific integrations:

1. Use the in-code SDK setup, specifying integrations_enabled=False.
2. Manually enable the integration with ddtrace.patch() at the top of the entrypoint file of your LLM application:

from ddtrace import patch
from ddtrace.llmobs import LLMObs

LLMObs.enable(integrations_enabled=False, ...)
patch(<INTEGRATION_NAME_IN_LOWERCASE>=True)

Note: Use botocore as the name of the AWS Bedrock integration when manually enabling.

OpenAI

The OpenAI integration provides automatic tracing for the OpenAI Python SDK’s completion and chat completion endpoints.

Traced methods

The OpenAI integration instruments the following methods, including streamed calls:

• Completions:
  • OpenAI().completions.create()
  • AsyncOpenAI().completions.create()
• Chat completions:
  • OpenAI().chat.completions.create()
  • AsyncOpenAI().chat.completions.create()

LangChain

The LangChain integration provides automatic tracing for the LangChain Python SDK’s LLM, chat model, and chain calls.

Traced methods

The LangChain integration instruments the following methods:

• LLMs:
  • llm.invoke(), llm.ainvoke()
• Chat models
  • chat_model.invoke(), chat_model.ainvoke()
• Chains/LCEL
  • chain.invoke(), chain.ainvoke()
  • chain.batch(), chain.abatch()
• Embeddings
  • OpenAI : OpenAIEmbeddings.embed_documents(), OpenAIEmbeddings.embed_query()

Note: The LangChain integration does not yet support tracing streamed calls.

AWS Bedrock

The AWS Bedrock integration provides automatic tracing for the AWS Bedrock Runtime Python SDK’s chat model calls (using Boto3/Botocore).

Traced methods

The AWS Bedrock integration instruments the following methods:

• Chat messages:
  • InvokeModel
• Streamed chat messages:
  • InvokeModelWithResponseStream

Note: The AWS Bedrock integration does not yet support tracing embedding calls.

Anthropic

The Anthropic integration provides automatic tracing for the Anthropic Python SDK’s chat message calls.

Traced methods

The Anthropic integration instruments the following methods:

• Chat messages (including streamed calls):
  • Anthropic().messages.create(), AsyncAnthropic().messages.create()
• Streamed chat messages:
  • Anthropic().messages.stream(), AsyncAnthropic().messages.stream()

Further Reading

Additional helpful documentation, links, and articles:

Learn about the LLM Observability SDK for PythonDOCUMENTATION