Overview

This guide uses the LLM Observability SDKs for Python and Node.js. If your application is written in another language, you can create traces by calling the API instead.

Setup

Jupyter notebooks

To better understand LLM Observability terms and concepts, you can explore the examples in the LLM Observability Jupyter Notebooks repository. These notebooks provide a hands-on experience, and allow you to apply these concepts in real time.

Command line

To generate an LLM Observability trace, you can run a Python or Node.js script.

Prerequisites

• LLM Observability requires a Datadog API key. For more information, see the instructions for creating an API key.
• The following example script uses OpenAI, but you can modify it to use a different provider. To run the script as written, you need:
  • An OpenAI API key stored in your environment as OPENAI_API_KEY. To create one, see Account Setup and Set up your API key in the official OpenAI documentation.
  • The OpenAI Python library installed. See Setting up Python in the official OpenAI documentation for instructions.

Note: DD_LLMOBS_AGENTLESS_ENABLED is only required if you do not have the Datadog Agent running. If the Agent is running in your production environment, make sure this environment variable is unset.

4. View the trace of your LLM call on the Traces tab of the LLM Observability page in Datadog.

The trace you see is composed of a single LLM span. The ddtrace-run or NODE_OPTIONS="--import dd-trace/initialize.mjs" command automatically traces your LLM calls from Datadog’s list of supported integrations.

If your application consists of more elaborate prompting or complex chains or workflows involving LLMs, you can trace it using the Setup documentation and the SDK documentation.

Tracing distributed proxy or gateway services

Like any traditionally application, LLM applications can be implemented across multiple different microservices. With LLM Observability, if one of these services is a LLM proxy or gateway service, you can trace the LLM calls made by individual LLM applications in a complete end-to-end trace.

Enabling LLM Observability for a proxy or gateway service

To enable LLM Observability for a proxy or gateway service that might be called from several different ML applications, you can enable LLM Observability without specifying an ML application name.

In the proxy service, enable LLM Observability without specifying an ML application name. Optionally, you can specify a service name.

# proxy.py
from ddtrace.llmobs import LLMObs

LLMObs.enable(service="chat-proxy")

# proxy-specific logic, including guardrails, sensitive data scans, and the LLM call

// proxy.js
const tracer = require('dd-trace').init({
  llmobs: true
});
const llmobs = tracer.llmobs;

// proxy-specific logic, including guardrails, sensitive data scans, and the LLM call

In your specific applications that orchestrate the ML applications that make calls to the proxy or gateway service, enable LLM Observability with the ML application name, and wrap the proxy call in a task span:

# application.py
from ddtrace.llmobs import LLMObs
LLMObs.enable(ml_app_name="my-ml-app")

import requests

if __name__ == "__main__":
    with LLMObs.workflow(name="run-chat"):
      # other application-specific logic - RAG steps, parsing, etc.

      with LLMObs.task(name="chat-proxy"): # wrap the proxy call in a task span
        response = requests.post("http://localhost:8080/chat", json={
          # data to pass to the proxy service
        })

      # other application-specific logic handling the response

// application.js
const tracer = require('dd-trace').init({
  llmobs: {
    mlApp: 'my-ml-app'
  }
});
const llmobs = tracer.llmobs;

const axios = require('axios');

async function main () {
  llmobs.trace({ name: 'run-chat', kind: 'workflow' }, async () => {
    // other application-specific logic - RAG steps, parsing, etc.

    // wrap the proxy call in a task span
    const response = llmobs.trace({ name: 'chat-proxy', kind: 'task' }, async () => {
      return await axios.post('http://localhost:8080/chat', {
        // data to pass to the proxy service
      });
    });

    // other application-specific logic handling the response
  });
}

main();

When making requests to the proxy or gateway service, the LLM Observability SDKs automatically propagate the ML application name from the original LLM application. Additionally, the LLM Observability SDKs automatically tag the task span capturing the proxy or gateway service call with the tag ml-proxy:custom.

Observing LLM gateway and proxy services

To observe the tasks performed by a variety of ML applications in the proxy service:

1. In the LLM trace view, view All Applications from the top-left dropdown.
2. Switch to the All Spans view in the top-right dropdown.
3. Filter the list by the ml-proxy:custom tag.

The spans listed are the parent spans of the any operations executed by the ML applications. To see all spans, and not just the top-level task spans, from the proxy service, you can instead filter by the service tag:

Observing end-to-end usage of LLM applications making calls to a proxy or gateway service

To observe the complete end-to-end usage of an LLM application that makes calls to a proxy or gateway service, you can filter for traces with that ML application name:

1. In the LLM trace view, select the ML application name of interest from the top-left dropdown.
2. Switch to the Traces view in the top-right dropdown.

Further Reading

Additional helpful documentation, links, and articles:

Learn about LLM ObservabilityDOCUMENTATION