- 필수 기능
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- 디지털 경험
- 소프트웨어 제공
- 보안
- 로그 관리
- 관리
- 인프라스트럭처
- ci
- containers
- csm
- ndm
- otel_guides
- overview
- slos
- synthetics
- tests
- 워크플로
In the context of LLM applications, it’s important to track user feedback and evaluate the quality of your LLM application’s responses. While LLM Observability provides a few out-of-the-box evaluations for your traces, you can submit your own evaluations to LLM Observability in two ways: with Datadog’s Python SDK, or with the LLM Observability API. See Naming custom metrics for guidelines on how to choose an appropriate label for your evaluations.
To submit evaluations from your traced LLM application to Datadog, you’ll need to associate it with a span using the below steps:
LLMObs.export_span(span)
. If span
is not provided (as when using function decorators), the SDK exports the current active span. See Exporting a span for more details.LLMObs.submit_evaluation()
with the extracted span context and evaluation information. See Submitting evaluations in the SDK documentation for details.from ddtrace.llmobs import LLMObs
from ddtrace.llmobs.decorators import llm
@llm(model_name="claude", name="invoke_llm", model_provider="anthropic")
def llm_call():
completion = ... # user application logic to invoke LLM
span_context = LLMObs.export_span(span=None)
LLMObs.submit_evaluation(
span_context,
label="sentiment",
metric_type="score",
value=10,
)
return completion
You can use the evaluations API provided by LLM Observability to send evaluations associated with spans to Datadog. See the Evaluations API for more details on the API specifications.
{
"data": {
"type": "evaluation_metric",
"attributes": {
"metrics": [
{
"span_id": "61399242116139924211",
"trace_id": "13932955089405749200",
"timestamp": 1609459200,
"metric_type": "categorical",
"label": "Sentiment",
"categorical_value": "Positive"
},
{
"span_id": "20245611112024561111",
"trace_id": "13932955089405749200",
"metric_type": "score",
"label": "Accuracy",
"score_value": 3
}
]
}
}
}