Overview
If you experience unexpected behavior with Datadog Observability Pipelines (OP), there are a few common issues you can investigate, and this guide may help resolve issues quickly. If you continue to have trouble, reach out to Datadog support for further assistance.
View Observability Pipelines Worker stats and logs
To view information about the Observability Pipelines Workers running for an active pipeline:
- Navigate to Observability Pipelines.
- Select your pipeline.
- Click the Workers tab to see the Workers’ memory and CPU utilization, traffic stats, and any errors.
- To view the Workers’ statuses and versions, click the Latest Deployment & Setup tab.
- To see the Workers’ logs, click the cog at the top right side of the page, then select View OPW Logs. See Logs Search Syntax for details on how to filter your logs. To see logs for a specific Worker, add
@op_work.id:<worker_id>
to the search query.
Inspect events sent through your pipeline to identify setup issues
If you can access your Observability Pipelines Workers locally, use the tap
command to see the raw data sent through your pipeline’s source and processors.
Enable the Observability Pipelines Worker API
The Observability Pipelines Worker API allows you to interact with the Worker’s processes with the tap
command. If you are using the Helm charts provided when you set up a pipeline, then the API has already been enabled. Otherwise, make sure the environment variable DD_OP_API_ENABLED
is set to true
in /etc/observability-pipelines-worker/bootstrap.yaml
. See Bootstrap options for more information. This sets up the API to listen on localhost
and port 8686
, which is what the CLI for tap
is expecting.
Use top
to find the component ID
You need the source’s or processor’s component ID to tap
into it. Use the top
command to find the ID of the component you want to tap
into:
observability-pipelines-worker top
Use tap
to see your data
If you are on the same host as the Worker, run the following command to tap
the output of the component:
observability-pipelines-worker tap <component_ID>
If you are using a containerized environment, use the docker exec
or kubectl exec
command to get a shell into the container to run the above tap
command.