Overview
In Observability Pipelines, a pipeline is a sequential path with three types of components: source, processors, and destinations. The Observability Pipeline source receives logs from your log source (for example, the Datadog Agent). The processors enrich and transform your data, and the destination is where your processed logs are sent. For some templates, your logs are sent to more than one destination. For example, if you use the Archive Logs template, your logs are sent to a cloud storage provider and another specified destination.
Set up a pipeline
Set up your pipelines and its sources, processors, and destinations in the Observability Pipelines UI. The general setup steps are:
- Navigate to Observability Pipelines.
- Select a template:
- Select and set up your source.
- Select and set up your destinations.
- Set up your processors.
- Install the Observability Pipelines Worker.
- Enable monitors for your pipeline.
See Advanced Configurations for bootstrapping options and for details on setting up the Worker with Kubernetes.
After you have set up your pipeline, see Update Existing Pipelines if you want to make any changes to it.
Clone a pipeline
- Navigate to Observability Pipelines.
- Select the pipeline you want to clone.
- Click the cog at the top right side of the page, then select Clone.
Delete a pipeline
- Navigate to Observability Pipelines.
- Select the pipeline you want to delete.
- Click the cog at the top right side of the page, then select Delete.
Note: You cannot delete an active pipeline. You must stop all Workers for a pipeline before you can delete it.
Further Reading
Additional helpful documentation, links, and articles:
