Overview

Observability Pipelines allows you to collect and process logs within your own infrastructure, before routing them to downstream integrations. Use out-of-the-box templates to build and deploy pipelines based on your use case.

The Observability Pipelines Worker is the software that runs in your infrastructure. It centrally aggregates, processes, and routes your logs based on your use case. This means you can redact sensitive data, pre-process logs, and determine which destinations they should go to, all before the logs leave your environment.

The Observability Pipelines UI provides a control plane to manage your Observability Pipelines Workers. You build and edit pipelines and deploy pipeline changes to your Workers from there. You can also enable out-of-the-box monitors for your pipelines so that you can evaluate their health.

Explore Observability Pipelines

Build pipelines with out-of-the-box templates

Build pipelines with out-of-the-box templates for the following use cases:

• Archive Logs
• Dual Ship Logs
• Generate Metrics
• Log Enrichment
• Log Volume Control
• Sensitive Data Redaction
• Split Logs

Build pipelines in the Observability Pipelines UI

Build your pipelines in the Observability Pipelines UI. After you select one of the out-the-box templates, the onboarding workflow walks you through setting up your source, processors, and destinations. The installation page provides instructions on how to install the Worker in your environment (Docker, Kubernetes, Linux, or CloudFormation).

Enable out-of-the-box monitors for your pipeline components

After you create your pipeline, enable out-of-the box monitors to get alerted when:

• There are increasing error rates for a component. This could happen because the component is processing data in unexpected formats.
• The Observability Pipelines Worker has high CPU usage or memory usage.
• There are spikes in data dropped by a component.

Get started

You must enable Remote Configuration to use Observability Pipelines.

See Set Up Pipelines to set up a pipeline in the UI. You can also set up pipelines using the Observability Pipelines API or Terraform.

See Advanced Configurations for bootstrapping options and for details on setting up the Worker with Kubernetes.

Further Reading

Documentation, liens et articles supplémentaires utiles:

Log collection and integrationsDOCUMENTATION Log Management data securityDOCUMENTATION Sensitive Data ScannerDOCUMENTATION Dual shipping with Observability PipelinesDOCUMENTATION Strategies for Reducing Log VolumeDOCUMENTATION Redact sensitive data from your logs on-prem by using Observability PipelinesBLOG Dual ship logs with Datadog Observability PipelinesBLOG Control your log volumes with Datadog Observability PipelinesBLOG Archive your logs with Observability Pipelines for a simple and affordable migration to DatadogBLOG Aggregate, process, and route logs easily with Datadog Observability PipelinesBLOG Stream logs in the OCSF format to your preferred security vendors or data lakes with Observability PipelinesBLOG Simplify your SIEM migration to Microsoft Sentinel with Datadog Observability PipelinesBLOG How state, local, and education organizations can manage logs flexibly and efficiently using Datadog Observability PipelinesBLOG How to optimize high-volume log data without compromising visibilityBLOG Search your historical logs more efficiently with Datadog Archive SearchBLOG