Overview

Log Pipeline Scanner allows you to scan log pipelines in real time, trace specific logs, and identify which pipelines and processing rules made changes to its fields. Organizations rely on log pipelines to process extensive log volumes, each team restructuring and enriching logs differently for their specific use cases, such as security monitoring, compliance audits, and DevOps.

Use Pipeline Scanner to:

• Troubleshoot log processing issues, such as unparsed logs, missing tags, or unexpected changes to the log structure.
• Determine and remove conflicting or redundant processing rules.
• Ensure that logs going through log pipelines are meeting security and compliance requirements.

Identify pipelines and processors modifying a log

The Pipeline Scanner samples and annotates logs matching the search query with the different processing steps they are going through, making it possible to identify all changes to your logs.

1. Navigate to Log Explorer.
2. Click on a log where you want to find out which pipelines and processors are modifying it.
3. Click the Pipeline Scanner icon in the upper right corner of the panel. If you hover over the icon, it says View pipelines for similar logs. Alternatively, click on an attribute in the log panel and select Scan pipelines for.
4. You can further refine your query in the Pipeline Scanner page. This query cannot be changed after a session is started.
5. Click Launch this session.
   For the next 15 minutes, logs matching your query are tagged with information about which pipelines and processors are modifying those logs. The Live Tail in the scanner shows which pipelines and how many pipelines match each of the logs.
6. Click a log to see the list of pipelines and processors matched to that log. Live Tail is paused at this point.

You can modify the pipelines and processors in the right side panel. The modifications you make do not affect the logs that have already been processed. Click Play to view the new logs that have been modified by the updated pipelines and processors.

You can also access Pipeline Scanner from the Log Pipelines page:

1. Navigate to Log Pipelines.
2. Click Pipeline Scanner.
3. Define the query for the logs you want to inspect.

Limitations

The logs_write_pipelines permission is required to use the Pipeline Scanner. See Log Management RBAC permissions for more information.

The number of sessions you can launch is limited to:

• A maximum of 3 concurrent sessions.
• A maximum of 12 sessions over a 12-hour sliding window.

Further reading

Additional helpful documentation, links, and articles:

Investigate your log processing with the Datadog Log Pipeline ScannerBLOG Learn more about log pipelinesDOCUMENTATION Learn more about log processorsDOCUMENTATION