Overview

This document explains bootstrapping for the Observability Pipelines Worker.

Bootstrap Options

Bootstrap the Observability Pipelines Worker within your infrastructure before you set up a pipeline. These environment variables are separate from the pipeline environment variables. The location of the related directories and files:

• Default data directory: var/lib/observability-pipelines-worker
• Bootstrap file: /etc/observability-pipelines-worker/bootstrap.yaml
• Environment variables file: /etc/default/observability-pipelines-worker

Note: DD_OP_DATA_DIR can only be owned by a single Observability Pipelines Worker. If you have multiple Workers, you must use unique data directories.

To set bootstrap options, do one of the following:

• Use environmental variables.
• Create a bootstrap.yaml and start the Worker instance with --bootstrap-config /path/to/bootstrap.yaml.

The following is a list of bootstrap options, their related pipeline environment variables, and which variables have a higher precedence (priority).

api

Pipeline environment variable: DD_OP_API_ENABLED

Priority: DD_OP_API_ENABLED

An example configuration:

    api:
        enabled: true
        address: "127.0.0.1:8686" # optional

Note: Setting address is optional. It is the network address to which the API should bind. If you’re running the Worker in a Docker container, bind to 0.0.0.0. Otherwise, the API is not exposed outside of the container.

Description: Enable the Observability Pipelines Worker API so you can see the Worker’s processes with the tap or top command. See Run, tap, or top the Worker for more information. If you are using the Helm charts provided when you set up a pipeline, then the API has already been enabled. Otherwise, make sure the environment variable DD_OP_API_ENABLED is set to true in /etc/observability-pipelines-worker/bootstrap.yaml, which:

• Sets up the API to listen on localhost and port 8686, which is what the CLI for tap is expecting.

• Exposes the /health endpoint. Configure a load balancer’s health check with the /health endpoint to check that the Worker is up and running.

api_key

Pipeline environment variable: DD_API_KEY

Priority: DD_API_KEY

Description: Create a Datadog API key for this environment variable. Remote Configuration must be enabled for the API key.

pipeline_id

Pipeline environment variable: DD_OP_PIPELINE_ID

Priority: DD_OP_PIPELINE_ID

Description: Create an Observability Pipelines pipeline ID for this environment variable.

site

Pipeline environment variable: DD_SITE

Priority: DD_SITE

Description: Your Datadog site (optional, default: datadoghq.com).

See Getting Started with Sites for more information.

data_dir

Pipeline environment variable: DD_OP_DATA_DIR

Priority: DD_OP_DATA_DIR

Description: The data directory (optional, default: /var/lib/observability-pipelines-worker). This is the file system directory that the Observability Pipelines Worker uses for local state.

tags: []

Pipeline environment variable: DD_OP_TAGS

Priority: DD_OP_TAGS

Description: The tags reported with internal metrics and can be used to filter Observability Pipelines instances for Remote Configuration deployments.

threads

Pipeline environment variable: DD_OP_THREADS

Priority: DD_OP_THREADS

Description: The number of threads to use for processing (optional, default: the number of available cores).

proxy

Pipeline environment variables: DD_PROXY_HTTP, DD_PROXY_HTTPS, DD_PROXY_NO_PROXY

Set proxy servers for the Observability Pipelines Worker. The proxy configuration for the Worker works in the same way as it does for the Datadog Agent.

Priority: The settings are applied to the entire Worker process. The HTTP proxy and HTTPS values are resolved in this order:
   1. DD_PROXY_HTTP(S)
   2. HTTP(S)_PROXY
   3. proxy :

An example proxy configuration:

    proxy:
        enabled: true
        https: https://foo.bar:3128

Description: The Observability Pipelines Worker can route external requests through forward proxies, such as Squid. Forward proxies forward client requests from the Observability Pipelines Worker to the internet. You might use them as a web firewall to forbid or allow certain domains, ports, or protocols. Forward proxies usually do not terminate SSL and therefore do not have access to the request content. They only pass packets back and forth between the client and the destination. HTTP tunnels are used to secure communication through a forward proxy.

Notes:

This option is available for Observability Pipelines Worker 2.1 and later.

The Observability Pipelines Worker cannot route external requests through reverse proxies, such as HAProxy and NGINX.

The DD_PROXY_HTTP(S) and HTTP(S)_PROXY environment variables need to be already exported in your environment for the Worker to resolve them. They cannot be prepended to the Worker installation script.

Further reading

추가 유용한 문서, 링크 및 기사:

Log volume control with Observability PipelinesDOCUMENTATION Dual ship logs with Observability PipelinesDOCUMENTATION Archive logs with Observability PipelinesDOCUMENTATION Split logs with Observability PipelinesDOCUMENTATION Redact data redaction with Observability PipelinesDOCUMENTATION Update existing pipelinesDOCUMENTATION