Overview

This document goes over bootstrapping the Observability Pipelines Worker and referencing files in Kubernetes.

Bootstrap Options

Bootstrap the Observability Pipelines Worker within your infrastructure before you set up a pipeline. These environment variables are separate from the pipeline environment variables. The location of the related directories and files:

• Default data directory: var/lib/observability-pipelines-worker
• Bootstrap file: /etc/observability-pipelines-worker/bootstrap.yaml
• Environment variables file: /etc/default/observability-pipelines-worker

To set bootstrap options, do one of the following:

• Use environmental variables.
• Create a bootstrap.yaml and start the Worker instance with --bootstrap-config /path/to/bootstrap.yaml.

The following is a list of bootstrap options, their related pipeline environment variables, and which variables have a higher precedence (priority).

api_key

Pipeline environment variable: DD_API_KEY

Priority: DD_API_KEY

Create a Datadog API key for this environment variable.

pipeline_id

Pipeline environment variable: DD_OP_PIPELINE_ID

Priority: DD_OP_PIPELINE_ID

Create an Observability Pipelines pipeline ID for this environment variable.

site

Pipeline environment variable: DD_SITE

Priority: DD_SITE

Your Datadog site (optional, default: datadoghq.com).

See Getting Started with Sites for more information.

data_dir

Pipeline environment variable: DD_OP_DATA_DIR

Priority: DD_OP_DATA_DIR

The data directory (optional, default: /var/lib/observability-pipelines-worker). This is the file system directory that the Observability Pipelines Worker uses for local state.

tags: []

Pipeline environment variable: DD_OP_TAGS

Priority: DD_OP_TAGS

The tags reported with internal metrics and can be used to filter Observability Pipelines instances for Remote Configuration deployments.

threads

Pipeline environment variable: DD_OP_THREADS

Priority: DD_OP_THREADS

The number of threads to use for processing (optional, default: the number of available cores).

proxy

This option is available for Observability Pipelines Worker 2.1 and later.

Pipeline environment variables: DD_PROXY_HTTP, DD_PROXY_HTTPS, DD_PROXY_NO_PROXY

Set proxy servers for the Observability Pipelines Worker. The proxy configuration for the Worker works in the same way as it does for the Datadog Agent.

Priority: The settings are applied to the entire Worker process. The HTTP proxy and HTTPS values are resolved in this order:
   1. DD_PROXY_HTTP(S)
   2. HTTP(S)_PROXY
   3. proxy :

An example proxy configuration:

    proxy:
        enabled: true
        https: https://foo.bar:3128

Note: The DD_PROXY_HTTP(S) and HTTP(S)_PROXY environment variables need to be already exported in your environment for the Worker to resolve them. They cannot be prepended to the Worker installation script.

Referencing files in Kubernetes

If you are referencing files in Kubernetes for Google Cloud Storage authentication, TLS certificates for certain sources, or an enrichment table processor, you need to use volumeMounts[*].subPath to mount files from a configMap or secret.

For example, if you have a secret defined as:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  credentials1.json: bXktc2VjcmV0LTE=
  credentials2.json: bXktc2VjcmV0LTI=

Then you need to override extraVolumes and extraVolumeMounts in the values.yaml file to mount the secret files to Observability Pipelines Worker pods using subPath:

# extraVolumes -- Specify additional Volumes to use.
extraVolumes:
  - name: my-secret-volume
    secret:
      secretName: my-secret

# extraVolumeMounts -- Specify Additional VolumeMounts to use.
extraVolumeMounts:
  - name: my-secret-volume
    mountPath: /var/lib/observability-pipelines-worker/config/credentials1.json
    subPath: credentials1.json
  - name: my-secret-volume
    mountPath: /var/lib/observability-pipelines-worker/config/credentials2.json
    subPath: credentials2.json

Note: If you override thedatadog.dataDir parameter, you need to override the mountPath as well.

Further reading

추가 유용한 문서, 링크 및 기사:

Log volume control with Observability PipelinesDOCUMENTATION Dual ship logs with Observability PipelinesDOCUMENTATION Archive logs with Observability PipelinesDOCUMENTATION Split logs with Observability PipelinesDOCUMENTATION Redact data redaction with Observability PipelinesDOCUMENTATION Update existing pipelinesDOCUMENTATION