This guide explains how to build a Datadog Agent image with additional OpenTelemetry components not included in the default Datadog Agent. To see a list of components already included in the Agent by default, see Included components.

Prerequisites

To complete this guide, you need the following:

• Docker
• GitHub and access to the Datadog Agent source code.
• The OpenTelemetry components you plan to include in the Agent must be compatible with embedded OpenTelemetry Collector version.

Recommended:

• Familiarity with building a custom collector and OpenTelemetry Collector Builder (OCB).
• Basic understanding of the Go compilation process and Go modules.

Download the Dockerfile

Download the Dockerfile template:

1. Go to your preferred file location in a terminal. Run the following commands to create a new folder (for example, named agent-with-otel) and cd into it.

   mkdir -p agent-with-otel
   cd agent-with-otel
   

2. Download the Dockerfile

   curl -o Dockerfile https://raw.githubusercontent.com/DataDog/datadog-agent/refs/tags/7.62.2/Dockerfiles/agent-ot/Dockerfile.agent-otel
   

The Dockerfile:

• Creates a multi-stage build with Ubuntu 24.04 and datadog/agent:7.62.2-ot-beta-jmx.
• Installs Go, Python, and necessary dependencies.
• Downloads and unpacks the Datadog Agent source code.
• Creates a virtual environment and installs required Python packages.
• Builds the OpenTelemetry agent and copies the resulting binary to the final image.

Create an OpenTelemetry Collector Builder manifest

Create and customize an OpenTelemetry Collector Builder (OCB) manifest file, which defines the components to be included in your custom Datadog Agent.

1. Download the Datadog default manifest:

   curl -o manifest.yaml https://raw.githubusercontent.com/DataDog/datadog-agent/refs/tags/7.62.2/comp/otelcol/collector-contrib/impl/manifest.yaml
   

2. Open the manifest.yaml file and add the additional OpenTelemetry components to the corresponding sections (extensions, exporters, processors, receivers, or connectors). The highlighted line in this example adds a metrics transform processor:

   dist:
     module: github.com/DataDog/comp/otelcol/collector-contrib
     name: otelcol-contrib
     description: Datadog OpenTelemetry Collector
     version: 0.115.0
     output_path: ./comp/otelcol/collector-contrib/impl
     otelcol_version: 0.115.0
   
   extensions:
   # You will see a list of extensions already included by Datadog
   # Add your desired extensions here
   
   exporters:
   # You will see a list of exporters already included by Datadog
   # Add your desired exporters here
   
   processors:
   # adding metrics transform processor to modify metrics
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/metricstransformprocessor v0.115.0
   
   receivers:
     - gomod: go.opentelemetry.io/collector/receiver/nopreceiver v0.115.0
     - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/receivercreator v0.115.0
     - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.115.0
   
   connectors:
   # You will see a list of connectors already included by Datadog
   # Add your desired connectors here

3. Save your changes to the manifest file.

Build and push the Agent image

The custom Agent image you build needs to be stored in your organization’s private container registry for your clusters to access it. Additionally, this build process must be repeated each time you update the Agent version to maintain compatibility with new Agent releases.

Build your custom Datadog Agent image and push it to a container registry.

1. Build the image with Docker:

   docker build . -t agent-otel --no-cache \
     --build-arg AGENT_VERSION="7.62.2-ot-beta-jmx" \
     --build-arg AGENT_BRANCH="7.62.x"
   

2. Tag and push the image:

   docker tag agent-otel <IMAGE-NAME>/<IMAGE-TAG>
   docker push <IMAGE-NAME>/<IMAGE-TAG>
   

   Replace <IMAGE-NAME> and <IMAGE-TAG> with your image name and desired tag. If the target repository is not Docker Hub, you need to include the repository name.
3. For a Helm chart installation, set the image tag in your values file:

   datadog-values.yaml

   Copy

   agents:
     image:
       repository: <YOUR-REPO>
       tag: <IMAGE-TAG>
       doNotCheckTag: true
      

   Replace <YOUR-REPO> and <IMAGE-TAG> with your repository name and desired image tag.

Test and validate

Create a sample configuration file and run your custom Agent to ensure everything is working correctly.

1. Create a sample OpenTelemetry configuration file with the additional components. The following example configures an additional metrics transform processor:

   receivers:
     otlp:
       protocols:
         http:
           endpoint: "0.0.0.0:4318"
         grpc:
           endpoint: "0.0.0.0:4317"
   
   processors:
     batch:
       send_batch_max_size: 1000
       send_batch_size: 100
       timeout: 10s
     # Rename system.cpu.usage to system.cpu.usage_time
     metricstransform:
       transforms:
         - include: system.cpu.usage
           action: update
           new_name: system.cpu.usage_time
   
   exporters:
     datadog:
       api:
         site: ${env:DD_SITE}
         key: ${env:DD_API_KEY}
   
   connectors:
       datadog/connector:
           traces:
             compute_top_level_by_span_kind: true
             peer_tags_aggregation: true
             compute_stats_by_span_kind: true
   
   service:
     pipelines:
       metrics:
         receivers: [otlp, datadog/connector]
         processors: [metricstransform, batch]
         exporters: [datadog]
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [datadog/connector]
       traces/2:
         receivers: [datadog/connector]
         processors: [batch]
         exporters: [datadog]
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [datadog]
   

2. Run the Agent using the following Docker command.

   docker run -it \
     -e DD_API_KEY=XX \
     -e DD_SITE=datadoghq.com \
     -e DD_HOSTNAME=datadog \
     -v $(pwd)/config.yaml:/config.yaml \
     -p 4317:4317 \
     -p 4318:4318 \
     --entrypoint otel-agent \
     agent-otel --config /config.yaml
   

3. If the Agent starts, then the build process was successful.

You can now use this new image to install the Agent. This enables Datadog monitoring capabilities along with the additional OpenTelemetry components you’ve added.

For detailed instructions on installing and configuring the Agent with added OpenTelemetry components, see the Install the Datadog Agent with Embedded OpenTelemetry Collector guide.

Troubleshooting

This section discusses some common issues you might encounter while building and running your custom Datadog Agent, along with their solutions:

Compatibility issues with awscontainerinsightreceiver

Problem: You may encounter errors related to awscontainerinsightreceiver during the build:

#0 0.879 go: downloading github.com/tidwall/gjson v1.17.1
#0 0.889 go: downloading code.cloudfoundry.org/go-diodes v0.0.0-20240604201846-c756bfed2ed3
#0 0.916 go: downloading github.com/hashicorp/go-retryablehttp v0.7.5
#0 0.940 go: downloading github.com/tidwall/pretty v1.2.1
#0 88.24 # github.com/opencontainers/runc/libcontainer/cgroups/ebpf
#0 88.24 /go/pkg/mod/github.com/opencontainers/runc@v1.1.12/libcontainer/cgroups/ebpf/ebpf_linux.go:190:3: unknown field Replace in struct literal of type link.RawAttachProgramOptions
#0 89.14 # github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver/internal/k8sapiserver
#0 89.14 /go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver@v0.115.0/internal/k8sapiserver/k8sapiserver.go:47:68: undefined: record.EventRecorderLogger
------

Solution: Remove awscontainerinsightreceiver from the manifest.yaml file. This receiver has incompatible libraries and cannot be included in the build.

Build process failures

Problem: You receive the following error:

ERROR: failed to solve: process "/bin/sh -c . venv/bin/activate && invoke otel-agent.build" did not complete successfully: chown /var/lib/docker/overlay2/r75bx8o94uz6t7yr3ae6gop0b/work/work: no such file or directory

Solution: Run the build command again:

docker build . -t agent-otel --no-cache

Insufficient disk space

Problem: You may encounter errors related to insufficient disk space, such as:

no space left on device

Solution: Clear up Docker space:

docker system prune -a

Further reading

Additional helpful documentation, links, and articles:

Use Custom OpenTelemetry Components with Datadog AgentDOCUMENTATION