OpenTelemetry

Supported OS Linux Windows Mac OS

OpenTelemetry Collector

Overview

Important: OpenTelemetry Collector Contrib v0.95.0 introduces a breaking change that disables Trace Metrics computation in the Datadog Exporter. Follow Datadog's migration guide when upgrading.

OpenTelemetry is a vendor-agnostic standard for telemetry data. Datadog supports ingesting OpenTelemetry data through the OpenTelemetry Collector and the Datadog Agent. This tile documents how to export data to Datadog through the OpenTelemetry Collector with Datadog Exporter OpenTelemetry collector Datadog exporter. Also see OTLP ingest in Datadog Agent for further information on ingesting OTLP traces with Datadog Agent.

The OpenTelemetry Collector is a vendor-agnostic agent process that, through the Datadog exporter, exports telemetry data directly to Datadog servers (no Agent installation required). It reports metrics and traces from instrumented applications as well as general system metrics.

Host metrics are shown in the OpenTelemetry Host Metrics default dashboard, but you can send arbitrary metrics to Datadog using the OpenTelemetry Collector. Metrics under system.* and process.*, such as those generated by the host metrics receiver, are renamed to otel.system.* and otel.process.* to prevent collisions with metrics from the Datadog Agent. Additionally, OpenTelemetry Collector metrics are shown in the OpenTelemetry Collector Metrics default dashboard.

Setup

Installation

Follow the OpenTelemetry Collector documentation to install the opentelemetry-collector-contrib distribution, or any other distribution that includes the Datadog Exporter.

The Datadog Agent is not needed to export telemetry data to Datadog in this setup. See OTLP Ingest in Datadog Agent if you want to use the Datadog Agent instead.

Configuration

To export telemetry data to Datadog from the OpenTelemetry Collector, add the Datadog exporter to your metrics and traces pipelines. The only required setting is your API key.

A minimal configuration file to retrieve system metrics is as follows.

receivers:
  hostmetrics:
    scrapers:
      load:
      cpu:
      disk:
      filesystem:
      memory:
      network:
      paging:
      process:

processors:
  batch:
    timeout: 10s

exporters:
  datadog:
    api:
      key: "<Your API key goes here>"
      
service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [batch]
      exporters: [datadog]

For further information on the Datadog exporter settings and how to configure the pipeline, see Datadog exporter for OpenTelemetry Collector.

See the Metrics section for metrics types and metadata.csv for a list of metrics provided by this check. If you’re using the hostmetrics receiver as in the sample configuration above. You can send arbitrary metrics with other OpenTelemetry Collector components.

Different groups of metrics can be enabled and customized by following the hostmetrics receiver instructions. CPU and disk metrics are not available on macOS.

Validation

Check the OpenTelemetry Collector logs to see the Datadog exporter being enabled and started correctly. For example, with the configuration above you should find logging messages similar to the following.

Exporter is enabled.	{"component_kind": "exporter", "exporter": "datadog"}
Exporter is starting...	{"component_kind": "exporter", "component_type": "datadog", "component_name": "datadog"}
Exporter started.	{"component_kind": "exporter", "component_type": "datadog", "component_name": "datadog"}
Everything is ready. Begin running and processing data.

Data Collected

Metrics

otel.system.cpu.time
(count)
Total CPU seconds broken down by different states
Shown as second
otel.system.cpu.load_average.1m
(gauge)
Average CPU Load over 1 minute
Shown as unit
otel.system.cpu.load_average.5m
(gauge)
Average CPU Load over 5 minutes
Shown as unit
otel.system.cpu.load_average.15m
(gauge)
Average CPU Load over 15 minutes
Shown as unit
otel.system.memory.usage
(count)
Bytes of memory in use
Shown as byte
otel.system.disk.io
(count)
Disk bytes transferred
Shown as byte
otel.system.disk.operations
(count)
Disk operations count
Shown as unit
otel.system.disk.io_time
(count)
Time disk spent activated
Shown as unit
otel.system.disk.operation_time
(count)
Time spent in disk operations
Shown as second
otel.system.disk.merged
(count)
The number of disk reads merged into single physical disk access operations.
Shown as request
otel.system.disk.pending_operations
(gauge)
The queue size of pending I/O operations.
Shown as unit
otel.system.filesystem.usage
(gauge)
Filesystem bytes used.
Shown as byte
otel.system.filesystem.inodes.usage
(gauge)
Filesystem inodes used
Shown as inode
otel.system.network.dropped
(count)
Packets that are dropped or discarded even though there was no error
Shown as packet
otel.system.network.packets
(count)
The number of packets transferred
Shown as packet
otel.system.network.errors
(gauge)
The number of errors encountered
Shown as error
otel.system.network.io
(count)
The number of bytes transmitted and received
Shown as byte
otel.system.network.connections
(gauge)
The number of connections
Shown as connection
otel.system.processes.count
(gauge)
Total number of processes in each state
Shown as process
otel.system.processes.created
(count)
Total number of created processes
Shown as process
otel.process.cpu.time
(count)
Total CPU seconds broken down by different states.
Shown as second
otel.process.memory.physical_usage
(count)
The amount of physical memory in use
Shown as byte
otel.process.memory.virtual_usage
(gauge)
Virtual memory size
Shown as byte
otel.process.disk.io
(count)
Disk bytes transferred per second
Shown as byte
otel.system.paging.usage
(gauge)
Unix swap or windows pagefile usage
Shown as byte
otel.system.paging.faults
(count)
The number of paging faults
Shown as occurrence
otel.system.paging.operations
(count)
The number of paging operations
Shown as unit
otel.datadog_exporter.metrics.running
(gauge)
Reports that metrics are being sent from a host through the Datadog exporter for the OpenTelemetry Collector
Shown as unit
otel.datadog_exporter.traces.running
(gauge)
Reports that traces are being sent from a host through the Datadog exporter for the OpenTelemetry Collector
Shown as unit
otelcol_process_uptime
(count)
Uptime of the process
Shown as unit
otelcol_process_memory_rss
(gauge)
Total physical memory (resident set size)
Shown as unit
otelcol_exporter_queue_size
(gauge)
Current size of the retry queue (in batches)
Shown as unit
otelcol_exporter_sent_spans
(count)
Number of spans successfully sent to destination
Shown as unit
otelcol_exporter_send_failed_metric_points
(count)
Number of metric points in failed attempts to send to destination
Shown as unit
otelcol_exporter_send_failed_spans
(count)
Number of spans in failed attempts to send to destination
Shown as unit
otelcol_process_cpu_seconds
(count)
Total CPU user and system time in seconds
Shown as unit
otelcol_receiver_refused_spans
(count)
Number of spans that could not be pushed into the pipeline
Shown as unit
otelcol_exporter_queue_capacity
(gauge)
Fixed capacity of the retry queue (in batches)
Shown as unit
otelcol_receiver_accepted_spans
(count)
Number of spans successfully pushed into the pipeline
Shown as unit
otelcol_exporter_sent_metric_points
(count)
Number of metric points successfully sent to destination
Shown as unit
otelcol_exporter_enqueue_failed_spans
(count)
Number of spans failed to be added to the sending queue
Shown as unit
otelcol_scraper_errored_metric_points
(count)
Number of metric points that were unable to be scraped
Shown as unit
otelcol_scraper_scraped_metric_points
(count)
Number of metric points successfully scraped
Shown as unit
otelcol_receiver_refused_metric_points
(count)
Number of metric points that could not be pushed into the pipeline
Shown as unit
otelcol_receiver_accepted_metric_points
(count)
Number of metric points successfully pushed into the pipeline
Shown as unit
otelcol_process_runtime_heap_alloc_bytes
(gauge)
Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
Shown as byte
otelcol_process_runtime_total_alloc_bytes
(count)
Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
Shown as byte
otelcol_exporter_enqueue_failed_log_records
(count)
Number of log records failed to be added to the sending queue
Shown as unit
otelcol_processor_batch_timeout_trigger_send
(count)
Number of times the batch was sent due to a timeout trigger
Shown as unit
otelcol_exporter_enqueue_failed_metric_points
(count)
Number of metric points failed to be added to the sending queue
Shown as unit
otelcol_process_runtime_total_sys_memory_bytes
(gauge)
Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
Shown as byte
otelcol_processor_batch_batch_size_trigger_send
(count)
Number of times the batch was sent due to a size trigger
Shown as unit
otelcol_exporter_sent_log_records
(count)
Number of log records successfully sent to destination
Shown as unit
otelcol_receiver_refused_log_records
(count)
Number of log records that could not be pushed into the pipeline
Shown as unit
otelcol_receiver_accepted_log_records
(count)
Number of log records successfully pushed into the pipeline
Shown as unit
runtime.uptime
(count)
Milliseconds since application was initialized
Shown as millisecond
process.runtime.go.goroutines
(gauge)
Number of goroutines that currently exist
Shown as invocation
process.runtime.go.cgo.calls
(gauge)
Number of cgo calls made by the current process
Shown as invocation
process.runtime.go.mem.heap_alloc
(gauge)
Bytes of allocated heap objects
Shown as byte
process.runtime.go.mem.heap_idle
(gauge)
Bytes in idle (unused) spans
Shown as byte
process.runtime.go.mem.heap_inuse
(gauge)
Bytes in in-use spans
Shown as byte
process.runtime.go.mem.heap_objects
(gauge)
Number of allocated heap objects
Shown as object
process.runtime.go.mem.heap_released
(gauge)
Bytes of idle spans whose physical memory has been returned to the OS
Shown as byte
process.runtime.go.mem.heap_sys
(gauge)
Bytes of heap memory obtained from the OS
Shown as byte
process.runtime.go.mem.lookups
(count)
Number of pointer lookups performed by the runtime
Shown as unit
process.runtime.go.mem.live_objects
(gauge)
Number of live objects is the number of cumulative Mallocs - Frees
Shown as object
process.runtime.go.gc.count
(count)
Number of completed garbage collection cycles
Shown as garbage collection
process.runtime.go.gc.pause_total_ns
(count)
Cumulative nanoseconds in GC stop-the-world pauses since the program started
Shown as nanosecond
otel.process.runtime.go.goroutines
(gauge)
Number of goroutines that currently exist
Shown as invocation
otel.process.runtime.go.cgo.calls
(gauge)
Number of cgo calls made by the current process
Shown as invocation
otel.process.runtime.go.mem.heap_alloc
(gauge)
Bytes of allocated heap objects
Shown as byte
otel.process.runtime.go.mem.heap_idle
(gauge)
Bytes in idle (unused) spans
Shown as byte
otel.process.runtime.go.mem.heap_inuse
(gauge)
Bytes in in-use spans
Shown as byte
otel.process.runtime.go.mem.heap_objects
(gauge)
Number of allocated heap objects
Shown as object
otel.process.runtime.go.mem.heap_released
(gauge)
Bytes of idle spans whose physical memory has been returned to the OS
Shown as byte
otel.process.runtime.go.mem.heap_sys
(gauge)
Bytes of heap memory obtained from the OS
Shown as byte
otel.process.runtime.go.mem.lookups
(count)
Number of pointer lookups performed by the runtime
Shown as unit
otel.process.runtime.go.mem.live_objects
(gauge)
Number of live objects is the number of cumulative Mallocs - Frees
Shown as object
otel.process.runtime.go.gc.count
(count)
Number of completed garbage collection cycles
Shown as garbage collection
otel.process.runtime.go.gc.pause_total_ns
(count)
Cumulative nanoseconds in GC stop-the-world pauses since the program started
Shown as nanosecond
process.runtime.jvm.memory.usage_after_last_gc
(gauge)
Measure of memory used as measured after the most recent garbage collection event on this pool
Shown as byte
process.runtime.jvm.classes.loaded
(count)
Number of classes loaded since JVM start
Shown as unit
process.runtime.jvm.classes.unloaded
(count)
Number of classes unloaded since JVM start
Shown as unit
process.runtime.jvm.classes.current_loaded
(gauge)
Number of classes currently loaded
Shown as unit
process.runtime.jvm.cpu.utilization
(gauge)
Recent CPU utilization for the process
Shown as unit
process.runtime.jvm.system.cpu.utilization
(gauge)
Recent CPU utilization for the whole system
Shown as unit
process.runtime.jvm.system.cpu.load_1m
(gauge)
Average CPU load of the whole system for the last minute
Shown as unit
process.runtime.jvm.buffer.usage
(gauge)
Measure of memory used by buffers
Shown as byte
process.runtime.jvm.buffer.limit
(gauge)
Measure of total memory capacity of buffers
Shown as byte
process.runtime.jvm.buffer.count
(gauge)
Number of buffers in the pool
Shown as buffer
process.runtime.jvm.memory.usage
(gauge)
The total Java memory used.
Shown as byte
process.runtime.jvm.memory.committed
(gauge)
The total Java memory committed to be used.
Shown as byte
process.runtime.jvm.memory.init
(gauge)
The initial Java memory allocated.
Shown as byte
process.runtime.jvm.memory.limit
(gauge)
The maximum Java memory available.
Shown as byte
process.runtime.jvm.threads.count
(count)
The number of live threads.
Shown as thread
process.runtime.jvm.gc.duration
(gauge)
The approximate accumulated garbage collection time elapsed.
Shown as millisecond
otel.process.runtime.jvm.memory.usage_after_last_gc
(gauge)
Measure of memory used as measured after the most recent garbage collection event on this pool
Shown as byte
otel.process.runtime.jvm.classes.loaded
(count)
Number of classes loaded since JVM start
Shown as unit
otel.process.runtime.jvm.classes.unloaded
(count)
Number of classes unloaded since JVM start
Shown as unit
otel.process.runtime.jvm.classes.current_loaded
(gauge)
Number of classes currently loaded
Shown as unit
otel.process.runtime.jvm.cpu.utilization
(gauge)
Recent CPU utilization for the process
Shown as unit
otel.process.runtime.jvm.system.cpu.utilization
(gauge)
Recent CPU utilization for the whole system
Shown as unit
otel.process.runtime.jvm.system.cpu.load_1m
(gauge)
Average CPU load of the whole system for the last minute
Shown as unit
otel.process.runtime.jvm.buffer.usage
(gauge)
Measure of memory used by buffers
Shown as byte
otel.process.runtime.jvm.buffer.limit
(gauge)
Measure of total memory capacity of buffers
Shown as byte
otel.process.runtime.jvm.buffer.count
(gauge)
Number of buffers in the pool
Shown as buffer
otel.process.runtime.jvm.memory.usage
(gauge)
The total Java memory used.
Shown as byte
otel.process.runtime.jvm.memory.committed
(gauge)
The total Java memory committed to be used.
Shown as byte
otel.process.runtime.jvm.memory.init
(gauge)
The initial Java memory allocated.
Shown as byte
otel.process.runtime.jvm.memory.limit
(gauge)
The maximum Java memory available.
Shown as byte
otel.process.runtime.jvm.threads.count
(count)
The number of live threads.
Shown as thread
otel.process.runtime.jvm.gc.duration
(gauge)
The approximate accumulated garbage collection time elapsed.
Shown as millisecond
process.runtime.dotnet.thread_pool.threads.count
(gauge)
The number of threads
Shown as thread
process.runtime.dotnet.monitor.lock_contention.count
(count)
The number of times a thread stopped to wait on a lock
process.runtime.dotnet.exceptions.count
(count)
The number of first-chance exceptions
Shown as exception
process.runtime.dotnet.gc.heap.size
(gauge)
The heap size (including fragmentation) as observed during the latest garbage collection
Shown as byte
process.runtime.dotnet.gc.collections.count
(count)
Number of garbage collections that have occured since process start
Shown as garbage collection
process.runtime.dotnet.gc.objects.size
(gauge)
Count of bytes currently in use by objects in the GC heap that haven't been collected yet
Shown as byte
process.runtime.dotnet.gc.allocations.size
(count)
Count of bytes allocated on the managed GC heap since the process start
Shown as byte
process.runtime.dotnet.gc.heap.fragmentation.size
(gauge)
The heap fragmentation as observed during the latest garbage collection
Shown as byte
process.runtime.dotnet.jit.il_compiled.size
(count)
Count of bytes of intermediate language that have been compiled since the process start
Shown as byte
process.runtime.dotnet.jit.methods_compiled.count
(count)
The number of times the JIT compiler compiled a method since the process start
Shown as method
process.runtime.dotnet.jit.compilation_time
(count)
The amount of time the JIT compiler has spent compiling methods since the process start
Shown as nanosecond
process.runtime.dotnet.thread_pool.completed_items.count
(count)
The number of work items that have been processed by the thread pool since the process start
Shown as item
process.runtime.dotnet.thread_pool.queue.length
(gauge)
The number of work items that are currently queued to be processed by the thread pool
Shown as item
process.runtime.dotnet.timer.count
(gauge)
The number of timer instances that are currently active
Shown as unit
process.runtime.dotnet.assemblies.count
(gauge)
The number of .NET assemblies that are currently loaded
Shown as unit
process.runtime.dotnet.gc.committed_memory.size
(gauge)
The amount of committed virtual memory for the managed GC heap as observed during the latest garbage collection
Shown as byte
otel.process.runtime.dotnet.thread_pool.threads.count
(gauge)
The number of threads
Shown as thread
otel.process.runtime.dotnet.monitor.lock_contention.count
(count)
The number of times a thread stopped to wait on a lock
otel.process.runtime.dotnet.exceptions.count
(count)
The number of first-chance exceptions
Shown as exception
otel.process.runtime.dotnet.gc.heap.size
(gauge)
The heap size (including fragmentation) as observed during the latest garbage collection
Shown as byte
otel.process.runtime.dotnet.gc.collections.count
(count)
Number of garbage collections that have occured since process start
Shown as garbage collection
otel.process.runtime.dotnet.gc.objects.size
(gauge)
Count of bytes currently in use by objects in the GC heap that haven't been collected yet
Shown as byte
otel.process.runtime.dotnet.gc.allocations.size
(count)
Count of bytes allocated on the managed GC heap since the process start
Shown as byte
otel.process.runtime.dotnet.gc.heap.fragmentation.size
(gauge)
The heap fragmentation as observed during the latest garbage collection
Shown as byte
otel.process.runtime.dotnet.jit.il_compiled.size
(count)
Count of bytes of intermediate language that have been compiled since the process start
Shown as byte
otel.process.runtime.dotnet.jit.methods_compiled.count
(count)
The number of times the JIT compiler compiled a method since the process start
Shown as method
otel.process.runtime.dotnet.jit.compilation_time
(count)
The amount of time the JIT compiler has spent compiling methods since the process start
Shown as nanosecond
otel.process.runtime.dotnet.thread_pool.completed_items.count
(count)
The number of work items that have been processed by the thread pool since the process start
Shown as item
otel.process.runtime.dotnet.thread_pool.queue.length
(gauge)
The number of work items that are currently queued to be processed by the thread pool
Shown as item
otel.process.runtime.dotnet.timer.count
(gauge)
The number of timer instances that are currently active
Shown as unit
otel.process.runtime.dotnet.assemblies.count
(gauge)
The number of .NET assemblies that are currently loaded
Shown as unit
otel.process.runtime.dotnet.gc.committed_memory.size
(gauge)
The amount of committed virtual memory for the managed GC heap as observed during the latest garbage collection
Shown as byte
process.runtime.cpython.cpu_time
(count)
Number of seconds executing in the kernel
Shown as second
process.runtime.cpython.gc_count
(count)
Number of completed garbage collection cycles
Shown as resource
process.runtime.cpython.memory
(gauge)
Resident set memory
Shown as byte
otel.process.runtime.cpython.cpu_time
(count)
Number of seconds executing in the kernel
Shown as second
otel.process.runtime.cpython.gc_count
(count)
Number of completed garbage collection cycles
Shown as resource
otel.process.runtime.cpython.memory
(gauge)
Resident set memory
Shown as byte

Service Checks

The OpenTelemetry Collector does not include any service checks.

Events

The OpenTelemetry Collector does not include any events.

Troubleshooting

Need help? Contact Datadog support.

PREVIEWING: mervebolat/span-id-preprocessing