Docker Metrics

Overview

OpenTelemetry Docker metrics in a Containers dashboard

To collect container metrics, configure the Docker stats receiver in your Datadog Exporter.

For more information, see the OpenTelemetry project documentation for the Docker stats receiver.

Setup

The Docker stats receiver needs access to the Docker socket. By default, the receiver looks for the Docker socket at unix:///var/run/docker.sock. If this is not the Docker socket path, specify the path in the endpoint configuration line.

Add the following lines to your Collector configuration:

receivers:
  docker_stats:
    endpoint: unix:///var/run/docker.sock # (default)
    metrics:
      container.network.io.usage.rx_packets:
        enabled: true
      container.network.io.usage.tx_packets:
        enabled: true
      container.cpu.usage.system:
        enabled: true
      container.memory.rss:
        enabled: true
      container.blockio.io_serviced_recursive:
        enabled: true
      container.uptime:
        enabled: true
      container.memory.hierarchical_memory_limit:
        enabled: true

Note: If you are using the collector image, you may need to configure additional permissions for the collector to have access to the Docker socket.

The Docker stats receiver needs access to the Docker socket. In Kubernetes, if you are running Docker as a runtime, mount the Docker socket:

Add the following lines to values.yaml:

extraVolumes:
 - name: docker-sock
   hostPath:
     path: /var/run/docker.sock
extraVolumeMounts:
 - name: docker-sock
   mountPath: /var/run/docker.sock

Add the following in the Collector configuration:

receivers:
  docker_stats:
    endpoint: unix:///var/run/docker.sock # default
    metrics:
      container.network.io.usage.rx_packets:
        enabled: true
      container.network.io.usage.tx_packets:
        enabled: true
      container.cpu.usage.system:
        enabled: true
      container.memory.rss:
        enabled: true
      container.blockio.io_serviced_recursive:
        enabled: true
      container.uptime:
        enabled: true
      container.memory.hierarchical_memory_limit:
        enabled: true

Data collected

The Docker Stats receiver generates container metrics for the OpenTelemetry Collector. The Datadog Exporter translates container metrics to their Datadog counterparts for use in the following views:

Note: To correlate trace and container metrics, configure Universal Service Monitoring attributes for each service, and set the following resource attributes for each service:

  • k8s.container.name
  • k8s.pod.name
  • container.name
  • container.id

Learn more about mapping between OpenTelemetry and Datadog semantic conventions for resource attributes.

The following table shows what Datadog container metric names are associated with corresponding OpenTelemetry container metric names

OTELDATADOGDESCRIPTIONFILTER
container.blockio.io_serviced_recursivecontainer.io.read.operationsNumber of IOs (bio) issued to the disk by the group and descendant groups (Only available with cgroups v1).operation: read
container.blockio.io_serviced_recursivecontainer.io.write.operationsNumber of IOs (bio) issued to the disk by the group and descendant groups (Only available with cgroups v1).operation: write
container.cpu.throttling_data.throttled_periodscontainer.cpu.throttled.periodsNumber of periods when the container hits its throttling limit.
container.cpu.throttling_data.throttled_timecontainer.cpu.throttledAggregate time the container was throttled.
container.cpu.usage.systemcontainer.cpu.systemSystem CPU usage, as reported by docker.
container.cpu.usage.totalcontainer.cpu.usageTotal CPU time consumed.
container.cpu.usage.usermodecontainer.cpu.userTime spent by tasks of the cgroup in user mode (Linux). Time spent by all container processes in user mode (Windows).
container.memory.active_anoncontainer.memory.kernelThe amount of anonymous memory that has been identified as active by the kernel.
container.memory.hierarchical_memory_limitcontainer.memory.limitThe maximum amount of physical memory that can be used by the processes of this control group (Only available with cgroups v1).
container.memory.rsskubernetes.memory.rssThe amount of memory that doesn’t correspond to anything on disk: stacks, heaps, and anonymous memory maps (Only available with cgroups v1).
container.memory.total_cachecontainer.memory.cacheTotal amount of memory used by the processes of this cgroup (and descendants) that can be associated with a block on a block device. Also accounts for memory used by tmpfs (Only available with cgroups v1).
container.memory.usage.limitcontainer.memory.soft_limitMemory limit of the container.
container.memory.usage.totalcontainer.memory.usageMemory usage of the container. This excludes the cache.
container.network.io.usage.rx_bytescontainer.net.rcvdBytes received by the container.
container.network.io.usage.rx_packetscontainer.net.rcvd.packetsPackets received.
container.network.io.usage.tx_bytescontainer.net.sentBytes sent.
container.network.io.usage.tx_packetscontainer.net.sent.packetsPackets sent.

See OpenTelemetry Metrics Mapping for more information.

Full example configuration

For a full working example configuration with the Datadog exporter, see docker-stats.yaml.

Example logging output

Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
     -> container.runtime: Str(docker)
     -> container.hostname: Str(be51776e036e)
     -> container.id: Str(be51776e036e04461169fce2847d4e77be3d83856b474ad544143afc3d48e9e5)
     -> container.image.name: Str(sha256:9bdff337981de15f8cdf9e73b24af64a03e2e6dd1f156a274a15c1d8db98ab79)
     -> container.name: Str(redis-otel)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/dockerstatsreceiver 0.89.0-dev
Metric #6
Descriptor:
     -> Name: container.cpu.utilization
     -> Description: Percent of CPU used by the container.
     -> Unit: 1
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 2023-11-20 14:58:17.522765 +0000 UTC
Timestamp: 2023-11-20 14:58:19.550208 +0000 UTC
Value: 0.170933
PREVIEWING: domalessi/docs-10186