CoreDNS

Supported OS Linux

Integration version5.0.0

Overview

Get metrics from CoreDNS in real time to visualize and monitor DNS failures and cache hits or misses.

Setup

Starting with version 1.11.0, this OpenMetrics-based integration has a latest mode (enabled by setting openmetrics_endpoint to point to the target endpoint) and a legacy mode (enabled by setting prometheus_url instead). To get all the most up-to-date features, Datadog recommends enabling the latest mode. For more information, see Latest and Legacy Versioning For OpenMetrics-based Integrations.

The latest mode of the CoreDNS check requires Python 3 and submits .bucket metrics and submits the .sum and .count histogram samples as monotonic count type. These metrics were previously submitted as gauge type in the legacy mode. See the metadata.csv file for a list of metrics available in each mode.

For hosts unable to use Python 3, or if you previously implemented this integration mode, see the legacy mode configuration example. For Autodiscovery users relying on the coredns.d/auto_conf.yaml file, this file enables the prometheus_url option for the legacy mode of the check by default. See the sample coredns.d/auto_conf.yaml for the default configuration options and the sample coredns.d/conf.yaml.example for all available configuration options.

Installation

The CoreDNS check is included in the Datadog Agent package, so you don’t need to install anything else on your servers.

Configuration

Docker

To configure this check for an Agent running on a container:

Metric collection

Set Autodiscovery Integration Templates as Docker labels on your application container:

LABEL "com.datadoghq.ad.check_names"='["coredns"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='[{"openmetrics_endpoint":"http://%%host%%:9153/metrics", "tags":["dns-pod:%%host%%"]}]'

To enable the legacy mode of this OpenMetrics-based check, replace openmetrics_endpoint with prometheus_url:

LABEL "com.datadoghq.ad.instances"='[{"prometheus_url":"http://%%host%%:9153/metrics", "tags":["dns-pod:%%host%%"]}]' 

Notes:

  • The shipped coredns.d/auto_conf.yaml file enables the prometheus_url option by default for legacy mode.
  • The dns-pod tag keeps track of the target DNS pod IP. The other tags are related to the Datadog Agent that is polling the information using the service discovery.
  • The service discovery annotations need to be done on the pod. In case of a deployment, add the annotations to the metadata of the template’s specifications. Do not add it at the outer specification level.

Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Docker Log Collection.

Then, set Log Integrations as Docker labels:

LABEL "com.datadoghq.ad.logs"='[{"source":"coredns","service":"<SERVICE_NAME>"}]'

Kubernetes

To configure this check for an Agent running on Kubernetes:

Metric collection

Set Autodiscovery Integrations Templates as pod annotations on your application container. Alternatively, you can configure templates with a file, configmap, or key-value store.

Annotations v1 (for Datadog Agent < v7.36)

apiVersion: v1
kind: Pod
metadata:
  name: coredns
  annotations:
    ad.datadoghq.com/coredns.check_names: '["coredns"]'
    ad.datadoghq.com/coredns.init_configs: '[{}]'
    ad.datadoghq.com/coredns.instances: |
      [
        {
          "openmetrics_endpoint": "http://%%host%%:9153/metrics", 
          "tags": ["dns-pod:%%host%%"]
        }
      ]      
  labels:
    name: coredns
spec:
  containers:
    - name: coredns

Annotations v2 (for Datadog Agent v7.36 or later)

apiVersion: v1
kind: Pod
metadata:
  name: coredns
  annotations:
    ad.datadoghq.com/coredns.checks: |
      {
        "coredns": {
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:9153/metrics", 
              "tags": ["dns-pod:%%host%%"]
            }
          ]
        }
      }      
  labels:
    name: coredns
spec:
  containers:
    - name: coredns

To enable the legacy mode of this OpenMetrics-based check, replace openmetrics_endpoint with prometheus_url:

Annotations v1 (for Datadog Agent < v7.36)

    ad.datadoghq.com/coredns.instances: |
      [
        {
          "prometheus_url": "http://%%host%%:9153/metrics", 
          "tags": ["dns-pod:%%host%%"]
        }
      ]      

Annotations v2 (for Datadog Agent v7.36 or later)

          "instances": [
            {
              "prometheus_url": "http://%%host%%:9153/metrics", 
              "tags": ["dns-pod:%%host%%"]
            }
          ]

Notes:

  • The shipped coredns.d/auto_conf.yaml file enables the prometheus_url option by default for legacy mode.
  • The dns-pod tag keeps track of the target DNS pod IP. The other tags are related to the Datadog Agent that is polling the information using the service discovery.
  • The service discovery annotations need to be done on the pod. In case of a deployment, add the annotations to the metadata of the template’s specifications. Do not add it at the outer specification level.

Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Kubernetes Log Collection.

Then, set Log Integrations as pod annotations. Alternatively, you can configure this with a file, configmap, or key-value store.

Annotations v1/v2

apiVersion: v1
kind: Pod
metadata:
  name: coredns
  annotations:
    ad.datadoghq.com/coredns.logs: '[{"source": "coredns", "service": "<SERVICE_NAME>"}]'
  labels:
    name: coredns

ECS

To configure this check for an Agent running on ECS:

Metric collection

Set Autodiscovery Integrations Templates as Docker labels on your application container:

{
  "containerDefinitions": [{
    "name": "coredns",
    "image": "coredns:latest",
    "dockerLabels": {
      "com.datadoghq.ad.check_names": "[\"coredns\"]",
      "com.datadoghq.ad.init_configs": "[{}]",
      "com.datadoghq.ad.instances": "[{\"openmetrics_endpoint\":\"http://%%host%%:9153/metrics\", \"tags\":[\"dns-pod:%%host%%\"]}]"
    }
  }]
}

To enable the legacy mode of this OpenMetrics-based check, replace openmetrics_endpoint with prometheus_url:

      "com.datadoghq.ad.instances": "[{\"prometheus_url\":\"http://%%host%%:9153/metrics\", \"tags\":[\"dns-pod:%%host%%\"]}]"

Notes:

  • The shipped coredns.d/auto_conf.yaml file enables the prometheus_url option by default for legacy mode.
  • The dns-pod tag keeps track of the target DNS pod IP. The other tags are related to the Datadog Agent that is polling the information using the service discovery.
  • The service discovery annotations need to be done on the pod. In case of a deployment, add the annotations to the metadata of the template’s specifications. Do not add it at the outer specification level.
Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see ECS Log Collection.

Then, set Log Integrations as Docker labels:

{
  "containerDefinitions": [{
    "name": "coredns",
    "image": "coredns:latest",
    "dockerLabels": {
      "com.datadoghq.ad.logs": "[{\"source\":\"coredns\",\"service\":\"<SERVICE_NAME>\"}]"
    }
  }]
}

Validation

Run the Agent’s status subcommand and look for coredns under the Checks section.

Data Collected

Metrics

coredns.acl.allowed_requests
(count)
[OpenMetrics V1] Counter of DNS requests being allowed.
Shown as request
coredns.acl.allowed_requests.count
(count)
[OpenMetrics V2] Counter of DNS requests being allowed.
Shown as request
coredns.acl.blocked_requests
(count)
[OpenMetrics V1] Counter of DNS requests being blocked.
Shown as request
coredns.acl.blocked_requests.count
(count)
[OpenMetrics V2] Counter of DNS requests being blocked.
Shown as request
coredns.autopath.success_count
(count)
[OpenMetrics V1] Counter of requests that did autopath.
Shown as request
coredns.autopath.success_count.count
(count)
[OpenMetrics V2] Counter of requests that did autopath.
Shown as request
coredns.build_info
(gauge)
[OpenMetrics V1 and V2] A metric with a constant '1' value labeled by version, revision, and goversion from which CoreDNS was built.
coredns.cache_drops_count
(count)
[OpenMetrics V1] Counter of responses excluded from the cache due to request/response question name mismatch.
Shown as response
coredns.cache_drops_count.count
(count)
[OpenMetrics V2] Counter of responses excluded from the cache due to request/response question name mismatch.
Shown as response
coredns.cache_hits_count
(count)
[OpenMetrics V1] Counter of cache hits by cache type
Shown as hit
coredns.cache_hits_count.count
(count)
[OpenMetrics V2] Counter of cache hits by cache type
Shown as hit
coredns.cache_misses_count
(count)
[OpenMetrics V1] Counter of cache misses.
Shown as miss
coredns.cache_misses_count.count
(count)
[OpenMetrics V2] Counter of cache misses.
Shown as miss
coredns.cache_prefetch_count
(count)
[OpenMetrics V1] The number of time the cache has prefetched a cached item.
coredns.cache_prefetch_count.count
(count)
[OpenMetrics V2] The number of time the cache has prefetched a cached item.
coredns.cache_request_count
(count)
[OpenMetrics V1] Counter of cache requests.
Shown as request
coredns.cache_request_count.count
(count)
[OpenMetrics V2] Counter of cache requests.
Shown as request
coredns.cache_size.count
(gauge)
[OpenMetrics V1 and V2]
Shown as entry
coredns.cache_stale_count
(count)
[OpenMetrics V1] Counter of requests served from stale cache entries.
Shown as request
coredns.cache_stale_count.count
(count)
[OpenMetrics V2] Counter of requests served from stale cache entries.
Shown as request
coredns.dnssec.cache_hits
(count)
[OpenMetrics V1] Counter of cache hits.
Shown as hit
coredns.dnssec.cache_hits.count
(count)
[OpenMetrics V2] Counter of cache hits.
Shown as hit
coredns.dnssec.cache_misses
(count)
[OpenMetrics V1] Counter of cache misses.
Shown as miss
coredns.dnssec.cache_misses.count
(count)
[OpenMetrics V2] Counter of cache misses.
Shown as miss
coredns.dnssec.cache_size
(gauge)
[OpenMetrics V1 and V2] Total elements in the cache, type is signature.
coredns.forward_healthcheck_broken_count
(count)
[OpenMetrics V1] counter of when all upstreams are unhealthy
Shown as entry
coredns.forward_healthcheck_broken_count.count
(count)
[OpenMetrics V2] counter of when all upstreams are unhealthy
Shown as entry
coredns.forward_healthcheck_failure_count
(count)
[OpenMetrics V1] number of failed health checks per upstream
Shown as entry
coredns.forward_healthcheck_failure_count.count
(count)
[OpenMetrics V2] number of failed health checks per upstream
Shown as entry
coredns.forward_max_concurrent_rejects
(count)
[OpenMetrics V1] Counter of the number of queries rejected because the concurrent queries were at maximum.
Shown as query
coredns.forward_max_concurrent_rejects.count
(count)
[OpenMetrics V2] Counter of the number of queries rejected because the concurrent queries were at maximum.
Shown as query
coredns.forward_request_count
(count)
[OpenMetrics V1] query count per upstream
Shown as request
coredns.forward_request_count.count
(count)
[OpenMetrics V2] query count per upstream
Shown as request
coredns.forward_request_duration.seconds.bucket
(count)
[OpenMetrics V2] duration per upstream interaction
Shown as second
coredns.forward_request_duration.seconds.count
(count)
[OpenMetrics V1 and V2] duration per upstream interaction
Shown as second
coredns.forward_request_duration.seconds.sum
(count)
[OpenMetrics V1 and V2] duration per upstream interaction
Shown as second
coredns.forward_response_rcode_count
(count)
[OpenMetrics V1] count of RCODEs per upstream
Shown as response
coredns.forward_response_rcode_count.count
(count)
[OpenMetrics V2] count of RCODEs per upstream
Shown as response
coredns.forward_sockets_open
(gauge)
[OpenMetrics V1 and V2] number of sockets open per upstream
Shown as connection
coredns.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2] Count of the GC invocation durations.
Shown as second
coredns.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2] Quantiles of the GC invocation durations.
Shown as second
coredns.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of the GC invocation durations.
Shown as second
coredns.go.goroutines
(gauge)
[OpenMetrics V1 and V2] Number of goroutines that currently exist.
Shown as thread
coredns.go.info
(gauge)
[OpenMetrics V1 and V2] Information about the Go environment.
coredns.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes allocated and still in use.
Shown as byte
coredns.go.memstats.alloc_bytes_total
(count)
[OpenMetrics V1] Total number of bytes allocated even if freed.
Shown as byte
coredns.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the profiling bucket hash table.
Shown as byte
coredns.go.memstats.frees_total
(count)
[OpenMetrics V1] Total number of frees.
coredns.go.memstats.frees_total.count
(count)
[OpenMetrics V2] Total number of frees.
coredns.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2] CPU taken up by GC
Shown as percent
coredns.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for garbage collection system metadata.
Shown as byte
coredns.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Bytes allocated to the heap
Shown as byte
coredns.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2] Number of idle bytes in the heap
Shown as byte
coredns.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of Bytes in the heap
Shown as byte
coredns.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2] Number of objects in the heap
Shown as object
coredns.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes released to the system in the last gc
Shown as byte
coredns.go.memstats.heap_released_bytes.count
(count)
[OpenMetrics V2] Count of bytes released to the system in the last gc
Shown as byte
coredns.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the heap
Shown as byte
coredns.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2] Length of last GC
Shown as second
coredns.go.memstats.lookups_total
(count)
[OpenMetrics V1] Number of lookups
Shown as operation
coredns.go.memstats.lookups_total.count
(count)
[OpenMetrics V2] Number of lookups
Shown as operation
coredns.go.memstats.mallocs_total
(count)
[OpenMetrics V1] Number of mallocs
Shown as operation
coredns.go.memstats.mallocs_total.count
(count)
[OpenMetrics V2] Number of mallocs
Shown as operation
coredns.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mcache structures.
Shown as byte
coredns.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mcache structures obtained from system.
Shown as byte
coredns.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mspan structures.
Shown as byte
coredns.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mspan structures obtained from system.
Shown as byte
coredns.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of heap bytes when next garbage collection will take place
Shown as byte
coredns.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for other system allocations
Shown as byte
coredns.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by the stack allocator
Shown as byte
coredns.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system for stack allocator
Shown as byte
coredns.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system
Shown as byte
coredns.go.threads
(gauge)
[OpenMetrics V1 and V2] Number of OS threads created.
Shown as thread
coredns.grpc.request_count
(count)
[OpenMetrics V1] Query count per upstream.
coredns.grpc.request_count.count
(count)
[OpenMetrics V2] Query count per upstream.
coredns.grpc.response_rcode_count
(count)
[OpenMetrics V1] Count of RCODEs per upstream. and we are randomly (this always uses the random policy) spraying to an upstream.
coredns.grpc.response_rcode_count.count
(count)
[OpenMetrics V2] Count of RCODEs per upstream. and we are randomly (this always uses the random policy) spraying to an upstream.
coredns.health_request_duration.bucket
(count)
[OpenMetrics V2] Sample for the histogram of the time (in seconds) each request took.
coredns.health_request_duration.count
(count)
[OpenMetrics V1 and V2] Count for the histogram of the time (in seconds) each request took.
coredns.health_request_duration.sum
(count)
[OpenMetrics V1 and V2] Sum for the histogram of the time (in seconds) each request took.
coredns.hosts.entries_count
(gauge)
[OpenMetrics V1 and V2] The combined number of entries in hosts and Corefile.
coredns.hosts.reload_timestamp
(gauge)
[OpenMetrics V1 and V2] The timestamp of the last reload of hosts file.
Shown as second
coredns.panic_count.count
(count)
[OpenMetrics V1 and V2]
Shown as entry
coredns.plugin_enabled
(gauge)
[OpenMetrics V1 and V2] A metric that indicates whether a plugin is enabled on per server and zone basis.
coredns.process.cpu_seconds_total
(count)
[OpenMetrics V1 and V2] Total user and system CPU time spent in seconds.
Shown as second
coredns.process.cpu_seconds_total.count
(count)
[OpenMetrics V2] Count of user and system CPU time spent in seconds.
Shown as second
coredns.process.max_fds
(gauge)
[OpenMetrics V1 and V2] Maximum number of open file descriptors.
Shown as file
coredns.process.open_fds
(gauge)
[OpenMetrics V1 and V2] Number of open file descriptors.
Shown as file
coredns.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Resident memory size in bytes.
Shown as byte
coredns.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2] Start time of the process since unix epoch in seconds.
Shown as second
coredns.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Virtual memory size in bytes.
Shown as byte
coredns.proxy_request_count
(count)
[OpenMetrics V1] query count per upstream.
Shown as request
coredns.proxy_request_count.count
(count)
[OpenMetrics V2] query count per upstream.
Shown as request
coredns.proxy_request_duration.seconds.bucket
(count)
[OpenMetrics V2] sample of duration per upstream interaction
Shown as second
coredns.proxy_request_duration.seconds.count
(count)
[OpenMetrics V1 and V2] duration per upstream interaction
Shown as second
coredns.proxy_request_duration.seconds.sum
(count)
[OpenMetrics V1 and V2] duration per upstream interaction
Shown as second
coredns.reload.failed_count
(count)
[OpenMetrics V1] Counts the number of failed reload attempts.
coredns.reload.failed_count.count
(count)
[OpenMetrics V2] Counts the number of failed reload attempts.
coredns.request_count
(count)
[OpenMetrics V1] total query count.
Shown as request
coredns.request_count.count
(count)
[OpenMetrics V2] total query count.
Shown as request
coredns.request_duration.seconds.bucket
(count)
[OpenMetrics V2] sample duration to process each query
Shown as second
coredns.request_duration.seconds.count
(count)
[OpenMetrics V1 and V2] duration to process each query
Shown as second
coredns.request_duration.seconds.sum
(count)
[OpenMetrics V1 and V2] duration to process each query
Shown as second
coredns.request_size.bytes.bucket
(count)
[OpenMetrics V2] sample size of the request in bytes
Shown as byte
coredns.request_size.bytes.count
(count)
[OpenMetrics V1 and V2] size of the request in bytes
Shown as byte
coredns.request_size.bytes.sum
(count)
[OpenMetrics V1 and V2] size of the request in bytes
Shown as byte
coredns.request_type_count
(count)
[OpenMetrics V1] counter of queries per zone and type
coredns.request_type_count.count
(count)
[OpenMetrics V2] counter of queries per zone and type
coredns.response_code_count
(count)
[OpenMetrics V1] number of responses per zone and rcode
coredns.response_code_count.count
(count)
[OpenMetrics V2] number of responses per zone and rcode
coredns.response_size.bytes.bucket
(count)
[OpenMetrics V2] sample size of the request in bytes
Shown as byte
coredns.response_size.bytes.count
(count)
[OpenMetrics V1 and V2] size of the request in bytes
Shown as byte
coredns.response_size.bytes.sum
(count)
[OpenMetrics V1 and V2] size of the request in bytes
Shown as byte
coredns.template.failures_count
(count)
[OpenMetrics V1] The number of times the Go templating failed.
Shown as error
coredns.template.failures_count.count
(count)
[OpenMetrics V2] The number of times the Go templating failed.
Shown as error
coredns.template.matches_count
(count)
[OpenMetrics V1] The total number of matched requests by regex.
coredns.template.matches_count.count
(count)
[OpenMetrics V2] The total number of matched requests by regex.
coredns.template.rr_failures_count
(count)
[OpenMetrics V1] The number of times the templated resource record was invalid and could not be parsed.
Shown as error
coredns.template.rr_failures_count.count
(count)
[OpenMetrics V2] The number of times the templated resource record was invalid and could not be parsed.
Shown as error

Events

The CoreDNS check does not include any events.

Service Checks

coredns.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

PREVIEWING: esther/docs-9478-fix-split-after-example