Overview

Get metrics from Kubernetes in real time to:

• Visualize and monitor Kubernetes states
• Be notified about Kubernetes failovers and events

For Kubernetes, it’s recommended to run the Agent in a DaemonSet. A Docker image is available with both the Docker and the Kubernetes integrations enabled.

You can also just run the Datadog Agent on your host and configure it to gather your Kubernetes metrics.

Set up Kubernetes

Installation

Container installation

Thanks to Kubernetes, you can take advantage of DaemonSets to automatically deploy the Datadog Agent on all your nodes (or on specific nodes by using nodeSelectors).

If DaemonSets are not an option for your Kubernetes cluster, install the Datadog Agent as a deployment on each Kubernetes node.

If your Kubernetes has RBAC enabled, see how to configure RBAC permissions with your Datadog-Kubernetes integration.

• Create the following dd-agent.yaml manifest:

apiVersion: extensions/v1beta1
metadata:
  name: dd-agent
spec:
  template:
    metadata:
      labels:
        app: dd-agent
      name: dd-agent
    spec:
      containers:
      - image: gcr.io/datadoghq/docker-dd-agent:latest
        imagePullPolicy: Always
        name: dd-agent
        ports:
          - containerPort: 8125
            name: dogstatsdport
            protocol: UDP
        env:
          - name: API_KEY
            value: "DATADOG_API_KEY"
          - name: KUBERNETES
            value: "yes"
        volumeMounts:
          - name: dockersocket
            mountPath: /var/run/docker.sock
          - name: procdir
            mountPath: /host/proc
            readOnly: true
          - name: cgroups
            mountPath: /host/sys/fs/cgroup
            readOnly: true
      volumes:
        - hostPath:
            path: /var/run/docker.sock
          name: dockersocket
        - hostPath:
            path: /proc
          name: procdir
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroups

Replace DATADOG_API_KEY with your api key or use Kubernetes secrets to set your API key as an environment variable.

• Deploy the DaemonSet with the command:

  kubectl create -f dd-agent.yaml
  

Note: This manifest enables the Autodiscovery, auto-configuration feature. To disable it, remove the SD_BACKEND environment variable definition. To learn how to configure Autodiscovery, see the Autodiscovery documentation.

Host installation

Install the dd-check-kubernetes package manually or with your favorite configuration manager.

Configuration

Edit the kubernetes.yaml file to point to your server and port, set the masters to monitor:

instances:
    host: localhost
    port: 4194
    method: http

See the example kubernetes.yaml for all available configuration options.

Validation

Container running

To verify the Datadog Agent is running in your environment as a DaemonSet, execute:

kubectl get daemonset

If the Agent is deployed, a similar output to the text below is displayed, where desired and current are equal to the number of nodes running in your cluster.

NAME       DESIRED   CURRENT   NODE-SELECTOR   AGE
dd-agent   3         3         <none>          11h

Agent check running

Run the Agent’s info subcommand and look for kubernetes under the Checks section:

Checks
======
    kubernetes
    -----------
      - instance #0 [OK]
      - Collected 39 metrics, 0 events & 7 service checks

Setup Kubernetes state

Installation

Container installation

If you are running Kubernetes >= 1.2.0, you can use the kube-state-metrics project to provide additional metrics (identified by the kubernetes_state prefix in the metrics list below) to Datadog.

To run kube-state-metrics, create a kube-state-metrics.yaml file using the following manifest to deploy the kube-state-metrics service:

apiVersion: extensions/v1beta1
metadata:
  name: kube-state-metrics
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: gcr.io/google_containers/kube-state-metrics:v1.2.0
        ports:
        - name: metrics
          containerPort: 8080
        resources:
          requests:
            memory: 30Mi
            cpu: 100m
          limits:
            memory: 50Mi
            cpu: 200m
---
apiVersion: v1
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  labels:
    app: kube-state-metrics
  name: kube-state-metrics
spec:
  ports:
  - name: metrics
    port: 8080
    targetPort: metrics
    protocol: TCP
  selector:
    app: kube-state-metrics

Then deploy it by running:

kubectl create -f kube-state-metrics.yaml

The manifest above uses Google’s publicly available kube-state-metrics container, which is also available on Quay. If you want to build it manually, see the official project documentation.

If you configure your Kubernetes State Metrics service to run on a different URL or port, you can configure the Datadog Agent by setting the kube_state_url parameter in conf.d/kubernetes_state.yaml, then restarting the Agent. For more information, see the kubernetes_state.yaml.example file. If you have enabled Autodiscovery, the kube state URL is configured and managed automatically.

Host installation

Install the dd-check-kubernetes_state package manually or with your favorite configuration manager (On CentOS/AWS, download the rpm package and see the installation instructions. Then edit the kubernetes_state.yaml file to point to your server and port and set the masters to monitor. See the example kubernetes_state.yaml for all available configuration options.

Validation

Container validation

To verify the Datadog Agent is running in your environment as a DaemonSet, execute:

kubectl get daemonset

If the Agent is deployed, you receive a similar output to the text below, where desired and current are equal to the number of running nodes in your cluster.

NAME       DESIRED   CURRENT   NODE-SELECTOR   AGE
dd-agent   3         3         <none>          11h

Agent check validation

Run the Agent’s info subcommand and look for kubernetes_state under the Checks section:

Checks
======
    kubernetes_state
    -----------
      - instance #0 [OK]
      - Collected 39 metrics, 0 events & 7 service checks

Setup Kubernetes DNS

Installation

Install the dd-check-kube_dns package manually or with your favorite configuration manager.

Configuration

Edit the kube_dns.yaml file to point to your server and port, set the masters to monitor. See the sample kube_dns.yaml for all available configuration options.

Using with service discovery

If you are using one dd-agent pod per Kubernetes worker node, you could use the following annotations on your kube-dns pod to retrieve the data automatically.

apiVersion: v1
metadata:
  annotations:
    service-discovery.datadoghq.com/kubedns.check_names: '["kube_dns"]'
    service-discovery.datadoghq.com/kubedns.init_configs: '[{}]'
    service-discovery.datadoghq.com/kubedns.instances: '[[{"prometheus_endpoint":"http://%%host%%:10055/metrics", "tags":["dns-pod:%%host%%"]}]]'

Notes:

• Notice the “dns-pod” tag keeps track of the target DNS pod IP. The other tags are related to the dd-agent that is polling the information using service discovery.
• The service discovery annotations need to be applied to the pod. In case of a deployment, add the annotations to the metadata of the template’s spec.

Validation

Run the Agent’s info subcommand and look for kube_dns under the Checks section:

Checks
======
    kube_dns
    -----------
      - instance #0 [OK]
      - Collected 39 metrics, 0 events & 7 service checks