Kubernetes Tag Extraction

The Datadog Agent can automatically assign tags to metrics, traces, and logs emitted by a pod (or an individual container within a pod) based on labels or annotations.

Out-of-the-box tags

The list of automatically-assigned tags depends on the Agent’s cardinality configuration.

TagCardinalitySourceRequirement
container_idHighPod statusN/A
display_container_nameHighPod statusN/A
pod_nameOrchestratorPod metadataN/A
oshift_deploymentOrchestratorPod annotation openshift.io/deployment.nameOpenShift environment and pod annotation must exist
kube_ownerref_nameOrchestratorPod ownerrefPod must have an owner
kube_jobOrchestratorPod ownerrefPod must be attached to a cronjob
kube_jobLowPod ownerrefPod must be attached to a job
kube_replica_setLowPod ownerrefPod must be attached to a replica set
kube_serviceLowKubernetes service discoveryPod is behind a Kubernetes service
kube_daemon_setLowPod ownerrefPod must be attached to a DaemonSet
kube_container_nameLowPod statusN/A
kube_namespaceLowPod metadataN/A
kube_app_nameLowPod label app.kubernetes.io/namePod label must exist
kube_app_instanceLowPod label app.kubernetes.io/instancePod label must exist
kube_app_versionLowPod label app.kubernetes.io/versionPod label must exist
kube_app_componentLowPod label app.kubernetes.io/componentPod label must exist
kube_app_part_ofLowPod label app.kubernetes.io/part-ofPod label must exist
kube_app_managed_byLowPod label app.kubernetes.io/managed-byPod label must exist
envLowPod label tags.datadoghq.com/env or container envvar (DD_ENV or OTEL_RESOURCE_ATTRIBUTES)Unified service tagging enabled
versionLowPod label tags.datadoghq.com/version or container envvar (DD_ENV or OTEL_RESOURCE_ATTRIBUTES)Unified service tagging enabled
serviceLowPod label tags.datadoghq.com/service or container envvar (DD_SERVICE, OTEL_RESOURCE_ATTRIBUTES, or OTEL_SERVICE_NAME)Unified service tagging enabled
pod_phaseLowPod statusN/A
oshift_deployment_configLowPod annotation openshift.io/deployment-config.nameOpenShift environment and pod annotation must exist
kube_ownerref_kindLowPod ownerrefPod must have an owner
kube_deploymentLowPod ownerrefPod must be attached to a deployment
kube_replication_controllerLowPod ownerrefPod must be attached to a replication controller
kube_stateful_setLowPod ownerrefPod must be attached to a statefulset
persistentvolumeclaimLowPod specA PVC must be attached to the pod
kube_cronjobLowPod ownerrefPod must be attached to a cronjob
image_nameLowPod specN/A
short_imageLowPod specN/A
image_tagLowPod specN/A
eks_fargate_nodeLowPod specEKS Fargate environment
kube_runtime_classLowPod specPod must be attached to a runtime class
gpu_vendorLowPod specContainer must be attached to a GPU resource

Host tag

The Agent can attach Kubernetes environment information as “host tags”.

TagCardinalitySourceRequirement
kube_cluster_nameLowDD_CLUSTER_NAME envvar or cloud provider integrationDD_CLUSTER_NAME envvar or cloud provider integration enabled
kube_node_roleLowNode label node-role.kubernetes.io/<role>Node label must exist

Tag Autodiscovery

Starting with Agent v6.10+, the Agent can autodiscover tags from Pod annotations. It allows the Agent to associate tags to all data emitted by the entire pods or an individual container within this pod.

As a best practice in containerized environments, Datadog recommends using unified service tagging to help unify tags. Unified service tagging ties Datadog telemetry together through the use of three standard tags: env, service, and version. To learn how to configure your environment with unified tagging, refer to the dedicated unified service tagging documentation.

To apply a <TAG_KEY>:<TAG_VALUE> tag to all data emitted by a given pod and collected by the Agent use the following annotation on your pod:

annotations:
  ad.datadoghq.com/tags: '{"<TAG_KEY>": "<TAG_VALUE>","<TAG_KEY_1>": "<TAG_VALUE_1>"}'

If you want to apply a <TAG_KEY>:<TAG_VALUE> tag to an individual container <CONTAINER_NAME> within a pod, use the following annotation on your pod:

annotations:
  ad.datadoghq.com/<CONTAINER_NAME>.tags: '{"<TAG_KEY>": "<TAG_VALUE>","<TAG_KEY_1>": "<TAG_VALUE_1>"}'

Starting with Agent v7.17+, the Agent can Autodiscover tags from Docker labels. This process allows the Agent to associate custom tags to all data emitted by a container, without modifying the Agent configuration.

com.datadoghq.ad.tags: '["<TAG_KEY>:TAG_VALUE", "<TAG_KEY_1>:<TAG_VALUE_1>"]'

Tag extraction

Node labels as tags

Starting with Agent v6.0+, the Agent can collect labels for a given node and use them as tags to attach to all metrics, traces, and logs emitted associated with this host in Datadog:

To extract a given node label <NODE_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Operator’s DatadogAgent configuration in datadog-agent.yaml:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    nodeLabelsAsTags:
      <NODE_LABEL>: <TAG_KEY>

For example, you could set up:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    nodeLabelsAsTags:
      kubernetes.io/arch: arch

For Agent v7.24.0+, use the following environment variable configuration to add all node labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    nodeLabelsAsTags:
      "*": <PREFIX>_%%label%%

To extract a given node label <NODE_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Helm datadog-values.yaml file:

datadog:
  nodeLabelsAsTags:
    <NODE_LABEL>: <TAG_KEY>

For example, you could set up:

datadog:
  nodeLabelsAsTags:
    kubernetes.io/arch: arch

For Agent v7.24.0+, use the following environment variable configuration to add all node labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

datadog:
  nodeLabelsAsTags:
    "*": <PREFIX>_%%label%%

To extract a given node label <NODE_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following environment variable to the Datadog Agent:

DD_KUBERNETES_NODE_LABELS_AS_TAGS='{"<NODE_LABEL>": "<TAG_KEY>"}'

For example, you could set up:

DD_KUBERNETES_NODE_LABELS_AS_TAGS='{"kubernetes.io/arch":"arch"}'

For Agent v7.24.0+, use the following environment variable configuration to add all node labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

DD_KUBERNETES_NODE_LABELS_AS_TAGS='{"*":"<PREFIX>_%%label%%"}'

Note: Custom metrics may impact billing. See the custom metrics billing page for more information.

Pod labels as tags

Starting with Agent v6.0+, the Agent can collect labels for a given pod and use them as tags to attach to all metrics, traces, and logs emitted by this pod:

To extract a given pod label <POD_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Operator’s DatadogAgent configuration in datadog-agent.yaml:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    podLabelsAsTags:
      <POD_LABEL>: <TAG_KEY>

For example, you could set up:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    podLabelsAsTags:
      app: kube_app

For Agent v7.24.0+, use the following environment variable configuration to add all pod labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    podLabelsAsTags:
      "*": <PREFIX>_%%label%%

To extract a given pod label <POD_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Helm datadog-values.yaml file:

datadog:
  podLabelsAsTags:
    <POD_LABEL>: <TAG_KEY>

For example, you could set up:

datadog:
  podLabelsAsTags:
    app: kube_app

For Agent v7.24.0+, use the following environment variable configuration to add all pod labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

datadog:
  podLabelsAsTags:
    "*": <PREFIX>_%%label%%

To extract a given pod label <POD_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following environment variable to the Datadog Agent:

DD_KUBERNETES_POD_LABELS_AS_TAGS='{"<POD_LABEL>": "<TAG_KEY>"}'

For example, you could set up:

DD_KUBERNETES_POD_LABELS_AS_TAGS='{"app":"kube_app"}'

For Agent v6.8.0+, use the following environment variable configuration to add all pod labels as tags to your metrics. In this example, the tags names are prefixed by <PREFIX>_:

DD_KUBERNETES_POD_LABELS_AS_TAGS='{"*":"<PREFIX>_%%label%%"}'

Note: Custom metrics may impact billing. See the custom metrics billing page for more information.

Pod annotations as tags

Starting with Agent v6.0+, the Agent can collect annotations for a given pod and use them as tags to attach to all metrics, traces, and logs emitted by this pod:

To extract a given pod annotation <POD_ANNOTATION> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Operator’s DatadogAgent configuration in datadog-agent.yaml

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    podAnnotationsAsTags:
      <POD_ANNOTATION>: <TAG_KEY>

For example, you could set up:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    podAnnotationsAsTags:
      app: kube_app

For Agent v7.24.0+, use the following environment variable configuration to add all pod annotations as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    podAnnotationsAsTags:
      "*": <PREFIX>_%%label%%

To extract a given pod annotation <POD_ANNOTATION> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Helm datadog-values.yaml file:

datadog:
  podAnnotationsAsTags:
    <POD_ANNOTATION>: <TAG_KEY>

For example, you could set up:

datadog:
  podAnnotationsAsTags:
    app: kube_app

For Agent v7.24.0+, use the following environment variable configuration to add all pod annotation as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

datadog:
  podAnnotationsAsTags:
    "*": <PREFIX>_%%label%%

To extract a given pod annotation <POD_ANNOTATION> and transform it as a tag key <TAG_KEY> within Datadog, add the following environment variable to the Datadog Agent:

DD_KUBERNETES_POD_ANNOTATIONS_AS_TAGS='{"<POD_ANNOTATION>": "<TAG_KEY>"}'

For example, you could set up:

DD_KUBERNETES_POD_ANNOTATIONS_AS_TAGS='{"app":"kube_app"}'

For Agent v7.24.0+, use the following environment variable configuration to add all pod annotations as tags to your metrics. In this example, the tags names are prefixed by <PREFIX>_:

DD_KUBERNETES_POD_ANNOTATIONS_AS_TAGS='{"*":"<PREFIX>_%%annotation%%"}'

Note: Custom metrics may impact billing. See the custom metrics billing page for more information.

Namespace labels as tags

Starting with Agent 7.55.0+, the Agent can collect labels for a given namespace and use them as tags to attach to all metrics, traces, and logs emitted by all pods in this namespace:

To extract a given namespace label <NAMESPACE_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Operator’s DatadogAgent configuration in datadog-agent.yaml:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    namespaceLabelsAsTags:
      <NAMESPACE_LABEL>: <TAG_KEY>

For example, you could set up:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    namespaceLabelsAsTags:
      app: kube_app

For Agent v7.24.0+, use the following environment variable configuration to add all namespace labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    namespaceLabelsAsTags:
      "*": <PREFIX>_%%label%%

To extract a given namespace label <NAMESPACE_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Helm datadog-values.yaml file:

datadog:
  namespaceLabelsAsTags:
    <NAMESPACE_LABEL>: <TAG_KEY>

For example, you could set up:

datadog:
  namespaceLabelsAsTags:
    app: kube_app

For Agent v7.24.0+, use the following environment variable configuration to add all namespace labels as tags to your metrics. In this example, the tags’ names are prefixed by <PREFIX>_:

datadog:
  namespaceLabelsAsTags:
    "*": <PREFIX>_%%label%%

To extract a given namespace label <NAMESPACE_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following environment variable to the Datadog Agent:

DD_KUBERNETES_NAMESPACE_LABELS_AS_TAGS='{"<NAMESPACE_LABEL>": "<TAG_KEY>"}'

For example, you could set up:

DD_KUBERNETES_NAMESPACE_LABELS_AS_TAGS='{"app":"kube_app"}'

Use the following environment variable configuration to add all namespace labels as tags to your metrics. In this example, the tag names are prefixed by <PREFIX>_:

DD_KUBERNETES_NAMESPACE_LABELS_AS_TAGS='{"*":"<PREFIX>_%%label%%"}'

Note: Custom metrics may impact billing. See the custom metrics billing page for more information.

Container environment variables as tags

Starting with Agent v7.32+, the Agent can collect container environment variables and use them as tags to attach to all metrics, traces, and logs corresponding to the container. Both docker and containerd containers are supported:

To extract a given environment variable <ENV_VAR> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Operator’s DatadogAgent configuration in datadog-agent.yaml:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      env:
        - name: DD_CONTAINER_ENV_AS_TAGS
          value: '{"<ENV_VAR>": "<TAG_KEY>"}'

For example, you could set up:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      env:
        - name: DD_CONTAINER_ENV_AS_TAGS
          value: '{"app":"kube_app"}'

To extract a given environment variable <ENV_VAR> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Helm datadog-values.yaml file:

datadog:
  env:
    - name: DD_CONTAINER_ENV_AS_TAGS
      value: '{"<ENV_VAR>": "<TAG_KEY>"}'

For example, you could set up:

datadog:
  env:
    - name: DD_CONTAINER_ENV_AS_TAGS
      value: '{"app":"kube_app"}'

To extract a given environment variable <ENV_VAR> and transform it as a tag key <TAG_KEY> within Datadog, add the following environment variable to the Datadog Agent:

DD_CONTAINER_ENV_AS_TAGS='{"<ENV_VAR>": "<TAG_KEY>"}'

For example:

DD_CONTAINER_ENV_AS_TAGS='{"app":"kube_app"}'

Note: Custom metrics may impact billing. See Custom Metrics Billing for more details.

Container labels as tags

Starting with Agent v7.33+, the Agent can collect container labels and use them as tags. The agent attaches the tags to all metrics, traces, and logs associated with the container.

The Agent can generate tags from container labels for both docker and containerd containers. In the case of containerd, the minimum supported version is v1.5.6, because previous releases do not propagate labels correctly.

To extract a given container label <CONTAINER_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Operator’s DatadogAgent configuration in datadog-agent.yaml:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      env:
        - name: DD_CONTAINER_LABELS_AS_TAGS
          value: '{"<CONTAINER_LABEL>": "<TAG_KEY>"}'

For example, you could set up:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      env:
        - name: DD_CONTAINER_LABELS_AS_TAGS
          value: '{"app":"kube_app"}'

To extract a given container label <CONTAINER_LABEL> and transform it as a tag key <TAG_KEY> within Datadog, add the following configuration to your Helm datadog-values.yaml file:

datadog:
  env:
    - name: DD_CONTAINER_LABELS_AS_TAGS
      value: '{"<CONTAINER_LABEL>": "<TAG_KEY>"}'

For example, you could set up:

datadog:
  env:
    - name: DD_CONTAINER_LABELS_AS_TAGS
      value: '{"app":"kube_app"}'

To extract a given container label <CONTAINER_LABEL> and transform it to a tag key <TAG_KEY>, add the following environment variable to the Datadog Agent:

DD_CONTAINER_LABELS_AS_TAGS='{"<CONTAINER_LABEL>":"<TAG_KEY>"}'

For example:

DD_CONTAINER_LABELS_AS_TAGS='{"app":"kube_app"}'

Note: Custom metrics may impact billing. See Custom Metrics Billing for more details.

Further Reading

PREVIEWING: rtrieu/product-analytics-ui-changes