Single Step APM Instrumentation on Kubernetes

Overview

In a Kubernetes environment, use Single Step Instrumentation (SSI) for APM to install the Datadog Agent and instrument your applications in one step, with no additional configuration required.

Requirements

  • Kubernetes v1.20+
  • Helm for deploying the Datadog Operator.
  • Kubectl CLI for installing the Datadog Agent.
Single Step Instrumentation for Kubernetes is GA for Agent versions 7.64+, and in Preview for Agent versions <=7.63.

Enable APM on your applications

You can enable APM by installing the Agent with either:

  • Datadog Operator
  • Datadog Helm chart
Single Step Instrumentation doesn't instrument applications in the namespace where you install the Datadog Agent. It's recommended to install the Agent in a separate namespace in your cluster where you don't run your applications.

    Follow these steps to enable Single Step Instrumentation across your entire cluster with the Datadog Operator. This automatically sends traces for all applications in the cluster that are written in supported languages.

    Note: To configure Single Step Instrumentation for specific namespace or pods, see Advanced options.

    To enable Single Step Instrumentation with the Datadog Operator:

    1. Install the Datadog Operator with Helm:

      helm repo add datadog https://helm.datadoghq.com
helm repo update
helm install my-datadog-operator datadog/datadog-operator

    2. Create a Kubernetes secret to store your Datadog API key:

      kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY

    3. Create datadog-agent.yaml with the spec of your Datadog Agent deployment configuration. The simplest configuration is as follows:

      apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  override:
    clusterAgent:
      image:
        tag: 7.64.1
  global:
    site: <DATADOG_SITE>
    tags:
      - env:<AGENT_ENV>
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
  features:
    apm:
      instrumentation:
        enabled: true
        targets:
          - name: "default-target"
            ddTracerVersions:
              java: "1"
              dotnet: "3"
              python: "2"
              js: "5"
              php: "1"

      Replace <DATADOG_SITE> with your Datadog site and <AGENT_ENV> with the environment your Agent is installed on (for example, env:staging).

      See Advanced options for more options.

    4. Run the following command:

      kubectl apply -f /path/to/your/datadog-agent.yaml

    5. After waiting a few minutes for the Datadog Cluster Agent changes to apply, restart your applications.

    Follow these steps to enable Single Step Instrumentation across your entire cluster with Helm. This automatically sends traces for all applications in the cluster that are written in supported languages.

    Note: To configure Single Step Instrumentation for specific namespace or pods, see Advanced options.

    To enable Single Step Instrumentation with Helm:

    1. Add the Helm Datadog repo:

       helm repo add datadog https://helm.datadoghq.com
 helm repo update

    2. Create a Kubernetes secret to store your Datadog API key:

      kubectl create secret generic datadog-secret --from-literal api-key=$DD_API_KEY

    3. Create datadog-values.yaml and add the following configuration:

      datadog:
 apiKeyExistingSecret: datadog-secret
 site: <DATADOG_SITE>
 tags:
      - env:<AGENT_ENV>
 apm:
   instrumentation:
      enabled: true
      targets:
        - name: "default-target"
          ddTracerVersions:
            java: "1"
            dotnet: "3"
            python: "2"
            js: "5"
            php: "1"

      Replace <DATADOG_SITE> with your Datadog site and <AGENT_ENV> with the environment your Agent is installed on (for example, env:staging).

      See Advanced options for more options.

    4. Run the following command:

      helm install datadog-agent -f datadog-values.yaml datadog/datadog

    5. After waiting a few minutes for the Datadog Cluster Agent changes to apply, restart your applications.

    After you complete these steps, you may want to enable runtime metrics or view observability data from your application in the Software Catalog.

    Advanced options

    When you run the one-line installation command, there are a few options to customize your experience:

      Configure instrumentation for namespaces and pods

      By default, Single Step Instrumentation instruments all services in all namespaces in your cluster. Alternatively, you can create targeting blocks with the targets label to specify which workloads to instrument and what configurations to apply.

      Each target block has the following keys:

      KeyDescription
      nameThe name of the target block. This has no effect on monitoring state and is used only as metadata.
      namespaceSelectorThe namespace(s) to instrument. Specify using one or more of:
      - matchNames: A list of one or more namespace name(s).
      - matchLabels: A list of one or more label(s) defined in {key,value} pairs.
      - matchExpressions: A list of namespace selector requirements.

      Namespaces must meet all criteria to match. For more details, see the Kubernetes selector documentation.
      podSelectorThe pod(s) to instrument. Specify using one or more of:
      - matchLabels: A list of one or more label(s) defined in {key,value} pairs.
      - matchExpressions: A list of pod selector requirements.

      Pods must meet all criteria to match. For more details, see the Kubernetes selector documentation.
      ddTraceVersionsThe Datadog APM SDK version to use for each language.
      ddTraceConfigsAPM SDK configs that allow setting Unified Service Tags, enabling Datadog products beyond tracing, and customizing other APM settings. See full list of options.

      The file you need to configure depends on how you enabled Single Step Instrumentation:

      • If you enabled SSI with Datadog Operator, edit datadog-agent.yaml.
      • If you enabled SSI with Helm, edit datadog-values.yaml.

      Example configurations

      Review the following examples demonstrating how to select specific services:

      This configuration:

      • enables APM for all namespaces except the jenkins namespace.
      • instructs Datadog to instrument the Java applications with the default Java APM SDK and Python applications with v.3.1.0 of the Python APM SDK.
         apm:
     instrumentation:
       enabled: true
       disabledNamespaces:
         - "jenkins"
       targets:
         - name: "all-remaining-services"
           ddTraceVersions:
             java: "default"
             python: "3.1.0"

      This configuration creates two targets blocks:

      • The first block (named login-service_namespace):
        • enables APM for services in the namespace login-service.
        • instructs Datadog to instrument services in this namespace with the default version of the Java APM SDK.
        • sets environment variables – DD_SERVICE, DD_ENV, and DD_PROFILING_ENABLED – for this target group.
      • The second block (named billing-service_apps)
        • enables APM for services in the namespace(s) with label app:billing-service.
        • instructs Datadog to instrument this set of services with v3.1.0 of the Python APM SDK.
        • sets environment variables – DD_SERVICE and DD_ENV – for this target group.
        apm:
    instrumentation:
      enabled: true
      targets:
        - name: "login-service_namespace"
          namespaceSelector:
            matchNames:
              - "login-service"
          ddTraceVersions:
            java: "default"
          ddTraceConfigs:
            - name: "DD_SERVICE"
              value: "login-service"
            - name: "DD_ENV"
              value: "prod"
            - name: "DD_PROFILING_ENABLED"  ## profiling is enabled for all services in this namespace
              value: "auto"
        - name: "billing-service_apps"
          namespaceSelector:
            matchLabels:
              app: "billing-service"
          ddTraceVersions:
            python: "3.1.0"
          ddTraceConfigs:
            - name: "DD_SERVICE"
              value: "billing-service"
            - name: "DD_ENV"
              value: "prod

      This configuration does the following:

      • enables APM for pods with the following labels:
        • app:db-user, which marks pods running the db-user application.
        • webserver:routing, which marks pods running the request-router application.
      • instructs Datadog to use the default versions of the Datadog Tracer SDKs.
      • sets several Datadog environment variables to apply to each target group.
         apm:
     instrumentation:
       enabled: true
       targets:
         - name: "db-user"
           podSelector:
             matchLabels:
               app: "db-user"
           ddTraceVersions:
             java: "default"
           ddTraceConfigs:   ## trace configs set for services in matching pods
             - name: "DD_SERVICE"
               value: "db-user"
             - name: "DD_ENV"
               value: "prod"
             - name: "DD_DSM_ENABLED"
               value: "true"
         - name: "user-request-router"
           podSelector:
             matchLabels:
               webserver: "user"
           ddTraceVersions:
             php: "default"
           ddTraceConfigs:
             - name: "DD_SERVICE"
               value: "user-request-router"
             - name: "DD_ENV"
               value: "prod

      This configuration:

      • enables APM for pods labeled app:password-resolver inside the login-service namespace.
      • instructs Datadog to use the default version of the Datadog Java Tracer SDK.
      • sets several Datadog environment variables to apply to this target.
         apm:
     instrumentation:
       enabled: true
       targets:
         - name: "login-service-namespace"
           namespaceSelector:
             matchNames:
               - "login-service"
           podSelector:
             matchLabels:
               app: "password-resolver"
           ddTraceVersions:
             java: "default"
           ddTraceConfigs:
             - name: "DD_SERVICE"
               value: "password-resolver"
             - name: "DD_ENV"
               value: "prod"
             - name: "DD_PROFILING_ENABLED"
               value: "auto"

      This configuration enables APM for all pods except those that have either of the labels app=app1 or app=app2.

         apm:
     instrumentation:
       enabled: true
       targets:
         - name: "default-target"
           matchExpressions:
             - key: app
               operator: NotIn
               values:
               - app1
               - app2

      Enable or disable instrumentation for namespaces

      You can choose to enable or disable instrumentation for applications in specific namespaces. You can only set enabledNamespaces or disabledNamespaces, not both.

      The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:

      To enable instrumentation for specific namespaces, add enabledNamespaces configuration to datadog-agent.yaml:

         features:
     apm:
       instrumentation:
         enabled: true
         enabledNamespaces: # Add namespaces to instrument
           - default
           - applications

      To disable instrumentation for specific namespaces, add disabledNamespaces configuration to datadog-agent.yaml:

         features:
     apm:
       instrumentation:
         enabled: true
         disabledNamespaces: # Add namespaces to not instrument
           - default
           - applications

      To enable instrumentation for specific namespaces, add enabledNamespaces configuration to datadog-values.yaml:

         datadog:
      apm:
        instrumentation:
          enabled: true
          enabledNamespaces: # Add namespaces to instrument
             - namespace_1
             - namespace_2

      To disable instrumentation for specific namespaces, add disabledNamespaces configuration to datadog-values.yaml:

         datadog:
      apm:
        instrumentation:
          enabled: true
          disabledNamespaces: # Add namespaces to not instrument
            - namespace_1
            - namespace_2

      Specify tracing library versions

      Starting with Datadog Cluster Agent v7.52.0+, you can automatically instrument a subset of your applications, based on the tracing libraries you specify.

      Specify Datadog tracing libraries and their versions to automatically instrument applications written in those languages. You can configure this in two ways, which are applied in the following order of precedence:

      1. Specify at the service level, or
      2. Specify at the cluster level.

      Default: If you don’t specify any library versions and apm.instrumentation.enabled=true, applications written in supported languages are automatically instrumented using the latest tracing library versions.

      Specify at the service level

      To automatically instrument applications in specific pods, add the appropriate language annotation and library version for your application in your pod spec:

      LanguagePod annotation
      Javaadmission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
      Node.jsadmission.datadoghq.com/js-lib.version: "<CONTAINER IMAGE TAG>"
      Pythonadmission.datadoghq.com/python-lib.version: "<CONTAINER IMAGE TAG>"
      .NETadmission.datadoghq.com/dotnet-lib.version: "<CONTAINER IMAGE TAG>"
      Rubyadmission.datadoghq.com/ruby-lib.version: "<CONTAINER IMAGE TAG>"
      PHPadmission.datadoghq.com/php-lib.version: "<CONTAINER IMAGE TAG>"

      Replace <CONTAINER IMAGE TAG> with the desired library version. Available versions are listed in the Datadog container registries and tracer source repositories for each language:

      Exercise caution when using the latest tag, as major library releases may introduce breaking changes.

      For example, to automatically instrument Java applications:

      apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    # ...
spec:
  template:
    metadata:
      annotations:
        admission.datadoghq.com/java-lib.version: "<CONTAINER IMAGE TAG>"
    spec:
      containers:
        - # ...

      Specify at the cluster level

      If you don’t enable automatic instrumentation for specific pods using annotations, you can specify which languages to instrument across the entire cluster using the Single Step Instrumentation configuration. When apm.instrumentation.libVersions is set, only applications written in the specified languages will be instrumented, using the specified library versions.

      The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:

      For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-agent.yaml file:

         features:
     apm:
       instrumentation:
         enabled: true
         libVersions: # Add any libraries and versions you want to set
            dotnet: "3.2.0"
            python: "1.20.6"
            js: "4.17.0"

      For example, to instrument .NET, Python, and Node.js applications, add the following configuration to your datadog-values.yaml file:

         datadog:
     apm:
       instrumentation:
         enabled: true
         libVersions: # Add any libraries and versions you want to set
            dotnet: "3.2.0"
            python: "1.20.6"
            js: "4.17.0"

      Container registries

      Datadog publishes instrumentation libraries images on gcr.io, Docker Hub, and Amazon ECR:

      Languagegcr.iohub.docker.comgallery.ecr.aws
      Javagcr.io/datadoghq/dd-lib-java-inithub.docker.com/r/datadog/dd-lib-java-initgallery.ecr.aws/datadog/dd-lib-java-init
      Node.jsgcr.io/datadoghq/dd-lib-js-inithub.docker.com/r/datadog/dd-lib-js-initgallery.ecr.aws/datadog/dd-lib-js-init
      Pythongcr.io/datadoghq/dd-lib-python-inithub.docker.com/r/datadog/dd-lib-python-initgallery.ecr.aws/datadog/dd-lib-python-init
      .NETgcr.io/datadoghq/dd-lib-dotnet-inithub.docker.com/r/datadog/dd-lib-dotnet-initgallery.ecr.aws/datadog/dd-lib-dotnet-init
      Rubygcr.io/datadoghq/dd-lib-ruby-inithub.docker.com/r/datadog/dd-lib-ruby-initgallery.ecr.aws/datadog/dd-lib-ruby-init
      PHPgcr.io/datadoghq/dd-lib-php-inithub.docker.com/r/datadog/dd-lib-php-initgallery.ecr.aws/datadog/dd-lib-php-init

      The DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_CONTAINER_REGISTRY environment variable in the Datadog Cluster Agent configuration specifies the registry used by the Admission Controller. The default value is gcr.io/datadoghq.

      You can pull the tracing library from a different registry by changing it to docker.io/datadog, public.ecr.aws/datadog, or another URL if you are hosting the images in a local container registry.

      For instructions on changing your container registry, see Changing Your Container Registry.

      Remove Single Step APM instrumentation from your Agent

      If you don’t want to collect trace data for a particular service, host, VM, or container, complete the following steps:

      Remove instrumentation for specific services

      To remove APM instrumentation and stop sending traces from a specific service, you can do one of the following:

      With workload selection (available for Agent v7.64+), you can enable and disable tracing for specific applications. See configuration details here.

      Use the Datadog Admission Controller

      As an alternative, or for a version of the agent that does not support workload selection, you can also disable pod mutation by adding a label to your pod.

      In addition to disabling SSI, the following steps disable other mutating webhooks. Use with caution.
      1. Set the admission.datadoghq.com/enabled: label to "false" for the pod spec:
        spec:
  template:
    metadata:
      labels:
        admission.datadoghq.com/enabled: "false"
      2. Apply the configuration:
        kubectl apply -f /path/to/your/deployment.yaml
      3. Restart the services you want to remove instrumentation for.

      Remove APM for all services on the infrastructure

      To stop producing traces, uninstall APM and restart the infrastructure:

      The file you need to configure depends on if you enabled Single Step Instrumentation with Datadog Operator or Helm:

        1. Set instrumentation.enabled=false in datadog-agent.yaml:

          features:
  apm:
    instrumentation:
      enabled: false

        2. Deploy the Datadog Agent with the updated configuration file:

          kubectl apply -f /path/to/your/datadog-agent.yaml

        1. Set instrumentation.enabled=false in datadog-values.yaml:

          datadog:
  apm:
    instrumentation:
      enabled: false

        2. Run the following command:

          helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog

        Further reading

        Additional helpful documentation, links, and articles:

        Enable Runtime MetricsDOCUMENTATION more more
        PREVIEWING: domalessi/docs-10444