The Agent’s main configuration file has transitioned from INI to YAML format to support complex configurations and provide a consistent experience across the Agent and checks.
Agent v5 datadog.conf –> Agent v6 datadog.yaml
To transition between Agent configuration paths and formats, use the Agent command:
sudo -u dd-agent -- datadog-agent import
This command parses an existing datadog.conf and converts supported parameters to the new format in datadog.yaml. The command also copies configuration files for checks that are enabled. For more details, see Upgrade to Datadog Agent v6.
The following Agent configuration options were changed or removed in Agent v6. Configuration options removed were either superseded by other options, or related to features that work differently from previous versions.
Agent v6 loads any valid YAML file in: <AGENT_DIRECTORY>/conf.d/<CHECK_NAME>.d/. This enables complex configurations to be broken down into multiple files.
For example, configuration files for the http_check could be:
Autodiscovery template files (auto_conf.yaml) are stored in the configuration folder as well. This is an example of the redisdb check configuration folder:
The YAML files within a <CHECK_NAME>.d folder can have any name, as long as they have a .yaml or .yml extension. The standard name is conf.yaml.
To keep backwards compatibility, the Agent picks up configuration files in the form: <AGENT_DIRECTORY>/conf.d/<CHECK_NAME>.yaml but migrating to the updated layout is strongly recommended.
Agent v6 supports the following options in a check’s instance section:
Option
Description
min_collection_interval
Set a different run interval in seconds, for checks that should run less frequently than the default, 15-second interval.
empty_default_hostname
Submit metrics, events, and service checks with no hostname when set to true.
tags
Send custom tags in addition to the tags sent by the check.
Most of the environment variables used in Agent v6 are different from previous versions. See the list of environment variables for Agent v6.
Note: DD_TAGS is the same tag but in Agent v6, the format is space-separated. Previous versions were comma-separated, example for v6: DD_TAGS="simple-tag-0 tag-key-1:tag-value-1"
For v6.4.0+, the Agent proxy settings can be overridden with the following environment variables:
Env variable
Description
DD_PROXY_HTTP
The URL to use as a proxy for http requests.
DD_PROXY_HTTPS
The URL to use as a proxy for https requests.
DD_PROXY_NO_PROXY
A space-separated list of URLs for which no proxy should be used.
The standard environment variables (HTTP_PROXY, HTTPS_PROXY, and NO_PROXY) are supported in Agent v6, but using the DD_PROXY_* variables is recommended. DD_PROXY_* variables have precedence over the other proxy variables.
The precedence order of Agent v6 proxy options is different from previous versions:
Agent v6 uses the environment variables first and then the configuration file.
Agent v6 overrides the values from the configuration file with the ones in the environment. For example, if both proxy.http and proxy.https are set in the configuration file but only DD_PROXY_HTTPS is set in the environment, the Agent uses the https value from the environment and the http value from the configuration file.
Only the Agent’s lifecycle commands (start/stop/restart/status) should be run with sudo service/sudo initctl/sudo systemctl.
All other commands must be invoked on the Agent binary, located in the PATH (/usr/bin) as datadog-agent by default. The dd-agent command is no longer available.
The info sub-command has been renamed to status.
Agent v6 does not ship a SysV-init script (previously located at /etc/init.d/datadog-agent).
With Agent v6+, other functionalities are provided by the Agent binary itself as sub-commands and shouldn’t be invoked with service/systemctl/initctl. Here are a few examples:
The Agent v5 Windows Agent Manager GUI was replaced with a browser-based, cross-platform manager. For details, see the Datadog Agent Manager for Windows.
The main executable file is agent.exe (previously ddagent.exe).
Commands should be run with the command line "%ProgramFiles%\datadog\datadog agent\embedded\agent.exe" <COMMAND> from an Administrator command prompt.
The Windows service is started as “Automatic-Delayed”. It is started automatically on boot, but after all other services. This results in a small delay in reporting metrics after a reboot.
The Windows GUI and Windows system tray icon are implemented separately. See the Datadog Agent Manager for Windows for more details.
The major changes for Agent v6 on MacOS are:
The lifecycle commands (formerly datadog-agent start/stop/restart/status) are replaced by launchctl commands on the com.datadoghq.agent service, and should be run under the logged-in user. For these commands, you can also use the Datadog Agent systray app.
All the other commands can still be run with the datadog-agent binary located in the PATH (/usr/local/bin/) by default.
The info command has been renamed to status.
The configuration GUI is a web-based application, which can be accessed by running the command datadog-agent launch-gui or using the systray app.
Example changes:
Agent v5 Command
Agent v6 Command
Description
datadog-agent start
launchctl start com.datadoghq.agent or systray app
The APM Agent is shipped by default with Agent v6 for Linux, MacOS, and Windows packages.
The APM Agent is enabled by default on Linux. To enable it on other platforms or disable it on Linux, update the apm_config key in your datadog.yaml:
apm_config:enabled:true
For the Docker image, the APM Agent is disabled by default. Enable it by setting DD_APM_ENABLED to true. It listens to all interfaces by default. If you want to listen to non-local traffic on any other platform, set DD_APM_NON_LOCAL_TRAFFIC to true. For more details, see Tracing Docker Applications.
The Process Agent is shipped by default with Agent v6 for the Linux packages only.
The Process Agent is not enabled by default. To enable it, update your datadog.yaml file with the following:
process_config:enabled:"true"
The enabled value is a string with the following options:
"true": Enable the Process Agent to collect processes and containers.
"false": Only collect containers if available (the default).
For Agent v6, Docker versions 1.12.1+ are supported.
The Docker check was rewritten in Go to take advantage of the internal architecture of the Agent. Therefore, the Python version (docker_daemon) is deprecated.
The new check is named docker. The Agent import command imports settings from the legacy docker_daemon.yaml configuration. All features are ported, except the following:
ecs_tags, performance_tags, and container_tags are deprecated. All relevant tags are collected by default.
Using collect_container_count to enable the docker.container.count metric is not supported. Use docker.containers.running and .stopped.
Some options have moved from docker_daemon.yaml to the main datadog.yaml:
collect_labels_as_tags was renamed to docker_labels_as_tags and supports high cardinality tags. See Tag Assignment and Extraction for details.
exclude and include were renamed to ac_include and ac_exclude. To make filtering consistent across all components of the Agent, there is no longer filtering on arbitrary tags. Tag filtering is only supported for image (image name) and name (container name). Regexp filtering is still available. See Container Discovery Management for details.
The docker_root option has been split in two options: container_cgroup_root and container_proc_root.
exclude_pause_container was added to exclude pause containers on Kubernetes and Openshift (defaults to true).
For Agent v6, Kubernetes versions 1.3+ are supported.
The Kubernetes integration provides insights by combining:
The Agent import command (v6.2+) imports settings from the legacy kubernetes.yaml configuration. The following options are deprecated:
API Server credentials (api_server_url, apiserver_client_crt, apiserver_client_key, apiserver_ca_cert): Instead, provide a kubeconfig file to the Agent with kubernetes_kubeconfig_path.
use_histogram: Contact Datadog support to determine the best alternative for you.
namespaces and namespace_name_regexp: Agent v6 collects metrics from all available namespaces.
The upgraded logic enables Prometheus metric collection compatible with Kubernetes versions 1.7.6+. If you run an older version or want to revert to the cadvisor collection logic, set cadvisor_port to 4194 (the port where your kubelet exposes cadvisor).
While Agent v5 automatically collected every pod label as tags, Agent v6 needs an allowlist. This is done with the kubernetes_pod_labels_as_tags option in datadog.yaml. See Tag Assignment and Extraction for details.
The following options and tags are deprecated:
label_to_tag_prefix is superseded by kubernetes_pod_labels_as_tags.
container_alias tags are not collected.
kube_replicate_controller is only added if the pod is created by a replication controller. Instead, use a relevant creator tag such as kube_deployment, or kube_daemon_set.
Agent v6 ships JMXFetch, with the following changes:
Troubleshooting command syntax has changed. These commands are available for v6.2.0+, for earlier versions, see JMX Agent troubleshooting:
Command
Description
sudo -u dd-agent datadog-agent jmx list matching
List attributes that match at least one of your instance configurations.
sudo -u dd-agent datadog-agent jmx list limited
List attributes that match one of your instance configurations but are not being collected because it would exceed the number of metrics that can be collected.
sudo -u dd-agent datadog-agent jmx list collected
List attributes that are collected by your current instances configuration.
sudo -u dd-agent datadog-agent jmx list not-matching
List attributes that don’t match any of your instance configurations.
sudo -u dd-agent datadog-agent jmx list everything
List every attribute available that has a type supported by JMXFetch.
sudo -u dd-agent datadog-agent jmx collect
Start the collection of metrics based on your current configuration and display them in the console.
Note: By default, these commands run on all the configured JMX checks. To specify a check, use the --checks flag, for example:
sudo datadog-agent jmx list collected --checks tomcat
Only affects Windows Agents
For the Windows Agent v5, system.mem.pagefile.* metrics display inconsistent units (off by 10^6).
This problem is fixed on the Windows Agent v6. However, the Agent v5 discrepancy remains for backwards compatibility. Therefore, reported values (and associated monitors) are different when upgrading from Agent v5 to Agent v6.
The Autodiscovery system was reworked for Agent v6. Also, container runtimes and orchestrators were decoupled to be more flexible. This includes the move from docker_images to ad_identifiers in templates.
When using Kubernetes, Autodiscovery sources information from the kubelet instead of Docker daemon. This allows Autodiscovery to work without access to the Docker socket. Also, the default behavior is to source Autodiscovery templates from pod annotations. You can enable the docker config-provider to use container labels, and replace the kubelet listener with the Docker one if you need Autodiscovery on containers running out of pods.
When specifying Autodiscovery templates in pod annotations, the annotation name prefix is ad.datadoghq.com/. The previous annotation prefix (service-discovery.datadoghq.com/) is still supported for Agent v6 but support is set to be removed in future versions.
The identifier override label has been renamed from com.datadoghq.sd.check.id to com.datadoghq.ad.check.id for consistency. The previous name is still supported for Agent v6 but support is set to be removed in future versions.
For Agent v6, all check-related Python code is imported from the datadog_checksnamespace. Most Python libraries included with Agent v5 are shipped with Agent v6. The following has changed:
util.py and its associated functions were removed from the Agent v6.
util.headers(...) is still included with Agent v6, but implemented in C and Go and passed through to the check.
Note: All official integrations were updated to remove obsolete modules, so these changes only affect custom checks.
A lot of the utils directory was removed from Agent v6, but most of the removed content was not directly related to checks. The flare module, for example, was removed and reimplemented in Go, but is uncommon to have been used by anyone in a custom check. For more details, read the custom checks developer guide.
The base class for Python checks (AgentCheck) is imported from datadog_checks.base.checks. There are several things that have been removed or changed in the class API. In addition, each check instance is its own instance of the class. So you cannot share state between them.
The following methods in the AgentCheck class are not implemented:
For Agent v6, the official checks take precedence over custom checks (checks in <AGENT_DIRECTORY>/checks.d). Custom checks with the same name as official checks are ignored.
To fix your custom check setup with Agent v6, rename affected custom checks to a new, unused name, and rename the related .yaml configuration files accordingly.
If you use custom checks, there’s a chance your code depends on Python code no longer bundled with Agent v6. These packages are no longer bundled with the Agent:
backports.ssl-match-hostname
datadog
decorator
future
futures
google-apputils
pycurl
pyOpenSSL
python-consul
python-dateutil
python-etcd
python-gflags
pytz
PyYAML
rancher-metadata
tornado
uptime
websocket-client
If your code depends on any of these packages, install missing packages by running:
Similarly, you may have added a PIP package to meet a requirement for a custom check while on Agent v5. If the added PIP package had inner dependencies with packages already bundled with Agent v5 (see list above), those dependencies are missing after upgrading to Agent v6. Install the missing dependencies as described above.
