Overview

This integration monitors your Cloudera Data Platform through the Datadog Agent, allowing you to submit metrics and service checks on the health of your Cloudera Data Hub clusters, hosts, and roles.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

The Cloudera check is included in the Datadog Agent package. No additional installation is needed on your server.

Configuration

Requirements

The Cloudera check requires version 7 of Cloudera Manager.

Prepare Cloudera Manager

1. In Cloudera Data Platform, navigate to the Management Console and click on the User Management tab.

2. Click on Actions, then Create Machine User to create the machine user that queries the Cloudera Manager through the Datadog Agent.

3. If the workload password hasn’t been set, click on Set Workload Password after the user is created.

Clusters Discovery

You can configure how your clusters are discovered with the clusters configuration option with the following parameters:

• limit : Maximum number of items to be autodiscovered.
  Default value: None (all clusters will be processed)

• include : Mapping of regular expression keys and component config values to autodiscover.
  Default value: empty map

• exclude : List of regular expressions with the patterns of components to exclude from autodiscovery.
  Default value: empty list

• interval : Validity time in seconds of the last list of clusters obtained through the endpoint.
  Default value: None (no cache used)

Examples:

Process a maximum of 5 clusters with names that start with my_cluster:

clusters:
  limit: 5
  include:
    - 'my_cluster.*'

Process a maximum of 20 clusters and exclude those with names that start with tmp_:

clusters:
  limit: 20
  include:
    - '.*'
  exclude:
    - 'tmp_.*'

Custom Queries

You can configure the Cloudera integration to collect custom metrics that are not be collected by default by running custom timeseries queries. These queries use the tsquery language to retrieve data from Cloudera Manager.

Example:

Collect JVM garbage collection rate and JVM free memory with cloudera_jvm as a custom tag:

custom_queries:
- query: select last(jvm_gc_rate) as jvm_gc_rate, last(jvm_free_memory) as jvm_free_memory
  tags: cloudera_jvm

Note: These queries can take advantage of metric expressions, resulting in queries such as total_cpu_user + total_cpu_system, 1000 * jvm_gc_time_ms / jvm_gc_count, and max(total_cpu_user). When using metric expressions, make sure to also include aliases for the metrics, otherwise the metric names may be incorrectly formatted. For example, SELECT last(jvm_gc_count) results in the metric cloudera.<CATEGORY>.last_jvm_gc_count. You can append an alias like in the following example: SELECT last(jvm_gc_count) as jvm_gc_count to generate the metric cloudera.<CATEGORY>.jvm_gc_count.

Validation

Run the Agent’s status subcommand and look for cloudera under the Checks section.

Data Collected

Metrics

Events

The Cloudera integration collects events that are emitted from the /events endpoint from the Cloudera Manager API. The event levels are mapped as the following:

Service Checks

cloudera.can_connect
Returns OK if the check is able to connect to the Cloudera Manager API and collect metrics, CRITICAL otherwise.
Statuses: ok, critical

cloudera.cluster.health
Returns OK if the cluster is in good health or is starting, WARNING if the cluster is stopping or the health is concerning, CRITICAL if the cluster is down or in bad health, and UNKNOWN otherwise.
Statuses: ok, critical, warning, unknown

cloudera.host.health
Returns OK if the host is in good health or is starting, WARNING if the host is stopping or the health is concerning, CRITICAL if the host is down or in bad health, and UNKNOWN otherwise.
Statuses: ok, critical, warning, unknown

Troubleshooting

Collecting metrics of Datadog integrations on Cloudera hosts

To install the Datadog Agent on a Cloudera host, make sure that the security group associated with the host allows SSH access. Then, you need to use the root user cloudbreak when accessing the host with the SSH key generated during the environment creation:

sudo ssh -i "/path/to/key.pem" cloudbreak@<HOST_IP_ADDRESS>

The workload username and password can be used to access Cloudera hosts through SSH, although only the cloudbreak user can install the Datadog Agent. Trying to use any user that is not cloudbreak may result in the following error:

<NON_CLOUDBREAK_USER> is not allowed to run sudo on <CLOUDERA_HOSTNAME>.  This incident will be reported.

Config errors when collecting Datadog metrics

If you see something similar to the following in the Agent status when collecting metrics from your Cloudera host:

  Config Errors
  ==============
    zk
    --
      open /etc/datadog-agent/conf.d/zk.d/conf.yaml: permission denied

You need to change the ownership of the conf.yaml to dd-agent:

[cloudbreak@<CLOUDERA_HOSTNAME> ~]$ sudo chown -R dd-agent:dd-agent /etc/datadog-agent/conf.d/zk.d/conf.yaml

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

• Gain visibility into your Cloudera clusters with Datadog