Overview

This check monitors Ray through the Datadog Agent. Ray is an open-source unified compute framework that makes it easy to scale AI and Python workloads, from reinforcement learning to deep learning to tuning, and model serving.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

Starting from Agent release 7.49.0, the Ray check is included in the Datadog Agent package. No additional installation is needed on your server.

WARNING: This check uses OpenMetrics to collect metrics from the OpenMetrics endpoint Ray can expose, which requires Python 3.

Configuration

labels:
  com.datadoghq.ad.checks: '{"ray":{"instances":[{"openmetrics_endpoint":"http://%%host%%:8080"}]}}'

apiVersion: v1
kind: Pod
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/ray.checks: |-
      {
        "ray": {
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:8080"
            }
          ]
        }
      }      
    # (...)
spec:
  containers:
    - name: 'ray'
# (...)

Ray metrics are available on the OpenMetrics endpoint. Additionally, Ray allows you to export custom application-level metrics. You can configure the Ray integration to collect these metrics using the extra_metrics option. All Ray metrics, including your custom metrics, use the ray. prefix.

Note: Custom Ray metrics are considered standard metrics in Datadog.

This example demonstrates a configuration leveraging the extra_metrics option:

init_config:
  ...
instances:
  - openmetrics_endpoint: http://<RAY_ADDRESS>:8080
    # Also collect your own Ray metrics
    extra_metrics:
      - my_custom_ray_metric

More info on how to configure this option can be found in the sample ray.d/conf.yaml configuration file.

Validation

Run the Agent’s status subcommand and look for ray under the Checks section.

Data Collected

Metrics

Events

The Ray integration does not include any events.

Service Checks

ray.openmetrics.health
Returns CRITICAL if the check cannot access the openmetrics metrics endpoint of Ray.
Statuses: ok, critical

Logs

The Ray integration can collect logs from the Ray service and forward them to Datadog.

apiVersion: v1
kind: Pod
metadata:
  name: ray
  annotations:
    ad.datadoghq.com/apache.logs: '[{"source":"ray","service":"ray"}]'
spec:
  containers:
    - name: ray

For more information about the logging configuration with Ray and all the log files, see the official Ray documentation.

Troubleshooting

Need help? Contact Datadog support.