Overview

This check monitors TorchServe through the Datadog Agent.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

Starting from Agent release 7.47.0, the TorchServe check is included in the Datadog Agent package. No additional installation is needed on your server.

Prerequisites

The TorchServe check collects TorchServe’s metrics and performance data using three different endpoints:

• The Inference API to collect the overall health status of your TorchServe instance.
• The Management API to collect metrics on the various models you are running.
• The OpenMetrics endpoint exposed by TorchServe.

You can configure these endpoints using the config.properties file, as described in the TorchServe documentation. For example:

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
metrics_mode=prometheus
number_of_netty_threads=32
default_workers_per_model=10
job_queue_size=1000
model_store=/home/model-server/model-store
workflow_store=/home/model-server/wf-store
load_models=all

This configuration file exposes the three different endpoints that can be used by the integration to monitor your instance.

OpenMetrics endpoint

To enable the Prometheus endpoint, you need to configure two options:

• metrics_address: Metrics API binding address. Defaults to http://127.0.0.1:8082
• metrics_mode: Two metric modes are supported by TorchServe: log and prometheus. Defaults to log. You have to set it to prometheus to collect metrics from this endpoint.

For instance:

metrics_address=http://0.0.0.0:8082
metrics_mode=prometheus

In this case, the OpenMetrics endpoint is exposed at this URL: http://<TORCHSERVE_ADDRESS>:8082/metrics.

Configuration

These three different endpoints can be monitored independently and must be configured separately in the configuration file, one API per instance. See the sample torchserve.d/conf.yaml for all available configuration options.

init_config:
  ...
instances:
  - openmetrics_endpoint: http://<TORCHSERVE_ADDRESS>:8082/metrics

init_config:
  ...
instances:
  - inference_api_url: http://<TORCHSERVE_ADDRESS>:8080

init_config:
  ...
instances:
  - management_api_url: http://<TORCHSERVE_ADDRESS>:8081

init_config:
  ...
instances:
  - management_api_url: http://<TORCHSERVE_ADDRESS>:8081
    limit: 25
    include: 
      - my_model.* 

init_config:
  ...
instances:
  - management_api_url: http://<TORCHSERVE_ADDRESS>:8081
    exclude: 
      - test.* 

Complete configuration

init_config:
  ...
instances:
  - openmetrics_endpoint: http://<TORCHSERVE_ADDRESS>:8082/metrics
    # Also collect your own TorchServe metrics
    extra_metrics:
      - my_custom_torchserve_metric
  - inference_api_url: http://<TORCHSERVE_ADDRESS>:8080
  - management_api_url: http://<TORCHSERVE_ADDRESS>:8081
    # Include all the model names that match this regex   
    include:
      - my_models.*
    # But exclude all the ones that finish with `-test`
    exclude: 
      - .*-test 
    # Refresh the list of models only every hour
    interval: 3600

labels:
  com.datadoghq.ad.checks: '{"torchserve":{"instances":[{"openmetrics_endpoint":"http://%%host%%:8082/metrics","extra_metrics":["my_custom_torchserve_metric"]},{"inference_api_url":"http://%%host%%:8080"},{"management_api_url":"http://%%host%%:8081","include":["my_models.*"],"exclude":[".*-test"],"interval":3600}]}}'

apiVersion: v1
kind: Pod
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/torchserve.checks: |-
      {
        "torchserve": {
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:8082/metrics",
              "extra_metrics": [
                "my_custom_torchserve_metric"
              ]
            },
            {
              "inference_api_url": "http://%%host%%:8080"
            },
            {
              "management_api_url": "http://%%host%%:8081",
              "include": [
                ".*"
              ],
              "exclude": [
                ".*-test"
              ],
              "interval": 3600
            }
          ]
        }
      }      
    # (...)
spec:
  containers:
    - name: 'torchserve'
# (...)

Validation

Run the Agent’s status subcommand and look for torchserve under the Checks section.

Data Collected

Metrics

Metrics are prefixed using the API they are coming from:

• torchserve.openmetrics.* for metrics coming from the OpenMetrics endpoint.
• torchserve.inference_api.* for metrics coming from the Inference API.
• torchserve.management_api.* for metrics coming from the Management API.

Events

The TorchServe integration include three events using the Management API:

• torchserve.management_api.model_added: This event fires when a new model has been added.
• torchserve.management_api.model_removed: This event fires when a model has been removed.
• torchserve.management_api.default_version_changed: This event fires when a default version has been set for a given model.

Service Checks

torchserve.openmetrics.health
Returns CRITICAL if the Agent is unable to connect to the OpenMetrics endpoint, otherwise returns OK.
Statuses: ok, critical

torchserve.inference_api.health
Returns CRITICAL if the Agent is unable to connect to the Inference API endpoint or if it is unhealthy, otherwise returns OK.
Statuses: ok, critical

torchserve.management_api.health
Returns CRITICAL if the Agent is unable to connect to the Management API endpoint, otherwise returns OK.
Statuses: ok, critical

Logs

The TorchServe integration can collect logs from the TorchServe service and forward them to Datadog.

1. Collecting logs is disabled by default in the Datadog Agent. Enable it in your datadog.yaml file:

   logs_enabled: true
   

2. Uncomment and edit the logs configuration block in your torchserve.d/conf.yaml file. Here’s an example:

   logs:
     - type: file
       path: /var/log/torchserve/model_log.log
       source: torchserve
       service: torchserve
     - type: file
       path: /var/log/torchserve/ts_log.log
       source: torchserve
       service: torchserve
   

See the example configuration file on how to collect all logs.

For more information about the logging configuration with TorchServe, see the official TorchServe documentation.

Troubleshooting

Need help? Contact Datadog support.