Previous

Next

Fiddler Dashboard in DataDog

Model Accuracy Chart

Model Analytics

Performing counterfactual explanations

Data Drift Charts

Model Explanations

Overview

Fiddler’s Model Performance Management platform monitors Machine Learning model performance by sending real-time alerts when model performance metrics drop, allowing users to analyze inference data to understand why model performance is degrading. This integration includes metrics and an out-of-the-box dashboard that displays performance metrics such as accuracy, traffic, and drift.

Setup

The Fiddler check is not included in the Datadog Agent package, so you need to install it.

Installation

For Agent v7.21+ / v6.21+, follow the instructions below to install the Fiddler check on your host. See Use Community Integrations to install with the Docker Agent or earlier versions of the Agent.

1. Run the following command to install the Agent integration:

   datadog-agent integration install -t datadog-fiddler==3.0.0
   

2. Configure your integration similar to Agent-based integrations.

Configuration

1. Edit the fiddler.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your Fiddler performance data. See the sample fiddler.d/conf.yaml for all available configuration options. The url, org, and fiddler_api_key parameters need to be updated for the Fiddler environment you wish the integration to query. Fiddler also suggests setting the minimum_collection_interval setting in the conf.yaml file to 300 (5 minutes).

2. Restart the Agent.

Validation

Run the Agent’s status subcommand and look for fiddler under the Checks section.

Data Collected

Metrics

See metadata.csv for a list of metrics provided by this check.

Service Checks

fiddler.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored Fiddler instance. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

Need help? Contact Fiddler support.