Fiddler Dashboard in DataDog
Performing counterfactual explanations
Overview
Fiddler’s Model Performance Management platform monitors Machine Learning model performance by sending real-time alerts when model performance metrics drop, allowing users to analyze inference data to understand why model performance is degrading. This integration includes metrics and an out-of-the-box dashboard that displays performance metrics such as accuracy, traffic, and drift.
Setup
The Fiddler check is not included in the Datadog Agent package, so you need to install it.
Installation
For Agent v7.21+ / v6.21+, follow the instructions below to install the Fiddler check on your host. See Use Community Integrations to install with the Docker Agent or earlier versions of the Agent.
Run the following command to install the Agent integration:
datadog-agent integration install -t datadog-fiddler==3.0.0
Configure your integration similar to Agent-based integrations.
Configuration
Edit the fiddler.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your Fiddler performance data. See the sample fiddler.d/conf.yaml for all available configuration options. The url, org, and fiddler_api_key parameters need to be updated for the Fiddler environment you wish the integration to query. Fiddler also suggests setting the minimum_collection_interval setting in the conf.yaml file to 300 (5 minutes).
Restart the Agent.
Validation
Run the Agent’s status subcommand and look for fiddler under the Checks section.
Data Collected
Metrics
See metadata.csv for a list of metrics provided by this check.
Service Checks
fiddler.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored Fiddler instance. Returns OK otherwise.
Statuses: ok, critical
Troubleshooting
Need help? Contact Fiddler support.
