Delivering value

After you’ve set up data ingestion, you can take several additional steps to maximize the value for your clients. Here are some key areas to focus on.

Setting up monitors and downtimes

Monitors and alerts draw human attention to particular systems and services that require inspection and intervention. To generate alerts, Datadog offers:

• Monitors - the definitions for alert conditions
• Downtimes - time periods when alerts should be raised or suppressed

To familiarize yourself with the concept of monitors in general, see the following resources:

• Alerting
• Monitoring 101 - Alerting on what matters (blog).
• Intro to Monitoring (Training).

Monitor migrations

Service providers often need to migrate a client from a different monitoring or observability platform to Datadog. In such cases, it might seem logical to replicate any monitor from the previous solution in Datadog. This approach often results in many of Datadog’s most useful features going unused. In particular, you don’t want to miss features that improve issue detection and resolution times or reduce alert fatigue.

Before starting a migration project, review existing alert and threshold definitions to answer the following questions:

• Does the metric have a time-based variation? An anomaly monitor might be a better approach.
• Does the metric have a load-based variation? An arithmetic monitor might be the best approach by combining a metric with a load-indicating metric. For example, load on the systems might be higher if there are more users using a service.
• Is the absolute value of the metric less important than the rate of change? A change monitor or a forecast monitor might be the best approach.
• Is the value of the metric itself less important than whether it is different from the value for other hosts or entities? For example, is one node in a cluster experiencing high latency while other nodes are not? The outlier monitor is the best approach in this scenario.
• Do only a combination of multiple metrics present an actionable situation? Composite monitors offer a solution that doesn’t require scripting.

Programmatic monitor management

As a service provider, management of monitors for you and your clients is best accomplished programmatically through one of the following ways:

• Datadog Monitors API
• Terraform
  • How to Manage Datadog Resources Using Terraform (Video)
  • Automate Monitoring with the Terraform Datadog Provider (HashiCorp Tutorial)

Ensure that you tag monitors to make the management of large numbers of monitors easier.

Recommended monitors

You might run into technologies your clients operate that you do not have a lot of experience with. Datadog offers Recommended Monitors to help you onboard new technologies quickly and confidently.

To find out more about monitors, see:

• Manage Monitors
• Monitors
• Creating Dynamic Alerts Using Tag Values (Video)
• Monitor settings changes not taking effect

Downtimes

A common problem with monitors and alerts is alert fatigue, where an overabundance of alarms or notifications causes desensitization to the alarms. One way to combat alert fatigue is to limit the number of false positive alarms. This is especially pertinent in controlled situations such as planned system shutdowns, maintenance, or upgrade windows.

Datadog’s Downtimes offer you and your clients a way to mute monitors during times of planned (or ad hoc) maintenance.

To find out more about managing downtimes, especially programmatically, see:

• Downtime
• Mute Datadog Alerts for Planned Downtimes (Blog)
• Managing Datadog with Terraform (Blog)
• Downtime API
• Prevent alerts from Monitors that were in downtime

Notifications

Some general guidelines for notifications:

• Alert liberally; page judiciously
• Page on symptoms, rather than causes

Datadog offers various channels through which you or your clients can notify users about important alerts:

• Email notifications
• Integrations, such as:
  • Slack
  • PagerDuty
  • Flowdock
  • ServiceNow
  • Google Chat
  • Microsoft Teams
  • And many more

You can also invoke any REST API using the generic Webhooks integration. You use a Webhooks integration to not only notify users, but also to trigger automatic remediation workflows.

To find out more about notifications, see:

• Notifications
• Send SMS Alerts with Webhooks and Twilio (Blog)

Setting up visualizations with dashboards

Visualizations are a great way to give your clients a clear picture of complex tech stacks and the abundance of metrics and events being collected. Dashboards are a natural starting point to investigate a potential issue you or your client was notified about by a monitor.

Out-of-the-box dashboards

The moment that an Agent or Cloud Integration is set up, Datadog automatically enables out-of-the-box dashboards for the newly integrated service or technology, providing immediate insights. You can also clone an out-of-the-box dashboard, giving you a great starting point for a custom dashboard.

Building custom dashboards

You can provide extra value and distinguish yourself from your competitors by providing a business-centric perspective, tailored to different personas.

Here are some dashboard best practices to take into consideration when building your dashboards:

• Focus on work metrics rather than too many resource metrics. To understand the difference, see Monitoring 101: Collecting the right data (blog).
• Make use of event overlays to correlate metrics and events.
• Annotate the dashboards with free text information on what data is shown and what to do in case the dashboard indicates an issue.

To find out more about Dashboards, see:

• Building Better Dashboards (Training)
• Use the Datadog Notebooks feature to gather data in an exploratory fashion and draft dashboards
• Monitor Datacenters and Networks with Datadog (Blog)
• Use Associated Template Variables (Blog)
• The Datadog Dashboard API
• Configure Observability as Code with Terraform & Datadog (HashiCorp webinar)

Visualizations for users without Datadog access

Depending on your business model, your clients might not require access to Datadog themselves. However, you might still want to provide Datadog visualizations to your clients. You have the following options for providing Datadog visualizations:

• Dashboard sharing: Provide a status page to your clients by sharing a public URL to a read-only dashboard, or share the dashboard privately using an individual email address.
  • As a service provider, your business needs to be able to scale. Managing shared dashboards using Datadog’s APIs is the most efficient approach.
• Embeddable graphs: If you have a client portal in which you want to present Datadog information, embeddable graphs are the way to go. Using parameters, you can filter data according to your needs. For more information, see:
  • Embeddable Graphs API
  • Embeddable Graphs with Template Variables

Setting up service-level objectives

It is a good idea to continually present the quality and level of your services to your clients in a transparent way. Service-level objectives (SLOs) are a great way to monitor and visualize the service quality on behalf of your clients, and also help your clients implement service level-based reporting internally.

The following material might be helpful for you when setting up and managing SLOs:

• To get started, see Service Level Objectives 101: Establishing Effective SLOs (Blog).
• SLO Checklist
• Best practices for managing your SLOs with Datadog (Blog)
• Track the status of all your SLOs in Datadog (Blog)
• The Datadog SLO API

Using Watchdog

Watchdog is an algorithmic feature that automatically detects potential application and infrastructure issues.

Set up a Watchdog Monitor for your own staff or your client with a notification whenever Watchdog has detected a new irregularity.

For more information, see Watchdog.

What’s next?

Find out how to monitor individual client and aggregate usage of the Datadog platform in multi-organization account setups with Billing and Usage Reporting.