Overview

Notifications are a key component of monitors that keep your team informed of issues and support troubleshooting. When creating your monitor, add to the Configure notifications and automations section.

Configure notifications and automations

Use the Configure notifications and automations section to:

• Send notifications to your team through email, Slack, PagerDuty, and other integrations.
• Trigger a workflow or create a workflow from a monitor.
• Add a case to your monitor.

Title

Add a unique title to your monitor (required). For multi alert monitors, some tags identifying your triggering scope are automatically inserted. Additionally, you can use tag variables.

Message

The message field allows standard Markdown formatting and variables. Use conditional variables to modulate the notification text sent to different contacts with @notifications.

A common use-case for the monitor message is to include a step-by-step way to resolve the problem, for example:

Steps to free up disk space:
1. Remove unused packages
2. Clear APT cache
3. Uninstall unnecessary applications
4. Remove duplicate files

Notifications

Use an @notification to add a team member, integration, workflow, or case to your notification. As you type, Datadog recommends existing options in a drop-down menu. Click an option to add it to your notification. Alternatively, click @ Add Mention, Add Workflow, or Add Case.

Note: An @notification must have a space between it and the last line character, for example:

Disk space is low @ops-team@company.com

@notifications can be sent to:

Email

• Notify an active Datadog user by email with @<DD_USER_EMAIL_ADDRESS>.

  Note: An email address associated with a pending Datadog user invitation or a disabled user is considered inactive and does not receive notifications. Blocklists, IP or domain filtering, spam filtering, or email security tools may also cause missing notifications.

• Notify any non-Datadog user by email with @<EMAIL>.

Teams

If a notification channel is set, you can route notifications to a specific Team. Monitor alerts targeting @team-handle are redirected to the selected communication channel. For more information on setting a notification channel to your Team, see the Teams documentation.

Integrations

Notify your team through connected integrations by using the format @<INTEGRATION_NAME>-<VALUES>.

This table lists prefixes and example links:

Handles that include parentheses ((, )) are not supported. When a handle with parentheses is used, the handle is not parsed and no alert is created.

Workflows

You can trigger a workflow automation or create a new workflow from a monitor. Before you add a workflow to a monitor, add a monitor trigger to the workflow.

To add an existing workflow to a monitor:

1. Click Add Workflow and search for it in the drop-down menu.
2. Alternatively, in the message section, add the full workflow mention name. The mention name should start with @workflow-. For example, @workflow-my-workflow.

To pass trigger variables into the workflow, use a comma-separated list with the syntax @workflow-name(key=value, key=value). You can use message template variables as trigger variables. For example, @workflow-my-workflow(hostname=host.name).

For more information on triggering a workflow, see Trigger a workflow.

To create a workflow:

1. Click Add Workflow.
2. Click the + icon and select a Blueprint, or select Start From Scratch.

For more information on building a workflow, see Build workflows.

Priority

Add a priority (optional) associated with your monitors. Values range from P1 through P5, with P1 being the highest priority and the P5 being the lowest. To override the monitor priority in the notification message, use {{override_priority 'Pi'}} where Pi is between P1 and P5.

For example, you can set different priorities for alert and warning notifications:

{{#is_alert}}
{{override_priority 'P1'}}
 ...
{{/is_alert}}
{{#is_warning}}
{{override_priority 'P4'}}
...
{{/is_warning}}

Toggle additional content

Monitor notifications include content such as the monitor’s query, the @-mentions used, metric snapshots (for metric monitors), and links back to relevant pages in Datadog. You have the option to choose which content you would like to include or exclude from notifications for individual monitors.

The options are:

• Default: No content is hidden.
• Hide Query: Remove the monitor’s query from the notification message.
• Hide Handles: Remove the @-mentions that are used in the notification message.
• Hide All: Notification message does not include query, handles, any snapshots (for metric monitors), or additional links in footers.

Note: Depending on the integration, some content may not be displayed by default.

Metadata

Add metadata (Priority, Tags, Datadog Team) to your monitor. Monitor Priority allows you to set the importance of your monitor through P-level (P1 to P5). Monitor tag–which are different from metric tags–are used in the UI to group and search for monitors. If tag policies are configured, the required tags and tag values need to be added. To learn more, see Tag Policies. Datadog Teams allows you to set a layer of ownership to this monitor and view all the monitors linked to your team. To learn more, see Datadog Teams.

Renotify

Enable monitor renotification (optional) to remind your team that a problem is not solved.

Configure the renotify interval, the monitor states from which the monitor renotifies (within alert, no data, and warn) and optionally set a limit to the number of renotification messages sent.

For example, configure the monitor to stop renotifying after 1 occurrence to receive a single escalation message after the main alert. Note: Attribute and tag variables in the renotification are populated with the data available to the monitor during the time period of the renotification.

If renotification is enabled, you are given the option to include an escalation message that is sent if the monitor remains in one of the chosen states for the specified time period.

The escalation message can be added in the following ways:

• In the {{#is_renotify}} block in the original notification message (recommended).
• In the Renotification message field in the Configure notifications and automations section.
• With the escalation_message attribute in the API.

If you use the {{#is_renotify}} block, the original notification message is also included in the renotification, so:

1. Include only extra details in the {{#is_renotify}} block and don’t repeat the original message details.
2. Send the escalation message to a subset of groups.

Learn how to configure your monitors for those use cases in the example section.

Audit notifications

An audit event is created anytime a monitor is created, modified, silenced, or deleted. Under the Define permissions and audit notifications section, select Notify to alert team members, chat services, and the monitor creator of these events.

Test notifications

Test notifications are supported for the monitor types: host, metric, anomaly, outlier, forecast, logs, rum, apm, integration (check only), process (check only), network (check only), custom check, event, and composite.

Run the test

1. After defining your monitor, test the notifications with the Test Notifications button at the bottom right of the monitor page.

2. From the test notifications pop-up, choose the monitor transition to test and the group (available only if the query has grouping). You can only test states that are available in the monitor’s configuration for the thresholds specified in the alerting conditions. Recovery thresholds are an exception, as Datadog sends a recovery notification once the monitor either is no longer in alert, or it has no warn conditions.

3. Click Run Test to send notifications to the people and services listed in the monitor.

Events

Test notifications produce events that can be searched within the event explorer. These notifications indicate who initiated the test in the message body with [TEST] in notification title.

Tag variables are only populated in the text of Datadog child events. The parent event only displays an aggregation summary.

Variables

Message variables auto-populate with a randomly selected group based on the scope of your monitor’s definition, for example:

{{#is_alert}}
{{host.name}} <-- will populate
{{/is_alert}}

Further Reading

Additional helpful documentation, links, and articles:

Create monitorsDOCUMENTATION Manage monitorsDOCUMENTATION