Migrating to the New Events Features
Datadog's legacy event stream and event monitors retire on June 30, 2022. Datadog is migrating all customers to a new and improved events experience. This page contains important information about this migration. Before the retirement date, follow the steps on this page to ensure that your existing event visualizations and monitors continue to work properly.
Why change Events features?
Launched more than 10 years ago, Datadog’s event stream is one of its earliest features. The new events experience includes many new features that allow you to get even more value from your events. These include event analytics, the ability to generate metrics from your events, the ability to create pipelines to post-process events, and a much friendlier and more intuitive query syntax that is better aligned with other Datadog products, such as Log Management and APM.
What is the migration timeline?
March 2022 - New Events Explorer and analytics are accessible. Datadog begins to migrate customer dashboards and monitors that are not managed with APIs.
May 5, 2022 - The event stream is retired in favor of the Event Explorer.
May 19, 2022 - Starting on this date, while Datadog continues to evaluate event monitors that have not been migrated, they are no longer editable. New event monitors must use the new syntax.
June 30, 2022 - Datadog stops evaluating event monitors that have not been migrated. Legacy event monitors stop working.
What action do I need to take?
If you do not manage your dashboard or monitors using external API-based tools (such as Terraform or scripts), then no action is required on your end. Datadog will migrate your dashboards and monitors before April 30, 2022. Datadog will leave your old monitors in place but they will be muted and Datadog will stop evaluating them by June 30, 2022 at the latest.
If you use Terraform or other API-based scripts to manage all or some of your dashboards, Datadog will migrate queries in your event widgets and overlays to the new syntax, but you will need to update your scripts to keep them in sync before June 30, 2022.
If you use Terraform or other API-based scripts to manage all or some of your monitors, you have until June 30, 2022, to update them. After this date, Datadog will create new versions of any non-migrated monitors and mute the existing monitors to ensure that you continue to have alerting.
Datadog can also assist you with migrating your monitors by suggesting updates or by applying the updates to your monitors.
What’s new?
Events Explorer
The Events Explorer displays the most recent events generated by your infrastructure and services or monitoring alerts. It replaces the Events Stream and provides a friendlier and more intuitive query syntax. See Events Explorer to learn more.
Event Analytics
In addition to viewing and searching events in the Explorer, you can now graph as a timeseries, top list or table and group the number of events for a given query. See Event Analytics to learn more.
You may also generate metrics with 15-month retention from any event search query to create monitors and alerts based on historical events.
Graph events in dashboards
You can now graph events for a given query inside your dashboards as a timeseries graph, query value, top list, table and more.
For example, check out the Monitor Notifications Overview dashboard, which analyzes monitor alert event trends to help you improve your configuration and reduce alert fatigue.
New event monitors experience
Event monitors have been ported to use the same standardized set of features as other products (Logs, RUM, APM) and have additional capabilities.
When you create event monitors, the new query search field uses autocomplete, rather than the legacy fill-in-the-blank query.
The new query search allows you to use complex queries in event monitors with new capabilities such as Boolean operators or wildcards.
Pipelines
Datadog automatically parses JSON-formatted events. When events are not JSON-formatted, they are parsed and enriched by chaining them sequentially through a processing pipeline. Processors extract meaningful information or attributes from semi-structured text to reuse as facets. Each event that comes through the pipelines is tested against every pipeline filter. If it matches a filter, then all the processors are applied sequentially before moving to the next pipeline.
Reserved attributes
This list describes automatically ingested reserved attributes with events.
| Attribute | Description |
|---|
host | The name of the originating host as defined in metrics. Datadog automatically retrieves corresponding host tags from the matching host in Datadog and applies them to your events. The Agent sets this value automatically. |
source | This corresponds to the integration name, or the technology from which the event originated. When it matches an integration name, Datadog automatically installs the corresponding parsers and facets. For example: nginx, postgresql, and more. |
status | This corresponds to the level or severity of an event. |
service | The name of the application or service generating the events. |
message | By default, Datadog ingests the value of the message attribute as the body of the event entry. |
What Changed?
Note: The process for sending events remains the same. You can continue sending events using the API, the Agent, or the events via email feature as before.
Datadog no longer automatically performs Event aggregation, and no longer groups Events by the aggregation_key attribute. The UI no longer displays Event aggregation.
Comments created using the API with the user_update event type will be displayed as normal events.
Status remapping in queries
Some status values have changed:
| Legacy status | New status |
|---|
| success | ok |
| warning | warn |
| info | info |
| error | error |
Source remapping in queries
Many Event source names have changed. See the full list of affected source names.
The monitor evaluation window is limited to 48 hours.
Your monitors are not evaluated beyond a 48 hour window. If you need to use a longer evaluation window, you can generate custom metrics from events and use a metrics monitor, where the evaluation window can be up to one month.
You can group by only up to 4 facets.
(Previously: Unlimited groups) Top-values, the highest frequency values of a group, are limited based on the total number of groups. For example, if a monitor triggers more times than the facet limit, it sorts by top group and shows only the top N groups. For example, N = 30 resulting hosts if grouping by two facets and one facet is host.
- One facet results in a limit of 1000 top values.
- Two facets results in a limit of 30 top values per facet (at most 900 groups)
- Three facets results in a limit of 10 top values per facet (at most 1000 groups)
- Four facets results in a limit of 5 top values per group (at most 625 groups)
Recovery thresholds in monitors are no longer supported
Event monitor thresholds no longer support the warning_recovery and critical_recovery threshold types. Recovery thresholds need to be removed in new Event monitors.
If you are using these features, contact Support to get help finding an alternative solution.
Examples
Event query syntax examples before and after
- Show events from GitHub or Chef
- Legacy syntax
sources:github,chef - New syntax
source:(github OR chef) - Show events tagged with
env-prod - Legacy syntax
tags:env-prod - New syntax
tags:env-prod - Show events tagged with
#env-prod or #db - Legacy syntax
tags:env-prod,db, tags:env-prod OR db - New syntax
tags:(env-prod OR db) - Show events tagged with
#security-group:sg-123 and #role:common-node - Legacy syntax
tags:security-group:sg-123 AND role:common-node - New syntax
tags:(security-group:sg-123 AND role:common-node) - Use wildcards to search for prefixes and suffixes
- Legacy syntaxNot available
- New syntax
*web matches all event messages ending with websource:amazon* matches all events that source starts with amazon
Event monitors API syntax examples before and after
The Event monitors API has a new monitor query syntax (See “Event V2 Alert Query” section), with Average and Cardinality rollup methods and fewer required attributes.
- No Slack events in the past 24 hours
- Legacy syntax
events('priority:all sources:slack').rollup('count').last('1d') < 1 - New syntax
events("source:slack").rollup("count").last("1d") < 1 - EC2 Instance marked for maintenance
- Legacy syntax
events('priority:all "Upcoming AWS maintenance event"').by('name,host').rollup('count').last('2d') >= 1 - New syntax
events("Upcoming AWS maintenance event").rollup("count").by("name,host").last("2d") >= 1 - Zabbix or Prometheus has triggered an alert for a service today
- Legacy syntax
events('tags:service priority:all status:error sources:prometheus sources:zabbix).rollup('count').last('1d') > 0 - New syntax
events("source:(prometheus OR zabbix) status:error tags:service").rollup("count").last("1d") > 0 - No events received in a datacenter for service
datadog-agent - Legacy syntaxLegacy Event Monitors do not support cardinality rollup.
- New syntax
events("service:datadog-agent").rollup("cardinality", "datacenter").by("service").last("15m") < 1
Further reading
Additional helpful documentation, links, and articles:
