How to update an anomaly detection monitor to account for local time zone

Datadog monitors use UTC time, which by default does not track local time zones. Depending on the type of system you have, your data could be impacted by local activity happening in your time zone. For example, a lunch surge could cause a spike in the middle of the day, and this surge could be detected as an unexpected anomaly. If you are getting unexpected anomalies caused by local activity, update your anomaly detection monitor to account for your local timezone.

If you are using the agile or robust anomaly detection algorithms with weekly or daily seasonality, you can update your anomaly detection monitor to account for a local timezone using both the API and the UI.

Here is an example of a monitor before it is set to account for a local timezone:

DST tracking turned off

Here is an example of a monitor when daylight savings time is being taken into account:

DST tracking turned on

UI

To update an anomaly detection monitor to account for a local timezone in the UI, navigate to New monitor > Anomaly monitor. Under the Set Alert Conditions section, open the Advanced panel and toggle on the switch to take daylight savings into account while evaluating the monitor. Then, set the timezone dropdown to match the timezone you want tracked

DST toggle option in the UI

API

  1. You need the following information to make the update request through the monitor API:
    • Your Datadog API key and application key for authentication.
    • The monitor ID and query from your anomaly detection monitor. In the app, navigate to the monitor you are querying to find the monitor ID in the URL.
    • The TZ identification string for the time zone related to your metric, for example America/New_York or Europe/Paris. Locate your preferred time zone in the TZ column on the List of tz database time zones (canonical format recommended).

  2. Create an updated version of the monitor query by adding a timezone argument to the anomalies() function call. For example, if you wanted to change the query shown above to use New York’s local time, the query would be updated to:
    avg(last_4h):anomalies(avg:system.cpu.user{role:trace-cassandra} by {host}, 'basic', 2, direction='both', alert_window='last_15m', interval=60, count_default_zero='true', timezone='America/New_York') >= 1
    
  3. Use the Edit a Monitor API to update the monitor’s definition.
    • Examples are available in Python, Ruby, and cURL.
    • Only include the ID and query in the request to avoid overriding existing settings. The name, message, options, and tags are not required.

Further Reading

PREVIEWING: rtrieu/product-analytics-ui-changes