Overview
Error Tracking provides fine-grained control of which errors to ingest, helping you reduce noise and avoid unexpected costs.
You can define what data is included in Error Tracking in two ways:
You can configure both rules and rate limits on the Error Tracking > Settings page.
Rules
Rules allow you to select which errors are ingested into Error Tracking. They apply to both billable and non-billable errors.
Each rule consists of:
- A scope: an inclusion filter, which contains a search query, such as
service:my-web-store. - Optionally, one or more nested exclusion filters to further refine the rule and ignore some of the matching events. For example, an exclusion filter might use the
env:staging query to exclude staging errors.
A given rule can be toggled on or off. An error event is included if it matches a query in one of the active inclusion filters and it does not match any active nested exclusion queries.
Each error event is checked against the rules in order. The event is processed only by the first active rule it matches, and all subsequent rules are ignored. If the matched rule has an exclusion filter, the event is excluded; otherwise, the event is included.
Note: Error events that get accepted by a rule might still be excluded from Error Tracking if they lack the required attributes.
Evaluation order
Rules are evaluated in order, with the evaluation stopping at the first matching rule. The priority of the rules and their nested filters depends on their order in the list.
Given a list of rules:
The processing flow is as follows:
An event with env:prod service:my-service status:warn
- will match rule 1 and go to its exclusion filters
- will not match exclusion 1-1 so will go to exclusion 1-2
- at exclusion 1-2, it will be a match, so the event will be discarded
An event with env:staging service:web
will not match rule 1, so will go to rule 2
at rule 2, it will be a match, so the event will be kept
Default rules
By default, Error Tracking has an * inclusion filter and no exclusion filters. This means all error with the requirements to be fingerprinted are ingested into Error Tracking.
Add a rule
To add a rule (inclusion filter):
- Navigate to Error Tracking Settings.
- Click Add New Rule.
- Choose the Error Tracking source the rule should be applied to.
- Enter a search query in the Define scope field.
- Optionally, Add Exclusion filters and a description to the rule.
- Click Save Changes
- Optionally, reorder the rules to change their evaluation order. Click and drag the six-dot icon on a given rule to move the rule up or down in the list.
Rate limits
Rate limits allow you to control the number of billable errors included in Error Tracking per day. This cap applies to all errors that match the filters of a rule.
After the daily cap is reached, ingestion stops until the next day. You can modify or remove the cap at any time.
Set a rate limit
To set a rate limit:
- Navigate to Error Tracking > Settings.
- Click Rate Limits.
- Edit the errors/month field.
- Click Save Rate Limit.
A Rate limit applied event is generated when you reach the rate limit. See the Event Management documentation for details on viewing and using events.
Monitoring usage
You can monitor your Error Tracking on Logs usage by setting up monitors and alerts for the datadog.estimated_usage.error_tracking.logs.events metric, which tracks the number of ingested error logs.
This metric is available by default at no additional cost, and its data is retained for 15 months.
Dynamic Sampling
Because Error Tracking billing is based on the number of errors, large increases in the errors for a single issue can quickly consume your Error Tracking budget. Dynamic Sampling protects you by establishing a threshold for the error rate per issue based on your daily rate limit and historical error volumes, sampling errors when that threshold is reached. Dynamic Sampling automatically deactivates when the error rate of your issue decreases below the given threshold.
Setup
Dynamic Sampling is automatically enabled with Error Tracking with a default intake threshold based on your daily rate limit and historical volume.
For best results, set up a daily rate limit on the Error Tracking Rate Limits page: Click Edit Rate Limit and enter a new value.
Disable Dynamic Sampling
Dynamic Sampling can be disabled on the Error Tracking Settings page.
Monitor Dynamic Sampling
A Dynamic Sampling activated event is generated when Dynamic Sampling is applied to an issue. See the Event Management documentation for details on viewing and using events.
Investigation and mitigation steps
When Dynamic Sampling is applied, the following steps are recommended:
- Check which issue is consuming your quota. The issue to which Dynamic Sampling is applied is linked in the event generated in Event Management.
- If you’d like to collect additional samples for this issue, raise your daily quota on the Error Tracking Rate Limits page.
- If you’d like to avoid collecting samples for this issue in the future, consider creating an exclusion filter to prevent additional events from being ingested into Error Tracking.
Further Reading
Additional helpful documentation, links, and articles:
