Define the search query

1. Construct a search query using the same logic as a CI Pipeline explorer search.
2. Select the CI Pipeline events level:
   • Pipeline: Evaluates the execution of an entire pipeline, usually composed of one or more jobs.
   • Stage: Evaluates the execution of a group of one or more jobs in CI providers that support it.
   • Job: Evaluates the execution of a group of commands.
   • Command: Evaluates manually instrumented custom command events, which are individual commands being executed in a job.
   • All: Evaluates all types of events.
3. Choose to monitor over a CI Pipeline event count, facet, or measure:
   • CI Pipeline event count: Use the search bar (optional) and do not select a facet or measure. Datadog evaluates the number of CI Pipeline events over a selected time frame, then compares it to the threshold conditions.
   • Dimension: Select dimension (qualitative facet) to alert over the Unique value count of the facet.
   • Measure: Select measure (quantitative facet) to alert over the numerical value of the CI Pipeline measure (similar to a metric monitor). Select the aggregation (min, avg, sum, median, pc75, pc90, pc95, pc98, pc99, or max).
4. Group CI Pipeline events by multiple dimensions (optional):
   • All CI Pipeline events matching the query are aggregated into groups based on the value of up to four facets.
5. Configure the alerting grouping strategy (optional):
   • If the query has a group by, multi alerts apply the alert to each source according to your group parameters. An alerting event is generated for each group that meets the set conditions. For example, you could group a query by @ci.pipeline.name to receive a separate alert for each CI Pipeline name when the number of errors is high.

Using formulas and functions

You can create CI Pipeline monitors using formulas and functions. This can be used, for example, to create monitors on the rate of an event happening, such as the rate of a pipeline failing (error rate).

The following example is of a pipeline error rate monitor using a formula that calculates the ratio of “number of failed pipeline events” (ci.status=error) over “number of total pipeline events” (no filter), grouped by ci.pipeline.name (to be alerted once per pipeline). To learn more, see the Functions Overview.

Define the search query

1. Common monitor types: (optional) Provides a template query for each of the New Flaky Test, Test Failures, and Test Performance common monitor types, which you can then customize. Learn more about this feature by reading Track new flaky tests.
2. Construct a search query using the same logic as a CI Test explorer search. For example, you can search failed tests for the main branch of the myapp test service using the following query: @test.status:fail @git.branch:main @test.service:myapp.
3. Choose to monitor over a CI Test event count, facet, or measure:
   • CI Test event count: Use the search bar (optional) and do not select a facet or measure. Datadog evaluates the number of CI Pipeline test events over a selected time frame, then compares it to the threshold conditions.
   • Dimension: Select dimension (qualitative facet) to alert over the Unique value count of the facet.
   • Measure: Select measure (quantitative facet) to alert over the numerical value of the CI Pipeline facet (similar to a metric monitor). Select the aggregation (min, avg, sum, median, pc75, pc90, pc95, pc98, pc99, or max).
4. Group CI Test events by multiple dimensions (optional):
   • All CI Test events matching the query are aggregated into groups based on the value of up to four facets.
5. Configure the alerting grouping strategy (optional):
   • If the query has a group by, an alert is sent for every source according to the group parameters. An alerting event is generated for each group that meets the set conditions. For example, you could group a query by @test.full_name to receive a separate alert for each CI Test full name when the number of errors is high. Test full name is a combination of a test suite and test name, for example: MySuite.myTest. In Swift, test full name is a combination of a test bundle, and suite and name, for example: MyBundle.MySuite.myTest.

Test runs with different parameters or configurations

Use @test.fingerprint in the monitor group by when you have tests with the same test full name, but different test parameters or configurations. This way, alerts trigger for test runs with specific test parameters or configurations. Using @test.fingerprint provides the same granularity level as the Test Stats, Failed, and Flaky Tests section on the Commit Overview page.

For example, if a test with the same full name failed on Chrome, but passed on Firefox, then using the fingerprint only triggers the alert on the Chrome test run.

Using @test.full_name in this case triggers the alert, even though the test passed on Firefox.

Formulas and functions

You can create CI Test monitors using formulas and functions. For example, this can be used to create monitors on the rate of an event happening, such as the rate of a test failing (error rate).

The following example is a test error rate monitor using a formula that calculates the ratio of “number of failed test events” (@test.status:fail) over “number of total test events” (no filter), grouped by @test.full_name (to be alerted once per test). To learn more, see the Functions Overview.

Using CODEOWNERS for notifications

You can send the notification to different teams using the CODEOWNERS information available in the test event.

The example below configures the notification with the following logic:

• If the test code owner is MyOrg/my-team, then send the notification to the my-team-channel Slack channel.
• If the test code owner is MyOrg/my-other-team, then send the notification to the my-other-team-channel Slack channel.

{{#is_match "citest.attributes.test.codeowners" "MyOrg/my-team"}}
  @slack-my-team-channel
{{/is_match}}
{{#is_match "citest.attributes.test.codeowners" "MyOrg/my-other-team"}}
  @slack-my-other-team-channel
{{/is_match}}

In the Notification message section of your monitor, add text similar to the code snippet above to configure monitor notifications. You can add as many is_match clauses as you need. For more information on Notification variables, see Monitors Conditional Variables.