Glossary

Welcome to our Glossary! This is a work in progress and we are actively building up a complete term list, which will take time. If you have feedback about how this glossary works, or terms you'd like to see defined, please click Feedback and let us know.

A

absolute change

CI-CD

Related terms: baseline mean , test duration , relative change

In Datadog Test Optimization, an absolute change is the absolute difference between the test duration and the baseline mean. For more information, see the documentation.

approval wait time

CI-CD

Related terms: queue time

Approval wait time is the duration that pipelines and jobs or stages within a pipeline are blocked, pending manual approval. For more information, see the documentation.

B

baseline mean

CI-CD

Related terms: test run

In Datadog Test Optimization, a baseline mean is the mean duration of the same test in the default branch, calculated over the last week of test runs. For more information, see the documentation.

baseline standard deviation

CI-CD

Related terms: test run

In Datadog Test Optimization, a baseline standard deviation is the standard deviation of the same test in the default branch, calculated over the last week of test runs. For more information, see the documentation.

C

custom measure

CI-CD

Related terms: custom tag

A custom measure is a quantitative facet (a numerical value) that can be used to aggregate values from multiple pipeline traces, filter pipeline traces using a range, or sort pipeline traces against a value. For more information, see the documentation.

custom span

CI-CD

A custom span is the ability to integrate command-level events in CI Pipeline Visibility, which are displayed in a pipeline’s flame graph visualization. For more information, see the documentation.

custom tag

CI-CD

Related terms: custom measure

A custom tag is a qualitative facet (a string value) that can be used to aggregate values from multiple pipeline traces, filter pipeline traces using a range, or sort pipeline traces against a value. For more information, see the documentation.

F

flaky test

CI-CD

A flaky test is a test that exhibits both a passing and failing status across multiple test runs for the same commit. If you commit some code and run it through CI, and a test fails, and you run it through CI again and the test passes, that test is unreliable as proof of quality code. For more information, see the documentation.

J

job log

CI-CD
LOG MANAGEMENT

A job log is a record of all events, outputs, and actions that occur when a job executes in a CI/CD pipeline. Job logs include information such as command outputs, system messages, error messages, and status updates. For more information, see the documentation.

M

manual step

CI-CD

A manual step is when there is a job with a manual approval phase in the pipeline. For more information, see the documentation.

P

parameter

CI-CD

A parameter is any user-defined value that can be passed to a CI pipeline at runtime. Parameters allow you to customize and control a pipeline’s execution behavior, and are useful for situations such as deploying to different environments, selecting specific test suites to run, or defining build versions. For more information, see the documentation.

partial retry

CI-CD

Related terms: running pipeline

A partial retry is when a subset of stages, jobs, or steps in a CI pipeline are re-executed after a failure, as opposed to retrying the entire pipeline. For more information, see the documentation.

performance regression

CI-CD

Related terms: test regression , test service

In Datadog Test Optimization, a performance regression is a measurable decline in performance metrics for a test service. For more information, see the documentation.

pipeline execution time

CI-CD

Pipeline execution time is the amount of time a pipeline has been actively running jobs. Pipeline execution time represents a pipeline trace, which provides an overview of the pipeline’s execution history, including the start and end times of each job and any idle periods between jobs.

A diagram illustrating the pipeline execution time for a CI pipeline. The pipeline takes five minutes to run, but the total pipeline execution time of individual jobs is one minute and forty five seconds

For example, consider a pipeline with 3 jobs: A, B, and C. During the first minute both A and B are running, so that minute counts towards execution time. The next 30 seconds, A stops while B is still running, so that time also counts. Then, there is a gap until C starts running, so the execution time is not increased during this period since there are no jobs running. Finally, C runs for 15 seconds.

The final execution time calculation is 1m30s+15s=1m45s.

This metric is available at the pipeline level only under the name @ci.execution_time.

pipeline failure

CI-CD

A pipeline failure occurs when one or more stages or jobs in a CI/CD pipeline does not complete successfully. For more information, see the documentation.

Q

queue time

CI-CD

Queue time is the amount of time a pipeline or job is in the queue before execution. For more information, see the documentation.

R

relative change

CI-CD

Related terms: baseline mean , test duration , absolute change

In Datadog Test Optimization, a relative change is the relative difference between the test duration and the baseline mean. For more information, see the documentation.

running pipeline

CI-CD

Related terms: partial retry

A running pipeline is a pipeline that is executing in a CI/CD environment. During its execution, the pipeline processes various jobs or tasks sequentially or in parallel, depending on the pipeline configuration. For more information, see the documentation.

S

standard deviation change

CI-CD

Related terms: baseline mean

In Datadog Test Optimization, a standard deviation change is the number of standard deviation above the baseline mean. For more information, see the documentation.

T

test duration

CI-CD

Related terms: absolute change , relative change

In Datadog Test Optimization, a test duration is the length of time for a CI test to complete. For more information, see the documentation.

test regression

CI-CD

Related terms: performance regression

In Datadog Test Optimization, a test run is marked as a regression when its duration is both five times the mean and greater than the max duration for the same test in the default branch. Additionally, the test run must have a minimum duration of 500ms to be considered a regression. A benchmark test run (@test.type:benchmark) is marked as a regression when its duration is five times the standard deviation above the mean for the same test in the default branch.

The mean and maximum durations of the default branch are calculated based on the test runs from the past week. For the algorithm to consider a test run, there must be a minimum of 100 test runs on the default branch. For benchmark test runs, the minimum number of test runs is 10. For more information on Test Optimization, see the documentation.

test run

SYNTHETIC MONITORING
CI-CD

In Datadog Synthetic Monitoring and Test Optimization, a test run is an executed set of tests on software to ensure its functionality. A browser test run is a simulation of a web transaction, up to 25 steps. For more information, see the documentation.

test service

CI-CD
SYNTHETIC MONITORING

Related terms: test suite

In Datadog Test Optimization, a test service is generally a group of tests associated with, for example, a project or repo. It contains all the individual tests for your code, optionally organized into test suites, which are like folders for your tests. For more information on Test Optimization, see the documentation.

test suite

CI-CD
SYNTHETIC MONITORING

Related terms: test service

In Datadog Synthetic Monitoring and Test Optimization, a test suite is generally a group of tests exercising the same unit of code depending on your language and testing framework. You can access an example of a test suite which corresponds to a test file in the datadog-ci repository. For more information, see the documentation.

PREVIEWING: ida.adjivon/DOCS-9781-V2