Selecting a service on the Service Catalog leads you to the detailed service page. A service is a set of processes that do the same job - for example a web framework or database (read more about how services are defined in Getting Started with APM).
The Service Health banner and alert badges provides a real-time summary of service signals to help you understand if a service needs your attention.
Service health considers many types of signals (including monitors, incidents, Watchdog insights, and error tracking issues), and surfaces the most critical alerts on the Service Page.
View the Service Health banner at the top of the page and the related alert badges in the vertical navigation.
The Service Health banner displays the status of your service as Warning, or Alert if at least one of the following conditions is met:
Status
Condition
Alert
Monitors: - A non-muted monitor with a paging integration setup (PagerDuty or Opsgenie) is triggered.
Incidents: - An incident of any severity is active.
Watchdog Insights: - A faulty deployment is active. - An ongoing APM latency/error rate alert is active.
Warning
Monitors: - A non-muted warning monitor with a paging integration setup (PagerDuty or Opsgenie) is triggered.
Incidents: - An incident of any severity is in a stable state.
Watchdog Insights: - An ongoing log anomaly alert is active.
Error Tracking Issues: - A new issue (within 48 hours) requires review.
No Alerts
No signal from critical or alert state is active.
Service monitor
The Service monitor panel surfaces active Monitors and Synthetics tests linked to your service.
Datadog also proposes a list of monitors depending on your service type:
Enable them directly or create your own APM monitors.
Note: Tag any monitor or Synthetic Test with service:<SERVICE_NAME> to attach it to an APM service.
Watchdog Insights
The Watchdog Insights carousel surfaces anomalies and outliers detected on specific tags, enabling you to investigate the root cause of an issue. Insights are discovered from APM, Continuous Profiler, Log Management, and Infrastructure data that include the service tag. These insights are the same insights that appear in each of the product pages. For example, the same Log outliers on the service page can be found in the Log Explorer.
Click on an insight to see more details, such as the time frame of the insight, related logs or traces, and suggested next steps.
Summary cards
The service page features summary cards with highlights on your service health. Easily spot potential faulty deployments, click into the card to view details or traces of the latest deployment, or view all deployments on this service. See new issues flagged on your service through our integration with Error Tracking, where errors are automatically aggregated into issues.
Our Service Level Objectives (SLOs) and Incidents summaries allow you to monitor the status of SLOs and ongoing incidents, so that you can keep performance goals top of mind. Click the cards to create a new SLO on the service or declare an incident. The security signals summary highlights how your services react to application threats.
Out-of-the-box graphs
Datadog provides out-of-the-box graphs for any given service. Use the dropdown above each graph to change the displayed information.
Requests and Errors
The Requests and Errors graph displays the total number of requests (hits) and errors over time. Using the dropdown menu, you can also view:
Requests by Version: Breakdown of requests across different service versions.
Requests per Second by Version: The rate of requests for each version.
Requests and Errors Per Second: The rate of requests (hits) and errors per second.
Errors
The Errors graph displays the total count of errors over time. Using the dropdown menu, you can also view:
Errors by Version: The error counts for each service version side by side.
Errors per Second by Version: The error rate (errors per second) for each service version over time.
Errors per Second: The overall error rate for the service, per second.
% Error Rate by Version: The percentage of requests resulting in errors for each service version.
% Error Rate: The overall error rate for the service, as a percentage.
Latency
The Latency graph displays the latency percentiles as a timeseries. Using the dropdown menu, you can also view:
Latency by Version: Latency broken down by service version.
Historical Latency: Comparison of the current latency distribution with the previous day and week.
Latency Distribution: The distribution of latencies over the selected time frame.
Latency by Error: The latency of requests over time, segmented by whether the requests resulted in errors.
Apdex (Application Performance Index): The Apdex score over time.
Avg Time per Request
For services involving multiple downstream services, a fourth graph breaks down the average execution time spent per request. This graph is built on sampled trace data, unlike the other top graphs which use unsampled data sources.
Using the dropdown menu, you can also view:
Total Time Spent: The cumulative time spent in each downstream service over time.
% of Time Spent: The percentage of time spent in each downstream service relative to the total time.
For services like Postgres or Redis, which are final operations that do not call other services, there is no sub-services graph. Watchdog performs automatic anomaly detection on the Requests, Latency, and Error graphs. If an anomaly is detected, an overlay appears on the graph. Clicking the Watchdog icon provides more details in a side panel.
Export
On the upper-right corner of each graph click on the arrow in order to export your graph into a pre-existing dashboard:
Resources
See Requests, Latency, and Error graphs broken down by resource to identify problematic resources. Resources are particular actions for your services (typically individual endpoints or queries). Read more in Getting Started with APM.
Below, there’s a list of resources associated with your service. Sort the resources for this service by requests, latency, errors, and time, to identify areas of high traffic or potential trouble. Note that these metric columns are configurable (see image below).
Click on a resource to open a side panel that displays the resource’s out-of-the-box graphs (about requests, errors, and latency), a resource dependency map, and a span summary table. Use keyboard navigation keys to toggle between resources on the Resources list and compare resources in a service. To view the full resource page, click Open Full Page.
Requests: Absolute amount of requests traced (per seconds)
Requests per second: Absolute amount of requests traced per second
Total time: Sum of all time spend in this resource
Avg/p75/p90/p95/p99/Max Latency: The Avg/p75/p90/p95/p99/Max latency of your traced requests
Errors: Absolute amount of error for a given resource
Error Rate: Percent of error for a given resource
Dependencies
Visualize upstream and downstream dependencies that the service interacts with from the dependency map. The map is powered by APM metrics to surface accurate request counts, error rates, and latency numbers. The map automatically groups dependencies by operation name. For instance, if a service calls two downstream services using gRPC, these services are grouped together. The table on the left-hand side of the map shows requests and error rates over time, useful to identify failing dependencies.
Inferred service dependencies like databases, queues, or third-party services are represented with a purple background node.
Note: Service overrides display along the edges (connecting lines) between nodes in the dependency map to keep visibility over the actual remote service, database, or queue the service is interacting with.
Investigating a specific service dependency
Click on a dependency service node to investigate one of the service’s dependencies. In the side panel, request count, errors, and latency metrics are scoped to requests going from the service to the downstream dependency.
Use the following table to see which resources are invoked in client calls from the service to the downstream dependency. For instance, if the service is calling a database, see the breakdown of queries that are made to the database, and use the table to identify slow or failing queries.
Additional sections
Deployments
A service configured with version tags will show versions in the Deployment tab. The version section shows all versions of the service that were active during the selected time interval, with active versions at the top.
By default, you can see:
The version names deployed for this service over the timeframe.
The times at which traces that correspond to this version were first and last seen.
An Error Types indicator, which shows how many types of errors appear in each version that did not appear in the immediately previous version.
Note: This indicator shows errors that were not seen in traces from the previous version. It doesn’t mean that this version necessarily introduced these errors. Looking into new error types can be a great way to begin investigating errors.
Requests per second.
Error rate as a percentage of total requests.
You can add columns to or remove columns from this overview table and your selections will be saved. The additional available columns are:
Endpoints that are active in a version that were not in the previous version.
Time active, showing the length of time from the first trace to the last trace sent to Datadog for that version.
Total number of Requests.
Total number of Errors.
Latency measured by p50, p75, p90, p95, p99, or max.
View issues on your service, which are similar errors aggregated together to turn a noisy stream of errors into manageable issues and help you assess the impact of your service’s errors. Read more about issues in Error Tracking.
This tab has overview graphs that show which resources have the most issues and a list of the most common issues occurring in your service. Click on an issue in the list to see details in a side panel, including its stack trace, related code versions, and total error occurrences since inception.
Security
Understand the security posture of the service, including known vulnerabilities exposed in the service’s libraries and security signals on your service, which are automatically created when Datadog detects application attacks impacting your services. The signals identify meaningful threats for you to review instead of assessing each individual attack attempt. Read more about Application Security.
The top section of the security tab has overview graphs that show the number and severity of vulnerabilities, a timeline of attacks, the types of attacks, and attacker information (client IP or authenticated user).
The next section of the panel lists all the vulnerabilities and signals concerning the service. Click on a security vulnerability to open a side panel with relevant details to investigate further and remediate the vulnerability. Click on a security signal to get information about what the detected threat is and what actions you can take to remediate it.
Databases
View the list of downstream database dependencies identified by Database Monitoring and identify latency or load outliers.
Learn more about connecting DBM and APM.
Infrastructure
If your service is running on Kubernetes, you can see an Infrastructure tab on the Service Page. The live Kubernetes Pods table displays detailed information on your pods, such as if memory usage is close to its limit, and allows you to improve resource allocation by seeing if provisioned compute resources exceed what is required for optimal application performance.
The Kubernetes Metrics section contains a high level summary of your infrastructure health for the selected time period, and includes CPU, Memory, Network, and Disk metrics.
If runtime metrics are enabled in the tracing client, you’ll see a Runtime metrics tab corresponding to the runtime language of your service. Read more in Runtime Metrics.
Profiling
You’ll see a Profiling tab if the Continuous Profiler is set up for your service.
Use the information in the Profiling tab to correlate a latency and throughput change to a code performance change.
In this example, you can see how latency is linked to a lock contention increase on /GET train that is caused by the following line of code:
Thread.sleep(DELAY_BY.minus(elapsed).toMillis());
Traces
View the list of traces associated with the service in the traces tab, which is already filtered on your service, environment, and operation name. Drill down to problematic spans using core facets such as status, resource, and error type. For more information, click a span to view a flame graph of its trace and more details.
Log patterns
View common patterns in your service’s logs, and use facets like status in the search bar to filter the list of patterns. Click on a pattern to open the side panel to view more details, such as what events triggered the cascade. Read more in Log patterns.
