Getting Started with Google Cloud
This page is not yet available in Spanish. We are working on its translation.
If you have any questions or feedback about our current translation project,
feel free to reach out to us!Overview
Use this guide to get started monitoring your Google Cloud environment at a folder or organization level. This approach simplifies the setup for Google Cloud environments with multiple projects, ensuring that there are no gaps in monitoring.
Setup
Prerequisites
- Create a Datadog account
- Set up a Service Account in any of your Google Cloud projects
- Review these Google Cloud Prerequisites:
◆ If your organization restricts identities by domain, you must add Datadog’s customer identity C0147pk0i as an allowed value in your policy.
◆ The Google Cloud integration requires the below APIs to be enabled for the folder or organization you want to monitor:
- Cloud Monitoring API
- Allows Datadog to query your Google Cloud metric data.
- Compute Engine API
- Allows Datadog to discover compute instance data.
- Cloud Asset API
- Allows Datadog to request Google Cloud resources and link relevant labels to metrics as tags.
- Cloud Resource Manager API
- Allows Datadog to append metrics with the correct resources and tags.
- IAM API
- Allows Datadog to authenticate with Google Cloud.
- Google Cloud Billing API
- Allows developers to manage billing for their Google Cloud Platform projects programmatically.
Organization-level metric collection
Org-level (or folder-level) monitoring is recommended for comprehensive coverage of all projects, including any future projects that may be created in an org or folder. To set up monitoring for individual projects, see the main Google Cloud integration page.
Note: You must have the Admin role assigned to your Cloud Identity user account at the desired scope (for example, Organization Admin).
- Open your Google Cloud console.
- Navigate to IAM & Admin > Service Accounts.
- Click Create service account at the top.
- Give the service account a unique name.
- Click Done to complete creating the service account.
- In the Google Cloud console, go to the IAM page.
- Select a folder or organization.
- To grant a role to a principal that does not already have other roles on the resource, click Grant Access, then enter the email of the service account you created earlier.
- Enter the service account’s email address.
- Assign the following roles:
- Compute Viewer provides read-only access to get and list Compute Engine resources
- Monitoring Viewer provides read-only access to the monitoring data availabile in your Google Cloud environment
- Cloud Asset Viewer provides read-only access to cloud assets metadata
- Browser provides read-only access to browse the hierarchy of a project
- Click Save.
Note: The Browser role is only required in the default project of the service account. Other projects require only the other listed roles.
Note: If you previously configured access using a shared Datadog principal, you can revoke the permission for that principal after you complete these steps.
- In Datadog, navigate to Integrations > Google Cloud Platform.
- Click Add Google Cloud Account.
If you have no configured projects, you are automatically redirected to this page.
- Copy your Datadog principal and keep it for the next section.
Note: Keep this window open for Section 4.
- In the Google Cloud console, under the Service Accounts menu, find the service account you created in Section 1.
- Go to the Permissions tab and click Grant Access.
- Paste your Datadog principal into the New principals text box.
- Assign the role of Service Account Token Creator.
- Click Save.
- In your Google Cloud console, navigate to the Service Account > Details tab. On this page, find the email associated with this Google service account. It has the format
<SA_NAME>@<PROJECT_ID>.iam.gserviceaccount.com. - Copy this email.
- Return to the integration configuration tile in Datadog (where you copied your Datadog principal in the previous section).
- In the box under Add Service Account Email, paste the email you previously copied.
- Click Verify and Save Account.
After finishing these steps, metrics appear in Datadog after approximately 15 minutes.
Validation
To view your metrics, use the left menu to navigate to Metrics > Summary and search for gcp:
Google Cloud integrations
The Google Cloud integration collects all available Google Cloud metrics from your projects through the Google Cloud Monitoring API. Integrations are installed automatically when Datadog recognizes data being ingested in from your Google Cloud account. <EXAMPLE?>
For deep dives into monitoring many of the more popular services, check out the blogs linked below.
- Cloud Armor
- Google Cloud Armor is a network security service protecting against DDoS and application attacks.
- BigQuery
- BigQuery is a serverless and multi-cloud data warehouse that can provide you with valuable insights from your business data.
- Cloud Run
- Cloud Run is a fully-managed platform that lets you run your code directly on scalable infrastructure in Google Cloud.
- Cloud SQL
- Cloud SQL is a fully-managed relational database service that works with MySQL, PostgreSQL, and SQL Server.
- Compute Engine
- Compute Engine is a computing and hosting service that provides you with the ability to create and run virtual machines in Google Cloud.
- Dataflow
- Dataflow is a fully-managed streaming analytics service that uses autoscaling and real-time data processing.
- Eventarc
- Eventarc is a fully-managed service enabling you to build event-driven architectures.
- Google Kubernetes Engine (GKE)
- GKE is a fully-managed Kubernetes service.
- Private Service Connect
- Private Service Connect lets you access managed Google services privately from within your VPC network.
- Security Command Center
- Security Command Center provides posture management and threat detection for code, identities, and data.
- Vertex AI
- Vertex AI is a fully-managed generative AI development platform.
Limit metric collection filters
You can limit metric collection to only the specific hosts, Cloud Run instances, or Google Cloud integrations valuable to your organization. This can help control costs by reducing the number of API calls made on your behalf.
Under the Metric Collection tab in Datadog’s Google Cloud integration page, deselect the metric namespaces to exclude.
- Assign a tag (such as
datadog:true) to the hosts or Cloud Run instances you want to monitor with Datadog. - Under the Metric Collection tab in Datadog’s Google Cloud integration page, enter the tags in the Limit Metric Collection Filters textbox. Only hosts that match one of the defined tags are imported into Datadog. You can use wildcards (
? for single character, * for multi-character) to match many hosts, or ! to exclude certain hosts. This example includes all c1* sized instances, but excludes staging hosts:
datadog:monitored,env:production,!env:staging,instance-type:c1.*
See Google’s documentation on Creating and managing labels for more details.
In the below example, only Google Cloud hosts with the label datadog:true are monitored by Datadog:
Log collection
Forwarding logs from your Google Cloud environment enables near real-time monitoring of the resources and activities taking place in your organization or folder. You can set up log monitors to be notified of issues, use Cloud SIEM to detect threats, or leverage Watchdog to identify unknown issues or anomalous behavior.
Use the Datadog Dataflow template to batch and compresses your log events before forwarding them to Datadog through Google Cloud Dataflow. This is the most network-efficient way to forward your logs. To specify which logs are forwarded, configure the Google Cloud Logging sink with any inclusion or exclusion queries using Google Cloud’s Logging query language.
Follow the instructions listed here to set up Log Collection. You can also use the Stream logs from Google Cloud to Datadog guide in the Google Cloud architecture center, for a more detailed explanation of the steps and architecture involved in log forwarding. For a deep dive into the benefits of the Pub/Sub to Datadog template, read Stream your Google Cloud logs to Datadog with Dataflow in the Datadog blog.
The
Dataflow API must be enabled to use Google Cloud Dataflow. See
Enabling APIs in the Google Cloud documentation for more information.
Resource changes collection
Resource changes collection allows you to monitor changes in your Google Cloud environment. You receive resource events in Datadog when Google’s Cloud Asset Inventory detects changes in your cloud resources. These events are forwarded to Datadog through a Cloud Pub/Sub topic and subscription.
For detailed setup instructions, see the resource changes collection section of the Google Cloud integration documentation.
Leveraging the Datadog Agent
After the Google Cloud integration is configured, Datadog automatically starts collecting Google Cloud metrics. However, you can leverage the Datadog Agent to gather deeper insights into your infrastructure.
The Datadog Agent provides the most granular, low-latency metrics from your infrastructure, delivering real-time insights into CPU, memory, disk usage, and more for your Google Cloud hosts.
The Agent can be installed on any host, including GKE.
The Agent also supports a wide range of integrations, enabling you to extend visibility into specific services and databases running on your hosts.
Traces collected through the Agent enable comprehensive Application Performance Monitoring (APM), helping you understand end-to-end service performance.
Logs collected through the Agent provide visibility into your Google Cloud resources, and the activities taking place in your Google Cloud environment.
For the full list of benefits of installing the Agent on your cloud instances, see Why should I install the Datadog Agent on my cloud instances?
Private Service Connect
Private Service Connect is only available for the US5 and EU Datadog sites.
Use the Google Cloud Private Service Connect integration to visualize connections, data transferred, and dropped packets through Private Service Connect. This gives you visibility into important metrics from your Private Service Connect connections, both for producers as well as consumers.
Private Service Connect (PSC) is a Google Cloud networking product that enables you to access Google Cloud services, third-party partner services, and company-owned applications directly from your Virtual Private Cloud (VPC).
See Access Datadog privately and monitor your Google Cloud Private Service Connect usage in the Datadog blog for more information.
Google Cloud Run
Use the Google Cloud Run integration to get detailed information on your Cloud Run containers, such as metrics and audit logs.
Cloud Cost Management (CCM)
Datadog’s Google Cloud Cost Management provides insights for engineering and finance teams to understand how infrastructure changes impact costs, allocate spend across your organization, and identify potential improvements.
Security
Cloud SIEM
Cloud SIEM provides real-time analysis of operational and security logs, while using out-of-the-box integrations and rules to detect and investigate threats.
To use this feature, see Getting Started with Cloud SIEM.
To view security findings from Google Cloud Security Command Center in Cloud SIEM, toggle the Enable collection of security findings option under the Security Findings tab & follow the setup instructions on the Google Cloud Security Command Center guide.
Cloud Security Management
Datadog Cloud Security Management (CSM) delivers real-time threat detection and continuous configuration audits across your entire cloud infrastructure.
Check out the Setting up Cloud Security Management guide to get started.
After setting up CSM, toggle the Enable Resource Collection option under the Resource Collection tab to start collecting configuration data for the Resource Catalog and CSM. Then, follow these instructions to enable Misconfigurations and Identity Risks (CIEM) on Google Cloud.
Further reading
Más enlaces, artículos y documentación útiles:
