Organization-level (or folder-level) monitoring is recommended for comprehensive coverage of all projects, including any future projects that may be created in an org or folder.

Note: Your Google Cloud Identity user account must have the Admin role assigned to it at the desired scope to complete the setup in Google Cloud (for example, Organization Admin).

3. Add the Datadog principal to your service account

Note: If you previously configured access using a shared Datadog principal, you can revoke the permission for that principal after you complete these steps.

1. In Datadog, navigate to Integrations > Google Cloud Platform.
2. Click Add Google Cloud Account. If you have no configured projects, you are automatically redirected to this page.
3. Copy your Datadog principal and keep it for the next section.

Note: Keep this window open for Section 4.

4. In the Google Cloud console, under the Service Accounts menu, find the service account you created in Section 1.
5. Go to the Permissions tab and click Grant Access.

6. Paste your Datadog principal into the New principals text box.
7. Assign the role of Service Account Token Creator.
8. Click Save.

Metrics appear in Datadog approximately 15 minutes after setup.

Best practices for monitoring multiple projects

Enable per-project cost and API quota attribution

By default, Google Cloud attributes the cost of monitoring API calls, as well as API quota usage, to the project containing the service account for this integration. As a best practice for Google Cloud environments with multiple projects, enable per-project cost attribution of monitoring API calls and API quota usage. With this enabled, costs and quota usage are attributed to the project being queried, rather than the project containing the service account. This provides visibility into the monitoring costs incurred by each project, and also helps to prevent reaching API rate limits.

To enable this feature:

1. Ensure that the Datadog service account has the Service Usage Consumer role at the desired scope (folder or organization).
2. Click the Enable Per Project Quota toggle in the Projects tab of the Google Cloud integration page.

You can use service account impersonation and automatic project discovery to integrate Datadog with Google Cloud.

This method enables you to monitor all projects visible to a service account by assigning IAM roles in the relevant projects. You can assign these roles to projects individually, or you can configure Datadog to monitor groups of projects by assigning these roles at the organization or folder level. Assigning roles in this way allows Datadog to automatically discover and monitor all projects in the given scope, including any new projects that may be added to the group in the future.

1. Create a Google Cloud service account

1. Open your Google Cloud console.
2. Navigate to IAM & Admin > Service Accounts.
3. Click on Create service account at the top.
4. Give the service account a unique name, then click Create and continue.
5. Add the following roles to the service account:
   • Monitoring Viewer provides read-only access to the monitoring data availabile in your Google Cloud environment
   • Compute Viewer provides read-only access to get and list Compute Engine resources
   • Cloud Asset Viewer provides read-only access to cloud assets metadata
   • Browser provides read-only access to discover accessible projects
6. Click Continue, then Done to complete creating the service account.

Forward logs from your Google Cloud services to Datadog using Google Cloud Dataflow and the Datadog template. This method provides both compression and batching of events before forwarding to Datadog.

You can use the terraform-gcp-datadog-integration module to manage this infrastructure through Terraform, or follow the instructions in this section to:

1. Create a Pub/Sub topic and pull subscription to receive logs from a configured log sink
2. Create a custom Dataflow worker service account to provide least privilege to your Dataflow pipeline workers
3. Create a log sink to publish logs to the Pub/Sub topic
4. Create a Dataflow job using the Datadog template to stream logs from the Pub/Sub subscription to Datadog

You have full control over which logs are sent to Datadog through the logging filters you create in the log sink, including GCE and GKE logs. See Google’s Logging query language page for information about writing filters. For a detailed examination of the created architecture, see Stream logs from Google Cloud to Datadog in the Cloud Architecture Center.

Note: You must enable the Dataflow API to use Google Cloud Dataflow. See Enabling APIs in the Google Cloud documentation for more information.

To collect logs from applications running in GCE or GKE, you can also use the Datadog Agent.

1. Create a Cloud Pub/Sub topic and subscription

1. Go to the Cloud Pub/Sub console and create a new topic. Select the option Add a default subscription to simplify the setup.

   Note: You can also manually configure a Cloud Pub/Sub subscription with the Pull delivery type. If you manually create your Pub/Sub subscription, leave the Enable dead lettering box unchecked. For more details, see Unsupported Pub/Sub features.

2. Give that topic an explicit name such as export-logs-to-datadog and click Create.

3. Create an additional topic and default subscription to handle any log messages rejected by the Datadog API. The name of this topic is used within the Datadog Dataflow template as part of the path configuration for the outputDeadletterTopic template parameter. When you have inspected and corrected any issues in the failed messages, send them back to the original export-logs-to-datadog topic by running a Pub/Sub to Pub/Sub template job.

4. Datadog recommends creating a secret in Secret Manager with your valid Datadog API key value, for later use in the Datadog Dataflow template.

Warning: Cloud Pub/Subs are subject to Google Cloud quotas and limitations. If the number of logs you have exceeds those limitations, Datadog recommends you split your logs over several topics. See the Monitor the Pub/Sub Log Forwarding section for information on setting up monitor notifications if you approach those limits.

2. Create a custom Dataflow worker service account

The default behavior for Dataflow pipeline workers is to use your project’s Compute Engine default service account, which grants permissions to all resources in the project. If you are forwarding logs from a Production environment, you should instead create a custom worker service account with only the necessary roles and permissions, and assign this service account to your Dataflow pipeline workers.

1. Go to the Service Accounts page in the Google Cloud console and select your project.
2. Click CREATE SERVICE ACCOUNT and give the service account a descriptive name. Click CREATE AND CONTINUE.
3. Add the roles in the required permissions table and click DONE.

Required permissions

Dataflow Admin

roles/dataflow.admin
Allow this service account to perform Dataflow administrative tasks.

Dataflow Worker

roles/dataflow.worker
Allow this service account to perform Dataflow job operations.

Pub/Sub Viewer

roles/pubsub.viewer
Allow this service account to view messages from the Pub/Sub subscription with your Google Cloud logs.

Pub/Sub Subscriber

roles/pubsub.subscriber
Allow this service account to consume messages from the Pub/Sub subscription with your Google Cloud logs.

Pub/Sub Publisher

roles/pubsub.publisher
Allow this service account to publish failed messages to a separate subscription, which allows for analysis or resending the logs.

Secret Manager Secret Accessor

roles/secretmanager.secretAccessor
Allow this service account to access the Datadog API key in Secret Manager.

Storage Object Admin

roles/storage.objectAdmin
Allow this service account to read and write to the Cloud Storage bucket specified for staging files.

Note: If you don’t create a custom service account for the Dataflow pipeline workers, ensure that the default Compute Engine service account has the required permissions above.

3. Export logs from Google Cloud Pub/Sub topic

1. Go to the Logs Explorer page in the Google Cloud console.

2. From the Log Router tab, select Create Sink.

3. Provide a name for the sink.

4. Choose Cloud Pub/Sub as the destination and select the Cloud Pub/Sub topic that was created for that purpose. Note: The Cloud Pub/Sub topic can be located in a different project.

5. Choose the logs you want to include in the sink with an optional inclusion or exclusion filter. You can filter the logs with a search query, or use the sample function. For example, to include only 10% of the logs with a severity level of ERROR, create an inclusion filter with severity="ERROR" AND sample(insertId, 0.1).

6. Click Create Sink.

Note: It is possible to create several exports from Google Cloud Logging to the same Cloud Pub/Sub topic with different sinks.

4. Create and run the Dataflow job

1. Go to the Create job from template page in the Google Cloud console.

2. Give the job a name and select a Dataflow regional endpoint.

3. Select Pub/Sub to Datadog in the Dataflow template dropdown, and the Required parameters section appears. a. Select the input subscription in the Pub/Sub input subscription dropdown. b. Enter the following in the Datadog Logs API URL field:

   https://

   Note: Ensure that the Datadog site selector on the right of the page is set to your Datadog site before copying the URL above.

   c. Select the topic created to receive message failures in the Output deadletter Pub/Sub topic dropdown. d. Specify a path for temporary files in your storage bucket in the Temporary location field.

4. Under Optional Parameters, check Include full Pub/Sub message in the payload.

5. If you created a secret in Secret Manager with your Datadog API key value as mentioned in step 1, enter the resource name of the secret in the Google Cloud Secret Manager ID field.

See Template parameters in the Dataflow template for details on using the other available options:

• apiKeySource=KMS with apiKeyKMSEncryptionKey set to your Cloud KMS key ID and apiKey set to the encrypted API key
• Not recommended: apiKeySource=PLAINTEXT with apiKey set to the plaintext API key

6. If you created a custom worker service account, select it in the Service account email dropdown.

7. Click RUN JOB.

Note: If you have a shared VPC, see the Specify a network and subnetwork page in the Dataflow documentation for guidelines on specifying the Network and Subnetwork parameters.

Validation

New logging events delivered to the Cloud Pub/Sub topic appear in the Datadog Log Explorer.

Note: You can use the Google Cloud Pricing Calculator to calculate potential costs.

Monitor the Cloud Pub/Sub log forwarding

The Google Cloud Pub/Sub integration provides helpful metrics to monitor the status of the log forwarding:

• gcp.pubsub.subscription.num_undelivered_messages for the number of messages pending delivery
• gcp.pubsub.subscription.oldest_unacked_message_age for the age of the oldest unacknowledged message in a subscription

Use the metrics above with a metric monitor to receive alerts for the messages in your input and deadletter subscriptions.

Monitor the Dataflow pipeline

Use Datadog’s Google Cloud Dataflow integration to monitor all aspects of your Dataflow pipelines. You can see all your key Dataflow metrics on the out-of-the-box dashboard, enriched with contextual data such as information about the GCE instances running your Dataflow workloads, and your Pub/Sub throughput.

You can also use a preconfigured Recommended Monitor to set up notifications for increases in backlog time in your pipeline. For more information, read Monitor your Dataflow pipelines with Datadog in the Datadog blog.

Collecting Google Cloud logs with a Pub/Sub Push subscription is in the process of being deprecated.

The above documentation for the Push subscription is only maintained for troubleshooting or modifying legacy setups.

Use a Pull subscription with the Datadog Dataflow template as described under Dataflow Method to forward your Google Cloud logs to Datadog instead.