Overview

Datadog Cloud SIEM applies detection rules to all processed logs in Datadog to detect threats, like a targeted attack, a threat intel listed IP communicating with your systems, or an insecure resource modification. The threats are surfaced as Security Signals in the Security Signals Explorer for triaging.

Use Google Cloud Dataflow and the Datadog template to forward logs from your Google Cloud services to Datadog. This guide walks you through the following steps so that you can start detecting threats with your Google Cloud audit logs:

1. Enable Data Access audit logs
2. Create a Google Cloud publish/subscription (Pub/Sub) topic and pull subscription to receive logs from a configured log sink
3. Create a custom Dataflow worker service account
4. Create a log sink to publish logs to the Pub/Sub
5. Create and run the Dataflow job
6. Use Cloud SIEM to triage Security Signals

Enable Data Access audit logs

1. Navigate to the IAM & Admin Console > Audit Log.
2. Select the services for which you want to enable data access logs.
3. In the Log Types panel, enable Admin Read, Data Read, and Data Write.
4. Click Save.

Change default configuration for new services

If a new Google Cloud service is added, it inherits your default audit configuration.

To ensure that Data Access audit logs are captured for new Google Cloud services, modify your default audit configuration:

1. Navigate to the IAM & Admin Console > Audit Log.
2. Enable Admin Read, Data Read, and Data Write.
3. Click Save.

Create a Google Cloud publish/subscription (Pub/Sub) system

1. Navigate to Pub/Sub > Topics.
2. Click Create Topic.
3. Enter a descriptive topic name. For example, export-audit-logs-to-datadog.
4. Leave Add a default subscription selected, which creates a subscription with default configuration values. The name of the subscription is automatically generated as your topic name with “-sub” appended to it. This subscription name is used when you create your Dataflow job later.
5. Click Create.

Create an additional topic and subscription for outputDeadletterTopic parameter

Create an additional topic and default subscription to handle any log messages rejected by the Datadog API. This topic is used when you set up the Dataflow job later.

1. Navigate back to Pub/Sub > Topics
2. Click Create Topic.
3. Enter a descriptive topic name.
4. Leave Add a default subscription selected.
5. Click Create.

Warning: Pub/subs are subject to Google Cloud quotas and limitations. If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics. See Monitor the Log Forwarding for information on how to set up a monitor to notify when you are close to those limits.

Create a secret in Secret Manager

Datadog recommends creating a secret in Secret Manager with your valid Datadog API key value. This secret is used when you set up the Dataflow job later.

1. Navigate to Security > Secret Manager.
2. Click Create Secret.
3. Enter a name for the secret.
4. Copy your Datadog API key and paste it into the Secret value section.
5. Optionally, set the other configurations based on your use case.
6. Click Create Secret.

Create a custom Dataflow worker service account

The default behavior for Dataflow pipeline workers is to use your project’s Compute Engine default service account, which grants permissions to all resources in the project. If you are forwarding logs from a production environment, create a custom worker service account with only the necessary roles and permissions, and assign this service account to your Dataflow pipeline workers.

Note: If you are not creating a custom service account for the Dataflow pipeline workers, ensure that the default Compute Engine service account has the required permissions below.

1. Navigate to Google Cloud’s Service Account page.
2. Select your project.
3. Click Create Service Account.
4. Enter a descriptive name for the service account.
5. Click Create and Continue.
6. Add the following roles:

   Required permissions

   Role	Path	Description
   Dataflow Admin	roles/dataflow.admin	Allow this service account to perform Dataflow administrative tasks
   Dataflow Worker	roles/dataflow.worker	Allow this service account to perform Dataflow job operations
   Pub/Sub Viewer	roles/pubsub.viewer	Allow this service account to view messages from the Pub/Sub subscription with your Google Cloud logs
   Pub/Sub Subscriber	roles/pubsub.subscriber	Allow this service account to consume messages from the Pub/Sub subscription with your Google Cloud logs
   Pub/Sub Publisher	roles/pubsub.publisher	Allow this service account to publish failed messages to a separate subscription, which allows for analysis or resending the logs
   Secret Manager Secret Accessor	roles/secretmanager.secretAccessor	Allow this service account to access the Datadog API key in Secret Manager
   Storage Object Admin	roles/storage.objectAdmin	Allow this service account to read and write to the Cloud Storage bucket specified for staging files

7. Click Continue.
8. Click Done.

Create a log sink to publish logs to the Pub/Sub

1. Navigate to Google Cloud’s Logs Explorer.
2. Select Log Router in the left side menu.
3. Click Create Sink.
4. Enter a descriptive name for the sink.
5. Click Next.
6. In the Select Sink Service dropdown menu, select Cloud Pub/Sub topic.
   Note: The Cloud Pub/Sub topic can be located in a different project.
7. In the Select a Cloud Pub/Sub topic, select the Pub/Sub created earlier.
8. Click Next.
9. Enter an inclusion filter for the logs you want to send to Datadog.
10. Click Next.
11. Optionally, enter an exclusion filter to exclude logs you do not want sent to Datadog.
12. Click Create Sink.

Note: You can create multiple exports from Google Cloud Logging to the same Pub/Sub topic with different sinks.

Create and run the Dataflow job

1. Navigate to Google Cloud Dataflow.
2. Click Create job from template.
3. Enter a name for the job.
4. Select a regional endpoint.
5. In the Dataflow template dropdown menu, select Pub/Sub to Datadog.
6. In Required Parameters section:
   a. In the Pub/Sub input subscription dropdown menu, select the default subscription that was created earlier when you created a new Pub/Sub system.
   b. Enter the following in the Datadog Logs API URL field:

   https://
   

   Note: Ensure that the Datadog site selector on the right of this documentation page is set to your Datadog site before copying the URL above.
   c. In the Output deadletter Pub/Sub topic field, select the additional topic you created earlier for receiving messages rejected by the Datadog API.
   d. Specify a path for temporary files in your storage bucket in the Temporary location field.
7. If you created a secret in Secret Manager for your Datadog API key value earlier:
   a. Click Optional Parameters to see the additional fields.
   b. Enter the resource name of the secret in the Google Cloud Secret Manager ID field.
   To get the resource name, go to your secret in Secret Manager. Click on your secret. Click on the three dots under Action and select Copy resource name.
   c. Enter SECRET_MANAGER in the Source of the API key passed field.
8. If you are not using a secret for your Datadog API key value:
   • Recommended:
     • Set Source of API key passed to KMS.
     • Set Google Cloud KMS key for the API key to your Cloud KMS key ID.
     • Set Logs API Key to the encrypted API key.
   • Not recommended: Source of API key passed set to PLAINTEXT with Logs API Key set to the plaintext API key.
9. See Template parameters in the Dataflow template for details on other available options.
10. If you created a custom worker service account, select it in the Service account email dropdown menu.
11. Click Run Job.

See new logging events delivered to the Cloud Pub/Sub topic in the Datadog Log Explorer.

Use Cloud SIEM to triage Security Signals

Cloud SIEM applies out-of-the-box detection rules to all processed logs, including the Google Cloud audit logs you have just set up. When a threat is detected with a detection rule, a Security Signal is generated and can be viewed in the Security Signals Explorer.

• Go to the Cloud SIEM Signals Explorer to view and triage threats. See Security Signals Explorer for further details.
• You can also use the Google Cloud Audit Log dashboard to investigate anomalous activity.
• See out-of-the-box detection rules that are applied to your logs.
• Create new rules to detect threats that match your specific use case.

Further reading

Additional helpful documentation, links, and articles:

Explore Cloud SIEM default detection rulesDOCUMENTATION Learn about the Security Signals ExplorerDOCUMENTATION Create new detection rulesDOCUMENTATION Collect Google Cloud Platform logsDOCUMENTATION Visualize activity in your Google Cloud environment with Datadog Cloud SIEM InvestigatorBLOG