Send AWS Services Logs With The Datadog Lambda Function

Docs > Log Management > Logs Guides > Send AWS Services Logs With The Datadog Lambda Function

AWS service logs can be collected with the Datadog Forwarder Lambda function. This Lambda—which triggers on S3 Buckets, CloudWatch log groups, and EventBridge events—forwards logs to Datadog.

To start collecting logs from your AWS services:

Set up the Datadog Forwarder Lambda function in your AWS account.
Enable logging for your AWS service (most AWS services can log to a S3 bucket or CloudWatch Log Group).
Set up the triggers that cause the Forwarder Lambda to execute when there are new logs to be forwarded. There are two ways to configure the triggers.

Notes:

You can use AWS PrivateLink to send your logs over a private connection.
CloudFormation creates an IAM policy which includes KMS:Decrypt for all resources, and does not align with AWS Security Hub’s best practice. This permission is used to decrypt objects from KMS-encrypted S3 buckets to set up the Lambda function, and the KMS key used to encrypt the S3 buckets cannot be predicted. You can safely delete this permission after the installation finishes successfully.

Enable logging for your AWS service

Any AWS service that generates logs into a S3 bucket or a CloudWatch Log Group is supported. Find setup instructions for the most used services in the table below:

AWS service	Activate AWS service logging	Send AWS logs to Datadog
API Gateway	Enable Amazon API Gateway logs	Manual and automatic log collection.
Cloudfront	Enable Amazon CloudFront logs	Manual and automatic log collection.
CloudTrail	Enable AWS CloudTrail logs	Manual and automatic log collection. See AWS Configuration for Cloud SIEM if you are setting up AWS CloudTrail for Cloud SIEM.
DynamoDB	Enable Amazon DynamoDB logs	Manual log collection.
EC2	`-`	Use the Datadog Agent to send your logs to Datadog.
ECS	`-`	Use the docker agent to gather your logs.
Elastic Load Balancing (ELB)	Enable Amazon ELB logs	Manual and automatic log collection.
Lambda	`-`	Manual and automatic log collection.
RDS	Enable Amazon RDS logs	Manual log. collection.
Route 53	Enable Amazon Route 53 logs	Manual log collection.
S3	Enable Amazon S3 logs	Manual and automatic log collection.
SNS	SNS does not provide logs, but you can process logs and events that are transiting through to the SNS Service.	Manual log collection.
SSM	`-`	Automatic log collection.
RedShift	Enable Amazon Redshift logs	Manual and automatic log collection.
Verified Access	Enable Verified Access logs	Manual log collection.
VPC	Enable Amazon VPC logs	Manual log collection.
Step Functions	Enable Amazon Step Functions logs	Manual log collection.
Web Application Firewall	Enable Amazon WAF logs	Manual and automatic log collection.
MWAA	Enable Amazon MWAA logs	Manual and automatic log collection.

Set up triggers

There are two options when configuring triggers on the Datadog Forwarder Lambda function:

Automatically: Datadog automatically retrieves the log locations for the selected AWS services and adds them as triggers on the Datadog Forwarder Lambda function. Datadog also keeps the list up to date.
Manually: Set up each trigger yourself.

Automatically set up triggers

Datadog can automatically configure triggers on the Datadog Forwarder Lambda function to collect AWS logs from the following sources and locations:

Source	Location
Apache Airflow (MWAA)	Cloudwatch
API Gateway Access Logs	CloudWatch
API Gateway Execution Logs	CloudWatch
Application ELB Access Logs	S3
Classic ELB Access Logs	S3
CloudFront Access Logs	S3
Cloudtrail Logs	S3, Cloudwatch
Lambda Logs	CloudWatch
Lambda@Edge Logs	Cloudwatch
Redshift Logs	S3
S3 Access Logs	S3
SSM Command Logs	Cloudwatch
Step Functions	CloudWatch
Web Application Firewall	S3, CloudWatch

Note: Subscription filters are not created automatically by the DatadogForwarder. Create them directly on a Log Group.

If you haven’t already, set up the Datadog log collection AWS Lambda function.

Ensure the policy of the IAM role used for Datadog-AWS integration has the following permissions. Information on how these permissions are used can be found in the descriptions below:

"airflow:ListEnvironments",
"airflow:GetEnvironment",
"cloudfront:GetDistributionConfig",
"cloudfront:ListDistributions",
"cloudtrail:GetTrail",
"cloudtrail:ListTrails",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"lambda:InvokeFunction",
"lambda:List*",
"lambda:GetPolicy",
"redshift:DescribeClusters",
"redshift:DescribeLoggingStatus",
"s3:GetBucketLogging",
"s3:GetBucketLocation",
"s3:GetBucketNotification",
"s3:ListAllMyBuckets",
"s3:PutBucketNotification",
"ssm:GetServiceSetting",
"ssm:ListCommands",
"states:ListStateMachines",
"states:DescribeStateMachine",
"wafv2:ListLoggingConfigurations",
"logs:PutSubscriptionFilter",
"logs:DeleteSubscriptionFilter",
"logs:DescribeSubscriptionFilters"

AWS Permission	Description
`airflow:ListEnvironments`	List all MWAA environment names
`airflow:GetEnvironment`	Get information about a MWAA environment
`cloudfront:GetDistributionConfig`	Get the name of the S3 bucket containing CloudFront access logs.
`cloudfront:ListDistributions`	List all CloudFront distributions.
`cloudtrail:GetTrail`	Get Trail logging information.
`cloudtrail.ListTrails`	List all Cloudtrail trails.
`elasticloadbalancing:` `DescribeLoadBalancers`	List all load balancers.
`elasticloadbalancing:` `DescribeLoadBalancerAttributes`	Get the name of the S3 bucket containing ELB access logs.
`lambda:InvokeFunction`	Invoke a Lambda function.
`lambda:List*`	List all Lambda functions.
`lambda:GetPolicy`	Get the Lambda policy when triggers are to be removed.
`redshift:DescribeClusters`	List all Redshift clusters.
`redshift:DescribeLoggingStatus`	Get the name of the S3 bucket containing Redshift Logs.
`s3:GetBucketLogging`	Get the name of the S3 bucket containing S3 access logs.
`s3:GetBucketLocation`	Get the region of the S3 bucket containing S3 access logs.
`s3:GetBucketNotification`	Get existing Lambda trigger configurations.
`s3:ListAllMyBuckets`	List all S3 buckets.
`s3:PutBucketNotification`	Add or remove a Lambda trigger based on S3 bucket events.
`ssm:GetServiceSetting`	Get the SSM service setting for customer script log group name.
`ssm:ListCommands`	List all SSM commands.
`states:ListStateMachines`	List all Step Functions.
`states:DescribeStateMachine`	Get logging details about a Step Function.
`wafv2:ListLoggingConfigurations`	List all logging configurations of the Web Application Firewall.
`logs:PutSubscriptionFilter`	Add a Lambda trigger based on CloudWatch Log events
`logs:DeleteSubscriptionFilter`	Remove a Lambda trigger based on CloudWatch Log events
`logs:DescribeSubscriptionFilters`	List the subscription filters for the specified log group.

In the AWS Integration page, select the AWS Account to collect logs from and click on the Log Collection tab.
Enter the ARN of the Lambda created in the previous section and click Add.
Select the services from which you’d like to collect logs and click Save. To stop collecting logs from a particular service, deselect the log source.
If you have logs across multiple regions, you must create additional Lambda functions in those regions and enter them in this page.
To stop collecting all AWS logs, hover over a Lambda and click the Delete icon. All triggers for that function are removed.
Within a few minutes of this initial setup, your AWS Logs appear in the Datadog Log Explorer.

Manually set up triggers

Collecting logs from CloudWatch log group

If you are collecting logs from a CloudWatch log group, configure the trigger to the Datadog Forwarder Lambda function using one of the following methods:

In the AWS console, go to Lambda.
Click Functions and select the Datadog Forwarder.
Click Add trigger and select CloudWatch Logs.
Select the log group from the dropdown menu.
Enter a name for your filter, and optionally specify a filter pattern.
Click Add.
Go to the Datadog Log section to explore any new log events sent to your log group.

For Terraform users, you can provision and manage your triggers using the aws_cloudwatch_log_subscription_filter resource. See sample code below.

data "aws_cloudwatch_log_group" "some_log_group" {
  name = "/some/log/group"
}

resource "aws_lambda_permission" "lambda_permission" {
  action        = "lambda:InvokeFunction"
  function_name = "datadog-forwarder" # this is the default but may be different in your case
  principal     = "logs.amazonaws.com" # or logs.amazonaws.com.cn for China*
  source_arn    = data.aws_cloudwatch_log_group.some_log_group.arn
}

resource "aws_cloudwatch_log_subscription_filter" "datadog_log_subscription_filter" {
  name            = "datadog_log_subscription_filter"
  log_group_name  = <CLOUDWATCH_LOG_GROUP_NAME> # for example, /some/log/group
  destination_arn = <DATADOG_FORWARDER_ARN> # for example,  arn:aws:lambda:us-east-1:123:function:datadog-forwarder
  filter_pattern  = ""
}

*All use of Datadog Services in (or in connection with environments within) mainland China is subject to the disclaimer published in the Restricted Service Locations section on our website.

For AWS CloudFormation users, you can provision and manage your triggers using the CloudFormation AWS::Logs::SubscriptionFilter resource. See sample code below.

The sample code also work for AWS SAM and Serverless Framework. For Serverless Framework, put the code under the resources section within your serverless.yml.

Resources:
  MyLogSubscriptionFilter:
    Type: "AWS::Logs::SubscriptionFilter"
    Properties:
      DestinationArn: "<DATADOG_FORWARDER_ARN>"
      LogGroupName: "<CLOUDWATCH_LOG_GROUP_NAME>"
      FilterPattern: ""

Collecting logs from S3 buckets

If you are collecting logs from an S3 bucket, configure the trigger to the Datadog Forwarder Lambda function using one of the following methods:

Once the Lambda function is installed, manually add a trigger on the S3 bucket that contains your logs in the AWS console:
Select the bucket and then follow the AWS instructions:
Set the correct event type on S3 buckets:

Once done, go into your Datadog Log section to start exploring your logs!

For Terraform users, you can provision and manage your triggers using the aws_s3_bucket_notification resource. See the sample code below.

resource "aws_s3_bucket_notification" "my_bucket_notification" {
  bucket = my_bucket
  lambda_function {
    lambda_function_arn = "<DATADOG_FORWARDER_ARN>"
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSLogs/"
    filter_suffix       = ".log"
  }
}

For CloudFormation users, you can configure triggers using the CloudFormation NotificationConfiguration for your S3 bucket. See the sample code below.

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: "<MY_BUCKET>"
      NotificationConfiguration:
        LambdaConfigurations:
        - Event: 's3:ObjectCreated:*'
          Function: "<DATADOG_FORWARDER_ARN>"

You can scrub emails or IP address from logs sent by the Lambda function, or define a custom scrubbing rule in the Lambda parameters. You can also exclude or send only those logs that match a specific pattern by using the filtering option.