Overview
This guide provides an overview of the process for integrating an Amazon Web Services(AWS) account with Datadog using Datadog’s CloudFormation template.
At a high level, this involves creating an IAM role and associated policy to enable Datadog’s AWS account to make API calls into your AWS account for collecting or pushing data. The template also deploys the Datadog Forwarder Lambda function for sending logs to Datadog. Using the CloudFormation template provides all the tools needed to send this data to your Datadog account, and Datadog maintains the CloudFormation template to provide the latest functionality.
After the initial connection is established, you can enable individual AWS service integrations relevant to your AWS environment. With a single click, Datadog provisions the necessary resources in your AWS account and begins querying metrics and events for the services you use. For popular AWS services you are using, Datadog provisions out-of-the-box dashboards, providing immediate and customizable visibility. This guide demonstrates setting up the integration and installing the Datadog Agent on an Amazon Linux EC2 instance, as well as providing a broad overview of the integration’s capabilities. See the Enable integrations for individual AWS service section for a list of the available sub-integrations.
This process can be repeated for as many AWS accounts as necessary, or you can also use the API, AWS CLI, or Terraform to set up multiple accounts at once. For more information, read the Datadog-Amazon CloudFormation guide.
Prerequisites
Before getting started, ensure you have the following prerequisites:
An AWS account. Your AWS user needs the following IAM permissions to successfully run the CloudFormation template:
- cloudformation:CreateStack
- cloudformation:CreateUploadBucket
- cloudformation:DeleteStack
- cloudformation:DescribeStacks
- cloudformation:DescribeStackEvents
- cloudformation:GetStackPolicy
- cloudformation:GetTemplateSummary
- cloudformation:ListStacks
- cloudformation:ListStackResources
- ec2:DescribeSecurityGroups
- ec2:DescribeSubnets
- ec2:DescribeVpcs
- iam:AttachRolePolicy
- iam:CreatePolicy
- iam:CreateRole
- iam:DeleteRole
- iam:DeleteRolePolicy
- iam:DetachRolePolicy
- iam:GetRole
- iam:GetRolePolicy
- iam:PassRole
- iam:PutRolePolicy
- iam:TagRole
- iam:UpdateAssumeRolePolicy
- kms:Decrypt
- lambda:AddPermission
- lambda:CreateFunction
- lambda:DeleteFunction
- lambda:GetCodeSigningConfig
- lambda:GetFunction
- lambda:GetFunctionCodeSigningConfig
- lambda:GetLayerVersion
- lambda:InvokeFunction
- lambda:PutFunctionConcurrency
- lambda:RemovePermission
- lambda:TagResource
- logs:CreateLogGroup
- logs:DeleteLogGroup
- logs:DescribeLogGroups
- logs:PutRetentionPolicy
- oam:ListSinks
- oam:ListAttachedLinks
- s3:CreateBucket
- s3:DeleteBucket
- s3:DeleteBucketPolicy
- s3:GetEncryptionConfiguration
- s3:GetObject
- s3:GetObjectVersion
- s3:PutBucketPolicy
- s3:PutBucketPublicAccessBlock
- s3:PutEncryptionConfiguration
- s3:PutLifecycleConfiguration
- secretsmanager:CreateSecret
- secretsmanager:DeleteSecret
- secretsmanager:GetSecretValue
- secretsmanager:PutSecretValue
- serverlessrepo:CreateCloudFormationTemplate
Setup
Go to the AWS integration configuration page in Datadog and click Add AWS Account.
Configure the integration’s settings under the Automatically using CloudFormation option.
a. Select the AWS regions to integrate with.
b. Add your Datadog API key.
c. Optionally, send logs and other data to Datadog with the Datadog Forwarder Lambda.
d. Optionally, enable Cloud Security Management Misconfigurations to scan your cloud environment, hosts, and containers for misconfigurations and security risks.
Click Launch CloudFormation Template. This opens the AWS Console and loads the CloudFormation stack. All the parameters are filled in based on your selections in the prior Datadog form, so you do not need to edit those unless desired.
Note: The DatadogAppKey parameter enables the CloudFormation stack to make API calls to Datadog to add and edit the Datadog configuration for this AWS account. The key is automatically generated and tied to your Datadog account.
Check the required boxes from AWS and click Create stack. This launches the creation process for the Datadog stack along with three nested stacks. This could take several minutes. Ensure that the stack is successfully created before proceeding.
After the stack is created, go back to the AWS integration tile in Datadog and click Ready!
Wait up to 10 minutes for data to start being collected, and then view the out-of-the-box AWS overview dashboard to see metrics sent by your AWS services and infrastructure:
Enable integrations for individual AWS services
See the Integrations page for a full listing of the available sub-integrations. Many of these integrations are installed by default when Datadog recognizes data coming in from your AWS account.
Send logs
There are two ways of sending AWS service logs to Datadog:
- Amazon Data Firehose destination: Use the Datadog destination in your Amazon Data Firehose delivery stream to forward logs to Datadog. It is recommended to use this approach when sending logs from CloudWatch in a very high volume.
- Forwarder Lambda function: Deploy the Datadog Forwarder Lambda function, which subscribes to S3 buckets or your CloudWatch log groups and forwards logs to Datadog. You must use this approach to send traces, enhanced metrics, or custom metrics from Lambda functions asynchronously through logs. Datadog also recommends you use this approach to sending logs from S3 or other resources that cannot directly stream data to Kinesis.
Read the Enable logging for your AWS service section to get logs flowing for the most-used AWS services.
Validation
Once you have enabled logs, find them in the Log Explorer using either the source or service facets from the facet panel, such as this example from S3:
Deeper visibility with the Datadog Agent on EC2
By default the Datadog AWS integration crawls the CloudWatch API for AWS-provided metrics, but you can gain even deeper visibility into your EC2 instances with the Datadog Agent. The Agent is a lightweight daemon that reports metrics and events, and can also be configured for logs and traces. The Agent Installation section of the Datadog application provides instructions for installing the Agent on a wide variety of operating systems. Many operating systems (for example, Amazon Linux) have one-step installation commands that you can run from the instance terminal to install the Agent:
Once the Agent is installed, it’s graphically represented within the Infrastructure List with a bone icon:
The screen shot above shows the host with the Datadog Agent reporting data from the System and NTP checks. The System check provides metrics around CPU, memory, filesystem, and I/O, providing additional insights into the host. You can enable additional integrations to suit the environment and use case, or additionally use DogStatsD to send custom metrics directly to Datadog.
See the FAQ on why you should install the Datadog Agent on your cloud instances for more information about the benefits of this approach.
Using the Datadog Agent with Amazon Container Services
For containerized environments, you can use the Datadog Agent, whether you’re managing your instances or leveraging Fargate for a serverless environment.
ECS with EC2 launch type
Use the Amazon ECS documentation to run the Datadog Docker Agent on the EC2 instances in your ECS cluster. Review the Amazon ECS Data Collection documentation to see the metrics and events reported to your Datadog account.
ECS with Fargate launch type
Use the Amazon ECS on AWS Fargate documentation to run the Agent as a container in the same task definition as your application. Note: Datadog Agent version 6.1.1 or higher is needed to take full advantage of the Fargate integration.
AWS Batch with Fargate orchestration type
Use the Amazon ECS on AWS Fargate for AWS Batch documentation to run the Agent as a container in the same AWS Batch job definition as your application. Note: Datadog Agent version 6.1.1 or higher is needed to take full advantage of the Fargate integration.
EKS
You don’t need any specific configuration for Amazon Elastic Kubernetes Service (EKS), as mentioned in the Kubernetes Distributions documentation. Use the dedicated Kubernetes documentation to deploy the Agent in your EKS cluster.
EKS with Fargate
Because Fargate pods are managed by AWS, they exclude host-based system checks like CPU and memory. To collect data from your AWS Fargate pods, use the Amazon EKS on AWS Fargate documentation to run the Agent as a sidecar of your application pod with custom role-based access control (RBAC). Note: This requires Datadog Agent version 7.17 or higher.
EKS Anywhere
Use the EKS Anywhere documentation for on-premises Kubernetes clusters.
Create additional Datadog resources
In addition to using the Datadog UI or API, you can create many Datadog resources with the CloudFormation Registry. For visibility and troubleshooting, use dashboards to display key data, apply Functions, and find Metric Correlations.
To get notified of any unwanted or unexpected behavior in your account, create monitors. Monitors consistently evaluate the data reported to your account, and send Notifications to ensure that the right information gets to the right team members. Review the List of Notification Integrations for all the ways to notify your team.
Serverless
You can unify the metrics, traces, and logs from your AWS Lambda functions running serverless applications in Datadog. Check out Serverless for instructions on instrumenting your application, installing Serverless Libraries and Integrations, implementing Distributed Tracing with Serverless Applications, or Serverless Troubleshooting.
APM
To dig even deeper and gather more data from your applications and AWS services, enable collecting distributed traces from either the AWS X-Ray integration or from a host with the Datadog Agent using APM. Then, read the APM documentation for a better understanding of how to use this data to gain insights into your application performance.
Additionally, you can use Watchdog, an algorithmic feature for APM performance and infrastructure metrics, to automatically detect and be notified about potential application issues.
Security
Cloud SIEM
Review Getting Started with Cloud SIEM to evaluate your logs against the out-of-the-box Log Detection Rules. These rules are customizable, and when threats are detected, they generate security signals which can be accessed on the Security Signals Explorer. To ensure that the correct team is notified, use Notification Rules to configure notification preferences across multiple rules.
Cloud Security Management Misconfigurations
Use the Setting Up CSM Misconfigurations guide to learn about detecting and assessing misconfigurations in your cloud environment. Resource configuration data is evaluated against the out-of-the-box Cloud and Infrastructure compliance rules to flag attacker techniques and potential misconfigurations, allowing for fast response and remediation.
Troubleshooting
If you encounter the error Datadog is not authorized to perform sts:AssumeRole, see its dedicated troubleshooting page. For any other issues, see the AWS integration troubleshooting guide.
Further Reading
Additional helpful documentation, links, and articles:
