Amazon S3 Destination

The Amazon S3 destination is available for the Archive Logs template. Use this destination to send your logs in Datadog-rehydratable format to an Amazon S3 bucket for archiving. You need to set up Datadog Log Archives if you haven’t already, and then set up the destination in the pipeline UI.

Configure Log Archives

If you already have a Datadog Log Archive configured for Observability Pipelines, skip to Set up the destination for your pipeline.

You need to have Datadog’s AWS integration installed to set up Datadog Log Archives.

Create an Amazon S3 bucket

  1. Navigate to Amazon S3 buckets.
  2. Click Create bucket.
  3. Enter a descriptive name for your bucket.
  4. Do not make your bucket publicly readable.
  5. Optionally, add tags.
  6. Click Create bucket.

Set up an IAM policy that allows Workers to write to the S3 bucket

  1. Navigate to the IAM console.
  2. Select Policies in the left side menu.
  3. Click Create policy.
  4. Click JSON in the Specify permissions section.
  5. Copy the below policy and paste it into the Policy editor. Replace <MY_BUCKET_NAME> and <MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1> with the information for the S3 bucket you created earlier.
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DatadogUploadAndRehydrateLogArchives",
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:GetObject"],
            "Resource": "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*"
        },
        {
            "Sid": "DatadogRehydrateLogArchivesListBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<MY_BUCKET_NAME>"
        }
    ]
}
  6. Click Next.
  7. Enter a descriptive policy name.
  8. Optionally, add tags.
  9. Click Create policy.

    Create an IAM user

    Create an IAM user and attach the IAM policy you created earlier to it.

    1. Navigate to the IAM console.
    2. Select Users in the left side menu.
    3. Click Create user.
    4. Enter a username.
    5. Click Next.
    6. Select Attach policies directly.
    7. Choose the IAM policy you created earlier to attach to the new IAM user.
    8. Click Next.
    9. Optionally, add tags.
    10. Click Create user.

    Create access credentials for the new IAM user. The AWS access key and AWS secret access key are added as environment variables in the Install the Observability Pipelines Worker step.

    Create a service account

    Create a service account to use the policy you created above.

    Create an IAM user

    Create an IAM user and attach the IAM policy you created earlier to it.

    1. Navigate to the IAM console.
    2. Select Users in the left side menu.
    3. Click Create user.
    4. Enter a username.
    5. Click Next.
    6. Select Attach policies directly.
    7. Choose the IAM policy you created earlier to attach to the new IAM user.
    8. Click Next.
    9. Optionally, add tags.
    10. Click Create user.

    Create access credentials for the new IAM user. The AWS access key and AWS secret access key are added later as environment variables when you install the Observability Pipelines Worker.

    Create an IAM user

    Create an IAM user and attach the IAM policy you created earlier to it.

    1. Navigate to the IAM console.
    2. Select Users in the left side menu.
    3. Click Create user.
    4. Enter a username.
    5. Click Next.
    6. Select Attach policies directly.
    7. Choose the IAM policy you created earlier to attach to the new IAM user.
    8. Click Next.
    9. Optionally, add tags.
    10. Click Create user.

    Create access credentials for the new IAM user. The AWS access key and AWS secret access key are added as environment variables in the Install the Observability Pipelines Worker step.

    Connect the S3 bucket to Datadog Log Archives

    1. Navigate to Datadog Log Forwarding.
    2. Click New archive.
    3. Enter a descriptive archive name.
    4. Add a query that filters out all logs going through log pipelines so that none of those logs go into this archive. For example, add the query observability_pipelines_read_only_archive, assuming no logs going through the pipeline have that tag added.
    5. Select AWS S3.
    6. Select the AWS account that your bucket is in.
    7. Enter the name of the S3 bucket.
    8. Optionally, enter a path.
    9. Check the confirmation statement.
    10. Optionally, add tags and define the maximum scan size for rehydration. See Advanced settings for more information.
    11. Click Save.

    See the Log Archives documentation for additional information.

    Set up the destination for your pipeline

    Set up the Amazon S3 destination and its environment variables when you set up an Archive Logs pipeline. The information below is configured in the pipelines UI.

    1. Enter the S3 bucket name for the S3 bucket you created earlier.
    2. Enter the AWS region the S3 bucket is in.
    3. Enter the key prefix. Prefixes are useful for partitioning objects. For example, you can use a prefix as an object key to store objects under a particular directory. If using a prefix for this purpose, it must end in / to act as a directory path; a trailing / is not automatically added.
    4. Select the storage class for your S3 bucket in the Storage Class dropdown menu.

    Your AWS access key ID and AWS secret access key are set as environment variables when you install the Worker later.

    Set the environment variables

    Enter the AWS access key ID and AWS secret access key for the S3 archive bucket you created earlier.

    How the destination works

    Event batching

    A batch of events is flushed when one of these parameters is met. See event batching for more information.

    Max EventsMax BytesTimeout (seconds)
    None100,000,000900
    PREVIEWING: piotr_wolski/update-dsm-docs