Amazon Kinesis

Overview

Amazon Kinesis is a fully managed, cloud-based service for real-time processing of large, distributed data streams.

Enable this integration to see in Datadog all your Kinesis metrics, and collect custom Kinesis tags.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first. There are no other installation steps that need to be performed.

Metric collection

In the AWS integration page, ensure that Kinesis is enabled under the Metric Collection tab.
Add those permissions to your Datadog IAM policy in order to collect Amazon Kinesis metrics:
- kinesis:ListStreams: List available streams.
- kinesis:DescribeStream: Add tags and new metrics for kinesis streams.
- kinesis:ListTagsForStream: Add custom tags.
For more information, see the Kinesis policies on the AWS website.
Install the Datadog - Amazon Kinesis integration.

Log collection

Enable logging

Datadog is one of the default destinations for Amazon Data Firehose delivery streams. AWS fully manages Amazon Data Firehose, so you don’t need to maintain any additional infrastructure or forwarding configurations for streaming logs.

You can set up an Amazon Data Firehose delivery stream in the AWS Firehose console, or automatically set up the destination using a CloudFormation template:

However, if you log to an S3 bucket, then use the AWS Lambda function. Make sure that amazon_kinesis is set as Target prefix.

If you haven’t already, set up the Datadog Forwarder Lambda function.
Once the Lambda function is installed, manually add a trigger on the S3 bucket or CloudWatch log group that contains your Amazon Kinesis logs in the AWS console:
- Add a manual trigger on the S3 bucket
- Add a manual trigger on the CloudWatch Log Group

Data Collected

Metrics

aws.kinesis.get_records_bytes (gauge)	Average number of bytes per GetRecords operation Shown as byte
aws.kinesis.get_records_bytes.maximum (gauge)	Maximum number of bytes per GetRecords operation Shown as byte
aws.kinesis.get_records_bytes.minimum (gauge)	Minimum number of bytes per GetRecords operation Shown as byte
aws.kinesis.get_records_bytes.sum (count)	Total number of bytes returned over all GetRecords operations Shown as byte
aws.kinesis.get_records_iterator_age (gauge)	Average difference between the current time and when the last record of a GetRecords call was written to the stream. Shown as second
aws.kinesis.get_records_iterator_age.maximum (gauge)	Maximum difference between the current time and when the last record of a GetRecords call was written to the stream. Shown as second
aws.kinesis.get_records_iterator_age_milliseconds (gauge)	Difference between the current time and when the last record of a GetRecords call was written to the stream. Shown as millisecond
aws.kinesis.get_records_iterator_age_milliseconds.maximum (gauge)	Maximum difference between the current time and when the last record of a GetRecords call was written to the stream. Shown as millisecond
aws.kinesis.get_records_latency (gauge)	Average time taken per GetRecords operation. Shown as millisecond
aws.kinesis.get_records_latency.maximum (gauge)	Maximum time taken per GetRecords operation. Shown as millisecond
aws.kinesis.get_records_records (gauge)	Average number of records per GetRecords operation Shown as record
aws.kinesis.get_records_records.maximum (gauge)	Maximum number of records per GetRecords operation Shown as record
aws.kinesis.get_records_records.minimum (gauge)	Minimum number of records per GetRecords operation Shown as record
aws.kinesis.get_records_records.sum (count)	Total number of records returned over all GetRecords operations Shown as record
aws.kinesis.get_records_success (count)	Sum of successful GetRecords operations per stream. Shown as event
aws.kinesis.get_records_success.average (count)	Average number of successful GetRecords operations per stream. Shown as event
aws.kinesis.incoming_bytes (gauge)	Average number of bytes successfully put to the Amazon Kinesis stream per operation. Shown as byte
aws.kinesis.incoming_bytes.sum (count)	Total number of bytes successfully put to the Amazon Kinesis stream. Shown as byte
aws.kinesis.incoming_records (count)	Total number of records sucessfully put to the Amazon Kinesis stream. Shown as record
aws.kinesis.incoming_records.average (gauge)	Average number of records sucessfully put to the Amazon Kinesis stream per operation. Shown as record
aws.kinesis.iterator_age_milliseconds (gauge)	The age of the last record in all GetRecords calls made against a shard, measured over the specified time period. Shown as millisecond
aws.kinesis.outgoing_bytes (gauge)	Average number of bytes retrieved in a GetRecords operation Shown as byte
aws.kinesis.outgoing_bytes.sum (count)	Total number of bytes retrieved from the Amazon Kinesis stream Shown as byte
aws.kinesis.outgoing_records (count)	The number of records retrieved from the shard, measured over the specified time period. Shown as record
aws.kinesis.outgoing_records.average (gauge)	The average number of records retrieved from the shard, measured over the specified time period. Shown as record
aws.kinesis.put_record_bytes (gauge)	Average number of bytes per PutRecord operation Shown as byte
aws.kinesis.put_record_bytes.maximum (gauge)	Maximum number of bytes per PutRecord operation Shown as byte
aws.kinesis.put_record_bytes.minimum (gauge)	Minimum number of bytes per PutRecord operation Shown as byte
aws.kinesis.put_record_bytes.sum (count)	Total number of bytes for all PutRecord operation Shown as byte
aws.kinesis.put_record_latency (gauge)	Average time taken per PutRecord operation. Shown as millisecond
aws.kinesis.put_record_latency.maximum (gauge)	Maximum time taken per PutRecord operation. Shown as millisecond
aws.kinesis.put_record_success (count)	Sum of successful PutRecord operations per Amazon Kinesis stream. Shown as event
aws.kinesis.put_record_success.average (count)	Average number of successful PutRecord operations per Amazon Kinesis stream. Shown as event
aws.kinesis.put_records_bytes (gauge)	Average number of bytes per PutRecords operation. Shown as byte
aws.kinesis.put_records_bytes.maximum (gauge)	Maximum number of bytes per PutRecords operation. Shown as byte
aws.kinesis.put_records_bytes.minimum (gauge)	Minimum number of bytes per PutRecords operation. Shown as byte
aws.kinesis.put_records_bytes.sum (count)	Total number of bytes for all PutRecords operation. Shown as byte
aws.kinesis.put_records_failed_records (gauge)	The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, summed over the specified time period. Shown as record
aws.kinesis.put_records_failed_records.average (gauge)	The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, averaged over the specified time period. Shown as record
aws.kinesis.put_records_failed_records.maximum (gauge)	The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, maximum over the specified time period. Shown as record
aws.kinesis.put_records_failed_records.minimum (gauge)	The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, minimum over the specified time period. Shown as record
aws.kinesis.put_records_latency (gauge)	Average time taken per PutRecords operation. Shown as millisecond
aws.kinesis.put_records_latency.maximum (gauge)	Maximum time taken for all PutRecords operation. Shown as millisecond
aws.kinesis.put_records_records (gauge)	Average number of records per PutRecords operation Shown as record
aws.kinesis.put_records_records.maximum (gauge)	Maximum number of records for PutRecords operations Shown as record
aws.kinesis.put_records_records.minimum (gauge)	Minimum number of records for PutRecords operations Shown as record
aws.kinesis.put_records_records.sum (count)	Total number of records for PutRecords operations Shown as record
aws.kinesis.put_records_success (count)	Sum of successful PutRecords operations per Amazon Kinesis stream. Shown as event
aws.kinesis.put_records_success.average (count)	Average number of successful PutRecords operations per Amazon Kinesis stream. Shown as event
aws.kinesis.put_records_successful_records (gauge)	The number of successful records in a PutRecords operation per Kinesis data stream, summed over the specified time period. Shown as record
aws.kinesis.put_records_successful_records.average (gauge)	The number of successful records in a PutRecords operation per Kinesis data stream, averaged over the specified time period. Shown as record
aws.kinesis.put_records_successful_records.maximum (gauge)	The number of successful records in a PutRecords operation per Kinesis data stream, maximum over the specified time period. Shown as record
aws.kinesis.put_records_successful_records.minimum (gauge)	The number of successful records in a PutRecords operation per Kinesis data stream, minimum over the specified time period. Shown as record
aws.kinesis.put_records_throttled_records (gauge)	The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, summed over the specified time period. Shown as record
aws.kinesis.put_records_throttled_records.average (gauge)	The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, averaged over the specified time period. Shown as record
aws.kinesis.put_records_throttled_records.maximum (gauge)	The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, maximum over the specified time period. Shown as record
aws.kinesis.put_records_throttled_records.minimum (gauge)	The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, minimum over the specified time period. Shown as record
aws.kinesis.put_records_total_records (gauge)	The total number of records sent in a PutRecords operation per Kinesis data stream, summed over the specified time period. Shown as record
aws.kinesis.put_records_total_records.average (gauge)	The total number of records sent in a PutRecords operation per Kinesis data stream, averaged over the specified time period. Shown as record
aws.kinesis.put_records_total_records.maximum (gauge)	The total number of records sent in a PutRecords operation per Kinesis data stream, maximum over the specified time period. Shown as record
aws.kinesis.put_records_total_records.minimum (gauge)	The total number of records sent in a PutRecords operation per Kinesis data stream, minimum over the specified time period. Shown as record
aws.kinesis.read_provisioned_throughput_exceeded (count)	Number of GetRecords calls throttled for the stream Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.average (count)	Average of GetRecords calls throttled for the stream Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.maximum (count)	Maximum number of GetRecords calls throttled for the stream Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.minimum (count)	Minimum number of GetRecords calls throttled for the stream Shown as record
aws.kinesis.subscribe_to_shard_event_bytes (gauge)	The number of bytes received from the shard, averaged over the specified time period. Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.maximum (gauge)	The number of bytes received from the shard, maximum over the specified time period. Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.minimum (gauge)	The number of bytes received from the shard, minimum over the specified time period. Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.sum (gauge)	The number of bytes received from the shard, summed over the specified time period. Shown as byte
aws.kinesis.subscribe_to_shard_event_millis_behind_latest (gauge)	The difference between the current time and when the last record of the SubscribeToShard event was written to the stream. Shown as millisecond
aws.kinesis.subscribe_to_shard_event_millis_behind_latest.maximum (gauge)	The difference between the current time and when the last record of the SubscribeToShard event was written to the stream. Shown as millisecond
aws.kinesis.subscribe_to_shard_event_records (count)	The number of records received from the shard, averaged over the specified time period. Shown as record
aws.kinesis.subscribe_to_shard_event_records.maximum (count)	The number of records received from the shard, maximum over the specified time period. Shown as record
aws.kinesis.subscribe_to_shard_event_records.minimum (count)	The number of records received from the shard, minimum over the specified time period. Shown as record
aws.kinesis.subscribe_to_shard_event_records.sum (count)	The number of records received from the shard, summed over the specified time period. Shown as record
aws.kinesis.subscribe_to_shard_event_success (count)	This metric is emitted every time an event is published successfully. It is only emitted when there's an active subscription. Shown as event
aws.kinesis.subscribe_to_shard_event_success.average (count)	This metric is emitted every time an event is published successfully. It is only emitted when there's an active subscription. Shown as event
aws.kinesis.subscribe_to_shard_rate_exceeded (count)	This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation. Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.average (count)	This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation. Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.maximum (count)	This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation. Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.minimum (count)	This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation. Shown as record
aws.kinesis.subscribe_to_shard_success (gauge)	This metric records whether the SubscribeToShard subscription was successfully established. The subscription only lives for at most 5 minutes. Therefore, this metric gets emitted at least once every 5 minutes. Shown as event
aws.kinesis.subscribe_to_shard_success.average (gauge)	This metric records whether the SubscribeToShard subscription was successfully established. The subscription only lives for at most 5 minutes. Therefore, this metric gets emitted at least once every 5 minutes. Shown as event
aws.kinesis.write_provisioned_throughput_exceeded (count)	Number of records rejected due to throttling for the stream Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.average (count)	Average of records rejected due to throttling for the stream Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.maximum (count)	Maximum number of records rejected due to throttling for the stream Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.minimum (count)	Minimum number of records rejected due to throttling for the stream Shown as record

Each of the metrics retrieved from AWS are assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

Events

The Amazon Kinesis integration does not include any events.

Service Checks

The Amazon Kinesis integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.