Amazon Data Firehose

Overview

Amazon Data Firehose is the easiest way to load streaming data into AWS.

Enable this integration to see in Datadog all your Firehose metrics.

Note: Amazon Data Firehose was formerly known as Amazon Kinesis Data Firehose. Read the AWS What’s New post to learn more.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Metric collection

  1. In the AWS integration page, ensure that Firehose is enabled under the Metric Collection tab.
  2. Install the Datadog - Amazon Data Firehose integration.

Log collection

Enable logging

Configure Amazon Data Firehose to send logs either to a S3 bucket or to CloudWatch.

Note: If you log to a S3 bucket, make sure that amazon_firehose is set as Target prefix.

Send logs to Datadog

  1. If you haven’t already, set up the Datadog Forwarder Lambda function.

  2. Once the Lambda function is installed, manually add a trigger on the S3 bucket or CloudWatch log group that contains your Amazon Data Firehose logs in the AWS console:

Data Collected

Metrics

aws.firehose.backup_to_s_3bytes
(count)
The number of bytes delivered to Amazon S3 for backup over the specified time period.
Shown as byte
aws.firehose.backup_to_s_3data_freshness
(gauge)
Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup.
Shown as second
aws.firehose.backup_to_s_3records
(count)
The number of records delivered to Amazon S3 for backup over the specified time period.
aws.firehose.backup_to_s_3success
(gauge)
Fraction of successful Amazon S3 put commands for backup.
Shown as fraction
aws.firehose.backup_to_s_3success.sum
(count)
The total number of successful Amazon S3 put commands for backup.
aws.firehose.bytes_per_second_limit
(count)
The current maximum number of bytes per second that a delivery stream can ingest before throttling.
aws.firehose.data_read_from_kinesis_stream_bytes
(count)
When the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream. This number includes rereads due to failovers.
Shown as byte
aws.firehose.data_read_from_kinesis_stream_records
(count)
When the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream.
aws.firehose.data_read_from_kinesis_stream_records.average
(gauge)
When the data source is a Kinesis data stream, this metric indicates the average number of records read from that data stream.
aws.firehose.delivery_to_elasticsearch_bytes
(count)
The total number of bytes indexed to Amazon ElasticSearch.
Shown as byte
aws.firehose.delivery_to_elasticsearch_records
(count)
The total number of records indexed to Amazon ElasticSearch.
aws.firehose.delivery_to_elasticsearch_success
(gauge)
Fraction of records successfully indexed to Amazon ElasticSearch.
Shown as fraction
aws.firehose.delivery_to_elasticsearch_success.sum
(count)
Total number of records successfully indexed to Amazon ElasticSearch.
aws.firehose.delivery_to_http_endpoint_bytes
(count)
The number of bytes delivered successfully to the HTTP endpoint.
Shown as byte
aws.firehose.delivery_to_http_endpoint_data_freshness
(gauge)
Age of the oldest record in Kinesis Data Firehose.
Shown as second
aws.firehose.delivery_to_http_endpoint_processed_bytes
(count)
The number of attempted processed bytes including retries.
Shown as byte
aws.firehose.delivery_to_http_endpoint_processed_records
(count)
The number of attempted records including retries.
aws.firehose.delivery_to_http_endpoint_processed_records.average
(gauge)
The average number of attempted records including retries.
aws.firehose.delivery_to_http_endpoint_records
(count)
The number of records delivered successfully to the HTTP endpoint.
aws.firehose.delivery_to_http_endpoint_records.average
(gauge)
The average number of records delivered successfully to the HTTP endpoint.
aws.firehose.delivery_to_http_endpoint_success
(gauge)
The fraction successful data delivery requests to the HTTP endpoint.
Shown as fraction
aws.firehose.delivery_to_http_endpoint_success.sum
(count)
The total number of all successful data delivery requests to the HTTP endpoint.
Shown as fraction
aws.firehose.delivery_to_redshift_bytes
(gauge)
The average number of bytes copied to Amazon Redshift per delivery.
Shown as byte
aws.firehose.delivery_to_redshift_bytes.sum
(count)
The total number of bytes copied to Amazon Redshift.
Shown as byte
aws.firehose.delivery_to_redshift_records
(count)
The total number of records copied to Amazon Redshift.
aws.firehose.delivery_to_redshift_records.average
(gauge)
The average number of records copied to Amazon Redshift per delivery.
aws.firehose.delivery_to_redshift_success
(gauge)
Fraction of successful Redshift COPY commands
Shown as fraction
aws.firehose.delivery_to_redshift_success.sum
(count)
Total number of successful Redshift COPY commands
aws.firehose.delivery_to_s_3bytes
(gauge)
The average number of bytes delivered to Amazon S3 per delivery.
Shown as byte
aws.firehose.delivery_to_s_3bytes.sum
(count)
The total number of bytes delivered to Amazon S3.
Shown as byte
aws.firehose.delivery_to_s_3data_freshness
(gauge)
Age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to the S3 bucket.
Shown as second
aws.firehose.delivery_to_s_3records
(count)
The total number of records delivered to Amazon S3.
aws.firehose.delivery_to_s_3records.average
(gauge)
The average number of records delivered to Amazon S3 per delivery.
aws.firehose.delivery_to_s_3success
(gauge)
Fraction of successful deliveries to S3
Shown as fraction
aws.firehose.delivery_to_s_3success.sum
(count)
Sum of successful deliveries to S3
aws.firehose.delivery_to_splunk_bytes
(gauge)
The average number of bytes delivered to Splunk per delivery.
Shown as byte
aws.firehose.delivery_to_splunk_bytes.sum
(count)
The total number of bytes delivered to Splunk.
Shown as byte
aws.firehose.delivery_to_splunk_data_ack_latency
(gauge)
The average approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data.
Shown as second
aws.firehose.delivery_to_splunk_data_ack_latency.maximum
(gauge)
The maximum approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data.
Shown as second
aws.firehose.delivery_to_splunk_data_freshness
(gauge)
Average age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to Splunk.
Shown as second
aws.firehose.delivery_to_splunk_data_freshness.maximum
(gauge)
Maximum age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to Splunk.
Shown as second
aws.firehose.delivery_to_splunk_records
(count)
The total number of records delivered to Splunk.
aws.firehose.delivery_to_splunk_records.average
(gauge)
The average number of records delivered to Splunk per delivery.
aws.firehose.delivery_to_splunk_success
(gauge)
Fraction of successfully indexed records
Shown as fraction
aws.firehose.delivery_to_splunk_success.maximum
(gauge)
Maximum value for fraction of successfully indexed records
Shown as fraction
aws.firehose.delivery_to_splunk_success.minimum
(gauge)
Minimum value for fraction of successfully indexed records
Shown as fraction
aws.firehose.describe_delivery_stream_latency
(gauge)
The average time taken per DescribeDeliveryStream operation.
Shown as millisecond
aws.firehose.describe_delivery_stream_latency.maximum
(gauge)
The maximum time taken per DescribeDeliveryStream operation.
Shown as millisecond
aws.firehose.describe_delivery_stream_requests
(count)
Total number of DescribeDeliveryStream requests.
aws.firehose.incoming_bytes
(gauge)
The number of bytes ingested into the Firehose delivery stream.
Shown as byte
aws.firehose.incoming_put_requests
(count)
The number of successful PutRecord and PutRecordBatch requests over the specified period of time after throttling.
aws.firehose.incoming_records
(count)
The number of records ingested into the Firehose delivery stream.
aws.firehose.kinesis_millis_behind_latest
(gauge)
When the data source is a Kinesis data stream, this metric indicates the average time that the last read record is behind the newest record in the Kinesis data stream.
Shown as millisecond
aws.firehose.kinesis_millis_behind_latest.maximum
(gauge)
When the data source is a Kinesis data stream, this metric indicates the maximum time that the last read record is behind the newest record in the Kinesis data stream.
Shown as millisecond
aws.firehose.kmskey_access_denied
(count)
The number of times the service encounters a KMSAccessDeniedException for the delivery stream.
Shown as millisecond
aws.firehose.kmskey_disabled
(count)
The number of times the service encounters a KMSDisabledException for the delivery stream.
Shown as millisecond
aws.firehose.kmskey_invalid_state
(count)
The number of times the service encounters a KMSInvalidStateException for the delivery stream.
Shown as millisecond
aws.firehose.kmskey_not_found
(count)
The number of times the service encounters a KMSNotFoundException for the delivery stream.
Shown as millisecond
aws.firehose.list_delivery_stream_latency
(gauge)
The average time taken per ListDeliveryStream operation.
Shown as millisecond
aws.firehose.list_delivery_stream_latency.maximum
(gauge)
The maximum time taken per ListDeliveryStream operation.
Shown as millisecond
aws.firehose.list_delivery_stream_requests
(count)
Total number of ListDeliveryStream requests.
aws.firehose.list_delivery_streams_latency
(gauge)
The average time taken per ListDeliveryStreams operation.
Shown as millisecond
aws.firehose.list_delivery_streams_latency.maximum
(gauge)
The maximum time taken per ListDeliveryStreams operation.
Shown as millisecond
aws.firehose.list_delivery_streams_requests
(count)
Total number of ListDeliveryStreams requests.
aws.firehose.put_record_batch_bytes
(count)
The total number of bytes put to the Firehose delivery stream using PutRecordBatch.
Shown as byte
aws.firehose.put_record_batch_bytes.average
(gauge)
The average number of bytes per PutRecordBatch request.
Shown as byte
aws.firehose.put_record_batch_latency
(gauge)
The average time taken per PutRecordBatch operation.
Shown as millisecond
aws.firehose.put_record_batch_latency.maximum
(gauge)
The maximum time taken per PutRecordBatch operation.
Shown as millisecond
aws.firehose.put_record_batch_records
(count)
Total number of records from PutRecordBatch operations.
aws.firehose.put_record_batch_records.average
(gauge)
Average number of records per PutRecordBatch operation.
aws.firehose.put_record_batch_requests
(count)
Total number of PutRecordBatch requests.
aws.firehose.put_record_bytes
(count)
The total number of bytes put to the Firehose delivery stream using PutRecord.
Shown as byte
aws.firehose.put_record_bytes.average
(gauge)
The average number of bytes per PutRecord request.
Shown as byte
aws.firehose.put_record_latency
(gauge)
The average time taken per PutRecord operation.
Shown as millisecond
aws.firehose.put_record_latency.maximum
(gauge)
The maximum time taken per PutRecord operation.
Shown as millisecond
aws.firehose.put_record_requests
(count)
Total number of PutRecord requests.
aws.firehose.put_requests_per_second_limit
(count)
The maximum number of put requests per second that a delivery stream can handle before throttling.
aws.firehose.records_per_second_limit
(count)
The current maximum number of records per second that a delivery stream can ingest before throttling.
aws.firehose.throttled_describe_stream
(count)
Total number of times the DescribeStream operation is throttled when the data source is a Kinesis data stream.
Shown as millisecond
aws.firehose.throttled_get_records
(count)
The total number of times the GetRecords operation is throttled when the data source is a Kinesis data stream.
Shown as millisecond
aws.firehose.throttled_get_shard_iterator
(count)
The total number of times the GetShardIterator operation is throttled when the data source is a Kinesis data stream.
Shown as millisecond
aws.firehose.throttled_records
(count)
The number of records that were throttled because data ingestion exceeded one of the delivery stream limits.
aws.firehose.update_delivery_stream_latency
(gauge)
The average time taken per UpdateDeliveryStream operation.
Shown as millisecond
aws.firehose.update_delivery_stream_latency.maximum
(gauge)
The maximum time taken per UpdateDeliveryStream operation.
Shown as millisecond
aws.firehose.update_delivery_stream_requests
(count)
Total number of UpdateDeliveryStream requests.
aws.firehose.update_firehose_latency
(gauge)
The average time taken per UpdateFirehose operation.
Shown as millisecond
aws.firehose.update_firehose_latency.maximum
(gauge)
The maximum time taken per UpdateFirehose operation.
Shown as millisecond
aws.firehose.update_firehose_requests
(count)
Total number of UpdateFirehose requests.

Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

Events

The Amazon Data Firehose integration does not include any events.

Service Checks

The Amazon Data Firehose integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.

PREVIEWING: may/unit-testing