Amazon Kinesis

개요

Amazon Kinesis는 완전 관리형 클라우드 기반 서비스로 대규모 분산 데이터 스트림을 실시간으로 프로세싱합니다.

이 통합을 활성화해 Datadog에서 모든 Kinesis 메트릭을 확인하고 커스텀 Kinesis 태그를 수집하세요.

설정

설치

이미 하지 않은 경우 먼저 Amazon Web Services를 설정합니다. 다른 설치 단계는 수행할 필요가 없습니다.

메트릭 수집

  1. AWS 통합 페이지에서 Kinesis가 Metric Collection 탭 아래 활성화되어 있는지 확인하세요.

  2. Amazon Kinesis 메트릭을 수집하려면 해당 권한을 Datadog IAM 정책에 추가하세요.

    • kinesis:ListStreams: 사용 가능한 스트림을 목록화합니다.
    • kinesis:DescribeStream: Kinesis 스트림을 위한 태그와 새로운 메트릭을 추가합니다.
    • kinesis:ListTagsForStream: 커스텀 태그를 추가합니다.

    자세한 정보는 AWS 웹사이트에서 Kinesis 정책을 참조하세요.

  3. Datadog - Amazon Kinesis 통합을 설치하세요.

로그 수집

로깅 활성화

Datadog는 Amazon Data Firehose 전송 스트림을 위한 기본 목적지 중 하나입니다. AWS는 Amazon Data Firehose를 완전히 관리하므로 로그 스트리밍을 위해 추가 인프라스트럭처나 포워딩 설정을 유지관리할 필요가 없습니다.

AWS Firehose 콘솔에서 Amazon Data Firehose 전송 스트림을 설정하거나 CloudFormation 템플릿을 사용해 자동으로 목적지를 설정할 수 있습니다.

하지만 S3 버킷에 로깅한 다음 AWS 람다 함수를 사용할 수 있습니다. amazon_kinesis가 _대상 접두어_로 설정되어 있는지 확인하세요.

  1. 이미 하지 않은 경우 Datadog 포워더(Forwarder) 람다 함수을 설정하세요.

  2. 람다 함수가 설치되면 AWS 콘솔에서 Amazon Kinesis 로그를 포함하는 S3 버킷 또는 클라우드와치(CloudWatch) 로그 그룹에 대한 트리거를 수동으로 추가할 수 있습니다.

수집한 데이터

메트릭

aws.kinesis.get_records_bytes
(gauge)		Average number of bytes per GetRecords operation
Shown as byte
aws.kinesis.get_records_bytes.maximum
(gauge)		Maximum number of bytes per GetRecords operation
Shown as byte
aws.kinesis.get_records_bytes.minimum
(gauge)		Minimum number of bytes per GetRecords operation
Shown as byte
aws.kinesis.get_records_bytes.sum
(count)		Total number of bytes returned over all GetRecords operations
Shown as byte
aws.kinesis.get_records_iterator_age
(gauge)		Average difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as second
aws.kinesis.get_records_iterator_age.maximum
(gauge)		Maximum difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as second
aws.kinesis.get_records_iterator_age_milliseconds
(gauge)		Difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as millisecond
aws.kinesis.get_records_iterator_age_milliseconds.maximum
(gauge)		Maximum difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as millisecond
aws.kinesis.get_records_latency
(gauge)		Average time taken per GetRecords operation.
Shown as millisecond
aws.kinesis.get_records_latency.maximum
(gauge)		Maximum time taken per GetRecords operation.
Shown as millisecond
aws.kinesis.get_records_records
(gauge)		Average number of records per GetRecords operation
Shown as record
aws.kinesis.get_records_records.maximum
(gauge)		Maximum number of records per GetRecords operation
Shown as record
aws.kinesis.get_records_records.minimum
(gauge)		Minimum number of records per GetRecords operation
Shown as record
aws.kinesis.get_records_records.sum
(count)		Total number of records returned over all GetRecords operations
Shown as record
aws.kinesis.get_records_success
(count)		Sum of successful GetRecords operations per stream.
Shown as event
aws.kinesis.get_records_success.average
(count)		Average number of successful GetRecords operations per stream.
Shown as event
aws.kinesis.incoming_bytes
(gauge)		Average number of bytes successfully put to the Amazon Kinesis stream per operation.
Shown as byte
aws.kinesis.incoming_bytes.sum
(count)		Total number of bytes successfully put to the Amazon Kinesis stream.
Shown as byte
aws.kinesis.incoming_records
(count)		Total number of records sucessfully put to the Amazon Kinesis stream.
Shown as record
aws.kinesis.incoming_records.average
(gauge)		Average number of records sucessfully put to the Amazon Kinesis stream per operation.
Shown as record
aws.kinesis.iterator_age_milliseconds
(gauge)		The age of the last record in all GetRecords calls made against a shard, measured over the specified time period.
Shown as millisecond
aws.kinesis.outgoing_bytes
(gauge)		Average number of bytes retrieved in a GetRecords operation
Shown as byte
aws.kinesis.outgoing_bytes.sum
(count)		Total number of bytes retrieved from the Amazon Kinesis stream
Shown as byte
aws.kinesis.outgoing_records
(count)		The number of records retrieved from the shard, measured over the specified time period.
Shown as record
aws.kinesis.outgoing_records.average
(gauge)		The average number of records retrieved from the shard, measured over the specified time period.
Shown as record
aws.kinesis.put_record_bytes
(gauge)		Average number of bytes per PutRecord operation
Shown as byte
aws.kinesis.put_record_bytes.maximum
(gauge)		Maximum number of bytes per PutRecord operation
Shown as byte
aws.kinesis.put_record_bytes.minimum
(gauge)		Minimum number of bytes per PutRecord operation
Shown as byte
aws.kinesis.put_record_bytes.sum
(count)		Total number of bytes for all PutRecord operation
Shown as byte
aws.kinesis.put_record_latency
(gauge)		Average time taken per PutRecord operation.
Shown as millisecond
aws.kinesis.put_record_latency.maximum
(gauge)		Maximum time taken per PutRecord operation.
Shown as millisecond
aws.kinesis.put_record_success
(count)		Sum of successful PutRecord operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_record_success.average
(count)		Average number of successful PutRecord operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_records_bytes
(gauge)		Average number of bytes per PutRecords operation.
Shown as byte
aws.kinesis.put_records_bytes.maximum
(gauge)		Maximum number of bytes per PutRecords operation.
Shown as byte
aws.kinesis.put_records_bytes.minimum
(gauge)		Minimum number of bytes per PutRecords operation.
Shown as byte
aws.kinesis.put_records_bytes.sum
(count)		Total number of bytes for all PutRecords operation.
Shown as byte
aws.kinesis.put_records_failed_records
(gauge)		The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_failed_records.average
(gauge)		The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_failed_records.maximum
(gauge)		The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_failed_records.minimum
(gauge)		The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.put_records_latency
(gauge)		Average time taken per PutRecords operation.
Shown as millisecond
aws.kinesis.put_records_latency.maximum
(gauge)		Maximum time taken for all PutRecords operation.
Shown as millisecond
aws.kinesis.put_records_records
(gauge)		Average number of records per PutRecords operation
Shown as record
aws.kinesis.put_records_records.maximum
(gauge)		Maximum number of records for PutRecords operations
Shown as record
aws.kinesis.put_records_records.minimum
(gauge)		Minimum number of records for PutRecords operations
Shown as record
aws.kinesis.put_records_records.sum
(count)		Total number of records for PutRecords operations
Shown as record
aws.kinesis.put_records_success
(count)		Sum of successful PutRecords operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_records_success.average
(count)		Average number of successful PutRecords operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_records_successful_records
(gauge)		The number of successful records in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_successful_records.average
(gauge)		The number of successful records in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_successful_records.maximum
(gauge)		The number of successful records in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_successful_records.minimum
(gauge)		The number of successful records in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records
(gauge)		The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records.average
(gauge)		The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records.maximum
(gauge)		The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records.minimum
(gauge)		The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.put_records_total_records
(gauge)		The total number of records sent in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_total_records.average
(gauge)		The total number of records sent in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_total_records.maximum
(gauge)		The total number of records sent in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_total_records.minimum
(gauge)		The total number of records sent in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded
(count)		Number of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.average
(count)		Average of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.maximum
(count)		Maximum number of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.minimum
(count)		Minimum number of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.subscribe_to_shard_event_bytes
(gauge)		The number of bytes received from the shard, averaged over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.maximum
(gauge)		The number of bytes received from the shard, maximum over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.minimum
(gauge)		The number of bytes received from the shard, minimum over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.sum
(gauge)		The number of bytes received from the shard, summed over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_millis_behind_latest
(gauge)		The difference between the current time and when the last record of the SubscribeToShard event was written to the stream.
Shown as millisecond
aws.kinesis.subscribe_to_shard_event_millis_behind_latest.maximum
(gauge)		The difference between the current time and when the last record of the SubscribeToShard event was written to the stream.
Shown as millisecond
aws.kinesis.subscribe_to_shard_event_records
(count)		The number of records received from the shard, averaged over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_records.maximum
(count)		The number of records received from the shard, maximum over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_records.minimum
(count)		The number of records received from the shard, minimum over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_records.sum
(count)		The number of records received from the shard, summed over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_success
(count)		This metric is emitted every time an event is published successfully. It is only emitted when there's an active subscription.
Shown as event
aws.kinesis.subscribe_to_shard_event_success.average
(count)		This metric is emitted every time an event is published successfully. It is only emitted when there's an active subscription.
Shown as event
aws.kinesis.subscribe_to_shard_rate_exceeded
(count)		This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.average
(count)		This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.maximum
(count)		This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.minimum
(count)		This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_success
(gauge)		This metric records whether the SubscribeToShard subscription was successfully established. The subscription only lives for at most 5 minutes. Therefore, this metric gets emitted at least once every 5 minutes.
Shown as event
aws.kinesis.subscribe_to_shard_success.average
(gauge)		This metric records whether the SubscribeToShard subscription was successfully established. The subscription only lives for at most 5 minutes. Therefore, this metric gets emitted at least once every 5 minutes.
Shown as event
aws.kinesis.write_provisioned_throughput_exceeded
(count)		Number of records rejected due to throttling for the stream
Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.average
(count)		Average of records rejected due to throttling for the stream
Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.maximum
(count)		Maximum number of records rejected due to throttling for the stream
Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.minimum
(count)		Minimum number of records rejected due to throttling for the stream
Shown as record

AWS에서 검색된 각 메트릭에는 호스트 이름, 보안 그룹 등을 포함하되 이에 국한되지 않고 AWS 콘솔에 표시되는 동일한 태그가 할당됩니다.

이벤트

Amazon Kinesis 통합에는 이벤트가 포함되어 있지 않습니다.

서비스 검사

Amazon Kinesis 통합에는 서비스 점검이 포함되어 있지 않습니다.

트러블슈팅

도움이 필요하신가요? Datadog 고객 지원팀에 문의해주세요.

PREVIEWING: may/unit-testing