Amazon Kinesis

Información general

Amazon Kinesis es un servicio totalmente gestionado, basado en la nube utilizado para procesar grandes flujos de datos distribuidos en tiempo real.

Habilita esta integración para ver en Datadog todas tus métricas Kinesis y recopilar etiquetas (tags) personalizadas de Kinesis.

Configuración

Instalación

Si aún no lo has hecho, configura la integración Amazon Web Services. No es necesario realizar ningún otro paso de instalación.

Recopilación de métricas

  1. En la página de la integración AWS, asegúrate de que Kinesis está habilitado en la pestaña Metric Collection.

  2. Añade estos permisos a tu política IAM de Datadog para recopilar métricas Amazon Kinesis:

    • kinesis:ListStreams: Enumera los flujos (flows) disponibles.
    • kinesis:DescribeStream: Añade etiquetas y nuevas métricas para flujos Kinesis.
    • kinesis:ListTagsForStream: Añade etiquetas personalizadas.

    Para obtener más información, consulta las políticas de Kinesis en el sitio web AWS.

  3. Instala la integración Amazon Kinesis en Datadog.

APM

Activar logging

Datadog es uno de los destinos predeterminados para los flujos de entrega de Amazon Data Firehose. AWS gestiona enteramente Amazon Data Firehose, por lo que no es necesario mantener ninguna configuración adicional de infraestructura o de reenvío para la transmisión de logs.

Puedes configurar un flujo de entrega de Amazon Data Firehose en la consola de AWS Firehose o puedes configurar automáticamente el destino utilizando una plantilla de CloudFormation:

Sin embargo, si generas logs en un bucket de S3, utiliza la función AWS Lambda y asegúrate de que amazon_kinesis esté configurado como Prefijo de destino.

  1. Si aún no lo has hecho, configura la [función Lambda del Datadog Forwarder8.

  2. Una vez instalada la función de Lambda, añade manualmente un activador en el bucket de S3 o en el grupo de logs de CloudWatch que contiene tus logs de Amazon Kinesis en la consola de AWS.

Datos recopilados

Métricas

aws.kinesis.get_records_bytes
(gauge)
Average number of bytes per GetRecords operation
Shown as byte
aws.kinesis.get_records_bytes.maximum
(gauge)
Maximum number of bytes per GetRecords operation
Shown as byte
aws.kinesis.get_records_bytes.minimum
(gauge)
Minimum number of bytes per GetRecords operation
Shown as byte
aws.kinesis.get_records_bytes.sum
(count)
Total number of bytes returned over all GetRecords operations
Shown as byte
aws.kinesis.get_records_iterator_age
(gauge)
Average difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as second
aws.kinesis.get_records_iterator_age.maximum
(gauge)
Maximum difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as second
aws.kinesis.get_records_iterator_age_milliseconds
(gauge)
Difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as millisecond
aws.kinesis.get_records_iterator_age_milliseconds.maximum
(gauge)
Maximum difference between the current time and when the last record of a GetRecords call was written to the stream.
Shown as millisecond
aws.kinesis.get_records_latency
(gauge)
Average time taken per GetRecords operation.
Shown as millisecond
aws.kinesis.get_records_latency.maximum
(gauge)
Maximum time taken per GetRecords operation.
Shown as millisecond
aws.kinesis.get_records_records
(gauge)
Average number of records per GetRecords operation
Shown as record
aws.kinesis.get_records_records.maximum
(gauge)
Maximum number of records per GetRecords operation
Shown as record
aws.kinesis.get_records_records.minimum
(gauge)
Minimum number of records per GetRecords operation
Shown as record
aws.kinesis.get_records_records.sum
(count)
Total number of records returned over all GetRecords operations
Shown as record
aws.kinesis.get_records_success
(count)
Sum of successful GetRecords operations per stream.
Shown as event
aws.kinesis.get_records_success.average
(count)
Average number of successful GetRecords operations per stream.
Shown as event
aws.kinesis.incoming_bytes
(gauge)
Average number of bytes successfully put to the Amazon Kinesis stream per operation.
Shown as byte
aws.kinesis.incoming_bytes.sum
(count)
Total number of bytes successfully put to the Amazon Kinesis stream.
Shown as byte
aws.kinesis.incoming_records
(count)
Total number of records sucessfully put to the Amazon Kinesis stream.
Shown as record
aws.kinesis.incoming_records.average
(gauge)
Average number of records sucessfully put to the Amazon Kinesis stream per operation.
Shown as record
aws.kinesis.iterator_age_milliseconds
(gauge)
The age of the last record in all GetRecords calls made against a shard, measured over the specified time period.
Shown as millisecond
aws.kinesis.outgoing_bytes
(gauge)
Average number of bytes retrieved in a GetRecords operation
Shown as byte
aws.kinesis.outgoing_bytes.sum
(count)
Total number of bytes retrieved from the Amazon Kinesis stream
Shown as byte
aws.kinesis.outgoing_records
(count)
The number of records retrieved from the shard, measured over the specified time period.
Shown as record
aws.kinesis.outgoing_records.average
(gauge)
The average number of records retrieved from the shard, measured over the specified time period.
Shown as record
aws.kinesis.put_record_bytes
(gauge)
Average number of bytes per PutRecord operation
Shown as byte
aws.kinesis.put_record_bytes.maximum
(gauge)
Maximum number of bytes per PutRecord operation
Shown as byte
aws.kinesis.put_record_bytes.minimum
(gauge)
Minimum number of bytes per PutRecord operation
Shown as byte
aws.kinesis.put_record_bytes.sum
(count)
Total number of bytes for all PutRecord operation
Shown as byte
aws.kinesis.put_record_latency
(gauge)
Average time taken per PutRecord operation.
Shown as millisecond
aws.kinesis.put_record_latency.maximum
(gauge)
Maximum time taken per PutRecord operation.
Shown as millisecond
aws.kinesis.put_record_success
(count)
Sum of successful PutRecord operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_record_success.average
(count)
Average number of successful PutRecord operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_records_bytes
(gauge)
Average number of bytes per PutRecords operation.
Shown as byte
aws.kinesis.put_records_bytes.maximum
(gauge)
Maximum number of bytes per PutRecords operation.
Shown as byte
aws.kinesis.put_records_bytes.minimum
(gauge)
Minimum number of bytes per PutRecords operation.
Shown as byte
aws.kinesis.put_records_bytes.sum
(count)
Total number of bytes for all PutRecords operation.
Shown as byte
aws.kinesis.put_records_failed_records
(gauge)
The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_failed_records.average
(gauge)
The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_failed_records.maximum
(gauge)
The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_failed_records.minimum
(gauge)
The number of records rejected due to internal failures in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.put_records_latency
(gauge)
Average time taken per PutRecords operation.
Shown as millisecond
aws.kinesis.put_records_latency.maximum
(gauge)
Maximum time taken for all PutRecords operation.
Shown as millisecond
aws.kinesis.put_records_records
(gauge)
Average number of records per PutRecords operation
Shown as record
aws.kinesis.put_records_records.maximum
(gauge)
Maximum number of records for PutRecords operations
Shown as record
aws.kinesis.put_records_records.minimum
(gauge)
Minimum number of records for PutRecords operations
Shown as record
aws.kinesis.put_records_records.sum
(count)
Total number of records for PutRecords operations
Shown as record
aws.kinesis.put_records_success
(count)
Sum of successful PutRecords operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_records_success.average
(count)
Average number of successful PutRecords operations per Amazon Kinesis stream.
Shown as event
aws.kinesis.put_records_successful_records
(gauge)
The number of successful records in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_successful_records.average
(gauge)
The number of successful records in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_successful_records.maximum
(gauge)
The number of successful records in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_successful_records.minimum
(gauge)
The number of successful records in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records
(gauge)
The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records.average
(gauge)
The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records.maximum
(gauge)
The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_throttled_records.minimum
(gauge)
The number of records rejected due to throttling in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.put_records_total_records
(gauge)
The total number of records sent in a PutRecords operation per Kinesis data stream, summed over the specified time period.
Shown as record
aws.kinesis.put_records_total_records.average
(gauge)
The total number of records sent in a PutRecords operation per Kinesis data stream, averaged over the specified time period.
Shown as record
aws.kinesis.put_records_total_records.maximum
(gauge)
The total number of records sent in a PutRecords operation per Kinesis data stream, maximum over the specified time period.
Shown as record
aws.kinesis.put_records_total_records.minimum
(gauge)
The total number of records sent in a PutRecords operation per Kinesis data stream, minimum over the specified time period.
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded
(count)
Number of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.average
(count)
Average of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.maximum
(count)
Maximum number of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.read_provisioned_throughput_exceeded.minimum
(count)
Minimum number of GetRecords calls throttled for the stream
Shown as record
aws.kinesis.subscribe_to_shard_event_bytes
(gauge)
The number of bytes received from the shard, averaged over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.maximum
(gauge)
The number of bytes received from the shard, maximum over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.minimum
(gauge)
The number of bytes received from the shard, minimum over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_bytes.sum
(gauge)
The number of bytes received from the shard, summed over the specified time period.
Shown as byte
aws.kinesis.subscribe_to_shard_event_millis_behind_latest
(gauge)
The difference between the current time and when the last record of the SubscribeToShard event was written to the stream.
Shown as millisecond
aws.kinesis.subscribe_to_shard_event_millis_behind_latest.maximum
(gauge)
The difference between the current time and when the last record of the SubscribeToShard event was written to the stream.
Shown as millisecond
aws.kinesis.subscribe_to_shard_event_records
(count)
The number of records received from the shard, averaged over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_records.maximum
(count)
The number of records received from the shard, maximum over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_records.minimum
(count)
The number of records received from the shard, minimum over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_records.sum
(count)
The number of records received from the shard, summed over the specified time period.
Shown as record
aws.kinesis.subscribe_to_shard_event_success
(count)
This metric is emitted every time an event is published successfully. It is only emitted when there's an active subscription.
Shown as event
aws.kinesis.subscribe_to_shard_event_success.average
(count)
This metric is emitted every time an event is published successfully. It is only emitted when there's an active subscription.
Shown as event
aws.kinesis.subscribe_to_shard_rate_exceeded
(count)
This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.average
(count)
This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.maximum
(count)
This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_rate_exceeded.minimum
(count)
This metric is emitted when a new subscription attempt fails because there already is an active subscription by the same consumer or if you exceed the number of calls per second allowed for this operation.
Shown as record
aws.kinesis.subscribe_to_shard_success
(gauge)
This metric records whether the SubscribeToShard subscription was successfully established. The subscription only lives for at most 5 minutes. Therefore, this metric gets emitted at least once every 5 minutes.
Shown as event
aws.kinesis.subscribe_to_shard_success.average
(gauge)
This metric records whether the SubscribeToShard subscription was successfully established. The subscription only lives for at most 5 minutes. Therefore, this metric gets emitted at least once every 5 minutes.
Shown as event
aws.kinesis.write_provisioned_throughput_exceeded
(count)
Number of records rejected due to throttling for the stream
Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.average
(count)
Average of records rejected due to throttling for the stream
Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.maximum
(count)
Maximum number of records rejected due to throttling for the stream
Shown as record
aws.kinesis.write_provisioned_throughput_exceeded.minimum
(count)
Minimum number of records rejected due to throttling for the stream
Shown as record

A cada una de las métricas recuperadas de AWS se le asignan las mismas etiquetas que aparecen en la consola de AWS, donde se incluyen el nombre del host y los grupos de seguridad, entre otras cosas.

Eventos

La integración Amazon Kinesis no incluye eventos.

Checks de servicios

La integración Amazon Kinesis no incluye checks de servicios.

Resolución de problemas

¿Necesitas ayuda? Ponte en contacto con Soporte técnico de Datadog.

PREVIEWING: mervebolat/span-id-preprocessing