Google Cloud Composer

Présentation

Google Cloud Composer est un service d’orchestration de workflows entièrement géré qui vous permet de créer, de planifier et de surveiller vos pipelines dans plusieurs clouds ou des centres de données sur site.

Utilisez l’intégration Datadog/Google Cloud Platform pour recueillir des métriques de Google Composer.

Implémentation

Installation

Si vous ne l’avez pas déjà fait, configurez d’abord l’intégration Google Cloud Platform. Aucune autre procédure d’installation n’est requise.

Collecte de logs

Les logs Google Cloud Composer sont recueillis avec Google Cloud Logging et envoyés à un Cloud Pub/Sub via un forwarder Push HTTP. Si vous ne l’avez pas déjà fait, configurez un Cloud Pub/Sub à l’aide d’un forwarder Push HTTP.

Une fois cette opération effectuée, exportez vos logs Google Cloud Composer depuis Google Cloud Logging vers le Pub/Sub :

  1. Accédez à la page Google Cloud Logging et filtrez les logs Google Cloud Composer.
  2. Cliquez sur Create Export et nommez le récepteur.
  3. Choisissez Cloud Pub/Sub comme destination et sélectionnez le Pub/Sub créé à cette fin. Remarque : le Pub/Sub peut se situer dans un autre projet.
  4. Cliquez sur Create et attendez que le message de confirmation s’affiche.

Données collectées

Métriques

gcp.composer.environment.api.request_count
(count)
Number of Composer API requests seen so far.
Shown as request
gcp.composer.environment.api.request_latencies.avg
(gauge)
Distribution of Composer API call latencies.
Shown as millisecond
gcp.composer.environment.api.request_latencies.samplecount
(count)
Sample count for API request latencies
Shown as millisecond
gcp.composer.environment.api.request_latencies.sumsqdev
(gauge)
Sum of squared deviation for API request latencies
Shown as second
gcp.composer.environment.dag_processing.parse_error_count
(count)
Number of errors raised during parsing DAG files
Shown as error
gcp.composer.environment.dag_processing.processes
(gauge)
Number of currently running DAG parsing processes
Shown as process
gcp.composer.environment.dag_processing.processesor_timeout_count
(count)
Number of file processors terminated due to processing timeout.
Shown as timeout
gcp.composer.environment.dag_processing.total_parse_time
(gauge)
Number of seconds taken to scan and import all DAG files once
Shown as second
gcp.composer.environment.dagbag_size
(gauge)
The current DAG bag size
gcp.composer.environment.database.cpu.reserved_cores
(gauge)
Number of cores reserved for the database instance
Shown as core
gcp.composer.environment.database.cpu.usage_time
(count)
CPU usage time of the database instance, in seconds
Shown as second
gcp.composer.environment.database.cpu.utilization
(gauge)
CPU utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.disk.bytes_used
(gauge)
Used disk space on the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.quota
(gauge)
Maximum data disk size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.disk.utilization
(gauge)
Disk quota usage ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.memory.bytes_used
(gauge)
Memory usage of the database instance in bytes
Shown as byte
gcp.composer.environment.database.memory.quota
(gauge)
Maximum RAM size of the database instance, in bytes
Shown as byte
gcp.composer.environment.database.memory.utilization
(gauge)
Memory utilization ratio (from 0.0 to 1.0) of the database instance
gcp.composer.environment.database.network.received_bytes_count
(count)
Number of bytes received by the database instance
Shown as byte
gcp.composer.environment.database.network.sent_bytes_count
(count)
Number of bytes sent by the database instance
Shown as byte
gcp.composer.environment.database_health
(gauge)
Healthiness of Composer Airflow database
gcp.composer.environment.executor.open_slots
(gauge)
Number of open slots on executor
gcp.composer.environment.executor.queued_tasks
(gauge)
Number of queued tasks on executor
Shown as task
gcp.composer.environment.executor.running_tasks
(gauge)
Number of running tasks on executor
Shown as task
gcp.composer.environment.finished_task_instance_count
(count)
Overall number of finished task instances
Shown as instance
gcp.composer.environment.healthy
(gauge)
Healthiness of Composer environment.
gcp.composer.environment.num_celery_workers
(gauge)
Number of Celery workers.
Shown as worker
gcp.composer.environment.num_workflows
(gauge)
Number of workflows.
gcp.composer.environment.scheduler_heartbeat_count
(count)
Scheduler heartbeats
gcp.composer.environment.task_queue_length
(gauge)
Number of tasks in queue.
Shown as task
gcp.composer.environment.unfinished_task_instances
(gauge)
Overall task instances in not finished state
Shown as instance
gcp.composer.environment.web_server.cpu.reserved_cores
(gauge)
Number of cores reserved for the web server instance
Shown as core
gcp.composer.environment.web_server.cpu.usage_time
(count)
CPU usage time of the web server instance, in seconds
Shown as second
gcp.composer.environment.web_server.health
(gauge)
Healthiness of Airflow web server
gcp.composer.environment.web_server.memory.bytes_used
(gauge)
Memory usage of the web server instance in bytes
Shown as byte
gcp.composer.environment.web_server.memory.quota
(gauge)
Maximum RAM size of the web server instance, in bytes
Shown as byte
gcp.composer.environment.worker.max_workers
(gauge)
Maximum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.min_workers
(gauge)
Minimum number of Airflow workers
Shown as worker
gcp.composer.environment.worker.pod_eviction_count
(count)
Number of Airflow worker pods evictions
Shown as eviction
gcp.composer.environment.worker.scale_factor_target
(gauge)
Scale factor for Airflow workers count
gcp.composer.environment.zombie_task_killed_count
(count)
Number of zombie tasks killed
Shown as task
gcp.composer.workflow.run_count
(count)
Number of workflow runs completed so far.
gcp.composer.workflow.run_duration
(gauge)
Duration of workflow run completion.
Shown as second
gcp.composer.workflow.task.run_count
(count)
Number of workflow tasks completed so far.
Shown as task
gcp.composer.workflow.task.run_duration
(gauge)
Duration of task completion.
Shown as second

Événements

L’intégration Google Cloud Composer n’inclut aucun événement.

Checks de service

L’intégration Google Cloud Composer n’inclut aucun check de service.

Dépannage

Besoin d’aide ? Contactez l’assistance Datadog.

PREVIEWING: may/unit-testing