Hive

Supported OS Linux Mac OS

통합 버전2.1.0

개요

이 점검은 Hive의 두 부분인 Hive Metastore와 HiveServer2를 모니터링합니다.

설정

설치

Hive 점검은 Datadog 에이전트 패키지에 포함되어 있습니다. 서버에 추가 설치가 필요하지 않습니다.

설정

Hive 설정

  1. HIVE_HOME/conf/hive-site.xml]3에서 Hive 설정 파일을 편집하여 다음 속성을 추가하여 Hive 메타스토어 및 HiveServer2 메트릭 를 활성화합니다:

    <property>
  <name>hive.metastore.metrics.enabled</name>
  <value>true</value>
</property>
<property>
  <name>hive.server2.metrics.enabled</name>
  <value>true</value>
</property>

  2. HiveServer2 및/또는 Hive Metastore에 대해 JMX 원격 연결을 활성화합니다. 예를 들어 HADOOP_CLIENT_OPTS 환경 변수를 설정합니다.

    export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.port=8808"

    그런 다음 HiveServer2 또는 Hive Metastore를 다시 시작합니다. Hive Metastore와 HiveServer2는 동일한 JMX 연결을 공유할 수 없습니다.

    호스트

    호스트에서 실행 중인 에이전트에 이 점검을 구성하는 방법:

    호스트에서 실행 중인 Agent에 대해 이 검사를 구성하려면 아래 지침을 따르세요. 컨테이너화된 환경의 경우 Containerized 섹션을 참조하세요.

    메트릭 수집

    1. 에이전트의 설정 디렉토리 루트의 conf.d/ 폴더에 있는 hive.d/conf.yaml 파일을 편집하여 Hive 성능 데이터 수집을 시작합니다. 사용 가능한 모든 설정 옵션은 hive.d/conf.yaml 샘플을 참조하세요.

      이 점검는 인스턴스당 350개 메트릭으로 제한됩니다. 반환된 메트릭 수는 상태 페이지에 표시됩니다. 아래 설정을 편집하여 관심 있는 메트릭을 지정할 수 있습니다. 수집할 메트릭을 커스터마이즈하는 방법을 알아보려면 JMX 점검 설명서에서 자세한 지침을 참조하세요. 더 많은 메트릭을 모니터링하려면 Datadog 지원팀에 문의하세요.

    2. 에이전트를 재시작하세요.

    로그 수집

    에이전트 버전 > 6.0에서 사용 가능

    1. Datadog 에이전트에서는 로그 수집이 기본적으로 비활성화되어 있습니다. datadog.yaml 파일에서 활성화해야 합니다.

      logs_enabled: true

    2. 이 설정 블록을 hive.d/conf.yaml 파일에 추가하여 Hive 로그 수집을 시작하세요.

        logs:
    - type: file
      path: /tmp/<USER>/hive.log
      source: hive
      service: '<SERVICE_NAME>'
      log_processing_rules:
        - type: multi_line
          name: new_log_start_with_date
          pattern: \d{4}\-\d{2}\-\d{2}

      path 및 service 파라미터 값을 변경하고 환경에 맞춰 설정합니다. 사용 가능한 모든 설정 옵션은 hive.d/conf.yaml 샘플을 참조하세요.

    3. 에이전트를 재시작하세요.

    컨테이너화된 환경

    컨테이너화된 환경의 경우 자동탐지 통합 템플릿에 아래 파라미터를 적용하는 방법이 안내되어 있습니다.

    메트릭 수집

    Datadog-Hive 통합을 사용하여 메트릭 수집하려면 JMX를 사용한 자동탐지 가이드를 참조하세요.

    로그 수집

    에이전트 버전 > 6.0에서 사용 가능

    Datadog 에이전트에서 기본적으로 로그 수집이 비활성화되어 있습니다. 활성화하려면 [쿠버네티스 로그 수집]을 확인하세요.

    파라미터값
    <LOG_CONFIG>{"source": "hive", "service": "<SERVICE_NAME>", "log_processing_rules":{"type":"multi_line","name":"new_log_start_with_date", "pattern":"\d{4}\-\d{2}\-\d{2}"}}

    검증

    에이전트의 상태 하위 명령을 실행하고 점검 섹션에서 Hive를 찾습니다.

    수집한 데이터

    메트릭

    hive.metastore.api.alter_table
    (rate)    		API call to alter table.
    Shown as task
    hive.metastore.api.alter_table.active_call
    (gauge)    		Active API call to alter table.
    Shown as unit
    hive.metastore.api.create_table
    (rate)    		API call to create a table.
    Shown as task
    hive.metastore.api.create_table.active_call
    (gauge)    		Active API call to create a table.
    Shown as unit
    hive.metastore.api.drop_table
    (rate)    		API call to drop a table.
    Shown as task
    hive.metastore.api.drop_table.active_call
    (gauge)    		Active API call to drop a table.
    Shown as unit
    hive.metastore.api.flushcache
    (rate)    		API flushcache.
    Shown as task
    hive.metastore.api.flushcache.active_call
    (gauge)    		Active API flushcache.
    Shown as unit
    hive.metastore.api.get_all_databases
    (rate)    		API call to get all databases.
    Shown as task
    hive.metastore.api.get_all_databases.active_call
    (gauge)    		Active API call to get all databases.
    Shown as unit
    hive.metastore.api.get_all_functions
    (rate)    		API call to get all functions.
    Shown as task
    hive.metastore.api.get_all_functions.active_call
    (gauge)    		Active API call to get all functions.
    Shown as unit
    hive.metastore.api.get_all_tables
    (rate)    		API call to get all tables.
    Shown as task
    hive.metastore.api.get_all_tables.active_call
    (gauge)    		Active API call to get all tables.
    Shown as unit
    hive.metastore.api.get_database
    (rate)    		API call to get a database.
    Shown as task
    hive.metastore.api.get_database.active_call
    (gauge)    		Active API call to get a database.
    Shown as unit
    hive.metastore.api.get_table
    (rate)    		API call to get a table.
    Shown as task
    hive.metastore.api.get_table.active_call
    (gauge)    		Active API call to get a table.
    Shown as unit
    hive.metastore.api.get_table_req
    (rate)
    Shown as task
    hive.metastore.api.get_table_req.active_call
    (gauge)
    Shown as unit
    hive.metastore.api.init
    (rate)    		API initialization.
    Shown as task
    hive.metastore.api.init.active_call
    (gauge)    		Active API initialization.
    Shown as unit
    hive.metastore.api.shutdown
    (rate)    		API shutdown.
    Shown as task
    hive.metastore.api.shutdown.active_call
    (gauge)    		Active API shutdown.
    Shown as unit
    hive.metastore.db.created
    (rate)    		Total number of created database.
    Shown as item
    hive.metastore.db.deleted
    (rate)    		Total number of deleted database.
    Shown as item
    hive.metastore.db.init
    (rate)    		Number of initialized database.
    Shown as item
    hive.metastore.directsql_errors
    (gauge)    		Number of SQL error.
    Shown as unit
    hive.metastore.open_connections
    (gauge)    		Number of connection opened.
    Shown as connection
    hive.metastore.partition.created
    (rate)    		Total number of created partition.
    Shown as item
    hive.metastore.partition.deleted
    (rate)    		Total number of deleted partition.
    Shown as item
    hive.metastore.partition.init
    (rate)    		Number of initialized partition.
    Shown as item
    hive.metastore.table.created
    (rate)    		Total number of created table.
    Shown as table
    hive.metastore.table.deleted
    (rate)    		Total number of deleted table.
    Shown as table
    hive.metastore.table.init
    (rate)    		Number of initialized table.
    Shown as table
    hive.server.api.operation.initialized.75percentile
    (gauge)    		P75 time to init an operation.
    Shown as millisecond
    hive.server.api.operation.initialized.95percentile
    (gauge)    		P95 time to init an operation.
    Shown as millisecond
    hive.server.api.operation.initialized.active_call
    (gauge)    		Active initialized operation.
    Shown as user
    hive.server.api.operation.initialized.count
    (rate)    		Number of operation initialized.
    Shown as operation
    hive.server.api.operation.initialized.max
    (gauge)    		Max time to init an operation.
    Shown as millisecond
    hive.server.api.operation.initialized.mean
    (gauge)    		Mean time to init an operation.
    Shown as millisecond
    hive.server.api.operation.initialized.meanrate
    (gauge)    		Operation initialization rate.
    Shown as operation
    hive.server.api.operation.initialized.min
    (gauge)    		Min time to init an operation.
    Shown as millisecond
    hive.server.api.operation.pending.75percentile
    (gauge)    		P75 time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.pending.95percentile
    (gauge)    		P95 time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.pending.active_call
    (gauge)    		Active pending operation.
    Shown as operation
    hive.server.api.operation.pending.count
    (rate)    		Number of operation in pending state.
    Shown as operation
    hive.server.api.operation.pending.max
    (gauge)    		Max time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.pending.mean
    (gauge)    		Mean time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.pending.meanrate
    (gauge)    		Operation pending rate.
    Shown as operation
    hive.server.api.operation.pending.min
    (gauge)    		Min time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.running.75percentile
    (gauge)    		P75 time in running state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.running.95percentile
    (gauge)    		P95 time in running state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.running.active_call
    (gauge)    		Active running operation.
    Shown as operation
    hive.server.api.operation.running.count
    (rate)    		Number of operation in running state.
    Shown as millisecond
    hive.server.api.operation.running.max
    (gauge)    		Max time in running state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.running.mean
    (gauge)    		Mean time in running state for an sql operation.
    Shown as millisecond
    hive.server.api.operation.running.meanrate
    (gauge)    		Operation running rate.
    Shown as operation
    hive.server.api.operation.running.min
    (gauge)    		Min time in running state for an sql operation.
    Shown as millisecond
    hive.server.api.queries.executing.active_call
    (gauge)    		Active executing queries.
    Shown as query
    hive.server.api.sql_operation.pending.75percentile
    (gauge)    		P75 time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.pending.95percentile
    (gauge)    		P95 time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.pending.active_call
    (gauge)    		Active pending sql operation.
    Shown as operation
    hive.server.api.sql_operation.pending.count
    (rate)    		Number of sql operation in pending state.
    Shown as operation
    hive.server.api.sql_operation.pending.max
    (gauge)    		Max time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.pending.mean
    (gauge)    		Mean time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.pending.meanrate
    (gauge)    		Pending sql operation rate.
    Shown as operation
    hive.server.api.sql_operation.pending.min
    (gauge)    		Min time in pending state for an sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.running.75percentile
    (gauge)    		P75 time for running state for a sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.running.95percentile
    (gauge)    		P95 time for running state for a sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.running.active_call
    (gauge)    		Active running sql operation.
    Shown as operation
    hive.server.api.sql_operation.running.count
    (rate)    		Number of sql operation in running state.
    Shown as operation
    hive.server.api.sql_operation.running.max
    (gauge)    		Max time for running state for a sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.running.mean
    (gauge)    		Mean time for running state for a sql operation.
    Shown as millisecond
    hive.server.api.sql_operation.running.meanrate
    (gauge)    		Running sql operation rate.
    Shown as operation
    hive.server.api.sql_operation.running.min
    (gauge)    		Min time for running state for a sql operation.
    Shown as millisecond
    hive.server.memory.heap.committed
    (gauge)    		Committed memory for the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.heap.init
    (gauge)    		Memory used at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.heap.max
    (gauge)    		Maximum memory that can be used by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.heap.used
    (gauge)    		Memory used by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.non_heap.committed
    (gauge)    		Committed memory for the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.non_heap.init
    (gauge)    		Memory used at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.non_heap.max
    (gauge)    		Maximum memory that can be used by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.non_heap.used
    (gauge)    		Memory used by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.total.committed
    (gauge)    		Total committed memory for the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.total.init
    (gauge)    		Total memory at the initialization by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.total.max
    (gauge)    		Total maximum memory that can be used for the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.memory.total.used
    (gauge)    		Total memory used by the HiveServer2 or Metastore. Depends on the connected instance type
    Shown as byte
    hive.server.open_operations
    (gauge)    		Operation opened in the HiveServer2.
    Shown as operation
    hive.server.operation.completed.closed
    (rate)    		Number of closed operation.
    Shown as operation
    hive.server.operation.completed.finished
    (rate)    		Number of finished operation.
    Shown as operation
    hive.server.queries.compiling.75percentile
    (gauge)    		P75 time for compiling a query.
    Shown as millisecond
    hive.server.queries.compiling.95percentile
    (gauge)    		P95 time for compiling a query.
    Shown as millisecond
    hive.server.queries.compiling.active_call
    (gauge)    		Active compiling queries.
    Shown as query
    hive.server.queries.compiling.count
    (rate)    		Number of compiled query.
    Shown as query
    hive.server.queries.compiling.max
    (gauge)    		Max time for compiling a query.
    Shown as millisecond
    hive.server.queries.compiling.mean
    (gauge)    		Mean time for compiling a query.
    Shown as millisecond
    hive.server.queries.compiling.meanrate
    (gauge)    		Compiling query rate.
    Shown as query
    hive.server.queries.compiling.min
    (gauge)    		Min time for compiling a query.
    Shown as millisecond
    hive.server.queries.executing.75percentile
    (gauge)    		P75 time for executing a query.
    Shown as millisecond
    hive.server.queries.executing.95percentile
    (gauge)    		P95 time for executing a query.
    Shown as millisecond
    hive.server.queries.executing.count
    (rate)    		Number of executed queries.
    Shown as query
    hive.server.queries.executing.max
    (gauge)    		Max time for executing a query.
    Shown as millisecond
    hive.server.queries.executing.mean
    (gauge)    		Mean time for executing a query.
    Shown as millisecond
    hive.server.queries.executing.meanrate
    (gauge)    		Executing query rate.
    Shown as query
    hive.server.queries.executing.min
    (gauge)    		Min time for executing a query.
    Shown as millisecond
    hive.server.queries.submitted.75percentile
    (gauge)    		P75 time for a submitted query.
    Shown as millisecond
    hive.server.queries.submitted.95percentile
    (gauge)    		P95 time for a submitted query.
    Shown as millisecond
    hive.server.queries.submitted.active_call
    (gauge)    		Active submitted queries.
    Shown as query
    hive.server.queries.submitted.count
    (rate)    		Number of submitted query.
    Shown as query
    hive.server.queries.submitted.max
    (gauge)    		Max time for a submitted query.
    Shown as millisecond
    hive.server.queries.submitted.mean
    (gauge)    		Mean time for a submitted query.
    Shown as millisecond
    hive.server.queries.submitted.meanrate
    (gauge)    		Submitted query rate.
    Shown as query
    hive.server.queries.submitted.min
    (gauge)    		Min time for a submitted query.
    Shown as millisecond
    hive.server.queries.succeeded.count
    (rate)    		Number of succeeded queries.
    Shown as query
    hive.server.queries.succeeded.meanrate
    (gauge)    		Succeeded queries rate.
    Shown as query
    hive.server.session.active
    (gauge)    		Number of active session.
    Shown as session
    hive.server.session.active.time_mean
    (gauge)    		Average time a session has been active.
    Shown as millisecond
    hive.server.session.open
    (gauge)    		Number of opened session.
    Shown as session
    hive.server.session.open.time_mean
    (gauge)    		Average time a session has been opened.
    Shown as millisecond
    hive.server.sql_operation.completed.closed
    (rate)    		Number of closed sql operation.
    Shown as operation
    hive.server.sql_operation.completed.finished
    (rate)    		Number of finished sql operation.
    Shown as operation
    hive.server.sql_operation.user.active
    (gauge)    		Number of active user.
    Shown as user

    이벤트

    하이브 점검에는 이벤트가 포함되어 있지 않습니다.

    서비스 점검

    hive.can_connect
    Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored HiveServer2/Hive Metastore instance, WARNING if no metrics are collected, and OK otherwise.
    Statuses: ok, critical, warning

    트러블슈팅

    도움이 필요하신가요? Datadog 지원팀에 문의하세요.

    PREVIEWING: jack.davenport/custom-resource-code-blocks