Pulsar

Supported OS Windows Mac OS

통합 버전3.2.0

개요

본 점검은 Datadog 에이전트를 통해 Pulsar을 모니터링합니다.

설정

아래 지침을 따라 호스트에서 실행되는 에이전트에 대해 이 점검을 설치하고 설정하세요. 컨테이너화된 환경의 경우 이러한 지침을 적용하는 데 가이드가 필요하면 오토파일럿 통합 템플릿을 참조하세요.

설치

Pulsar 점검은 Datadog 에이전트 패키지에 포함되어 있습니다. 서버에 추가 설치가 필요하지 않습니다.

구성

  1. 에이전트의 설정 디렉터리 루트의 conf.d/ 폴더에 있는 pulsar.d/conf.yaml 파일을 편집하여 pulsar 성능 데이터 수집을 시작합니다. 사용 가능한 모든 설정 옵션은 pulsar.d/conf.yaml 샘플을 참고하세요.

  2. Agent를 재시작합니다.

검증

에이전트 상태 하위 명령을 실행하고 점검 섹션에서 pulsar를 찾습니다.

수집한 데이터

메트릭

pulsar.active_connections
(gauge)		The number of active connections.
Shown as connection
pulsar.authentication_failures_count.count
(count)		The number of failing authentication operations.
pulsar.authentication_success_count.count
(count)		The number of successful authentication operations.
pulsar.bookie_DELETED_LEDGER_COUNT.count
(count)		The total number of ledgers deleted since the bookie has started.
pulsar.bookie_READ_BYTES.count
(count)		The total number of bytes read from the bookie.
Shown as byte
pulsar.bookie_SERVER_STATUS
(gauge)		The server status for bookie server. 1: the bookie is running in writable mode.0: the bookie is running in readonly mode.
pulsar.bookie_WRITE_BYTES.count
(count)		The total number of bytes written to the bookie.
Shown as byte
pulsar.bookie_entries_count
(gauge)		The total number of entries stored in the bookie.
pulsar.bookie_flush
(gauge)		The table flush latency of bookie memory.
pulsar.bookie_journal_JOURNAL_CB_QUEUE_SIZE
(gauge)		The total number of callbacks pending in the callback queue.
pulsar.bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE
(gauge)		The total number of force write (fsync) requests pending in the force-write queue.
Shown as request
pulsar.bookie_journal_JOURNAL_QUEUE_SIZE
(gauge)		The total number of requests pending in the journal queue.
Shown as request
pulsar.bookie_journal_JOURNAL_SYNC_count.count
(count)		The total number of journal fsync operations happening at the bookie. The success label is used to distinguish successes and failures.
pulsar.bookie_ledger_writable_dirs
(gauge)		The number of writable directories in the bookie.
pulsar.bookie_ledgers_count
(gauge)		The total number of ledgers stored in the bookie.
pulsar.bookie_read_cache_size
(gauge)		The bookie read cache size (in bytes).
Shown as byte
pulsar.bookie_throttled_write_requests.count
(count)		The number of write requests to be throttled.
Shown as request
pulsar.bookie_write_cache_size
(gauge)		The bookie write cache size (in bytes).
Shown as byte
pulsar.bookkeeper_server_ADD_ENTRY_count.count
(count)		The total number of ADD_ENTRY requests received at the bookie. The success label is used to distinguish successes and failures.
Shown as request
pulsar.bookkeeper_server_BOOKIE_QUARANTINE_count.count
(count)		The number of bookie clients to be quarantined.
pulsar.bookkeeper_server_READ_ENTRY_count.count
(count)		The total number of READ_ENTRY requests received at the bookie. The success label is used to distinguish successes and failures.
Shown as request
pulsar.brk_ml_cursor_nonContiguousDeletedMessagesRange
(gauge)		The number of non-contiguous deleted messages ranges.
pulsar.brk_ml_cursor_persistLedgerErrors
(gauge)		The number of ledger errors occurred when acknowledgment states fail to be persistent to the ledger.
Shown as error
pulsar.brk_ml_cursor_persistLedgerSucceed
(gauge)		The number of acknowledgment states that is persistent to a ledger.
pulsar.brk_ml_cursor_persistZookeeperErrors
(gauge)		The number of ledger errors occurred when acknowledgment states fail to be persistent to ZooKeeper.
Shown as error
pulsar.brk_ml_cursor_persistZookeeperSucceed
(gauge)		The number of acknowledgment states that is persistent to ZooKeeper.
pulsar.brk_ml_cursor_readLedgerSize
(gauge)		The size of read from ledger.
pulsar.brk_ml_cursor_writeLedgerLogicalSize
(gauge)		The size of write to ledger (accounting for without replicas).
pulsar.brk_ml_cursor_writeLedgerSize
(gauge)		The size of write to ledger.
pulsar.broker_load_manager_bundle_assignment
(gauge)		The summary of latency of bundles ownership operations.
pulsar.broker_lookup.count
(count)		Number of samples of the latency of all lookup operations.
pulsar.broker_lookup.quantle
(count)		Latency of all lookup operations.
pulsar.broker_lookup.sum
(count)		Total latency of all lookup operations.
pulsar.broker_lookup_answers.count
(count)		The number of lookup responses (i.e. not redirected requests).
Shown as response
pulsar.broker_lookup_failures.count
(count)		The number of lookup failures.
pulsar.broker_lookup_pending_requests
(gauge)		The number of pending lookups in broker. When it is up to the threshold, new requests are rejected.
Shown as request
pulsar.broker_lookup_redirects.count
(count)		The number of lookup redirected requests.
Shown as request
pulsar.broker_throttled_connections
(gauge)		The number of throttled connections.
Shown as connection
pulsar.broker_throttled_connections_global_limit
(gauge)		The number of throttled connections because of per-connection limit.
Shown as connection
pulsar.broker_topic_load_pending_requests
(gauge)		The load of pending topic operations.
pulsar.bundle_consumer_count
(gauge)		The consumer count of the topics in this bundle.
pulsar.bundle_msg_rate_in
(gauge)		The total message rate coming into the topics in this bundle (messages/second).
pulsar.bundle_msg_rate_out
(gauge)		The total message rate going out from the topics in this bundle (messages/second).
pulsar.bundle_msg_throughput_in
(gauge)		The total throughput coming into the topics in this bundle (bytes/second).
pulsar.bundle_msg_throughput_out
(gauge)		The total throughput going out from the topics in this bundle (bytes/second).
pulsar.bundle_producer_count
(gauge)		The producer count of the topics in this bundle.
pulsar.bundle_topics_count
(gauge)		The topic count in this bundle.
pulsar.compaction_compacted_entries_count
(gauge)		The total number of the compacted entries.
pulsar.compaction_compacted_entries_size
(gauge)		The total size of the compacted entries.
pulsar.compaction_duration_time_in_mills
(gauge)		The duration time of the compaction.
pulsar.compaction_failed_count
(gauge)		The total number of failures of the compaction.
pulsar.compaction_read_throughput
(gauge)		The read throughput of the compaction.
pulsar.compaction_removed_event_count
(gauge)		The total number of removed events of the compaction.
pulsar.compaction_succeed_count
(gauge)		The total number of successes of the compaction.
pulsar.compaction_write_throughput
(gauge)		The write throughput of the compaction.
pulsar.connection_closed_total_count
(gauge)		The total number of closed connections.
Shown as connection
pulsar.connection_create_fail_count
(gauge)		The number of failed connections.
Shown as connection
pulsar.connection_create_success_count
(gauge)		The number of successfully created connections.
Shown as connection
pulsar.connection_created_total_count
(gauge)		The total number of connections.
Shown as connection
pulsar.consumer_available_permits
(gauge)		The available permits for for a consumer.
pulsar.consumer_blocked_on_unacked_messages
(gauge)		Indicate whether a consumer is blocked on unacknowledged messages or not. 1 means the consumer is blocked on waiting unacknowledged messages to be acked.0 means the consumer is not blocked on waiting unacknowledged messages to be acked.
pulsar.consumer_msg_rate_out
(gauge)		The total message dispatch rate for a consumer (messages/second).
pulsar.consumer_msg_rate_redeliver
(gauge)		The total message rate for message being redelivered (messages/second).
pulsar.consumer_msg_throughput_out
(gauge)		The total message dispatch throughput for a consumer (bytes/second).
pulsar.consumer_unacked_messages
(gauge)		The total number of unacknowledged messages of a consumer (messages).
Shown as message
pulsar.consumers_count
(gauge)		The number of active consumers of the namespace connected to this broker.
pulsar.expired_token_count.count
(count)		The number of expired tokens in Pulsar.
pulsar.function_last_invocation
(gauge)		The timestamp of the last invocation of the function.
pulsar.function_processed_successfully_total.count
(count)		The total number of messages processed successfully.
Shown as message
pulsar.function_processed_successfully_total_1min.count
(count)		The total number of messages processed successfully in the last 1 minute.
Shown as message
pulsar.function_received_total.count
(count)		The total number of messages received from source.
Shown as message
pulsar.function_received_total_1min.count
(count)		The total number of messages received from source in the last 1 minute.
Shown as message
pulsar.function_system_exceptions_total.count
(count)		The total number of system exceptions.
pulsar.function_system_exceptions_total_1min.count
(count)		The total number of system exceptions in the last 1 minute.
pulsar.function_user_exceptions_total.count
(count)		The total number of user exceptions.
pulsar.function_user_exceptions_total_1min.count
(count)		The total number of user exceptions in the last 1 minute.
pulsar.in_bytes_total
(gauge)		The number of messages in bytes received for this topic.
Shown as byte
pulsar.in_messages_total
(gauge)		The total number of messages received for this topic.
Shown as message
pulsar.jetty_async_dispatches_total.count
(count)		Number of requested that have been asynchronously dispatched.
pulsar.jetty_async_requests_total.count
(count)		Total number of async requests.
Shown as request
pulsar.jetty_async_requests_waiting
(gauge)		Currently waiting async requests.
Shown as request
pulsar.jetty_async_requests_waiting_max
(gauge)		Maximum number of waiting async requests.
Shown as request
pulsar.jetty_dispatched_active
(gauge)		Number of dispatches currently active.
pulsar.jetty_dispatched_active_max
(gauge)		Maximum number of active dispatches being handled.
pulsar.jetty_dispatched_time_max
(gauge)		Maximum time spent in dispatch handling.
pulsar.jetty_dispatched_time_seconds_total.count
(count)		Total time spent in dispatch handling.
Shown as second
pulsar.jetty_dispatched_total.count
(count)		Number of dispatches.
pulsar.jetty_expires_total.count
(count)		Number of async requests requests that have expired.
Shown as request
pulsar.jetty_request_time_max_seconds
(gauge)		Maximum time spent handling requests.
Shown as second
pulsar.jetty_request_time_seconds_total.count
(count)		Total time spent in all request handling.
Shown as second
pulsar.jetty_requests_active
(gauge)		Number of requests currently active.
Shown as request
pulsar.jetty_requests_active_max
(gauge)		Maximum number of requests that have been active at once.
Shown as request
pulsar.jetty_requests_total.count
(count)		Number of requests.
Shown as request
pulsar.jetty_responses_bytes_total.count
(count)		Total number of bytes across all responses.
Shown as byte
pulsar.jetty_responses_total.count
(count)		Number of responses, labeled by status code. The code label can be "1xx", "2xx", "3xx", "4xx", or "5xx".
Shown as response
pulsar.jetty_stats_seconds
(gauge)		Time in seconds stats have been collected for.
Shown as second
pulsar.lb_bandwidth_in_usage
(gauge)		The broker inbound bandwith usage (in percent).
pulsar.lb_bandwidth_out_usage
(gauge)		The broker outbound bandwith usage (in percent).
pulsar.lb_bundles_split_count.count
(count)		bundle split count in this bundle splitting check interval
pulsar.lb_cpu_usage
(gauge)		The broker cpu usage (in percent).
pulsar.lb_directMemory_usage
(gauge)		The broker process direct memory usage (in percent).
pulsar.lb_memory_usage
(gauge)		The broker process memory usage (in percent).
pulsar.lb_unload_broker_count.count
(count)		Unload broker count in this bundle unloading
pulsar.lb_unload_bundle_count.count
(count)		Bundle unload count in this bundle unloading
pulsar.ml_AddEntryBytesRate
(gauge)		The bytes/s rate of messages added
pulsar.ml_AddEntryErrors
(gauge)		The number of addEntry requests that failed
Shown as request
pulsar.ml_AddEntryMessagesRate
(gauge)		The msg/s rate of messages added
pulsar.ml_AddEntrySucceed
(gauge)		The number of addEntry requests that succeeded
Shown as request
pulsar.ml_AddEntryWithReplicasBytesRate
(gauge)		The bytes/s rate of messages added with replicas
pulsar.ml_MarkDeleteRate
(gauge)		The rate of mark-delete ops/s
pulsar.ml_NumberOfMessagesInBacklog
(gauge)		The number of backlog messages for all the consumers
Shown as message
pulsar.ml_ReadEntriesBytesRate
(gauge)		The bytes/s rate of messages read
pulsar.ml_ReadEntriesErrors
(gauge)		The number of readEntries requests that failed
Shown as request
pulsar.ml_ReadEntriesRate
(gauge)		The msg/s rate of messages read
pulsar.ml_ReadEntriesSucceeded
(gauge)		The number of readEntries requests that succeeded
Shown as request
pulsar.ml_StoredMessagesSize
(gauge)		The total size of the messages in active ledgers (accounting for the multiple copies stored)
pulsar.ml_cache_evictions
(gauge)		The number of cache evictions during the last minute.
Shown as eviction
pulsar.ml_cache_hits_rate
(gauge)		The number of cache hits per second on the broker side.
Shown as hit
pulsar.ml_cache_hits_throughput
(gauge)		The amount of data is retrieved from the cache on the broker side (in byte/s).
pulsar.ml_cache_misses_rate
(gauge)		The number of cache misses per second on the broker side.
Shown as miss
pulsar.ml_cache_misses_throughput
(gauge)		The amount of data is not retrieved from the cache on the broker side (in byte/s).
pulsar.ml_cache_pool_active_allocations
(gauge)		The number of currently active allocations in direct arena
pulsar.ml_cache_pool_active_allocations_huge
(gauge)		The number of currently active huge allocation in direct arena
pulsar.ml_cache_pool_active_allocations_normal
(gauge)		The number of currently active normal allocations in direct arena
pulsar.ml_cache_pool_active_allocations_small
(gauge)		The number of currently active small allocations in direct arena
pulsar.ml_cache_pool_allocated
(gauge)		The total allocated memory of chunk lists in direct arena
pulsar.ml_cache_pool_used
(gauge)		The total used memory of chunk lists in direct arena
pulsar.ml_cache_used_size
(gauge)		The size in byte used to store the entries payloads
Shown as byte
pulsar.ml_count
(gauge)		The number of currently opened managed ledgers
pulsar.out_bytes_total
(gauge)		The total number of messages in bytes read from this topic.
Shown as byte
pulsar.out_messages_total
(gauge)		The total number of messages read from this topic.
Shown as message
pulsar.producers_count
(gauge)		The number of active producers of the namespace connected to this broker.
pulsar.proxy_active_connections
(gauge)		Number of connections currently active in the proxy.
Shown as connection
pulsar.proxy_binary_bytes.count
(count)		Counter of proxy bytes.
Shown as byte
pulsar.proxy_binary_ops.count
(count)		Counter of proxy operations.
pulsar.proxy_new_connections.count
(count)		Counter of connections being opened in the proxy.
Shown as connection
pulsar.proxy_rejected_connections.count
(count)		Counter for connections rejected due to throttling.
Shown as connection
pulsar.rate_in
(gauge)		The total message rate of the namespace coming into this broker (messages/second).
pulsar.rate_out
(gauge)		The total message rate of the namespace going out from this broker (messages/second).
pulsar.replication_backlog
(gauge)		The total backlog of the namespace replicating to remote cluster (messages).
Shown as message
pulsar.replication_connected_count
(gauge)		The count of replication-subscriber up and running to replicate to remote cluster.
pulsar.replication_delay_in_seconds
(gauge)		Time in seconds from the time a message was produced to the time when it is about to be replicated.
Shown as second
pulsar.replication_rate_expired
(gauge)		Total rate of messages expired (messages/second).
pulsar.replication_rate_in
(gauge)		The total message rate of the namespace replicating from remote cluster (messages/second).
pulsar.replication_rate_out
(gauge)		The total message rate of the namespace replicating to remote cluster (messages/second).
pulsar.replication_throughput_in
(gauge)		The total throughput of the namespace replicating from remote cluster (bytes/second).
pulsar.replication_throughput_out
(gauge)		The total throughput of the namespace replicating to remote cluster (bytes/second).
pulsar.sink_last_invocation
(gauge)		The timestamp of the last invocation of the sink.
pulsar.sink_received_total.count
(count)		The total number of records that a sink has received from Pulsar topics.
pulsar.sink_received_total_1min.count
(count)		The total number of messages that a sink has received from Pulsar topics in the last 1 minute.
Shown as message
pulsar.sink_sink_exception
(gauge)		The exception from a sink.
pulsar.sink_sink_exceptions_total.count
(count)		The total number of sink exceptions.
pulsar.sink_sink_exceptions_total_1min.count
(count)		The total number of sink exceptions in the last 1 minute.
pulsar.sink_system_exception
(gauge)		The exception from system code.
pulsar.sink_system_exceptions_total.count
(count)		The total number of system exceptions.
pulsar.sink_system_exceptions_total_1min.count
(count)		The total number of system exceptions in the last 1 minute.
pulsar.sink_written_total.count
(count)		The total number of records processed by a sink.
pulsar.sink_written_total_1min.count
(count)		The total number of records processed by a sink in the last 1 minute.
pulsar.source_last_invocation
(gauge)		The timestamp of the last invocation of the source.
pulsar.source_received_total.count
(count)		The total number of records received from source.
pulsar.source_received_total_1min.count
(count)		The total number of records received from source in the last 1 minute.
pulsar.source_source_exception
(gauge)		The exception from a source.
pulsar.source_source_exceptions_total.count
(count)		The total number of source exceptions.
pulsar.source_source_exceptions_total_1min.count
(count)		The total number of source exceptions in the last 1 minute.
pulsar.source_system_exception
(gauge)		The exception from system code.
pulsar.source_system_exceptions_total.count
(count)		The total number of system exceptions.
pulsar.source_system_exceptions_total_1min.count
(count)		The total number of system exceptions in the last 1 minute.
pulsar.source_written_total.count
(count)		The total number of records written to a Pulsar topic.
pulsar.source_written_total_1min.count
(count)		The total number of records written to a Pulsar topic in the last 1 minute.
pulsar.split_bytes_read.count
(count)		Number of bytes read from BookKeeper.
Shown as byte
pulsar.split_num_messages_deserialized.count
(count)		Number of messages deserialized.
Shown as message
pulsar.split_num_record_deserialized.count
(count)		Number of records deserialized.
pulsar.storage_backlog_quota_limit
(gauge)		The total amount of the data in this topic that limit the backlog quota (bytes).
Shown as byte
pulsar.storage_backlog_size
(gauge)		The total backlog size of the topics of this namespace owned by this broker (messages).
pulsar.storage_logical_size
(gauge)		The storage size of topics in the namespace owned by the broker without replicas (in bytes).
pulsar.storage_offloaded_size
(gauge)		The total amount of the data in this namespace offloaded to the tiered storage (bytes).
pulsar.storage_read_rate
(gauge)		The total message batches (entries) read from the storage for this namespace (message batches / second).
pulsar.storage_size
(gauge)		The total storage size of the topics in this namespace owned by this broker (bytes).
pulsar.storage_write_rate
(gauge)		The total message batches (entries) written to the storage for this namespace (message batches / second).
pulsar.subscription_back_log
(gauge)		The total backlog of a subscription (messages).
Shown as message
pulsar.subscription_blocked_on_unacked_messages
(gauge)		Indicate whether a subscription is blocked on unacknowledged messages or not. 1 means the subscription is blocked on waiting unacknowledged messages to be acked.0 means the subscription is not blocked on waiting unacknowledged messages to be acked.
pulsar.subscription_delayed
(gauge)		The total message batches (entries) are delayed for dispatching.
pulsar.subscription_msg_rate_out
(gauge)		The total message dispatch rate for a subscription (messages/second).
pulsar.subscription_msg_rate_redeliver
(gauge)		The total message rate for message being redelivered (messages/second).
pulsar.subscription_msg_throughput_out
(gauge)		The total message dispatch throughput for a subscription (bytes/second).
pulsar.subscription_unacked_messages
(gauge)		The total number of unacknowledged messages of a subscription (messages).
Shown as message
pulsar.subscriptions_count
(gauge)		The number of Pulsar subscriptions of the namespace served by this broker.
pulsar.throughput_in
(gauge)		The total throughput of the namespace coming into this broker (bytes/second).
pulsar.throughput_out
(gauge)		The total throughput of the namespace going out from this broker (bytes/second).
pulsar.topics_count
(gauge)		The number of Pulsar topics of the namespace owned by this broker.
pulsar.txn_aborted_count.count
(count)		Number of aborted transactions of this coordinator.
Shown as transaction
pulsar.txn_active_count
(gauge)		Number of active transactions.
Shown as transaction
pulsar.txn_append_log_count.count
(count)		Number of append transaction logs.
pulsar.txn_committed_count.count
(count)		Number of committed transactions.
Shown as transaction
pulsar.txn_created_count.count
(count)		Number of created transactions.
Shown as transaction
pulsar.txn_timeout_count.count
(count)		Number of timeout transactions.
Shown as transaction

로그 수집

  1. Pulsar 로그 통합은 Pulsar의 기본 로그 형식을 지원합니다. 다른 형식이 있을 경우에는 통합 파이프라인을 복제하고 편집하세요.

  2. Datadog Agent에서 로그 수집은 기본적으로 비활성화되어 있으므로 datadog.yaml 파일에서 활성화합니다.

    logs_enabled: true

  3. pulsar.d/conf.yaml 파일에서 로그 구성 블록의 주석 처리를 제거하고 편집합니다. 환경에 따라 경로 파라미터 값을 변경합니다. 사용 가능한 모든 구성 옵션은 샘플 pulsar.d/conf.yaml을 참고하세요.

     logs:
   - type: file
     path: /pulsar/logs/pulsar.log
     source: pulsar

  4. Agent를 재시작합니다

이벤트

Pulsar 통합에는 이벤트가 포함되어 있지 않습니다.

서비스 점검

pulsar.openmetrics.health
Returns CRITICAL if the Agent is unable to connect to the OpenMetrics endpoint, otherwise returns OK.
Statuses: ok, critical

트러블슈팅

도움이 필요하신가요? Datadog 고객 지원팀에 문의하세요.

PREVIEWING: guacbot/translation-pipeline