Overview

CloudPrem uses a decoupled architecture which separates the compute (indexing and searching) and data on an object storage. This allows for independent scaling and optimization of different cluster components based on workload demands.

Components

The CloudPrem cluster, typically deployed on Kubernetes (EKS), consists of several components:

Indexers

Responsible for receiving logs from Datadog Agents. Indexers process, index, and store logs in index files called splits to the object storage (for example, Amazon S3).

Searchers

Handle search queries from the Datadog UI, reading metadata from Metastore and fetching data from the object storage.

Metastore

Stores metadata about the indexes, including split locations on the object storage. CloudPrem uses PostgreSQL for this purpose.

Control Plane

Schedules indexing jobs called indexing pipelines on indexers.

Janitor

Performs maintenance tasks, applying retention policies, garbage collecting expired splits, and running delete query jobs.

Further reading

Additional helpful documentation, links, and articles:

CloudPrem OverviewDOCUMENTATION Install CloudPrem and Send Logs with the AgentDOCUMENTATION Configure CloudPrem IngressDOCUMENTATION Configure AWSDOCUMENTATION Configure CloudPrem Log ProcessingDOCUMENTATION Learn more about Cluster Sizing and OperationsDOCUMENTATION TroubleshootingDOCUMENTATION