Overview

Choose a configuration option below to begin ingesting your logs. If you are already using a log-shipper daemon, refer to the dedicated documentation for Rsyslog, Syslog-ng, NXlog, FluentD, or Logstash.

Consult the list of available Datadog log collection endpoints if you want to send your logs directly to Datadog.

Note: When sending logs in a JSON format to Datadog, there is a set of reserved attributes that have a specific meaning within Datadog. See the Reserved Attributes section to learn more.

Setup

• Host
• Application
• Container
• Serverless
• Cloud/Integration

1. Install the Datadog Agent.
2. To enable log collection, change logs_enabled: false to logs_enabled: true in your Agent’s main configuration file (datadog.yaml). See the Host Agent Log collection documentation for more information and examples.
3. Once enabled, the Datadog Agent can be configured to tail log files or listen for logs sent over UDP/TCP, filter out logs or scrub sensitive data, and aggregate multi-line logs.

1. Install the Datadog Agent.
2. To enable log collection, change logs_enabled: false to logs_enabled: true in your Agent’s main configuration file (datadog.yaml). See the Host Agent Log collection documentation for more information and examples.
3. Follow your application language installation instructions to configure a logger and start generating logs:

Choose a container or orchestrator provider and follow their dedicated log collection instructions:

Notes:

• The Datadog Agent can collect logs directly from container stdout/stderr without using a logging driver. When the Agent’s Docker check is enabled, container and orchestrator metadata are automatically added as tags to your logs.

• It is possible to collect logs from all your containers or only a subset filtered by container image, label, or name.

• Autodiscovery can also be used to configure log collection directly in the container labels.

• In Kubernetes environments, you can also leverage the daemonset installation.

Use the Datadog Forwarder, an AWS Lambda function that ships logs from your environment to Datadog. To enable log collection in your AWS serverless environment, refer to the Datadog Forwarder documentation.

Select your Cloud provider below to see how to automatically collect your logs and forward them to Datadog:

Datadog integrations and log collection are tied together. You can use an integration’s default configuration file to enable dedicated processors, parsing, and facets in Datadog. To begin log collection with an integration:

1. Select an integration from the Integrations page and follow the setup instructions.
2. Follow the integration’s log collection instructions. This section covers how to uncomment the logs section in that integration’s conf.yaml file and configure it for your environment.

Reduce data transfer fees

Use Datadog’s Cloud Network Monitoring to identify your organization’s highest throughput applications. Connect to Datadog over supported private connections and send data over a private network to avoid the public internet and reduce your data transfer fees. After you switch to private links, use Datadog’s Cloud Cost Management tools to verify the impact and monitor the reduction in your cloud costs.

For more information, see How to send logs to Datadog while reducing data transfer fees.

Additional configuration options

Logging endpoints

Datadog provides logging endpoints for both SSL-encrypted connections and unencrypted connections. Use the encrypted endpoint when possible. The Datadog Agent uses the encrypted endpoint to send logs to Datadog. More information is available in the Datadog security documentation.

Supported endpoints

Use the site selector dropdown on the right side of the page to see supported endpoints by Datadog site.

Custom log forwarding

Any custom process or logging library able to forward logs through TCP or HTTP can be used in conjunction with Datadog Logs.

gnutls-cli intake.logs.datadoghq.com:10516

openssl s_client -connect intake.logs.datadoghq.com:10516

<DATADOG_API_KEY> Log sent directly using TLS

<DATADOG_API_KEY> {"message":"json formatted log", "ddtags":"env:my-env,user:my-user", "ddsource":"my-integration", "hostname":"my-hostname", "service":"my-service"}

gnutls-cli tcp-intake.logs.datadoghq.eu:443

openssl s_client -connect tcp-intake.logs.datadoghq.eu:443

<DATADOG_API_KEY> Log sent directly using TLS

<DATADOG_API_KEY> {"message":"json formatted log", "ddtags":"env:my-env,user:my-user", "ddsource":"my-integration", "hostname":"my-hostname", "service":"my-service"}

Notes:

• The HTTPS API supports logs of sizes up to 1MB. However, for optimal performance, it is recommended that an individual log be no greater than 25K bytes. If you use the Datadog Agent for logging, it is configured to split a log at 256kB (256000 bytes).
• A log event should not have more than 100 tags, and each tag should not exceed 256 characters for a maximum of 10 million unique tags per day.
• A log event converted to JSON format should contain less than 256 attributes. Each of those attribute’s keys should be less than 50 characters, nested in less than 20 successive levels, and their respective value should be less than 1024 characters if promoted as a facet.
• Log events can be submitted with a timestamp that is up to 18h in the past.

Log events that do not comply with these limits might be transformed or truncated by the system or not indexed if outside the provided time range. However, Datadog tries to preserve as much user data as possible.

There is an additional truncation in fields that applies only to indexed logs: the value is truncated to 75 KiB for the message field and 25 KiB for non-message fields. Datadog still stores the full text, and it remains visible in regular list queries in the Logs Explorer. However, the truncated version will be displayed when performing a grouped query, such as when grouping logs by that truncated field or performing similar operations that display that specific field.

Attributes and tags

Attributes prescribe logs facets, which are used for filtering and searching in Log Explorer. See the dedicated attributes and aliasing documentation for a list of reserved and standard attributes and to learn how to support a naming convention with logs attributes and aliasing.

Attributes for stack traces

When logging stack traces, there are specific attributes that have a dedicated UI display within your Datadog application such as the logger name, the current thread, the error type, and the stack trace itself.

To enable these functionalities use the following attribute names:

Note: By default, integration Pipelines attempt to remap default logging library parameters to those specific attributes and parse stack traces or traceback to automatically extract the error.message and error.kind.

For more information, see the complete source code attributes documentation.

Next steps

Once logs are collected and ingested, they are available in Log Explorer. Log Explorer is where you can search, enrich, and view alerts on your logs. See the Log Explorer documentation to begin analyzing your log data, or see the additional log management documentation below.

Further Reading

Additional helpful documentation, links, and articles:

How to manage log files using LogrotateBLOG Advanced log collection configurationsDOCUMENTATION Discover how to process your logsDOCUMENTATION Learn more about parsingDOCUMENTATION Datadog live tail functionalityDOCUMENTATION See how to explore your logsDOCUMENTATION Logging Without Limits*DOCUMENTATION