Host

To configure this check for an Agent running on a host:

Metric collection

1. Edit the fluentd.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your Fluentd metrics. See the sample fluentd.d/conf.yaml for all available configuration options.

   init_config:
   
   instances:
     ## @param monitor_agent_url - string - required
     ## Monitor Agent URL to connect to.
     #
     - monitor_agent_url: http://example.com:24220/api/plugins.json
   

   Copy

2. Restart the Agent.

Log collection

You can use the Datadog Fluentd plugin to forward the logs directly from Fluentd to your Datadog account.

Add metadata to your logs

Proper metadata (including hostname and source) is the key to unlocking the full potential of your logs in Datadog. By default, the hostname and timestamp fields should be properly remapped with the remapping for reserved attributes.

Source and custom tags

Add the ddsource attribute with the name of the log integration in your logs in order to trigger the integration automatic setup in Datadog. Host tags are automatically set on your logs if there is a matching hostname in your infrastructure list. Use the ddtags attribute to add custom tags to your logs:

Setup Example:

  # Match events tagged with "datadog.**" and
  # send them to Datadog

<match datadog.**>
  @type datadog
  @id awesome_agent
  api_key <your_api_key>

  # Optional
  include_tag_key true
  tag_key 'tag'

  # Optional tags
  dd_source '<INTEGRATION_NAME>'
  dd_tags '<KEY1:VALUE1>,<KEY2:VALUE2>'

  <buffer>
          @type memory
          flush_thread_count 4
          flush_interval 3s
          chunk_limit_size 5m
          chunk_limit_records 500
  </buffer>
</match>

By default, the plugin is configured to send logs through HTTPS (port 443) using gzip compression. You can change this behavior by using the following parameters:

• use_http: Set this to false if you want to use TCP forwarding and update the host and port accordingly (default is true)
• use_compression: Compression is only available for HTTP. Disable it by setting this to false (default is true)
• compression_level: Set the compression level from HTTP. The range is from 1 to 9, 9 being the best ratio (default is 6)

Additional parameters can be used to change the endpoint used in order to go through a proxy:

• host: The proxy endpoint for logs not directly forwarded to Datadog (default value: http-intake.logs.datadoghq.com).
• port: The proxy port for logs not directly forwarded to Datadog (default value: 80).
• ssl_port: The port used for logs forwarded with a secure TCP/SSL connection to Datadog (default value: 443).
• use_ssl: Instructs the Agent to initialize a secure TCP/SSL connection to Datadog (default value: true).
• no_ssl_validation: Disables SSL hostname validation (default value: false).

Note: Set host and port to your region http-intake.logs.datadoghq.com 443.

<match datadog.**>

  #...
  host 'http-intake.logs.datadoghq.eu'

</match>

Kubernetes and Docker tags

Datadog tags are critical to be able to jump from one part of the product to another. Having the right metadata associated with your logs is therefore important in jumping from a container view or any container metrics to the most related logs.

If your logs contain any of the following attributes, these attributes are automatically added as Datadog tags on your logs:

• kubernetes.container_image
• kubernetes.container_name
• kubernetes.namespace_name
• kubernetes.pod_name
• docker.container_id

While the Datadog Agent collects Docker and Kubernetes metadata automatically, Fluentd requires a plugin for this. Datadog recommends using fluent-plugin-kubernetes_metadata_filter to collect this metadata.

Configuration example:

# Collect metadata for logs tagged with "kubernetes.**"
 <filter kubernetes.*>
   type kubernetes_metadata
 </filter>