There are a number of common issues that can get in the way when sending new container logs to Datadog with the Container Agent or with a locally installed Host Agent. If you experience issues sending new logs to Datadog, this guide helps you troubleshoot. If you continue to have trouble, contact our support team for further assistance.
The Agent collects Docker logs from the log files on disk by default in versions 6.33.0/7.33.0+ so long as the log files on disk are accessible by the Agent. DD_LOGS_CONFIG_DOCKER_CONTAINER_USE_FILE can be set to false to disable this behavior.
The Agent tails containers from the Docker socket if it cannot access the logs file path. If the Agent ever collected logs from a particular container using the Docker socket, it will continue to do so (including after Agent restarts) in order to avoid sending duplicate logs. To force the Agent to collect logs from file, set DD_LOGS_CONFIG_DOCKER_CONTAINER_FORCE_USE_FILE to true. This setting may cause duplicate logs to appear in Datadog.
When collecting Docker container logs from file, the Agent falls back on collection from the Docker socket if it cannot read from the directory where Docker container logs are stored (/var/lib/docker/containers on Linux). In some circumstances, the Datadog Agent may fail to collect logs from file. To diagnose this, check the Logs Agent status and look for a file type entry showing an error similar to the following:
- Type: file
Identifier: ce0bae54880ad75b7bf320c3d6cac1ef3efda21fc6787775605f4ba8b6efc834
Path: /var/lib/docker/containers/ce0bae54880ad75b7bf320c3d6cac1ef3efda21fc6787775605f4ba8b6efc834/ce0bae54880ad75b7bf320c3d6cac1ef3efda21fc6787775605f4ba8b6efc834-json.log
Status: Error: file /var/lib/docker/containers/ce0bae54880ad75b7bf320c3d6cac1ef3efda21fc6787775605f4ba8b6efc834/ce0bae54880ad75b7bf320c3d6cac1ef3efda21fc6787775605f4ba8b6efc834-json.log does not exist
This status means that the Agent is unable to find a log file for a given container. To resolve this issue, check that the folder containing Docker container logs is correctly exposed to the Datadog Agent container. On Linux, it corresponds to -v /var/lib/docker/containers:/var/lib/docker/containers:ro on the command line starting the Agent container, whereas on Windows it corresponds to -v c:/programdata/docker/containers:c:/programdata/docker/containers:ro. Note that the directory relative to the underlying host may be different due to specific configuration of the Docker daemon—this is not an issue pending a correct Docker volume mapping. For example, use -v /data/docker/containers:/var/lib/docker/containers:ro if the Docker data directory has been relocated to /data/docker on the underlying host.
If logs are collected but single lines appear to be split, check that the Docker daemon is using the JSON logging driver.
Note: When you install the Agent on the host, the Agent does not have permission to access /var/lib/docker/containers. Therefore, the Agent collects logs from the Docker socket when it is installed on the host.
For Agent version prior to 7.17, if the Docker Daemon starts while the host Agent is already running, restart the Agent to retrigger container collection.
In order for the Container Agent to collect logs from Docker containers, it needs to have access to the Docker socket. If it doesn’t have access, the following logs appear in agent.log:
2019-10-09 14:10:58 UTC | CORE | INFO | (pkg/logs/input/container/launcher.go:51 in NewLauncher) | Could not setup the docker launcher: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2019-10-09 14:10:58 UTC | CORE | INFO | (pkg/logs/input/container/launcher.go:58 in NewLauncher) | Could not setup the kubernetes launcher: /var/log/pods not found
2019-10-09 14:10:58 UTC | CORE | INFO | (pkg/logs/input/container/launcher.go:61 in NewLauncher) | Container logs won't be collected
Relaunch the Agent container with the following option: -v /var/run/docker.sock:/var/run/docker.sock:ro to allow access to the Docker socket.
If the Logs Agent status looks like the example in Check the Agent status but your logs still aren’t reaching the Datadog platform, there could be a problem with one of the following:
The required port (10516) for sending logs to Datadog is being blocked.
Your container is using a different logging driver than the Agent expects.
The Datadog Agent sends its logs to Datadog over TCP using port 10516. If that connection is not available, logs fail to be sent and an error is recorded in the agent.log file to that effect.
You can manually test your connection using OpenSSL, GnuTLS, or another SSL/TLS client. For OpenSSL, run the following command:
If opening the port 10516 is not an option, it is possible to configure the Datadog Agent to send logs through HTTPS by setting the DD_LOGS_CONFIG_FORCE_USE_HTTP environment variable to true:
Docker’s default is the json-file logging driver so the Container Agent tries to read from this first. If your containers are set to use a different logging driver, the Logs Agent indicates that it is able to successfully find your containers but it isn’t able to collect their logs. The Container Agent can also be configured to read from the journald logging driver.
If you’re unsure of which logging driver your containers are using, use docker inspect <CONTAINER_NAME> to see what logging driver you have set. The following block appears in the Docker Inspect when the container is using the JSON logging driver:
Mount the YAML file into your container following the instructions in the Docker Agent documentation. For more information on setting log drivers for Docker containers, see this documentation.
The Docker daemon can have performances issues while it is trying to retrieve logs for containers for which it has already stored large logs files on disk. This could lead to read timeouts when the Datadog Agent is gathering the containers’ logs from the Docker daemon.
When it occurs, the Datadog Agent outputs a log containing Restarting reader after a read timeout for a given container every 30 seconds and stops sending logs from that container while it is actually logging messages.
The default read timeout is set to 30 seconds, increasing this value gives more time to the Docker daemon to reply to the Datadog Agent. This value can be set in datadog.yaml using the logs_config.docker_client_read_timeout parameter or by using the environment variable DD_LOGS_CONFIG_DOCKER_CLIENT_READ_TIMEOUT. This value is a duration in seconds, find below an example increasing it to 60 seconds:
If you’re using the Host Agent, the user dd-agent needs to be added to the Docker group to have permission to read from the Docker socket. If you see the following error logs in the agent.log file:
2019-10-11 09:17:56 UTC | CORE | INFO | (pkg/autodiscovery/autoconfig.go:360 in initListenerCandidates) | docker listener cannot start, will retry: temporary failure in dockerutil, will retry later: could not determine docker server API version: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/version: dial unix /var/run/docker.sock: connect: permission denied
2019-10-11 09:17:56 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider docker: temporary failure in dockerutil, will retry later: could not determine docker server API version: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/version: dial unix /var/run/docker.sock: connect: permission denied
To add the host Agent to the Docker user group, perform the following command: usermod -a -G docker dd-agent.
