This guide describes how to set up the CSM Threats eBPF-less solution for eBPF disabled environments, such as AWS Fargate. The eBPF-less solution uses a ptrace-based Datadog Agent.
This guide also describes some advantages of the ptrace solution.
Threat Detection for Linux Without eBPF Support is in Preview. Reach out to your Datadog representative to sign up.
Summary of Agent options
CSM Threats includes two Agent options for threat detection and response:
eBPF solution
eBPF-less solution with ptrace: This version is only available where eBPF is not (Linux kernel versions 3.4 to 4.14).
eBPF improves safety by validating each program through the Linux kernel verifier. This ensures that a program can’t crash, fall into infinite loops, or harm the system.
eBPF is JIT (Just In Time) compiled and the output bytecode is executed on a eBPF VM sandbox. This prevents any kernel crash and provides competitive performance.
Easy to debug and maintain, can dynamically load programs, and has access to all information needed to trace the user space.
Some environments use instances with old kernels that do not have eBPF at all. The ptrace solution is provided for these environments.
The following features are not available in the eBPF-less Agent:
Security profiles, providing:
Anomaly detection
Auto-suppression of normal behavior for signal triaging
Malware detection
Network detections
The current implementation supports amd64 and arm64 architecture and ABIs, but can be extended to 32-bit ABIs.
Advantages of ptrace solution
A ptrace-based solution achieves a balance between robust threat detection and unwavering service availability. Some of the advantages of the ptrace-based solution are:
Precise process control: ptrace provides detailed inspection of memory and registers, safeguarding critical application workloads. This granular visibility is essential for identifying sophisticated threats. The Datadog procfs (Process Filesystem) scanner monitors all system-wide executions, enabling the surgical termination of malicious processes. Together, these tools protect from malicious activity.
Operational stability: Operating in user space, ptrace avoids the complexities and risks of kernel space, providing a safer and more manageable approach. In the event of a failure, a ptrace-based agent defaults to a fail-open state at the OS layer, keeping the system unaffected, even if the application hangs.
Performance efficiency: Recent benchmarks conducted by Datadog’s engineering team demonstrate that the Datadog ptrace-based implementation shows comparable performance to kernel-based solutions. Specifically, it introduces only a minimal overhead of around 3% for PostgreSQL workloads and negligible impacts for Redis operations, making it very efficient for most use cases.
Open source verification: Datadog has open-sourced the ptrace-based and eBPF Agent, allowing clients and the security community to verify its safety and effectiveness themselves, fostering transparency and trust in the solution.
eBPF-less Agent setup
You can set up the eBPF-less Agent on various platforms, including Docker and Linux hosts.
The eBPF-less Agent is designed for environments where eBPF is disabled, using ptrace for runtime security, and supports arm64/amd64 architectures.
Custom installation commands and configurations are required for deploying the eBPF-less Agent. Specific instructions are provided in this section for Docker and Linux host installations.
The eBPF-less solution includes two tracing modes for applications:
Wrap mode: Traces applications from the start.
Attach mode: Attaches to already running applications, but comes with more performance overhead and limitations.
eBPF-less setup steps
An additional environment variable is required on Docker. Add the following line to your docker installation command:
Alternatively, to manually install the .deb/.rmp provided custom build packages, modify the /etc/datadog-agent/system-probe.yaml file to enable CWS and eBPF-less mode as follows:
Specify additional configurations from the previous eBPF-less agent setup sections to install the custom version and enable eBPF-less mode.
Verify setup
To validate your Agent installation and setup, connect to your Linux host or Docker container and run:
sudo /opt/datadog-agent/embedded/bin/system-probe config|grep -A 1 ebpfless
You should see the output:
ebpfless:
enabled: true
Set up application tracing with eBPF-less Agent
After the eBPF-less Agent is installed and set up to use the eBPF-Free mode, you can set up how your application is traced. This section provides you two different methods:
Wrap mode: (Recommended) In this mode, your application is launched by the Datadog wrapper that traces it from the beginning using ptrace.
All spawned children are traced also.
A seccomp profile is applied to drastically reduce the ptracing overhead.
Attach mode: In this mode, you can specify a list of PIDs to attach to your application processes. This should be done quickly because your application is not ptraced until this is done.
In this mode, a seccomp profile cannot be applied. Consequently, there is a small amount ptracing overhead.
Both modes use the cws-instrumentation binary packaged with the Datadog Agent, and located at /opt/datadog-agent/embedded/bin/cws-instrumentation.
This tracer communicates with system-probe (part of the Datadog Agent) on localhost using port 5678. The system-probe address can be configured with the --probe-addr=host:port cws-instrumentation option. The server-side address can be updated through the runtime_security_config.ebpfless.socket option of the /etc/datadog-agent/system-probe.yaml Agent config file.
In wrap mode, the Datadog wrapper launchs the application. Here is an example:
If you already have an init script, here is a simple example of the required changes:
#!/bin/sh
set -e
### BEGIN INIT INFO# Provides: my_app# Required-Start: $network# Required-Stop: $network# Default-Start: 2 3 4 5# Default-Stop: 0 1 6# Short-Description: My application# Description: My application### END INIT INFO# Start the servicestart(){echo"Starting my app" /opt/datadog-agent/embedded/bin/cws-instrumentation trace -- /usr/bin/myapp &}# Stop the servicestop(){echo"Stopping my app" pkill -f /usr/bin/myapp
}### main logic ###case"$1" in
start) start
;; stop) stop
;; restart) stop
start
;; *)echo $"Usage: $0 {start|stop|restart}"exit1esacexit0
Docker
For Docker application deployments, you should modify your Dockerfile to wrap your application like this:
FROM gcr.io/datadoghq/agent:7 AS datadogagent
FROM ubuntu:latest
COPY --from=datadogagent /opt/datadog-agent/embedded/bin/cws-instrumentation .
ENTRYPOINT ["/cws-instrumentation", "trace", "--"]CMD ["/bin/bash", "-c", "while true; do sleep 1; echo my app is running; done"]
When running your docker application, it’s important to give it an additional capability by adding --cap-add=SYS_PTRACE to your docker run command.
You also have to connect the container to Datadog on port 5678 by doing one of the following:
Launch both containers with the --network host option.
Use the [Docker network][6] feature to run both containers on the same bridge network.
The wrap mode is recommended because the attach mode has the following limitations:
It misses all initialization made by the application until Datadog attaches to it.
When attaching, Datadog cannot set up a seccomp profile.
More performance overhead.
If the traced application restarts, Datadog must ensure that the tracer restarts also.
The attach mode differs from the wrap mode by attaching directly the tracer on an already running application, like this:
