Overview

If you experience unexpected behavior with Datadog Application Security Management (ASM), there are common issues you can investigate, as mentioned below. If you continue to have trouble, reach out to Datadog support for further assistance.

ASM rate limits

ASM traces are rate-limited to 100 traces per second. Traces sent after the limit are not reported. Contact Datadog support if you need to change the limit.

No security traces detected by ASM

There are a series of steps that must run successfully for threat information to appear in the ASM Trace and Signals Explorer. It is important to check each step when investigating this issue. Additional troubleshooting steps for specific languages are in the language tab at the end.

Confirm ASM is enabled

You can use the metric datadog.apm.appsec_host to check if ASM is running.

  1. Go to Metrics > Summary in Datadog.
  2. Search for the metric datadog.apm.appsec_host. If the metric doesn’t exist, then there are no services running ASM. If the metric exists, the services are reported with the metric tags host and service.
  3. Select the metric, and in the Tags section, search for service to see which services are running ASM.

If you are not seeing datadog.apm.appsec_host, check the in-app instructions to confirm that all steps for the initial setup are complete.

ASM data is sent with APM traces. See APM troubleshooting to confirm APM setup and check for connection errors.

Send a test attack to your application

To test your ASM setup, trigger the Security Scanner Detected rule by running a file that contains the following curl script:

for ((i=1;i<=250;i++));
do
# Target existing service's routes
curl https://your-application-url/existing-route -A dd-test-scanner-log;
# Target non existing service's routes
curl https://your-application-url/non-existing-route -A dd-test-scanner-log;
done

A few minutes after you enable your application and exercise it, and if it’s successful, threat information appears in the Trace and Signals Explorer.

Security Signal details page showing tags, metrics, suggested next steps, and attacker IP addresses associated with a threat.

Check if required tracer integrations are deactivated

ASM relies on certain tracer integrations. If they are deactivated, ASM won’t work. To see if there are deactivated integrations, look for disabled_integrations in your startup logs.

The required integrations vary by language.

For Python, the WSGI integration is required along with the integration for the framework you’re using, such as the Django or Flask integration.

Check Datadog Agent configuration

To troubleshoot this step of the process, do the following:

  • Check the details of the running Agent at this address http://<agent-machine-name>:<agent-port>/info, usually http://localhost:8126/info.
  • Ensure there are no Agent transmission errors related to spans in your tracer logs.
  • If the Agent is installed on a separate machine, check that DD_AGENT_HOST and, optionally, DD_TRACE_AGENT_PORT are set, or that DD_TRACE_AGENT_URL is set for the application tracing library.

Check if spans are successfully transmitted to Datadog

ASM data is sent over spans. To confirm that spans are successfully transmitted to Datadog, check that your tracer logs contain logs that look similar to this:

2021-11-29 21:19:58 CET | TRACE | INFO | (pkg/trace/info/stats.go:111 in LogStats) | [lang:.NET lang_version:5.0.10 interpreter:.NET tracer_version:1.30.1.0 endpoint_version:v0.4] -> traces received: 2, traces filtered: 0, traces amount: 1230 bytes, events extracted: 0, events sampled: 0

If spans are not being transmitted, then the tracer logs will contain logs similar to this:

2021-11-29 21:18:48 CET | TRACE | INFO | (pkg/trace/info/stats.go:104 in LogStats) | No data received

Troubleshooting by language

Below are additional troubleshooting steps for specific languages.

If you don’t see ASM threat information in the Trace and Signals Explorer for your Python application, check that ASM is running and that your tracer is working.

  1. Set your application’s log level to DEBUG to confirm that ASM is running:

    import logging
    logging.basicConfig(level=logging.DEBUG)
    

    Then, run any HTTP call to the application. You should see the following log:

    DEBUG:ddtrace.appsec.processor:[DDAS-001-00] Executing AppSec In-App WAF with parameters:
    

    If this log is not present, ASM is not running.

  2. Is the tracer working? Can you see relevant traces on the APM dashboard?

    ASM relies on the tracer. If you don’t see traces, then the tracer might not be working. See APM Troubleshooting.

No vulnerabilities detected by Software Composition Analysis

There are a series of steps that must run successfully for vulnerability information to appear either in the Software Catalog Security View or in the Vulnerability Explorer. It is important to check each step when investigating this issue.

Confirm ASM is enabled

You can use the metric datadog.apm.appsec_host to check if ASM is running.

  1. Go to Metrics > Summary in Datadog.
  2. Search for the metric datadog.apm.appsec_host. If the metric doesn’t exist, then there are no services running ASM. If the metric exists, the services are reported with the metric tags host and service.
  3. Select the metric, and in the Tags section, search for service to see which services are running ASM.

If you are not seeing datadog.apm.appsec_host, check the in-app instructions to confirm that all steps for the initial setup are complete.

ASM data is sent with APM traces. See APM troubleshooting to confirm APM setup and check for connection errors.

Confirm tracer versions are updated

See the Application Security product set up documentation to validate you you are using the right version of the tracer. These minimum versions are required to start sending telemetry data that includes library information.

Ensure the communication of telemetry data

Ensure the DD_INSTRUMENTATION_TELEMETRY_ENABLED environment variable (DD_TRACE_TELEMETRY_ENABLED for Node.js) is set to true, or the corresponding system property for your language is enabled. For example in Java: -Ddd.instrumentation.telemetry.enabled=true

Disabling threat management and protection

To disable threat management, remove the DD_APPSEC_ENABLED=true environment variable from your application configuration, and restart your service.

If no DD_APPSEC_ENABLED=true environment variable is set for your service, do one of the following:

  • If it’s a PHP service: explicitly set the environment variable to DD_APPSEC_ENABLED=false, and restart your service.
  • If threat management was activated using Remote Configuration, do the following:
    1. Go to Services (ASM > Catalog > Services).
    2. Select Threat Management in Monitoring Mode.
    3. In the Threat Management facet, enable Monitoring Only, No data, and Ready to block.
    4. Click on a service.
    5. In the service details, in Threat Detection, click Deactivate.
If threat management was activated using Remote Configuration, you can use a Deactivate button. If threat management was activated using local configuration, the Deactivate button is not an option.
  • To disable threat management on your services in bulk, do the following:
    1. Go to Services.
    2. In the Threat Management facet, enable Monitoring Only, No data, and Ready to block.
    3. Select the check boxes for the services where you want to disable threat detection.
    4. In Bulk Actions, select Deactivate Threat detection on (number of) services.

Disabling Code Security

To disable Code Security, remove the DD_IAST_ENABLED=true environment variable from your application configuration or set it to false as DD_IAST_ENABLED=false, and restart your service.

Need more help?

If you continue to have issues with ASM, contact Datadog support with the following information:

Further Reading

PREVIEWING: hannahkm/clarify-v2-docs