Missing profiles in the profile search page

If you’ve configured the profiler and don’t see profiles in the profile search page, turn on debug mode and open a support ticket with debug files and the following information:

• Operating system type and version (for example, Linux Ubuntu 20.04)
• Runtime type, version, and vendor (for example, Java OpenJDK 11 AdoptOpenJDK)

Reduce overhead from default setup

If the default setup overhead is not acceptable, you can use the profiler with minimal configuration settings. Minimal configuration has the following changes compared to the default:

• Increases sampling threshold to 500ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
• Disables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the minimal configuration ensure you have dd-java-agent version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=minimal -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Increase profiler information granularity

If you want more granularity in your profiling data, you can specify the comprehensive configuration. Note that this approach will increase your profiler overhead at the cost of further granularity. Comprehensive configuration has the following changes compared to the default:

• Reduces sampling threshold to 10ms for ThreadSleep, ThreadPark, and JavaMonitorWait events compared to 100ms default
• Enables ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, ExceptionSample, ExceptionCount events

To use the comprehensive configuration ensure you have dd-trace-java version 0.70.0 then change your service invocation to the following:

java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.jfr-template-override-file=comprehensive -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Enabling the allocation profiler

On Java 15 and lower, the allocation profiler is turned off by default because it can overwhelm the profiler in allocation-heavy applications.

To enable the allocation profiler, start your application with the -Ddd.profiling.allocation.enabled=true JVM setting or the DD_PROFILING_ALLOCATION_ENABLED=true environment variable.

Alternatively, you can enable the following events in your jfp override template file:

jdk.ObjectAllocationInNewTLAB#enabled=true
jdk.ObjectAllocationOutsideTLAB#enabled=true

Learn how to use override templates.

Enabling the heap profiler

Alternatively, you can enable the following events in your jfp override template file:

jdk.OldObjectSample#enabled=true

Learn how to use override templates.

Enabling the heap histogram metrics

To enable the heap histogram metrics, start your application with the -Ddd.profiling.heap.histogram.enabled=true JVM setting or the DD_PROFILING_HEAP_HISTOGRAM_ENABLED=true environment variable.

Removing sensitive information from profiles

If your system properties contain sensitive information such as user names or passwords, turn off the system property event by creating a jfp override template file with jdk.InitialSystemProperty disabled:

jdk.InitialSystemProperty#enabled=false

Learn how to use override templates.

Large allocation events overwhelming the profiler

To turn off allocation profiling, disable the following events in your jfp override template file:

jdk.ObjectAllocationInNewTLAB#enabled=false
jdk.ObjectAllocationOutsideTLAB#enabled=false

Learn how to use override templates.

Memory leak detection slowing down garbage collector

jdk.OldObjectSample#enabled=false

# track only 10% of the allocation samples
java -javaagent:dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.ddprof.liveheap.enabled=true -Ddd.profiling.ddprof.liveheap.sample_percent=10 -jar <YOUR_SERVICE>.jar <YOUR_SERVICE_FLAGS>

Exceptions overwhelming the profiler

The Datadog exception profiler has a small footprint and overhead under normal conditions. If a lot of exceptions are created and thrown, it can cause significant overhead for the profiler. This can happen when you use exceptions for control flow. If you have an unusually high exception rate, turn off exception profiling temporarily until you fix the cause.

To disable exception profiling, start the tracer with the -Ddd.integration.throwables.enabled=false JVM setting.

Remember to turn this setting back on after you’ve returned to a more typical rate of exceptions.

Java 8 support

The following OpenJDK 8 vendors are supported for Continuous Profiling because they include JDK Flight Recorder in their latest versions:

If your vendor is not on the list, open a support ticket, as other vendors may be in development or available in Preview support.

Creating and using a JFR template override file

Override templates let you specify profiling properties to override. However, the default settings are balanced for a good tradeoff between overhead and data density that cover most use cases. To use an override file, perform the following steps:

1. Create an override file in a directory accessible by dd-java-agent at service invocation:

   touch dd-profiler-overrides.jfp
   

2. Add your desired overrides to the jfp file. For example, to disable allocation profiling and JVM system properties, your dd-profiler-overrides.jfp file would look like the following:

   jdk.ObjectAllocationInNewTLAB#enabled=false
   jdk.ObjectAllocationOutsideTLAB#enabled=false
   jdk.InitialSystemProperty#enabled=false
   

3. When running your application with dd-java-agent, your service invocation must point to the override file with -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp>, for example:

   java -javaagent:/path/to/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.profiling.jfr-template-override-file=</path/to/override.jfp> -jar path/to/your/app.jar
   

Further Reading

Additional helpful documentation, links, and articles:

APM TroubleshootingDOCUMENTATION