Connect RUM and Traces

RUM and Traces

Overview

The APM integration with Real User Monitoring allows you to link requests from your web and mobile applications to their corresponding backend traces. This combination enables you to see your full frontend and backend data through one lens.

Use frontend data from RUM, as well as backend, infrastructure, and log information from trace ID injection to pinpoint issues anywhere in your stack and understand what your users are experiencing.

To start sending just your iOS application’s traces to Datadog, see iOS Trace Collection.

Usage

Prerequisites

  • You have set up APM tracing on the services targeted by your RUM applications.
  • Your services use an HTTP server.
  • Your HTTP servers are using a library that supports distributed tracing.
  • You have the following set up based on your SDK:
    • With the Browser SDK, you have added the XMLHttpRequest (XHR) or Fetch resources on the RUM Explorer to your allowedTracingUrls.
    • With the Mobile SDK, you have added the Native or XMLHttpRequest (XHR) to your firstPartyHosts.
  • You have a corresponding trace for requests to allowedTracingUrls or firstPartyHosts.

Setup RUM

Note: Configuring RUM and Traces makes use of APM paid data in RUM, which may impact your APM billing.

  1. Set up RUM Browser Monitoring.

  2. Initialize the RUM SDK. Configure the allowedTracingUrls initialization parameter with the list of internal, first-party origins called by your browser application.

    For npm install:

    import { datadogRum } from '@datadog/browser-rum'
    
    datadogRum.init({
      clientToken: '<CLIENT_TOKEN>',
      applicationId: '<APPLICATION_ID>',
      site: 'datadoghq.com',
      //  service: 'my-web-application',
      //  env: 'production',
      //  version: '1.0.0',
      allowedTracingUrls: ["https://api.example.com", /https:\/\/.*\.my-api-domain\.com/, (url) => url.startsWith("https://api.example.com")]
      sessionSampleRate: 100,
      sessionReplaySampleRate: 100, // if not specified, defaults to 100
      trackResources: true,
      trackLongTasks: true,
      trackUserInteractions: true,
    })
    

    For CDN install:

    window.DD_RUM.init({
       clientToken: '<CLIENT_TOKEN>',
       applicationId: '<APPLICATION_ID>',
       site: 'datadoghq.com',
       //  service: 'my-web-application',
       //  env: 'production',
       //  version: '1.0.0',
       allowedTracingUrls: ["https://api.example.com", /https:\/\/.*\.my-api-domain\.com/, (url) => url.startsWith("https://api.example.com")]
       sessionSampleRate: 100,
       sessionReplaySampleRate: 100, // if not included, the default is 100
       trackResources: true,
       trackLongTasks: true,
       trackUserInteractions: true,
     })
    

    To connect RUM to Traces, you need to specify your browser application in the service field.

    allowedTracingUrls matches the full URL (<scheme>://<host>[:<port>]/<path>[?<query>][#<fragment>]). It accepts the following types:

    • string: matches any URL that starts with the value, so https://api.example.com matches https://api.example.com/v1/resource.
    • RegExp: executes a test with the provided RegExp and the URL.
    • function: evaluates with the URL as parameter. Returning a boolean set to true indicates a match.
  3. (Optional) Configure the traceSampleRate initialization parameter to keep a defined percentage of the backend traces. If not set, 100% of the traces coming from browser requests are sent to Datadog. To keep 20% of backend traces, for example:

    import { datadogRum } from '@datadog/browser-rum'
    
    datadogRum.init({
        ...otherConfig,
        traceSampleRate: 20
    })
    

Note: traceSampleRate does not impact RUM sessions sampling. Only backend traces are sampled out.

  1. (Optional) If you set a traceSampleRate, to ensure backend services’ sampling decisions are still applied, configure the traceContextInjection initialization parameter to sampled (set to all by default).

    For example, if you set the traceSampleRate to 20% in the Browser SDK:

    • When traceContextInjection is set to all, 20% of backend traces are kept and 80% of backend traces are dropped.
    traceContextInjection set to all
  • When traceContextInjection is set to sampled, 20% of backend traces are kept. For the remaining 80%, the browser SDK does not inject a sampling decision. The decision is made on the server side and is based on the tracing library head-based sampling configuration. In the example below, the backend sample rate is set to 40%, and therefore 32% of the remaining backend traces are kept.

    traceContextInjection set to sampled
End-to-end tracing is available for requests fired after the Browser SDK is initialized. End-to-end tracing of the initial HTML document and early browser requests is not supported.
  1. Set up RUM Android Monitoring.

  2. Set up Android Trace Collection.

  3. Add the Gradle dependency to the dd-sdk-android-okhttp library in the module-level build.gradle file:

    dependencies {
        implementation "com.datadoghq:dd-sdk-android-okhttp:x.x.x"
    }
    
  4. Configure the OkHttpClient interceptor with the list of internal, first-party origins called by your Android application.

    val tracedHosts = listOf("example.com", "example.eu")
    
    val okHttpClient = OkHttpClient.Builder()
        .addInterceptor(DatadogInterceptor(tracedHosts))
        .addNetworkInterceptor(TracingInterceptor(tracedHosts))
        .eventListenerFactory(DatadogEventListener.Factory())
        .build()
    

    By default, all subdomains of listed hosts are traced. For instance, if you add example.com, you also enable the tracing for api.example.com and foo.example.com.

  5. (Optional) Configure the traceSampler parameter to keep a defined percentage of the backend traces. If not set, 20% of the traces coming from application requests are sent to Datadog. To keep 100% of backend traces:

    val okHttpClient = OkHttpClient.Builder()
       .addInterceptor(DatadogInterceptor(traceSampler = RateBasedSampler(100f)))
       .build()

Note:

  • traceSampler does not impact RUM sessions sampling. Only backend traces are sampled out.
  • If you define custom tracing header types in the Datadog configuration and are using a tracer registered with GlobalTracer, make sure the same tracing header types are set for the tracer in use.
  1. Set up RUM iOS Monitoring.

  2. Enable RUM with the urlSessionTracking option and firstPartyHostsTracing parameter:

    RUM.enable(
        with: RUM.Configuration(
            applicationID: "<rum application id>",
            urlSessionTracking: .init(
                firstPartyHostsTracing: .trace(
                    hosts: [
                        "example.com",
                        "api.yourdomain.com"
                    ]
                )
            )
        )
    )
    
  3. Enable URLSession instrumentation for your SessionDelegate type, which conforms to URLSessionDataDelegate protocol:

    URLSessionInstrumentation.enable(
        with: .init(
            delegateClass: <YourSessionDelegate>.self
        )
    )
    
  4. Initialize URLSession as stated in Setup:

    let session =  URLSession(
        configuration: ...,
        delegate: <YourSessionDelegate>(),
        delegateQueue: ...
    )
    

    By default, all subdomains of listed hosts are traced. For instance, if you add example.com, you also enable tracing for api.example.com and foo.example.com.

    Trace ID injection works when you are providing a URLRequest to the URLSession. Distributed tracing does not work when you are using a URL object.

  5. (Optional) Set the sampleRate parameter to keep a defined percentage of the backend traces. If not set, 20% of the traces coming from application requests are sent to Datadog.

    To keep 100% of backend traces:

    RUM.enable(
        with: RUM.Configuration(
            applicationID: "<rum application id>",
            urlSessionTracking: .init(
                firstPartyHostsTracing: .trace(
                    hosts: [
                        "example.com",
                        "api.yourdomain.com"
                    ],
                    sampleRate: 100
                )
            )
        )
    )
    

Note: sampleRate does not impact RUM sessions sampling. Only backend traces are sampled out.

  1. Set up RUM React Native Monitoring.

  2. Set the firstPartyHosts initialization parameter to define the list of internal, first-party origins called by your React Native application:

    const config = new DatadogProviderConfiguration(
        // ...
    );
    config.firstPartyHosts = ["example.com", "api.yourdomain.com"];
    

    By default, all subdomains of listed hosts are traced. For instance, if you add example.com, you also enable tracing for api.example.com and foo.example.com.

  3. (Optional) Set the resourceTracingSamplingRate initialization parameter to keep a defined percentage of the backend traces. If not set, 20% of the traces coming from application requests are sent to Datadog.

    To keep 100% of backend traces:

    const config = new DatadogProviderConfiguration(
        // ...
    );
    config.resourceTracingSamplingRate = 100;
    

    Note: resourceTracingSamplingRate does not impact RUM sessions sampling. Only backend traces are sampled out.

  1. Set up RUM Flutter Monitoring.

  2. Follow the instructions under Automatic Resource Tracking to include the Datadog Tracking HTTP Client package and enable HTTP tracking. This includes the following changes to your initialization to add a list of internal, first-party origins called by your Flutter application:

    final configuration = DatadogConfiguration(
      // ...
      // added configuration
      firstPartyHosts: ['example.com', 'api.yourdomain.com'],
    )..enableHttpTracking()
    

RUM for Roku is not available on the US1-FED Datadog site.

  1. Set up RUM Roku Monitoring.

  2. Use the datadogroku_DdUrlTransfer component to perform your network requests.

        ddUrlTransfer = datadogroku_DdUrlTransfer(m.global.datadogRumAgent)
        ddUrlTransfer.SetUrl(url)
        ddUrlTransfer.EnablePeerVerification(false)
        ddUrlTransfer.EnableHostVerification(false)
        result = ddUrlTransfer.GetToString()
    

Verifying setup

To verify you’ve configured the APM integration with RUM, follow the steps below based on the SDK you installed RUM with.

  1. Visit a page in your application.
  2. In your browser’s developer tools, go to the Network tab.
  3. Check the request headers for a resource request that you expect to be correlated contains the correlation headers from Datadog.
  1. Run your application from Android Studio.
  2. Visit a screen in your application.
  3. Open Android Studio’s Network Inspector.
  4. Check the request headers for a RUM resource and verify that the required headers are set by the SDK.
  1. Run your application from Xcode.
  2. Visit a screen in your application.
  3. Open Xcode’s Network Connections and HTTP Traffic instrument.
  4. Check the request headers for a RUM resource and verify that the required headers are set by the SDK.
  1. Run your application from Xcode (iOS) or Android Studio (Android).
  2. Visit a screen in your application.
  3. Open Xcode’s Network Connections and HTTP Traffic instrument or Android Studio’s Network Inspector.
  4. Check the request headers for a RUM resource and verify that the required headers are set by the SDK.
  1. Run your application using your preferred IDE or flutter run.
  2. Visit a screen in your application.
  3. Open Flutter’s Dev Tools and navigate to Network View.
  4. Check the request headers for a RUM resource and verify that the required headers are set by the SDK.

Supported libraries

Below is a list of the supported backend libraries that need to be on the services receiving the network requests.

OpenTelemetry support

RUM supports several propagator types to connect resources with backends that are instrumented with OpenTelemetry libraries.

The default injection style is tracecontext, Datadog.

Note: If you are using a backend framework such as Next.js/Vercel that uses OpenTelemetry, follow these steps.

  1. Set up RUM to connect with APM as described above.

  2. Modify allowedTracingUrls as follows:

    import { datadogRum } from '@datadog/browser-rum'
    
    datadogRum.init({
        ...otherConfig,
        allowedTracingUrls: [
          { match: "https://api.example.com", propagatorTypes: ["tracecontext"]}
        ]
    })
    

    match accepts the same parameter types (string, RegExp or function) as when used in its simple form, described above.

    propagatorTypes accepts a list of strings for desired propagators:

  1. Set up RUM to connect with APM as described above.

  2. Use .traceWithHeaders(hostsWithHeaders:sampleRate:) instead of .trace(hosts:sampleRate:) as follows:

      RUM.enable(
          with: RUM.Configuration(
              applicationID: "<rum application id>",
              urlSessionTracking: .init(
                  firstPartyHostsTracing: .traceWithHeaders(
                      hostsWithHeaders: [
                          "api.example.com": [.tracecontext]
                      ],
                      sampleRate: 100
                  )
              )
          )
      )
    

    .traceWithHeaders(hostsWithHeaders:sampleRate:) takes Dictionary<String, Set<TracingHeaderType>> as a parameter, where the key is a host and the value is a list of supported tracing header types.

    TracingHeaderType in an enum representing the following tracing header types:

  1. Set up RUM to connect with APM as described above.

  2. Configure the OkHttpClient interceptor with the list of internal, first-party origins and the tracing header type to use as follows:

    val tracedHosts = mapOf("example.com" to setOf(TracingHeaderType.TRACECONTEXT),
                          "example.eu" to setOf(TracingHeaderType.DATADOG))
    
    val okHttpClient = OkHttpClient.Builder()
        .addInterceptor(DatadogInterceptor(tracedHosts))
        .addNetworkInterceptor(TracingInterceptor(tracedHosts))
        .eventListenerFactory(DatadogEventListener.Factory())
        .build()
    

    TracingHeaderType is an enum representing the following tracing header types:

  1. Set up RUM to connect with APM.

  2. Configure the RUM SDK with the list of internal, first-party origins and the tracing header type to use as follows:

    const config = new DatadogProviderConfiguration(
        // ...
    );
    config.firstPartyHosts = [{
        match: "example.com",
        propagatorTypes: [
            PropagatorType.TRACECONTEXT,
            PropagatorType.DATADOG
        ]
    }];
    

    PropagatorType is an enum representing the following tracing header types:

  1. Set up RUM to connect with APM as described above.

  2. Use firstPartyHostsWithTracingHeaders instead of firstPartyHosts as follows:

    final configuration = DatadogConfiguration(
      // ...
      // added configuration
      firstPartyHostsWithTracingHeaders: {
        'example.com': { TracingHeaderType.tracecontext },
      },
    )..enableHttpTracking()
    

    firstPartyHostsWithTracingHeaders takes Map<String, Set<TracingHeaderType>> as a parameter, where the key is a host and the value is a list of supported tracing header types.

    TracingHeaderType in an enum representing the following tracing header types:

How are RUM resources linked to traces?

Datadog uses the distributed tracing protocol and sets up the HTTP headers below. By default, both trace context and Datadog-specific headers are used.

x-datadog-trace-id
Generated from the Real User Monitoring SDK. Allows Datadog to link the trace with the RUM resource.
x-datadog-parent-id
Generated from the Real User Monitoring SDK. Allows Datadog to generate the first span from the trace.
x-datadog-origin: rum
To make sure the generated traces from Real User Monitoring don’t affect your APM Index Spans counts.
x-datadog-sampling-priority: 1
To make sure that the Agent keeps the trace.
traceparent: [version]-[trace id]-[parent id]-[trace flags]
version: The current specification assumes version is set to 00.
trace id: 128 bits trace ID, hexadecimal on 32 characters. The source trace ID is 64 bits to keep compatibility with APM.
parent id: 64 bits span ID, hexadecimal on 16 characters.
trace flags: Sampled (01) or not sampled (00)
Example:
traceparent: 00-00000000000000008448eb211c80319c-b7ad6b7169203331s-01
b3: [trace id]-[span id]-[sampled]
trace id: 64 bits trace ID, hexadecimal on 16 characters.
span id: 64 bits span ID, hexadecimal on 16 characters.
sampled: True (1) or False (0)
Example for b3 single header:
b3: 8448eb211c80319c-b7ad6b7169203331-1
Example for b3 multiple headers:
X-B3-TraceId: 8448eb211c80319c
X-B3-SpanId: b7ad6b7169203331
X-B3-Sampled: 1

These HTTP headers are not CORS-safelisted, so you need to configure Access-Control-Allow-Headers on your server handling requests that the SDK is set up to monitor. The server must also accept preflight requests (OPTIONS requests), which are made by the SDK prior to every request.

How are APM quotas affected?

Connecting RUM and traces may significantly increase the APM-ingested volumes. Use the initialization parameter traceSampleRate to keep a share of the backend traces starting from browser and mobile requests.

How long are traces retained?

These traces are available for 15 minutes in the Live Search explorer. To retain the traces for a longer period of time, create retention filters. Scope these retention filters on any span tag to retain traces for critical pages and user actions.

Further Reading

PREVIEWING: may/unit-testing