Enable Data Jobs Monitoring for Databricks

이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Data Jobs Monitoring gives visibility into the performance and reliability of your Apache Spark and Databricks jobs.

Setup

Follow these steps to enable Data Jobs Monitoring for Databricks.

  1. Configure the Datadog-Databricks integration with your Databricks API token.
  2. Install the Datadog Agent on your Databricks cluster(s).

Configure the Datadog-Databricks integration

  1. In your Databricks workspace, click on your profile in the top right corner and go to Settings. Select Developer in the left side bar. Next to Access tokens, click Manage.

  2. Click Generate new token, enter “Datadog Integration” in the Comment field, remove the default value in Lifetime (days), and click Generate. Take note of your token.

    Important:

    • Make sure you delete the default value in Lifetime (days) so that the token doesn’t expire and the integration doesn’t break.
    • Ensure the account generating the token has CAN VIEW access for the Databricks jobs and clusters you want to monitor.

    As an alternative, follow the official Databricks documentation to generate access token for a service principal.

  3. In Datadog, open the Databricks integration tile.

  4. On the Configure tab, click Add New.

  5. Enter a workspace name, your Databricks workspace URL, and the Databricks token you generated.

    In the Datadog-Databricks integration tile, a Databricks workspace is displayed. This workspace has a name, URL, and API token.

Install the Datadog Agent on your Databricks cluster(s)

You can choose to install the Agent globally, or on a specific Databricks cluster.

    1. In Databricks, click your display name (email address) in the upper right corner of the page.

    2. Select Settings and click the Compute tab.

    3. In the All purpose clusters section, next to Global init scripts, click Manage.

    4. Click Add. Name your script. Then, in the Script field, copy and paste the following script, remembering to replace the placeholders with your parameter values.

      #!/bin/bash

# Required parameters
export DD_API_KEY=<YOUR API KEY>
export DD_SITE=<YOUR DATADOG SITE>
export DATABRICKS_WORKSPACE="<YOUR WORKSPACE NAME>"

# Download and run the latest init script
bash -c "$(curl -L https://dd-data-jobs-monitoring-setup.s3.amazonaws.com/scripts/databricks/databricks_init_latest.sh)" || true

      The script above sets the required parameters, downloads and runs the latest init script for Data Jobs Monitoring in Databricks. If you want to pin your script to a specific version, you can replace the file name in the URL with databricks_init_1.5.1.sh to use the last stable version.

    5. To enable the script for all new and restarted clusters, toggle Enabled.

      Databricks UI, admin settings, global init scripts. A script called 'install-datadog-agent' is in a list with an enabled toggle.

    6. Click Add.

    Set the required init script parameters

    Provide the values for the init script parameters at the beginning of the global init script.

    export DD_API_KEY=<YOUR API KEY>
export DD_SITE=<YOUR DATADOG SITE>
export DATABRICKS_WORKSPACE="<YOUR WORKSPACE NAME>"

    Optionally, you can also set other init script parameters and Datadog environment variables here, such as DD_ENV and DD_SERVICE. The script can be configured using the following parameters:

    VariableDescriptionDefault
    DD_API_KEYYour Datadog API key.
    DD_SITEYour Datadog site.
    DATABRICKS_WORKSPACEName of your Databricks Workspace. It should match the name provided in the Datadog-Databricks integration step. Enclose the name in double quotes if it contains whitespace.
    DRIVER_LOGS_ENABLEDCollect spark driver logs in Datadog.false
    WORKER_LOGS_ENABLEDCollect spark workers logs in Datadog.false
    DD_DJM_ADD_LOGS_TO_FAILURE_REPORTInclude init script logs for debugging when reporting a failure back to Datadog.false

    1. In Databricks, create a init script file in Workspace with the following content. Be sure to make note of the file path.

      #!/bin/bash

# Download and run the latest init script
bash -c "$(curl -L https://dd-data-jobs-monitoring-setup.s3.amazonaws.com/scripts/databricks/databricks_init_latest.sh)" || true

      The script above downloads and runs the latest init script for Data Jobs Monitoring in Databricks. If you want to pin your script to a specific version, you can replace the file name in the URL with databricks_init_1.3.1.sh to use the last stable version.

    2. On the cluster configuration page, click the Advanced options toggle.

    3. At the bottom of the page, go to the Init Scripts tab.

      Databricks UI, cluster configuration advanced options, Init Scripts tab. A 'Destination' drop-down and an 'Init script path' file selector.
      - Under the **Destination** drop-down, select `Workspace`. - Under **Init script path**, enter the path to your init script. - Click **Add**.

    Set the required init script parameters

    1. In Databricks, on the cluster configuration page, click the Advanced options toggle.

    2. At the bottom of the page, go to the Spark tab.

      Databricks UI, cluster configuration advanced options, Spark tab. A textbox titled 'Environment variables' contains values for DD_API_KEY and DD_SITE.

      In the Environment variables textbox, provide the values for the init script parameters.

      DD_API_KEY=<YOUR API KEY>
DD_SITE=<YOUR DATADOG SITE>
DATABRICKS_WORKSPACE="<YOUR WORKSPACE NAME>"

      Optionally, you can also set other init script parameters and Datadog environment variables here, such as DD_ENV and DD_SERVICE. The script can be configured using the following parameters:

    VariableDescriptionDefault
    DD_API_KEYYour Datadog API key.
    DD_SITEYour Datadog site.
    DATABRICKS_WORKSPACEName of your Databricks Workspace. It should match the name provided in the Datadog-Databricks integration step. Enclose the name in double quotes if it contains whitespace.
    DRIVER_LOGS_ENABLEDCollect spark driver logs in Datadog.false
    WORKER_LOGS_ENABLEDCollect spark workers logs in Datadog.false
    DD_DJM_ADD_LOGS_TO_FAILURE_REPORTInclude init script logs for debugging when reporting a failure back to Datadog.false
    1. Click Confirm.

    Validation

    In Datadog, view the Data Jobs Monitoring page to see a list of all your Databricks jobs.

    Advanced Configuration

    Tag spans at runtime

    런타임의 Spark 스팬(span)에서 태그를 설정할 수 있습니다. 본 태그는 태그가 추가된 후 실행되는 스팬(span)에만 적용됩니다.

    // 다음 모든 Spark 컴퓨팅에 태그 추가
sparkContext.setLocalProperty("spark.datadog.tags.key", "value")
spark.read.parquet(...)

    런타임 태그를 제거하려면 다음을 수행합니다.

    // 다음 모든 Spark 컴퓨팅에서 태그 제거
sparkContext.setLocalProperty("spark.datadog.tags.key", null)

    Aggregate cluster metrics from one-time job runs

    This configuration is applicable if you want cluster resource utilization data about your jobs and create a new job and cluster for each run via the one-time run API endpoint (common when using orchestration tools outside of Databricks such as Airflow or Azure Data Factory).

    If you are submitting Databricks Jobs via the one-time run API endpoint, each job run will have a unique job ID. This can make it difficult to group and analyze cluster metrics for jobs that use ephemeral clusters. To aggregate cluster utilization from the same job and assess performance across multiple runs, you must set the DD_JOB_NAME variable inside the spark_env_vars of every new_cluster to the same value as your request payload’s run_name.

    Here’s an example of a one-time job run request body:

    {
   "run_name": "Example Job",
   "idempotency_token": "8f018174-4792-40d5-bcbc-3e6a527352c8",
   "tasks": [
      {
         "task_key": "Example Task",
         "description": "Description of task",
         "depends_on": [],
         "notebook_task": {
            "notebook_path": "/Path/to/example/task/notebook",
            "source": "WORKSPACE"
         },
         "new_cluster": {
            "num_workers": 1,
            "spark_version": "13.3.x-scala2.12",
            "node_type_id": "i3.xlarge",
            "spark_env_vars": {
               "DD_JOB_NAME": "Example Job"
            }
         }
      }
   ]
}

    Set up Data Jobs Monitoring with Databricks Networking Restrictions

    With Databricks Networking Restrictions, Datadog may not have access to your Databricks APIs, which is required to collect traces for Databricks job executions along with tags and other metadata.

    If you are controlling Databricks API access through IP access lists, allow-listing Datadog’s specific IP addresses allows your cluster to perform all these interactions with Datadog services. Please see Databricks documentation for more details on how to manage IP access lists in Databricks.

    If you are using Databricks Private Connectivity, the steps to configure the connection depend on your cloud provider.

    Refer to the guide for your cloud environment:


    For further assistance, contact the Datadog support team.

    Further Reading

    추가 유용한 문서, 링크 및 기사:

    Data Jobs MonitoringDOCUMENTATION more more
    PREVIEWING: rtrieu/product-analytics-ui-changes