- 필수 기능
- 시작하기
- Glossary
- 표준 속성
- Guides
- Agent
- 통합
- 개방형텔레메트리
- 개발자
- API
- Datadog Mobile App
- CoScreen
- Cloudcraft
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- APM
- Continuous Profiler
- 스팬 시각화
- 데이터 스트림 모니터링
- 데이터 작업 모니터링
- 디지털 경험
- 소프트웨어 제공
- 보안
- AI Observability
- 로그 관리
- 관리
Data Jobs Monitoring gives visibility into the performance and reliability of Apache Spark applications on Google Cloud Dataproc.
This guide is for Dataproc clusters on Compute Engine. If you are using Dataproc on GKE, refer to the Kubernetes Installation Guide instead.
Dataproc Release 2.0.x or later is required. All of Debian, Rocky Linux, and Ubuntu Dataproc standard images are supported.
Follow these steps to enable Data Jobs Monitoring for GCP Dataproc.
dd_api_key
.Compute Engine default service account
. To grant access, copy the associated service account Principal, and click Grant Access on the Permissions tab of the secret’s page. Assign the secretmanager.secretAccessor
role, or any other one that has secretmanager.versions.access
permission. See the IAM roles documentation for a full description of available roles.When you create a new Dataproc Cluster on Compute Engine in the Google Cloud Console, add an initialization action on the Customize cluster page:
Save the following script to a GCS bucket that your Dataproc cluster can read. Take note of the path to this script.
#!/bin/bash
# Set required parameter DD_SITE
DD_SITE=
# Set required parameter DD_API_KEY with Datadog API key.
# The commands below assumes the API key is stored in GCP Secret Manager, with the secret name as dd_api_key and the project <PROJECT_NAME>.
# IMPORTANT: Modify if you choose to manage and retrieve your secret differently.
# Change the project name, which you can find on the secrets page. The resource ID is in the format "projects/<PROJECT_NAME>/secrets/<SECRET_NAME>".
PROJECT_NAME=<PROJECT_NAME>
gcloud config set project $PROJECT_NAME
SECRET_NAME=dd_api_key
DD_API_KEY=$(gcloud secrets versions access latest --secret $SECRET_NAME)
# Optional parameters
# Uncomment the following line to allow adding init script logs when reporting a failure back to Datadog. A failure is reported when the init script fails to start the Datadog Agent successfully.
# export DD_DJM_ADD_LOGS_TO_FAILURE_REPORT=true
# Download and run the latest init script
DD_SITE=$DD_SITE DD_API_KEY=$DD_API_KEY bash -c "$(curl -L https://dd-data-jobs-monitoring-setup.s3.amazonaws.com/scripts/dataproc/dataproc_init_latest.sh)" || true
The script above sets the required parameters, and downloads and runs the latest init script for Data Jobs Monitoring in Dataproc. If you want to pin your script to a specific version, you can replace the file name in the URL with dataproc_init_<version_tag>.sh
, such as dataproc_init_1.5.0.sh
to use the specific version you want.
On the Customize cluster page, locate the Initialization Actions section. Enter the path where you saved the script from the previous step.
When your cluster is created, this initialization action installs the Datadog Agent and downloads the Java tracer on each node of the cluster.
Tagging enables you to better filter, aggregate, and compare your telemetry in Datadog. You can configure tags by passing -Ddd.service
, -Ddd.env
, -Ddd.version
, and -Ddd.tags
options to your Spark driver and executor extraJavaOptions
properties.
In Datadog, each job’s name corresponds to the value you set for -Ddd.service
.
spark-submit \
--conf spark.driver.extraJavaOptions="-Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION> -Ddd.tags=<KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>" \
--conf spark.executor.extraJavaOptions="-Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION> -Ddd.tags=<KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>" \
application.jar
In Datadog, view the Data Jobs Monitoring page to see a list of all your data processing jobs.
런타임의 Spark 스팬(span)에서 태그를 설정할 수 있습니다. 본 태그는 태그가 추가된 후 실행되는 스팬(span)에만 적용됩니다.
// 다음 모든 Spark 컴퓨팅에 태그 추가
sparkContext.setLocalProperty("spark.datadog.tags.key", "value")
spark.read.parquet(...)
런타임 태그를 제거하려면 다음을 수행합니다.
// 다음 모든 Spark 컴퓨팅에서 태그 제거
sparkContext.setLocalProperty("spark.datadog.tags.key", null)
추가 유용한 문서, 링크 및 기사: