Google Cloud Run with serverless-init

Overview

If you are running multiple containers per Google Cloud Run application, Datadog recommends using the Datadog sidecar: Instrument Google Cloud Run.

Google Cloud Run is a fully managed serverless platform for deploying and scaling container-based applications. Datadog provides monitoring and log collection for Cloud Run through the Google Cloud integration. Datadog also provides a solution for instrumenting your Cloud Run applications with a purpose-built Agent to enable tracing, custom metrics, and direct log collection.

Prerequisites

Make sure you have a Datadog API key and are using a programming language supported by a Datadog tracing library.

Instrument your application

You can instrument your application in one of two ways: Dockerfile or buildpack.

Dockerfile

Datadog publishes new releases of the serverless-init container image to Google’s gcr.io, AWS’s ECR, and on Docker Hub:

dockerhub.iogcr.iopublic.ecr.aws
datadog/serverless-initgcr.io/datadoghq/serverless-initpublic.ecr.aws/datadog/serverless-init

Images are tagged based on semantic versioning, with each new version receiving three relevant tags:

  • 1, 1-alpine: use these to track the latest minor releases, without breaking changes
  • 1.x.x, 1.x.x-alpine: use these to pin to a precise version of the library
  • latest, latest-alpine: use these to follow the latest version release, which may include breaking changes

How serverless-init works

The serverless-init application wraps your process and executes it as a subprocess. It starts a DogStatsD listener for metrics and a Trace Agent listener for traces. It collects logs by wrapping the stdout/stderr streams of your application. After bootstrapping, serverless-init then launches your command as a subprocess.

To get full instrumentation, ensure you are calling datadog-init as the first command that runs inside your Docker container. You can do this through by setting it as the entrypoint, or by setting it as the first argument in CMD.

Add the following instructions and arguments to your Dockerfile.

COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
RUN pip install --target /dd_tracer/python/ ddtrace
ENV DD_SERVICE=datadog-demo-run-python
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1
ENTRYPOINT ["/app/datadog-init"]
CMD ["/dd_tracer/python/bin/ddtrace-run", "python", "app.py"]

Explanation

  1. Copy the Datadog serverless-init into your Docker image.

    COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
    
  2. Install the Datadog Python tracer.

    RUN pip install --target /dd_tracer/python/ ddtrace
    

    If you install the Datadog tracer library directly in your application, as outlined in the manual tracer instrumentation instructions, omit this step.

  3. (Optional) Add Datadog tags.

    ENV DD_SERVICE=datadog-demo-run-python
    ENV DD_ENV=datadog-demo
    ENV DD_VERSION=1
    
  4. Change the entrypoint to wrap your application in the Datadog serverless-init process. Note: If you already have an entrypoint defined inside your Dockerfile, see the alternative configuration.

    ENTRYPOINT ["/app/datadog-init"]
    
  5. Execute your binary application wrapped in the entrypoint, launched by the Datadog trace library. Adapt this line to your needs.

    CMD ["/dd_tracer/python/bin/ddtrace-run", "python", "app.py"]
    

Alternative configuration

If you already have an entrypoint defined inside your Dockerfile, you can instead modify the CMD argument.

COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
RUN pip install --target /dd_tracer/python/ ddtrace
ENV DD_SERVICE=datadog-demo-run-python
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1
CMD ["/app/datadog-init", "/dd_tracer/python/bin/ddtrace-run", "python", "app.py"]

If you require your entrypoint to be instrumented as well, you can swap your entrypoint and CMD arguments instead. For more information, see How serverless-init works.

COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
RUN pip install --target /dd_tracer/python/ ddtrace
ENV DD_SERVICE=datadog-demo-run-python
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1
ENTRYPOINT ["/app/datadog-init"]
CMD ["your_entrypoint.sh", "/dd_tracer/python/bin/ddtrace-run", "python", "app.py"]

As long as your command to run is passed as an argument to datadog-init, you will receive full instrumentation.

Buildpack

Pack Buildpacks provide a convenient way to package your container without using a Dockerfile.

First, manually install your tracer:

Then, build your application by running the following command:

pack build --builder=gcr.io/buildpacks/builder \
--buildpack from=builder \
--buildpack datadog/serverless-buildpack:latest \
gcr.io/<YOUR_PROJECT>/<YOUR_APP_NAME>

Note: Buildpack instrumentation is not compatible with Alpine images.

Configure your application

Once the container is built and pushed to your registry, the last step is to set the required environment variables for the Datadog Agent:

  • DD_API_KEY: Datadog API key, used to send data to your Datadog account. It should be configured as a Google Cloud Secret for privacy and safety.
  • DD_SITE: Datadog endpoint and website. Select your site on the right side of this page. Your site is: datadoghq.com.

For more environment variables and their function, see Environment Variables.

The following command deploys the service and allows any external connection to reach it. In this example, your service listening is set to port 8080. Ensure that this port number matches the exposed port inside of your Dockerfile.

shell
gcloud run deploy <APP_NAME> --image=gcr.io/<YOUR_PROJECT>/<APP_NAME> \
  --port=8080 \
  --update-env-vars=DD_API_KEY=$DD_API_KEY \
  --update-env-vars=DD_SITE=$DD_SITE \

See all arguments and flags for gcloud run deploy.

Results

Once the deployment is completed, your metrics and traces are sent to Datadog. In Datadog, navigate to Infrastructure > Serverless to see your serverless metrics and traces.

Additional configurations

  • Advanced Tracing: The Datadog Agent already provides some basic tracing for popular frameworks. Follow the advanced tracing guide for more information.

  • Logs: If you use the Google Cloud integration, your logs are already being collected. Alternatively, you can set the DD_LOGS_ENABLED environment variable to true to capture application logs through the serverless instrumentation directly.

  • Custom Metrics: You can submit custom metrics using a DogStatsD client. For monitoring Cloud Run and other serverless applications, use distribution metrics. Distributions provide avg, sum, max, min, and count aggregations by default. On the Metric Summary page, you can enable percentile aggregations (p50, p75, p90, p95, p99) and also manage tags. To monitor a distribution for a gauge metric type, use avg for both the time and space aggregations. To monitor a distribution for a count metric type, use sum for both the time and space aggregations.

Environment Variables

VariableDescription
DD_API_KEYDatadog API key - Required
DD_SITEDatadog site - Required
DD_LOGS_ENABLEDWhen true, send logs (stdout and stderr) to Datadog. Defaults to false.
DD_LOGS_INJECTIONWhen true, enrich all logs with trace data for supported loggers in Java, Node, .NET, and PHP. See additional docs for Python, Go, and Ruby.
DD_SERVICESee Unified Service Tagging.
DD_VERSIONSee Unified Service Tagging.
DD_ENVSee Unified Service Tagging.
DD_SOURCESee Unified Service Tagging.
DD_TAGSSee Unified Service Tagging.

Troubleshooting

This integration depends on your runtime having a full SSL implementation. If you are using a slim image, you may need to add the following command to your Dockerfile to include certificates.

RUN apt-get update && apt-get install -y ca-certificates

Further reading

Additional helpful documentation, links, and articles:

PREVIEWING: aaron.hoffman/SDCD-952_write-rollback-cd-docs