Instrumenting Node.js Serverless Applications Using the Datadog Forwarder

Overview

If you are a new user of Datadog Serverless, follow the instructions to instrument your Lambda functions using the Datadog Lambda Extension instead. If you have setup Datadog Serverless with the Datadog Forwarder before Lambda offered out-of-the-box functionality, use this guide to maintain your instance.

Prerequisites

The Datadog Forwarder Lambda function is required to ingest AWS Lambda traces, enhanced metrics, custom metrics, and logs.

Configuration

The Datadog CLI modifies existing Lambda functions’ configurations to enable instrumentation without requiring a new deployment. It is the quickest way to get started with Datadog’s serverless monitoring.

You can also add the command to your CI/CD pipelines to enable instrumentation for all your serverless applications. Run the command after your normal serverless application deployment, so that changes made by the Datadog CLI command are not overridden.

Install

Install the Datadog CLI with NPM or Yarn:

# NPM
npm install -g @datadog/datadog-ci

# Yarn
yarn global add @datadog/datadog-ci

Instrument

To instrument the function, run the following command with your AWS credentials.

datadog-ci lambda instrument -f <functionname> -f <another_functionname> -r <aws_region> -v <layer_version> --forwarder	<forwarder_arn>

To fill in the placeholders:

  • Replace <functionname> and <another_functionname> with your Lambda function names.
  • Replace <aws_region> with the AWS region name.
  • Replace <layer_version> with the desired version of the Datadog Lambda Library. The latest version is 121.
  • Replace <forwarder_arn> with the Forwarder ARN (see the Forwarder documentation).

For example:

datadog-ci lambda instrument -f my-function -f another-function -r us-east-1 -v 121 --forwarder "arn:aws:lambda:us-east-1:000000000000:function:datadog-forwarder"

If your Lambda function is configured to use code signing, you must add Datadog’s Signing Profile ARN (arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc) to your function’s Code Signing Configuration before you can instrument it with the Datadog CLI.

More information and additional parameters can be found in the CLI documentation.

The Datadog Serverless Plugin automatically adds the Datadog Lambda library to your functions using layers, and configures your functions to send metrics, traces, and logs to Datadog through the Datadog Forwarder.

If your Lambda function is configured to use code signing, you must add Datadog’s Signing Profile ARN (arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc) to your function’s Code Signing Configuration before you install the Datadog Serverless Plugin.

To install and configure the Datadog Serverless Plugin, follow these steps:

  1. Install the Datadog Serverless Plugin:
    yarn add --dev serverless-plugin-datadog
  2. In your serverless.yml, add the following:
    plugins:
  - serverless-plugin-datadog
  3. In your serverless.yml, also add the following section:
    custom:
  datadog:
    forwarderArn: # The Datadog Forwarder ARN goes here.
    More information on the Datadog Forwarder ARN or installation can be found here. For additional settings, see the plugin documentation.

Note: You need to follow these additional configuration steps if your Lambda function is simultaneously using Datadog’s tracing libraries and webpack.

The Datadog CloudFormation macro automatically transforms your SAM application template to add the Datadog Lambda library to your functions using layers, and configure your functions to send metrics, traces, and logs to Datadog through the Datadog Forwarder.

Install

Run the following command with your AWS credentials to deploy a CloudFormation stack that installs the macro AWS resource. You only need to install the macro once for a given region in your account. Replace create-stack with update-stack to update the macro to the latest version.

aws cloudformation create-stack \
  --stack-name datadog-serverless-macro \
  --template-url https://datadog-cloudformation-template.s3.amazonaws.com/aws/serverless-macro/latest.yml \
  --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM

The macro is now deployed and ready to use.

Instrument

In your template.yml, add the following under the Transform section, after the AWS::Serverless transform for SAM.

Transform:
  - AWS::Serverless-2016-10-31
  - Name: DatadogServerless
    Parameters:
      stackName: !Ref "AWS::StackName"
      nodeLayerVersion: "121"
      forwarderArn: "<FORWARDER_ARN>"
      service: "<SERVICE>" # Optional
      env: "<ENV>" # Optional

To fill in the placeholders:

  • Replace <FORWARDER_ARN> with Forwarder ARN (see the Forwarder documentation).
  • Replace <SERVICE> and <ENV> with your service and environment values.

If your Lambda function is configured to use code signing, you must add Datadog’s Signing Profile ARN (arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc) to your function’s Code Signing Configuration before you can use the macro.

More information and additional parameters can be found in the macro documentation.

The Datadog CDK Constructs automatically configure ingestion of metrics, traces, and logs from your serverless applications by:

  • Installing and configuring the Datadog Lambda library for your Python and Node.js Lambda functions.
  • Enabling the collection of traces and custom metrics from your Lambda functions.
  • Managing subscriptions from the Datadog Forwarder to your Lambda function log groups.

Install

Run the following Yarn or NPM command in your CDK project to install the Datadog CDK Constructs library:

#Yarn
yarn add --dev datadog-cdk-constructs-v2

#NPM
npm install datadog-cdk-constructs-v2 --save-dev

Instrument

To instrument the function, import the datadog-cdk-construct module in your AWS CDK app and add the following configurations (this example is TypeScript, but usage in other languages is similar):

import * as cdk from "@aws-cdk/core";
import { DatadogLambda } from "datadog-cdk-constructs-v2";

class CdkStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    const datadogLambda = new DatadogLambda(this, "DatadogLambda", {
      nodeLayerVersion: 121,
      forwarderArn: "<FORWARDER_ARN>",
      service: "<SERVICE>",  // Optional
      env: "<ENV>",  // Optional
    });
    datadog.addLambdaFunctions([<LAMBDA_FUNCTIONS>])
  }
}

To fill in the placeholders:

  • Replace <FORWARDER_ARN> with Forwarder ARN (see the Forwarder documentation).
  • Replace <SERVICE> and <ENV> with your service and environment values.

If your Lambda function is configured to use code signing, you must add Datadog’s Signing Profile ARN (arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc) to your function’s Code Signing Configuration before you can use the macro.

More information and additional parameters can be found in the Datadog CDK NPM page.

Install

If you are deploying your Lambda function as a container image, you cannot use the Datadog Lambda Library as a layer. Instead, you must install the Datadog Lambda Library as a dependency of your function within the image. If you are using Datadog tracing, you must also install dd-trace.

NPM:

npm install --save datadog-lambda-js dd-trace

Yarn:

yarn add datadog-lambda-js dd-trace

Note: The minor version of the datadog-lambda-js package always matches the layer version. For example, datadog-lambda-js v0.5.0 matches the content of layer version 5.

Configure

Follow these steps to configure the function:

  1. Set your image’s CMD value to node_modules/datadog-lambda-js/dist/handler.handler. You can set this in AWS or directly in your Dockerfile. Note: The value set in AWS overrides the value in the Dockerfile if you set both.
  2. Set the following environment variables in AWS:
  • Set DD_LAMBDA_HANDLER to your original handler, for example, myfunc.handler.
  • Set DD_TRACE_ENABLED to true.
  • Set DD_FLUSH_TO_LOG to true.
  1. Optionally add service and env tags with appropriate values to your function.

Subscribe

Subscribe the Datadog Forwarder Lambda function to each of your functions’ log groups in order to send metrics, traces, and logs to Datadog.

  1. Install the Datadog Forwarder if you haven’t.
  2. Subscribe the Datadog Forwarder to your function’s log groups.

Install

The Datadog Lambda Library can be imported as a layer or JavaScript package.

The minor version of the datadog-lambda-js package always matches the layer version. For example, datadog-lambda-js v0.5.0 matches the content of layer version 5.

Using the layer

Configure the layers for your Lambda function using the ARN in the following format.

# For us,us3,us5,eu, and ap1 regions
arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:<VERSION>

# For us-gov regions
arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-<RUNTIME>:<VERSION>

The available RUNTIME options are: Node18-x, Node20-x, Node22-x. The latest VERSION is 121. For example:

arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Node22-x:121

If your Lambda function is configured to use code signing, you must add Datadog’s Signing Profile ARN (arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc) to your function’s Code Signing Configuration before you can add the Datadog Lambda library as a layer.

Using the package

NPM:

npm install --save datadog-lambda-js

Yarn:

yarn add datadog-lambda-js

See the latest release.

Configure

Follow these steps to configure the function:

  1. Set your function’s handler to /opt/nodejs/node_modules/datadog-lambda-js/handler.handler if using the layer, or node_modules/datadog-lambda-js/dist/handler.handler if using the package.
  2. Set the environment variable DD_LAMBDA_HANDLER to your original handler, for example, myfunc.handler.
  3. Set the environment variable DD_TRACE_ENABLED to true.
  4. Set the environment variable DD_FLUSH_TO_LOG to true.
  5. Optionally add a service and env tag with appropriate values to your function.

Note: You need to follow these additional configuration steps if your Lambda function is simultaneously using Datadog’s tracing libraries and webpack.

Subscribe

Subscribe the Datadog Forwarder Lambda function to each of your function’s log groups, in order to send metrics, traces, and logs to Datadog.

  1. Install the Datadog Forwarder if you haven’t.
  2. Subscribe the Datadog Forwarder to your function’s log groups.

Tag

Although it’s optional, Datadog recommends tagging you serverless applications with the env, service, and version tags following the unified service tagging documentation.

Explore

After configuring your function following the steps above, view your metrics, logs, and traces on the Serverless homepage.

Monitor custom business logic

If you would like to submit a custom metric or span, see the sample code below:

const { sendDistributionMetric, sendDistributionMetricWithDate } = require("datadog-lambda-js");
const tracer = require("dd-trace");

// submit a custom span named "sleep"
const sleep = tracer.wrap("sleep", (ms) => {
  return new Promise((resolve) => setTimeout(resolve, ms));
});

exports.handler = async (event) => {
  // add custom tags to the lambda function span,
  // does NOT work when X-Ray tracing is enabled
  const span = tracer.scope().active();
  span.setTag('customer_id', '123456');

  await sleep(100);

  // submit a custom span
  const sandwich = tracer.trace('hello.world', () => {
    console.log('Hello, World!');
  });

  // submit a custom metric
  sendDistributionMetric(
    "coffee_house.order_value", // metric name
    12.45, // metric value
    "product:latte", // tag
    "order:online", // another tag
  );

  // submit a custom metric with timestamp
  sendDistributionMetricWithDate(
    "coffee_house.order_value", // metric name
    12.45, // metric value
    new Date(Date.now()), // date, must be within last 20 mins
    "product:latte", // tag
    "order:online", // another tag
  );

  const response = {
    statusCode: 200,
    body: JSON.stringify("Hello from serverless!"),
  };
  return response;
};

For more information on custom metric submission, see Serverless Custom Metrics. For additional details on custom instrumentation, see the Datadog APM documentation for custom instrumentation.

Further Reading

Additional helpful documentation, links, and articles:

PREVIEWING: brett.blue/reorg-otel