Use Observability Pipelines’ HTTP/S Server source to collect HTTP client logs. Select and set up this source when you set up a pipeline.

You can also send AWS vended logs with Datadog Lambda Forwarder to Observability Pipelines.

Prerequisites

To use Observability Pipelines’ HTTP/S Server source, you need the following information available:

• The HTTP/S server address, such as 0.0.0.0:9997. The Observability Pipelines Worker listens to this socket address for your HTTP client logs.
• The HTTP authentication password.

Set up the source in the pipeline UI

Select and set up this source when you set up a pipeline. The information below is for the source settings in the pipeline UI.

To configure your HTTP/S Server source, enter the following:

1. Select your authorization strategy.
2. Select the decoder you want to use on the HTTP messages. Your HTTP client logs must be in this format. Note: If you select bytes decoding, the raw log is stored in the message field.
3. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
   Note: All file paths are made relative to the configuration data directory, which is /var/lib/observability-pipelines-worker/config/ by default. See Advanced Configurations for more information. The file must be owned by the observability-pipelines-worker group and observability-pipelines-worker user, or at least readable by the group or user.
   • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.

Send AWS vended logs with the Datadog Lambda Forwarder to Observability Pipelines

To send AWS vended logs to Observability Pipelines with the HTTP/S Server source:

• Set up a pipeline with the HTTP/S Server source.
• Deploy the Datadog Forwarder.

Note: This is available for Worker versions 2.51 or later.

Set up a pipeline

1. Navigate to Observability Pipelines.
2. Select the HTTP Server source:
   1. Select None in the dropdown menu for Authorization strategy.
   2. Select JSON in the dropdown menu for Decoding.
3. After you set up your destinations and processors, click Next: Install.
4. On the Install page, enter the HTTP/S Server listener address, such as 0.0.0.0:9997.
5. Follow the rest of the instructions on the page to install the Worker based on your platform.

Deploy the Datadog Lambda Forwarder

Follow the Datadog Forwarder CloudFormation installation instructions to deploy the Datadog Forwarder. After you have filled in DdApiKey and selected the appropriate DdSite, go to the Log Forwarding (Optional) section. Enter the following information in that section to set up the Forwarder to send logs to Observability Pipelines:

1. In the DdUrl field, enter your load balancer address. Datadog recommends fronting your Worker with a load balancer.
2. In the DdPort field, enter 80.
3. If you want to encrypt traffic from the Datadog Forwarder to your Observability Pipelines Worker load balancer, you need to create an HTTPS listener and manage a certificate on your load balancer. See Create an HTTPS listener for your Application Load Balancer for more information.
4. If you do not need to maintain encryption between the Datadog Forwarder and load balancer, select true in the DdNoSsl dropdown menu.
5. Click Create stack, and wait for the creation to complete.

You can find the installed forwarder Lambda function under the stack’s Resources tab with the logical ID Forwarder.

Set up triggers for the installed Datadog Forwarder.