Get Started with the Custom Processor

Overview

Observability Pipelines enables you to transform your logs before sending them to your destinations. Use the Custom Processor to create scripts with custom functions that conditionally modify log fields, values, and events.

This guide walks you through how to use the following functions in your Custom Processor script:

It also goes over example scripts that address common use cases, such as:

Decode Base64

For incoming log fields or events encoded in Base64, use the decode_base64 function to decode the field or event. This function’s syntax also works for decode_base16.

Example

Input

Example log event containing a Base64 field to decode:

{
    "timestamp": "2025-05-28T19:30:00Z",
    "level": "info",
    "source": "auth-service",
    "user_id": "user_9876",
    "payload": "VXNlciByZXF1ZXN0ZWQgYWNjZXNzIHRvIHByb3RlY3RlZCByZXNvdXJjZQ=="
}

Custom function

Use the decode_base64 function to decode payload and store the result in a new field called decoded_payload.

.decoded_payload = decode_base64!(.payload)

Alternatively, you can rewrite the original payload value with the decoded value by replacing decoded_payload in the previous function with payload.

.payload = decode_base64!(.payload)

Output

The output when you use decoded_payload to store the decoded value.

{
    "timestamp": "2025-05-28T19:30:00Z",
    "level": "info",
    "source": "auth-service",
    "user_id": "user_9876",
    "payload": "VXNlciByZXF1ZXN0ZWQgYWNjZXNzIHRvIHByb3RlY3RlZCByZXNvdXJjZQ==",
    "decoded_payload": "User requested access to protected resource"
}

Decode an entire Base64-encoded event

Example

Input

Example input of an event encoded in Base64:

{
    "raw": "eyJ0aW1lc3RhbXAiOiAiMjAyNS0wNS0yOFQxOTozMDowMFoiLCAibGV2ZWwiOiAiaW5mbyIsICJtessagemIjogIlVzZXIgbG9naW4gc3VjY2Vzc2Z1bCJ9"
}

Custom function

The script to decode the entire Base64-encoded event raw.

.json_string = decode_base64!(.raw)`
.full_event = parse_json!(.json_string)
. = .full_event

Note: The syntax . = .full_event is shorthand for replacing the entire event with the contents of a field.

Output

{
    "timestamp": "2025-05-28T19:30:00Z",
    "level": "info",
    "message": "User login successful"
}

Encode Base64

For outgoing log fields or events that you want to encode in Base64, use the encode_base64 function to encode the field or event. This function’s syntax also works for encode_base16.

Example

Input

Example log event containing the message field that you want to encode in Base64:

{
    "timestamp": "2025-05-28T19:30:00Z",
    "level": "info",
    "source": "auth-service",
    "user_id": "user_9876",
    "message": "User login successful"
}

Custom function

Use the encode_base64 function to decode message and store the result in a new field called encoded_message.

.encoded_message = encode_base64!(.message)

Alternatively, you can overwrite the original message field (message) with the decoded value by replacing encoded_message in the previous function with message.

.message = encode_base64!(.message)

Output

The output when you use encoded_message to store the encoded value.

{
    "timestamp": "2025-05-28T19:30:00Z",
    "level": "info",
    "source": "auth-service",
    "message": "User login successful",
    "encoded_message": "VXNlciBsb2dpbiBzdWNjZXNzZnVs"
}

Remap timestamps for historical logs

If you want to migrate archived logs from other platforms, ensuring those logs have the correct historical timestamp is essential. Remapping logs with historical timestamps enables you to handle older logs stored for compliance, audit, and archive purposes.

Example

Input

If the Worker does not find the timestamp field on a log, the timestamp of when the Worker received the log is used. This is an example of a log showing the timestamp of when the Worker received the log, as well as the log’s historical timestamp (historical_ts), which is the value that the Worker is looking for.

{
    "timestamp": "2025-05-28T19:30:00Z",
    "historical_ts": "2019-03-14T17:30:00Z",
    "level": "info",
    "message": "User login successful"
}

Custom function

For the above example, you can create a function to store the ingested timestamp in a new field and remap timestamp to the historical_ts value.

#Create a new field for the ingested/processed timestamp
.ingested_ts = {{.timestamp}}

#Remap timestamp to be the historical field
.timestamp = {{.historical_ts}}

#Remove the original historical timestamp
del(.historical_ts)

Output

{
    "timestamp": "2019-03-14T17:30:00Z",
    "ingested_ts": "2025-05-28T19:30:00Z",
    "level": "info",
    "message": "User login successful"
}

Extract a field from the Datadog tags array

Fields nested within the Datadog tags (ddtags) array can contain useful information. You may want to extract these fields as top level key-value pairs, or as values for other fields.

Example

Input

Sample log containing the ddtags array with Datadog tags.

{
    "timestamp": "2025-005-27T05:26:18.205Z",
    "status": "info",
    "service": "chaos-engineering",
    "ddsource": "python",
    "hostname": "gke-prod-node-abc123.internal",
    "message": "2025-05-27 05:26:17,609 -- Sending request to rails: checkout_v2",
    "source_type": "datadog_agent",
    "ddtags": [
        "env:prod",
        "team:sre",
        "service:chaos-engineering",
        "version:1.0.0",
        "pod_name:load-generator-main-abcde"
    ]
}

Custom function to extract the env field

#Extract a tag from ddtags array and elevate as log attribute
.my_tag, err = filter(array!(.ddtags)) -> |_index, value| {
    #Keep any elements that have the key name "env"
    starts_with(value, "env:")
}
#Assign env to be value of the key
.env = split!(.my_tag[0], ":")[1]
del(.my_tag)

Output

{
   "ddsource": "python",
   "ddtags": [
       "env:prod",
       "team:sre",
       "service:chaos-engineering",
       "version:1.0.0",
       "pod_name:load-generator-main-abcde"
   ],
   "env": "prod",
   "hostname": "gke-prod-node-abc123.internal",
   "message": "2025-05-27 05:26:17,609 -- Sending request to rails: checkout_v2",
   "service": "chaos-engineering",
   "source_type": "datadog_agent",
   "status": "info",
   "timestamp": "2025-005-27T05:26:18.205Z"
}

Reference another field’s value

If you want a field’s value to be based on another field, you can dynamically reference the other field’s value.

Example

Input

For this example, you have a service field that contains an incorrect service name, and you want to use the value of app_id for the service instead.

{
    "timestamp": "2025-05-27T05:26:18.205Z",
    "status": "info",
    "service": "mywrongservice",
    "app_id": "web-store"
}

Custom function

#Overwrite service to be the value of app_id
.service = {{.app_id}}

Output

{
  "timestamp": "2025-05-27T05:26:18.205Z",
  "status": "info",
  "service": "web-store",
  "app_id": "web-store"
}

Further reading

Additional helpful documentation, links, and articles:

Learn more about the Custom ProcessorDOCUMENTATION more more Set up pipelinesDOCUMENTATION more more
PREVIEWING: guacbot/translation-pipeline