Cette page n'est pas encore disponible en français, sa traduction est en cours. Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.
Observability Pipelines is not available on the US1-FED Datadog site.
A sink is a destination for events. Each sink’s design and transmission method is determined by the downstream service with which it interacts. For example, the socket sink streams individual events, while the aws_s3 sink buffers and flushes data.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
exchange
required
string
A templated field.
The exchange to publish messages to.
properties
optional
<oneOf>
Configure the AMQP message properties.
AMQP message properties.
Option 1
optional
object
Configure the AMQP message properties.
AMQP properties configuration.
content_encoding
optional
string,null
Content-Encoding for the AMQP messages.
content_type
optional
string,null
Content-Type for the AMQP messages.
routing_key
optional
<oneOf>
A templated field.
Template used to generate a routing key which corresponds to a queue binding.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
connection_string
required
string
URI for the AMQP server.
The URI has the format of
amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.
The default vhost can be specified by using a value of %2f.
To connect over TLS, a scheme of amqps can be specified instead. For example,
amqps://.... Additional TLS settings, such as client certificate verification, can be
configured under the tls section.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
uri
The URI for the AppSignal API to send data to.
push_api_key
required
string
A valid app-level AppSignal Push API key.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
group_name
required
string
The group name of the target CloudWatch Logs stream.
request
optional
Outbound HTTP request settings.
headers
optional
object
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
stream_name
required
string
The stream name of the target CloudWatch Logs stream.
There can only be one writer to a log stream at a time. If multiple instances are writing to
the same log group, the stream name must include an identifier that is guaranteed to be
unique per instance.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
The default namespace to use for metrics that do not have one.
Metrics with the same name can only be differentiated by their namespace, and not all
metrics have their own namespace.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
Configuration of the authentication strategy for interacting with AWS services.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
request_retry_partial
optional
boolean
Whether or not to retry successful requests containing partial failures.
stream_name
optional
string
The stream name of the target Kinesis Firehose delivery stream.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
partition_key_field
optional
<oneOf>
The log field used as the Kinesis record’s partition key value.
If not specified, a unique partition key is generated for each Kinesis record.
Option 1
optional
string
A wrapper around OwnedValuePath that allows it to be used in Vector config.
This requires a valid path to be used. If you want to allow optional paths,
use [optional_path::OptionalValuePath].
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
Configuration of the authentication strategy for interacting with AWS services.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
request_retry_partial
optional
boolean
Whether or not to retry successful requests containing partial failures.
stream_name
optional
string
The stream name of the target Kinesis Firehose delivery stream.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
Configuration of the authentication strategy for interacting with AWS services.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
bucket
required
string
The S3 bucket name.
This must not include a leading s3:// or a trailing /.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Some cloud storage API clients and browsers handle decompression transparently, so
depending on how they are accessed, files may not always appear to be compressed.
Whether or not to append a UUID v4 token to the end of the object key.
The UUID is appended to the timestamp portion of the object key, such that if the object key
generated is date=2022-07-18/1658176486, setting this field to true results
in an object key that looks like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.
This ensures there are no name collisions, and can be useful in high-volume workloads where
object keys must be unique.
filename_extension
optional
string,null
The filename extension to use in the object key.
This overrides setting the extension based on the configured compression.
filename_time_format
optional
string
The timestamp format for the time component of the object key.
By default, object keys are appended with a timestamp that reflects when the objects are
sent to S3, such that the resulting object key is functionally equivalent to joining the key
prefix with the formatted timestamp, such as date=2022-07-18/1658176486.
This would represent a key_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022
20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders
timestamps in seconds since the Unix epoch.
Supports the common strftime specifiers found in most
languages.
When set to an empty string, no timestamp is appended to the key prefix.
key_prefix
optional
string
A prefix to apply to all object keys.
Prefixes are useful for partitioning objects, such as by creating an object key that
stores objects under a particular directory. If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
The bucket/object owner is granted the FULL_CONTROL permission, and no one else has
access.
This is the default.
public-read
optional
public-read
Bucket/object can be read publicly.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ permission.
public-read-write
optional
public-read-write
Bucket/object can be read and written publicly.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ and WRITE permissions.
This is generally not recommended.
aws-exec-read
optional
aws-exec-read
Bucket/object are private, and readable by EC2.
The bucket/object owner is granted the FULL_CONTROL permission, and the AWS EC2 service is
granted the READ permission for the purpose of reading Amazon Machine Image (AMI) bundles
from the given bucket.
authenticated-read
optional
authenticated-read
Bucket/object can be read by authenticated users.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AuthenticatedUsers grantee group is granted the READ permission.
bucket-owner-read
optional
bucket-owner-read
Object is private, except to the bucket owner.
The object owner is granted the FULL_CONTROL permission, and the bucket owner is granted the READ permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
bucket-owner-full-control
optional
bucket-owner-full-control
Object is semi-private.
Both the object owner and bucket owner are granted the FULL_CONTROL permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
log-delivery-write
optional
log-delivery-write
Bucket can have logs written.
The LogDelivery grantee group is granted WRITE and READ_ACP permissions.
Only relevant when specified for a bucket: this canned ACL is otherwise ignored when
specified for an object.
Overrides what content encoding has been applied to the object.
Directly comparable to the Content-Encoding HTTP header.
If not specified, the compression scheme used dictates this value.
content_type
optional
string,null
Overrides the MIME type of the object.
Directly comparable to the Content-Type HTTP header.
If not specified, the compression scheme used dictates this value.
When compression is set to none, the value text/x-log is used.
grant_full_control
optional
string,null
Grants READ, READ_ACP, and WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata, as well as read and
modify the ACL on the created objects.
grant_read
optional
string,null
Grants READ permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata.
grant_read_acp
optional
string,null
Grants READ_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the ACL on the created objects.
grant_write_acp
optional
string,null
Grants WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to modify the ACL on the created objects.
server_side_encryption
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
The Server-side Encryption algorithm used when storing these objects.
Option 1
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
More information on each algorithm can be found in the AWS documentation.
AES256
optional
AES256
Each object is encrypted with AES-256 using a unique key.
This corresponds to the SSE-S3 option.
aws:kms
optional
aws:kms
Each object is encrypted with AES-256 using keys managed by AWS KMS.
Depending on whether or not a KMS key ID is specified, this corresponds either to the
SSE-KMS option (keys generated/managed by KMS) or the SSE-C option (keys generated by
the customer, managed by KMS).
ssekms_key_id
optional
string,null
Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed
customer master key (CMK) that is used for the created objects.
Only applies when server_side_encryption is configured to use KMS.
If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
framing
optional
<oneOf>
Framing configuration.
Option 1
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
message_deduplication_id
optional
string,null
The message deduplication ID value to allow AWS to identify duplicate messages.
This value is a template which should result in a unique string for each event. See the AWS
documentation for more about how AWS does message deduplication.
message_group_id
optional
string,null
The tag that specifies that a message belongs to a specific message group.
Can be applied only to FIFO queues.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
blob_append_uuid
optional
boolean,null
Whether or not to append a UUID v4 token to the end of the blob key.
The UUID is appended to the timestamp portion of the object key, such that if the blob key
generated is date=2022-07-18/1658176486, setting this field to true results
in an blob key that looks like
date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.
This ensures there are no name collisions, and can be useful in high-volume workloads where
blob keys must be unique.
blob_prefix
optional
string
A prefix to apply to all blob keys.
Prefixes are useful for partitioning objects, such as by creating a blob key that
stores blobs under a particular directory. If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
blob_time_format
optional
string,null
The timestamp format for the time component of the blob key.
By default, blob keys are appended with a timestamp that reflects when the blob are sent to
Azure Blob Storage, such that the resulting blob key is functionally equivalent to joining
the blob prefix with the formatted timestamp, such as date=2022-07-18/1658176486.
This would represent a blob_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022
20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders
timestamps in seconds since the Unix epoch.
Supports the common strftime specifiers found in most
languages.
When set to an empty string, no timestamp is appended to the blob prefix.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Authentication with access key is the only supported authentication method.
Either storage_account, or this field, must be specified.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
container_name
required
string
The Azure Blob Storage Account container name.
endpoint
optional
string,null
The Azure Blob Storage Endpoint URL.
This is used to override the default blob storage endpoint URL in cases where you are using
credentials read from the environment/managed identities or access tokens without using an
explicit connection_string (which already explicitly supports overriding the blob endpoint
URL).
This may only be used with storage_account and is ignored when used with
connection_string.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
storage_account
optional
string,null
The Azure Blob Storage Account name.
Attempts to load credentials for the account in the following ways, in order:
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
framing
optional
<oneOf>
Framing configuration.
Option 1
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
azure_resource_id
optional
string,null
The Resource ID of the Azure resource the data should be associated with.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
The record type of the data that is being submitted.
Can only contain letters, numbers, and underscores (_), and may not exceed 100 characters.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Use this option to customize the log field used as TimeGenerated in Azure.
The setting of log_schema.timestamp_key, usually timestamp, is used here by default.
This field should be used in rare cases where TimeGenerated should point to a specific log
field. For example, use this field to set the log field source_timestamp as holding the
value that should be used as TimeGenerated on the Azure side.
Option 1
optional
string
An optional path that deserializes an empty string to None.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
print_interval_secs
optional
integer
The interval between reporting a summary of activity.
Set to 0 to disable reporting.
rate
optional
integer,null
The number of events, per second, that the sink is allowed to consume.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The database that contains the table that data is inserted into.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
date_time_best_effort
optional
boolean
Sets date_time_input_format to best_effort, allowing ClickHouse to properly parse RFC3339/ISO 8601.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URI component of a request.
The endpoint of the ClickHouse server.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
skip_unknown_fields
optional
boolean
Sets input_format_skip_unknown_fields, allowing ClickHouse to discard fields not present in the table schema.
table
required
string
A templated field.
The table that data is inserted into.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
framing
optional
<oneOf>
Framing configuration.
Option 1
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
The database that contains the table that data is inserted into.
encoding
optional
Configures how events are encoded into raw bytes.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URI component of a request.
The endpoint of the Databend server.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
table
required
string
The table that data is inserted into.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
aws_s3
optional
<oneOf>
S3-specific configuration options.
Option 1
optional
S3-specific configuration options.
auth
optional
Configuration of the authentication strategy for interacting with AWS services.
The bucket/object owner is granted the FULL_CONTROL permission, and no one else has
access.
This is the default.
public-read
optional
public-read
Bucket/object can be read publicly.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ permission.
public-read-write
optional
public-read-write
Bucket/object can be read and written publicly.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ and WRITE permissions.
This is generally not recommended.
aws-exec-read
optional
aws-exec-read
Bucket/object are private, and readable by EC2.
The bucket/object owner is granted the FULL_CONTROL permission, and the AWS EC2 service is
granted the READ permission for the purpose of reading Amazon Machine Image (AMI) bundles
from the given bucket.
authenticated-read
optional
authenticated-read
Bucket/object can be read by authenticated users.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AuthenticatedUsers grantee group is granted the READ permission.
bucket-owner-read
optional
bucket-owner-read
Object is private, except to the bucket owner.
The object owner is granted the FULL_CONTROL permission, and the bucket owner is granted the READ permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
bucket-owner-full-control
optional
bucket-owner-full-control
Object is semi-private.
Both the object owner and bucket owner are granted the FULL_CONTROL permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
log-delivery-write
optional
log-delivery-write
Bucket can have logs written.
The LogDelivery grantee group is granted WRITE and READ_ACP permissions.
Only relevant when specified for a bucket: this canned ACL is otherwise ignored when
specified for an object.
Grants READ, READ_ACP, and WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata, as well as read and
modify the ACL on the created objects.
grant_read
optional
string,null
Grants READ permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata.
grant_read_acp
optional
string,null
Grants READ_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the ACL on the created objects.
grant_write_acp
optional
string,null
Grants WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to modify the ACL on the created objects.
server_side_encryption
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
The Server-side Encryption algorithm used when storing these objects.
Option 1
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
More information on each algorithm can be found in the AWS documentation.
AES256
optional
AES256
Each object is encrypted with AES-256 using a unique key.
This corresponds to the SSE-S3 option.
aws:kms
optional
aws:kms
Each object is encrypted with AES-256 using keys managed by AWS KMS.
Depending on whether or not a KMS key ID is specified, this corresponds either to the
SSE-KMS option (keys generated/managed by KMS) or the SSE-C option (keys generated by
the customer, managed by KMS).
ssekms_key_id
optional
string,null
Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed
customer master key (CMK) that is used for the created objects.
Only applies when server_side_encryption is configured to use KMS.
If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
key_prefix
optional
string,null
A prefix to apply to all object keys.
Prefixes are useful for partitioning objects, such as by creating an object key that
stores objects under a particular directory. If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
service
required
string
The name of the object storage service to use.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
default_api_key
required
string
The default Datadog API key to use in authentication of HTTP requests.
If an event has a Datadog API key set explicitly in its metadata, it takes
precedence over this setting.
endpoint
optional
string,null
The endpoint to send observability data to.
The endpoint must contain an HTTP scheme, and may specify a
hostname or IP address and port.
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
region
optional
<oneOf>
DEPRECATED: The Datadog region to send logs to.
Option 1
optional
<oneOf>
A Datadog region.
us
optional
us
US region.
eu
optional
eu
EU region.
request
optional
Outbound HTTP request settings.
headers
optional
object
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
default_api_key
required
string
The default Datadog API key to use in authentication of HTTP requests.
If an event has a Datadog API key set explicitly in its metadata, it takes
precedence over this setting.
endpoint
optional
string,null
The endpoint to send observability data to.
The endpoint must contain an HTTP scheme, and may specify a
hostname or IP address and port.
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_namespace
optional
string,null
Sets the default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
region
optional
<oneOf>
DEPRECATED: The Datadog region to send metrics to.
Option 1
optional
<oneOf>
A Datadog region.
us
optional
us
US region.
eu
optional
eu
EU region.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
default_api_key
required
string
The default Datadog API key to use in authentication of HTTP requests.
If an event has a Datadog API key set explicitly in its metadata, it takes
precedence over this setting.
endpoint
optional
string,null
The endpoint to send observability data to.
The endpoint must contain an HTTP scheme, and may specify a
hostname or IP address and port.
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
default_api_key
required
string
The default Datadog API key to use in authentication of HTTP requests.
If an event has a Datadog API key set explicitly in its metadata, it takes
precedence over this setting.
endpoint
optional
string,null
The endpoint to send observability data to.
The endpoint must contain an HTTP scheme, and may specify a
hostname or IP address and port.
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
api_version
optional
<oneOf>
The API version of Elasticsearch.
auto
optional
auto
Auto-detect the API version.
If the cluster state version endpoint isn't reachable, a warning is logged to
stdout, and the version is assumed to be V6 if the suppress_type_name option is set to
true. Otherwise, the version is assumed to be V8. In the future, the sink instead
returns an error during configuration parsing, since a wrongly assumed version could lead to
incorrect API calls.
Automatically routes events by deriving the data stream name using specific event fields.
The format of the data stream name is <type>-<dataset>-<namespace>, where each value comes
from the data_stream configuration field of the same name.
If enabled, the value of the data_stream.type, data_stream.dataset, and
data_stream.namespace event fields are used if they are present. Otherwise, the values
set in this configuration are used.
dataset
optional
string
A templated field.
The data stream dataset used to construct the data stream at index time.
namespace
optional
string
A templated field.
The data stream namespace used to construct the data stream at index time.
sync_fields
optional
boolean
Automatically adds and syncs the data_stream.* event fields if they are missing from the event.
This ensures that fields match the name of the data stream that is receiving events.
type
optional
string
A templated field.
The data stream type used to construct the data stream at index time.
distribution
optional
<oneOf>
Options for determining the health of an endpoint.
Option 1
optional
object
Options for determining the health of an endpoint.
retry_initial_backoff_secs
optional
integer
Initial delay between attempts to reactivate endpoints once they become unhealthy.
retry_max_duration_secs
optional
integer
Maximum delay between attempts to reactivate endpoints once they become unhealthy.
This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to
set this option since Elasticsearch has removed it.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string,null
The Elasticsearch endpoint to send logs to.
DEPRECATED: The endpoint must contain an HTTP scheme, and may specify a
hostname or IP address and port.
endpoints
optional
[string]
A list of Elasticsearch endpoints to send logs to.
The endpoint must contain an HTTP scheme, and may specify a
hostname or IP address and port.
id_key
optional
<oneOf>
The name of the event key that should map to Elasticsearch’s _id field.
By default, the _id field is not set, which allows Elasticsearch to set this
automatically. Setting your own Elasticsearch IDs can hinder performance.
Option 1
optional
string
A wrapper around OwnedValuePath that allows it to be used in Vector config.
This requires a valid path to be used. If you want to allow optional paths,
use [optional_path::OptionalValuePath].
metrics
optional
<oneOf>
Configuration for the metric_to_log transform.
Option 1
optional
object
Configuration for the metric_to_log transform.
host_tag
optional
string,null
Name of the tag in the metric to use for the source host.
If present, the value of the tag is set on the generated log event in the host field,
where the field key uses the global host_key option.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments as
described by the native_json codec.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
timezone
optional
<oneOf>
The name of the time zone to apply to timestamp conversions that do not contain an explicit
time zone.
This overrides the global timezone option. The time zone name may be
any name in the TZ database or local to indicate system local time.
Option 1
optional
<oneOf>
Timezone reference.
This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.
Ingests documents in bulk, using the bulk API index action.
data_stream
optional
data_stream
Ingests documents in bulk, using the bulk API create action.
Elasticsearch Data Streams only support the create action.
pipeline
optional
string,null
The name of the pipeline to apply.
query
optional
object,null
Custom parameters to add to the query string for each HTTP request sent to Elasticsearch.
request
optional
Outbound HTTP request settings.
headers
optional
object
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
request_retry_partial
optional
boolean
Whether or not to retry successful requests containing partial failures.
To avoid duplicates in Elasticsearch, please use option id_key.
suppress_type_name
optional
boolean
Whether or not to send the type field to Elasticsearch.
DEPRECATED: The type field was deprecated in Elasticsearch 7.x and removed in Elasticsearch 8.x.
If enabled, the doc_type option is ignored.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
customer_id
required
uuid
The Unique identifier (UUID) corresponding to the Chronicle instance.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string,null
The endpoint to send data to.
log_type
required
string
The type of log entries in a request.
This must be one of the supported log types, otherwise
Chronicle rejects the entry with an error.
region
optional
<oneOf>
The GCP region to use.
Option 1
optional
<oneOf>
Google Chronicle regions.
eu
optional
eu
EU region.
us
optional
us
US region.
asia
optional
asia
APAC region.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Whether or not to append a UUID v4 token to the end of the object key.
The UUID is appended to the timestamp portion of the object key, such that if the object key
generated is date=2022-07-18/1658176486, setting this field to true results
in an object key that looks like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.
This ensures there are no name collisions, and can be useful in high-volume workloads where
object keys must be unique.
filename_extension
optional
string,null
The filename extension to use in the object key.
If not specified, the extension is determined by the compression scheme used.
filename_time_format
optional
string
The timestamp format for the time component of the object key.
By default, object keys are appended with a timestamp that reflects when the objects are
sent to S3, such that the resulting object key is functionally equivalent to joining the key
prefix with the formatted timestamp, such as date=2022-07-18/1658176486.
This would represent a key_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022
20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders
timestamps in seconds since the Unix epoch.
Supports the common strftime specifiers found in most
languages.
When set to an empty string, no timestamp is appended to the key prefix.
key_prefix
optional
string,null
A prefix to apply to all object keys.
Prefixes are useful for partitioning objects, such as by creating an object key that
stores objects under a particular directory. If using a prefix for this purpose, it must end
in / in order to act as a directory path. A trailing / is not automatically added.
metadata
optional
object,null
The set of metadata key:value pairs for the created objects.
For more information, see the custom metadata documentation.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
storage_class
optional
<oneOf>
The storage class for created objects.
For more information, see the storage classes documentation.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
framing
optional
<oneOf>
Framing configuration.
Option 1
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The endpoint to which to publish events.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the GCP Pub/Sub API are used.
The trailing slash / must not be included.
project
required
string
The project name to which to publish events.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
topic
required
string
The topic within the project to which to publish events.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
log_id
required
string
The log ID to which to publish logs.
This is a name you create to identify this log stream.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
resource
required
A monitored resource.
The monitored resource to associate the logs with.
The field of the log event from which to take the outgoing log’s severity field.
The named field is removed from the log event if present, and must be either an integer
between 0 and 800 or a string containing one of the severity level names (case
is ignored) or a common prefix such as err.
If no severity key is specified, the severity of outgoing records is set to 0 (DEFAULT).
A wrapper around OwnedValuePath that allows it to be used in Vector config.
This requires a valid path to be used. If you want to allow optional paths,
use [optional_path::OptionalValuePath].
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_namespace
optional
string
The default namespace to use for metrics that do not have one.
Metrics with the same name can only be differentiated by their namespace, and not all
metrics have their own namespace.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
resource
required
A monitored resource.
The monitored resource to associate the metrics with.
type
required
string
The monitored resource type.
For example, the type of a Compute Engine VM instance is gce_instance.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
api_key
required
string
The API key that is used to authenticate against Honeycomb.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
dataset
required
string
The dataset to which logs are sent.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
DEPRECATED: A list of custom headers to add to each request.
method
optional
<oneOf>
HTTP method.
The HTTP method to use when making the request.
get
optional
get
GET.
head
optional
head
HEAD.
post
optional
post
POST.
put
optional
put
PUT.
delete
optional
delete
DELETE.
options
optional
options
OPTIONS.
trace
optional
trace
TRACE.
patch
optional
patch
PATCH.
payload_prefix
optional
string
A string to prefix the payload with.
This option is ignored if the encoding is not character delimited JSON.
If specified, the payload_suffix must also be specified and together they must produce a valid JSON object.
payload_suffix
optional
string
A string to suffix the payload with.
This option is ignored if the encoding is not character delimited JSON.
If specified, the payload_prefix must also be specified and together they must produce a valid JSON object.
request
optional
Outbound HTTP request settings.
headers
optional
object
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
uri
required
string
The full URI to make HTTP requests to.
This should include the protocol and host, but can also include the port, path, and any other valid part of a URI.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
framing
optional
<oneOf>
Framing configuration.
Option 1
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The base URL of the Humio instance.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
event_type
optional
<oneOf>
The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.
If unset, Humio defaults it to none.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
host_key
optional
string
Overrides the name of the log field used to retrieve the hostname to send to Humio.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
indexed_fields
optional
[string]
Event fields to be added to Humio’s extra fields.
Can be used to tag events by specifying fields starting with #.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
source
optional
<oneOf>
The source of events sent to this sink.
Typically the filename the logs originated from. Maps to @source in Humio.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
timestamp_key
optional
string
Overrides the name of the log field used to retrieve the timestamp to send to Humio.
Overrides the name of the log field used to retrieve the nanosecond-enabled timestamp to send to Humio.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
event_type
optional
<oneOf>
The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.
If unset, Humio defaults it to none.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
host_key
optional
string
Overrides the name of the log field used to retrieve the hostname to send to Humio.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
indexed_fields
optional
[string]
Event fields to be added to Humio’s extra fields.
Can be used to tag events by specifying fields starting with #.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
source
optional
<oneOf>
The source of events sent to this sink.
Typically the filename the metrics originated from. Maps to @source in Humio.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
token
required
string
The Humio ingestion token.
host_tag
optional
string,null
Name of the tag in the metric to use for the source host.
If present, the value of the tag is set on the generated log event in the host field,
where the field key uses the global host_key option.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments as
described by the native_json codec.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
timezone
optional
<oneOf>
The name of the time zone to apply to timestamp conversions that do not contain an explicit
time zone.
This overrides the global timezone option. The time zone name may be
any name in the TZ database or local to indicate system local time.
Option 1
optional
<oneOf>
Timezone reference.
This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The endpoint to send data to.
This should be a full HTTP URI, including the scheme, host, and port.
host_key
optional
<oneOf>
Use this option to customize the key containing the hostname.
The setting of log_schema.host_key, usually host, is used here by default.
Option 1
optional
string
An optional path that deserializes an empty string to None.
measurement
optional
string,null
The name of the InfluxDB measurement that is written to.
message_key
optional
<oneOf>
Use this option to customize the key containing the message.
The setting of log_schema.message_key, usually message, is used here by default.
Option 1
optional
string
An optional path that deserializes an empty string to None.
namespace
optional
string,null
The namespace of the measurement name to use.
DEPRECATED: When specified, the measurement name is <namespace>.vector.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
source_type_key
optional
<oneOf>
Use this option to customize the key containing the source_type.
The setting of log_schema.source_type_key, usually source_type, is used here by default.
Option 1
optional
string
An optional path that deserializes an empty string to None.
tags
optional
[string]
The list of names of log fields that should be added as tags to each measurement.
By default Vector adds metric_type as well as the configured log_schema.host_key and
log_schema.source_type_key options.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_namespace
optional
string,null
Sets the default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
endpoint
required
string
The endpoint to send data to.
This should be a full HTTP URI, including the scheme, host, and port.
quantiles
optional
[number]
The list of quantiles to calculate when sending distribution metrics.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tags
optional
object,null
A map of additional tags, in the key/value pair format, to add to each measurement.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
bootstrap_servers
required
string
A comma-separated list of Kafka bootstrap servers.
These are the servers in a Kafka cluster that a client should use to bootstrap its
connection to the cluster, allowing discovery of all the other hosts in the cluster.
Must be in the form of host:port, and comma-separated.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
headers_key
optional
<oneOf>
The log field name to use for the Kafka headers.
If omitted, no headers are written.
Option 1
optional
string
A wrapper around OwnedTargetPath that allows it to be used in Vector config
with prefix default to PathPrefix::Event
key_field
optional
<oneOf>
The log field name or tag key to use for the topic key.
If the field does not exist in the log or in the tags, a blank value is used. If
unspecified, the key is not sent.
Kafka uses a hash of the key to choose the partition or uses round-robin if the record has
no key.
Option 1
optional
string
A wrapper around OwnedTargetPath that allows it to be used in Vector config
with prefix default to PathPrefix::Event
librdkafka_options
optional
object
A map of advanced options to pass directly to the underlying librdkafka client.
Default timeout, in milliseconds, for network requests.
topic
required
string
A templated field.
The Kafka topic name to write events to.
sasl
optional
<oneOf>
Configuration for SASL authentication when interacting with Kafka.
Option 1
optional
object
Configuration for SASL authentication when interacting with Kafka.
enabled
optional
boolean,null
Enables SASL authentication.
Only PLAIN- and SCRAM-based mechanisms are supported when configuring SASL authentication using sasl.*. For
other mechanisms, librdkafka_options.* must be used directly to configure other librdkafka-specific values.
If using sasl.kerberos.* as an example, where * is service.name, principal, kinit.md, etc., then
librdkafka_options.* as a result becomes librdkafka_options.sasl.kerberos.service.name,
librdkafka_options.sasl.kerberos.principal, etc.
Wrapper for sensitive strings containing credentials
username
optional
string,null
The SASL username.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
api_key
required
string
The Ingestion API key.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_app
optional
string
The default app that is set for events that do not contain a file or app field.
default_env
optional
string
The default environment that is set for events that do not contain an env field.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The HTTP endpoint to send logs to.
Both IP address and hostname are accepted formats.
hostname
required
string
A templated field.
The hostname that is attached to each batch of events.
ip
optional
string,null
The IP address that is attached to each batch of events.
mac
optional
string,null
The MAC address that is attached to each batch of events.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tags
optional
array,null
The tags that are attached to each batch of events.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The base URL of the Loki instance.
The path value is appended to this.
labels
optional
object
A set of labels that are attached to each batch of events.
Both keys and values are templateable, which enables you to attach dynamic labels to events.
Valid label keys include *, and prefixes ending with *, to allow for the expansion of
objects into multiple labels. See Label expansion for more information.
Note: If the set of labels has high cardinality, this can cause drastic performance issues
with Loki. To prevent this from happening, reduce the number of unique label keys and
values.
out_of_order_action
optional
<oneOf>
Out-of-order event behavior.
Some sources may generate events with timestamps that aren't in chronological order. Even though the
sink sorts the events before sending them to Loki, there is a chance that another event could come in
that is out of order with the latest events sent to Loki. Prior to Loki 2.4.0, this
was not supported and would result in an error during the push request.
If you're using Loki 2.4.0 or newer, Accept is the preferred action, which lets Loki handle
any necessary sorting/reordering. If you're using an earlier version, then you must use Drop
or RewriteTimestamp depending on which option makes the most sense for your use case.
drop
optional
drop
Drop the event.
rewrite_timestamp
optional
rewrite_timestamp
Rewrite the timestamp of the event to the timestamp of the latest event seen by the sink.
accept
optional
accept
Accept the event.
The event is not dropped and is sent without modification.
Requires Loki 2.4.0 or newer.
path
optional
string
The path to use in the URL of the Loki instance.
remove_label_fields
optional
boolean
Whether or not to delete fields from the event when they are used as labels.
remove_timestamp
optional
boolean
Whether or not to remove the timestamp from the event payload.
The timestamp is still sent as event metadata for Loki to use for indexing.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
When running Loki locally, a tenant ID is not required.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
api_key
required
string
The Ingestion API key.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_app
optional
string
The default app that is set for events that do not contain a file or app field.
default_env
optional
string
The default environment that is set for events that do not contain an env field.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The HTTP endpoint to send logs to.
Both IP address and hostname are accepted formats.
hostname
required
string
A templated field.
The hostname that is attached to each batch of events.
ip
optional
string,null
The IP address that is attached to each batch of events.
mac
optional
string,null
The MAC address that is attached to each batch of events.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tags
optional
array,null
The tags that are attached to each batch of events.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Configuration of the authentication strategy when interacting with NATS.
Option 1
optional
<oneOf>
Configuration of the authentication strategy when interacting with NATS.
UserPassword
optional
object
Username/password authentication.
strategy
required
user_password
Username/password authentication.
user_password
required
object
Username and password configuration.
password
required
string
Password.
user
required
string
Username.
Token
optional
object
Token authentication.
strategy
required
token
Token authentication.
token
required
object
Token configuration.
value
required
string
Token.
CredentialsFile
optional
object
Credentials file authentication. (JWT-based)
credentials_file
required
object
Credentials file configuration.
path
required
string
Path to credentials file.
strategy
required
credentials_file
Credentials file authentication. (JWT-based)
Nkey
optional
object
NKey authentication.
nkey
required
object
NKeys configuration.
nkey
required
string
User.
Conceptually, this is equivalent to a public key.
seed
required
string
Seed.
Conceptually, this is equivalent to a private key.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
api
required
<oneOf>
New Relic API endpoint.
events
optional
events
Events API.
metrics
optional
metrics
Metrics API.
logs
optional
logs
Logs API.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
license_key
required
string
A valid New Relic license key.
region
optional
<oneOf>
New Relic region.
Option 1
optional
<oneOf>
New Relic region.
us
optional
us
US region.
eu
optional
eu
EU region.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URI component of a request.
The TCP endpoint to send logs to.
keepalive
optional
<oneOf>
TCP keepalive settings for socket-based components.
Option 1
optional
object
TCP keepalive settings for socket-based components.
time_secs
optional
integer,null
The time to wait before starting to send TCP keepalive probes on an idle connection.
process
optional
string
A templated field.
The value to use as the process in Papertrail.
send_buffer_bytes
optional
integer,null
Configures the send buffer size using the SO_SNDBUF option on the socket.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
address
optional
string
The address to expose for scraping.
The metrics are exposed at the typical Prometheus exporter path, /metrics.
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
buckets
optional
[number]
Default buckets to use for aggregating distribution metrics into histograms.
default_namespace
optional
string,null
The default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with an underscore (_).
While distributions as a lossless way to represent a set of samples for a
metric is supported, Prometheus clients (the application being scraped, which is this sink) must
aggregate locally into either an aggregated histogram or aggregated summary.
flush_period_secs
optional
integer
The interval, in seconds, on which metrics are flushed.
On the flush interval, if a metric has not been seen since the last flush interval, it is
considered expired and is removed.
Be sure to configure this value higher than your client’s scrape interval.
quantiles
optional
[number]
Quantiles to use for aggregating distribution metrics into a summary.
suppress_timestamp
optional
boolean
Suppresses timestamps on the Prometheus output.
This can sometimes be useful when the source of metrics leads to their timestamps being too
far in the past for Prometheus to allow them, such as when aggregating metrics over long
time periods, or when replaying old metrics from a disk buffer.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Authentication strategies.
Option 1
optional
<oneOf>
Authentication strategies.
Basic
optional
object
HTTP Basic Authentication.
password
required
string
Basic authentication password.
strategy
required
basic
HTTP Basic Authentication.
user
required
string
Basic authentication username.
Bearer
optional
object
Bearer authentication.
A bearer token (OAuth2, JWT, etc) is passed as-is.
strategy
required
bearer
Bearer authentication.
A bearer token (OAuth2, JWT, etc) is passed as-is.
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
buckets
optional
[number]
Default buckets to use for aggregating distribution metrics into histograms.
compression
optional
<oneOf>
Supported compression types for Prometheus Remote Write.
snappy
optional
snappy
Snappy.
gzip
optional
gzip
Gzip.
zstd
optional
zstd
Zstandard.
default_namespace
optional
string,null
The default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with an underscore (_).
The endpoint should include the scheme and the path to write to.
quantiles
optional
[number]
Quantiles to use for aggregating distribution metrics into a summary.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tenant_id
optional
<oneOf>
The tenant ID to send.
If set, a header named X-Scope-OrgID is added to outgoing requests with the value of this setting.
This may be used by Cortex or other remote services to identify the tenant making the request.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auth
optional
<oneOf>
Authentication configuration.
Option 1
optional
object
Authentication configuration.
name
optional
string,null
Basic authentication name/username.
This can be used either for basic authentication (username/password) or JWT authentication.
When used for JWT, the value should be token.
oauth2
optional
<oneOf>
OAuth2-specific authentication configuration.
Option 1
optional
object
OAuth2-specific authentication configuration.
audience
optional
string,null
The OAuth2 audience.
credentials_url
required
string
The credentials URL.
A data URL is also supported.
issuer_url
required
string
The issuer URL.
scope
optional
string,null
The OAuth2 scope.
token
optional
<oneOf>
Basic authentication password/token.
This can be used either for basic authentication (username/password) or JWT authentication.
When used for JWT, the value should be the signed JWT, in the compact representation.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
max_events
optional
integer,null
The maximum amount of events in a batch before it is flushed.
Note this is an unsigned 32 bit integer which is a smaller capacity than
many of the other sink batch settings.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The endpoint to which the Pulsar client should connect to.
The endpoint should specify the pulsar protocol and port.
partition_key_field
optional
<oneOf>
The log field name or tags key to use for the partition key.
If the field does not exist in the log event or metric tags, a blank value will be used.
If omitted, the key is not sent.
Pulsar uses a hash of the key to choose the topic-partition or uses round-robin if the record has no key.
Option 1
optional
string
An optional path that deserializes an empty string to None.
producer_name
optional
string,null
The name of the producer. If not specified, the default name assigned by Pulsar is used.
properties_key
optional
<oneOf>
The log field name to use for the Pulsar properties key.
If omitted, no properties will be written.
Option 1
optional
string
An optional path that deserializes an empty string to None.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
data_type
optional
<oneOf>
Redis data type to store messages in.
list
optional
list
The Redis list type.
This resembles a deque, where messages can be popped and pushed from either end.
This is the default.
channel
optional
channel
The Redis channel type.
Redis channels function in a pub/sub fashion, allowing many-to-many broadcasting and receiving.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URL of the Redis endpoint to connect to.
The URL must take the form of protocol://server:port/db where the protocol can either be
redis or rediss for connections secured via TLS.
key
required
string
A templated field.
The Redis key to publish messages to.
list_option
optional
<oneOf>
List-specific options.
Option 1
optional
object
List-specific options.
method
required
<oneOf>
The method to use for pushing messages into a list.
rpush
optional
rpush
Use the rpush method.
This pushes messages onto the tail of the list.
This is the default.
lpush
optional
lpush
Use the lpush method.
This pushes messages onto the head of the list.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string,null
The endpoint to send data to.
Setting this option overrides the region option.
region
optional
<oneOf>
The Sematext region to send data to.
us
optional
us
United States
eu
optional
eu
Europe
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_namespace
required
string
Sets the default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
endpoint
optional
string,null
The endpoint to send data to.
Setting this option overrides the region option.
region
optional
<oneOf>
The Sematext region to send data to.
us
optional
us
United States
eu
optional
eu
Europe
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The maximum number of pending acknowledgements from events sent to the Splunk HEC collector.
Once reached, the sink begins applying backpressure.
query_interval
optional
integer
The amount of time to wait between queries to the Splunk HEC indexer acknowledgement endpoint.
retry_limit
optional
integer
The maximum number of times an acknowledgement ID is queried for its status.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
auto_extract_timestamp
optional
boolean,null
Passes the auto_extract_timestamp option to Splunk.
This option is only relevant to Splunk v8.x and above, and is only applied when
endpoint_target is set to event.
Setting this to true causes Splunk to extract the timestamp from the message text
rather than use the timestamp embedded in the event. The timestamp must be in the format
yyyy-mm-dd hh:mm:ss.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
uri
The base URL of the Splunk instance.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
If not specified, the default index defined within Splunk is used.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
source
optional
<oneOf>
The source of events sent to this sink.
This is typically the filename the logs originated from.
If unset, the Splunk collector sets it.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
sourcetype
optional
<oneOf>
The sourcetype of events sent to this sink.
If unset, Splunk defaults to httpevent.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
timestamp_key
optional
string
Overrides the name of the log field used to retrieve the timestamp to send to Splunk HEC.
When set to “”, a timestamp is not set in the events sent to Splunk HEC.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
The maximum number of pending acknowledgements from events sent to the Splunk HEC collector.
Once reached, the sink begins applying backpressure.
query_interval
optional
integer
The amount of time to wait between queries to the Splunk HEC indexer acknowledgement endpoint.
retry_limit
optional
integer
The maximum number of times an acknowledgement ID is queried for its status.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
default_token
required
string
Default Splunk HEC token.
If an event has a token set in its metadata, it prevails over the one set here.
endpoint
required
uri
The base URL of the Splunk instance.
The scheme (http or https) must be specified. No path should be included since the paths defined
by the Splunk API are used.
host_key
optional
string
Overrides the name of the log field used to retrieve the hostname to send to Splunk HEC.
The name of the index where to send the events to.
If not specified, the default index defined within Splunk is used.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
source
optional
<oneOf>
The source of events sent to this sink.
This is typically the filename the logs originated from.
If unset, the Splunk collector sets it.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
sourcetype
optional
<oneOf>
The sourcetype of events sent to this sink.
If unset, Splunk defaults to httpevent.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
default_namespace
optional
string,null
Sets the default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.).
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
address
required
uri
The downstream Vector address to which to connect.
Both IP address and hostname are accepted formats.
The address must include a port.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
boolean
Whether or not to compress requests.
If set to true, requests are compressed with gzip.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit metric.
default: 1
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The time window used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
version
optional
<oneOf>
Version of the configuration.
Option 1
optional
<oneOf>
Marker type for the version two of the configuration for the vector sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, waits for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized/compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: null
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Prefixes are useful for partitioning objects, such as by creating a blob key that
stores blobs under a particular directory. If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
The final file path is in the format of {root}/{prefix}{suffix}.
root
optional
string
The root path for WebHDFS.
Must be a valid directory.
The final file path is in the format of {root}/{prefix}{suffix}.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer.
This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in
field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character
like \ (instead of escaping quotes by doubling them).
To use this double_quotes needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they
appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array, Object, and Regex are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary.
They are necessary when fields contain a quote, delimiter or record terminator.
Quotes are also necessary when writing an empty record
(which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric.
Namely, when writing a field that does not parse as a valid float or integer,
then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
framing
optional
<oneOf>
Framing configuration.
Option 1
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
