- 필수 기능
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- 디지털 경험
- 소프트웨어 제공
- 보안
- 로그 관리
- 관리
- 인프라스트럭처
- ci
- containers
- csm
- ndm
- otel_guides
- overview
- slos
- synthetics
- tests
- 워크플로
A sink is a destination for events. Each sink’s design and transmission method is determined by the downstream service with which it interacts. For example, the socket
sink streams individual events, while the aws_s3
sink buffers and flushes data.
Supports AMQP version 0.9.1
항목
required
유형
설명
acknowledgements:
enabled: null
encoding: ''
exchange: string
properties: ''
routing_key: ''
type: amqp
Configuration for the appsignal
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
uri
The URI for the AppSignal API to send data to.
push_api_key
required
string
A valid app-level AppSignal Push API key.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: gzip
encoding: {}
endpoint: 'https://appsignal-endpoint.net'
push_api_key: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
type: appsignal
Configuration for the aws_cloudwatch_logs
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
assume_role: string
auth:
imds:
connect_timeout_seconds: 1
max_attempts: 4
read_timeout_seconds: 1
load_timeout_secs: null
region: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
create_missing_group: true
create_missing_stream: true
encoding: ''
group_name: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
headers: {}
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
stream_name: string
tls: ''
type: aws_cloudwatch_logs
Configuration for the aws_cloudwatch_metrics
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
assume_role: string
auth:
imds:
connect_timeout_seconds: 1
max_attempts: 4
read_timeout_seconds: 1
load_timeout_secs: null
region: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
default_namespace: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
type: aws_cloudwatch_metrics
Configuration for the aws_kinesis_firehose
sink.
항목
required
유형
설명
batch:
max_bytes: null
max_events: null
timeout_secs: null
type: aws_kinesis_firehose
Configuration for the aws_kinesis_streams
sink.
항목
required
유형
설명
batch:
max_bytes: null
max_events: null
timeout_secs: null
partition_key_field: ''
type: aws_kinesis_streams
Configuration for the aws_s3
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
auth:
imds:
connect_timeout_seconds: 1
max_attempts: 4
read_timeout_seconds: 1
load_timeout_secs: null
region: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
bucket: string
compression: gzip
filename_append_uuid: true
filename_extension: string
filename_time_format: '%s'
key_prefix: date=%F
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
type: aws_s3
Configuration for the axiom
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
dataset
required
string
The Axiom dataset to write to.
org_id
optional
string,null
The Axiom organization ID.
Only required when using personal tokens.
request
optional
Outbound HTTP request settings.
headers
optional
object
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
token
required
string
The Axiom API token.
url
optional
uri
URI of the Axiom endpoint to send data to.
Only required if not using Axiom Cloud.
acknowledgements:
enabled: null
compression: none
dataset: string
org_id: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
headers: {}
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
token: string
url: string
type: axiom
Configuration for the azure_blob
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
blob_append_uuid: boolean
blob_prefix: blob/%F/
blob_time_format: string
compression: gzip
connection_string: ''
container_name: string
endpoint: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
storage_account: string
type: azure_blob
Configuration for the azure_monitor_logs
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
azure_resource_id
optional
string,null
The Resource ID of the Azure resource the data should be associated with.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullencoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
log_type
required
string
The record type of the data that is being submitted.
Can only contain letters, numbers, and underscores (_), and may not exceed 100 characters.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60time_generated_key
optional
<oneOf>
Use this option to customize the log field used as TimeGenerated
in Azure.
The setting of log_schema.timestamp_key
, usually timestamp
, is used here by default.
This field should be used in rare cases where TimeGenerated
should point to a specific log
field. For example, use this field to set the log field source_timestamp
as holding the
value that should be used as TimeGenerated
on the Azure side.
Option 1
optional
string
An optional path that deserializes an empty string to None
.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
azure_resource_id: string
batch:
max_bytes: null
max_events: null
timeout_secs: null
customer_id: string
encoding: {}
host: ods.opinsights.azure.com
log_type: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
shared_key: string
time_generated_key: ''
tls: ''
type: azure_monitor_logs
Configuration for the blackhole
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
print_interval_secs
optional
integer
The interval between reporting a summary of activity.
Set to 0
to disable reporting.
rate
optional
integer,null
The number of events, per second, that the sink is allowed to consume.
By default, there is no limit.
acknowledgements:
enabled: null
print_interval_secs: 1
rate: integer
type: blackhole
Configuration for the clickhouse
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
database
optional
<oneOf>
A templated field.
The database that contains the table that data is inserted into.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
date_time_best_effort
optional
boolean
Sets date_time_input_format
to best_effort
, allowing ClickHouse to properly parse RFC3339/ISO 8601.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URI component of a request.
The endpoint of the ClickHouse server.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60skip_unknown_fields
optional
boolean
Sets input_format_skip_unknown_fields
, allowing ClickHouse to discard fields not present in the table schema.
table
required
string
A templated field.
The table that data is inserted into.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
auth: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: gzip
database: ''
date_time_best_effort: boolean
encoding: {}
endpoint: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
skip_unknown_fields: boolean
table: string
tls: ''
type: clickhouse
Configuration for the console
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
target: stdout
type: console
Configuration for the databend
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
none
optional
none
No compression.
database
optional
string
The database that contains the table that data is inserted into.
encoding
optional
Configures how events are encoded into raw bytes.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URI component of a request.
The endpoint of the Databend server.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60table
required
string
The table that data is inserted into.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
auth: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
database: default
encoding:
codec: json
endpoint: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
table: string
tls: ''
type: databend
Configuration for the datadog_archives
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
aws_s3
optional
<oneOf>
S3-specific configuration options.
Option 1
optional
S3-specific configuration options.
auth
optional
Configuration of the authentication strategy for interacting with AWS services.
private
optional
private
Bucket/object are private.
The bucket/object owner is granted the FULL_CONTROL
permission, and no one else has
access.
This is the default.
public-read
optional
public-read
Bucket/object can be read publicly.
The bucket/object owner is granted the FULL_CONTROL
permission, and anyone in the
AllUsers
grantee group is granted the READ
permission.
public-read-write
optional
public-read-write
Bucket/object can be read and written publicly.
The bucket/object owner is granted the FULL_CONTROL
permission, and anyone in the
AllUsers
grantee group is granted the READ
and WRITE
permissions.
This is generally not recommended.
aws-exec-read
optional
aws-exec-read
Bucket/object are private, and readable by EC2.
The bucket/object owner is granted the FULL_CONTROL
permission, and the AWS EC2 service is
granted the READ
permission for the purpose of reading Amazon Machine Image (AMI) bundles
from the given bucket.
authenticated-read
optional
authenticated-read
Bucket/object can be read by authenticated users.
The bucket/object owner is granted the FULL_CONTROL
permission, and anyone in the
AuthenticatedUsers
grantee group is granted the READ
permission.
bucket-owner-read
optional
bucket-owner-read
Object is private, except to the bucket owner.
The object owner is granted the FULL_CONTROL
permission, and the bucket owner is granted the READ
permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when specified for a bucket.
bucket-owner-full-control
optional
bucket-owner-full-control
Object is semi-private.
Both the object owner and bucket owner are granted the FULL_CONTROL
permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when specified for a bucket.
log-delivery-write
optional
log-delivery-write
Bucket can have logs written.
The LogDelivery
grantee group is granted WRITE
and READ_ACP
permissions.
Only relevant when specified for a bucket: this canned ACL is otherwise ignored when specified for an object.
For more information about logs, see Amazon S3 Server Access Logging.
grant_full_control
optional
string,null
Grants READ
, READ_ACP
, and WRITE_ACP
permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata, as well as read and modify the ACL on the created objects.
grant_read
optional
string,null
Grants READ
permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata.
grant_read_acp
optional
string,null
Grants READ_ACP
permissions on the created objects to the named grantee.
This allows the grantee to read the ACL on the created objects.
grant_write_acp
optional
string,null
Grants WRITE_ACP
permissions on the created objects to the named grantee.
This allows the grantee to modify the ACL on the created objects.
server_side_encryption
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
The Server-side Encryption algorithm used when storing these objects.
Option 1
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
More information on each algorithm can be found in the AWS documentation.
AES256
optional
AES256
Each object is encrypted with AES-256 using a unique key.
This corresponds to the SSE-S3
option.
aws:kms
optional
aws:kms
Each object is encrypted with AES-256 using keys managed by AWS KMS.
Depending on whether or not a KMS key ID is specified, this corresponds either to the
SSE-KMS
option (keys generated/managed by KMS) or the SSE-C
option (keys generated by
the customer, managed by KMS).
ssekms_key_id
optional
string,null
Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed customer master key (CMK) that is used for the created objects.
Only applies when server_side_encryption
is configured to use KMS.
If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.
storage_class
required
<oneOf>
The storage class for the created objects.
For more information, see Using Amazon S3 storage classes.
STANDARD
optional
STANDARD
Standard Redundancy.
REDUCED_REDUNDANCY
optional
REDUCED_REDUNDANCY
Reduced Redundancy.
INTELLIGENT_TIERING
optional
INTELLIGENT_TIERING
Intelligent Tiering.
STANDARD_IA
optional
STANDARD_IA
Infrequently Accessed.
ONEZONE_IA
optional
ONEZONE_IA
Infrequently Accessed (single Availability zone).
GLACIER
optional
GLACIER
Glacier Flexible Retrieval.
DEEP_ARCHIVE
optional
DEEP_ARCHIVE
Glacier Deep Archive.
tags
optional
object,null
The tag-set for the object.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
default: nullazure_blob
optional
<oneOf>
ABS-specific configuration options.
Option 1
optional
object
ABS-specific configuration options.
connection_string
required
string
The Azure Blob Storage Account connection string.
Authentication with access key is the only supported authentication method.
bucket
required
string
The name of the bucket to store the archives in.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
gcp_cloud_storage
optional
<oneOf>
GCS-specific configuration options.
Option 1
optional
GCS-specific configuration options.
authenticated-read
optional
authenticated-read
Bucket/object can be read by authenticated users.
The bucket/object owner is granted the OWNER
permission, and anyone authenticated Google
account holder is granted the READER
permission.
bucket-owner-full-control
optional
bucket-owner-full-control
Object is semi-private.
Both the object owner and bucket owner are granted the OWNER
permission.
Only relevant when specified for an object: this predefined ACL is otherwise ignored when specified for a bucket.
bucket-owner-read
optional
bucket-owner-read
Object is private, except to the bucket owner.
The object owner is granted the OWNER
permission, and the bucket owner is granted the
READER
permission.
Only relevant when specified for an object: this predefined ACL is otherwise ignored when specified for a bucket.
private
optional
private
Bucket/object are private.
The bucket/object owner is granted the OWNER
permission, and no one else has
access.
project-private
optional
project-private
Bucket/object are private within the project.
Project owners and project editors are granted the OWNER
permission, and anyone who is
part of the project team is granted the READER
permission.
This is the default.
public-read
optional
public-read
Bucket/object can be read publically.
The bucket/object owner is granted the OWNER
permission, and all other users, whether
authenticated or anonymous, are granted the READER
permission.
metadata
optional
object,null
The set of metadata key:value
pairs for the created objects.
For more information, see Custom metadata.
STANDARD
optional
STANDARD
Standard storage.
This is the default.
NEARLINE
optional
NEARLINE
Nearline storage.
COLDLINE
optional
COLDLINE
Coldline storage.
ARCHIVE
optional
ARCHIVE
Archive storage.
api_key
optional
<oneOf>
An API key.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS
environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
credentials_path
optional
string,null
Path to a service account credentials JSON file.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS
environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
key_prefix
optional
string,null
A prefix to apply to all object keys.
Prefixes are useful for partitioning objects, such as by creating an object key that
stores objects under a particular directory. If using a prefix for this purpose, it must end
in /
to act as a directory path. A trailing /
is not automatically added.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60service
required
string
The name of the object storage service to use.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
aws_s3: ''
azure_blob: ''
bucket: string
encoding: {}
gcp_cloud_storage: ''
key_prefix: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
service: string
tls: ''
type: datadog_archives
Configuration for the datadog_events
sink.
항목
required
유형
설명
region: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: datadog_events
Configuration for the datadog_logs
sink.
항목
required
유형
설명
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: ''
encoding: {}
region: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
headers: {}
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: datadog_logs
Configuration for the datadog_metrics
sink.
항목
required
유형
설명
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_namespace: string
region: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: datadog_metrics
Configuration for the datadog_traces
sink.
항목
required
유형
설명
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: datadog_traces
Configuration for the elasticsearch
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
api_version
optional
<oneOf>
The API version of Elasticsearch.
auto
optional
auto
Auto-detect the API version.
If the cluster state version endpoint isn't reachable, a warning is logged to
stdout, and the version is assumed to be V6 if the suppress_type_name
option is set to
true
. Otherwise, the version is assumed to be V8. In the future, the sink instead
returns an error during configuration parsing, since a wrongly assumed version could lead to
incorrect API calls.
v6
optional
v6
Use the Elasticsearch 6.x API.
v7
optional
v7
Use the Elasticsearch 7.x API.
v8
optional
v8
Use the Elasticsearch 8.x API.
auth
optional
<oneOf>
Elasticsearch Authentication strategies.
Option 1
optional
<oneOf>
Elasticsearch Authentication strategies.
Basic
optional
object
HTTP Basic Authentication.
password
required
string
Basic authentication password.
strategy
required
basic
HTTP Basic Authentication.
user
required
string
Basic authentication username.
Aws
optional
Amazon OpenSearch Service-specific authentication.
strategy
required
aws
Amazon OpenSearch Service-specific authentication.
aws
optional
<oneOf>
Configuration of the region/endpoint to use when interacting with an AWS service.
Option 1
optional
object
Configuration of the region/endpoint to use when interacting with an AWS service.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
default: nullbatch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullbulk
optional
object
Elasticsearch bulk mode configuration.
action
optional
string
Action to use when making requests to the Elasticsearch Bulk API.
Only index
and create
actions are supported.
index
optional
string
A templated field.
The name of the index to write events to.
default: vector-%Y.%m.%dcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
data_stream
optional
<oneOf>
Elasticsearch data stream mode configuration.
Option 1
optional
object
Elasticsearch data stream mode configuration.
auto_routing
optional
boolean
Automatically routes events by deriving the data stream name using specific event fields.
The format of the data stream name is <type>-<dataset>-<namespace>
, where each value comes
from the data_stream
configuration field of the same name.
If enabled, the value of the data_stream.type
, data_stream.dataset
, and
data_stream.namespace
event fields are used if they are present. Otherwise, the values
set in this configuration are used.
dataset
optional
string
A templated field.
The data stream dataset used to construct the data stream at index time.
namespace
optional
string
A templated field.
The data stream namespace used to construct the data stream at index time.
sync_fields
optional
boolean
Automatically adds and syncs the data_stream.*
event fields if they are missing from the event.
This ensures that fields match the name of the data stream that is receiving events.
type
optional
string
A templated field.
The data stream type used to construct the data stream at index time.
distribution
optional
<oneOf>
Options for determining the health of an endpoint.
Option 1
optional
object
Options for determining the health of an endpoint.
retry_initial_backoff_secs
optional
integer
Initial delay between attempts to reactivate endpoints once they become unhealthy.
retry_max_duration_secs
optional
integer
Maximum delay between attempts to reactivate endpoints once they become unhealthy.
doc_type
optional
string
The doc_type
for your index data.
This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string,null
The Elasticsearch endpoint to send logs to.
DEPRECATED: The endpoint must contain an HTTP scheme, and may specify a hostname or IP address and port.
endpoints
optional
[string]
A list of Elasticsearch endpoints to send logs to.
The endpoint must contain an HTTP scheme, and may specify a hostname or IP address and port.
id_key
optional
<oneOf>
The name of the event key that should map to Elasticsearch’s _id
field.
By default, the _id
field is not set, which allows Elasticsearch to set this
automatically. Setting your own Elasticsearch IDs can hinder performance.
Option 1
optional
string
A wrapper around OwnedValuePath
that allows it to be used in Vector config.
This requires a valid path to be used. If you want to allow optional paths,
use [optional_path::OptionalValuePath].
metrics
optional
<oneOf>
Configuration for the metric_to_log
transform.
Option 1
optional
object
Configuration for the metric_to_log
transform.
host_tag
optional
string,null
Name of the tag in the metric to use for the source host.
If present, the value of the tag is set on the generated log event in the host
field,
where the field key uses the global host_key
option.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments as
described by the native_json
codec.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
timezone
optional
<oneOf>
The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone.
This overrides the global timezone
option. The time zone name may be
any name in the TZ database or local
to indicate system local time.
Option 1
optional
<oneOf>
Timezone reference.
This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.
mode
optional
<oneOf>
Elasticsearch Indexing mode.
bulk
optional
bulk
Ingests documents in bulk, using the bulk API index
action.
data_stream
optional
data_stream
Ingests documents in bulk, using the bulk API create
action.
Elasticsearch Data Streams only support the create
action.
pipeline
optional
string,null
The name of the pipeline to apply.
query
optional
object,null
Custom parameters to add to the query string for each HTTP request sent to Elasticsearch.
request
optional
Outbound HTTP request settings.
headers
optional
object
Additional HTTP headers to add to every HTTP request.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
request_retry_partial
optional
boolean
Whether or not to retry successful requests containing partial failures.
To avoid duplicates in Elasticsearch, please use option id_key
.
suppress_type_name
optional
boolean
Whether or not to send the type
field to Elasticsearch.
DEPRECATED: The type
field was deprecated in Elasticsearch 7.x and removed in Elasticsearch 8.x.
If enabled, the doc_type
option is ignored.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
api_version: auto
auth: ''
aws: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
bulk:
action: index
index: vector-%Y.%m.%d
compression: none
data_stream: ''
distribution: ''
doc_type: _doc
encoding: {}
endpoint: string
endpoints: []
id_key: ''
metrics: ''
mode: bulk
pipeline: string
query: object
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
headers: {}
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
request_retry_partial: boolean
suppress_type_name: boolean
tls: ''
type: elasticsearch
Configuration for the gcp_chronicle_unstructured
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
customer_id: string
encoding: ''
endpoint: string
log_type: string
region: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
type: gcp_chronicle_unstructured
Configuration for the gcp_cloud_storage
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
acl: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
bucket: string
compression: none
filename_append_uuid: true
filename_extension: string
filename_time_format: '%s'
key_prefix: string
metadata: object
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
storage_class: ''
tls: ''
type: gcp_cloud_storage
Configuration for the gcp_pubsub
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
encoding: ''
endpoint: 'https://pubsub.googleapis.com'
project: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
topic: string
type: gcp_pubsub
Configuration for the gcp_stackdriver_logs
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
encoding: {}
log_id: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
resource: ''
severity_key: ''
tls: ''
type: gcp_stackdriver_logs
Configuration for the gcp_stackdriver_metrics
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_namespace: namespace
project_id: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
resource: ''
tls: ''
type: gcp_stackdriver_metrics
Configuration for the honeycomb
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
api_key
required
string
The API key that is used to authenticate against Honeycomb.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nulldataset
required
string
The dataset to which logs are sent.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60acknowledgements:
enabled: null
api_key: string
batch:
max_bytes: null
max_events: null
timeout_secs: null
dataset: string
encoding: {}
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: honeycomb
Configuration for the http
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
auth: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
headers: object
method: post
payload_prefix: string
payload_suffix: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
headers: {}
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
uri: string
type: http
Configuration for the humio_logs
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The base URL of the Humio instance.
The scheme (http
or https
) must be specified. No path should be included since the paths defined
by the Splunk
API are used.
event_type
optional
<oneOf>
The type of events sent to this sink. Humio uses this as the name of the parser to use to ingest the data.
If unset, Humio defaults it to none.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
host_key
optional
string
Overrides the name of the log field used to retrieve the hostname to send to Humio.
By default, the global log_schema.host_key
option is used.
index
optional
<oneOf>
Optional name of the repository to ingest into.
In public-facing APIs, this must (if present) be equal to the repository used to create the ingest token used for authentication.
In private cluster setups, Humio can be configured to allow these to be different.
For more information, see Humio’s Format of Data.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
indexed_fields
optional
[string]
Event fields to be added to Humio’s extra fields.
Can be used to tag events by specifying fields starting with #
.
For more information, see Humio’s Format of Data.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60source
optional
<oneOf>
The source of events sent to this sink.
Typically the filename the logs originated from. Maps to @source
in Humio.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
timestamp_key
optional
string
Overrides the name of the log field used to retrieve the timestamp to send to Humio.
By default, the global log_schema.timestamp_key
option is used.
timestamp_nanos_key
optional
string,null
Overrides the name of the log field used to retrieve the nanosecond-enabled timestamp to send to Humio.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
token
required
string
The Humio ingestion token.
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
encoding: ''
endpoint: 'https://cloud.humio.com'
event_type: ''
host_key: host
index: ''
indexed_fields: []
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
source: ''
timestamp_key: timestamp
timestamp_nanos_key: '@timestamp.nanos'
tls: ''
token: string
type: humio_logs
Configuration for the humio_metrics
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
endpoint: 'https://cloud.humio.com'
event_type: ''
host_key: host
index: ''
indexed_fields: []
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
source: ''
tls: ''
token: string
type: humio_metrics
Configuration for the influxdb_logs
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
encoding: {}
endpoint: string
host_key: ''
measurement: string
message_key: ''
namespace: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
source_type_key: ''
tags: []
tls: ''
type: influxdb_logs
Configuration for the influxdb_metrics
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_namespace: string
endpoint: string
quantiles:
- 0.5
- 0.75
- 0.9
- 0.95
- 0.99
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tags: object
tls: ''
type: influxdb_metrics
Configuration for the kafka
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
bootstrap_servers: string
compression: none
encoding: ''
headers_key: ''
key_field: ''
librdkafka_options: {}
message_timeout_ms: 300000
socket_timeout_ms: 60000
topic: string
type: kafka
Configuration for the logdna
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
api_key
required
string
The Ingestion API key.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nulldefault_app
optional
string
The default app that is set for events that do not contain a file
or app
field.
default_env
optional
string
The default environment that is set for events that do not contain an env
field.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The HTTP endpoint to send logs to.
Both IP address and hostname are accepted formats.
hostname
required
string
A templated field.
The hostname that is attached to each batch of events.
ip
optional
string,null
The IP address that is attached to each batch of events.
mac
optional
string,null
The MAC address that is attached to each batch of events.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tags
optional
array,null
The tags that are attached to each batch of events.
acknowledgements:
enabled: null
api_key: string
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_app: vector
default_env: production
encoding: {}
endpoint: 'https://logs.mezmo.com/'
hostname: string
ip: string
mac: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tags: array
type: logdna
Configuration for the loki
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
Original
optional
<oneOf>
Compression configuration.
Basic compression.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
Extended
optional
<oneOf>
Loki-specific compression.
snappy
optional
snappy
Snappy compression.
This implies sending push requests as Protocol Buffers.
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The base URL of the Loki instance.
The path
value is appended to this.
labels
optional
object
A set of labels that are attached to each batch of events.
Both keys and values are templateable, which enables you to attach dynamic labels to events.
Valid label keys include *
, and prefixes ending with *
, to allow for the expansion of
objects into multiple labels. See Label expansion for more information.
Note: If the set of labels has high cardinality, this can cause drastic performance issues with Loki. To prevent this from happening, reduce the number of unique label keys and values.
out_of_order_action
optional
<oneOf>
Out-of-order event behavior.
Some sources may generate events with timestamps that aren't in chronological order. Even though the sink sorts the events before sending them to Loki, there is a chance that another event could come in that is out of order with the latest events sent to Loki. Prior to Loki 2.4.0, this was not supported and would result in an error during the push request.
If you're using Loki 2.4.0 or newer, Accept
is the preferred action, which lets Loki handle
any necessary sorting/reordering. If you're using an earlier version, then you must use Drop
or RewriteTimestamp
depending on which option makes the most sense for your use case.
drop
optional
drop
Drop the event.
rewrite_timestamp
optional
rewrite_timestamp
Rewrite the timestamp of the event to the timestamp of the latest event seen by the sink.
accept
optional
accept
Accept the event.
The event is not dropped and is sent without modification.
Requires Loki 2.4.0 or newer.
path
optional
string
The path to use in the URL of the Loki instance.
remove_label_fields
optional
boolean
Whether or not to delete fields from the event when they are used as labels.
remove_timestamp
optional
boolean
Whether or not to remove the timestamp from the event payload.
The timestamp is still sent as event metadata for Loki to use for indexing.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tenant_id
optional
<oneOf>
The tenant ID to specify in requests to Loki.
When running Loki locally, a tenant ID is not required.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
auth: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: snappy
encoding: ''
endpoint: string
labels: object
out_of_order_action: drop
path: /loki/api/v1/push
remove_label_fields: boolean
remove_timestamp: true
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tenant_id: ''
tls: ''
type: loki
Configuration for the mezmo
(formerly logdna
) sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
api_key
required
string
The Ingestion API key.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nulldefault_app
optional
string
The default app that is set for events that do not contain a file
or app
field.
default_env
optional
string
The default environment that is set for events that do not contain an env
field.
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string
The HTTP endpoint to send logs to.
Both IP address and hostname are accepted formats.
hostname
required
string
A templated field.
The hostname that is attached to each batch of events.
ip
optional
string,null
The IP address that is attached to each batch of events.
mac
optional
string,null
The MAC address that is attached to each batch of events.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tags
optional
array,null
The tags that are attached to each batch of events.
acknowledgements:
enabled: null
api_key: string
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_app: vector
default_env: production
encoding: {}
endpoint: 'https://logs.mezmo.com/'
hostname: string
ip: string
mac: string
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tags: array
type: mezmo
Configuration for the nats
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auth
optional
<oneOf>
Configuration of the authentication strategy when interacting with NATS.
Option 1
optional
<oneOf>
Configuration of the authentication strategy when interacting with NATS.
UserPassword
optional
object
Username/password authentication.
strategy
required
user_password
Username/password authentication.
user_password
required
object
Username and password configuration.
password
required
string
Password.
user
required
string
Username.
Token
optional
object
Token authentication.
strategy
required
token
Token authentication.
token
required
object
Token configuration.
value
required
string
Token.
CredentialsFile
optional
object
Credentials file authentication. (JWT-based)
credentials_file
required
object
Credentials file configuration.
path
required
string
Path to credentials file.
strategy
required
credentials_file
Credentials file authentication. (JWT-based)
Nkey
optional
object
NKey authentication.
nkey
required
object
NKeys configuration.
nkey
required
string
User.
Conceptually, this is equivalent to a public key.
seed
required
string
Seed.
Conceptually, this is equivalent to a private key.
strategy
required
nkey
NKey authentication.
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
url
required
string
The NATS URL to connect to.
The URL must take the form of nats://server:port
.
If the port is not specified it defaults to 4222.
acknowledgements:
enabled: null
auth: ''
connection_name: vector
encoding: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
subject: string
tls: ''
url: string
type: nats
Configuration for the new_relic
sink.
항목
required
유형
설명
account_id
required
string
The New Relic account ID.
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
api
required
<oneOf>
New Relic API endpoint.
events
optional
events
Events API.
metrics
optional
metrics
Metrics API.
logs
optional
logs
Logs API.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
encoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
license_key
required
string
A valid New Relic license key.
region
optional
<oneOf>
New Relic region.
Option 1
optional
<oneOf>
New Relic region.
us
optional
us
US region.
eu
optional
eu
EU region.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60account_id: string
acknowledgements:
enabled: null
api: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: gzip
encoding: {}
license_key: string
region: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: new_relic
Configuration for the papertrail
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URI component of a request.
The TCP endpoint to send logs to.
keepalive
optional
<oneOf>
TCP keepalive settings for socket-based components.
Option 1
optional
object
TCP keepalive settings for socket-based components.
time_secs
optional
integer,null
The time to wait before starting to send TCP keepalive probes on an idle connection.
process
optional
string
A templated field.
The value to use as the process
in Papertrail.
send_buffer_bytes
optional
integer,null
Configures the send buffer size using the SO_SNDBUF
option on the socket.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
encoding: ''
endpoint: string
keepalive: ''
process: vector
send_buffer_bytes: integer
tls: ''
type: papertrail
Configuration for the prometheus_exporter
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
address
optional
string
The address to expose for scraping.
The metrics are exposed at the typical Prometheus exporter path, /metrics
.
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
default_namespace
optional
string,null
The default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with an underscore (_
).
It should follow the Prometheus naming conventions.
distributions_as_summaries
optional
boolean
Whether or not to render distributions as an aggregated histogram or aggregated summary.
While distributions as a lossless way to represent a set of samples for a metric is supported, Prometheus clients (the application being scraped, which is this sink) must aggregate locally into either an aggregated histogram or aggregated summary.
flush_period_secs
optional
integer
The interval, in seconds, on which metrics are flushed.
On the flush interval, if a metric has not been seen since the last flush interval, it is considered expired and is removed.
Be sure to configure this value higher than your client’s scrape interval.
suppress_timestamp
optional
boolean
Suppresses timestamps on the Prometheus output.
This can sometimes be useful when the source of metrics leads to their timestamps being too far in the past for Prometheus to allow them, such as when aggregating metrics over long time periods, or when replaying old metrics from a disk buffer.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: '0.0.0.0:9598'
auth: ''
buckets:
- 0.005
- 0.01
- 0.025
- 0.05
- 0.1
- 0.25
- 0.5
- 1
- 2.5
- 5
- 10
default_namespace: string
distributions_as_summaries: boolean
flush_period_secs: 60
quantiles:
- 0.5
- 0.75
- 0.9
- 0.95
- 0.99
suppress_timestamp: boolean
tls: ''
type: prometheus_exporter
Configuration for the prometheus_remote_write
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auth
optional
<oneOf>
Authentication strategies.
Option 1
optional
<oneOf>
Authentication strategies.
Basic
optional
object
HTTP Basic Authentication.
password
required
string
Basic authentication password.
strategy
required
basic
HTTP Basic Authentication.
user
required
string
Basic authentication username.
Bearer
optional
object
Bearer authentication.
A bearer token (OAuth2, JWT, etc) is passed as-is.
strategy
required
bearer
Bearer authentication.
A bearer token (OAuth2, JWT, etc) is passed as-is.
token
required
string
The bearer token to send.
Aws
optional
Amazon Prometheus Service-specific authentication.
strategy
required
aws
Amazon Prometheus Service-specific authentication.
aws
optional
<oneOf>
Configuration of the region/endpoint to use when interacting with an AWS service.
Option 1
optional
object
Configuration of the region/endpoint to use when interacting with an AWS service.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
default: nullbatch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Supported compression types for Prometheus Remote Write.
snappy
optional
snappy
Snappy.
gzip
optional
gzip
Gzip.
zstd
optional
zstd
Zstandard.
default_namespace
optional
string,null
The default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with an underscore (_
).
It should follow the Prometheus naming conventions.
endpoint
required
string
The endpoint to send data to.
The endpoint should include the scheme and the path to write to.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tenant_id
optional
<oneOf>
The tenant ID to send.
If set, a header named X-Scope-OrgID
is added to outgoing requests with the value of this setting.
This may be used by Cortex or other remote services to identify the tenant making the request.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
auth: ''
aws: ''
batch:
max_bytes: null
max_events: null
timeout_secs: null
buckets:
- 0.005
- 0.01
- 0.025
- 0.05
- 0.1
- 0.25
- 0.5
- 1
- 2.5
- 5
- 10
compression: snappy
default_namespace: string
endpoint: string
quantiles:
- 0.5
- 0.75
- 0.9
- 0.95
- 0.99
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tenant_id: ''
tls: ''
type: prometheus_remote_write
Configuration for the pulsar
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auth
optional
<oneOf>
Authentication configuration.
Option 1
optional
object
Authentication configuration.
name
optional
string,null
Basic authentication name/username.
This can be used either for basic authentication (username/password) or JWT authentication.
When used for JWT, the value should be token
.
oauth2
optional
<oneOf>
OAuth2-specific authentication configuration.
Option 1
optional
object
OAuth2-specific authentication configuration.
audience
optional
string,null
The OAuth2 audience.
credentials_url
required
string
The credentials URL.
A data URL is also supported.
issuer_url
required
string
The issuer URL.
scope
optional
string,null
The OAuth2 scope.
token
optional
<oneOf>
Basic authentication password/token.
This can be used either for basic authentication (username/password) or JWT authentication. When used for JWT, the value should be the signed JWT, in the compact representation.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch before it is flushed.
default: nullmax_events
optional
integer,null
The maximum amount of events in a batch before it is flushed.
Note this is an unsigned 32 bit integer which is a smaller capacity than many of the other sink batch settings.
default: nullcompression
optional
<oneOf>
Supported compression types for Pulsar.
none
optional
none
No compression.
lz4
optional
lz4
LZ4.
zlib
optional
zlib
Zlib.
zstd
optional
zstd
Zstandard.
snappy
optional
snappy
Snappy.
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The endpoint to which the Pulsar client should connect to.
The endpoint should specify the pulsar protocol and port.
partition_key_field
optional
<oneOf>
The log field name or tags key to use for the partition key.
If the field does not exist in the log event or metric tags, a blank value will be used.
If omitted, the key is not sent.
Pulsar uses a hash of the key to choose the topic-partition or uses round-robin if the record has no key.
Option 1
optional
string
An optional path that deserializes an empty string to None
.
producer_name
optional
string,null
The name of the producer. If not specified, the default name assigned by Pulsar is used.
properties_key
optional
<oneOf>
The log field name to use for the Pulsar properties key.
If omitted, no properties will be written.
Option 1
optional
string
An optional path that deserializes an empty string to None
.
topic
required
string
A templated field.
The Pulsar topic name to write events to.
acknowledgements:
enabled: null
auth: ''
batch:
max_bytes: null
max_events: null
compression: none
encoding: ''
endpoint: string
partition_key_field: ''
producer_name: string
properties_key: ''
topic: string
type: pulsar
Configuration for the redis
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nulldata_type
optional
<oneOf>
Redis data type to store messages in.
list
optional
list
The Redis list
type.
This resembles a deque, where messages can be popped and pushed from either end.
This is the default.
channel
optional
channel
The Redis channel
type.
Redis channels function in a pub/sub fashion, allowing many-to-many broadcasting and receiving.
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
string
The URL of the Redis endpoint to connect to.
The URL must take the form of protocol://server:port/db
where the protocol can either be
redis
or rediss
for connections secured via TLS.
key
required
string
A templated field.
The Redis key to publish messages to.
list_option
optional
<oneOf>
List-specific options.
Option 1
optional
object
List-specific options.
method
required
<oneOf>
The method to use for pushing messages into a list
.
rpush
optional
rpush
Use the rpush
method.
This pushes messages onto the tail of the list.
This is the default.
lpush
optional
lpush
Use the lpush
method.
This pushes messages onto the head of the list.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
data_type: list
encoding: ''
endpoint: string
key: string
list_option: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
type: redis
Configuration for the sematext_logs
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullencoding
optional
object
Transformations to prepare an event for serialization.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
optional
string,null
The endpoint to send data to.
Setting this option overrides the region
option.
region
optional
<oneOf>
The Sematext region to send data to.
us
optional
us
United States
eu
optional
eu
Europe
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60token
required
string
The token that is used to write to Sematext.
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
encoding: {}
endpoint: string
region: us
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
token: string
type: sematext_logs
Configuration for the sematext_metrics
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nulldefault_namespace
required
string
Sets the default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.
).
endpoint
optional
string,null
The endpoint to send data to.
Setting this option overrides the region
option.
region
optional
<oneOf>
The Sematext region to send data to.
us
optional
us
United States
eu
optional
eu
Europe
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60token
required
string
The token that is used to write to Sematext.
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_namespace: string
endpoint: string
region: us
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
token: string
type: sematext_metrics
Configuration for the splunk_hec_logs
sink.
항목
required
유형
설명
acknowledgements
optional
Splunk HEC acknowledgement configuration.
indexer_acknowledgements_enabled
optional
boolean
Controls if the sink integrates with Splunk HEC indexer acknowledgements for end-to-end acknowledgements.
max_pending_acks
optional
integer
The maximum number of pending acknowledgements from events sent to the Splunk HEC collector.
Once reached, the sink begins applying backpressure.
query_interval
optional
integer
The amount of time to wait between queries to the Splunk HEC indexer acknowledgement endpoint.
retry_limit
optional
integer
The maximum number of times an acknowledgement ID is queried for its status.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
auto_extract_timestamp
optional
boolean,null
Passes the auto_extract_timestamp
option to Splunk.
This option is only relevant to Splunk v8.x and above, and is only applied when
endpoint_target
is set to event
.
Setting this to true
causes Splunk to extract the timestamp from the message text
rather than use the timestamp embedded in the event. The timestamp must be in the format
yyyy-mm-dd hh:mm:ss
.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
default_token
required
string
Default Splunk HEC token.
If an event has a token set in its secrets (splunk_hec_token
), it prevails over the one set here.
encoding
required
Configures how events are encoded into raw bytes.
avro
required
object
Apache Avro-specific encoder options.
schema
required
string
The Avro schema.
Csv
optional
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
csv
required
object
The CSV Serializer Options.
capacity
optional
integer
Set the capacity (in bytes) of the internal buffer used in the CSV writer. This defaults to a reasonable setting.
delimiter
optional
integer
The field delimiter to use when writing CSV.
double_quote
optional
boolean
Enable double quote escapes.
This is enabled by default, but it may be disabled. When disabled, quotes in field data are escaped instead of doubled.
escape
optional
integer
The escape character to use when writing CSV.
In some variants of CSV, quotes are escaped using a special escape character like \ (instead of escaping quotes by doubling them).
To use this double_quotes
needs to be disabled as well otherwise it is ignored
fields
required
[string]
Configures the fields that will be encoded, as well as the order in which they appear in the output.
If a field is not present in the event, the output will be an empty string.
Values of type Array
, Object
, and Regex
are not supported and the
output will be an empty string.
quote
optional
integer
The quote character to use when writing CSV.
quote_style
optional
<oneOf>
The quoting style to use when writing CSV data.
always
optional
always
This puts quotes around every field. Always.
necessary
optional
necessary
This puts quotes around fields only when necessary. They are necessary when fields contain a quote, delimiter or record terminator. Quotes are also necessary when writing an empty record (which is indistinguishable from a record with one empty field).
non_numeric
optional
non_numeric
This puts quotes around all fields that are non-numeric. Namely, when writing a field that does not parse as a valid float or integer, then quotes will be used even if they aren’t strictly necessary.
never
optional
never
This never writes quotes, even if it would produce invalid CSV data.
codec
required
csv
Encodes an event as a CSV message.
This codec must be configured with fields to encode.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
Native
optional
object
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
codec
required
native
Encodes an event in the native Protocol Buffers format.
This codec is experimental.
RawMessage
optional
object
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This encoding uses the message
field of a log event.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single
, only the last non-bare value of tags are displayed with the
metric. When set to full
, all metric tags are exposed as separate assignments.
single
optional
single
Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.
full
optional
full
All tags are exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This encoding uses the message
field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Be careful if you are modifying your log events (for example, by using a remap
transform) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
except_fields
optional
array,null
List of fields that are excluded from the encoded event.
only_fields
optional
array,null
List of fields that are included in the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
Option 1
optional
<oneOf>
The format in which a timestamp should be represented.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
endpoint
required
uri
The base URL of the Splunk instance.
The scheme (http
or https
) must be specified. No path should be included since the paths defined
by the Splunk
API are used.
endpoint_target
optional
<oneOf>
Splunk HEC endpoint configuration.
raw
optional
raw
Events are sent to the raw endpoint.
When the raw endpoint is used, configured event metadata is sent as
query parameters on the request, except for the timestamp
field.
event
optional
event
Events are sent to the event endpoint.
When the event endpoint is used, configured event metadata is sent directly with each event.
host_key
optional
string
Overrides the name of the log field used to retrieve the hostname to send to Splunk HEC.
By default, the global log_schema.host_key
option is used.
index
optional
<oneOf>
The name of the index to send events to.
If not specified, the default index defined within Splunk is used.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60source
optional
<oneOf>
The source of events sent to this sink.
This is typically the filename the logs originated from.
If unset, the Splunk collector sets it.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
sourcetype
optional
<oneOf>
The sourcetype of events sent to this sink.
If unset, Splunk defaults to httpevent
.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
timestamp_key
optional
string
Overrides the name of the log field used to retrieve the timestamp to send to Splunk HEC.
When set to “”
, a timestamp is not set in the events sent to Splunk HEC.
By default, the global log_schema.timestamp_key
option is used.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
indexer_acknowledgements_enabled: true
max_pending_acks: 1000000
query_interval: 10
retry_limit: 30
auto_extract_timestamp: boolean
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
default_token: string
encoding: ''
endpoint: string
endpoint_target: event
host_key: host
index: ''
indexed_fields: []
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
source: ''
sourcetype: ''
timestamp_key: timestamp
tls: ''
type: splunk_hec_logs
Configuration of the splunk_hec_metrics
sink.
항목
required
유형
설명
acknowledgements
optional
Splunk HEC acknowledgement configuration.
indexer_acknowledgements_enabled
optional
boolean
Controls if the sink integrates with Splunk HEC indexer acknowledgements for end-to-end acknowledgements.
max_pending_acks
optional
integer
The maximum number of pending acknowledgements from events sent to the Splunk HEC collector.
Once reached, the sink begins applying backpressure.
query_interval
optional
integer
The amount of time to wait between queries to the Splunk HEC indexer acknowledgement endpoint.
retry_limit
optional
integer
The maximum number of times an acknowledgement ID is queried for its status.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
Option 2
optional
object
Compression algorithm and compression level.
level
optional
enum
Compression level.
Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
default_namespace
optional
string,null
Sets the default namespace for any metrics sent.
This namespace is only used if a metric has no existing namespace. When a namespace is
present, it is used as a prefix to the metric name, and separated with a period (.
).
default_token
required
string
Default Splunk HEC token.
If an event has a token set in its metadata, it prevails over the one set here.
endpoint
required
uri
The base URL of the Splunk instance.
The scheme (http
or https
) must be specified. No path should be included since the paths defined
by the Splunk
API are used.
host_key
optional
string
Overrides the name of the log field used to retrieve the hostname to send to Splunk HEC.
By default, the global log_schema.host_key
option is used.
index
optional
<oneOf>
The name of the index where to send the events to.
If not specified, the default index defined within Splunk is used.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60source
optional
<oneOf>
The source of events sent to this sink.
This is typically the filename the logs originated from.
If unset, the Splunk collector sets it.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
sourcetype
optional
<oneOf>
The sourcetype of events sent to this sink.
If unset, Splunk defaults to httpevent
.
Option 1
optional
string
A templated field.
In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.
By using Template
, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log
. An example of a template string is my-file-{{key}}.log
, where {{key}}
is the key's value when the template is rendered into a string.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
acknowledgements:
indexer_acknowledgements_enabled: true
max_pending_acks: 1000000
query_interval: 10
retry_limit: 30
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: none
default_namespace: string
default_token: string
endpoint: string
host_key: host
index: ''
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
source: ''
sourcetype: ''
tls: ''
type: splunk_hec_metrics
Configuration for the statsd
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
default_namespace: string
type: statsd
Configuration for the vector
sink.
항목
required
유형
설명
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
enabled
optional
boolean,null
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, waits for events to be acknowledged by the sink before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements
configuration.
address
required
uri
The downstream Vector address to which to connect.
Both IP address and hostname are accepted formats.
The address must include a port.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that is processed by a sink.
This is based on the uncompressed size of the batched events, before they are serialized/compressed.
default: nullmax_events
optional
integer,null
The maximum size of a batch before it is flushed.
default: nulltimeout_secs
optional
number,null
The maximum age of a batch before it is flushed.
default: nullcompression
optional
boolean
Whether or not to compress requests.
If set to true
, requests are compressed with gzip
.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.
default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"initial_concurrency":1,"rtt_deviation_scale":2.5}decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0
and less than 1
. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0
and less than 1
.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.
default: 0.4initial_concurrency
optional
integer
The initial concurrency limit to use. If not specified, the initial limit will be 1 (no concurrency).
It is recommended to set this value to your service's average limit if you're seeing that it takes a
long time to ramp up adaptive concurrency after a restart. You can find this value by looking at the
adaptive_concurrency_limit
metric.
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0
, and we expect reasonable values to range from 1.0
to 3.0
.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
adaptive
optional
adaptive
Concurrency will be managed by Vector's Adaptive Request Concurrency feature.
Fixed
optional
integer
A fixed amount of concurrency will be allowed.
rate_limit_duration_secs
optional
integer,null
The time window used for the rate_limit_num
option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs
time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence is used to select future backoffs.
default: 1retry_max_duration_secs
optional
integer,null
The maximum amount of time to wait between retries.
default: 3600timeout_secs
optional
integer,null
The time a request can take before being aborted.
Datadog highly recommends that you do not lower this value below the service's internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
version
optional
<oneOf>
Version of the configuration.
Option 1
optional
<oneOf>
Marker type for the version two of the configuration for the vector
sink.
2
optional
2
Marker value for version two.
acknowledgements:
enabled: null
address: string
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: boolean
request:
adaptive_concurrency:
decrease_ratio: 0.9
ewma_alpha: 0.4
initial_concurrency: 1
rtt_deviation_scale: 2.5
rate_limit_duration_secs: 1
rate_limit_num: 9223372036854776000
retry_attempts: 9223372036854776000
retry_initial_backoff_secs: 1
retry_max_duration_secs: 3600
timeout_secs: 60
tls: ''
version: ''
type: vector
Configuration for the webhdfs
sink.
항목
required
유형
설명
acknowledgements:
enabled: null
batch:
max_bytes: null
max_events: null
timeout_secs: null
compression: gzip
endpoint: string
prefix: string
root: string
type: webhdfs