Integration version1.13.0
Overview
The Solr check tracks the state and performance of a Solr cluster. It collects metrics for the number of documents indexed, cache hits and evictions, average request times, average requests per second, and more.
Setup
Installation
The Solr check is included in the Datadog Agent package, so you don’t need to install anything else on your Solr nodes.
This check is JMX-based, so you need to enable JMX Remote on your Solr servers. See the JMX Check documentation for more details.
Configuration
Host
To configure this check for an Agent running on a host:
Edit the solr.d/conf.yaml
file, in the conf.d/
folder at the root of your Agent’s configuration directory. See the sample solr.d/conf.yaml for all available configuration options.
init_config:
## @param is_jmx - boolean - required
## Whether or not this file is a configuration for a JMX integration.
#
is_jmx: true
## @param collect_default_metrics - boolean - required
## Whether or not the check should collect all default metrics.
#
collect_default_metrics: true
instances:
## @param host - string - required
## Solr host to connect to.
- host: localhost
## @param port - integer - required
## Solr port to connect to.
port: 9999
Restart the Agent.
List of metrics
The conf
parameter is a list of metrics to be collected by the integration. Only 2 keys are allowed:
include
(mandatory): A dictionary of filters, any attribute that matches these filters are collected unless it also matches the exclude
filters (see below).exclude
(optional): A dictionary of filters, attributes that match these filters are not collected.
For a given bean, metrics get tagged in the following manner:
mydomain:attr0=val0,attr1=val1
In this example, your metric is mydomain
(or some variation depending on the attribute inside the bean) and has the tags attr0:val0
, attr1:val1
, and domain:mydomain
.
If you specify an alias in an include
key that is formatted as camel case, it is converted to snake case. For example, MyMetricName
is shown in Datadog as my_metric_name
.
The attribute filter
The attribute
filter can accept two types of values:
A dictionary whose keys are attributes names (see below). For this case, you can specify an alias for the metric that becomes the metric name in Datadog. You can also specify the metric type as a gauge or counter. If you choose counter, a rate per second is computed for the metric.
conf:
- include:
attribute:
maxThreads:
alias: tomcat.threads.max
metric_type: gauge
currentThreadCount:
alias: tomcat.threads.count
metric_type: gauge
bytesReceived:
alias: tomcat.bytes_rcvd
metric_type: counter
A list of attributes names (see below). For this case, the metric type is a gauge, and the metric name is jmx.\[DOMAIN_NAME].\[ATTRIBUTE_NAME]
.
conf:
- include:
domain: org.apache.cassandra.db
attribute:
- BloomFilterDiskSpaceUsed
- BloomFilterFalsePositives
- BloomFilterFalseRatio
- Capacity
- CompressionRatio
- CompletedTasks
- ExceptionCount
- Hits
- RecentHitRate
Older versions
List of filters is only supported in Datadog Agent > 5.3.0. If you are using an older version, use singletons and multiple include
statements instead.
# Datadog Agent > 5.3.0
conf:
- include:
domain: domain_name
bean:
- first_bean_name
- second_bean_name
# Older Datadog Agent versions
conf:
- include:
domain: domain_name
bean: first_bean_name
- include:
domain: domain_name
bean: second_bean_name
Containerized
For containerized environments, see the Autodiscovery with JMX guide.
Log collection
Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml
file:
Solr uses the log4j
logger by default. To customize the logging format, edit the server/resources/log4j2.xml
file. By default, Datadog’s integration pipeline supports the following conversion pattern:
%maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n
Clone and edit the integration pipeline if you have a different format.
Uncomment and edit the logs configuration block in your solr.d/conf.yaml
file. Change the type
, path
, and service
parameter values based on your environment. See the sample solr.d/solr.yaml for all available configuration options.
logs:
- type: file
path: /var/solr/logs/solr.log
source: solr
# To handle multi line that starts with yyyy-mm-dd use the following pattern
# log_processing_rules:
# - type: multi_line
# pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])
# name: new_log_start_with_date
Restart the Agent.
To enable logs for Kubernetes environments, see Kubernetes Log Collection.
Validation
Run the Agent’s status subcommand and look for solr
under the Checks section.
Data Collected
Metrics
solr.document_cache.evictions (gauge) | The total number of cache evictions per second. Shown as eviction |
solr.document_cache.hits (gauge) | The number of cache hits per second. Shown as hit |
solr.document_cache.inserts (gauge) | The total number of cache inserts per second. Shown as set |
solr.document_cache.lookups (gauge) | The total number of cache lookups per second. Shown as get |
solr.filter_cache.evictions (gauge) | The total number of cache evictions per second. Shown as eviction |
solr.filter_cache.hits (gauge) | The number of cache hits per second. Shown as hit |
solr.filter_cache.inserts (gauge) | The total number of cache inserts per second. Shown as set |
solr.filter_cache.lookups (gauge) | The total number of cache lookups per second. Shown as get |
solr.query_result_cache.evictions (gauge) | The total number of cache evictions per second. Shown as eviction |
solr.query_result_cache.hits (gauge) | The number of cache hits per second. Shown as hit |
solr.query_result_cache.inserts (gauge) | The total number of cache inserts per second. Shown as set |
solr.query_result_cache.lookups (gauge) | The total number of cache lookups per second. Shown as get |
solr.search_handler.errors (gauge) | Number of errors per second encountered by the handler. Shown as error |
solr.search_handler.request_times.50percentile (gauge) | Request processing time in milliseconds (50percentile). Shown as millisecond |
solr.search_handler.request_times.75percentile (gauge) | Request processing time in milliseconds (75percentile). Shown as millisecond |
solr.search_handler.request_times.95percentile (gauge) | Request processing time in milliseconds (95percentile). Shown as millisecond |
solr.search_handler.request_times.98percentile (gauge) | Request processing time in milliseconds (98percentile). Shown as millisecond |
solr.search_handler.request_times.999percentile (gauge) | Request processing time in milliseconds (999percentile). Shown as millisecond |
solr.search_handler.request_times.99percentile (gauge) | Request processing time in milliseconds (99percentile). Shown as millisecond |
solr.search_handler.request_times.mean (gauge) | The average time per request. Shown as millisecond |
solr.search_handler.request_times.mean_rate (gauge) | Average number of requests received per second since the Solr core was first created. Shown as request |
solr.search_handler.request_times.one_minute_rate (gauge) | Requests per second received over the past minutes. Shown as request |
solr.search_handler.requests (gauge) | Number of requests per second processed by the handler. Shown as request |
solr.search_handler.time (gauge) | The sum of all request processing times (in milliseconds) per second. |
solr.search_handler.timeouts (gauge) | Number of responses per second received with partial results. Shown as timeout |
solr.searcher.maxdocs (gauge) | One greater than the largest possible document number. Shown as document |
solr.searcher.numdocs (gauge) | The total number of indexed documents. Shown as document |
solr.searcher.warmup (gauge) | The time spent warming up. Shown as millisecond |
Events
The Solr check does not include any events.
Service Checks
solr.can_connect
Returns CRITICAL
if the Agent is unable to connect to and collect metrics from the monitored SolR instance, WARNING
if no metrics are collected, and OK
otherwise.
Statuses: ok, critical, warning
Troubleshooting
Commands to view the available metrics
The datadog-agent jmx
command was added in version 4.1.0.
- List attributes that match at least one of your instances configuration:
sudo datadog-agent jmx list matching
- List attributes that do match one of your instances configuration but that are not being collected because it would exceed the number of metrics that can be collected:
sudo datadog-agent jmx list limited
- List attributes expected to be collected by your current instances configuration:
sudo datadog-agent jmx list collected
- List attributes that don’t match any of your instances configuration:
sudo datadog-agent jmx list not-matching
- List every attributes available that has a type supported by JMXFetch:
sudo datadog-agent jmx list everything
- Start the collection of metrics based on your current configuration and display them in the console:
sudo datadog-agent jmx collect
Further Reading
Parsing a string value into a number
If your jmxfetch returns only string values like false and true and you want to transform it into a Datadog gauge metric for advanced usages. For instance if you want the following equivalence for your jmxfetch:
"myJmxfetch:false" = myJmxfetch:0
"myJmxfetch:true" = myJmxfetch:1
You may use the attribute
filter as follow:
# ...
attribute:
myJmxfetch:
alias: your_metric_name
metric_type: gauge
values:
"false": 0
"true": 1