Tomcat

Supported OS Linux Windows Mac OS

Integration version4.0.0

Tomcat Dashboard

Overview

This check collects Tomcat metrics, for example:

  • Overall activity metrics: error count, request count, processing times, etc.
  • Thread pool metrics: thread count, number of threads busy, etc.
  • Servlet processing times

Setup

Installation

The Tomcat check is included in the Datadog Agent package, so you don’t need to install anything else on your Tomcat servers.

This check is JMX-based, so you need to enable JMX Remote on your Tomcat servers. Follow the instructions in Monitoring and Managing Tomcat.

Configuration

Host

To configure this check for an Agent running on a host:

  1. Edit the tomcat.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to collect Tomcat metrics and logs. See the sample tomcat.d/conf.yaml for all available configuration options.

  2. Restart the Agent.

See the JMX Check documentation for a list of configuration options usable by all JMX-based checks.

List of metrics

The conf parameter is a list of metrics to be collected by the integration. Only two keys are allowed:

  • include (mandatory): A dictionary of filters. Any attribute that matches these filters is collected unless it also matches the exclude filters (see below).
  • exclude (optional): A dictionary of filters. Attributes that match these filters are not collected.

For a given bean, metrics get tagged in the following manner:

mydomain:attr0=val0,attr1=val1

In this example, your metric is mydomain (or some variation depending on the attribute inside the bean) and has the tags attr0:val0, attr1:val1, and domain:mydomain.

If you specify an alias in an include key that is formatted as camel case, it is converted to snake case. For example, MyMetricName is shown in Datadog as my_metric_name.

The attribute filter

The attribute filter can accept two types of values:

  • A dictionary whose keys are attributes names (see below). For this case, you can specify an alias for the metric that becomes the metric name in Datadog. You can also specify the metric type as a gauge or counter. If you choose counter, a rate per second is computed for the metric.

    conf:
      - include:
        attribute:
          maxThreads:
            alias: tomcat.threads.max
            metric_type: gauge
          currentThreadCount:
            alias: tomcat.threads.count
            metric_type: gauge
          bytesReceived:
            alias: tomcat.bytes_rcvd
            metric_type: counter
    
  • A list of attributes names (see below). For this case, the metric type is a gauge, and the metric name is jmx.\[DOMAIN_NAME].\[ATTRIBUTE_NAME].

    conf:
      - include:
        domain: org.apache.cassandra.db
        attribute:
          - BloomFilterDiskSpaceUsed
          - BloomFilterFalsePositives
          - BloomFilterFalseRatio
          - Capacity
          - CompressionRatio
          - CompletedTasks
          - ExceptionCount
          - Hits
          - RecentHitRate
    

Log collection

  1. To submit logs to Datadog, Tomcat uses the log4j logger. For versions of Tomcat before 8.0, log4j is configured by default. For Tomcat 8.0+, you must configure Tomcat to use log4j, see Using Log4j. In the first step of those instructions, edit the log4j.properties file in the $CATALINA_BASE/lib directory as follows:

      log4j.rootLogger = INFO, CATALINA
    
      # Define all the appenders
      log4j.appender.CATALINA = org.apache.log4j.DailyRollingFileAppender
      log4j.appender.CATALINA.File = /var/log/tomcat/catalina.log
      log4j.appender.CATALINA.Append = true
    
      # Roll-over the log once per day
      log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout
      log4j.appender.CATALINA.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n
    
      log4j.appender.LOCALHOST = org.apache.log4j.DailyRollingFileAppender
      log4j.appender.LOCALHOST.File = /var/log/tomcat/localhost.log
      log4j.appender.LOCALHOST.Append = true
      log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout
      log4j.appender.LOCALHOST.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n
    
      log4j.appender.MANAGER = org.apache.log4j.DailyRollingFileAppender
      log4j.appender.MANAGER.File = /var/log/tomcat/manager.log
      log4j.appender.MANAGER.Append = true
      log4j.appender.MANAGER.layout = org.apache.log4j.PatternLayout
      log4j.appender.MANAGER.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n
    
      log4j.appender.HOST-MANAGER = org.apache.log4j.DailyRollingFileAppender
      log4j.appender.HOST-MANAGER.File = /var/log/tomcat/host-manager.log
      log4j.appender.HOST-MANAGER.Append = true
      log4j.appender.HOST-MANAGER.layout = org.apache.log4j.PatternLayout
      log4j.appender.HOST-MANAGER.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n
    
      log4j.appender.CONSOLE = org.apache.log4j.ConsoleAppender
      log4j.appender.CONSOLE.layout = org.apache.log4j.PatternLayout
      log4j.appender.CONSOLE.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n
    
      # Configure which loggers log to which appenders
      log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost] = INFO, LOCALHOST
      log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] =\
        INFO, MANAGER
      log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager] =\
        INFO, HOST-MANAGER
    

    Then follow the remaining steps in the Tomcat docs for configuring log4j.

  2. By default, Datadog’s integration pipeline support the following conversion patterns:

      %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
      %d [%t] %-5p %c - %m%n
    

    Clone and edit the integration pipeline if you have a different format. See Logging in Tomcat for details on Tomcat logging capabilities.

  3. Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml file:

    logs_enabled: true
    
  4. Add this configuration block to your tomcat.d/conf.yaml file to start collecting your Tomcat Logs:

    logs:
      - type: file
        path: /var/log/tomcat/*.log
        source: tomcat
        service: "<SERVICE>"
        #To handle multi line that starts with yyyy-mm-dd use the following pattern
        #log_processing_rules:
        #  - type: multi_line
        #    name: log_start_with_date
        #    pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])
    

    Change the path and service parameter values and configure them for your environment. See the sample tomcat.yaml for all available configuration options.

  5. Restart the Agent.

Containerized

For containerized environments, see the Autodiscovery with JMX guide.

Validation

Run the Agent’s status subcommand and look for tomcat under the Checks section.

Data Collected

Metrics

tomcat.bytes_rcvd
(gauge)
Bytes per second received by all request processors.
Shown as byte
tomcat.bytes_sent
(gauge)
Bytes per second sent by all the request processors.
Shown as byte
tomcat.error_count
(gauge)
The number of errors per second on all request processors.
Shown as error
tomcat.jdbc.connection_pool.active
(gauge)
The number of established connections in the pool that are in use.
tomcat.jdbc.connection_pool.idle
(gauge)
The number of established connections in the pool that are idle.
tomcat.jdbc.connection_pool.max_active
(gauge)
The maximum number of open connections.
tomcat.jdbc.connection_pool.max_idle
(gauge)
The maximum number of idle connections.
tomcat.jdbc.connection_pool.min_idle
(gauge)
The minimum number of idle connections.
tomcat.jdbc.connection_pool.size
(gauge)
The number of established connections in the pool, idle and in use.
tomcat.jsp.count
(gauge)
The number of JSPs per second that have been loaded in the web module.
Shown as page
tomcat.jsp.reload_count
(gauge)
The number of JSPs per second that have been reloaded in the web module.
Shown as page
tomcat.max_time
(gauge)
The longest request processing time (in milliseconds).
Shown as millisecond
tomcat.min_time
(gauge)
The shortest request processing time (in milliseconds).
Shown as millisecond
tomcat.processing_time
(gauge)
The sum of request processing times across all requests handled by the request processors (in milliseconds) per second.
tomcat.request_count
(gauge)
The number of requests per second across all request processors.
Shown as request
tomcat.servlet.error_count
(gauge)
The number of erroneous requests received by the servlet per second.
Shown as error
tomcat.servlet.max_time
(gauge)
The maximum processing time of a request
Shown as millisecond
tomcat.servlet.min_time
(gauge)
The minimum processing time of a request
Shown as millisecond
tomcat.servlet.processing_time
(gauge)
The sum of request processing times across all requests to the servlet (in milliseconds) per second.
tomcat.servlet.request_count
(gauge)
The number of requests received by the servlet per second.
Shown as request
tomcat.string_cache.access_count
(gauge)
The number of accesses to the string cache per second.
Shown as get
tomcat.string_cache.cache_size
(gauge)
The size of the String cache
Shown as byte
tomcat.string_cache.hit_count
(gauge)
The number of string cache hits per second.
Shown as hit
tomcat.string_cache.max_size
(gauge)
The maximum size of the String cache
Shown as byte
tomcat.threads.busy
(gauge)
The number of threads that are in use.
Shown as thread
tomcat.threads.count
(gauge)
The number of threads managed by the thread pool.
Shown as thread
tomcat.threads.max
(gauge)
The maximum number of allowed worker threads.
Shown as thread
tomcat.threads.min
(gauge)
The minimum number of allowed worker threads.
Shown as thread
tomcat.web.cache.hit_count
(gauge)
The number of web resource cache hits per second.
Shown as hit
tomcat.web.cache.lookup_count
(gauge)
The number of lookups to the web resource cache per second.
Shown as get

Events

The Tomcat check does not include any events.

Service Checks

tomcat.can_connect
Returns CRITICAL if the Agent is unable to connect to and collect metrics from the monitored Tomcat instance, WARNING if no metrics are collected, and OK otherwise.
Statuses: ok, critical, warning

Troubleshooting

Missing tomcat.* metrics

The Datadog Agent collects JMX metrics with either Catalina or Tomcat as bean domain names with the Datadog Agent version 7.49.0 or later. Older versions only collect metrics with Catalina as the bean domain name. Standalone Tomcat deployments have metrics under domain Catalina, but embedded Tomcat deployments (such as with Spring Boot) have metrics under domain Tomcat.

If the Datadog Agent version is older than 7.49.0, and if the exposed Tomcat metrics are prefixed with a different bean domain name such as Tomcat, copy the default metrics from the metrics.yaml file to the conf section of the tomcat.d/conf.yaml file and modify the domain filter to use the applicable bean domain name.

- include:
    domain: Tomcat
    type: ThreadPool
    attribute:
      maxThreads:
        alias: tomcat.threads.max
        metric_type: gauge
      currentThreadCount:
        alias: tomcat.threads.count
        metric_type: gauge
      currentThreadsBusy:
        alias: tomcat.threads.busy
        metric_type: gauge

See the JMX Check documentation for more detailed information.

Commands to view the available metrics

The datadog-agent jmx command allows you to run troubleshooting commands on JMXFetch integrations. On Linux systems, you will need to prepend the command with sudo -u dd-agent so that the Datadog Agent runs as the correct user.

datadog-agent jmx collect

Running datadog-agent jmx collect starts the collection of metrics based on your current configuration and displays them in the console.

datadog-agent jmx list

The datadog-agent jmx list has a number of available subcommands:

  • collected - List attributes that will actually be collected by your current instance’s configuration.
  • everything - List every attribute available that has a type supported by JMXFetch.
  • limited - List attributes that match one of your instances’ configurations but that are not being collected because it would exceed the number of metrics that can be collected.
  • matching - List attributes that match at least one of your instances’ configurations.
  • not-matching - List attributes that don’t match any of your instances’ configurations.
  • with-metrics - List attributes and metrics data that match at least one of your instances’ configurations.
  • with-rate-metrics - List attributes and metrics data that match at least one of your instances’ configurations, including rates and counters.

Further Reading

Additional helpful documentation, links, and articles:

PREVIEWING: rtrieu/product-analytics-ui-changes