Your application has finally been tested and deployed in a production environment. However, the IT operators are complaining that the application is consuming more system resources than originally expected. The problem being described by the IT operators cannot be repeated in the development environment. To establish the cause of these problems, it's important to get enough information about the runtime environment, more specifically what is going on inside the Java Virtual Machine (JVM).

Before J2SE 5.0, the information available from the JVM was at best extremely limited. JSR 174 has added management and monitoring APIs in J2SE 5.0 that expose valuable JVM information. The exposed information ranges from JVM health indicators like memory and threads to class loading and garbage collection information. This article provides an introduction to manageability and describes how the monitoring and management APIs can be used to externally manage Java applications as well as build manageability in applications.

Introduction to Manageability
Manageability is defined as the capability that allows an application to be managed and controlled. Manageable applications provide mechanisms by which it is possible to probe, measure, track, and control it (see Figure 1). Typically these actions are:

• Monitoring: To capture runtime information from the application
• Tracking: To observe aspects of an application over a period of time
• Control: To alter the behavior of a runtime component

This article focuses on JMX as the technology to manage applications. The new monitoring and management APIs use JMX and reside in the java.lang.management package. The APIs expose data in areas like memory, thread, runtime, operating system, garbage collection, etc. JMX can be used to expose coarse business-level information to fine-grained information about specific classes and objects. The information exposed by JMX can be consumed via system management software like HP Openview, IBM Tivoli, or simpler tools like jconsole.

Basics of JMX
The JMX specification defines an API and architecture that provides a standard mechanism to management-enable Java applications. The JMX architecture (see Figure 2) has three layers:

• Instrumentation
• Agent
• Distributed

Agent Layer
The agent layer defines a JMX agent, which is a management entity that runs on a JVM. The agent acts as the liaison between the MBeans and the management application. A JMX agent is composed of an MBeans server, a set of MBeans representing managed resources, a minimum number of agent services implemented as MBeans, and typically at least one protocol adaptor or connector.

The MBean server is the keystone of the JMX architecture and the central registry for MBeans. All management operations on MBeans are brokered through the MBean server. The MBean server, MBeans, and adapters make use of the generic functionality that is available as services. Additional services can be dynamically added by the application or the management system to extend the functionality of a JMX agent. Some of the services included are monitoring, timer, relation, and dynamic class loading.

Distributed Layer
The distributed layer typically contains at least one protocol adapter or connector. The adapters and connectors make the agent accessible remotely. Adapters provide a view of the JMX agent through a protocol like HTTP or SNMP. Connectors, on the other hand, are used to connect the JMX Agent with a remote JMX-compliant management application using a distributed technology like RMI. The distributed layer also has tools that expose the management view of a JMX agent and the MBeans via another protocol like SNMP, and tools that interact with the JMX agent and MBeans via a connector for distributed applications. This layer is responsible for providing connection services, security, and consolidated management information from disparate JMX agents. The adapter tools interact with the MBeanServer to generate artifacts that are required by a particular protocol. For example, MOF files may be created by a Web Based Enterprise Management (WBEM) adapter tool.

Using the Monitoring and Management APIs
The monitoring and management APIs use JMX as the underlying mechanism to expose the JVM information. The use of JMX allows the information to be available locally and remotely to applications that support JMX. The performance impact of extracting the JVM information is extremely low. The information exposed by new APIs can be used to aid in:

• External monitoring and management: Allows external entities like management software, system administrators, IT operators, support engineers, and developers to monitor the JVM and Java applications.
• Internal monitoring and management: Allows the developers to add logic to self-monitor and manage the JVM to make the applications more manageable and robust.

The JMX agent available on the JVM is disabled by default. To enable out-of-the box local JMX monitoring, the application needs to be started with the following arguments.

$JAVA_HOME/bin/java -Dcom.sun.management.jmxremote ApplicationName

This will start the JMX agent and allow the application to be monitored from the local machine using an application such as jconsole.

To enable out-of-the-box remote monitoring without SSL and password authentication the following command line can be used.

$JAVA_HOME/bin/java -Dcom.sun.management.jmxremote.port=1097
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
ApplicationName

However, in most realistic situations the connections would be enabled with SSL and password authentication. The following command line can be used to start the application on a specific port.

$JAVA_HOME/bin/java -Dcom.sun.management.jmxremote.port=1097
ApplicationName

Apart from the port number, other configurations need to be set via property files. The JRE comes with property files that are located in $JRE/lib/management/. These files set the various properties required to set up a secure connection when connecting to the JVM. Table 2 outlines the various files required to enable secure out-of-the-box remote management.

Connecting to the MXBean Interface
To begin the process of monitoring and tracking, the external entity needs to make a connection to access the MXBean interface. This connection can be made via any supported adapter tool like the SNMP. However, the examples here will be more focused on implementations that use Java. There are two ways to connect: via a proxy or making a connection to the MBeanServer.

A connection can be made by constructing an MXBean proxy instance that forwards the method calls to an MBeanServer (as shown in Listing 1).

While the mechanism described in Listing 1 works by getting direct access to an MXBean interface, it's also possible to go through the MBeanServer to get indirect access to the MXBean interface (see Listing 2).

The advantage of connecting through the MBeanServer is that it allows the management application to discover all the MBeans that are registered. This allows the management application to discover a richer set of information in a more dynamic manner. While connecting directly to a specific MBean is simpler, it is more suited to scenarios where the MBeans to be accessed are known beforehand.

After a connection has been established, memory usage for deployed applications can be monitored and tracked by using a polling-based or event-driven mechanism. The JMX architecture supports both of these mechanisms and, depending on the use case, either the polling or the event notification mechanism can be used.

The memory usage can be polled regularly using the getUsage() method. To use the event notification-based mechanism to monitor the memory, the usage threshold can be set using

memoryPool.setUsageThreshold(MEMORY_LIMIT);

where MEMORY_LIMIT is the peak memory consumption value in bytes.

Setting the usage threshold will generate events of MemoryNotificationInfo when the memory usage exceeds the threshold. When these events are received, additional actions can be taken by IT operators. IT can get detailed information about the state of the application during the time when the application is not behaving properly and this can help developers find the root cause of the problem.

Unusual CPU consumption is one of the indicators that some of the threads in a Java application may not be behaving as expected. The thread-related information available from the ThreadMXBean interface can help provide insights into the underlying problem. The interface not only provides mechanisms to see the number of threads executing, but also provides methods for getting detailed information about each of the threads. The information can indicate the threads that are executing for periods of time that are out of the normal bounds or the threads that are deadlocked. The code in Listing 3 finds if there are any deadlocked threads and prints the information about the blocked and the blocking threads. (Listings 3-7 can be downloaded from www.sys-con.com/java/sourcec.cfm.)

The ThreadInfo class also provides methods to get valuable thread-related information that includes the state of the thread, elapsed time in blocked state, and the stack trace.

One of the important aspects of manageability is the ability to add a degree of control after a certain condition is met. Although the current API doesn't provide controls to tune the JVM, it does provide a mechanism to get additional information when abnormal runtime conditions are observed. This control is provided using the LoggingMXBean interface (see Listing 4). The logging interface allows you to retrieve the various loggers and set the log levels dynamically. This is an extremely important feature because it allows external monitoring entities to acquire detailed runtime information when the application doesn't behave as expected or consumes excessive system resources. It's also important to note that to exercise any kind of runtime control, the management application needs to connect using the authentication for control role.

The new APIs allow the JVM to be managed out-of-the-box by exposing information critical to the health of a JVM. By leveraging JMX as the underlying architecture, it's now possible to externally monitor, track, and control a Java application and this makes Java applications more manageable.

Internal Monitoring and Management
The monitoring and management APIs can also help create adaptive business logic. Until now it has been difficult to write application logic that takes into account the effects of the runtime environment. Without using the runtime information, it's easy for the application to be in situations in which there are not enough system resources (like memory) available. To make the application more robust, the applications can:

• Expose management information that can be used by IT operators via management software to aid in root cause analysis.
• Take compensatory actions to prevent the application from running into an unrecoverable state.

In this article I look into exposing more granular information at an object level that deals more closely with application logic. For example, collection classes like hashtables and vectors can be easily misused and result in memory leaks. Some of the newer data structures like PriorityBlockingQueue (java.util.concurrent) can also cause OutofMemoryError if not used properly. Based on application logic, information contained in critical classes and utilities like object pools can be exposed via MBeans.

Exposing Management Information
To expose information via MBeans the first step is to create a management interface.

public interface ObjectPoolMBean {
	public Integer availableObjects();
	public Integer size();
	public String poolName();

}

The ObjectPool class will contain the implementation and the logic that will return the attributes exposed by the management interface (see Listing 5).

When constructing an MBean it's important to ensure that:

• The MBean is a public nonabstract class.
• The MBean has at least one public constructor.
• Standard MBeans implement their own management interface and DynamicMBeans implement the javax.management.DynamicMBean interface.

//Get MBeanServer
MBeanServer platformMBeanserver=ManagementFactory.getPlatformMBeanServer();
//Register the ObjectPool MBean
ObjectName poolName = 
new ObjectName("com.foo:id=ObjectPool");
platformMBeanserver.registerMBean
    (new ObjectPool(),poolName);

By having the relevant sections of the code exposed as MBeans, it's possible to extract the information about the state of the critical parts of the application. When the application misbehaves, the information exposed via the application-specific MBeans can help narrow the root cause of the problem.

Taking Compensatory Actions
The exposed management information from an application can alert IT operators about impending problems. The application can also use the JVM information to throttle the application and prevent it from getting into an unrecoverable state. By throttling the application, the garbage collector may get a chance to run and thereby prevent memory or thread-related problems. The example below uses the usage threshold notification mechanism to demonstrate how self-managing logic can be used. In this example, whenever the memory threshold-exceeded notification is received, the application saves the requests so that they can be processed later.

The notification listener (javax.management.NotificationListener) can be extended to detect low memory conditions. The memory threshold notifications are only generated when the memory-usage limits are exceeded. The memory limits are set for the memory pool using setUsageThreshold(). It's also important to note that when a low memory situation continues to occur, additional notifications will not be generated. The next event will be generated only when the memory usage falls below the threshold and then exceeds it again.

Once the LowMemoryListener has been created (see Listing 6), the listener needs to be registered with the MemoryMXBean. The ManagementFactory is a factory class that provides static methods to get the standard management interfaces for JVM information. The factory can be used to get Memory beans to register the LowMemoryListener.

MemoryMXBean membean = ManagementFactory.getMemoryMXBean();
NotificationEmitter emitter 
                     = (NotificationEmitter) membean;
LowMemoryListener listener = new MyListener();
emitter.addNotificationListener(listener, null, null);

In this example, the saved requests need to be processed at a later time when memory usage is back to the normal level. A polling-based mechanism can be set up to retrieve saved requests and restore the request queue one request at a time (see Listing 7). After the saved request queue is empty, new requests can be accepted.

The example uses the new APIs to build manageability by making the application aware of runtime resource consumption. The application uses a polling-based mechanism to monitor the application state and ensure that all saved requests are processed before new requests are accepted. The memory-exceeded notification is used to track and control application behavior to prevent it from going into an unrecoverable state.

Guidelines for Exposing Management Data and Taking Adaptive Action
The new APIs enable any application to build manageability by exposing management information. It's important to expose the right type of data to realize the benefits of application manageability. Some of the factors to consider when exposing the management data are:

• The granularity of the data: The data can be exposed either at an object level or by collating data from multiple objects.
• Usefulness and relevancy of the exposed data: The information exposed via managed beans has to be helpful to the IT operators from the manageability perspective. A good rule of thumb is that if the information is helpful for debugging and doing a root cause analysis of a problem, it's a worthwhile attribute to expose.

When adding logic for adaptive business logic for self-management, some of the functional decision points to keep in mind are:

• Use runtime information to drive application logic: This will allow the application to slow down graciously and thereby allow IT operators to take appropriate actions without incurring application downtime.
• Use standard Java logging mechanism: This will allow IT operators to retrieve additional diagnostic information via a logging management interface when the application doesn't behave as expected.

IT operators can now provide detailed information to developers about the application state when the application doesn't behave normally. The JVM information can help developers write self-managing logic that uses the runtime resource consumption characteristics. The new monitoring and manageability API helps developers identify problems quickly, IT operators get detailed information about Java applications and, most important, increases application uptime significantly.

Further Information