TechNote (Archived)

JRun 4: General troubleshooting techniques for an unresponsive JRun server

There are a number of conditions that can result in the appearance that a JRun server is hanging or crashed, and a number of techniques that can be used to help isolate the source of the problem. This TechNote will discuss these conditions and provide a checklist of troubleshooting techniques that can help you better understand what's happening, and why.

Metrics logging
JVM heap size
Client request threads
JVM issues
Thread deadlock in container
System resources are low

Metrics logging

Metrics logging is an important monitoring mechanism provided with JRun. It allows you to obtain statistical information about the connection between a web server and a JRun container. These statistics include data such as number of requests handled, available threads for handling requests, and memory utilization. During periods when JRun appears to hang, this information will help determine the volume of requests being sent to JRun, and will provide memory utilization trends. An increasing number of requests and threads while JRun is non-responsive can indicate a concurrency issue within the application. A decreasing amount of free memory can signal issues related to excessive object creation or garbage collection tuning.

To enable metrics logging, locate and uncomment the following section in the server's jrun.xml configuration file:

 <!-- ============================================ --><!-- This Service provides metrics information    --><!-- ============================================ --><!-- To enable metrics: uncomment this service and  in LoggerService set metricsEnabled to true -->
		

	<!--<service outputclass="jrunx.metrics.MetricsService" name="MetricsService"><attribute name="bindToJNDI">true</attribute></service> -->

Next (also in jrun.xml), set the metricsEnabled property to "true", and set metricsLogFrequency to an appropriate value (this is the time interval in seconds between log file entries):

 ...
		

	<service outputclass="jrunx.logger.LoggerService" name="LoggerService"><attribute name="errorEnabled">true</attribute><attribute name="warningEnabled">true</attribute><attribute name="infoEnabled">true</attribute><attribute name="debugEnabled">false</attribute><attribute name="metricsEnabled">true</attribute><attribute name="metricsLogFrequency">10</attribute><attribute name="metricsFormat">Web threads (busy/total):   {jrpp.busyTh}/{jrpp.totalTh} Sessions: {sessions}   Total Memory={totalMemory} Free={freeMemory}</attribute> ...

The following example shows the log message for this format:

 02/14 16:11:53 metrics Web threads (busy/total): 0/2   Sessions: 2 Total Memory=7252 Free=3103

Output will be written into the {server_name}-event.log file, where {server-name} represents the name of your JRun server (ex. default-event.log).

For more information about JRun metrics logging, please refer to chapter 7 ("Connection Monitoring") of the JRun Administrator's guide.

JVM Heap Size

The heap is the amount of memory that is allocated to the JVM (both JRun and the application run inside the JVM). Heap size is controlled by parameters that are passed to the JVM process at startup time, and is subject to a configurable maximum size. Depending on the application and number of users, it is possible to exhaust the memory that's been allocated to the JVM. If that happens, you can increase the heap size by adjusting the JVM arguments.

Setting an appropriate heap size can have a positive impact on over-all system performance. A large heap can contain a relatively large number of objects, so garbage collection is required less frequently. However, when GC is run on a large heap, it will take longer (more objects to collect) and may have a noticeably negative effect on application response time. If you are using JDK version 1.4.x from Sun, you can take advantage of its new GC options and memory allocation switches. For more information, please see:

It is a good idea to test out different starting and maximum heap sizes to determine the optimum settings for a specific application.

To modify the JVM heap size, modify the command line arguments that are passed into the JVM. This is done by modifying thejvm.config file in the {JRun_home}/bin directory. For example:

java.args=-Xmx128M -Xms32m
-Xmx indicates the maximum amount of memory to allocate to the JVM heap (128MB in this example)
-Xms indicates the amount of memory to allocate to the heap at startup time (32MB in this example)

Additional information on setting an appropriate heap size can be found in the JRun: java.lang.OutOfMemoryErrors TechNote.

Note: It is highly recommended that the application developer also investigate potential causes of high memory usage (using a profiling tool such as JProbe or OptimizeIt) to ensure that this runtime condition is not the result of application coding.

Client request threads

Every incoming request is bound to and managed by a JRun thread. The number of JRun handler threads is configurable, and may include both external web server ("jrpp") threads and JRun web server threads ("web"). If the number of client requests exceeds the number of available threads, requests will be queued and users may experience delays in receiving a response from the server. If the number of requests significantly exceeds the number of available handler threads, requests may even be dropped completely rather than queued - giving the appearance that JRun is hung.

If the application handles high request volume, you can also modify the ActiveHandlerThreads and MaxHandlerThreads settings for the JRunProxyService (in the jrun.xml configuration file). It's always a good idea to perform load testing based on antipated volumes, and tune these parameters accordingly. This helps you better distinguish between normal threading activity and abnormal activity. Changing parameter values without first establishing a solid performance baseline is not a good idea.

To modify JRun threading parameters, edit the JRunProxyService section of the jrun.xml file:

 <service outputclass="jrun.servlet.jrpp.JRunProxyService" name="ProxyService"><attribute name="activeHandlerThreads">25</attribute><attribute name="backlog">500</attribute><attribute name="deactivated">false</attribute><attribute name="interface">*</attribute><attribute name="maxHandlerThreads">1000</attribute><attribute name="minHandlerThreads">1</attribute><attribute name="port">51000</attribute><attribute name="threadWaitTimeout">20</attribute><attribute name="timeout">300</attribute><!--<attribute name="keyStore">{jrun.rootdir}/lib/keystore</attribute><attribute name="keyStorePassword">changeit</attribute><attribute name="trustStore">{jrun.rootdir}/lib/trustStore</attribute><attribute name="socketFactoryName">jrun.servlet.jrpp.JRunProxySSLServerSocketFactory</attribute> --></service>

One technique for verifying whether communication problems exist between the web server and JRun is to make a request using the internal JRun web server (JWS). JWS uses a different port (8100 by default) and a different thread group to handle requests ("web"), and will continue to respond even if all jrpp threads are in use. If a request to JWS works while a request to the external web server fails, this indicates that there is a connection issue between JRun and the web server. To further verify this condition, enable Metrics logging for the JRun server (in the jrun.xml configuration file).

JVM issues

If JRun or the application appears to hang, this can sometimes indicate a problem with the JVM process. While the server is hanging, check to see if the JVM process is still running (at the operating system level). If it is not running that means the JVM has halted. Since JRun itself requires the JVM to run, neither JRun nor the user application can continue to function. In this case you may want to switch to a newer or different version of the JDK. Installing a new JDK is a relatively quick and simple procedure, and only requires that the java.home property in the jvm.config file be updated to point to the correct location of the new JDK:

 # java.home=C:\\bea\\JRockit80_141_32 # java.home=C:\\bea\\JRockit80_141_32\\jre # java.home=C:\\jdk1.3.1_07 # java.home=C:\\j2sdk1.4.1_01 java.home=C:\\j2sdk1.4.2

New JVMs may be obtained from a number of different vendors, including Sun: Java 2 Platform, Standard Edition (J2SE) downloads

Thread deadlock in container

Many components within a J2EE application require multi-threaded access, and are therefore subject to thread-related performance issues (ex. synchronization, race conditions, deadlocks). It is not uncommon that a thread deadlock within the application is at the source of a JRun hang. The best way to verify this it to generate a series of JVM thread dumps while the server is hanging (a good rule of thumb is to generate 3-5 dumps at 60 second intervals).

A thread dump will display the current state of each thread contained within the JVM. It can help identify potential deadlocks, as well as other conditions such as threads blocked on network access, database access, long running database queries, access to 3rd-party components such as a CORBA server, etc. If threads are blocked while trying to access a remote system or resource, you will need to investigate that system or resource to verify that it really is accessible (i.e no network problems, server is running, etc.).

Please refer to the following JRun TechNote for platform-specific information about how to produce and interpret a thread dump: JRun Server: TechNote Index for Thread Dump/Stack Trace Articles

System resources are low

Sometimes the hardware does not have enough power to provide the performance and responsiveness demanded by your user-base. As system resources become depleted, especially in a production environment, you can turn off certain JRun services to minimize contention. For example, Metrics and Debug logging can generate huge amounts of log file data. Large log files can negatively impact application performance, and may also cause the server to hang in some extreme cases.

If after reviewing the above suggestions you still have questions or need additional assistance, please contact Macromedia JRun Product Support to open a support incident. A JRun Support Engineer can work with you directly to help resolve the problem. Please be prepared to provide all metrics logs and thread dump files to the JRun Support Engineer.