Java Virtual Machine Performance Tuning

This page describes how to tune Java VM garbage collection for optimum performance and longevity as it frees dynamically allocated memory that is no longer referenced.

This is a companion to my pages on Java and performance tuning

Java Virtual Machine Specification, 2nd Edition by Tim Lindholm and Frank Yellin

Topics this page:

Site Map  
About this site  

Summary of Lessons Learned

I have learned several lessons in my experience with troubleshooting Java for stability and scalability:

• Configure servers for the maximum number of Unix inodes / Windows file descriptors. The limited number of file handles available can be consumed quickly under a full load.
• Use static code checkers such as SAP's JLin to catch issues such as release of resources. These classes require explicit release code:
  • Connection
  • InputStream (except for ByteArrayInputStream)
  • OutputStream (except for ByteArrayOutputStream)
  • Reader (except for StringReader)
  • Writer (except for StringWriter)

• Wait time during garbage collection is the battle ground, and is increasingly being including in SLA (Server Level Agreements).

• Instill change controls with application startup scripts across dev, test, and other machines so that appropriate JVM settings are included.

• Load machines up with memory. This is usually the limiting factor, especially when multi-threaded applications are not tuned.

• Compare performance under load against baseline measurements.

• Keep a baseline history so that you can compare the impact of new JVMs as they are released.

• Begin with simple settings. Use of new features (such as GC across multiple CPUs) is not always appropriate (because they often don't work the first time out and need time to mature).

• Begin tuning settings by changing the default setting for this is usually not enough, such as the Max permanent space that hold classes.

• Have extra machines available for debugging, with plenty of disk space for logs (otherwise, it may not be there when needed).

• The dirty little secret about Java is that most production shops assume we're not always going to be able to detect and fix memory leaks, so work around it by having more machines in a cluster so that they can be shut down and rebooted occassionally before the predicted time between failures.

Real Time Java

• If your application can't live with delays inevitable with full GCs, consider using a Real-Time Java VM such as Sun Mackinac and IBM Websphere. These were developed based on the Sun community's JSR-01 RTSJ (Real Time Specification for Java).

  With it you can designate areas of memory not subject to garbage collection and threads that cannot be preempted by the garbage collector.

  The Excelsior VM makes regular SE6 more regular by performing ahead-of-time compilation which also optimizes code execution speed. This enables Java programs to run as executables, without a JRE.

  The Azul's Zing JVM enables transparent pauseless GC -- a continuously concurrent compacting collector which manages the memory and GC on several app servers. The Zing virtual appliance expands memory. YOUTUBE: Understanding Java Garbage Collection and what you can do about it. It's free to open source developers and testers.

  jHiccup by Charles Nutter (@headius) measures and charts spikes due to GC.

Programming for Garbage Collection

One of the advantages of programming in Java language rather than programmiong malloc() / free() and constructor / destructor commands in C language is that the Java virtual machine (JVM) takes care of managing "garbage" — automatically freeing objects that are no longer referenced by the program.

As instances are completed, local variables instantiated within them are no longer considered "live" because they cannot be reached from anywhere in the running program. They become "garbage" because they no longer participate in the future course of program execution.

There are three ways programmers can make objects eligible for garbage collection:

1. Nulling an object by setting an object explicitly to null, which means the lack of a value. But the memory that object consumes is not made available to other objects until garbage collection occurs.
2. Reassigning a reference - This is done by setting the reference variable to refer to another object.
3. Isolating a reference - A class has an instance variable that becomes a reference variable to another instance of the same class. For example, if there are two instances like this and they refer to each other, and all other references to the objects are removed and the objects are eligible for collection even with remaining valid references.

Java doesn't allow programmers to add a finalize() method to classes as C programmers are required to do explicitly (or by using libraries to emulate Java-like garbage collection as a replacement for malloc() and free() that uses a garbage-collected heap:

The Boehm-Demers-Weiser conservative garbage collector for C
the GCMM (Garbage Collection and Memory Management) library

But Java programmers can specify a finalize method to associate a cleanup method with an object type. Since it is generally not possible to predict when a finalize method will be run, ill-considered use of finalize can lead to unpredictable race conditions.

Eliminating the need for programmers to track of what memory has been allocated and then "finalize" it to release its memory space avoids many headaches. Automatic garbage collection also ensure system integrity because programmers cannot accidentally (or purposely) crash the JVM by incorrectly freeing memory.

Since Garbage Collection (GC) identifies garbage space available again for new objects, a more accurate and up-to-date metaphor might be "memory recycling."

The garbage collector also combats fragmentation, when free blocks of heap memory remaining between blocks occupied by live objects are too small (does not have enough contiguous free space) for new objects to use. This results in the need to extend the size of the heap for new objects — causing extra paging that degrade the performance of the executing program.

jconsole executable that comes with Java 5 (in JDK_HOME/bin) uses JVM 1.5's JMX instrumentation agent to monitor either local java processID, remote host:port, or JMX agents of MBean servers.

The J2SE 1.5 Monitoring & Management Console provides a formatted Summary.

Among Java Performance Documentation is Performance Documentation for the Java HotSpot VM

Urban performance legends, revisited" 27 Sep 2005 (article in the Java theory and practice series) by Brian Goetz starts by claiming that:

allocation in modern JVMs is far faster than the best performing malloc implementations.

Different JVMs

Keeping JVM Garbage Collection pause times low is becoming more and more important to provide predictable performance.

Major middleware vendors provide their own JVM.

Microsoft .NET

Using System.Collections.Generic, the Microsoft .NET Framework has a way to (in C# language) display counts of how many objects are in each generation:

Console.WriteLine( GetTotalCollections() );

Static int GetTotalCollections(){
	return GC.CollectionCount(0) + GC.CollectionCount(1) + GC.CollectionCount(2) );
}

REFERENCE: All Methods in the .NET framework GC namespace

Microsoft's .NET languages provide the low-level System.Threading API for starting, stopping, and joining threads and the high-level System.Threading.Threads API for concurrent and async programming.

BEA JRockit

Rob Levy, the CTO at BEA Systems claims that BEA's JRockit JVM for BEA WebLogic Server 9.0 "continues to impress and set the industry benchmark as the world's fastest JVM for large-scale, mission-critical, server side applications."

BEA's JRockit 5.0 Memory Leak Detector hooks into Java 5.0 SP1 (not 1.4) apps started (with the -Xmanagement option) in real-time production mode (through default port 7091) because the trace tool can be connected and then disconnected dynamically from the Java MBeans (JMX) management server rather than being specified in JVM startup parameters. What a concept!

BEA's JRockit Runtime Analyzer hooks into Java 1.4.2+ apps.

Plus, JRockit 5.0 R26 increases the maximum heap size on Windows to almost 3 GB.

BEA WebLogic Real Time (WLRT) (for Windows or Linux) uses JRockit 5.0 R26 Deterministic Garbage Collection (DetGC) (JVM setting "-Xgcprio:deterministic") to enable short GC pause times averaging 30 milliseconds vs. unpredictably longer pauses (averaging 275 ms) in Sun JVM's Incremental Garbage Collection (IncGC),

These timings were reported by Tom Barnes using a Grinder Jython client to measure his "Trader" app which uses the Java Spring open source framework 1.2.6.

BEA achieves this by applying QoS (Quality of Service) alogrithms limiting the total number of pauses within a prescribed window.

MBeans are managed on a separate Management Console server (installed by default to JRockit_JDK/bin).

JRockit home page

JRockit Product docs

BEA WebLogic JMS Performance Guide" white paper by Tom Barnes, senior software engineer on BEA's WebLogic Performance Team.

IBM

The IBM JVM can be downloaded [with required registration] as part of the IBM developer kit for Linux

IBM Java on Linux, Java 2 version 1.4.1Technology Edition by Pascal Chong on 25 January 2004

IBM Java Linux newsgroup (Usenet server: news.software.ibm.com, Groupname: ibm.software.java.linux

IBM Diagnostics Guides for
JVM 1.4.2 (562 pages) and other versions.

Try This: Default values

Parameter	Win 32-bit	Win 64-bit
-Xms	3670k
-Xmx	64m
-Xms	3670k
-Xmx	64m
-XX:MinHeapFreeRatio=	40
-XX:MaxHeapFreeRatio=	70

32-bit Windows address space is limited to 1.3 GB, spread out in

If you are running SAP's Sherlok, use 1.1 GB.

Instead of huge heaps, use additional server nodes.

On Unix machines, if you have 4GB RAM, start with this:

-Xmx3550m -Xms3550m -Xmn2g -Xss128k -ParallelGCThreads=8 -XX:+UseParallelOldGC

Values for 64-bit OS are about 30 percent higher.

Command Line Argument Flags

Arguments with Java can be specified in two places:

1. After the command line invoking java.
   Here is the output when the IBM 1.4.2 java is invoked on RHEL without any parameters:

   Usage: java [-options] class [args...]
              (to execute a class)
      or  java [-jar] [-options] jarfile [args...]
              (to execute a jar file)
   
   where options include:
       -cp -classpath ≪directories and zip/jar files separated by :>
                 set search path for application classes and resources
       -D=
                 set a system property
       -verbose[:class|gc|jni]
                 enable verbose output
       -version  print product version
       -showversion  print product version and continue
       -? -help  print this help message
       -X        print help on non-standard options
       -assert   print help on assert options
   

   Here is the output from java -X

       -Xargencoding     Allow Unicode escape sequences in args
       -Xbootclasspath: <directories and zip/jar files separated by :>
                         Set search path for bootstrap classes and resources
       -Xbootclasspath/a:<directories and zip/jar files separated by :>
                         Append to end of bootstrap class path
       -Xbootclasspath/p:<directories and zip/jar files separated by :>
                         Prepend in front of bootstrap class path
       -Xcheck:jni       Perform additional checks for JNI functions
       -Xcheck:nabounds  Perform additional checks for JNI array operations
       -Xcomp            Compile all methods on first use (z/OS only)
       -Xdisableexplicitgc Disable explicit GCs
       -Xdisablejavadump Use system dump rather than java dump on error
       -Xlp              Try to allocate Java heap using large pages
       -Xgcpolicy[:optthruput]|[:optavgpause]
                         Control garbage collector behavior
       -Xms        Set initial Java heap size
       -Xmx        Set maximum Java heap size
       -Xnoclassgc       Disable class garbage collection
       -Xcompactexplicitgc   Run full compaction in every forced GC (System.gc)
       -Xnocompactexplicitgc Don't run compaction in any forced GC
       -Xnosigcatch      Disable JVM recovery code
       -Xnosigchain      Disable chaining of signal handlers
       -Xoptionsfile= File containing JVM options and defines
       -Xoss       Set maximum Java stack size for any thread
       -Xpd              Load the Problem Determination JRE libraries
       -Xquickstart      Improve startup time by delaying compilation
       -Xrs              Reduce the use of OS signals
       -Xrunhprof[:help]|[:=, ...]
                         Perform heap, cpu, or monitor profiling
       -Xrunjdwp[:help]|[:=, ...]
                         Load debugging libraries for remote debug
       -Xss        Set maximum native stack size for any thread
       -Xverify          Enable strict class checking of all class loads
       -Xverify:none     Disable strict class checking
       -Xverify:remote   Enable strict class checking of remote class loads
       -Xdebug           Enable remote debugging
   The -X options are non-standard and subject to change without notice.
   

2. In the $JAVA_ARGS variable, such as in the .profile or wrapper.pl file.

To get a thread dump from Java:
1. Add these parameters on the JVM at launch time:

   -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintClassHistogram

2. While the java program is running, hit CTRL-BREAK/Pause
3. Look in the program's stdout file for the thread dump and class histogram.

Error Messages

Did you get this message after changing a parameter?

Exception in thread "main" java.lang.NoClassDefFoundError: XX:MaxPermSize=128m

The exception is in "main" because that's where Java parses its startup parameters.

The "java.lang.NoClassDefFoundError" appears because Java uses parameter names as class names. (It trusts us more than it should)

In this case, there is a dash missing from the parameter. So try "-XX:MaxPermSize=128m"

Verbosity Flags

Place verbosity flags before others.

Each JVM manufacturer (Sun, IBM, BEA) outputs GC logs in their own format.

Adding -verbose:gc to Sun's JVM will result in the jvm.log being filled with a line each time GC occurs.
Adding -verbosegc sets IBM's JVM to fill stderr.log.

Adding flag -XX:+PrintGCTimeStamps to Sun's JVM additionally prints a time stamp before each collection line. The sample below occured one minute (60 seconds) after the app was invoked.

60.042 [GC 106231K->33739K(362112K), 0.0509500 secs]
61.042 [Full GC 106231K->33739K(362112K), 0.0509500 secs]

The number of "secs" is the time spent in collection, when applications are paused and thus unresponsive to users. This affects Throughput — the percentage of total time not spent in garbage collection, considered over long periods of time.

The -> arrow separates the space before and after a collection. The larger the different between these two numbers, the more memory was recovered by a collection. The space remaining after a collection can't be reclaimed.

XX:PrintFLSStatistics=1 combines with other verbosity flags to print Free space in the old generation, Num chunks (number of non-contiguous free chunks of memory), and Max chunk size (the largest chunk -- the biggest single allocation we can satisfy without a pause).

(Total Heap Size)

In parentheses is the total JVM heap space, which is based on the value of flags such as -Xms3670k -Xmx1400m. This example specifies a minimum of 3.67MB and a maximum of 1.4GB. Numbers can include 'm' or 'M' for megabytes, 'k' or 'K' for kilobytes, and 'g' or 'G' for gigabytes. For example, 32k is the same as 32768 (32 * 1024).

The JVM automatically expands the space (from the Xms value up to the Xmx value) when the percentage of free space falls below the minimum defined by flag -XX:MinHeapFreeRatio=. Setting smaller than the default 40 percent would prevent expansion.

To prevent shrinking, set the -XX:MaxHeapFreeRatio=<maximum> flag at or above the default 70 percent.

Alka Gupta and Michael Doyle, "Turbo-charging the Java HotSpot Virtual Machine, v1.4x to Improve the Performance and Scalability of Application Servers, (PrintGCStats and ThreadAnalyzer)

Nagendra Nagarajayya and Steven Mayer, "Improving Java Application Performance and Scalability by Reducing Garbage Collection Times and Sizing Memory Using JDK 1.4.1

Alka Gupta, "Garbage Collection (GC) Analysis and Performance Tuning Using the GC Portal", July 2003

SAP's Sherlok

SAP developed "Sherlok" to determine the resource consumption of Java processes.

Sherlok 1.4 can output to telnet port (by default) or SAP Portal iViews (html frames) running on SAP J2EE Engine 6.20 with sherlok.so/.sl/.dll in J2EE Instance/cluster/server.

The most important parameters in the sherlok.properties file are displayed with
lss

From New/Young to Old/Tenured

JVM GC Illustration Pop-up this diagram to its own window

The total heap holds three types of memory: permanent, "young", and "tenured" generations of objects.

The young generation is also called the new generation, as new allocations of memory are created for apps in the eden-space within the young generation.

When eden runs out of space, a minor "scavenge" is done to one of two alternating survior spaces within the young generation. One survivor space is empty at any time, and serves as a destination of the next, alternating between to-space and from-space.

The allocation of space among the three areas within the new generation can be controlled by a flag such as -XX:SurvivorRatio=6 which sets a 1:6 ratio between each survivor space and eden, making each survivor space one eighth of the young generation (not one seventh, because there are two survivor spaces).

SAP recommends -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=90 The first options controls how big is eden space compared to survivor space (the latter contains two semispaces); the second option sets the desired percentage of the survivor space heap which must be used before objects are promoted to the old generation.

The JVM aims to keep survivor spaces half full by recalculating at each garbage collection a threshold number of times an object can be copied before being tenured.

Adding flag -XX:+PrintTenuringDistribution displays this threashold value, plus a "lifetime distribution" of space consumed by objects surviving each age (cycle) of garbage collection. An example:

Desired survivor size 131072 bytes, new threshold 31 (max 31)
- age 1: 5496 bytes, 5496 total
- age 13: 760 bytes, 6256 total
- age 14: 760 bytes, 7016 total
- age 17: 760 bytes, 7776 total
- age 20: 14720 bytes, 22496 total

-Xminf40 and -Xmaxf70 sets the minimum and maximum percentage of the heap space available after each collection run to their defaults.

Since most objects have short lifetimes, objects are copied between survivor spaces until they become old enough to be copied to the Tenured Generation, also called the old generation, a larger area collected less often.

The tenured generation also receives objects when the to-space is full.

Garbage in the tenured generation is collected during a major collection.

Young/Old Sizing

In jdk 1.4.1 and later, the amount of space used by the new generation is calculated:

min( MaxNewSize, max( NewSize, heap/(NewRatio+1) ) )

The -XX:NewSize and -XX:MaxNewSize flags explictly specify the minimum and maximum space allocated to the new generation. This should be no more than half the total space.

SAP Note 696410 recommends setting both NewSize and MaxNewSize to approximately 1/6 of the heap. For example:
-XX:NewSize=160m -XX:MaxNewSize=160m for 1 GB heap size and
-XX:NewSize=320m -XX:MaxNewSize=320m for 2 GB heap size.

Alternately, since JVM 1.4.0, a flag such as -Xmn1000m sets the absolute size of the new generation to 1 GB.

If a flag such as -XX:NewRatio=2 is not specified, the jvm assigns a default ratio of space for new and old generation depending on the type of machine: 12 for Intel machines [sparc -server: 2, sparc -client: 4 (v1.3) 8 (v1.3.1+)]. NewRatio=3 means that the ratio between the young and tenured generation is 1:3. In other words, the combined size of the eden and survivor spaces will be one fourth of the total heap size.

In detailed jvm log entries, space available in the new/young generation is labeled "DefNew" in parentheses (the middle set of numbers)

Collector Details

-XX:+PrintGCDetails (available since Sun JVM 1.4.2) provides more details about the type of garbage collector used. I haven't heard of negative effects from it.

0.000: [Full GC 0.007:
[Tenured: 0K->5335K(1417856K), 0.1498780 secs]
40221K->5335K(1524224K),
[Perm : 5273K->5273K(262144K)], 0.1500390 secs]
2.293: [GC 2.293:
[DefNew: 94591K->1301K(106368K), 0.0275230 secs]
99927K->6637K(1524224K), 0.0276280 secs]

The example above appears when the default "Tenured" garbage collector is used with the "DefNew" default new generation collector (described in the next section).

The Old/Tenure generation is the difference between the heap number and the New number. In the example above at time 2.293, the old generation went

• from 5336 (99927 - 94591)
• to 5536 (6637 - 1301)
• out of 1417856 (1524224 - 106368)
• in 0.000105 (0.0276280 - 0.0275230) seconds.

A cycle is formed when two or more objects refer to one another. For example, a parent object that has a reference to its child object, which has a reference back to its parent.

Promptness

Promptness — the time between when an object becomes dead and when the resource becomes available again — is especially important for applications that rely on JVM finalization to close file descriptors.

There are two ways to improve promptness when taking advantage of multiple CPU servers that can execute multiple threads at a time.

Concurrency for Big Machines

Add flag -XX:+UseParallelGC in conjunction with flags -server (to invoke the optimizing JIT Compiler) and -XX:+AggressiveHeap, which makes use of XX:+UseAdaptiveSizePolicy by default.

The SAP J2EE Startup Framework doesn't accept "-server" or "-client" command line parameters. Instead, set entry jstartup/vm/type=server in the instance profile.

-XX:+UseParallelGC is the default for Sun's GC for 1.5 JVM. others.

-XX:+UseParallelOldGC option was added with J2SE 1.5.0_06.

These settings conflict with -Xms and -Xmx options because they instruct the JVM to push memory use to the limit: the overall heap is more than 3850MB, the allocation area of each thread is 256K, the memory management policy defers collection as long as possible, and (beginning with J2SE 1.3.1_02) some GC activity is done in parallel.

Because it makes the JVM greedy with memory resources, it is not appropriate for many programs and makes it easier to run out of memory. For example, by using most of the 4GB address space for heap, some programs will run out of space for stack memory. In such cases -Xss may sometimes help by reducing stack requirements.

Since only one collector can work at a time, the flags described above should not be specified with flags specifying the incremental train collector specified by flag -Xincgc for the Solaris Exact VM 1.3.0 obsoleted by the Java HotSpot VM.

ParNew

-XX:+UseParNewGC enables the parallel version of the copying collector (newly available with JVM 1.4.2) rather than the default single threaded young generation copying collector.

But UseParNewGC doesn't work (and therefore shouldn't be used) with JDK 1.4.2_07 or older on Windows2003/IA64 platform (see SAP note 716604).

Its companion is flag -XX:+UseConcMarkSweepGC to specify use of the Concurrent Mark Sweep (CMS) collector, also referred to as the "concurrent low pause collector" because it aims to minimize pauses due to garbage collection by doing tenured garbage collection simultaneously with application threads.

This works with flag -XX:+CMSParallelRemarkEnabled and
flag -XX:ParallelGCThreads=4 when running on servers with 4 cpus, since parallel collectors use as many garbage collector (GC) threads as there are processors on the machine.

With the IBM 1.4.2 JVM, flag -Xgcthreads3 sets the number of helper threads (n-1) when running on servers with 4 CPUs.

Concurrent marking is set with flag -Xgcpolicy:optavgpause instead of the default -Xgcpolicy:optthruput.

However, if collection happens too quickly, the heap can become fragmented.

JNI objects are pinned and cannot be moved until unpinned by JNI.::

A fragmented heap can cause allocation requests to fail even though a lot of free space is available. The IBM 1.4.2 JVM lists pinnned and dosed objects when run with the flag

IBM_JAVA_OPTIONS -Dibm.dg.trc.print=st_compact_verbose

If the concurrent collector is unable to finish before the tenured generation fills up, all application threads are paused for a non-concurrent Full GC, also called STW (Stop-The-World) because it suspends all other threads while it works. Such collections are a sign that some adjustments need to be made to the concurrent collection parameters.

A Full GC can also be invoked from within an application invoking System.gc() or Runtime.gc(). This is not advisable. Flag -XX:+DisableExplicitGC makes the GC ignore such calls.

The JVM issues a out of memory error only when a Full GC fails to recover usable space.

??? "floating debris" may be created

JVM GC Illustration


Techniques used in the concurrent collector (for the collection of the tenured generation)

How to detect and troubleshoot garbage collection issues with the IBM Java Virtual Machine by Sumit Chawla [Jan 2003]

With the Mark-Sweep-Compact (MSC) algorithm:
During Mark, all the objects that are "reachable," or alive, are marked, starting with "roots" such as objects on thread stacks.
During Sweep, all the objects that were allocated but are not marked anymore are swept away, the space used by them is reclaimed.
During Compat, holes or fragments in the heap are removed by moving the live objects together (to create contiguous freespace in the heap).

See this Java applet simulating a garbage-collected heap, described in
Chapter 9 of Inside the Java Virtual Machine by Bill Venners

A Generational Mostly-concurrent Garbage Collector [Printezis/Detlefs, ISMM2000]

The Permanent Generation

If an OutOfMemory error is raised even though there is a lot of available heap space, it's very likely caused by a surfeit of permanent generation space.

The default 32MB maximum is usually not enough for complex programs, programs that dynamically generate and load many classes (such as some implementations of JSPs), or several applications are deployed.

The permanent generation is sized independently from the other generations because it's where the JVM allocates classes, methods, and other "reflection" objects.

These objects are not really permanent unless you use the -noclassgc flag, which is rarely used.

Specify flag -XX:PermSize=16m to allow the JVM to start with enough memory so that it doesn't have to pause apps to allocate more memory.

When running on servers with multiple cpus, multiply the memory by the number of cpu's. For example, to run a 4 way CPU using the default:

-XX:MaxPermSize=128m sets the JVM to have a maximum of 131072K (128 * 1024) bytes to grow into.

-XX:MaxPermSize=256m -XX:PermSize=256m is SAP's recommendation for SAP 7.0 on 32-bit Windows (Note 723909).
-XX:MaxPermSize=512m -XX:PermSize=512m is recommended for SAP 7.0 on 64 bit machines.

JVM GC Illustration


Java application performance tuning IV: Using JVMTI, BCI and JMX by Joseph Coha

Other Issues

Footprint is the working set of a process, measured in pages and cache lines. On systems with limited physical memory or many processes, footprint may dictate scalability.

JVM specifications do not define how the heap the garbage collector must work. The designer of each JVM must decide how to implement garbage collection.

If your application uses a lot of threads, see if contention for heap locks is reduced with flags -XX:-UseTLE and -XX:TLESize= to assign specific threads to a "Thread Local Eden".

-XX:+UseTLAB (Thread Local Allocation of oBjects) is SAP recommended. On Solaris and HP-UX this switch is set by default. On Windows and Linux you have to set it explicitly.

-Djava.awt.headless=true should be added if your app is "headless" (a back-end services app that interacts with operators only through the command line and not graphically). This saves the memory of loading graphic libraries not needed.

-Xbatch ???

-XX:SoftRefLRUPolicyMSPerMB=1 is recommended by SAP Note 723909.

-Dsun.io.useCanonCaches=false SAP Note 723909 requires for its Knowledge Management servers to guarantee functional correctness of CM repositories and file system repositories with file system persistence.

The .NET Framework provides several Perfmon counters described in these notes:

.NET CLR / # of Exceps Thrown
If this counter is higher than 1 or 2 when your application is under stress, investigate the source of the exceptions.

.NET Memory / % Time in GC
This counter shows you the amount of time your .NET code is spending in the garbage collector. There are some different opinions on what this counter should be. I have customers shoot for and average of under 10% when the application is near full stress.

.NET Memory / # of pinned objects
When an object is pinned, the garbage collector can not free OR move the object during a collection. This is primarilly used in .NET applications that interop with native code. If you are having memory shortage / fragmentation issues check this counter. A high number of pinned objects can severly impact the GC's ability to manage the heaps under stress.

Tuning performance with IBM Websphere app server JVM settings

Rico Mariani's .NET Performance Tidbits describes how to use tasklist to get the process id for

VADump (Virtual Address Dump) command-line tool to show the type, size, state, and protection of each segment of virtual address space. It's used to make sure virtual address space is not over-allocated. described here

and windbg (Windows Debugger) with the mscorwks.dll loaded from the same place that hold SOS.

CLRProfiler to see the allocation profile of .NET managed applications. It provides a histogram of allocated types, allocation and call graphs, a time line showing GCs of various generations and the resulting state of the managed heap after those collections, and a call tree showing per-method allocations and assembly loads.

Java Benchmarks

1. CaffeineMark from Pendragon Software Corporation, Libertyville, IL

2. JMark from Ziff-Davis, Inc., New York, .

3. SPECjvm98 from The Standard Performance Evaluation Corporation (SPEC), Manassas, VA

4. VolanoMark from Volano LLC, San Francisco, CA

MS.NET Benchmarks

Microsoft has two .NET garbage collectors:
mscorwrk.dll for workstation machines.

mscorsvr.dll for server machines.

Graphing GC logs

Each JVM manufacturer (Sun, IBM, BEA) outputs GC logs in their own format.

IBM provides a Memory Dump Diagnostic for Java (MDD for Java) which analyzes heap dumps. It can also compare two heap dumps to detect a memory leak. MDD for Java requires IBM Support Assistant (ISA) to provide extra tools and components for troubleshooting as well as providing a place to write problems (PMRs).

Graphing GC logs using Excel

1. Get a copy of the jvm.log.
2. Open it with WordPad rather than Notepad if the jvm.log comes from a UNIX machine.
   1. Within Windows Explorer, right click on the file
   2. select "Open with...".
   3. Click "Choose Program" and wait.
   4. Scroll down to the bottom of the list to select "WordPad".
3. Save the file as a text file (named, for example, "jvm.txt").
4. Press PgDn to scan down the file to remove text messages such as "Unloading class" or "CMS-initial-mark".
5. Press Ctrl-H to find and replace all characters between numbers with a separator character Excel will recognize (such as tabs or commas). I've worked out this sequence:
   • K-> to separator
   • K( to separator
   • K), to separator
   • secs] to separator
   • : [GC to separator
   • : [ParNew: to separator
   • : [Full GC to separator
   • : [CMS: to separator
   • [CMS Perm : to separator
   • )], to separator

   Wait until the hourglass disapppears and the OK pop-up confirms completion of each operation.

6. Save the modified file.
7. Press Ctrl-A to select all text in the file.
8. In Excel, open a new file, click cell B3, and paste the text,
9. Create headings:

   Without detail flagged:
   Time		ParNewFrom	ParNewTo	NewNew	NewSecs	HeapFrom	HeapTo	HeapTot	HeapSecs
   With detail flagged:
   Time	Time2	ParNewFrom	ParNewTo	NewNew	NewSecs	HeapFrom	HeapTo	HeapTot	HeapSecs

10. To the right of these, add columns for Full GC
11. Add columns for calculation "HeadDiff"
12. Lock the headings: Click on the line number (in the left-most grey column) on the line immediately under the heading. Select the Window menu and select "Freeze Panes".
13. Reposition values in Full GC lines:
    1. Cursor to the right-most cell containing a value,
    2. To find the next Full GC line, press Ctrl and press the down arrow.
    3. Copy the block of numbers to the right of the second time code and paste that over the extraneous time code cell.
14. In the cell in the first row under column heading "HeapDiff" (Q3) type formula =H3-I3.
15. Highlight the column Q.
16. Chart the difference column.
17. In
18. Numbers on the x axis (Elapsed Time) are too numerous, so in Chart Options, click tab "Axes" and unselect Primary axis Category (x) axis.
19. To chart diff and secs on the same page, in Chart Type, select "Custom Types" and "Lines on 2 Axes"
20. To avoid long collections from ...

    After data from multiple runs is put on different sheets, a chart can be created to visually compare lines from sheets to different runs.