Performance Tuning for Speed and Scalability

Here is my concise reference on the business of tuning (tweaking) servers and applications as part of performance monitoring and performance testing

Site Map  
About this site  

Aspects of Tuning

There are several factors (variables) that affect the performance and scalability of a system. The optimal Levels (values) for factor variables are determined from the result (experimental responses) of test runs under various conditions:

I. Architecture that scales smaller or larger [a scalable architecture]
“Vertical scaling” (up) adds additional RAM or CPUs within individual server boxes.
A web garden is where one large multiprocessor machine acts as a web server.

“Horizontal scaling” (out) adds additional server boxes to a cluster of servers obtaining work from a load balancer.
A web farm is where many machines work together as one web server.

An appliance (firewall and SSL processor) replaces a general purpose computer.

II. Individual machine Configuration

Amount of memory, Amount of caching, Number of CPUs on motherboard, Speed of CPU, Hard disk spin rate RPM, etc.

III. Network Settings

IV. Operating System Settings for Windows Linux Solaris

V. Database and Application Software Settings

Version of application or system (driver) software, Pooling of Threads & connections, JVM Garbage Collection parameters, Extent of Logging/Monitoring, Timeouts, Refresh Rates, etc.

VI. Source Code Configuration Tweaks

VII. Source Code Programming Tweaks

Code Execution Profiling to find multi-threaded problems, resource leaks, time-dependant boundary conditions, and caching errors. May also expose issues with efficiency of coding practices and team development approaches.

Scaling Out

Roger Wolter uses these illustrations in his Scaling Out SQL Server 2005:






Achieving Scalability and Availability with Peer-to-Peer Transactional Replication by Michael Blythe
Managing SQL Server 2005 Peer-to-Peer Replication by David Lindquist

Table partitioning:



Planning, Implementing, and Administering Scaleout Solutions with SQL Server 2005 by Bob Beauchemin

David Campbell's Service-Oriented Data Architecture: App-Server:Lite?

The Approach To Tuning

This flowchart covers the bases of tuning efforts. It is from Darren Broemmer's
J2EE Best Practices : Java Design Patterns, Automation, and Performance (Indianapolis, Ind., John Wiley & Sons, November 8, 2002)

Although the code examples in the book are dated (using the version 1.0 perform method rather than Struts 1.1), the summary is valid. The flowchart in table form:

Phase	Task	Deliverable
Project Planning	Evaluate	Project Objectives, High-level Requirements, Scalable Architecture
Analysis	Validate	Use-Case or Scenario Level Requirements
Design	Redesign	Scalable, Robust Design
Code	Optimize	Everyday Development Best Practices
Performance Test	Measure and Evaluate	[Measurements]
[Deployment]	Profile	Acceptable Results [that meet SLA's]

My plan is based on the

1. Network
2. Database
3. Web Servers
4. App Servers

"Premature optimization is the root of all evil." — Donald Knuth, pioneering computer scientist

Tuning Workflow

As with other LoadRunner modules, Mercury Tuning uses a separate VuGen program to create scripts. Those scripts are run by another program as part of a workflow of steps within Tuning Sessions. A third "analysis" program analyzes results from tuning sessions.

New Tuning Session step files (file extension .prs) are created from a template of steps called "cartridges" (file extension .__).
Below is my custom cartridge file, adapted from the "J2EEBreakdownCartridge" that comes with the product. This file should reside in your folder "C:\Program Files\Mercury\LoadRunner\Tuning Cartridges".

My session steps include tasks for capacity modeling

My Step		Based on	Details
	1. Setup_Environment		1.1 Create a Topology graph of the hierarchy of routers, servers, and other system elements [Moved up from their cartridge location] This is stored in a file suffixed with .tpl. 1.2 Define the logical name, host name, OS, Product, and Version of each element on the typology. + The IP address(es) of each element is noted in the Comment section for each element. 1.2 + Procure facilities, furniture, hardware, and software. 1.3 + Install OS, utilities, database, and app software on machines. 1.4 Install Mercury tuning agents and monitors (rstatd) on each host. 1.5 Install Database client and connect it to the Database server. 1.6 Collect settings from servers into the Tuning Console's "System Configuration" tab.
	2.1 Network_Health_Check measures and remembers the number of hops and latency using tracert between tuning agents on system components. 2.2 Bandwidth_Test measures the data throughput required to download files of known sizes using script "http_downstream_bandwidth" provided run for 20 vusers. 2.3 Network_Diagnosis 2.4 SSL_Test
	3. Setup_ J2EE_ Transaction_ Breakdown_ Diagnostics		3.1 Install MSXML (Microsoft Core XML Services 6.0) downloaded from Microsoft for Mercury's J2EE Transaction Breakdown Diagnostics Module. 3.2 Install Mercury's J2EE Diagnostic Mediator, activate it, and connect client to it.
	4. Database_Server_Diagnostics     SQL_Profiling	-	4.1
	5. System_Validation	-	5.1 + Create a shell/command script to stop and start all services, plus identify system information (disk space, etc.) before runs. 5.2 + Create a shell/command script to collect test run results.
	6. System_Snapshot	-	6.1 Create a load-testing script to log in users and initiate a new user session. 6.2 Create a load-testing script to emulate user tranactions. 6.3 Debug configuration settings. 6.4 Run a single user to capture monitoring information that identifies behavior under no load, such as "spikes" during 2-3 hour runs.
	7. [Max_]Benchmark		Use the load-testing script to 7.1 Identify the optimal number of vusers (load factors) while response time is adequate and no errors occur. 7.2 Identify the maximum number of vusers (load factors) when errors occur, response time becomes too slow, or (most commonly) processing rate declines. 7.3 Resolve transaction failures and errors 7.4 Resolve performance issues
	8. [Peak_]Benchmark		previously identified to 8.1 Ramp-up to optimal load factor 8.2 Archive (backup) script and run files so it can be used later as the benchmark to compare the impact of changes to the system configuration (to determine if there was an improvement in performance).
	9. Scan_Log_Files	-	Identify trends and errors using tuning agents reading server logs generated during test runs.
	Troubleshoot: 10.01 Transaction_Failure_Diagnostics 10.02 Performance_Diagnostics 10.03 Slow_Response_Diagnostics 10.04 Transaction Response Time Diagnostics 10.05 Sychronization Diagnostics evaluates threads holding locks. 10.06 Network_Load_Balancer 10.10 Application_Server_Diagnostics 10.11 Application_Server_CPU_Diagnostics 10.12 Application_Server_Memory_Diagnostics 10.13 Memory_Leaks_Diagnostics 10.14 Application_Server_IO_Diagnostics 10.15 JVM_Diagnostics 10.16 Network_Diagnostics
	11.1 Scan_Recommended_Values		Optimize applications by using tuning agents to compare actual settings against recommended settings. The recommendations from the Mercury Knowledge Base "according to vendor documentation" and can be customized and exported as part of the tuning cartridge.
	12.1 Application_Tuning 12.2 Server_Tuning 12.3 EJB_Unit_Tuning 12.4 Servlet_Tuning 12.5 JDBC_Optimization
	13. Generate_Report		Create the tuning session summary report (in MS-Word format) using the Tuning Console's Session Report feature.	-
	14. + Compare_Reports	-	Create reports that compare tuning sessions.

"Progress Dashboard" The Tuning Console tracks Each time you run this step, an entry is added to the progress dashboard to help you achieve your goals.

The Tuning Console lists all the logs and their properties for each host and service. You can also scan the logs for errors according to predefined rules. You can also define your own rules and export them as part of a cartridge (please refer to the product documentation). Note: 1. Before you can scan logs, you need to install and run tuning agents on the servers where the log files are situated. 2. The log scan step is available only for the services that are handled by the tuning agents. In other words, if the tuning agent cannot handle a particular service, the service's log file cannot be scanned, even if the server itself is running a tuning agent. 3. It is not recommended to scan large logs (>1MB), as this is time- and resource-consuming.

Server configuration tab. After you define the environment in the Topology window, the Console attempts to connect to the tuning agent running on each host in the topology. If a tuning agent has not been installed, or has been installed but not started, the attempt will fail. Make sure that the Console can connect to the tuning agents on the host machines. For details, see the section on tuning in the documentation.

Mercury's Performance Tuning Network at tuning.merc-int.com is not operational. I filed a bug report on this July 06.

lr8.lrtun81.notconcurrent.bmp

Network Speed Tweaks

First of all, I am assuming that you have looked at QoS appliances such as the Packeteer PacketShaper 2500 appliance deployed at both ends of remote office WAN links and use various techniques to move data more efficiently, especially for high-priority transactional users.

The efficiency of data throughput performance between network components is important because many network requests are required to process each client request.

To manually measure the speed (latency) of packets traveling over an IP Network, use command:

tracert wilsonmar.com

This reports the number of milliseconds (ms) between each hop between router IP/host address.

Add to the parameters:

-d	to not resolve addresses to hostnames.
-h 60	to specify a maximum_hops higher than the default 30 hops to search for target.
-j host-list	to specify a "loose" source route along host-list.
-w 30	to specify a wait timeout milliseconds for each reply.

Microsoft Windows 200x and Windows XP provides additional network loss information with
pathping wilsonmar.com

Performance thresholds are:

-	Good	Average	Bad
LAN	< 10 ms	~ 50 ms	> 100 ms
WAN	< 150 ms	~ 300 ms	> 600

Crimped line?

If your modem is slower than before, maybe your phone line is bad. If you can, go to your telephone company's box and plug in a new cable that connects directly to your computer. This bypasses the house wires. If your computer works better, you need to bypass each section of wire until you find a crimped wire or loose connection somewhere. This same troubleshooting technique can be used to find degraded computer network (category 5/6) cables and networking equipment.

OS Configuration Speed Tweaks

Fragmented hard disk space.

When you delete files, the space those files used are not reused immediately. If a file using that old space is bigger than the file that used to be there, the new file is split over several areas on the disk. Over time, files take longer and longer to retrieve. To fix this, use Windows Explorer and delete the contents of your temp folder (not the folder itself). Also, in your Internet Explorer, go to Tools, Options and delete files.

A program hogging up CPU think time.

"Spyware" programs can be loaded on your computer without you knowing about it. Sometimes it's easier to add a new hard disk and install the operating system and all applications all over again, then refer to your old data on the old drive.



Forward and Reverse Cache

To accelerate Web server performance, place reverse cache servers close (local) to the origin server -- to pre-load the web server that the enterprise maintains to host Web content. For example, a national news site can have a reverse cache engine on the East Coast, one in the Midwest, and one on the West Coast.

To reduce Internet traffice and latency time between clients and servers, reduce the need for all traffic to travel to one origin server by deploying global forward cache engines to mirror (duplicate) Web site content forward -- out geographically closer to clients.

Note, however, that content freshness checks may add up.

Different ways to architect a transparent to intercept (filter) traffic and redirect it to a cache:

1. A Layer 4 switch is a high-performance device parses HTTP packets and makes routing decisions based on various protocols.

2. The Web Cache Communication Protocol (WCCP), a proprietary standard developed by Cisco Systems, establishes and maintains the transparent redirection of selected types of traffic flowing through a group of routers. It serves a cache cluster to provide redundancy, interface aggregation, and distribution of the redirection load. The router monitors the availability of the cache engine. version 2 of WCCP allows multiple routers and multiple TCP port redirection by using Cisco Express Forwarding (CEF).

3. The newer Web Proxy Autodiscovery Protocol (WPAD), a proposed vendor-neutral standard protocol to automatically locate caches and services on networks without end-users configuration of browsers, streaming media clients, and other Internet client software.

Windows Client Machines TCP/IP Settings Are Tuned

There are many reasons why a computer runs slow. Before buying more hardware, check out your system. I'll list them in order of difficulty to fix:



TCPIP Parameters

In Windows XP clients, performance is boosted when Regedit is used to change these default settings in Windows Registry key:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters
• Interfaces\MTU = Maximum Transmission Unit (MTU) = none to 1492
• TcpWindowSize = 64240 to 255552
• DefaultTTL = 128 to 64
• EnablePMTUBDiscovery = none to 1

HKEY_CURRENT_USER\Microsoft\Windows\CurrentVersion\Internet Settings
but should be set for better performance:

• MaximumConnectionsPerServer = none to 10
• MaximumConnectionsPer1_0Server = none to 20

Historically, Windows 98 by default is set to handle a small number of connections. This can be increased by changing system registry key

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\InternetSettings
"MaxConnectionsPerServer"=dword:00000010
"MaxConnectionsPer1_0Server"=dword:00000010

Increase this to 100. More than that, machines get overloaded.



Disable Last Access Date Update

On Windows NTFS partition drives, each folder is tagged with the last access date and time whenever the folder structure is traversed. Traversing large disks with large folder structures is slowed by this date-and-time stamping process.

[_] To disable the last access update on NTFS partitions, in key
HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Control \FileSystem
add the NtfsDisableLastAccessUpdate value name with a REG_DWORD data type and a value of 1.



Allocate Space for MFT

NTFS uses the MFT (File Table) as a file index. File properties including location information and entire files, if the files are small, are stored in the MFT. The MFT stores at least one entry for every file on an NTFS partition. As files are added to a partition, the MFT grows. Because the MFT is a file itself, it is susceptible to fragmentation. Excessive fragmentation in files impacts performance. The standard disk defragmenter included with Windows 2000 does not defragment the MFT. Therefore, to avoid fragmentation in the MFT, NTFS reserves space for it. The default space reservation is adequate for relatively small partitions that will not contain many files.

[_] To increase contiguous space allocation for larger partitions that will contain many files, add the NtfsMftZoneReservation value name with a REG_DWORD data type and a value of 2, 3, or 4 to the HKEY_LOCAL_MACHINE \SYSTEM \CurrentControlSet \Control \FileSystem key. The larger the value of this registry entry, the more space that is allocated to the MFT.

Get this print framed for your wall!

• Microsoft Windows XP Performance Maintenance Newsgroup

Tweaking Windows Server Settings

[_] Page File Location

Create an additional pagefile on its own partition so that Windows can make multiple I/O requests quicker. The pagefile will not be fragmented. Choose "Create" and not "Move".

Set Windows Page File Size

Rather than letting Microsoft Windows dynamically obtain more space for its Windows Page File set from the Windows Start menu choose Settings Control Panel, Advanced tab, Performance options.
[_] In the Performance dialog box, change the page file setting for the amount of memory, plus 24MB. For example, a machine with 2GB RAM should have a page file size of 2024GB.

There should be at least a 127MB pagefile on the boot partition to accomodate memory dumps in case of Kernel mode STOP error, after which the page file is written only on the boot partition.

[_] Set Windows Server Virtual Memory Manager for Applications

Servers running applications should be set to "Applications" in the Performance options dialog box from Windows Start menu, Settings, Control Panel, Advanced tab, Performance options.

[_] Set Windows Screen Saver Off

Servers running applications should have screen saver set to off in the Display Properties dialog box within the Screen Saver tab from the Windows Start menu, Settings, Control Panel, Display (or right-click anywhere on the desktop and choose Properties).

Tweaking Linux

Performance Tuning for Linux Servers (IBM Press, 2005, 574 pages) by IBMers Sandra K. Johnson, Gerrit Huizenga, and Badari Pulavarty

Optimizing Linux Performance : A Hands-OnGuide to Linux Performance Tools (Prentice Hall PTR, March 14, 2005) by Phillip G. Ezolt, of Compaq's Alpha performance group and HP's representative to the SPEC CPU subcommittee.

Performance Assurance for IT Systems (Auerbach Publications, 2005, 376 pages) by Brian King

Introscope from Wily Technology pinpoints component-level bottlenecks in J2EE applications.

JProbe from Sitraka

Tweaking Solaris

Solaris Performance Administration (McGraw-Hill © 1998) by H. Frank Cervone

Application Settings Tweaks

Java App Server Application Tweaks

1. Tuning JVM Garbage Collection pause times

2. Thread Pool optimal size need to be identified through experimentation, since the efficiency from additional threads may be less than the overhead needed to maintain additional threads.

   This should be appropriate to the number of CPU's on the machine.

   If set too low, WebLogic may not be able to respond to requests in a timely manner. The thread count is too low if the sever never reaches 100 percent CPU utilization rate. If set too high, WebLogic spends most of its time switching between threads rather than performing real work.

   Product	Where	Setting
   BEA Weblogic 8.1	ExecuteQueue:	ThreadsMaximum, ThreadsIncrease QueueLength
   IBM WebSphere 5.1	Application Servers > "server" > Web Container > Thread Pools:	Minimum Size Maximum Size Inactivity Timeout
   JBOSS	MBean: org.jboss.util.threadpool.BasicThreadPool	MaximumQueueSize MinimumPoolSize MaximumPoolSize

3. JDBC Connection Pooling

   This setting is about 1/2 the thread pool size.

   During development, the server starts quicker when there are less connection pools. In production systems, the pool maximum capacity and the initial capacity are usually set to the same number so that all connection objects are created immediately.

   Product	Where	Setting
   BEA Weblogic 8.1	JDBCConnectionPool:	InitialCapacity MaximumCapacity
   IBM WebSphere 5.1	JDBC Providers > database > Data Sources > data source	Max Connections Min Connections
   JBOSS	In file dataSourceName-ds.xml:	<Min-Pool-Size>1</Min-Pool-Size> <Max-Pool-Size>100</Max-Pool-Size>

4. Set the minimum Java heap size equal to the maximum heap size. JVM Heap

   Product	Where	Setting
   BEA Weblogic 8.1	Administration Server scripts startWLS.sh (UNIX) and startWLS.cmd (Windows) in bin folder.	"%JAVA_HOME%\bin\java" -hotspot -Xms512m -Xmx512m -classpath %CLASSPATH%
   IBM WebSphere 5.1	-	-
   JBOSS	-	-

Mercury's recorded and upcoming Webinars
• Diagnostics Methodology & Configuration Related Problems by Reid Spencer, Chief Performance Consultant at Mercury.
• The Big 3 in Application Code Problems Neelesh Kamkolkar, Technology Evangelist, Mercury
• Common Architectural & Design Issues by Neelesh Kamkolkar, Technology Evangelist, Mercury

• Performance Analysis for Java? Web Sites ( Addison Wesley, September 10, 2002) by Stacy Joines, Ruth Willenborg, and Ken Hygh — consultants and developers at IBM Software Group at Research Triangle Park, North Carolina.

  J2EE Performance Testing (Apress, July 11, 2003) by Peter Zadrozny [English/Spanish Blog] (vice president and chief evangelist for Oracle Application Server at Oracle Corporation), Philip Aston (of BEA) (maintainer of
  The Grinder open source Java load testing framework used in the book), and Ted Osborne focus on BEA Weblogic servers with Panorama.

  High-Performance Java Platform Computing: Multithreaded and Networked Programming (Prentice Hall, (September, 2000) by Thomas Christopher & George K. Thiruvathukal

  Java 2 Performance and Idiom Guide (Prentice Hall, September, 1999) by Craig Larman & Rhett Guthrie is an almanac for the advanced programmer based on Java 1.1, but still mostly relevant.

  Server-Side Programming Techniques (Java(TM) Performance and Scalability, Volume 1) (Addison-Wesley June 2, 2000) by Dov Bulka

  Java Performance Tuning (2nd Edition) by Jack Shirazi covers Java 1.4

  Building Scalable and High-Performance Java Web Applications Using J2EE Technology (Addison-Wesley, December 27, 2001) by Greg Barish explains JSP, Servlets, EJB's, JMS, JDBC, etc. in the context of how they contribute to web application scalability and performance.

  Arcturus Tech's Autopilot WL provides intelligent SOA and J2EE monitoring in that their embedded "Knowledge Engine" auto-tunes WebLogic 8.1 and the JVMs they use to provide root cause analysis.

Architecture Tweaks

Before doing "open heart surgery" on source code, consider these strategies:

• Expedia, Orbitz, and most other travel booking sites throw up a "searching... please wait" screen. This buys your system 30 seconds before users get impatient.
• To avoid being swamped, many sites throw up a "Busy ... Please Try Again Later" screen so that users already in the back-end have a chance to finish normally. Such messages are thrown up by a front-end server.

• To spare the first user of the day (on a server than just got restarte) from waiting for classes to load, Consider having "fake" users perform transactions so classes are loaded.

Source Code Configuration

7.1 Whitespace and comments are removed from Javascript and CSS code


Whitespace (space characters, extra line breaks, and tab indents) make javascript code more human readable. Generally, the more whitespace, the easier it is to read a source code (which is a good thing).

However, whitespace can take up as much as half the total number of bytes in a source code file.

Since whitespace characters are not needed for programs, whitespace make files take longer to download, which slows performance.

Removing whitespace and comments make it more difficult for others to reverse engineer ("steal") the intellectual property in code.

Make text compression an easy and automatic by using shell/command scripts that also checks code into the source repository. This way, developers continue to work on commented source files while compressed files are in a separate folder referenced by users.

Whitespace in Javascript files can be removed quickly by compression utilities:

JSMIN

Static CSS code compression utilities include:

CSS Optimizer also converts multiple background, font, margin, padding and list attributes into a single attribute, which makes the file faster to render by internet browser clients.

A side benefit of compression is that when programmers know that whitespace will have no impact on performance, they can adopt a more expressive programming style.

Including comments make it more difficult for competitors to reverse engineer ("steal") code.

Source Code Programming Tweaks

Before doing "open heart surgery" on source code, consider these strategies:

• Expedia, Orbitz, and most other travel booking sites throw up a "searching... please wait" screen. This buys your system 30 seconds before users get impatient.
• To avoid being swamped, many sites throw up a "Busy ... Please Try Again Later" screen so that users already in the back-end have a chance to finish normally. Such messages are thrown up by a front-end server.

• To spare the first user of the day from waiting for classes to load, consider having "fake" users perform transactions so classes are loaded.

Java compilers, as with most other programming languages, use several optimization techniques:

• Removing code that can never be executed
• Removing constant operations from inside loops
• Computing common subexpressions only once
• inlining — Moving subroutine code inline (replacing a call to a method with the body of the non-sychronized method being called).
• Optimizing the use of registers and memory
• Optimizing the access of elements in an array

Within applications code:

• Threading to take advantage of additional processors.

• Messages for Transparancy to Developers, Testers, Operations, Users

• Exception Handling

• File I/O, such as tuning SAN data transfer parameters to align with the block size used by the OS and the database system.

• Database calling and Database result set processing

• Memory leak detection, thrashes & how to detect them

• Detecting Thrashing within alogorithms (looping, sorting, parsing, merging)

• Diagnosing object synchronization and deadlocks

Aprobe from OCSystems non-intrusively automatically modifies executables at runtime, while it is in memory, inserting commands at the machine-code level and executed while the application is in process.

RootCause from OCSystems runs on top of the Aprobe technology, providing a user interface that makes Aprobe simpler and easier to use. However, the full power of Aprobe is available under the hood. You can write application-specific probes to monitor anything, collect any data, and perform virtually any task you need.

JVM Enhancements

Major middleware vendors provide their own JVM

Rob Levy, chief technology officer, BEA Systems, claims that BEA's JRockit 5.0 JVM for BEA WebLogic Server 9.0 "continues to impress and set the industry benchmark as the world's fastest JVM for large-scale, mission-critical, server side applications."

On December 09, 2005, BEA and Mercury announced their "strategic alliance":
BEA WebLogic Server 9.0 and JRockit 5.0 JVM downloads will be bundled with, at no additional cost, the Mercury Diagnostics Profiler .
And Mercury Diagnostics (in the first quarter of 2006) will include BEA's Mission Control Suite diagnostic toolset. The JRockit Mission Control suite has diagnostic tools that give deep access into the WebLogic stack in order to monitor, manage, profile and detect memory leaks and performance issues that can occur during development.

JRocket captures and presents garbage collection pauses, memory utilization and CPU usage, as well as information from any JMX MBean deployed in the JVM internal MBean server. This gives developers and system administrators real-time visibility and control over the inner workings and behaviors of multiple JVM instances in a network.

Since capture is real-time, unlike Sun's JVM, initiating a troubleshooting session does not require a server restart.

BEA claims that the tracing is provided "without introducing the performance overhead normally associated with tools of this type."

That's because they count performance "overhead" as part of the standard overhead of the server. And that's as it should be.

So maybe now it won't require an "act of congress" to obtain JVM stats from production machines.

Code Execution Profiling

Profiling is the cental task to performance tuning because it identifies the frequency of class and method invocations, which provides a way to prioritize code optimization efforts that can be very time-consuming.

A 20% improvement to a section of code that executes 50% of the time will yield a 10% improvement.
But a 20% improvement in code that only executes 10% of the time will yield a 2% improvement.

Profiling .NET

Which methods call which methods?

Microsoft PM Gregor Noriskin on Profiling Managed Code with the
Common Language Runtime (CLR) Profiler (v2) for .NET applications

Writing Faster Managed Code: Know What Things Cost (June 2003) by Jan Gray, Microsoft CLR Performance Team describes a low-level cost model for managed code execution time using the

Download a zip file containing the million-plus lines of source code of Microsoft's Shared Source, (Microsoft internal codenamed "Rotor") standardized implementation of the ECMA-335 CLI and ECMA-334 C# language specification.

Release Notes

Shared Source Newsgroups at Microsoft, the University of Pisa, and Developmentor

Writing High-Performance Managed Applications: A Primer by Gregor Noriskin

Garbage Collector Basics and Performance Hints by Rico Mariani

Performance Tips and Tricks in .NET Applications by Emmanuel Schanzer

[Managed] Code Optimization, Sept. 10, 2002 .NET Show "Enter the Programmer" interview of Ivan Mladenov, Patrick Dussud, and Gregor Noriskin about their
Code Optimization Demos

Shared Source CLI Essentials (O'Reilly and Assoc., 2003) by David Stutz, Ted Neward, Geoff Shilling

The Common Language Infrastructure Annotated Standard by James S. Miller (Microsoft architect), Susann Ragsdale, Jim Miller

Invoke Virtual Address Dump (VaDump.exe) Platform SDK Tool with

vadump -p pid

This lists information about the memory usage of the process associated with the specific process id:

• Each address, along with its size, state, protection, and type.
• Total committed memory for the image.
• Total committed memory for the .EXE file.
• Total committed memory for each .DLL file, including system .DLL files.
• Total mapped committed memory.
• Total private committed memory.
• Total reserved memory.
• Information about the working set.
• Information about paged and nonpaged pool usage.

Real-time Application Analytics

LoadRunner Diagnostics with Probes

Panorama — which OPNET (NASDAQ: OPNT) bought in Aug. 2004 from Altaworks.com — identifies potential sources of poor performance by providing dashboards for ongoing monitoring and drill downs into specific application components.

Most impressive is its Analysis Engines which automatically determines which events and metrics are statistically related, and alerts people when deviations exceed norms.

BEA's JRockit Management Console ...

Log Replay

Rather than using artificial (synthetic) scripts to simulate a production load, as LoadRunner does, the "next generation" of software replays server logs to recreate loads.

AppSight from Identify corporation (formed in Petach-Tikva, Israel by ex-Mercury founder Yochi Slonim) has "black boxes" that spew out logs from Windows, .NET, and J2EE servers.

Performant's US$50-80,000 OptiBench 2 product (acquired for $22.5 million by Mercury in 2000) extracts performance data from live, running production J2EE apps by inserting instrumentation byte-code into jar files at each tier:

• HTTP requests from runclients
• JSP requests from web servers
• EJB requests to J2EE app servers
• JDBC requests to database servers

They say their instrumentation creates an overhead of 1-5% in HTTP web servers and 1-7% in the JVM, depending on the threashold set for when workload is captured to files in a repository.

Workload files are used to "playback" workloads within a test environment for diagnosis and for validating fixes/changes.

Note: "Performant" ( kinda like "compliant") is a French word that means "efficient" in English.

Application-Oriented IT Modeling Using Mercury Capacity Planning and Mercury LoadRunner Archive by Rajesh Radhakrishnan, Sr. Director Product Marketing, recorded December 6, 2004

Java 5.0 JMX Java Management (MBean) Extensions

Java 5.0 ushered in a new era for application monitoring and management with its Java Management Extensions (JMX) API implemented as the javax.management package that manipulates MBeans representing a device, an application, or other manageable unit

Platform MBean	Description
java.lang.management.ClassLoadingMXBean	Class loading system of the Java virtual machine.
java.lang.management.CompilationMXBean	Compilation system of the Java virtual machine.
java.lang.management.MemoryMXBean	Memory system of the Java virtual machine.
java.lang.management.MemoryManagerMXBean	Memory manager in the Java virtual machine.
java.lang.management.MemoryPoolMXBean	Memory pool in the Java virtual machine.
java.lang.management.GarbageCollectorMXBean	Garbage collector in the Java virtual machine.
java.lang.management.ThreadMXBean	Threading system of the Java virtual machine.
java.lang.management.RuntimeMXBean	Runtime system of the Java virtual machine.
java.lang.management.OperatingSystemMXBean	Operating system on which the Java virtual machine is running.
java.util.logging.LoggingMXBean	Logging facility.

These names are used by the Sun Reference Implementation (RI) that need to be added to the CLASSPATH. In its zip lib are jmxri.jar (containing implementations of JMX APIs and the agent), and jmxtools.jar (containing adaptor classes such as the HTTP protocol adaptor, class com.sun.jdmk.comm.HtmlAdaptorServer in Sun’s Java Dynamic Management Kit (JDMK)).
BEA's JRockit Management Console prefixes their JMX packages with "jrockit" rather than "java.lang".
IBM and the Adventnet Agent Toolkit JMX Edition use other approaches.

Because JMX is a generic, unified, and standardized framework (and protocol), it has enabled a new industry of JMX management clients (managers) in tools for J2EE tuning, testing, tracing, and monitoring.

• JXInsight and JDBInsight from JInspired from The Netherlands
• Arcturus
• AdventNet Agent Toolkit JMX Edition.

The services JMX provides for application management are:

• Configuration management: Enables querying and defining configuration setting at run time.
• Dynamic class loading: Enables loading management classes from XML files at runtime.
• Event notification: Enables broadcasting information across the management framework.
• Monitoring service: Defines off-the-shelf implementations of event notification broadcasters.
• Timer service: Enables triggering notification events at a specified time interval.

Images from http://skillport.books24x7.com/book/id_5115/toc.asp Managing Java Applications Using JMX API SkillSoft Press © 2003

There are 3 levels to the JMX architecture:

1. At the instrumentation level, MBeans expose the metadata (features) of the application resources to
2. the JMX agent MBean Server at the Agent level, which acts as a container for MBeans by maintaining a registry of MBeans.
3. remote management clients at the Manager Level locate MBeans exposed by the MBean Server through connectors and protocol adaptors.

The m-let service enables the dynamic loading of MBeans in the JMX agent at run time, and the relation service creates and maintains relationships between MBeans.

Different MBean interfaces differ in how they expose the management interface of the application resource to the agent:

• Standard MBeans use Java interfaces static structures (that does not change over time).
• DynamicMBeans interfaces expose metadata tp the getMBeanInfo() method to get various management attributes and operations at different run times.
• ModelMBean JMX interfaces extends the DynamicMBean interface by using descriptor objects to include MBean behavior characteristics, such as persistence and caching.
• OpenMBean is optionally defined and involves the overhead of class loading at run time. Open MBean CompositeData hashmap and TabularData (of primitive data in rows and columns) are derived from the "OpenType" base class.

public Object getAttribute(String attribute) throws AttributeNotFoundException, MBeanException, ReflectionException;

To create and register a standard MBean:

1. Defining the management interface.
2. Defining the standard MBean that will implement the management interface.
3. Instantiating the JMX agent.
4. Registering the MBean with the JMX agent.
5. access remotely.

http://www-128.ibm.com/developerworks/websphere/downloads/

Services for Runtime Settings

Microsoft's runtime for services manages the execution of services according to attributes or configuration settings you set:

• Instancing behavior determines how many instances of a service are created for clients.
• Concurrency behavior controls thread synchronization. For example, a service can indicate that its instances are not thread-safe.
• Throttling behavior puts limits on the number of connections, sessions, and threads.
• Error handling behavior controls how errors are handled and reported to clients.

Related Topics:
Java Garbage Collection 
LoadRunner 
SNMP 
Rational Robot 
Free Training! 
Tech Support