HP/Mercury Diagnostics

This explains (without the sales hype and repetition) how to install and use HP (previous Mercury) diagnostics, with comparison with similar tools (Wily Introscope within SAP Solution Manager). The wisdom here are based on lessons learned through experience.

valid:
Performance Monitoring
SAP Solution Manager
Microsoft Operations Manager
LoadRunner Architecture

Topics this page:

• Usage Scenarios
• How Pieces Fit
• Install Paths
• Machines & Ports
• Install Sequence
• Probes
• Instrumenters
• Profiler
• Diag. Server
• Diag. Reports, GUI
• Points Files, Layers
• LoadRunner Diags.
• BAC Diags.
• Your comments???

Site Map
About this site

HP/Mercury Product Terminology

The terminology used by programs performing similar tasks are different in different products:

Task	AD LoadRunner	Tuning	Profiler	Deep Diag.	AM BAC
Define options	Controller	Tuning Console	Profiler	Diagnostic Console
Run-time options	Scenarios, Schedules	Session Step	Profiler	Diagnostic Console Plans
Collect raw metrics	mdrv Agent	Agent	Probe	Probe	Agent
Emulate clients making requests	Load Generator	Console	n/a	Commander
Receive raw metrics	Controller	Mediator			Diagnostic Server
Calculate and graph results statistics	Analysis	Analysis	Probe	Diagnostic Console

Note:The LoadRunner Controller cannot run a load test scenario during a Tuning Console session.

Overall Installation Sequence

The recommended sequence of installation is this order: BAC > LoadRunner Diagnostics Add-in > Diagnostics Server > Probes > JRE Instrumenter > Diagnostics Collector

• For Business Availability Center 6.2 and Diagnostics to coexist on the same machine, BAC must be installed first.
• To make the drill-down from Real User Monitor available, first open Diagnostics from the menu, then open BAC.
• To avoid crashing LoadRunner 9, first make sure the Diagnostics server is running, then connect to it from LoadRunner..
1. To servers running Norton Anti-Virus, add kill.exe to the exclusion list so that it is not treated as a virus.
2. Install Adobe PDF viewer clients on the Diag. server so documentation can be read there.
3. Set all servers and clients to automatically sychronize time from a common time service
4. BAC
5. Install LoadRunner/Performance Center, if not installed already.
6. Install LoadRunner Diagnostics Add-in for J2EE & .NET
7. Install Diagnostics Server
8. Install Diagnostics Probes on app servers under test
   • Do not run JRE Instrumenter (until later).
   • Do not select "additional products" such as Managed Software Solution.
9. Install Diagnostics Collector for SAP R/3, Oracle 10g database, IBM MQ Series messaging
10. In Server/etc/server.properties file:
    • If diag. servers do not have an internet connection, to avoid sync signals being sent, set timemanager.time_source=SERVER or =NTP (the default) with timemanager.ntp.servers= a NTP server.
    • Set registered_hostname to ip address.
11. To protect changes from being overlaid during re-installation, copy the JavaProbe\etc\auto_detect.points file to a new name specified in the JVM start-up parm -Dprobe.points.file.name=mycompany
    Do not specify the file path nor ".points" file extension.

Unlike other HP/Mercury products, with the Diagnostics Server you can install a license .key file from HP during the initial installation from the CD rather than after installation.

Before installation, close any LoadRunner programs and the Windows Control Panel Services window.

With BAC 6.2, do not specify a password that contains an exclamation point (!). Otherwise, you cannot access functionality related to the Script Repository and transaction monitors in Monitor Administration.

Verify installation and troubleshoot connectivity issues between Diagnostics components by using the Health-Check Monitor Java Applet.

With v3.5, setup programs for Diagnostics Commander, Mediator, and Probe are downloaded from the BAC Admin > Diagnostics "Downloads" tab or http://Diag server/opal/stynamic/downloads.html The file downloaded is for the hostname and port number specified for the Diagnostic server.

To install a J2EE Diagnostics server on a Windows 2000 Professional machine, to avoid the error "Mercury Diagnostics Server cannot be installed on this platform." (KB 36971):

1. Copy the contents of the Windows installation CD onto the hard drive.
2. Go to the J2EE_App folder.
3. Open the tsinst.ini file in Notepad.
4. Change RequiredSoftware = IE4, NT to RequiredSoftware = IE4, NTProfessional.
5. Save the file and run setup.exe to install the software.

On Solaris, installed are ~/vpd.properties and ~/vpd.patches, directories: /var/sadm/pkg/IS* and /var/sadm/pkg/MERQ.

Probes

Information from probes are displayed in:

• LoadRunner Analysis module transaction breakdown feature ("TxBD" for short). or by the
• Business Availability Center "Memory Usage" reports
J2EE (not .NET) Diagnostics also provides a Memory Leak report.

Probe Installation

Diagnostics for Profiler J2EE Installation and User's Guide (redirected from the URL printed in the manual).

According to each version's ReadMe.htm file, each Diagnostics Probe takes:

Version	Platform	Additional Hard Disk Space on Install	Additional RAM to run
v4	J2EE	200 MB	50 MB
	.NET	200 MB	60 MB

Probe Installer steps by admin (not as Unix root operator):

1. Copy the Installation CD to a temporary Installer folder.
2. Edit the installer.properties file to change the value of default
   JAVAPROBE_HOME c:\\Program Files\\Mercury Interactive\\common\\JavaProbe
   to where you would like to install the probe, such as:
   JAVAPROBE_HOME=c:\\MercuryDiagnostics\\JAVAProbe
3. Invoke the DiagnosticsProbeSetup_....exe. It takes a minute or more before you see the first screen to accept HP's terms.
4. Leave the Probe Name entry blank so that the same name is not used for all JVM instances on the machine.
5. Specify a Probe Group Name or "Default" will be assigned by default. On a SAP system, use the 3-letter SAP system name.
6. The location of the Diagnostics Server in Mediator mode Port is by default 2612 and written to the dispatcher.properties and dynamic.properties files.
7. Selecting "Profiler Only" mode sets active.products=PRO in probe.properties and limits monitoring to 5 threads.
8. When "Finish" is clicked, the installer automatically creates a "etc.old" folder containing files in the etc folder. Manually rename it with a date.
9. Run the JRE Instrumenter and copy the appropriate JRE Instrumenter parameter to the start-up script or (WebLogic Remote Start value)
10. Edit the start-up script for each specific server,
11. Set permissions to make sure that users who start the application server has write permissions to the log directory writeable for all users on the system. On Unix:
    chmod a+w /opt/MercuryDiagnostics/JAVAProbe/log

Probe Configuration

A built-in probe is created for host with a fully qualified domain name created by combining the %COMPUTER_NAME% (machine name) + the DNS Suffix Search List from running the "ipconfig /all".

(KB 38840) Probes can be configured from the loadrunner \util\bin folder config.cmd (Windows) or config.sh (Unix) shell scripts.

1. Edit JavaProbe\etc\probe.properties file to set active.products=Enterprise.
2. Set registrar.url=http://CommanderHost:CommanderPort/registrar/ in
   etc\dispatcher.properties if there is no proxy, or
   etc\webserver.properties if there is a proxy.
3. Within the JavaProbe\etc\dispatcher.properties file, note the difference in capitalization of "Port" in entries specifying the range of ports:
   webser.port=35000
   webserver.maxPort=35100

If you open Control Panel > Add or Remove Programs, you'll see "InstallMIGenericHook".

The .NET Probe hooks into applications through two modules (KB 46307):

The first module hooks into the CLR by having its GUID in an environment variable that the CLR reads to see what modules it should find in the Windows Registry which it should then engage.

The second module uses the ADSI .NET API to recognize the applications running on the CLR. It looks for virtual directories and files with the ASP.NET extension, then places hooks there.

In addition to .NET "common" points files (asp.net.points, WebServices.points, lwmd.points), .NET automatically creates a custom points file with the name of the .NET app domain.

JRE Instrumenter

The JVM Instrumenter needs to be run once for each JVM, from All Programs > HP Diagnostics J2EE Probe > JRE Instrumenter

This executes jreinstrumenter.cmd.

Edit the start-up script for the application under test to identify the java path.

After clicking "Add JVM(s)" but before clicking "Search from here", specify "C:" to search the entire hard drive.

Click the path associated with the Java.exe used by the application under test.

Click "Copy Parameter" to put one of these into the Windows clipboard:
"-javaagent:C:\MercuryDiagnostics\JAVAProbe\lib\probeagent.jar"
"-javaagent:C:\BEA\jrocket90_150_04\mercuryprofiler\lib\probeagent.jar"
"-Xbootclasspath/p:C:\MercuryDiagnostics\JAVAProbe\classes\boot"

-Xbootclasspath/p:C:\MercuryDiagnostics\JAVAProbe\classes\boot
is the profiler's own JVM.

To use JRockit that comes with BEA:
-Xmanagement:class=com.mercury.opal.capture.proxy.JRockitManagement
"-Xbootclasspath/a:C:\MercuryDiagnostics\JAVAProbe\classes\boot"

Edit/Past Into Application Start-up Script JVM Options

Edit the app server's startup script and paste.
On SAP Portal servers, use
/usr/sap/SID/instance/j2ee/ConfigTool.

specify a different probe name:

-Dprobe.id=Japp_1
-Dprobe.id=Japp_2

-Xbootclasspath must be placed at the beginning of the Java parameters, following any JIT options such as -hotspot or -classic.

Instrumentation

If you click Memory Analysis and get this error message , add to the JVM parameters (on a Windows machine):
-agentpath:<MercuryDiagnosticsProbeInstallPath>\lib\x86-windows\jvmti.dll

JVMTI (Java Virtual Machine Tool Interface) was introduced with JVM 1.5 to replace the
JVMPI (Java VM Profiling Interface) and BCEL service in JVM 1.4.
These perform "automated" instrumentation needed to cause a system to produce output useful for diagnosis, where compiled jar files are searched and the byte codes automatically modified using "interposition" techniques which substitute a system component with a pass-through component that records the calls made to it. to capture calls at various points during program execution.

In constrast, "manual" instrumentation consists of programmers adding statements to source code (that is often difficult to fully remove),

Unlike desktop IDEs like JProbe, which weight heavily on a machine, HP Diagnostics is setup by default to track only those points which take the longest time or consume the most resources.

Starting (Launching) Probes

In a Windows Run command window, navigate to C:\MercuryDiagnostics\JAVAProbe\bin and invoke
startAgent.cmd
startAgent.sh

Internally, this invokes java with the -Xbootclasspath parameter that specifies \lib\probe.jar using "com.mercury.opal.capture.JavaProbe true".

Disabling Probes

There's reverse logic here, but if you want to stop profiler, set:
in etc\probe.properties disable.profiler=true

in etc\dispatcher.properties #ac.register.sink=false disables Profiler for method collection.

Profiler

The Profiler is a JETTY (Java-based open-source stand-alone) web server. It can be started from All Programs > HP Diagnostics J2EE Probe > Diagnostics Profiler for J2EE, which starts JavaProbe\bin\profiler.cmd

http://localhost:35000 retrieves the probe management app: (the default user name and password is admin/admin). When an internet browser enters the site, several components are automatically downloaded.

	Open Diagnostics	http://localhost:35000/profiler
	Click "Begin Profiling" if some of the tabs are greyed out.
	Advanced Options	http://localhost:35000/maintenance
	inst = Instrumentation Utilities	http://localhost:35000/inst
	files	http://localhost:35000/files
	security = Manage Authorization and Authentication	http://localhost:35000/security is redirected (on a new window) to the Diag. Server if one exists.

Diagnostics Server

http://localhost:2006 retrieves the Diagnostics Server management app:

	Open Diagnostics	http://localhost:2006/topaz/... This is the same Java Applet GUI as the LoadRunner Controller "Diagnostics for J2EE/.NET" tab.
	Configure Diagnostics (System Health Graph is within Registrar)	http://localhost:2006/maintenance Component (Maintenance) advanced (v7) configuration advanced (v7)
	Manage Authorization and Authentication	http://localhost:2006/security

http://localhost:2612 should not be accessed. It retrieves the Mediator app:

com.mercury.opal.common.io.mediator.EventChannel.Incoming

Diagnostics Server Installation

Since v4, the Mediator and Commander are installed using the same "Diagnostic Server" software.
Previously, in v3, mediator and commander were installed separately.

On UNIX:

• the default folder is /opt/MercuryDiagnostics. But some organizations do not provide enough room or not allow installations into opt, so you may have to use another folder under /usr or some other folder.
• Rather than running the installer as root, after su -someuser
• Unix also needs "chmod a+w /opt/MercuryDiagnostics/JAVAProbe/log".
• With IBM AIX, "set IBM_JAVA_OPTIONS" to clear them out before running the installer.

Both the Controller and Mediator sends (HTTP protocol) de/activation instructions to the Probe through port 2005. The Mediator also sends synchronization instructions to the Probe.

Because of their tight hand shaking, both the Mediator and the probe need to have the same logical LAN ID (e.g. LAN1, LAN2, or "DefaultLAN" if there is only one LAN).

When a Diagnostics Server is deployed in Commander mode, it has both commanding and mediating responsbilities. This separation is needed to accomodate large system landscapes with many servers, The Mediator pre-processes, filters, and aggregates real-time data before sending it to the Diagnostics server/LR Agent every 30 seconds (by default).

Real-user transaction data the Mediator sends offline every ten minutes.

LoadRunner's Agent process connects with the Mediator. The LoadRunner UpdateService is installed on a Windows 200X machine (separate from the Controller) with "J2EE Mediator Installation" in Start > All Programs.

Probes continuously send raw performance data (using a proprietary protocol) to the Mediator listening port.

If several application servers (and their JVMs) are monitored by a single Mediator, that Mediator should reside on a separate mediator host machine. Each Diagnostics Mediator can process a limited number of probes. The readme file has these recommendations:

Item	Platform	Total Number of Probes
		up to 10	up to 20	up to 30
Machines	Windows	1 x 1 GHz	1 x 2 GHz	2 x 2.4 GHz
	Solaris	1 x Ultrasparc 2	2 x Ultrasparc 2	2 x Ultrasparc 3
Memory	Windows	768 MB	1 GB	2 GB
	Solaris	1 GB	1.5 GB	2 GB
Java Heap		350 MB	700 MB	1400 MB

The Java heap -xmx setting should be less than 75% of physical memory (e.g., -xmx768m for a machine with 1GB of RAM installed).

The installer sets up the HP Diagnostics Server service to start automatically upon start-up from C:\MercuryDiagnostcs\Server\nanny\windows\bin\magentservice.exe. Alternately, this can be manually started or stopped from

Start > Programs > Diagnostics Server or from
Start > Settings > Control Panel > Administrative Services > Services

This service needs to be stopped before you can run the installer again, or you'll get this error message

The Diagnostics Server uses rhttp -- revershe HTTP -- to query metrics from J2EE probes.

HP Diagnostic Collector

The Diagnostic Collector collects metrics from SAP R/3, Oracle, and MQ series servers.

Create SAP JCo files within the folder before starting the installer.

Do not copy the librfc32.dll to windows/system32 unless it's a later version (6405.5.165.5297 vs. 7100.1.7.5371 from a more recent SAPGUI install).

Run localhost:2006/security/EncryptPassword.jsp to encrypt the password, which is pasted within file
C:\MercuryDiagnostics\Collector\etc\r3config.xml
tag <encrypted-password>
Then replace "ChangeMe" with information for the target system under test.

Within file \etc\monitor.properties, set xml.filename=sapr3.metrics.xml

From Start > HP Diagnostic Collector, Stop and start it again.

Diagnostics System Health Monitor

The "System Health Monitor" is a web page (graphic Java Applet) at http://localhost:2016/registrar/health (and providing default username and password "admin").



An application must be running before that node turns from Inactive gray to active green.

Activity level can be configured to be illustrated two ways after Right-clicking on the graph: "Show Load Using Circle" size or the default "Show Load Using Scale" length of the black line under the component.

Diag. Server Application Groups and Applications

The Entire Enterprise application predefined by default includes the entire managed environment context (all Diagnostics views and components).

Define application groups to organize groups of related composite applications and other application groups.

Define composite applications to logically group Diagnostics entities (such as probes, hosts, server requests, etc.).

Examples are Division A vs. B, HR vs. Finance.

Applications are not shown within LoadRunner or Load graphs.

Configure/Enable Reporting

To keep overhead low, by default several views are not shown or does not show all information. So they must be manually enabled.

CPU Time Java is enabled in Transaction, Probes, and Call Profile views if:

probe\etc\capture.properties file entry use.cpu.timestamps=true and
probe\etc\dynamic.properties file cpu.timestamp.collection.method=
1 only for server requests (the default), 2 for all transactions/methods, 0 for no timestamping

Profiler Light Weight Memory Diagnostics (LWMD) is used to identify the largest collections and the fastest growing collections within Java applications. Page 343 of the Diag. 7 User Guide says:

probe\etc\dynamic.properties file entry
lwm.diagnostics.capture=true and
lwm.diagnostics.top.n=15 (to only show the top 10 collections by size or growth)
lwmd.resource.replace (pattern replacement of code resources, as in uri.pattern.replace)
probe\etc\auto_detect.points file where key = lwmd set to active=true

Profiler Memory Analysis with Heap Walker (for JVM 1.5+) or Heap Breakdown (JVM 1.4) are used to identify objects that may be leaking. This doesn't work at all on IBM JVM (due to a defect on the IBM VM, per KB 40630).

Edit the app server's startup script to add (after -Xbootclasspath) -Xrunheapdump to enable Heap Dumps.

Edit
probe\etc\capture.properties file entry use.cpu.timestamps=true and
probe\etc\dynamic.properties file cpu.timestamp.collection.method=

1 only for server requests (the default), 2 for all transactions/methods, 0 for no timestamping

Thread Waited and Blocked Time columns can be included in the Probe Threads Table by setting

threads.contention.monitoring.enabled property=true

To graph latency for a specific method (point), specify in the Probe\etc\auto_detect.points file layertype = trended_method for each point

Memory Configuration

From Diagnostic Server Configuration > Memory Diagnostics > Worst Collection Limit (The number of "worst" collections that should be tracked) defaults to server.properties file parameter bucket.lwmd.topN = 10

Points for Exploration

Classes and methods displayed on views are specified in the file by specifications in the points file.

Define separate points files for Dev, QA, and Prod.
In QA, keep defaults (Synchronization inactive but core SAP points activated).
In Dev., activate Synchronization and inactivate core SAP points.
During pseudo-production measurement runs, do not use no Synchronization and inactivate core SAP points.

Save a single points file and copy it to each it to each of several identical clustered machines using a custom batch file.

Instead of directly editing the points file, click Profiler > Configuration for the screen on the right.

Check Capture Class Map.

In Change Probe Instrumentation Plan > Shared Instrumentation, click "Edit..." button for this GUI:


The red icon indicates that "active=false" is set in the Source for that point.

Pressing the Source tab shows the points file at the point selected (appearing between [square brackets] at the bottom of the Source screen).

"layer" properties for each point is the points file is the basis for the Layers shown in Load views.

The points file is saved to the file after both OK and "Accept" are pressed.

CrossVM > RMI correlates the call tree data from both calling and called virtual machines with this point specification. Notice that the keyword is displayed below the layer.

[RMI] 
keyword = rmi
layer = CrossVM
active = true

Instrumentation for OVTA (HP OpenView Transaction Analyzer) Java Servlet Filters and EJB local home methods are invoked by these points:

Business Tier
EJB
LocalHome
EJBLocalHome
Web Tier
Servlet
ServletFilter
HttpCorrelation2

[Synchronization], when activated, identifies a common source of performance problems.

Avoid creating a separate custom layer. Also, custom packages should ignore utility methods (get, set, is):

[MyClass_Example_1]
class  = !com.MyCompany\...
method = !.*
ignore_method = !(set.*)|(get.*)|(is.*)
signature = !.*
layer = MyCompany/LOB/MyClasses

The classes discovered during probe installation are specified in file lwmd.discovered.points within the probe\etc folder.

## In the absense of a ClassMap, this file contains all the classes which 
## will be instrumented for light-weight memory detection, or were detected 
## at runtime to be of interested to light-weight memory detection and will 
## be instrumented next time the application is restarted

## Each class which should be tracked for light-weight memory diagnostics
## should be listed here, each on its own line.

## If you remove a collection to prevent it from being instrumented on a
## subsequent run, you need to add it to the lwm.diagnostics.filter.exclude
## property inside dynamic.properties in order to prevent the collection from
## being discovered & added back into this list when the probe next restarts.

Other points file parameters:

class = <class/package name/s to capture>
ignore_cl = <classes to exclude>
method = <method name/s to capture>
ignore_method = <method prototypes to exclude>
signature = <signature/s of method/s in javap symbolic encoding>
layer = <layer name>
active = <true, false> #Enabled: checked in GUI

layerType = <layer type>
deep_mode = <soft or hard mode>
scope = <comma separated list of methods>
ignoreScope = <comma separated list of methods to exclude>
ignore_tree = args,caller,return <Exclude subclasses>
detail = <args,caller,return>
code-key = <key that you generate to secure the code snippet>
tierDefinition = <hex_identifier>
merge_classes = <class names>
merge_url_to = <comma separated list of methods>

[Leak] should be specified for only a single class. Do not simply activate it.

Layers

The layers in HP Diagnostics Points files

Layer	Sub Layer(s)	Protocol	Description
1. Client		HTTP/ HTTPS	requests to/from presentation layer web servers.
Portal	SAP	-	SAP Portal Core Components
Web Tier	Session	HTML	javax.servlet.http.HttpServletRequest
	Servlet	-	javax.servlet.http.HttpServlet javax.servlet.Filter	-
	JSP	-	javax.servlet.jsp.HttpJspPage
	SAP	-	SAP Portal Web Service Components
	Spring	-	-
	Struts	-	-
Application	Spring	-	keyword = soapfault
XML	SAX	-	org.xml.sax.XMLFilter
	DOM	-	(expensive!)
Web Services	Generic	SOAP/ REST	requests handled by web servers. keyword = soapfault
	SAP	SOAP	com.sap.engine.services.webservices. ... ...runtime.ProtocolChainProcessor ...jaxrpc.wsdl2java.soapbinding.MimeHttpBinding
SAP-RFC		RFC	keyword = saprfc
Directory Service		LDAP	JNDI (method: lookup) javax.naming.Context
JSP (Java Server Pages)		markup	-
Messaging	JMS	-	keyword = jms-client javax.jms.Connection, MessageConsumer, MessageListener, QueueSender
		(Java Mail Services) requests sent to email servers.	keyword = jms-server javax.jms.TopicPublisher
JTA (Java Transaction API)		-	-
Database	JDBC	SQL	java.sql.Connection & Execute via Oracle Net protocol
JDBC-SQL		SQL	keyword = jdbcsql
RMI (Remote Method Invocation)		RMI	-
RPC (Remote Procedure Call)		RPC	-
Logging	Log4j	-	javax.resource.cci.Connection
Logging	Log4j	-	org.apache.log4j.Logger Method: trace
JMX		JMX	javax.management

System Tiers

The flow of data Layers are illustrated here.

Diagnostics group performance metrics by layers and sub-layers based upon the resources that the application invokes to perform the processing. These layers make it easier for you to isolate and identify the areas of the system that may be contributing to performance issues.

Allocations

[Leak] point

Probe Profiler Tabs

Profiler supports IE only.

Summary

• Memory
• Load
• Shortest Requests

Hotspots

• Slowest Methods
• CPU Hotspots (Methods)
• Slowest SQL

Metrics

• System (Host) CPU, Memory Usage, PageInsPerSec, PageOutsPerSec, PageCutsPerSec, Disk, Network
• JVM: Probe: HeapFree, HeapTotal, HeapUsed
• Java Platform: Classes, GC, Threads
• Mercury System
• Weblogic: EJB, Execute Queues, JDBC, etc.

Threads (JVM 1.5)

All Methods

All SQL

Collections (using LWMD)

Exceptions

Server Requests

Web Services

Allocation Analysis

Memory Analysis (Heap Breakdown)

Configuration (without restarting app)

Diagnostic Report Views

It takes several minutes for diagnostics to appear.

The default left pane has:


View	Entity	Metric
Standard Views	Server Summary - Status, Aggregate Request Aggregate Requests - Latency Application Metrics - Latency Dependent Service - Latency Hosts - CPU Load Outbound calls (none) Probes - VM Heap SQL Statements - Latency Server Requests - Latency Topology Transactions - Latency Trended Methods - Latency	-
Portals	Server Summary - Portal Requests & Portal Components Portal Components - Latency	-
"Web services" (HTTP/SOAP or HTTPS/SOAP-RPC calls to/from remote components via internet)	Outboud / Consumer	-
	Inbound / Provider (aggregating multiple consumer IPs)	-
	Inbound / Provider by Consumer IPs of origination listed in JavaProbe/log/probe_name_addr.properties file (up to max.tracked.addresses.per.probe in JavaProbe\etc\dispatcher.properties)	-
SAP	NetWeaver Summary (Requests & Threads) R/3 Summary - NetWeaver Requests - HTTP Requests/Sec NetWeaver Threads - Application Threads/Active Threads R/3 Probes - Latency R/3 Server Requests - Latency	-
Oracle	Summary - Probes Wait Time	-
BEA Weblogic	Summary EJB Pooled Resource Connection JDBC Connection Status JDBC Resource Contention Server Threads	-
IBM WebSphere	-	-
CICS	-	-


Alerting
My Views

The Probe captures the call stack for the 3 slowest instances of each server request. It also captures a call stack representing all call instances for a type of service request and calculates the aggregated latency.

The default BEA Weblogic contains:

Diagnostics GUI Controls

My preference is to stay in My (customized) Views. We are pretty much forced to because changes to column sequence or which columns are visible or hidden (by clicking on the grey Header) are only saved within custom views. This dialog also lets you arrange things.

Default views are not customizable. Hide (Close) Views from the View Bar by clicking view bar open space, not an icon. Create View Groups by right-clicking on the white area within My Views.

Save Server Summary to My View after adding the Current Logons Count column.
Save Server Requests to My View after adding an Exceptions column.

Only 100 graph entities (probes, hosts, server-requests, etc.) are shown in each view's entity tables. Search for entity text by pressing Ctrl+F or use a different sort and hope the one you need shows up.

If you define a sort sequence with more than one column, save a screen shot of it because clicking the grey heading unselects the sort order. Clicking on the header again switches ascending to descending. Clicking a third time removes the column from sorting. To do all this with subsidiary sort sequences, hold down Ctrl while clicing the subsidiary header.

To close the Diag. View and go back to the Application screen, press Ctrl+F4.

To view (point-in-time) Analyze Snapshots, select Snapshot mode in the View Context Drop-Down.
For instructions on creating snapshots, select (Real Time) Monitoring mode.
You can't use Shift/Ctrl as in Windows Explorer, but right click each individual line to add to an active snapshot.

To ensure that metrics collected represent processing when applications are running in a steady state:

• warm up the system with a range of different transactions (to realistically populate the cache), then
• Press the J2EE Diagnostics Profiler "Reset" so it force-drops all call stacks captured and zeroes out baselines used to calculate instance counts, average latency, slowest latency, etc.
• Press the J2EE Diagnostics Profiler "Garbage Collection" trash to request a full garbage collection.

To go back one level in the breadcrumb trail, press Backspace or Alt+LeftArrow.

Export to HTML is designed well - file names contain details, including a date/time. This enables you to quickly save several pages.

Filter Time and Disk Space Usage

Time Range Resolution	Sub folder	Update Frequency
5 minutes	Hours\5m	5 seconds
20 minutes	Hours\20m	3 minutes
hour	Hours\1h	4 minutes
6 hours	Days\6h	8 minutes
day	Days\1d	11 minutes
week	Weeks\7d	29 minutes
month	Months\1M
3 months	Years\3M
year	Years\1Y

Diagnostics conserves disk space by overwriting the same "buckets" reporting values in a fixed number of time resolution zoom periods (from low to high resolution).

The "Sub folder" in the table is within the directory where, over time, *.summary.zip, *.trend.zip, and *.tree.zip files pile up (persist):
C:\MercuryDiagnostics\Server\archive \mediator-%COMPUTERNAME%\persistence \Mercury System_\...

This is also the schedule of how LoadRunner extends update frequency automatically during a long run. Update frequency is every 5 seconds at the beginning of a Whole Scenario, but extends to 4 minutes an hour into the scenario.

For Less frequent point duration (granularity of aggregation):

probe.trends.pull.interval (from every 5s to 30s)
probe.trees.pull.interval (from every 45s to 120s)

Change Diagnostics Server Mediator mode offline aggregation period in file Server/etc/mediator.properties

bucket.lr.offline.duration and
bucket.lr.offline.sr.duration

The Mediator stores logs in the log folder where it's installed (by default, in C:\MercuryDiagnostic\Server\log).

#	File Name	Description	Space	Note
1	server.log		-	This fills up fast
2	server-full.log		-	Usually 0 unless there's a problem.
3	server-metrics.log		-	Usually 0 unless there's a problem.
4	server-jetty.log	Event log entries suchs as: 2006-06-29 11:14:12,524: INFO jetty JettyStreamLogSink.log[177] : EVENT [Thread-8]org.mortbay.http.HttpServer.setStatsOn(HttpServer.java:1063)14> Statistics on = false for org.mortbay.jetty.Server@1b04ff7	-	This fills up fast
5	server-alerting.log		-	Usually 0 unless there's a problem.
6	server-useradmin.log		-	Usually 0 unless there's a problem.
7	server-system-properties.log	Lists the values from properties files configuring the Diagnostics Server.	-	Usually 3 KB.
8	server.out	Diagnostics webserver listening on 0.0.0.0:2006	-	Usually 3 KB.
9	server.err		-	Usually 0 unless there's a problem.

Call Profile Graph from Instance Trees

This Call Profile Graph displays instances trees collected during the last 5 minutes. They different types are:

• the slowest (maximum) and second slowest invocation
• the fastest (minimum) invocation
• the median (Average) invocation time
• a randomly selected end-to-end trace of the entire cross-VM tree which includes RMI and other outbound calls to an external VM. If the remote system is also instrumented, it includes the trees of remote systems.

To identify what methods take longer under load, compare graphs for Average versus Slowest method times

Aggregate Trees keep all samples collected and combines multiple possible execution paths (instance trees) into a single tree, so it's more difficult to understand what the application is doing.

Tuning Console for LoadRunner

I am told that this product has been deprecated by HP.

The ability to automatically invoke several runs of LoadRunner (with different configurations) is really helpful.

Even more importantly, comparison of results across several runs is great to have.

I've developed an internal tool to do this.

Diagnostics LoadRunner Add-in for J2EE & .NET

Diagnostics for J2EE & .NET for LoadRunner Read Me

As the LoadRunner Diagnostics tab initial screen instructs, Probes need to be enabled within LoadRunner's Diagnostics menu:

This brings up the dialog shown at right.

Check "Enable the following diagnostics" and adjust the "% of all the relevant Vusers in the current scenario" to 100 when you are exploring SAP.

Within the "Offline Diagnostics" area, clicking the "Configure" button for "Web Page Diagnostics" retrieves this dialog .

Clicking "Troubleshoot Diagnostics for J2EE/.NET connectivity" invokes a new internet browser window containing the System Health Monitor.

In the LoadRunner Launcher, when you select CONFIGURATION Diagnostics for J2EE/.NET Setup
expect to see a dialog such as this:


To create offline AM/Standalone datasets for LoadRunner Analysis to read, set Diagnostic Server > Configure Diagnostics > Configuration > Show Advanced Options > Online Cache to true (which automatically sets Server\etc\server.properties file parameter copy.ad.to.am = true).

For a list of files and runs that are open and still collecting data, go to Diagnostic Server > Configure Diagnostics > Configuration > query.

Within Diagnostic Server > Configuration > Aggregation modify LoadRunner Offline/Summary VU

Vu Aggregation Period	bucket.lr.trend.duration = 5s
server request aggregation period	bucket.lr.offline.sr.duration=5s
Business Availability Center (Topaz) BPM Aggregation Period (must be multiple of Summary Trend duration)	bucket.tz.vu.duration=30s

LoadRunner Transaction Breakdown Diagnostics Module

The J2EE Transaction Breakdown Diagnostics Module includes the following components:
• Console: Displays the transaction data.
• Mediator: Processes the transaction data, and then passes it to the Console.
• Probe: Resides on the application server. Gathers information about the monitored transactions and passes it to the mediator.

The J2EE Transaction Breakdown Diagnostics Module requires MSXML 3.0 or later (included in Internet Explorer 6.0) installed from this Microsoft MSDN Web page.

Probes collect time spent only within J2EE/.NET application servers, so their timings do not include user think time, time spent traveling the network, nor time in the web server. For this reason:

"J2EE/.NET - Transaction Response Time Server-Side Graph".

Diagnostics Usage Summary Report

The Summary Report is titled "Diagnostics Usage"

There are two groups of J2EE & .NET Diagnostics graphs:

• J2EE & .NET Server Diagnostics Graphs show the performance of requests and methods
  generated by real and virtual users without connection to any transaction:
  • J2EE/.NET - Server Request Response Time (a line for each request)
  • J2EE/.NET - Average Server Method Response Time (a line for each request)
  • J2EE/.NET - Server Requests per Second (a line for each request)
  • J2EE/.NET - Server Methods Calls per Second (a line for each request)
  • J2EE/.NET - Average Number of Exceptions on Server
  • J2EE/.NET - Average Number of Timeouts on Server
• J2EE & .NET Diagnostics Graphs show the performance of requests and methods
  generated by vuser transactions (but not by real users):
  • Average Method Response Time in Transactions
  • Transaction Time Spent in Element (per method/query)
  • Transactions Per Second (for all transactions)
  • Method calls per second in Transactions (a line for each Transaction)
  • Breakdown information by layer, class, and method
  • Chain of calls and call stack statistics
  • Average Number of Exceptions in Transactions
  • Average Number of Timeouts in Transactions

Rather than using automatic transactions specified in Runtime Settings, lr_start_transaction and lr_end_transaction statements must be inserted manually around each transaction in the Vuser script.

"Elements" within each transaction and request are :

• Layer (System Tier)
  • Class (associated with an object/bean)
    • Method (within each Class)

Filter values can be defined for these elements plus:

• Scenario Elapsed Time
• Transaction Name
• SQL Logical Name (shown in Measurement Description)

Transaction Chain of Calls and Call Stack




The most time consuming chain is shown in red.

Click on a method and click the icon for the reverse bottom-up view of the Method Call Stack:




A view group is pre-defined for each type of server (WebLogic, WebSphere, etc.).

Views can be by Load, Probe, and Server Requests.

Drill down from a view group to detail views for each probe or host.

The Metrics Inspector is like a properties GUI control used to specify the threshold values controlling whether each metric displays on the graph and detail table.

Business Availability Center (BAC) x.x Diagnostics for J2EE & .NET

Business Availability Center comes in several CD's:

• Diagnostics for J2EE & .NET v3.5 - Supporting Business Availability Center 5.x
  1. Unix Installation
  2. Windows Installation
  3. Patches
  4. Probes

BAC Performance (Dashboard) Customizing the KPI, Rules, parameters

BAC Versions

BAC Version	Diag. Version	Date	OS Support	App Server	Database	Web Server
6.3 (VuGen 8.1)	4.1	8/06	Windows 2000 Server/Advanced Server (SP4), Windows 2003 Standard/Enterprise Server (SP1) Solaris 8, 9, 10	WebLogic 8.0	Oracle 10.2.0.1, SQL Server 2000 Enterprise SP4	IIS 5,6; Apache 2.0.49; Sun Java System Web Server 6 SP 7
6.2 (VuGen 8.1)	4.1	6/06	Windows 2000 Server/Advanced Server (SP4), Windows 2003 Standard/Enterprise Server (SP1) Solaris 8, 9, 10	WebLogic 8.0	Oracle 10.2.0.1, SQL Server 2000 Enterprise SP4	IIS 5,6; Apache 2.0.49; Sun Java System Web Server 6 SP 7
6.1 (VuGen 8.1)	4.0 SP1	-	-	WebLogic 8.0	Oracle 10.2.0.1	Sun iPlanet
6.0 (VuGen 8.1)	4.0 SP1	-	-	WebLogic 8.0	Oracle 10.2.0.1	Sun iPlanet
5.1 (VuGen 8.0)	3.5	6/14/05	Solaris 10, Windows 2000 Server, Windows 2003 Server (all editions)	WebLogic 7.0 SP6
5.0	3.3	11/12/04	Solaris 9, Windows 2000 Server SP4, Windows 2003 Standard Edition	-

Types of Monitors

The types of monitors managed by BAC are classified by this hierarchy:

EUM - End-User Management monitors:
• BPM - Business Process Monitor
• RUM - Real User Monitor
• CM - Client Monitor

System monitors:

• SiteScope (SIS)
• EMS - Enterprise Management Systems such as SNMP, Peregrine, Remedy, Tivoli, BMC Patrol, HP OpenView)

The "IT Universe" within BAC consists of the Business, Monitor, and System.

Business items include:

• Business Processes
• Business Process Steps
• Group
• Line of Business

BAC Workflow

The BAC dashboard is a portal that presents HTML aggregated after consuming one or more portlets on WSRP (Web Services for Remote Portlets) service producers which publish WSDL to a WSRP Registry using UDDI and invoked using SOA, like other XML services. WSRP was first approved at OASIS standard 1.0 on September, 2003. v1.1 was approved May, 2004.

Monitors generate "samples" of activity data carried over a "bus".
Business Logic Engine (BLE) matches (filters) samples against the model of CIs (Configuration Items) stored in the CMDB (Configuration Management DataBase) central repository.

Matched samples that fit the model light up the "dashboard" of real-time calcuations and status.

The Alert Engine handles alerts associated with KPIs (Key Processor Indicators)

Diagnostics Management Database

Deep Diagnostics setup (file DeepDiag3.3.11.2VM.exe dated 11/24/2004) is a Java SWT app. It installs by default to folder "C:\MercuryDiagnostics". By default it installs MSDE 2000 and MDAC 2.7.

The Diagnostics Management database is not needed since v3.5.
With v3.3, if the Diagnostics management database is on a different machine than the machine hosting Diagnostics server, they both need to be set to the same time zone. The database is configured in Start > Programs > Mercury Diagnostics Server > Administration > Connect to Database

With v3.5, diagnostics components "can" store data from over 72 hours of runs with 20 SUT's running load tests for both .NET and J2EE probes. However, The System Health Monitor will not report an error condition on the graph when the Diagnostics components are out of disk space.

Web server

An open-source web server is installed using file jakarta-tomcat-3.2.3.tar.gz in the shared\third-party folder of installation CDs.

"Select the Web Server type you want Application Management to work with."

• Apache HTTP Server

The Diagnostics application only works with IE6 browsers, with privacy settings set to "Always accept cookies from this site."

The SMTP server name needs to be specified during installation. Its Sender Name is by default MercuryAM_Alert_Manager.

Configuration

Run JRE Instrumenter

BAC Diagnostic breakdowns need to be enabled for each BAC Business Process Transaction Monitor.

In each server's startup script, add (after the JIT option) a -Xbootclasspath Java command line parameter reference to the <probe_install_dir>\classes/boot

Also add to the CLASSPATH=%CLASSPATH%;%J2EE probe installation directory%\dat

With LR7.8SP1, from the J2EE Transaction Breakdown Home Directory\lib directory, run the Initializer

java -jar sipatool.jar

This puts ClassLoader.class under directory Probe Home Directory/classes/boot/java/lang.

Each probe writes to a <probe_id>.log file in the <probe_install_dir>/log folder.

The probe of each server must be enabled before it can monitor events.

Lastly, run the System Health Check to view the component's configuration information and performance metrics.

The Profile database stores transaction breakdown summary information in its event_meter table, em_download_data_size, em_s_network_fb_time, and em_s_connection_time columns for each transaction.

RUM (Real User Monitor)

Scripts for the Real User Monitor are developed by the VuGen software borrowed from the LoadRunner product.

The installer creates file %WINDDIR%\vpd.properties which should be deleted before the product is re-installed (after a mandatory uninstall).

Resources

Search for word:
in this site in all googled sites in Whatis.com in Wikipedia in Technorati in Amazon.com in Crosswalk.com

• Performance Testing
• Performance Tuning
• Performance Reports
• LoadRunner



• TCP/IP Addressing
• ISO-OSI Layers
• Routing & Switching