How I may help
LinkedIn Profile Email me!
Call me using Skype client on your machine

Permalink HP/Mercury Diagnostics

This explains (without the sales hype and repetition) how to install and use HP (previous Mercury) diagnostics, with comparison with similar tools (Wily Introscope within SAP Solution Manager). The wisdom here are based on lessons learned through experience.

RSS 2.0 XML feed on SAP valid:
Performance Monitoring
SAP Solution Manager
Microsoft Operations Manager
LoadRunner Architecture


Topics this page:

Site Map Go to a list of all pages on this sites
About this site About/Help on this site
First Topic

HP Diagnostics at HP:
product page
classes and certification

When HP bought Mecury in 2006, it kept the name BTO (Business Technology Optimization)A website external to this site to group various products from HP.

Set screen Diagnostics Usage Scenario

Go to top of page Why? The Benefits

    Diagnostics falls into the middle ground between Quality Assurance and Operations Performance Validation.

    For developers, having Diagnostics means that tracing code doesn't have to be added and removed. This is a big side effect of why diags can improve performance.

    The Launcher's Diagnostics tab reads:

    Diagnostics is the science of pinpointing the root cause of a problem. LoadRunner is the first load testing tool to provide a set of Diagnostics modules that trace, time, and troubleshoot end-user transactions across all tiers of the system. These modules extend LoadRunner to provide a unified view of both end-user experience and application component (method, SQL) level performance. The intuitive visual interface allows the user to drill down from a problematic business process all the way to the poorly performing component. This granularity of results ensures that every load test provides development with actionable results, thus reducing the cost and time required to optimize J2EE/.NET applications.

    Mouse over the Launcher's "Diagnostics for J2EE/.NET" for

    Diagnostics for J2EE/.NET provides complete visibility into the transaction performance of J2EE/.NET- based applications. Users can drill-down from a poorly performing end user transaction and view JNDI, web, EJB, and JDBC performance. Having identified the top-time layer, users can further drill-down to the Method or SQL level and pinpoint the slow SQL query or method impacting the end-user. This granularity of results data ensures that every load test provides development with actionable results thus reducing the cost and time required to optimize J2EE/.NET applications.

Go to top of page How the Pieces Fit Together

Go to top of page Tuning Console LoadRunner Controller Diagnostic Analysis Diagnostics Commander Files Load Generator Client Emulation Profiler Deep Diagnostic Console Deep Diagnostic Collector Server Under Test Rstat.daemon shell script JRE Instrumenter JRE Instrumenter Metrics Metrics Probe Mediator Diagnostic Server Filtering and Aggregation Business Availability Center Dashboard within BAC Business Process Manager Business Process Playback Dynamic Configuration SiteScope SiteScope SSH Full Metrics from SiteScope Other Metrics from SiteScope BLE RUM HP (Mercury) Diagnostics Go to top of page pop-up Pop-up in a new window.

  1. Various metricsanother page on this site (such as JVM heap size, garbage collection frequency, method invocation counts, etc.) are grabbed by
  2. Probes (aka the "systemmetrics" daemon) which pass metric data out to the
  3. Profiler web service (installed with and runs on the same server with the probe) to
  4. produce web pages in HTML or XML or format which can be parsed dynamically by
  5. Scriptsanother page on this site running within LoadRunner (Performance Center)another page on this site programmed to store diagnostics values as user-defined values along with metrics maintained by LoadRunner (such as the number of vusers running concurrently). This LoadRunner Controller sends scripts to
  6. Load Generators which creates artificial (synthetic) loads on servers by running
  7. scripts where virtual users emulate working clients.
  8. LoadRunner can determine the overall impact of load on servers from basic metrics obtained by emulating operators using SSH (Secure Shell) communications to
  9. rpc rstat.d daemons that collect metrics such as CPU Utilization on UNIX operating systems. However,
  10. Java apps running within Java Virtual Machines need their start script parameters modified with a parameter obtained from the
  11. JRE Instrumenter.
  12. Operator commands through SSH (Secure Shell) connections to collect data from servers are also used by HP's SiteScope product, which can also craft more complex requests and process responses than LoadRunner.
  13. Sitescope is able to obtain metrics from many other sources, including internal Oracle database stats. SiteScope's full range of measurement dataanother page on this site referenceable by LoadRunner and other software, such as the
  14. HP(Mercury) Tuning Console on this page product which tracks the impact of server configuration changes on metricsanother page on this site

  15. When many app servers are involved, the Diagnostics Add-inanother page on this site to LoadRunner displays
  16. metrics files obtained from the
  17. Diagnostic Server, also called the Commander, which stores data from the
  18. Deep Diagnosticsanother page on this site Collector and
  19. Mediator which filter and aggregate data obtained from probes on app servers.
  20. The Deep Diagnostics Console another page on this site View screenshot serves web pages containing metrics such as method call chains (call trees) that break down transaction data at each resource layer.
  21. The Diagnostic Server (Commander) also feeds "samples" (values at a particular point in time) to the BLE (Business Logic Engine / "Trinity" service) of the
  22. Business Availability Center (BAC) product's processing server, which stores data into the Configuration Management Data Base (CMDB) based on Business Rules and status Objectives.
  23. The BAC can dynamically configure diagnostics servers and use Analytics to display metrics.
  24. The BAC Dashboard presents (over time) raw data organized as Key Process Indicators (KPIs) associated with end-user business processes and other configuration items (CIs).

  25. During production, the Business Process Manager (BPM) issues "synthetic"
  26. Vuser Business Process (BP) playback requests to obtain timings from the client point of view (not to generate load).
  27. Actual user activities are captured by the Real User Monitor (RUM)another page on this site

Set screen HP/Mercury Product Terminology

    The terminology used by programs performing similar tasks are different in different products:

    Task AD LoadRunner Tuning Profiler Deep Diag. AM BAC
    Define options Controller Tuning Console Profiler Diagnostic Console
    Run-time options Scenarios, Schedules Session Step Profiler Diagnostic Console Plans
    Collect raw metrics another page on this site mdrv Agent Agent Probe Probe Agent
    Emulate clients making requests Load Generator Console n/a Commander
    Receive raw metrics another page on this site Controller Mediator Diagnostic Server
    Calculate and graph results statistics Analysis Analysis Probe Diagnostic Console

    Note:The LoadRunner Controller cannot run a load test scenario during a Tuning Console session.


Set screen Installation Paths


Set screen Machines and Ports

Set screen Versions Module Interactions

Set screen Overall Installation Sequence

    The recommended sequence of installation is this order: BAC > LoadRunner Diagnostics Add-in > Diagnostics Server > Probes > JRE Instrumenter > Diagnostics Collector

    • For Business Availability Center 6.2 and Diagnostics to coexist on the same machine, BAC must be installed first.
    • To make the drill-down from Real User Monitor available, first open Diagnostics from the menu, then open BAC.
    • To avoid crashing LoadRunner 9, first make sure the Diagnostics server is running, then connect to it from LoadRunner..

    1. To servers running Norton Anti-Virus, add kill.exe to the exclusion list so that it is not treated as a virus.
    2. Install Adobe PDF viewer clients on the Diag. server so documentation can be read there.
    3. Set all servers and clients to automatically sychronize time from a common time service another page on this site
    4. BAC
    5. Install LoadRunner/Performance Center, if not installed already.
    6. Install LoadRunner Diagnostics Add-in for J2EE & .NET
    7. Install Diagnostics Server
    8. Install Diagnostics Probes on app servers under test
      • Do not run JRE Instrumenter (until later).
      • Do not select "additional products" such as Managed Software Solution.
    9. Install Diagnostics Collector for SAP R/3, Oracle 10g database, IBM MQ Series messaging
    10. In Server/etc/ file:
      • If diag. servers do not have an internet connection, to avoid sync signals being sent, set timemanager.time_source=SERVER or =NTP (the default) with timemanager.ntp.servers= a NTP server.
      • Set registered_hostname to ip address.
    11. To protect changes from being overlaid during re-installation, copy the JavaProbe\etc\auto_detect.points file to a new name specified in the JVM start-up parm
      Do not specify the file path nor ".points" file extension.

    Unlike other HP/Mercury products, with the Diagnostics Server you can install a license .key file from HP during the initial installation from the CD rather than after installation.

    Caution! Before installation, close any LoadRunner programs and the Windows Control Panel Services window.

    Caution! With BAC 6.2, do not specify a password that contains an exclamation point (!). Otherwise, you cannot access functionality related to the Script Repository and transaction monitors in Monitor Administration.

    Verify installation and troubleshoot connectivity issues between Diagnostics components by using the Health-Check Monitor Java Applet.

    With v3.5, setup programs for Diagnostics Commander, Mediator, and Probe are downloaded from the BAC Admin > Diagnostics "Downloads" tab or http://Diag server/opal/stynamic/downloads.html The file downloaded is for the hostname and port number specified for the Diagnostic server.

    Do this! To install a J2EE Diagnostics server on a Windows 2000 Professional machine, to avoid the error "Mercury Diagnostics Server cannot be installed on this platform." (KB 36971):

    1. Copy the contents of the Windows installation CD onto the hard drive.
    2. Go to the J2EE_App folder.
    3. Open the tsinst.ini file in Notepad.
    4. Change RequiredSoftware = IE4, NT to RequiredSoftware = IE4, NTProfessional.
    5. Save the file and run setup.exe to install the software.

    On Solaris, installed are ~/ and ~/vpd.patches, directories: /var/sadm/pkg/IS* and /var/sadm/pkg/MERQ.

Go to Top of this page.
Next topic this page

Set screen Probes

    Information from probes are displayed in:

    • LoadRunner Analysis module transaction breakdown feature ("TxBD" for short). or by the
    • Business Availability Center "Memory Usage" reports
    J2EE (not .NET) Diagnostics also provides a Memory Leak report.

    Set screen Probe Installation

    webpage article Diagnostics for Profiler J2EE Installation and User's Guide (redirected from the URL printed in the manual).

    According to each version's ReadMe.htm file, each Diagnostics Probe takes:

    Version Platform Additional Hard Disk Space on Install Additional RAM to run
    v4 J2EE 200 MB 50 MB
    .NET 200 MB 60 MB

    Probe Installer steps by admin (not as Unix root operator):

    1. Copy the Installation CD to a temporary Installer folder.
    2. Edit the file to change the value of default
        JAVAPROBE_HOME c:\\Program Files\\Mercury Interactive\\common\\JavaProbe
        to where you would like to install the probe, such as:
    3. Invoke the DiagnosticsProbeSetup_....exe. It takes a minute or more before you see the first screen to accept HP's terms.
    4. Leave the Probe Name entry blank so that the same name is not used for all JVM instances on the machine.
    5. Specify a Probe Group Name or "Default" will be assigned by default. On a SAP system, use the 3-letter SAP system name.
    6. The location of the Diagnostics Server in Mediator mode Port is by default 2612 and written to the and files.
    7. Probe type selection Selecting "Profiler Only" mode sets active.products=PRO in and limits monitoring to 5 threads.
    8. When "Finish" is clicked, the installer automatically creates a "etc.old" folder containing files in the etc folder. Manually rename it with a date.
    9. Run the JRE Instrumenter and copy the appropriate JRE Instrumenter parameter to the start-up script or (WebLogic Remote Start value)
    10. Edit the start-up script for each specific server,
    11. Set permissions to make sure that users who start the application server has write permissions to the log directory writeable for all users on the system. On Unix:
        chmod a+w /opt/MercuryDiagnostics/JAVAProbe/log

    Set screen Probe Configuration

    A built-in probe is created for host with a fully qualified domain name created by combining the %COMPUTER_NAME% (machine name) + the DNS Suffix Search List from running the "ipconfig /all".

    (KB 38840) Probes can be configured from the loadrunner \util\bin folder config.cmd (Windows) or (Unix) shell scripts.

    1. Edit JavaProbe\etc\ file to set active.products=Enterprise.
    2. Set registrar.url=http://CommanderHost:CommanderPort/registrar/ in
      etc\ if there is no proxy, or
      etc\ if there is a proxy.
    3. Within the JavaProbe\etc\ file, note the difference in capitalization of "Port" in entries specifying the range of ports:

    If you open Control Panel > Add or Remove Programs, you'll see "InstallMIGenericHook".

    The .NET Probe hooks into applications through two modules (KB 46307):

      The first module hooks into the CLR by having its GUID in an environment variable that the CLR reads to see what modules it should find in the Windows Registry which it should then engage.

      The second module uses the ADSI .NET API to recognize the applications running on the CLR. It looks for virtual directories and files with the ASP.NET extension, then places hooks there.

      In addition to .NET "common" points files (, WebServices.points, lwmd.points), .NET automatically creates a custom points file with the name of the .NET app domain.

Go to Top of this page.
Next topic this page

Set screen JRE Instrumenter

    The JVM Instrumenter needs to be run once for each JVM, from All Programs > HP Diagnostics J2EE Probe > JRE Instrumenter

      This executes jreinstrumenter.cmd.

    Edit the start-up script for the application under test to identify the java path.

    After clicking "Add JVM(s)" but before clicking "Search from here", specify "C:" to search the entire hard drive.

    Click the path associated with the Java.exe used by the application under test.

    Click "Copy Parameter" to put one of these into the Windows clipboard:

    is the profiler's own JVM.

    To use JRockit that comes with BEA:

    Set screen Edit/Past Into Application Start-up Script JVM Options

    Edit the app server's startup script and paste.
      On SAP Portal servers, use

    specify a different probe name:

    -Xbootclasspath must be placed at the beginning of the Java parameters, following any JIT options such as -hotspot or -classic.

    Set screen Instrumentation

    If you click Memory Analysis and get this error message , add to the JVM parameters (on a Windows machine):

      JVMTI (Java Virtual Machine Tool Interface) was introduced with JVM 1.5 to replace the
      JVMPI (Java VM Profiling Interface) and BCEL service in JVM 1.4.
      These perform "automated" instrumentation needed to cause a system to produce output useful for diagnosis, where compiled jar files are searched and the byte codes automatically modified using "interposition" techniques which substitute a system component with a pass-through component that records the calls made to it. to capture calls at various points during program execution.

      In constrast, "manual" instrumentation consists of programmers adding statements to source code (that is often difficult to fully remove),

    Unlike desktop IDEs like JProbe, which weight heavily on a machine, HP Diagnostics is setup by default to track only those points which take the longest time or consume the most resources.


Go to Top of this page.
Next topic this page

Set screen Starting (Launching) Probes

    In a Windows Run command window, navigate to C:\MercuryDiagnostics\JAVAProbe\bin and invoke


      Internally, this invokes java with the -Xbootclasspath parameter that specifies \lib\probe.jar using "com.mercury.opal.capture.JavaProbe true".

    Set screen Disabling Probes

    There's reverse logic here, but if you want to stop profiler, set:
    in etc\ disable.profiler=true

    in etc\ #ac.register.sink=false disables Profiler for method collection.

Set screen Profiler

Set screen Diagnostics Server

    webpage article http://localhost:2006 retrieves the Diagnostics Server management app:

    open diagnostics icon Open Diagnostics webpage article http://localhost:2006/topaz/...
    This is the same Java Applet GUI as the LoadRunner Controller "Diagnostics for J2EE/.NET" tab.
    configure diagnostics icon Configure Diagnostics

    (System Health Graph is within Registrar)
    webpage article http://localhost:2006/maintenance

    Component (Maintenance) advanced (v7)
    configuration advanced (v7)
    manage diagnostics icon Manage Authorization and Authentication webpage article http://localhost:2006/security

    webpage article http://localhost:2612 should not be accessed. It retrieves the Mediator app:

    Set screen Diagnostics Server Installation

    Since v4, the Mediator and Commander are installed using the same "Diagnostic Server" software.
    Previously, in v3, mediator and commander were installed separately.

    On UNIX:

    • the default folder is /opt/MercuryDiagnostics. But some organizations do not provide enough room or not allow installations into opt, so you may have to use another folder under /usr or some other folder.
    • Rather than running the installer as root, after su -someuser
    • Unix also needs "chmod a+w /opt/MercuryDiagnostics/JAVAProbe/log".
    • With IBM AIX, "set IBM_JAVA_OPTIONS" to clear them out before running the installer.

    Both the Controller and Mediator sends (HTTP protocol) de/activation instructions to the Probe through port 2005. The Mediator also sends synchronization instructions to the Probe.

    Because of their tight hand shaking, both the Mediator and the probe need to have the same logical LAN ID (e.g. LAN1, LAN2, or "DefaultLAN" if there is only one LAN).

    When a Diagnostics Server is deployed in Commander mode, it has both commanding and mediating responsbilities. This separation is needed to accomodate large system landscapes with many servers, The Mediator pre-processes, filters, and aggregates real-time data before sending it to the Diagnostics server/LR Agent every 30 seconds (by default).

      Real-user transaction data the Mediator sends offline every ten minutes.

    LoadRunner's Agent process connects with the Mediator. The LoadRunner UpdateService is installed on a Windows 200X machine (separate from the Controller) with "J2EE Mediator Installation" in Start > All Programs.

    Probes continuously send raw performance data (using a proprietary protocol) to the Mediator listening port.

    If several application servers (and their JVMs) are monitored by a single Mediator, that Mediator should reside on a separate mediator host machine. Each Diagnostics Mediator can process a limited number of probes. The readme file has these recommendations:

      Item Platform Total Number of Probes
      up to 10 up to 20 up to 30
      Machines Windows 1 x 1 GHz 1 x 2 GHz 2 x 2.4 GHz
      Solaris 1 x Ultrasparc 2 2 x Ultrasparc 2 2 x Ultrasparc 3
      Memory Windows 768 MB 1 GB 2 GB
      Solaris 1 GB 1.5 GB 2 GB
      Java Heap 350 MB 700 MB 1400 MB

    Reminder The Java heapanother page on this site -xmx setting should be less than 75% of physical memory (e.g., -xmx768m for a machine with 1GB of RAM installed).

    The installer sets up the HP Diagnostics Server service to start automatically screen captured upon start-up from C:\MercuryDiagnostcs\Server\nanny\windows\bin\magentservice.exe. Alternately, this can be manually started or stopped from

      Start > Programs > Diagnostics Server or from
      Start > Settings > Control Panel > Administrative Services > Services

      This service needs to be stopped before you can run the installer again, or you'll get this error message screen captured

    The Diagnostics Server uses rhttp -- revershe HTTP -- to query metrics another page on this site from J2EE probes.

Go to Top of this page.
Next topic this page

Set screen HP Diagnostic Collector

    HP Diagnostic Collector Installation The Diagnostic Collector collects metrics from SAP R/3, Oracle, and MQ series servers.

    Create SAP JCo files within the folder before starting the installer.

    Do not copy the librfc32.dll to windows/system32 unless it's a later version (6405.5.165.5297 vs. 7100.1.7.5371 from a more recent SAPGUI install).

    Run localhost:2006/security/EncryptPassword.jsp to encrypt the password, which is pasted within file
    tag <encrypted-password>
    Then replace "ChangeMe" with information for the target system under test.

    Within file \etc\, set xml.filename=sapr3.metrics.xml

    From Start > HP Diagnostic Collector, Stop and start it again.

Go to Top of this page.
Next topic this page

Set screen Diagnostics System Health Monitor

    The "System Health Monitor" is a web page (graphic Java Applet) at webpage article http://localhost:2016/registrar/health (and providing default username and password "admin").

    health check sample screen

    An application must be running before that node turns from Inactive gray to active green.

    Activity level can be configured to be illustrated two ways after Right-clicking on the graph: "Show Load Using Circle" size or the default "Show Load Using Scale" length of the black line under the component.

Go to Top of this page.
Next topic this page

Set screen Diag. Server Application Groups and Applications

    The Entire Enterprise application predefined by default includes the entire managed environment context (all Diagnostics views and components).

    Define application groups to organize groups of related composite applications and other application groups.

    Define composite applications to logically group Diagnostics entities (such as probes, hosts, server requests, etc.).

    Examples are Division A vs. B, HR vs. Finance.

    Applications are not shown within LoadRunner or Load graphs.

Go to Top of this page.
Next topic this page

Set screen Configure/Enable Reporting

    To keep overhead low, by default several views are not shown or does not show all information. So they must be manually enabled.

    CPU Time Java is enabled in Transaction, Probes, and Call Profile views if:

      probe\etc\ file entry use.cpu.timestamps=true and
      probe\etc\ file cpu.timestamp.collection.method=
        1 only for server requests (the default), 2 for all transactions/methods, 0 for no timestamping

    Profiler Light Weight Memory Diagnostics (LWMD) is used to identify the largest collections and the fastest growing collections within Java applications. Page 343 of the Diag. 7 User Guide says:

      probe\etc\ file entry
        lwm.diagnostics.capture=true and (to only show the top 10 collections by size or growth)
        lwmd.resource.replace (pattern replacement of code resources, as in uri.pattern.replace)
      probe\etc\auto_detect.points file where key = lwmd set to active=true

    Profiler Memory Analysis with Heap Walker (for JVM 1.5+) or Heap Breakdown (JVM 1.4) are used to identify objects that may be leaking. This doesn't work at all on IBM JVM (due to a defect on the IBM VM, per KB 40630).

      Edit the app server's startup script to add (after -Xbootclasspath) -Xrunheapdump to enable Heap Dumps.

      probe\etc\ file entry use.cpu.timestamps=true and
      probe\etc\ file cpu.timestamp.collection.method=

        1 only for server requests (the default), 2 for all transactions/methods, 0 for no timestamping

    Thread Waited and Blocked Time columns can be included in the Probe Threads Table by setting

      threads.contention.monitoring.enabled property=true

    To graph latency for a specific method (point), specify in the Probe\etc\auto_detect.points file layertype = trended_method for each point

Go to Top of this page.
Next topic this page

Set screen Memory Configuration

    From Diagnostic Server Configuration > Memory Diagnostics > Worst Collection Limit (The number of "worst" collections that should be tracked) defaults to file parameter bucket.lwmd.topN = 10

Go to Top of this page.
Next topic this page

Set screen Points for Exploration

    Classes and methods displayed on views are specified in the file by specifications in the points file.

    Define separate points files for Dev, QA, and Prod.
    In QA, keep defaults (Synchronization inactive but core SAP points activated).
    In Dev., activate Synchronization and inactivate core SAP points.
    During pseudo-production measurement runs, do not use no Synchronization and inactivate core SAP points.

    Save a single points file and copy it to each it to each of several identical clustered machines using a custom batch file.

    Profiler7 Configuration Instead of directly editing the points file, click Profiler > Configuration for the screen on the right.

    Check Capture Class Map.

    In Change Probe Instrumentation Plan > Shared Instrumentation, click "Edit..." button for this GUI:
    Profiler Configuration Instrumentation Points GUI

    The red icon indicates that "active=false" is set in the Source for that point.

    Pressing the Source tab shows the points file at the point selected (appearing between [square brackets] at the bottom of the Source screen).

    "layer" properties for each point is the points file is the basis for the Layers shown in Load views.

    The points file is saved to the file after both OK and "Accept" are pressed.

    CrossVM > RMI correlates the call tree data from both calling and called virtual machines with this point specification. Notice that the keyword is displayed below the layer.

      keyword = rmi
      layer = CrossVM
      active = true

    Instrumentation for OVTA (HP OpenView Transaction Analyzer) Java Servlet Filters and EJB local home methods are invoked by these points:

      Business Tier
      Web Tier

    [Synchronization], when activated, identifies a common source of performance problems.

    Avoid creating a separate custom layer. Also, custom packages should ignore utility methods (get, set, is):

      class  = !com.MyCompany\...
      method = !.*
      ignore_method = !(set.*)|(get.*)|(is.*)
      signature = !.*
      layer = MyCompany/LOB/MyClasses

    The classes discovered during probe installation are specified in file lwmd.discovered.points within the probe\etc folder.

      ## In the absense of a ClassMap, this file contains all the classes which 
      ## will be instrumented for light-weight memory detection, or were detected 
      ## at runtime to be of interested to light-weight memory detection and will 
      ## be instrumented next time the application is restarted
      ## Each class which should be tracked for light-weight memory diagnostics
      ## should be listed here, each on its own line.
      ## If you remove a collection to prevent it from being instrumented on a
      ## subsequent run, you need to add it to the lwm.diagnostics.filter.exclude
      ## property inside in order to prevent the collection from
      ## being discovered & added back into this list when the probe next restarts.

    Other points file parameters:

      class = <class/package name/s to capture>
      ignore_cl = <classes to exclude>
      method = <method name/s to capture>
      ignore_method = <method prototypes to exclude>
      signature = <signature/s of method/s in javap symbolic encoding>
      layer = <layer name>
      active = <true, false> #Enabled: checked in GUI
      layerType = <layer type>
      deep_mode = <soft or hard mode>
      scope = <comma separated list of methods>
      ignoreScope = <comma separated list of methods to exclude>
      ignore_tree = args,caller,return <Exclude subclasses>
      detail = <args,caller,return>
      code-key = <key that you generate to secure the code snippet>
      tierDefinition = <hex_identifier>
      merge_classes = <class names>
      merge_url_to = <comma separated list of methods>

    [Leak] should be specified for only a single class. Do not simply activate it.

Go to Top of this page.
Next topic this page

Set screen Layers

    The layers in HP Diagnostics Points files

    Layer Sub Layer(s) Protocol Description
    1. Client HTTP/ HTTPS requests to/from presentation layer web servers.
    Portal SAP - SAP Portal Core Componentsanother page on this site
    Web Tier Session HTML javax.servlet.http.HttpServletRequest
    Servlet - javax.servlet.http.HttpServlet javax.servlet.Filter -
    JSP - javax.servlet.jsp.HttpJspPage
    SAP - SAP Portal Web Service Componentsanother page on this site
    Spring - -
    Struts - -
    Application Spring - keyword = soapfault
    XML SAX - org.xml.sax.XMLFilter
    DOM - (expensive!)
    Web Services Generic SOAP/ REST requests handled by web servers. keyword = soapfault
    SAP SOAP ...
    SAP-RFC RFC keyword = saprfc
    Directory Service LDAP JNDI (method: lookup) javax.naming.Context
    JSP (Java Server Pages) markup -
    Messaging JMS - keyword = jms-client javax.jms.Connection, MessageConsumer, MessageListener, QueueSender
    (Java Mail Services) requests sent to email servers. keyword = jms-server javax.jms.TopicPublisher
    JTA (Java Transaction API) - -
    Database JDBC SQL java.sql.Connection & Execute via Oracle Netanother page on this site protocol
    JDBC-SQL SQL keyword = jdbcsql
    RMI (Remote Method Invocation) RMI -
    RPC (Remote Procedure Call) RPC -
    Logging Log4j - javax.resource.cci.Connection
    Logging Log4j - org.apache.log4j.Logger Method: trace
    JMXanother page on this site JMX

Go to Top of this page.
Next topic this page

Set screen System Tiers

    System Tiers The flow of data Layers are illustrated here.

    Diagnostics group performance metrics by layers and sub-layers based upon the resources that the application invokes to perform the processing. These layers make it easier for you to isolate and identify the areas of the system that may be contributing to performance issues.

Go to Top of this page.
Next topic this page

Set screen Allocations


    [Leak] point

Go to Top of this page.
Next topic this page

Set screen Probe Profiler Tabs

    Profiler supports IE only.


    • Memory
    • Load
    • Shortest Requests


    • Slowest Methods
    • CPU Hotspots (Methods)
    • Slowest SQL


    • System (Host) CPU, Memory Usage, PageInsPerSec, PageOutsPerSec, PageCutsPerSec, Disk, Network
    • JVM: Probe: HeapFree, HeapTotal, HeapUsed
    • Java Platform: Classes, GC, Threads
    • Mercury System
    • Weblogic: EJB, Execute Queues, JDBC, etc.

    Threads (JVM 1.5)

    All Methods

    All SQL

    Collections (using LWMD)


    Server Requests

    Web Services

    Allocation Analysis

    Memory Analysis (Heap Breakdown)

    Configuration (without restarting app)

Go to Top of this page.
Next topic this page

Set screen Diagnostic Report Views

    It takes several minutes for diagnostics to appear.

    The default left pane has:
    View Entity Metric
    Standard Views Server Summary - Status, Aggregate Request
    Aggregate Requests - Latency
    Application Metrics - Latency
    Dependent Service - Latency
    Hosts - CPU
    Outbound calls (none)
    Probes - VM Heap
    SQL Statements - Latency
    Server Requests - Latency
    Transactions - Latency
    Trended Methods - Latency
    Portals Server Summary - Portal Requests & Portal Components
    Portal Components - Latency
    "Web services" (HTTP/SOAP or HTTPS/SOAP-RPC calls to/from remote components via internet) Outboud / Consumer -
    Inbound / Provider (aggregating multiple consumer IPs) -
    Inbound / Provider by Consumer IPs of origination listed in JavaProbe/log/ file (up to max.tracked.addresses.per.probe in JavaProbe\etc\ -
    SAP NetWeaver Summary (Requests & Threads)
    R/3 Summary -
    NetWeaver Requests - HTTP Requests/Sec
    NetWeaver Threads - Application Threads/Active Threads
    R/3 Probes - Latency
    R/3 Server Requests - Latency
    Oracle Summary -
    Wait Timeanother page on this site
    BEA Weblogic Summary
    EJB Pooled Resource Connection
    JDBC Connection Status
    JDBC Resource Contention
    Server Threads
    IBM WebSphere - -
    CICS - -

    My Views

    The Probe captures the call stack for the 3 slowest instances of each server request. It also captures a call stack representing all call instances for a type of service request and calculates the aggregated latency.

    The default BEA Weblogic contains:

Go to Top of this page.
Next topic this page

Set screen Diagnostics GUI Controls

    My preference is to stay in My (customized) Views. We are pretty much forced to because changes to column sequence or which columns are visible or hidden (by clicking on the grey Header) are only saved within custom views. This dialog also lets you arrange things.

    Default views are not customizable. Hide (Close) Views from the View Bar by clicking view bar open space, not an icon. Create View Groups by right-clicking on the white area within My Views.

      Save Server Summary to My View after adding the Current Logons Count column.
      Save Server Requests to My View after adding an Exceptions column.

    Only 100 graph entities (probes, hosts, server-requests, etc.) are shown in each view's entity tables. Search for entity text by pressing Ctrl+F or use a different sort and hope the one you need shows up.

    If you define a sort sequence with more than one column, save a screen shot of it because clicking the grey heading unselects the sort order. Clicking on the header again switches ascending to descending. Clicking a third time removes the column from sorting. To do all this with subsidiary sort sequences, hold down Ctrl while clicing the subsidiary header.

    To close the Diag. View and go back to the Application screen, press Ctrl+F4.

    To view (point-in-time) Analyze Snapshots, select Snapshot mode in the View Context Drop-Down.
    For instructions on creating snapshots, select (Real Time) Monitoring mode.
    You can't use Shift/Ctrl as in Windows Explorer, but right click each individual line to add to an active snapshot.

    To ensure that metrics collected represent processing when applications are running in a steady state:

    • warm up the system with a range of different transactions (to realistically populate the cache), then
    • Press the J2EE Diagnostics Profiler "Reset" so it force-drops all call stacks captured and zeroes out baselines used to calculate instance counts, average latency, slowest latency, etc.
    • Press the J2EE Diagnostics Profiler "Garbage Collection" trash to request a full garbage collection.

    To go back one level in the breadcrumb trail, press Backspace or Alt+LeftArrow.

    Export to HTML Export to HTML is designed well - file names contain details, including a date/time. This enables you to quickly save several pages.

Go to Top of this page.

Go to Top of this page.
Next topic this page

    Set screen Filter Time and Disk Space Usage

    Time Range
    5 minutes Hours\5m 5 seconds
    20 minutes Hours\20m 3 minutes
    hour Hours\1h 4 minutes
    6 hours Days\6h 8 minutes
    day Days\1d 11 minutes
    week Weeks\7d 29 minutes
    month Months\1M
    3 months Years\3M
    year Years\1Y

    Diagnostics conserves disk space by overwriting the same "buckets" reporting values in a fixed number of time resolution zoom periods (from low to high resolution).

    Caution! The "Sub folder" in the table is within the directory where, over time, *, *, and * files pile up (persist):
    C:\MercuryDiagnostics\Server\archive \mediator-%COMPUTERNAME%\persistence \Mercury System_\...

    This is also the schedule of how LoadRunner extends update frequency automatically during a long run. Update frequency is every 5 seconds at the beginning of a Whole Scenario, but extends to 4 minutes an hour into the scenario.

    For Less frequent point duration (granularity of aggregation):

      probe.trends.pull.interval (from every 5s to 30s)
      probe.trees.pull.interval (from every 45s to 120s)

    Change Diagnostics Server Mediator mode offline aggregation period in file Server/etc/ and

    The Mediator stores logs in the log folder where it's installed (by default, in C:\MercuryDiagnostic\Server\log).

    # File Name Description Space Note
    1 server.log - This fills up fast
    2 server-full.log - Usually 0 unless there's a problem.
    3 server-metrics.log - Usually 0 unless there's a problem.
    4 server-jetty.log Event log entries suchs as:
    2006-06-29 11:14:12,524: INFO jetty JettyStreamLogSink.log[177] : EVENT [Thread-8]org.mortbay.http.HttpServer.setStatsOn(> Statistics on = false for org.mortbay.jetty.Server@1b04ff7
    - This fills up fast
    5 server-alerting.log - Usually 0 unless there's a problem.
    6 server-useradmin.log - Usually 0 unless there's a problem.
    7 server-system-properties.log Lists the values from properties files configuring the Diagnostics Server. - Usually 3 KB.
    8 server.out Diagnostics webserver listening on - Usually 3 KB.
    9 server.err - Usually 0 unless there's a problem.

Go to Top of this page.
Next topic this page

Set screen Call Profile Graph from Instance Trees

    This Call Profile Graph displays instances trees collected during the last 5 minutes. They different types are:

    • the slowest (maximum) and second slowest invocation
    • the fastest (minimum) invocation
    • the median (Average) invocation time
    • a randomly selected end-to-end trace of the entire cross-VM tree which includes RMI and other outbound calls to an external VM. If the remote system is also instrumented, it includes the trees of remote systems.

    To identify what methods take longer under load, compare graphs for Average versus Slowest method times

    Aggregate Trees keep all samples collected and combines multiple possible execution paths (instance trees) into a single tree, so it's more difficult to understand what the application is doing.

Go to Top of this page.
Next topic this page

Set screen Tuning Console for LoadRunner

    I am told that this product has been deprecated by HP.

    The ability to automatically invoke several runs of LoadRunner (with different configurations) is really helpful.

    Even more importantly, comparison of results across several runs is great to have.

    I've developed an internal tool to do this.

Go to Top of this page.
Next topic this page

Set screen Diagnostics LoadRunner Add-in for J2EE & .NET

    webpage article Diagnostics for J2EE & .NET for LoadRunner Read Me

    Diagnostics Distribution As the LoadRunner Diagnostics tab initial screen screen captured instructs, Probes need to be enabled within LoadRunner's Diagnostics menu:
    LR8 menu
    This brings up the dialog shown at right.

    Check "Enable the following diagnostics" and adjust the "% of all the relevant Vusers in the current scenario" to 100 when you are exploring SAP.

    Within the "Offline Diagnostics" area, clicking the "Configure" button for "Web Page Diagnostics" retrieves this dialog screen captured.

    Clicking "Troubleshoot Diagnostics for J2EE/.NET connectivity" invokes a new internet browser window containing the System Health Monitor.

    In the LoadRunner Launcher, when you select CONFIGURATION Diagnostics for J2EE/.NET Setup
    expect to see a dialog such as this:
    commander config settings dialog screen

    To create offline AM/Standalone datasets for LoadRunner Analysis to read, set Diagnostic Server > Configure Diagnostics > Configuration > Show Advanced Options > Online Cache to true (which automatically sets Server\etc\ file parameter = true).

    For a list of files and runs that are open and still collecting data, go to Diagnostic Server > Configure Diagnostics > Configuration > query.

    Within Diagnostic Server > Configuration > Aggregation modify LoadRunner Offline/Summary VU
    Vu Aggregation Period = 5s
    server request aggregation period
    Business Availability Center (Topaz) BPM Aggregation Period (must be multiple of Summary Trend duration)

Go to Top of this page.
Previous topic this page
Next topic this page

    Set screen LoadRunner Transaction Breakdown Diagnostics Module

    The J2EE Transaction Breakdown Diagnostics Module includes the following components:

    • Console: Displays the transaction data.
    • Mediator: Processes the transaction data, and then passes it to the Console.
    • Probe: Resides on the application server. Gathers information about the monitored transactions and passes it to the mediator.

    The J2EE Transaction Breakdown Diagnostics Module requires MSXML 3.0 or later (included in Internet Explorer 6.0) installed from this Microsoft MSDN Web page.

    Probes collect time spent only within J2EE/.NET application servers, so their timings do not include user think time, time spent traveling the network, nor time in the web server. For this reason:

    "J2EE/.NET - Transaction Response Time Server-Side Graph".

Go to Top of this page.
Next topic this page

    Set screen Diagnostics Usage Summary Report

    The Summary Report is titled "Diagnostics Usage"

      Summary Report

    There are two groups of J2EE & .NET Diagnostics graphs:

    • J2EE & .NET Server Diagnostics Graphs show the performance of requests and methods
      generated by real and virtual users without connection to any transaction:

      • J2EE/.NET - Server Request Response Time (a line for each request)
      • J2EE/.NET - Average Server Method Response Time (a line for each request)
      • J2EE/.NET - Server Requests per Second (a line for each request)
      • J2EE/.NET - Server Methods Calls per Second (a line for each request)
      • J2EE/.NET - Average Number of Exceptions on Server
      • J2EE/.NET - Average Number of Timeouts on Server
    • J2EE & .NET Diagnostics Graphs show the performance of requests and methods
      generated by vuser transactions (but not by real users):

      • Average Method Response Time in Transactions
      • Transaction Time Spent in Element (per method/query)
      • Transactions Per Second (for all transactions)
      • Method calls per second in Transactions (a line for each Transaction)
      • Breakdown information by layer, class, and method
      • Chain of calls and call stack statistics
      • Average Number of Exceptions in Transactions
      • Average Number of Timeouts in Transactions
    • Rather than using automatic transactions specified in Runtime Settings, lr_start_transaction and lr_end_transaction statements must be inserted manually around each transaction in the Vuser script.

    "Elements" within each transaction and request are :

    • Layer (System Tier)
      • Class (associated with an object/bean)
        • Method (within each Class)

    Filter values can be defined for these elements plus:

    • Scenario Elapsed Time
    • Transaction Name
    • SQL Logical Name (shown in Measurement Description)

Go to Top of this page.
Next topic this page

    Set screen Transaction Chain of Calls and Call Stack

    chain icon
    chain of calls screen

    The most time consuming chain is shown in red.

    Click on a method and click the icon for the reverse bottom-up view of the Method Call Stack:

    method call icon
    Method call stack statistics screen

    A view group is pre-defined for each type of server (WebLogic, WebSphere, etc.).

    Views can be by Load, Probe, and Server Requests.

    Drill down from a view group to detail views for each probe or host.

    The Metrics Inspector is like a properties GUI control used to specify the threshold values controlling whether each metric displays on the graph and detail table.


Go to Top of this page.
Previous topic this page
Next topic this page

Set screen Business Availability Center (BAC) x.x Diagnostics for J2EE & .NET

    Business Availability Center comes in several CD's:

    • Diagnostics for J2EE & .NET v3.5 - Supporting Business Availability Center 5.x
      1. Unix Installation
      2. Windows Installation
      3. Patches
      4. Probes

    BAC Performance (Dashboard) Customizing the KPI, Rules, parameters

    Set screen BAC Versions

    BAC Version Diag. Version Date OS Support App Server Database Web Server
    6.3 (VuGen 8.1) 4.1 8/06 Windows 2000 Server/Advanced Server (SP4), Windows 2003 Standard/Enterprise Server (SP1) Solaris 8, 9, 10 WebLogic 8.0 Oracle, SQL Server 2000 Enterprise SP4 IIS 5,6; Apache 2.0.49; Sun Java System Web Server 6 SP 7
    6.2 (VuGen 8.1) 4.1 6/06 Windows 2000 Server/Advanced Server (SP4), Windows 2003 Standard/Enterprise Server (SP1) Solaris 8, 9, 10 WebLogic 8.0 Oracle, SQL Server 2000 Enterprise SP4 IIS 5,6; Apache 2.0.49; Sun Java System Web Server 6 SP 7
    6.1 (VuGen 8.1) 4.0 SP1 - - WebLogic 8.0 Oracle Sun iPlanet
    6.0 (VuGen 8.1) 4.0 SP1 - - WebLogic 8.0 Oracle Sun iPlanet
    5.1 (VuGen 8.0) 3.5 6/14/05 Solaris 10, Windows 2000 Server, Windows 2003 Server (all editions) WebLogic 7.0 SP6
    5.0 3.3 11/12/04 Solaris 9, Windows 2000 Server SP4, Windows 2003 Standard Edition -

    Set screen Types of Monitors

    The types of monitors managed by BAC are classified by this hierarchy:

      EUM - End-User Management monitors:

      • BPM - Business Process Monitor
      • RUM - Real User Monitor
      • CM - Client Monitor

      System monitors:

      • SiteScope (SIS)
      • EMS - Enterprise Management Systems such as SNMP, Peregrine, Remedy, Tivoli, BMC Patrol, HP OpenView)

    The "IT Universe" within BAC consists of the Business, Monitor, and System.

    Business items include:

    • Business Processes
    • Business Process Steps
    • Group
    • Line of Business

    Set screen BAC Workflow

    The BAC dashboard is a portal that presents HTML aggregated after consuming one or more portlets on WSRP (Web Services for Remote Portlets) service producers which publish WSDL to a WSRP Registry using UDDI and invoked using SOA, like other XML services. WSRP was first approved at OASIS standard 1.0 on September, 2003. v1.1 was approved May, 2004.

    Monitors generate "samples" of activity data carried over a "bus".
    Business Logic Engine (BLE) matches (filters) samples against the model of CIs (Configuration Items) stored in the CMDB (Configuration Management DataBase) central repository.

    Matched samples that fit the model light up the "dashboard" of real-time calcuations and status.

    The Alert Engine handles alerts associated with KPIs (Key Processor Indicators)

    Set screen Diagnostics Management Database

    Deep Diagnostics setup (file DeepDiag3.3.11.2VM.exe dated 11/24/2004) is a Java SWT app. It installs by default to folder "C:\MercuryDiagnostics". By default it installs MSDE 2000 and MDAC 2.7.

    The Diagnostics Management database is not needed since v3.5.
    Caution! With v3.3, if the Diagnostics management database is on a different machine than the machine hosting Diagnostics server, they both need to be set to the same time zone. The database is configured in Start > Programs > Mercury Diagnostics Server > Administration > Connect to Database

    With v3.5, diagnostics components "can" store data from over 72 hours of runs with 20 SUT's running load tests for both .NET and J2EE probes. However, The System Health Monitor will not report an error condition on the graph when the Diagnostics components are out of disk space.

    Set screen Web server

    An open-source web server is installed using file jakarta-tomcat-3.2.3.tar.gz in the shared\third-party folder of installation CDs.

    "Select the Web Server type you want Application Management to work with."

    • Apache HTTP Server

    The Diagnostics application only works with IE6 browsers, with privacy settings set to "Always accept cookies from this site."

    The SMTP server name needs to be specified during installation. Its Sender Name is by default MercuryAM_Alert_Manager.

    Set screen Configuration

    Run JRE Instrumenter

    BAC screen shot BAC Diagnostic breakdowns need to be enabled for each BAC Business Process Transaction Monitor.

    In each server's startup script, add (after the JIT option) a -Xbootclasspath Java command line parameter reference to the <probe_install_dir>\classes/boot

    Also add to the CLASSPATH=%CLASSPATH%;%J2EE probe installation directory%\dat

    With LR7.8SP1, from the J2EE Transaction Breakdown Home Directory\lib directory, run the Initializer

      java -jar sipatool.jar

      This puts ClassLoader.class under directory Probe Home Directory/classes/boot/java/lang.

    Each probe writes to a <probe_id>.log file in the <probe_install_dir>/log folder.

    The probe of each server must be enabled before it can monitor events.

    Lastly, run the System Health Check to view the component's configuration information and performance metrics.

    The Profile database stores transaction breakdown summary information in its event_meter table, em_download_data_size, em_s_network_fb_time, and em_s_connection_time columns for each transaction.

Go to Top of this page.
Previous topic this page
Next topic this page

Set screen RUM (Real User Monitor)

    Scripts for the Real User Monitor are developed by the VuGen software borrowed from the LoadRunner product.

    The installer creates file %WINDDIR%\ which should be deleted before the product is re-installed (after a mandatory uninstall).

Go to Top of this page.
Next topic this page

Set this at top of window. Resources

Go to Top of this page.
Next topic this page
Set screen

Go to Top of this page.
Previous topic this page
Link to Performance Engineer RSS 2.0 XML feed Atom 1.0 XML feed feeds
for Performance and Capacity Engineers...

How I may help

Send a message with your email client program

Your rating of this page:
Low High

Your first name:

Your family name:

Your location (city, country):

Your Email address: 

  Top of Page Go to top of page

Thank you!

Human verify:
Please retype:

Visitor comments are owned by the poster.
All trademarks and copyrights on this page are owned by their respective owners.
The rest ©Copyright 1996-2011 Wilson Mar. All rights reserved.
| | | |