How I may help
LinkedIn Profile Email me!
Call me using Skype client on your machine

Reload this page LoadTestTools.Info

This page analyzes the various ways to generate load for performance measurement. This is a companion to my other pages on Load Monitoring, Testing and Reportinganother page on this site of capacity management of IT Service Management (ITSM).


Topics this page:

  • Product Classifications
  • Load Gen Requirements
  • Open Source Tools
  • Commercial Tools
  • Virtualization
  • Labs
  • Your comments???

    RSS XML feed for load testers RSS preview for load testers Site Map List all pages on this site 
    About this site About this site 
    Go to first topic Go to Bottom of this page

    Set screen Load Testing Product Classifications

      I assess load-testing tools several ways:

    1. By functionality needed in load-testing tools

    2. By the technology being measured and their communication protocols:
      Internet Browsers' HTTP, Java Applets, SOA Web Services driven by Java J2EE or .NET , PHP, Python, Ruby, etc.

      A sub-classification is whether tools operating externally from the server being tested vs. internally in the server being tested.

    3. By vendor pricing strategy:


      webpage article Comparsion of load testing products

      Ancillary tools I've found useful:

      tool $29 Bare Tail displays text logs by automatically scrolling to the tail (bottom) of the file as they are updated by a running program. This utility also highlights lines when it finds "Failure", "Full GC", or whatever text values specified by the user.

      Go to Top of this page.
    Previous topic this page
    Next topic this page

    Set screen Load Generation Product Requirements:

      Test Preparations Test Drivers Test Results Test Follow-up Test Environment Management Test Team Collaboration This diagram (adapted from the work of Ross Collard), presents a framework of load-testing tools:

      1. "Front-end" tools to prepare the data, scripts, and other test assets needed to drive test actions are currently the most commonly sold category of product.
      2. "Back-end" Tools to collect and follow-up on the results of test runs is where integrated commercial package LoadRunner shines.

      4. Tools to manage the test environment are necessary for load testing efficiency and repeatability.

      6. Tools to enable collaboration (share assets and knowledge) within and outside the team is a category that is usually neglected at the price of the organization's efficiency at responding to the complexities of today's advanced technologies for high-security and high-availability under heavy demand.

      Test Preparation Tools


      Test Driver


      Test Results Manager


      Test Follow-up

      A load testing program needs these features automated in order to emulate or control many instances of a client on many machines:

      1. A way to distribute from a central location scripts and files to the many injector (load generator) machines.

      2. A way to invoke the script at specific times (all together). This is why an automated approach is needed.

      3. A way to define logic such as capturing sessionids to maintain state.

      4. A way to capture the response time for each transaction on each instance of each machine.

      5. A way to collect, to a central location, response time and other data from each instance of each machine.

      6. A way to specify the behavior of each script, such running a particular action a 30% percentage of time.

      7. A way to specify the verbosity (level of detail) of log entries.

      8. A way to stop, from a central location, scripts running on specific machines. This is the weakness of using simple batch command files.


      Go to Top of this page.
    Previous topic this page
    Next topic this page

    Set screen Open source load generation products and services:


    • TPTP — the TPTP (Test and Performance Tools Platform) IBM Eclipse project (formed in August 2004 by promoting and expanding the former Hyades project formed December 2002) uses JVMPI (Java Virtual Machine Profiler Interface)/Java 5 JVMTI Trace/Profiling and Common Base Event (CBE) Monitoring, and JUnit & Assisted Manual test tools.

    • Redstone


    Go to Top of this page.
    Previous topic this page
    Next topic this page

    Go to Top of this page.
    Previous topic this page
    Next topic this page

    Set screen tool Commercial stress testing products


    Go to Top of this page.
    Previous topic this page
    Next topic this page

    Set screen Performance Stats From Systems Management Packages

      Ideally, measuring performance is an on-going effort, from unit-testing through production. And performance metrics can be tracked and presented with continuity.

      Ideally, programmers, DBAs, and operations people would use a common toolset that serves the needs of each and provide the "glue" for easy communication among the various groups that have traditionally maintained its own "silos" of data and toolsets.

      HP's acquisition of Mercury presents some hope for unification under its OpenView. But Mercury tools are as well known as IBM Eclipse for developers and Quest for DBAs.

      So while IBM integrates Websphere into Rational and Tivoli, Quest software offers its own stack, Symantec i3 application performance management (APM) software is part of Symantec Data Center Foundation line.

      It is ironic that the applications that purports to monitor an "open" framework are themselves "closed" in many respects. The dashboards from each vendor do not always "play nice" with information from other vendors.

      So each user site must undertake an elaborate company-wide effort to integrate data and dashboards to overcome the mistakes and delays of everyone pasting data into Excel to integrate data.

      Symantec i3 application performance management (APM) software, formerly known as VERITAS i3 focuses on J2EE platforms such as BEA WebLogic Portal Server showing portlet performance.


    Go to Top of this page.
    Previous topic this page
    Next topic this page

    Set screen Virtualization & Products that come with the system under test

      A new approach to performance testing and capacity management is needed in a data center which has been "Virtualized" by software such as

      • EMC's VMWare ESXServer suite,
      • Microsoft's Virtual Server 2005,
      • open source projects QEMU, and
      • Xen, started at the University of Cambridge and adopted by Linux distributions, including Novell's SuSE Linux Enterprise Server.

      The Virtual Desktop Infrastructure Alliance

      Virtualized machines run as a guest OS operating on top of the host OS.

      In a virtualized data center, nothing is "static". In one given moment, an application can be running on dozens or hundreds of machines, with data on several other machines (typically over a SAN). In another moment, they vanish.

      EMC's VMotion™ technology lets virtual machines -- while it's running -- be moved "dynamically" from one host to another while maintaining continuous service availability.

      That's the whole point of virtualization -- to make computing power available as needed, then reclaim that power for others when it's no longer used.

      Rather than having lots of machines with a different set of reserve capacity sitting idle, computing power and storage is pooled for sharing.

      This is good news for functional testers becuase ESX Server software now requires a moveable working environment for development to occur, testers no longer have to beg for test machines and test time, since application instances can be easily created.

      Now that with virtualization managers don't have to make sure that an app is running on the right size machines, does the need for load testing go away?

      Maybe not. Since there is always a point where demand can exceed capacity, the job of measuring capacity and performance shifts to a central console, just as hundreds of servers can be managed from a single EMC VirtualCenter console.

      Capacity management includes a dynamic moment-by-moment activity rather than being just a long project with phases.

      To stay ahead of the game, analysis of performance need to be modeled rather than measured using a purely empirical approach running apps on a static set of servers.

      But before that, there is a need to validate the claims of vendors. New AMD and Intel CPU architectures have hardware virtualization capabilities called "para-virtualization" which modify the host operating system to support low-level calls needed by the guest OSes. Xen and User-mode Linux (UML) use this approach.

      Currently the VMware Virtual Center server provides a web page to expose status at any given point in time, but not over time.

      So people have to do things like write a MIB to query and send out SMNP notifications for a collector.

      Or buy nWorks SPI for VMWare to publish XML that is consumed by an nWorks Windows server such as MOM.

      Microsoft's Virtual Server 2005 R2 supports new Intel and AMD virtualization extensions and for Windows' Volume Shadow Copy Service.

      Open Nebula's front-end on UNIX uses XML-RPC API to remotely accesses machines managed by a Hypervisor running Xen, KVM and on-demand access to Amazon EC2.

    Go to Top of this page.
    Previous topic this page
    Next topic this page

    Set screen Labs

    Go to Top of this page.
    Previous topic this page
    Next topic this page