HP Quality Center : How To Make It Work
This is an unabashed review of how to use HP Quality Center (to version 9.2) despite annoyances.
This page is one in a series for QA Managers:
|
|
|
|
What is Quality Center?Quality Center (QC for short) is a software system offered by the Software division of HP (Hewlett-Packard), one of the largest corporations in the world.
QC was one of several products (along with LoadRunner, WinRunner, Quick Test Pro, as well as IT governance products) HP purchased as part of its acquisition of Mercury Interactive in 2003.
QC integrates management of Requirements, Test Plans, Test Execution, and Defects into a single system. This ambition makes it popular within corporations familiar with the "waterfall" approach of software developement. This expansive scope is reflected in the Quality Center product being renamed from "Test Director", a name still being used as a subset.
But despite this name change, many workers see the product as being designed to be primarily a device to input data for management reporting (a fancy timesheet) rather than as a tool for collaboration among workers.
Due to the amount of manual effort necessary to create management reports may make those down on the ground doing the work may feel they must make the "Sophie's Choice" of either disappointing executives who bought Quality Center (to "make your lives easier") or getting the real work (of testing) done.
The Annoyances (and what to do about them)
The decision to adopt Quality Center is often made between just C-level executives and HP salespeople who pitch how managers can get better visibility and control of the testing process. This management-oriented "business proposition" and "payback" sounds so enticing that executives often don't feel a need to get full input from technical people or to evaluate "minor technical" details (imagine the salesperson's hand waving). Why inject negativity?
This article warns of potential pitfalls and ways to avoid or work around them:
Backup On Your Own, But Can't DeleteQuality Center (at version 9.5) requires both a database server and a Windows 2003 Server, not Windows XP nor Vista. So to prevent people messing up the production environment with the result of experiments, a dev environment needs to be set up for QC admistrators to test out patches. This is necessary because patches have been known to fail and cause major problems.
The dev environment is in addition to production and an environment for production staging and fail-over. HP's dashboard requires yet another database.
To learn the system, users need a project with the set of data described in HP's User Guide documentation.
QC does not come with a Version Control system of its own. However, the QC database does have fields to accomodate versions. But until one is integrated into QC, each change in data wipes out whatever was in the database, with no hope of recovery.
Restore from a complete database backup is cumbersome because it's an "or or nothing" restore. So most sites create a stand-by environment to handle fail-over.
Managing Database GrowthAs with any database, the Quality Center database must be watched to ensure that it is built with the correct number of extents from the start and that those extents are extended as planned. This is to avoid disruptions from the database crashing when it unexplectedly runs out of disk space.
To anticipate turnover in DBA staff, have alerts sent to both the current DBA and the group distribution of system administrators and their managers.
All this is necessary becasuse the default workflow (business process) causes more disk space to be consumed than one would assume, due to the linkages among items Quality Center assumes is needed. For example, each test item must be copied to Test Lab.
The Fallacy of Coverge Percentage
Yes, Quality Center comes with a lot of reports and charts.
And it does come with a separate Dashboard that can be installed (on an additional server
using an additional database).
Default QC reports and graphs are conceived from a rather technical point of view, providing simple counts without personalization to vary for each manager only the fields, sorting, and filtering relevant to him/her. Simply put, the large complex organizations that typically use Quality Center need more complex reports and graphs than QC provides by default.
The "Coverage Progress" chart (at right) is prominent in the software to summarize progress toward "completion" at several moments over time.
At the last reporting period shown (on the far right at end of the ribbons), 100% of Requirements are shown as "Assigned" by the purple ribbon at the top. But those who have taken a system through implementation know that new Requirements inevitably crop up. So managers need QC to include quantification of "unknown unknowns" (an estimate of how much work might be added along the way).
It's difficult to figure out, but this chart shows the "Planned Coverage" (yellow ribbon) at 60%. Those presenting this chart need to be prepared to answer the naive but valid question of "why were only 60% planned coverage"? Many in our culture assume that 100% (or even 110%) is the usual goal. A skilled manager would ask for an analysis of the economic trade-offs between risk and work. There may not be payoff in doing everything planned, as there is less liklihodd of finding a defect.
Another common assumption is that whatever is planned (yellow line) should be executed (dark red line). Practically, however, it is often desirable to have plans for more work in case other items are blocked, which frees up time for additional work elsewhere. However, within the default Quality Center structure, it is difficult to differentiate between items commited to be executed versus items held in reserve.
I think it is a bad management tactic to measure test lab execution on a 100% scale because we want testers to define more tests than they plan on running so that if certain items get blocked, they would have already defined additional tests to run immediately rather than just waiting around. Judging testers based on a default 100% would discourage (indeed penalize) testers for being proactively productive.
Agile Ideas
Unlike a "heavyweight" test management approach such as Quality Center managing a backlog of verfication work by testers, the Agile approach finds wisdom in automated "Continuous Integration" (CI), where developers incorporate testing as part of their own work. An Agile developer cannot call a creation "done" unless it's also been tested. When a defect is identified, developers are supposed to stop and fix it immediately. Thus, there is no separate backlog of testing to prioritize and manage. The testing backlog is the same as the development backlog.
In the Agile world, testers are embedded among developers. So issues of "testability" are discussed and solved. 100% coverage is assumed with Agile.
Specialized Test Management Systems are an Agile Impediment —Elisabeth Hendrickson, October 6th, 2009... test automation created by a siloed QA team working in isolation to reverse-engineer existing software and automate tests against an untestable UI using proprietary tools accessible only to a few select team members is guaranteed to be incredibly expensive both to create and to maintain, and also ridiculously fragile. People still do that?! :-) No wonder "testers" get such a bad rap. Their mission is literally impossible. — Jason Huggins, April 4, 2008
The "test first" concept with Agile is that code to validate whether the code being developed works before that code is written. Thus, code is written to satisfy verifiable goals.
This includes testing on various browsers and against various internal conditions.
Linkages Among Components
To meet regulartory requirements, HP Quality Center provides a system to document "traceability" by establishing links for each and every item within a set of complex hierarchies. The set of coverages are illustrated here.
To track "Assigned Requirements" coverage, each requirement (product backlog item) must be linked to some Release Cycle. This makes replanning a cumbersome process. More troubling still is Quality Center provides no direct linkage between Requirements and Test Plans (called Test Sets). Requirements are instead linked to Test Set Folders which contain Test Sets.
To be able to calculate executed coverage, each test run must be manually created and designated with the test set it is based upon.
Metrics Managers WantThe most useful metrics are based on workflow tasks (what people do), and thus serve as predictors of where time and budgets might need adjustment.
Examples of more sophisticated metric reports include:
What Do You Call That, Again?
Different names for objects are used in different parts of the application.
So a certain amount of education is needed to clarify the vocabulary
among different names used in several places in the application:
The alphabetical list on the left (with objects in singluar form) is from the menu only admininistors use to define custom fields, which does not occur frequently. It's a good idea that administrators avoid using this as the basis for describing content in the database.
The hierarchy on the right is from TOOLS > Document Generator used to create detail reports. This is perhaps the best structure to use as a basis for defining a common vocabulary.
The names of tables and columns within the database need to be known to those crafting SQL for Excel reports.
Those coming from or transitioning to other systems (such as ScrumWorks or Microsoft TFS) would need to memorize the equivalence of terminology.
For example, ScrumWorks uses the word "Themes" within "Epics" to organize what needs to be achieved by each release.
|
Ownership by Department/WorkflowQC currenlty does not provide a mechanism for restricting actions on specific items. QC does not come with a field that defines the owner of each item, designating the role, organizatonal group, or individual who can delete or change or edit the item.
The result of this missing security functionality is that someone allowed to make deletions can delete any item within a project. This is a big hole in functionality considering that organizations that use QC are usually complex. The work-around:
Database SchemaThe Quality Center database is rather complex, with dozens of tables. So rather convulted SQL coding is often necessary to produce a usable report.
HP QC 9.2 quality_center_db.chm.
[After downloading, in Windows Explorer go into the Properties for the
file and unclidk the Read-Only property.
If you still have problems reading .chm files, see
this)]
Hierarchy RestructureIt is desirable for the hierarchy of test sets and test runs to take several forms over time.
Executives may want a hierarchy by version and release:
Those who need to analyze items across workgroups for duplication need an "upside down" hierarchy:
However, work is planned by those responsible for an area of responsibility, called a System or Workstream or PlanGroup:
What Are The Numbers Supposed To BeThe default reports only say how many of each type of data are in the database. But missing is what number is supposed to be there (at various moments of time during the project)?
Tracking the percentage of test plans (or other items) planned versus entered in the system is where SQL and Excel come in (next section), since this is not a default part of Quality Center.
Many don't bother with additional reports not because of the technical challenges, but because of politics. Putting out a projection would "hold their feet to the fire".
For some reason, most test managers are allowed to say that they can't predict how many defects will be produced. Unfortunately, most sales managers don't have such a luxury. Stockholders and the SEC require them to provide "guidance" each quarter.
Whether it's publicly voiced or not, organizations make some prediction about how many defects they expect in their staffing actions. If NO developer time has been allocated to analyze and fix bugs, then expect NO bugs to be found. If later there is a scamble to find time to address bugs found, then some "mismanagement" has occured.
Skill at predicting future events is a learned skill, and require insightful dialog among development and test managers, something that is lacking in many organizations.
Basic Burn-Down Charts
This gap is answered by software supporting Agile approaches which produce a burn-down chart such as this for points of functionality being developed by "Pigs".
In this example (produced by Rally software), each column illustrates the count of "function points" at the end of each day. The blue column is the backlog of how many remain to do.
In organizations that practice Agile, this chart is reviewed during daily scrums facilitated by a Scrum Master.
Each column defines the work for each sprint of fixed duration (1-3 weeks each).
The starting backlog is defined for each sprint are agreed upon based on a prioritized list of functionality maintained by the Product Owner.
Prioritization of features developed within each sprint is based on estimates of business value and effort obtained during Sprint Planning workshops and Product Backlog Refinement workshops.
The diagonal line in the basic burn-down chart is labeled "ideal" because it represents ideal circumstances (not management expectations). The difference between ideal and actual progress reflect blocks the team encounters.
The rate that features are worked off is (the team's capacity of items each sprint) is called its velocity.
Agile practioners prefer the horizontal time frame (including the length of each sprint) to not change to foster learning needed to improve the accuracy of estimating skill.
Changes To Quality Center Usage for AgileQuality Center was built to track individual tasks and the hours of work associated with each. It can also differentiate between completed (Accepted) items as green columns count versus items still to be processed as blue columns.
However, an important difference with Agile vs. traditional approaches to project management is that generalized points are used instead of absolute man-days so that estimation bias can be more easily adjusted for reality.
Such an approach requires custom fields in Quality Center.
Rally maintains this chart by establishing a two-way exchange of data with Quality Center.
But this should be done with the understanding that there is a fundamental difference between Quality Center and Agile/Scrum in that Quality Center forces estimates, whereas with Agile an "epic" can remain as such without being associated with a numerical budget of time.
Additional processing (by Rally) adjusts estimated completion dates based on Scrum estimating techniques.
True Agile practioners do not differentiate between testers who file bugs and developers who fix them, since both work toward the same definition of "done". Instead of saying "it's done when I say it's done" (after developers say they'r done), an interdisciplinary team gets done faster together without continuous integrations.
Enhanced Burn-Down Charts
ScrumWorks' Enhanced Burndown chart has a more informative presentation
which differentiates "headaches" (work to remove defects)
from original effort planned.
Some may find this too sophisticated / confusing.
The length of each bar — the total effort remaining after each period — is after a push down the original effort plus a push up to work off the headaches/defects remaining. An enhancement to this are different colors to depict new headaches/defects added during the period.
The red diagonal line displays the Best Fit line depicting the average velocity of burndown. Periods of slower progress would have bars floating above the red line. Periods of greater progress would have bars below the red line.
Producing a burn-down chart does not make an organization "agile" nor does it mean management is "in control". Burn-down charts do not address management of progress in all aspects of software development.
Being Agile takes a change in mindset about responding to change through constant communication and adjustment rather than pre-planned expository and adversarial contracts. An organization does not really get Agile if programmers report to a "Scrum Master" who maintain burn-down charts so managers can judge whether developers are creating detailed specs according to senior management expectations.
Becoming Agile: ...in an imperfect world (Paperback)
(Manning Publications, June 8, 2009)
by Greg Smith, Dr. Ahmed Sidky
explains Agile from a case-study perspective
within a 9 step process for adoption of agile practices.
and presents a
Agile Measurement Index
Succeeding with Agile by Mike Cohns
Excel From SQL
This is a "poor-man's" visualization capability. The Excel graphic reports I describe on another page are based on the structure of Excel chart files below:
The file "Defect Analysis Data R1 v3.xls" is replaced whenever it is generated from SQL within QC.
Each tab in the SQL within QC is produced as a separate tab in the output spreadsheet.
Another spreadsheet (the Base spreadsheet) references the Data spreadsheet.
A third spreadsheet then produces charts by referencing the Base spreadsheet.
The trouble with this approach (using Excel) is that these need to be run on a specific date (e.g., every Friday at noon) so that counts at each particular point in time can be captured when it's available.
A program to generate the count of each at various points in time can be generated, but would require more complex code.
Reports Generation OptionsTo it's credit HP does provide many ways to create custom reports. But there are so many different technologies to create reports that a manager may have to use several technologies to get a complete picture.
During my consulting engagements, I customize from working generic examples refined over several engagements.
Quality Center needs to be considered within the total set of enterprise applications operating within most large organizations:
The Architecture of External InterfacesTo get serious with managing software, I recommend that one treat data from Quality Center like they do with other corporate metrics, and get it integrated with the corporate dashboard (BI system).
Here is a flowchart of how Quality Center interacts with external data.
Integrations needs to be considered with:
VBScriptHP provides a rich VBScript API to retrieve data programmatically from the Quality Center database.
Data Uploads and Downloads (Import/Export)QC provides batch import capabilities in add-ins to Excel and Word.
Most people use Excel rather than Word (which requires each element to be marked).
The Excel interface is rather error-prone because importing is based on Excel column letters (A, B, C, etc.) rather than column names as in most CSV files. So here is my advice:
Certification and Training
Related Topics:
Performance Testing
NT Perfmon / UNIX rstatd Counters
Mercury LoadRunner
Mercury Virtual Table Server (VTS)
Mercury WinRunner
Rational Robot
Free Training!
Tech Support
|
| Your first name: Your family name: Your location (city, country): Your Email address: |
Top of Page 
Thank you! | |||
