Success criteria breed success

Page 1 of 10

Success criteria breed success

Karl Wiegers, Principal Consultant, Process Impact Summary: from the Rational Edge: Defining explicit success criteria during the project's Inception phase keeps stakeholders focused on shared objectives and establishes targets for evaluating progress. This article proposes a four-step process for defining project success criteria. Date: 15 Feb 2002 Level: Introductory Activity: 14474 views Comments: 0 (View | Add comment - Sign in) Average rating (63 votes) Rate this article Some rewarding journeys begin with no clear destination in mind, but if you don't know at the beginning of your software project how you're going to measure its business success, then you're headed for trouble. Defining explicit success criteria during the project's Inception phase keeps stakeholders focused on shared objectives and establishes targets for evaluating progress. For initiatives that involve multiple subprojects, success criteria help align each subproject with the big picture. In contrast, ill-defined, unrealistic, or poorly communicated success criteria can lead to disappointing business outcomes. Such vague objectives as "world class" or "killer app" are meaningless unless you can measure whether you've achieved them. This article describes a four-step process for defining project success criteria, as shown in Figure 1: 1. Identify project stakeholders. 2. Define business goals and objectives for both the development organization and its customers. 3. Identify constraints that limit the project manager's options. 4. Derive project success criteria that relate to the business objectives. Figure 1: A Four-Step Process for Defining Success Criteria

Step 1. Identify Stakeholders A project achieves success by delivering value to various stakeholders--people, groups, or organizations that are actively involved in a project, are affected by its outcome, or can influence its outcome. 1,2 Begin your quest for success by identifying these stakeholders and what they want. Value could translate

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 2 of 10

to time savings for a corporate department, market dominance for a commercial software vendor, or increased productivity for a user. Look for stakeholders both inside and outside the development organization, in the categories shown in Table 1.

Table 1: Possible Internal and External Stakeholder Categories Internal Project manager Program manager Executive sponsor or funding authority Team members, including developers, testers, analysts, tech writers, and support staff Company owners or shareholders Marketing Manufacturing Finance Legal Sales Support External

Various user classes Procuring customer Regulatory bodies Government agencies Subcontractors Venture capitalists Business partners

Next, do a stakeholder analysis to reveal the expectations each stakeholder group has for the project. First, identify each stakeholder's interests, or win conditions. Examples of interests include desired financial benefits, specific time to market, required functionality, performance targets, or other quality characteristics. Then, evaluate how the project will be affected if each interest is or is not satisfied. The project won't be able to fully satisfy everyone's interests, so this analysis will help you determine which ones are the most compelling. You should also assess the relative influence each stakeholder has on the project's decisions. Some stakeholders may wield great political power, whereas others might dictate essential product features, impose restricting constraints, or generate substantial revenue. Key stakeholders are those who can strongly influence the project's decisions--and those whom it is most important to satisfy. Ten years ago, my small software team was building a medium-sized information system for a large corporation's central research and development division. The stakeholder who represented our primary user class naturally wanted us to focus just on his department's needs. However, we identified the entire customer organization as another key stakeholder, which led us to include functionality that made the system valuable to additional user classes. The primary user class representative wasn't happy about the extra time it took to deliver "his" application, but stakeholder analysis helped us make the right strategic choices for the business as a whole. Expect to encounter conflicting stakeholder interests. Finance might want the lowest possible unit cost for the product, whereas the development team might wish to use expensive, cutting-edge technologies to meet stringent performance demands. Different market segments will value different combinations of time to market, feature richness, reliability, and usability. Identifying your key stakeholders will help you resolve such conflicts as they arise and to negotiate win-win solutions that maximize benefits for the greatest number of stakeholders. 3

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 3 of 10

Step 2. Define Business Goals and Objectives Business goals answer the question, "Why do we want to tackle this project?" You can transform each business goal into specific, attainable, verifiable, and prioritized objectives that you can use to assess whether the goal has been achieved. For example, a business goal might be to "Demonstrate competitive advantage with distinguishing features and technologies." A corresponding business objective might state that specific features must be operational by a particular date. Another objective might require the team to demonstrate by a specific date that the planned technologies are sufficiently robust for use in your product. Express your business objectives in terms of measurable business, political, or perhaps even social value. If you can't measure these objectives, then how can you tell whether you've achieved them? A project that satisfies its requirements specification and ships on schedule and budget is good; a product that achieves its intended business objectives is better. Well-crafted business objectives articulate the bottom-line outcomes that the product will deliver for stakeholders in both the customer and development organizations, including the expected time frame for achieving the objective, and how you will measure successful achievement. 4 Business objectives typically address: What the product must be and do (essential or distinguishing functions it will perform and how well it will perform them). Economic constraints (including development costs, cost of materials, and time to market). The product's operational and temporal context (for example, technologies required for compatibility in the target environment, backward compatibility, and future extensibility). Table 2 illustrates some financial and strategic business objective statements.

Table 2: Examples of Financial and Strategic Business Objectives Financial Capture a market share of X% within Y months. Increase market share in country X by Y% in Z months. Reach a sales volume of X units or revenue of $Y within Z months. Achieve X% profit or return on investment within Y months. Achieve positive cash flow on this product within Y months. Save $X per year by replacing a highmaintenance legacy system. Keep unit materials cost below X dollars per unit in the expected Y-year lifetime of the product. Reduce support costs by X% within Y months. Receive no more than X service calls per unit and Y warranty calls per unit Strategic Achieve a customer satisfaction measure of at least X within Y months of release. Process at least X transactions per day with at least Y% accuracy. Achieve a specified time to market that provides clear business advantages. Develop a robust platform for a family of related products. Develop specific core technology competencies in the organization. Have X positive product reviews appear in trade journals before a specified date. Be rated as the top product for reliability in published product reviews by a specified date. Maintain staff turnover below X% through the end of the project. Comply with a specific set of federal and state regulations. Reduce turnaround time to X hours on

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 4 of 10

within Z months after shipping.

Y% of customer support calls.

I know of one company that evaluates proposed software projects based on potential savings in three categories: external expenditures; internal staffing; other avoided costs. The project sponsor determines how much of the savings must come from each category. Each affected business unit measures its costs prior to launching the project and again after the new system is implemented. The business then tracks ongoing costs with the goal of confirming the projected savings by a previously established date, typically six months following implementation. Record your business goals and objectives in a strategic guiding document for the project, such as a vision and scope, 5 project overview, business plan, business case, project charter, or marketing requirements document. Some software project management plan templates include a section on management objectives and priorities, which could contain your business objectives and stakeholder analysis. As the team gets into technical design and implementation, they might discover that certain business objectives are not attainable. The cost of materials might be higher than planned, for example, or cutting-edge technologies might not work as expected. Business realities can also change along with evolving marketplace demands or reduced profit forecasts. Under such circumstances, you'll need to modify your business objectives and reevaluate to see whether the project is still worth undertaking. A telecommunications project I recall had as an objective to build a replacement interface unit for an existing product at a specified maximum unit cost and with a defined reusability goal. A review of the proposed architecture revealed that the product would exceed the unit cost and could not meet the reusability goal. Management revisited the business case, and concluded that the estimated unit cost was still low enough to justify the development. They simply revised the profit forecasts, dropped the reusability goal, and forged ahead. In short, the key stakeholders redefined "success" to correspond with business and technical realities.

Defining Success Criteria Using Planguage Too often, success criteria are worded imprecisely and cannot be assessed quantitatively, so it is difficult to judge objectively if a criterion has been satisfied. To address this problem, consultant and author Tom Gilb has developed a notation that he calls "Planguage," a planning language that permits much more precise statements of requirements and project goals. 8 Here is an example of how to express a success criterion using Planguage: TAG GIST SCALE METER MUST Performance.ResponseTime The response time of the application to database queries on the base user platform. Seconds. Elapsed time between pressing Enter to submit a query and the beginning of the display of query results. No more than ten seconds for all queries.

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 5 of 10

No more than three seconds for category one queries, eight seconds for all queries. No more than two seconds for all queries. WISH base user platform 400 mHz Intel Pentium II processor, 64 MB RAM, Microsoft Windows 98, QueryGen 3.3 running, single user, no other applications running on server. DEFINED PLAN Each success criterion receives a unique tag or label. The gist provides a more complete description of the criterion. Scale defines the units of measurement, and meter describes precisely how to make the measurements. You can specify several target values. The must criterion is the minimum acceptable achievement level for the item being defined. Success demands that every "must" be completely satisfied. The plan value is the nominal target, and the wish value represents the ideal outcome. Any specialized terms in the Planguage statement are defined to make them perfectly clear to the reader. Expressing your business objectives and technical project success criteria using Planguage provides a more precise way to know what "success" means on your project.

Step 3. Identify Constraints Constraints define the boundaries and restrictions within which the project manager must operate. Perhaps you've seen a sign in your local coffee shop that asks, "What do you want: good, fast, or cheap? Pick two." Although people often apply this classic tradeoff triangle to software, in fact a project manager must really make tradeoffs along five dimensions: features (or scope), quality, staff, cost, and schedule. 6 Each dimension is either a limiting constraint within which you must operate, a driver aligned with project success, or a degree of freedom that you can adjust within certain bounds. It is wise to classify the factors comprising the five dimensions into these three categories early in your project. Then, for constraints, state the immovable limit. For drivers, define the success goal. For degrees of freedom, identify the allowable range within which the project manager must operate. Managers often think of schedule as a constraint when it is really a driver. To make the distinction, consider the consequences of shipping late. If it would mean paying a contractual penalty or losing a unique market window forever, then schedule truly is a constraint. For example, an Internet commerce project that I know of delivered a set of Christmas baskets bundled with a digital camera and special software. Had those baskets and the Web site for purchasing them not been available in time for Christmas, then the project would have been a complete failure, with no return on investment. For holiday-related products and trade show unveilings, schedule really is a constraint, just as it was for Year 2000 remediation projects. With an understanding of a project's drivers and constraints, the project manager can make appropriate tradeoff decisions. Every tradeoff has a price: Adding more people is costly and doesn't shorten the schedule proportionately. 7 Increasing scope can degrade quality if the schedule isn't extended. Stringent quality demands increase the development and testing effort needed. Attempting to reduce cost by skimping on requirements development or design increases longterm costs and delays the release of an acceptable product. Keep in mind that not all project dimensions can be constraints. Project managers must have some flexibility to react to schedule slips, demands for increased functionality, staff turnover, and other

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 6 of 10

eventualities. The successful manager makes consistent decisions that will let him achieve his project's success criteria--its drivers--within the limits imposed by its constraints.

Step 4. Derive Project Success Criteria You can't judge whether a product satisfies all of its business objectives until some time after delivery. However, each business objective should imply technical success criteria that you can monitor prior to release. The development team cannot directly meet a business objective of "Capture a market share of forty percent within nine months," but they can decompose that objective into specific project actions that will help achieve the market share target. For example, business objectives pertaining to high customer satisfaction or competitive advantage might imply certain performance or reliability results. As development progresses, you can track your approach to these technical targets as an indicator of whether the product is likely to achieve its business objectives. If there is a disconnect between project success criteria and business objectives, then stakeholders might mistakenly think they are on target when they are actually chasing the wrong rainbow's pot of gold. Table 3 shows some simple project success criteria. Well-written success criteria are feasible, quantified, and verifiable (see Sidebar). Performance goals are often written against either internal benchmarks or external industry reference data. For example, a goal might compare a new compiler's performance to that of the prior version and also to the performance of a competing product under standard conditions. Such success criteria let the project team design tests that measure whether performance, reliability, transaction volume, or scalability goals are being met. Trends in these measures may provide early warning that you might miss a success target, so that the team can either take corrective action or redefine the success criteria. If you don't define clear priorities for the success criteria, then team members might end up working at cross-purposes, which leads to rework, frustration, and stress. Weight your success criteria based on how strongly they align with achieving critical business objectives, so team members will know which ones are essential and which are merely desirable. In one weighting scheme, stakeholders allocate 100 points to the success criteria associated with each business objective, with the more important items receiving more points. Consider summarizing all of your success criteria information in a table, listing each business objective, the stakeholder who provided it, corresponding project success criteria, and each criterion's method of measurement and priority weight (see Table 4 for an example). Use the stakeholder list to verify that you haven't inadvertently overlooked any important business objectives.

Table 3: Examples of Project Success Criteria. Examples of Project Success Criteria Total project cost does not exceed X% of the initial budget. The actual delivery schedule is within X percent of the initial deadline. All high-priority functionality defined in the requirements specification is delivered in the first release. The estimated number of residual defects does not exceed X per function point.

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 7 of 10

Load testing confirms successful scale-up to X concurrent users, with page download times no longer than Y seconds over a 56K dial-up connection. All unauthorized network intrusions are intercepted, prevented, logged, and reported. The mean time to failure under specified load conditions is at least X hours. The average installation time of the product is less than X minutes, with Y% of installations being error-free and requiring no support call. At least X% of software components were reused from previous applications. Prerelease development rework does not exceed ten percent of total development effort. Table 4: Sample Success Criteria Table. Business Objective Stakeholder Project Success Criteria Human Factors Department approves UI design through usability review. UI prototype evaluation with focus group results in at least Marketing a 4.2/5.0 rating and enables 90% of high-priority use cases to be performed. Passes acceptance criteria defined by 80% of beta site evaluators. Measurement All major severity UI design defects are corrected. Priority 30

Achieve a customer satisfaction measure of at least four on a scale of five within four months after release.

Survey of focus group members; count of 30 defined high-priority use cases. Pass/fail rating by beta 40 sites.

If you clearly define what success means at the beginning of your project, then you have a great chance of achieving it at the end. Understanding your stakeholders' interests, writing clear business objectives, and defining corresponding success criteria lay the foundation for a successful software project.

Acknowledgments The author appreciates the contributions made to this article by Ellen Gottesdiener, Michael Green, Karen King, Neil Potter, Kathy Rhode, Johanna Rothman, Erik Simmons, and Dan Starr.

Notes
1 Larry

W. Smith, "Project Clarity Through Stakeholder Analysis." CrossTalk, vol. 13, no. 12 (December 2000), pp. 4-9.
2A

Guide to the Project Management Body of Knowledge, 2000 Edition. Project Management Institute, 2000.
3 Barry

Boehm and Prasanta Bose,"A Collaborative Spiral Software Process Model Based on Theory W," Third International Conference on the Software Process, 1994. http://sunset.usc.edu/TechRpts/Papers/process.conf94.doc.ps [link no longer exists]

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 8 of 10

W. Wysocki, Robert Beck Jr., David B. Crane, Effective Project Management, 2nd Ed. John Wiley and Sons, 2000.
5 Karl

4 Robert

E. Wiegers, Software Requirements. Microsoft Press, 1999. A template for the vision and scope document is available from http://www.processimpact.com/goodies.shtml. E. Wiegers, Creating a Software Engineering Culture. Dorset House Publishing, 1996. P. Brooks, Jr., The Mythical Man-Month, Anniversary Edition. Addison-Wesley, 1995.

6 Karl

7 Frederick 8 Tom

Gilb, Competitive Engineering: A Handbook for Systems and Software Engineering Management Using Planguage. Addison-Wesley (forthcoming, 2002). About the author

As Principal Consultant for Process Impact in Portland, Oregon, Karl Wiegers specializes in requirements engineering, process improvement, peer reviews, and project management. Prior to founding Process Impact, he spent eighteen years as a research scientist, software developer, software manager, and process improvement leader at Eastman Kodak. In addition to authoring three books -Peer Reviews in Software: A Practical Guide (Addison-Wesley, 2002), Software Requirements (Microsoft Press, 1999), and Creating a Software Engineering Culture (Dorset House, 1996) -- he has published more than 140 articles on software, chemistry, and military history. He holds a B.S. in chemistry from Boise State College and M.S. and Ph.D. degrees in organic chemistry from the University of Illinois.

Close [x]

developerWorks: Sign in
If you do not have an IBM ID and password, register here. IBM ID: Forgot your IBM ID? Password: Forgot your password? Change your password? After sign in: Stay on the current page

By clicking Submit, you agree to the developerWorks terms of use.

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 9 of 10

Submit

Cancel

The first time you sign into developerWorks, a My developerWorks profile is created for you. This profile includes the first name, last name, and display name contained in the profile you created when you registered with My developerWorks. Selected information in your My developerWorks profile is displayed to the public, but you may edit the information at any time. Your first name, last name (unless you choose to hide them), and display name will accompany the content that you post. All information submitted is secure. Close [x]

Choose your display name

The first time you sign in to developerWorks a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks. Display name: (Must be between 3 31 characters.)

Note: Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email for privacy reasons.

By clicking Submit, you agree to the developerWorks terms of use.

Submit Cancel

All information submitted is secure. Average rating (63 votes) 1 star 2 stars 3 stars 4 stars 5 stars
Submit

1 star 2 stars 3 stars 4 stars 5 stars

Add comment:

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011

Success criteria breed success

Page 10 of 10

Sign in or register to leave a comment. Note: HTML elements are not supported within comments.

1000 characters left

Post

Be the first to add a comment Trademarks | My developerWorks terms and conditions

http://www.ibm.com/developerworks/rational/library/2950.html

2/6/2011