Vous êtes sur la page 1sur 99

PETER H.

ROSSI
HOWARD E. FREEMAN
MARK W LIPSEY

~LUATON
To the memory of Donald T. Campbell-
architect of modem evaluation theory and practice;
mentor, directly and indirectly, to ali evaluators

A SYSTEMATIC APPROACH

SAGE Publications
lnternationat Educalional and Professiona/ Pub/isher
Thousand Oaks London New Delhi
REVIZE 2004
Copyright 1999, 1993, 1989, 1985, 1982, 1979 by Sage Publications, Inc.

All rights reserved. No pllrt af this book may be reproduced or utilized in any farm or by any
means, electronic ar mechanical, including photocapying1 recording, ar by any infannation
storage and retrieval system, without pennission in writing from the publisher:

For informadon:
SAGE Publications, Inc.
CONTENTS
2455 Tuller Rcad
Thousand Oaks, Califomia 91320
E-mail: order@sagepub.com
SAGE Publications Ltd.
6 Bonhill Street
London EC2A 4PU Preface IX
United Kingdom
::JQ.3- P.CJ~~:;
MASARYKO': \ i .. 1.1 Y BRX
SAGE Publications India Pvt. Ltd. Fakulta s1_,~ ,,i: --~1 :;Ludi 1 Programs, Policies, and Evaluations 3
M-32 Market stfeJni l-s.ll1u\'nll What Is Evaluation Researchl 4
Greater I{ailash I Gorkeho 7
New Delhi li O 048 lndia 60200 BRNO A Brief History of Evaluation Research 9
An Overview of Prograrn Evaluation 20
Printed in the United States of America
Evaluation Research in Practice 27
Llbrary of Congress Cataloging-in-Publicadon Data Who Can Do Evaluations? 33
Rossi, Peter Henry, 1921- J
Surnrnary 35
Evaluation: a systematic approach I by Peter H. Rossi,
Howard E. Freeman, and Mark W Lipsey. - 6th ed. 2 'Thiloring Evaluations 37
p. cm.
Includes bibliographical references and index.. What Aspects of the Evaluation Plan Must Be
ISBN 0-761908935 (acid-lree paper) Tailored? 38
1. Evaluation research !Social action programs). 1. Freeman,
Howard E. li. Lipsefi Mark W III. Title. What Considerations Should Guide Evaluation
H62.R666 1998 Planningl 39
361.6'1'072-dc21 98-40244
The Nature of the Evaluator-Stalceholder Relationship 54
This book is printed on acid-free pape[ 1 Evaluation Questions and Evaluation Methods 62
Stitching It Ali Together 74
1
Surnrnary 76

0203041098765 3 Identifying lssues and Formulating Questions 79


What Malces a Good Evaluation Questionl 81
C, Dcbornh Lnughton
Acquiring Editor:
Editorial Assistnnt: Eileen Carr
Deterrnining the Questions on Which the Evaluation
Production Editor: Astrid Virdlng Should Focus 88
Designerf1Ypesetter: Janclle LeMaster
Covcr Designer: l.l.vi Balasuriya Collating Evaluation Questions and Setting Priorities 115
Surnrnary 116
4 Assessng the Need for a Program 119 Choosing the Right Impact Assessment Strategy 274
The Role of Evaluators in Diagnosing Social Summary 275
Condtions and Service Needs 120
Defining Social Problems 125 1 8 Randomzed Designs for lmpact Assessment 279
Specifying the Extent of the Problem: When, Where, .11 Unts of Analysis 279
andHowBig? 126 Experiments as an Impact Assessment Strategy 280
Defining and Identifying the Targets of Interventions 137 Analyzing Randornzed Experiments 292
Describing the Nature of Service Needs 146 Lirntations on the Use of Randomized Experiments 297
Summary 151 Summary 305

5 Expressng and Assessng Program Theory 155 9 Quasi-Experimental Impact Assessments 309
The Evaluability Assessment Perspective 157 Quasi-Experimental Impact Assessment 309
Eliciting and Expressing Program Theory 160 Constructing Comparison Groups in Quasi-
Assessing Program Theory 173 Experimental Evaluations 313
187 Some Cautions in Using Constructed Contrais 332
Summary
Summary 340
6 Montoring Program Process and Performance 191
What Is Program Montoring? 192 10 Assessment of Full-Coverage Programs 343
Perspectives on Program Montoring 203 Nonunform Full-Coverage Programs 34~

Montoring Service Utilization 207 Reflexive Contrais 347


Montoring Organizational Functions 214 Shadow Contrais 356
Montoring Program Outcomes 220 Summary 363
Collecting Data for Montoring 225
229 11 Measuring Efficiency 365
Analysis of Montoring Data
231 Key Concepts in Efficiency Analysis 367
Summary Methodology of Cost-Benefit Analysis 374
235 Cost-Effectiveness Analysis 390
7 Strateges for lmpact Assessment
Key Concepts in Impact Assessment 236 Summary 394
Extraneous Confounding Factors 241
244 12 The Social Context of Evaluaton 397
Design Effects The Purposefulness of Evaluation Activities 398
Design Strategies for Isolating the Effects of
257 The Social Ecology of Evaluations 400
Extraneous Factors
260 The Profession of Evaluation 417
A Catalog of Impact Assessment Designs
268 Evaluation Standards, Guidelines, and Ethcs 425
Judgmental Approaches to Impact Assessment
Utilization of Evaluation Results 431
Quantitative Versus Qualitative Data in Impact
Assessments 269 Epilogue 436
Inference Validty Issues in Impact Assessment 271 Summary 439
Glossary 441
References 451
Author Index 477
Subject Index 483 PREFACE
About the Authors 499

T hrnughout the six editions of this book, its


objectives have remained constant. It pro-
vides an introduction to the range of research
provide references to more advanced discus-
sions of critical topics for those readers who
want to pursue some topic in greater depth.
activities used in appraising the design, imple- When Howard Freeman died suddenly and
mentation, and utility of social programs. That prematurely shortly before the filth edi tion was
set of research procedures known as evaluation published, I was quite sure that there would
has become solidly incorporated into the rou- never be a sixth edition of which I would be a
tine activities of ali leveis of government living coauthm: Howard Freeman had been a
throughout the world, into the operations of person of great lmowledge, experience, and wit,
nongovernmental organizations, and into the working with whom was most of the reward of
discussions of social issues. We believe that collaboration. Without Howard to work with,
evaluation research has influenced social poli- the prospect of yet another revision seemed
cies and other efforts to improve the social painfully unrewarding.
conditions of the citizens of many co=uni- Howeve~ it became increasingly clear after
ties. It is also an exciting professional role a fewyears had passed that the fifth edition was
providing opportunities to advance social well- rapidly becoming out of date. Our noble Sage
being alongwith the exercise of technical skills. senior edito~ C. Deborah Laughton, started to
Evaluation: A Systematic Approach has urge me to consider a major revision. At. first I
strong ambitions to co=unicate the techni- resisted on the grounds that I had no collabo-
cal knowledge and collective experiences of rator with whom to work. Working with great
practicing evaluators to those who might con- subtlety, stealth, and a bit of benign deception,
sider engaging in evaluation and to those who she told Mark Lipsey that I was interested in
need to lmow what evaluation is ali about. Our having him as coauthor at the sarne time telling
intended audiences are students, practitioners, me tliat Mark was interested in becoming a
novice social researchers, public officials, spon- coauthm: Knowing the high quality of Mark
sors of social program.s, social commentators, Lipsey's evaluation work, I was quite flattered
and the legendary intelligent layperson. Al- at bis interest. He tells me that he was also
though some very experienced evaluators pleased that I was interested in worlng with
might Iind our book too elementary; we hope him. And so a new collaboration was forged.
that reading it will help others at earlier stages Mark cannot replace Howard: He brings to the
of their encounters with evaluation. We also collaboration a different baclcground and set of

ix
x EVALUATION

experiences, equal to Howard's in depth and We are grateful to the following review-
sophistication but dilierent in content in ways ers for their co=ents: Jack McKillip, Ron
that enrich the sixth edition. Workingwith him Andersen, Melissa Jonson-Reid, David MacPhee,
has been a great pleasure, greatly enhanced by and William Shadish. A special acknowl-
his sense of proportion and ready humm: I arn edgrnent is also extended to Kate Peterson for
grateful to C. Deborah Laughton for her skillful her extensive copyediting efforts.
matchrnaking and for the improvements in We have dedicated this edition to the mem-
this edition that it made possible. ory of Donald T. Carnpbell. In our view, there
Most of the new material that appears in is no one who more deeply influenced the
this revision is Mark Lipsey's contribution. development of evaluation. Not only are his
The previous edition covered quite sketchily publications cited in ali major works on evalu-
evaluation diagnostic procedures and how ation, but he was also an important mentm:
evaluations should be tailored to fit prograrns Several generations of his students now are
and social contexts. The current version has leading figures in the field and their students
greatly expanded coverage of those topics, add- are rising quickly to prominence. His influence
ing important detailed material on explicating on this volume can be seen in the discussion of
prograrn theory. The treatment in the first five impact assessment, which follows rather
chapters of this version carries the reader closely his exposition of research designs laid
through a sequence that follows more closely down in his 1966 publication (Carnpbell and
the typical steps that the development of evalu- Stanley, 1966). Ali evaluators can see farther
ations take. Lipsey has also updated the exhib- because we stand on his shoulders: Carnpbell
its that appear in this edition, often with exam- was a giant who gave us a lofty perspective on
ples of evaluations currently under way ar very how social science can advance the improve-
recently completed. ment of the human condition.
-P.H.R.

i
PROGRAMS, POLICIES,
AND EVALUATIONS
Social program; An organized, planned, and usualty ongoing effort designed to ameliorate a This chapter introduces program evaluation as a robust arena of activity ifuected at collect-
social intervention social problem ar improve social conditions. ing, analyzing, interpreting, and communicating information about the effectiveness of social
Program evaluation The use of social research procedures to systematically investigate the programs undertaken for the pwpose of improving social conditions. Evaluations are conducted
effectiveness of social intervention programs that is adapted to their political for a variety of practical reasons: to aid in decisions concerning whether programs should be
and organizational environments and designed to inform social action in ways continued, improved, expanded, or curtailed; to assess the utility of new programs and initia-
that improve social conditions. tives; to increase the effectiveness of program management and administration; and to satisfy
Procedures for studying social behavior devised by social scientists that are the accountability requirements of program sponsors. Evaluations also may contribute to
Social research methods
based on systematic observation and logical rules for drawing inferences
substantive and methodological social science knowledge.
from those observations.
Understanding evaluation in contemporary context requires some appreciation of its history,
its distinguishing concepts and purposes, and the inherent tensions and challenges that shape its
Comprehensive evaluation An assessment of a social program that covers the need for the program, its practice. Program evaluation represents an adaptation of social research methods to the task of
design, implementation, impact, and efficiency.
studying social intervention in its natural political and organizational circumstances so that
Evaluation sponsor The person(s), group, or organization that requests or requires the evaluation sound iudgments can be drawn about the need for intervention and the design, implementation,
and provides the resources to conduct it. impact, and efficiency of programs that address that need. Individual evaluation studies, and the
Individuais, groups, or organizations having a significant interest in how well cumulation of knowledge from many such studies, can make a vital contribution to informed
Stakeholders
a program functions, for instance, those with decision-making authority over it, social action aimed at improving the human condition.
funders and sponsors, administrators and personnel, and clients or intended The principal pwpose of program evaluation, therefore, is to provide valid findings about the
beneficiaries.
effectiveness of social programs to those persons with responsibilities or interests related to their
creation, continuation, or improvement.

ong before Sir Thomas More coined the lems and estimate their scope and which prob-
L word utopia in 1516, many persons had
tried to envision a perfect world. That their
lems are salient to us have changed over time
with shifts in values and lifestyles. And it is
aspirations, and ours, have not been realized is equally true that communities, societies, and
evident in the social problems and attendant cllltures differ widely in the attention they pay
personal problems that confront every country to particular problems. But now, as evei; to
in the world. TI-ue, how we define social prob- borrow Irom Charles Dickens, these are the

3
4 EVALUATION Programs, Policies, and Evaluations 5

best af times for some of us and tbe worst of ciency and tbat limes are rich in vitarnin C. What are the nature aod scope of the prob- lished in 1980 to fostertbe application of evalu-
Nevertheless, tbe intervention worked and lerol Where is it located, whom does it ation research to the program and policy review
times for otbers.
affect, and how does it affect therol functions that the GAO perforrns for Congress.
Since antiquity, organized efforts have been British seamen eventually were compelled to
undertaken to describe, understand, and ame- consume citrus fruit regularly, a practice that What is it about the problero ar its effects During its history, it has evaluated such en-
liorate tbe defects in tbe human condition. gave rise to tbe still-popular label limeys. Inci- that justifies new, expanded, ar modified deavors as the procurement and testing of mili-
This book is rooted in tbe tradition of scientific dentally, it took about 50 years before the cap- social prograrns l tary hardware, quality contrai for drinking
study of social problems-a tradition tbat has tain's "social prograrn" was widely adopted. What feasible interventions are likely to wate~ the maintenance of major highways, tbe
aspired to improve tbe quality of our physical Then, as now, dillusion and acceptance of sigoificantly arneliorate the problerol use of hormones to stimulate growth in beef
and social environments and enhance our in- evaluation findings did not come easily. What are the appropriate target populations cartle, and otber organized activities far afield
dividual and collective well-being tbrough tbe for interventionl from human services.
systematic creation and application of knowl- Indeed, tbe techniques described in this
Is a particular intervention reaching its tar-
edge. Altbough tbe term evaluadon reseGich is get populationl text have utility to virtually ali spheres of activ-
WHAT IS EVALUATION RESEARCH? ity in which issues of the effectiveness of orga-
a relatively recent invention, tbe activities tbat
Is theintervention beingimpleroented welll nized social action are raised. For example, the
we will consider under this rubric are not. They Are the intended services being providedl
can be traced to tbe very beginnings of modem A!though the broadest definition of evaluation mass co=mlication and advertising indus-
science. Three centuries ago, as Cronbach and includes ali efforts to place value on events, Is the intervention effective in attaining the tries use fundamentally tbe sarne approaches
desired goals ar benefitsl in developing media prograrns and marketing
colleagues j 1980) point out, Thomas Hobbes tbings, processes, ar people, we will be con-
and his contemporaries endeavored to devise cemed here with tbe evaluation of social pro- How much does the prograrn costl products; co=ercial and industrial corpora-
numerical measures to assess social conditions grams. For purposes of orientation, we offer a Is theprograrncostreasonableinrelation to tions evaluate the procedures tbey use in select-
and identify tbe causes of mortality, morbidity, preliminary definition of social prograrn evalu- its effectiveness and beoefitsl ing, training, and promoting employees and
and social disorganization. ation now and will present and discuss a more organizing their workforces; poltica! candi-
Even social experiments, tbe most techni- complete version !ater in this chapter: Program dates develop tbeir carnpaigns by evaluating
cally challenging form of contemporary evalu- evaluadon is the use of social reseGich proce- Exhibit 1-A conveys tbe views of one feisty tbe voter appeal of different strategies; con-
ation research, are hardly a recent invention. dures to systemadcally investigate the effec- senator about the need for evaluation evidence sumer products are tested for performance,
One of the earliest "social experiments" took dveness of social intervendon programs. More on prograrn effectiveness. Answers to questions durability, and safety; and administrators in
place in tbe l 700s when a British ship's captain specifically, evaluation researchers leva!uators) such as those above are necessary for local ar botb the public and private sectors often assess
observed the lack of scurvy among sailors ser- use social research metbods to study, appraise, specialized prograrns, such as job training in a the clerical, fiscal, and personnel practices of
ving on tbe naval ships of Mediterranean coun- and help improve social prograrns in ali their small town, a new mathematics cutriculum for their organizations.
tries. He noticed, too, tbat citrus fruit was part important aspects, including the diagnosis of elementary schools, ar tbe outpatient services The distinctions arnong tbese various ap-
of tbeir rations. Thereupon he made half his the social problems tbey address, tbeir concep- of a co=wlitymentalhealth clinic, as well as plications of evaluation lie primarily in tbe
crew consume limes while tbe otber half con- tualization and design, tbeir implementation for broad national ar state prograrns such as nature and goals of tbe endeavors being evalu-
tinued witb tbeir regular diet. Despite much and administration, their outcomes, and tbeir healtb care, farnily preservation, ar educational ated. Our emphasis in this text is on the evalu-
grumbling among tbe crew in tbe "treatrnent" efficiency. reforrn. Providing tbose answers is tbe work of ation of prograrns designed to benefit tbe hu-
group, the experim.ent was a success-it At various times, policymakers, funding persons in tbe prograrn evaluation field. man condition ratber tban witb such purposes
showed that consurning limes prevented organizations, planners, program managers, Altbough this text emphasizes tbe evalu- as increasing profits ar amassing influence and
scurvy. taxpayers, ar prograrn clientele need to distin- ation of social prograrns, especially human ser- power This choice of focus stems not from a
The good captain probably did not know guish worthwhile prograrns from ineffective vice programs, program evaluation is not re- sense of righteousness about tbe proper appli-
tbat he was evaluating a demonstration project ones and launch new prograrns ar revise exist- stricted to tbat arena. An excellent example of cation of social research metbods but from a
nor did he likely have an explicitimpact theory ing ones so as to achieve certain desirable re- tbe broad scope of prograrn evaluation is pro- desire to concentrate on a particularly signifi-
ia term we will discuss !ater), narnely, that sults. To do so, tbey must obtain answers to vided by tbe work of tbe Prograrn Evaluation cant and active area of evaluation combined
scurvy is a consequence of a vitarnin C defi- questions such as tbe following: Metbodology Division of tbe U.S. General Ac- witb a practical need to limit tbe scope of tbe
counting Office IGAO). This wlit was estab- book.
6 EVALUATION Programs, Policies, and Evaluations 7

But ali the while we were taking on this large-- to cost $930 miltion over five years, starting at once case and site variations are taken into account. ln other words, there are either negative
and, as we can now say, hugely successful-effort $60 million in fiscal year 1994. For three decades effects or not effects.
[deficit reduction ], we were constantly besieged 1 had been watching families come apart in our This is nothing new. Here is Peter Rossi's conclusion in his 1992 paper, "Assessing Family
by administration officials wanting us to add society; now 1was being told by seemingly every- Preservation Programs." Evaluations conducted to date "do not forma sufficient basis upon
money for this social program ar that social one on the new team that one more program which to firmly decide whether family preservation programs are either effective ar not. 11
program .... My favorite in this miscellany was would do the trick.... At the risk of indiscretion, May 1 say to you that there is nothing in the least surprising in either of these findingsl
something called "family preservation, 11 yet let me include in the record at this point a letter From the mid-60s on this has been the repeated, 1 almost want to say consistent, pattern of
another categorical aid program (there were a 1 wrote on July 28, 1993, to Dr. Laura D'Andrea evaluation studies. Either few effects ar negative effects. Thus the negative incarne tax
dozen in place already) which amounted to a Tyson, then the distinguished chairman of the experiments of the 1970s appeared to produce an increase in family breakup.
dollop of social services and a press release for Council of Economic Advisors, regarding the lhis pattern of "counterintuitive" findings first appeared in the 1 60s. Greeley and Rossi,
some subcommittee chairman. lhe program was Family Preservation program: some of my work, and Coleman's. lo this day l cannot decide whether we are dealing here
with an artifact of methodology ar a much larger and more intractable fact of social programs.
ln any event, by 1978 we had Rossi's lron Law. To wit: "lf there is any empirical law that is
Dear Dr. Tyson:
emerging from lhe past decade of widespread evaluation activity, it is that the expected
You will recall that lastThursdaywhen you so kindly joined us ata meeting of the Democratic value for any measured effect of a social program is zero. 11
Policy Committee you and 1 discussed the President's family preservation proposal. You 1 write you at such length for what 1 believe to be an important purpose. ln the last six
indicated how much he supports the measure. t assured you 1, too, support it, but went on months 1 have been repeatedly impressed by the number of members of the Clinton
to ask what evidence was there that it would have any effect. You assured me there were administration who have assured me with great vigor that something ar other is known in
such data. Just for fun, 1 asked for two citations. an area of social policy which, to the best of my understanding, is not known at ali. This
The next day we received a fax from Sharon Glied of your staff with a number of citations seems to me perilous. lt is quite possible to live with uncertainty, with the possibility, even
and a paper, "Evaluating the Results, 11 that appears to have been written by Frank Farrow of the likelihood that one is wrong. But beware of certainty where nane exists. ldeological
the Center for the Study of Social Policy here in Washington and Harold Richman at the certainty easily degenerates into an insistence upon ignorance.
Chapin Hall Center at the University of Chicago. The paper is quite direct: "Solid proof that The great strength of political conservatives at this time (and for a generation) is that they
11
family preservation services can affect a state's overall placement rates is still lacking. are open to the thought that matters are complex. Liberais got into a reflexive pattern of
Just yesterday, the sarne Chapin Hall Center released an "Evaluation of the Illinois Family denying this. 1 had hoped twelve years in the wilderness might have changed this; it may be
First Placement Prevention Program: Final Repor!." This was a large scale study of the Illinois it has only reinforced it. lf this is so, current reviva! of liberalism will be brief and inconse-
Family First initiative authorized by the Illinois Family Preservation Act of 1987. lt was quential.
"designed to test effects if this program on out-of-home placement and other outcomes, Respectfully,
such as subsequent child maltreatment.11 Data on case and service characteristics were
provided by Family First caseworkers on approximately 4,500 cases: approximately 1,600
Senator Daniel Patrick Moynihan
families participated in the randomized experiment. lhe findings are clear enough.
Overall, the Family First placement prevention program results in a slight increase in
placement rates (when data from ali experimental sites are combined). This effect disappears SOURCE: Adapted, with permission, from D. P. Moynihan, Miles to Go: A Personal History of Social Policy {Cambridge
MA: Harvard University Press, 1996), pp. 47-49. '

The importance of evaluating social pro- ciencies in the quality of human life, both in tives and improve existing ones comes the need yield demonstrable and proportionate social
grams-both those currently in effect and those industrialized countries and in less developed for evaluation to determine if those efforts are benefits. Moreove~ experiences of the past sev-
in various stages of design and pilot testing- nations, needs no elaboration here. But along worthwhile. Limited resources for social pro- eral decades have highlighted the barriers to
should not be underestimated. The continuing with the need for purposeful, practical, and grams in every country, including the United successful implementation of social programs
challenge of devising ways to remedy the defi- well-organized efforts to implement new initia- States, malce it critical that such investments and, correspondingly, the importance of assess-
8 EVALUATION Programs Policies, and Evaluations
1 9

ing the practicality of program design and the A small number of philanthropic advo- and increasing employment. Many of the pro- company reorganized its employees into inde-
effectiveness of program operations. cates of sclmol voucliers have initiated a pri- gram features studied were incorporated in the pendent work teams. Within the teams, work-
To 11put some meat on the bones 11 and make vately funded program in New York City for federal welfare reforms passed in 1996 (Per- ers designated and assigned tasks, recom-
the notion of program evaluation more con- poor fanlilies with children in the first three sonal Responsibility and Work Opportunity mended productivity quotas to management,
crete, we offer below examples of social pro- grades of more disadvantaged public schools. Reconciliation ActJ. and voted on the distribution of bonuses for
grams that have been evaluated under the ln spring 1997, scliolarships of $1,400 for a productivity and quality improvements. Infor-
sponsorship of local, state, and federal govem- period of three years were offered to eligible Fully two-thirds of the world's rural chil-
mation from an assessment of the program
mental agencies, intemational organizations, fanlilies togo toward tuition costs in the private dren suffer mild to severe malnutrition, with
revealed that it reduced days absent from the
private foundations and philanthropies, and schools of their clioice. Some 14,000 scholar- serious consequences for their health, physical
job, turnover rates, and similar measures of
both nonprofit and for-profit associations and ship applications were received, and 1, 500 suc- growth, and mental development. A major
employee inelliciency.
corporations. cessful candidates were chosen by random se- demonstration of the potential for improving
lection. Talcing advantage of this mode of children's health status and mental develop-
With support from the U.S. Department ment by providing dietary supplements was These short examples illustrate the diver-
selection, Mathematica Policy Research is re-
of Justice, the cities of Houston and Newark undertaken in Central America. Pregnant sity of social interventions that have been sys-
garding the program as a randomized experi-
instituted co=unity policing on a trial basis. women, lactating mothers, and children from tematically evaluated. Howeve~ ali of them
ment and intends to compare educational out-
ln Houston the police set up neighborhood birth through age 12 were provided with a daily involve one particular evaluation activity: the
comes among those selected and attending
substations, conducted doar-to-doar surveys of high-protein, high-calorie food supplement. assessment of program outcomes. As we will
private schools with outcomes among those
citizen problems, started newsletters, and held Results showed major gains in physical growth discuss late~ evaluation may also focus on the
who were not selected. The evaluation will be
co=unity meetings. ln Newark the police and modest increases in cognitive functioning. need for a program, its design, operation and
conducted over a three-year period.
established local walking police beats, dis- service delivery, ar elliciency. Before moving
persed groups of loiterers, and conducted ran- A co=unity mental health center in a Over the past two decades, the number aliead to more fully describe the nature and
dom checks of motorvehicles. ln both cities the medium-sized New England city developed an of reported cases of child abuse and neglect has range of program evaluation, we provide a brief
trial neighborhoods showed increases in citizen extensive program using local co=unity more than doubled in the United States. As a history of the development of the field to con-
confidence in the police and slight reductions members to counsel teenagers and adults about consequence more than half a million children vey a sense of the traditions in which current
in crimes when compareci with similar areas their emotional, sexual, and educational prob- are in foster or group care. Concemed that work is rooted.
without co=unity policing. lems. Compareci with persons treated by psy- removal from their fanlilies might be harmful,
chiatrists and social workers, the clients of the many child welfare agencies have provided
ln several major cities in the United ABRIEF HISTORY OF
indigenous counselors did as weli in terms of short-term intensive services to fanlilies with
States, a large private foundation provided the
need for hospitalization, maintenance of treat- abused ar neglected children with the goal of EVALUATION RESEARCH
initial operating costs to establish co=unity preventing removal of the children from their
ment, and self-reports of satisfaction with the
health centers in low-income areas. The cen- fanlilies while keeping them safe from further As we have noted, evaluation researcli is one
center
ters were intended as an altemative way for abuse ar neglect. Several evaluation studies facet of the general use of social research for
residents to obtain ambulatory patient care ln the past decade, the federal govem- have been undertalcen in whicli children at risk understanding and addressing social problems.
otherwise available to them only from hospital ment has allowed states to modify theirwelfare of being removed from their homes were ran- Howeve~ despite historical roots that extend to
outpatient clinics and emergency rooms at programs provided that the clianges were domly assigned to "family preservation" pro- the 1 7th century, systematic evaluation re-
great public cost. It was further hoped that by evaluated for their effects on clients and costs. grams ar to the usual service. Those assigned search is a relatively modem development. The
improving access to sucl1 care, the clinics might Some states instituted strong work and job to family preservation programs were no less application of social research methods to pro-
increase timely treatment and thus reduce the training requirements, others put time limits likely to end up being removed from their gram evaluation coincides with the growth and
need for lengthy and expensive hospital care. on benefits, and a few prohibited increases in homes, showing that these programs were in- refinement of the methods themselves as wel!
Evaluations indicated that some of these cen- benefits for children bom while on the welfare effective. as with ideological, political, and demographic
ters were cost-effective in comparison with ralis. Evaluation researcli showed that sucli changes that have occurred during this century.
hospital clinics. policies were capable of reducing welfare ralis ln an effort to increase worker satisfac- Of key importance were the emergence and
tion and product quality, a large manufacturing increased standing of the social sciences in
10 EVALUATION Programs, Policies, and Evaluations 11

particular boost provided by its contributions Rossi, and Wright, 1980; Levine et ai., 1981).
during World War II. Stouffer and his associates Expanding knowledge of the methods of social
worked with the U.S. Anny to develop proce- research, including sample surveys and ad-
dures for monitoring soldier morale and evalu- vanced statistical procedures, and increased
No one can deny the progress in the social upon as the creation of experimental laboratories ate personnel policies and propaganda tech- funding and administrative know-how, made
sciences. But with ali the exacting methods de- for the social scientists, and for the social workers, niques, whereas the Office of War Information possible even large-scale, multisite evaluation
veloped, the economists, sociologists, and politi- educators, and administrators who may profit used sample surveys to monitor civilian morale studies.
cal scientists have suffered from a lack oi large- from their research. (Stouffer et ai., 1949). A host of smaller studies
scale experimental set-ups to match the everyday These laboratories, sei up by the planning agen- Duriog the 1960s, the numbers of papers
assessed the efficacy of price contrais and me- and books about evaluation research grew dra-
resources of the scientists in the laboratory. cies of the New Deal, permita more effective use
dia campaigns to modify American eating hab- matically. Hayes's (1959) monograph on evalu-
The current enthusiasm over planning of the experimental method in the research its. Similar social science efforts were mounted ation research in less developed countries,
schemes now being devised by the alphabetical projects of the social scientists. This research, in
corporations of the federal government furnishes
in Britain and elsewhere. Suchman's (1967) review of evaluation re-
turn, would not only be an addition to science
some hope that this deficiency may be partially but would also be a form oi social auditing for searc11 methods, and Campbell's (1969) call for
remedied. The blueprints of these agencies and the planning authorities in noting and accounting The Boom Period in social experimentation are a few illustrations.
the carrying out of their plans may well be looked for the changes wrought by the programs. Evaluation Research ln the United States, a key impetus for the
spurt of interest in evaluation researc11 was tl1e
SOURCE: Adapled, with permission, from A. 5. S!ephan, "Prospects and Possibilities: lhe New Deal and the New Social Following World War II, numerous major federal war on poverty, initiated under Lyndon
Research," Social Forces, May 1935, 13:515, 51 B. programs were launched to meet needs for Tohnson's presidency. By the late 1960s, evalu-
urban development and housing, techoological ationresearchhad become, in thewords ofWall
and cultural education, occupational training, Street, a growth industry.
univers1t1es and increased support for social duce water boiling as a public health practice and preventive health activities. It was also ln the early 1970s, evaluation research
research. Social science departments in univer- in villages in the Middle East is a landmark in duriog this time that major commitments were emerged as a distinct specialty field in the social
sities became centers of early work in program the pre-World War II literature. And it was an made to international programs for farnily sciences. A variety of books appeared, including
evaluation and have continued to occupy an Arkansas sociology professor who first pleaded planning, health and nutrition, and rural deve]- the first texts ('Neiss, 1972), critiques of the
influential place in the field. for studies of President Roosevelt's New Deal opment. Expenditures were very large and con- methodological quality of evaluation studies
programs (see Exhibit 1-BJ. It was also during sequently were accompanied by demands for (Bernstein and Freeman, 1975), and discus-
this period that social experimentation "knowledge of results." sions of the organizational and structural con-
Evaluation Research as a By the end of the 1950s, program evalu-
emerged in psychology. Lewin's pioneering "ac- straints on evaluation research (Rieclcen and
Social Science Activity tion research" studies and Lippitt and White's ation research was commonplace. Social sci- Boruch, 197 4 J. The journal Evaluation Review
Commitment to the systematic evaluation work on democratic and authoritarian leader- entists were engaged in assessments of delin- was established in 197 6 and continues to be
of social programs first became commonplace ship, for example, were widely influential quency prevention programs, psychothera- widely read by evaluators. Other joumals fol-
in education and public health. Prior to World evaluative studies. The famous Western Elec- peutic and psychopharmacological treatments, lowed in rapid succession, and today there are
War I, the most significant efforts were directed tric experiments on worker productivity that public housing programs, educational activi- about a dozen devoted primarily to evaluation.
at assessing literacy and occupational training contributed the term Hawthome etfect to the ties, community organization initiatives, and During this period, special sessions on evalu-
programs and public health initiatives to re- social science lexicon date from this time as numerous other such areas. Studies were un- ation studies at the meetings of academic and
duce mortality and morbidity from infectious well. (See Bernstein and Freeman, 1975, for a dertalcen not only in the United States, Europe, practitioner groups became commonplace, and
diseases. By the 1930s, social scientists in vari- more extended discussion and Bulme~ 1982, and other industrialized countries but also in professional associations specilically for evalu-
ous disciplines were advocating the use of rig- Cronbach et ai., 1980, and Madaus and Stuf- less developed nations. lncreasingly, programs ation researchers were founded (see Exhibit 1-C
orous research methods to assess social pro- flebeam, 1989, for somewhat different histori- for farnily planning in Asia, nutrition and for a listing of the major jouroals and profes-
grams, and systematic evaluations became cal perspectives.J health care in Latin America, and agricultura! sional organizationsJ. By 1980, Cronbach and
more frequent (Freeman, 1977). ln sociology, From such beginnings, applied social re- and community development in Africa in- his associates were able to state, "Evaluation
for instance, Dodd's study of attempts to intra- search grew at an accelerating pace, with a cluded evaluation components (Freeman, has become the liveliest frontier of American
12 EVALUAT!ON
Programs, Policies, and Evaluations 13

ment of measurement and survey research pro- to better the human condition. ln thls regard,
cedures, and the development of electronic evaluation research must be seen as an integral
computers that made it possible to analyze part of the social policy and public adminis-
large numbers of variables by means of multi- tration movements.
Journals devoted primarily to program and policy eva[uation: variate statistics. The computer revolution was
11 Evafuation Review: A Journal of Applied Social Research (Sage Publications) an especially important stirnulus to the growth
Social Policy and Public
11 Evafuation Practice, renamed (1998) American ]ournaf oi Evafuation (JAI Press) of evaluation research (Nagel, 1986) and lias
New Directions for Evafuation (Jossey-Bass) facilitated not only data analysis but data col- Administration Movements
11 Eva/uation: The lnternational Journal ofTheory, Research, and Practice (Sage Publications Ltd.) lection as well (Gray, 1988). The dose relation- A full treatment of the development of the
11 Eva/uation and Program Planning (Pergamon) shlp between technological changes and tech- overlapping social policy and public adminis-
11 ]ourna/ of Policy Analysis and Management (John Wiley) nical developments in applied social research tration movements would require tracing the
a Canadian ]ournal of Program Evaluation (University of Calgary Press) continues today.
11 Evafuation ]ournal of Australasia (Australasian Evaluation Society) remarkable growth of population and industri-
But hlstory can obscure as well as illurni- alization in the United States during the first
11 Evaluation & the Health Professions (Sage Publications)
a Educationa/ Evafuation and Policy Anafysis (American Educational Research Association) nate. Although there is continuity in the devel- part of thls century. During thls period, chang-
a Assessment and Eva/uation in Higher Education (Carfax Publishing Ltd.) opment of the evaluation field, a qualitative ing social values resulted in a shlft of respon-
change occurred as it matured. ln its early sibility for co=unity welfare from family
Professional organizations for program and policy eva[uators: years, evaluation was an endeavor shaped members, church charities, and private philan-
mainly by the interests of social researchers. ln thropy to gove=ent agencies. At least a few
a American Evaluation Association (Web page: http://www.eval.org/)
!ater stages, howevei; the consumers of evalu- hlghlights are important here.
Association for Public Policy Analysis and Management
ation research have had a significant influence
(Web page: http://qsilver.queensu.ca/appam/)
a Canadian Evaluation Association
on the field. Evaluation is now sustained pri- The Emergence of
(Web page: http://www.unites.uqam.ca/ces/ces-sce.html) marily by policymakers, program planners, and Govemment Programs
Australasian Evaluation Society administrators who use the findings and be-
(Web page: http://www.parklane.eom.au/aes/) lieve in the worth of the evaluation enterprise. Social programs and the evaluation activi-
European Evaluation Society It is also supported by the interests of the ties that accompanied them emerged from the
(Web page: http://www.europeanevaluation.org) general public and the clients of the programs relatively recent transfer of responsibility for
U K Evaluation Society evaluated. Evaluations may not make front- the nation's social and environmental condi-
(Web page: http://www.evaluation.org.uk) page headlines, but their findings are often tions, and the quality of life of its citizens, to
11 German Evaluation Society matters of intense concern to informed citizens gove=ental bodies. AB Bremner (1956) lias
(Web page: http://www.fal.de/tissen/geproval.htm)
and those w hose lives are affected, directly or described, before World War I, except for war
ltalian Evaluation Society
indirectly, by the programs at issue. Over the veterans, the provision of human services was
(Web page: http://www.valutazione.it/)
years, these various consumers and sponsors of seen primarily as the obligation of individuais
evaluation have played an increasingly large and voluntary associations. Poor people, physi-
role in defining the nature of the field. cally and mentally disabled persons, and trou-
and statistics applicable to the study of social lncorporation of the consumer perspective bled families were the clients of local charities
social science" (pp. 12-13). Although the period
problems, social processes, and interpersonal into evaluation research lias moved the field staffed mainly by volunteers drawn from the
of rapid growth is ovei; evaluation continues to
relations. Conversely, the need for sophlsti- beyond tl1e study of social programs by applied ranks of the more fortuna te. Our image of these
be an important specialty area within the social
ca~ed methods for evaluating social programs social researchers. It lias also become a political volunteers as wealthy matrons toting baskets
sciences and is widely supported by public and
stirnulated methodological innovation. ln par- and managerial activity, an input into the com- of food and hand-me-down clothing to give to
private agencies.
ticular; two essential inputs contributed to the plex mosaic from whlch emerge policy deci- the poor and unfortunate is only somewhat
The development of the field of evaluation
evolution of the field: improvements in system- sions and resources for the planning, design, exaggerated. Along with civic associations and
in the postwar years was made possible to a
atic data collection brought about by the refine- implementation, and continuance of programs locally supported charity hospitais, county and
large extent by advances in research methods
14 EVALUATION Programs, Policies, and Evaluations 15

state asylums, locally supported public schools, Ali this began to change in the 1930s. vinced that concepts, techniques, and princi- ganizations. ln particul~ these training pro-
state normal schools, and sectarian old-age Human services grew at a rapid pace with the pies from economics, political science, and so- grams stress evaluation research, both as an
homes, volunteers were the bulwark of our advent of the Great Depression, and, of course, ciology could be useful. The study of the public assessment procedure and as a body of knowl-
huma.n service "system." so did gove=ent in general, especially during sector thus grew into the largely applied re- edge about the effectiveness of programs and
lndeed, gove=ent was comparatively the period surrounding World War II. ln part search specialty that is now most co=only practice in the sundry policy areas.
small before the l 930s, particularly the federal because of the unwieldiness that accompanied called 11policy science" ar "policy a.nalysis. 11 The importance of evaluation is now ac-
gove=ent. There were few national health, this accelerated growth, there was strong pres- Moreove~ as the federal gove=ent be- knowledged by those in political as well as
education, and welfare prograrns and no need sure to apply the concepts and techniques of came increasingly complex and technical, its executive roles. For exarople, the GAO, Con-
for an army of federal employees. The idea of so-called scientific management, which were programs could no longer be adequately man- gress's "watchdog, 11 made a major commitment
annual federal expenditures of billions of dol- well regarded in industry, to gove=ent pro- aged by persons hired as intelligent generalists to evaluation in 1980 in response to congres-
lars for health research, for instance, would grams and activities. These ideas first tookhold ar because of their connections with political sional interest in the assessment of govem-
have completely bewildered the gove=ent in the Department of Defense and then dif- patrons, relatives, ar friends. Most midlevel ment initiatives. ln addition, many federal
official of the 1920s. The notion of more bil- fused to other gove=ent organizations, in- management jobs and many senior executive agencies have their own evaluation units, as do
lions going to purchase medical care for the cluding human service agencies. Concepts and positions required specific substantive and a large number of their state counterparts. Even
aged ar for poor persons would have been even procedures for planning, budgeting, quality technical skills, and those who filled them more co=onplace at federal, state, and local
more mind-boggling. Federal fiscal support of contrai, and accountability, as well as late~ needed either training ar extensive experience leveis are procedures for commissioning pro-
public education was infinitesimal-more dol- more sophisticated notions of cost-benefit to do theirwork competently (see Exhibit 1-DJ. gra.m_ evaluation, as the need arises, on a con-
lars for public education currently flow from analysis and system modeling, became the ar- The state and local counterparts of federal tract basis from university researchers ar re-
Washington in six months than were spent in der of the day in the human resource area. agencies expanded ata similar rate, stimulated search firms and consultants.
the entire first decade of this century. Moreove~ in part by federal initiatives and funding, and ln short, although evaluation research con-
the scope and use of social and economic infor- The Development of Policy and they too required skilled staffs. ln response, tinues to have an academic side oriented to-
mation mirrored the sparseness of gove=ent Public Administration Specialists university social science departments mobi- ward training, methodology, theory, and rela-
program operations. Even in the late 1930s, lized to provide trained persons for gove=ent tively detached study of the nature and effects
federal expenditures for social science research During this sarne period, persons with so- positions as well as training researchers. of social programs, it is a field that now extends
and statistics were only $40-$50 million, as cial science training began to apply themselves Graduate schools of management, public well beyond university social science depart-
compared to 40 to 50 times that arnount today to understanding the political, organizational, health, and social work began programs to meet ments. Evaluation is generally practiced in a
(Lynn, 1980). and administrative decision making that took the need for executives and technicians, and context of policy making, program manage-
Finally, human. services and gove=ent place in executive departments and other gov- special schools, generally with "public admin- ment, and client ar consumer advocacy. Thus,
operated under different norms than today. Key emm_ental agencies. Also, economists were istration" in their titles, were established ar not only is its history intertwined with the
gove=ent officials usually were selected perfecting models for planning and refining expanded. social policy and public administration move-
without regard to objective competence crite- theories of macroeconomic social processes ln short, a new army of professionals ments, but its practice typically occurs in the
ria; indeed, there were few ways of objectively (Stokey and Zeckhause~ 1978). ln part, the emerged. Furthermore, the institutionalization sarne political and organizational arenas as
determining competence. The professional interests of social scientists in gove=ent at of policy analysis and public administration policy analysis and public administration.
civil service was a fraction of the size it is today, this time were purely academic. They wanted programs in universities has maintained the
most jobs did not require technical know-how, to know how gove=ent worked. Howeve~ momentum of the intertwined policy science
and formal training prograrns were not widely persons in leadership positions in gove=en- and public administration movements. Con-
available. Moreove~ because its activities and tal agencies, groping forways to deal with their cepts and methods from the social sciences The Great Society and Its
influence were comparatively small, there was large staffs and fulI coffers of funds, recognized have become the core of the educational pro- Aftermath: Political Ideology and
relatively little interest in what went on in a criticai need for orderly, explicit ways to han- grams from which are drawn many of our the Evaluation Enterprise
gove=ent, at least in terms of human service dle their policy, administrative, program, and public officials and program managers as well
programs. planning responsibilities. They became con- as the staffs of foundations, public and private Evaluation activities increased rapidly dur-
human service agencies, and intemational ar- ing the Kennedy and Johnson eras when social
16 EVALUATION Programs, Policies, and Evaluations 17

lhe steady growth in the number, variety, amounts of research, analysis, and data gathering lhe year 1965 was an important one in the The War on Poverty-Great Society devel-
complexity, and social importance of policy are being dane. evolution of ''policy analysis and evaluation opments initiated in 1965 represented a set of
issues confronting government is making Because the power to influence policy is research" as an independent branch of study. social interventions on an unprecedented scale.
increasing intellectual demands on public widely shared in our system of government, Two developments at the federal government Ali impacted by them wanted to know if they
officials and their staffs. What should be dane public officials seeking to influence policy-to level-the War on Poverty-Great Society initia- were working, and who was being affected by
about nuclear safety, teenage pregnancies, urban play the policy game well-must be persuasive. tive and the Executive Order establishing the them and how. Those with the skills to answer
decline, rising hospital costs, unemployment Because of the changing character of policy Planning-Programming-Budgeting (PPB) system- these questions found both financial support
among black youth, violence toward spouses and issues, it is probably harder to be persuasive than were of signal importance in this regard. Both and an interested audience for their efforts. And
children, and the disposal of toxic wastesl Many it used to be. Seniority, affability, and dever offered standing, /egitimacy, and financial sup- the social science community responded. lhe
of these subjects were not on the public agenda "wheeling and dealing
11
may be relatively less port to scholars who would tum their skills and sarne year saw government-wide adoption of the
20 years ago. They are priority issues now, and influential than being generally knowledgeable interests toward examining the efficiency with formal evaluation and analysis methods that had
new ones of a similar character emerge virtually and tough-minded, having the ability to offer which public measures allocate resources, their earlier been applied in Robert McNamara's De-
every year. For most elected and appointed ideas and solutions that can attract a wide impacts on individual behavior, their effective- fense Department in the Planning-Programming-
officials and their staffs, such complicated and following, ar having a reputation as a well- ness in attaining the objectives for which they Budgeting system. A presidential Executive Or-
controversial questions are outside the scope of informed critic. lncreasingly, officials from the were designed, and their effects on the well- der gave employment and financial support to
their judgment and previous experience. Yet the president on down \ase influence in policy being of rich versus poor, minority versus ma- thousands who wished to apply their analytical
questions cannot be sidestepped; government debates when they cannot gel their numbers jority, and North versus South. skills to such efficiency, effectiveness, and equity
executives are expected to deal with them right ar when their ideas and arguments are questions.
responsibly and effectively. successfully challenged by opposing experts.
To aid them in thinking about and deciding on lndeed, thorough and detailed command of an
such matters, public officials have been depend- issue ar problem is often mandatory. Legislatures SOURCE: Robert H. Haveman, "Policy Analysis and Evaluation Research After Twenty Years," Policy Sfudies Journal,
1987, 16:191-218.
ing to an increasing extent on knowledge derived are requiring executives to be experts in the
from research, policy analysis, program evalua- programs and issues under their jurisdiction.
tions, and statistics to inform ar buttress their Judges are requiring detailed proof that
views. More often than in the past, elected and administrative decisions are not arbitrary and programs undertaken under the rubrics of the 19 84). But more realistic expectations for social
appointed officials in the various branches and capricious. Budget officials demand positive War on Poverty and the Great Society provided programs only increase the importance of un-
leveis of governmEnt, from federal judges to town program evaluations. The public demands extensive resources to deal with unemploy- dertaking evaluation before putting prograrns
selectmen, are citing studies, official data, and accountability. lhus the dynamic processes ment, crime, urban deterioration, access to into place on a permanent and widespread basis
expert opinion in at least partia! justification for whereby our poltica] system confronts social
medicalcare, andmentalhealth treatment {see ar malcing signilicant modifications to them.
their actions. lheir staffs, which have been in- problems are perceptibly, if not dramatically,
Exhibit 1-E). These programs were often hur- Partly as a consequence of the apparent
creasing in size and responsibility in recent raising the standards of substantive and
riedly put into place, and at least a signilicant ineffectiveness of many initiatives, the decade
decades, include growing numbers of people managerial competence in the performance of
trained in ar familiar with analytic techniques to public responsibilities. portion were poorly conceived, improperly im- of the 1970s was marked by increasing resis-
gather and evaluate information. lncreasing plemented, and ineffectively administered. tance to the continued expansion of govem-
Findings of limited effectiveness and poor bene- ment prograrns {Freeman and Solomon, 1979).
fit-to-cost ratios for the large-scale federal ini- The reaction was most clear in referenda such
SOURCE: Adapted, with permission, from Laurence E. Lynn, Jr., Designing Public Policy {Santa Manica, CA: Scolt, tiatives of this era prompted widespread reap- as California's Proposition 13, which limited
Foresman, 1980). praisal of the magnitude of effects that can be real estate tax revenue, and in "sunset laws, 11
expected from social programs. Social interven- which required the automatic shutdown of in-
tion all too often yields small gains, much to effective programs {Adams and Sherman,
the chagrin of thosewho advocate them {Weick, 1978). Of course, some of the attacks on big
Programs, Policies, and Evaiuations 19
18 EVALUATION

phasis on institutionalization to efforts to in- program arena (Freeman, 1983). Rather, these
government were simply political campaign As should be apparent, social programs crease tolerance for street people and malce different political conditions raise different sets
rhetoric. And although a number of states and and, consequently, the evaluation enterprise heavier commitrnents of resources for their of evaluation questions corresponding to the
major cities enacted sunset laws, in only a few are shaped by the changing times. Political medical care, shelter, food, and other necessi- shifts in the concems emphasized by the stalce-
instances have programs actually been shut perspectives during the 198Ds, not only in the ties. Enumerations of the homeless, diagnoses holders. Regardless of political outlook, two
down. More often, only superficial and sym- United States but also in a number of Westem of their conditions, and demonstration pro- matters are clear about the 1990s. First, re-
bolic assessmen ts were undertalcen ar the pro- European countries, have brought about in- grams with accompanying evaluations are nu- straints on resources will continue to require
grams were given extensions to allow them to creased concem with tl1e balance of benefits merous and expanding, despite the budgetary choosing the social problem areas on which to
develop documentation for their effectiveness. and costs for social programs, even in social shortfalls at all levels of government. concentrate and the programs that should be
Nevertheless, it is clear that the rise of problem areas that receive generous funding. given priority. Second, intensive scrutiny of
fiscal conservatism in the 197Ds resulted in a On the intellectual front, both conservative existiog programs will continue because of the
decline in governmental support for new social and liberal critique of the Great Society pro-
The Evaluation Field
pressure to curtail ar dismantle those that do
programs and, to some extent, private support grams have had an impact on the evaluation in the 1990s not demonstrate that their services are effective
as well. This, in tum, brought about a change field. Although these criticisms were some- Fiscal conservatism, the devolution of re- and efficient. Moreover, both dissatisfaction
in emphasis in the evaluation field. ln particu- times based more on ideology than evidence, sponsibility to the states, and skepticism about with existing programs and shifts in political
lru:, increased attention has been given to as- tl1ey nevertheless have drawn on evaluation social programs dominate national policy malc- currents will result in new and modified pro-
sessing the expenditures of social programs in results in condemning social programs. For ing today. The Clinton Democratic presidency grams that come forward with promises of
compririson to their benefits and to dem- instance, evaluation research has been used to and the Republican majority Congress seem being more effective and less costly. Ali these
onstrating fiscal accountability and effective argue that tl1e major federal welfare program, detemiined to cut federal spending and hand circumstances will generate a need, and quite
management. ln the process, many fiscal and Aid to Faniilies With Dependent Children over major social programs to the adrninis- likely a demand, for evaluation research.
political conservatives, often skeptical about (AFDC), provides perverse incentives that in- tration of the states. For instance, welfare re- Major worldwide changes will also affect
social science, have joined the advocates of crease the social problem it was intended to forrn legislation, passed in 1996 (Personal the evaluation field. The globalization of eco-
social action programs in pressing for the infor- ameliorate (Murray, 1984). The evaluation Responsibility and Work Opportunity Recon- nomic activities may force nations with. gener-
mation that evaluations provide. field has thus been thrust into the middle of ciliation Act), that abolished the entitlement ous social welfare programs to cut baclc their
ln the 1980s, during both the Reagan and contentious debates about the very concept of status of AFDC required the states to adrninis- expenditures to remain competitive on world
Bush administrations, domestic federal expen- social intervention and faced with new chal- ter it under severe time eligibility restrictions markets. Nations emerging from totalitarian
ditures were curtailed in an attempt to contrai lenges to demonstrate that any major program and imposed an emphasis on moving benefici- socialist regimes, on the other hand, may have
inflation and reduce the federal deficit. Many initiative can be effective. aries into employment. Such changes have to launch new social initiatives. lndeed, in
of the largest cutbacks were targeted on social Meanwhile, new social problems are con- mixed implications for evaluation. On the one many developing nations throughout the world
programs. A similar posture was manifest in tinually emerging on the political landscape, hand, these major revisions and reforrns in there is intense pressure to develop and evalu-
many states and cities; indeed, some of the accompanied by demands that they receive social programs require evaluations if anything ate social programs virtually overnight. Af the
local and state reactions to their deteriorating progranimatic attention and that the efforts is to be leamed about their fiscal and social sarne time, evaluation itself is becoming in-
economic situations were particularly severe. made to ameliorate them be evaluated. A strik- impacts. On the other hand, much of the re- creasingly intemational (see Exhibit 1-FJ.
These developments were partly a consequence ing example is the issue of homelessness sponsibility for conducting evaluation has de- Perhaps more subtle, but at least as great a
of the distrust, hostility, and political actions of (Jencks, 1994; Rossi, 1989). At the time the volved to the states along with the programs, source of influence on programs and their
community members dismayed with the pain- first edition of this text was published (1979), and many states do not have the capabilities ar evaluations, are shifts in the values and self-in-
ful bite of incarne and property taxes. As we there was little public notice of the homeless, the will to undertalce the rigorous evaluation terests of community members and organiza-
have indicated, however, they were also influ- little political activity to initiate and fund pub- needed. tions. Surveys in the United States anda num-
enced by disenchantrnent with the modest ef- lic prograrns to better their lot, and, conse- Fundamentally, however, whether there is ber of Westem European countries, for
fects and poor implementation of many of the quently, little effort to estimate the number of a "liberal" ar 11conservative 11 outlookin govem- instance, documenta reduced emphasis among
programs ruost ardently championed by public sucl1 persons, their characteristics, ar the rea- ment and among the public should not change workers on earnings and an increased value
officials, planners, and politicians in the past sons for their condition. Tuday, views on how the role of evaluation research in the social placed on nonwork activities. As another illus-
severa! decades. to deal with the homeless range from an em-
20 EVALUATJON Programs, Policies 1 and Evaluations 21

Evaluation is the process of determining the does not just test products and report the test
Evaluation is becoming increasingly inter- in one part of the globe frequently affect people,
merit, worth, and value of things, and evaluations scores; it (i) rates or ranks by (ii) merit or
national, but in ways that go beyond previous institutions, and programs all around the
are the products of that process .... Evaluation cost-eflectiveness. To get to that kind of
conceptions of what international means. world. ... By transnariona/, we mean that the
is not the mere accumulation and summarizing conclusion requires an input of something
lnternational is no longer used only to describe problems and programs that we are called upon
of data that are clearly relevant for decision besides data, in the usual sense of that term. lhe
the efforts of particular evaluators in individual to evaluate today often extend beyond the
making, although there are still evaluation second element is required to get to conc/usions
countries around the world-although it certainly boundaries of any one nation, any one continent,
theorists who take that to be its definition .... ln about merit ar net benefits, and it consists of
is still used in this way. ... Today, evaluation is ar even one hemisphere . . . . These include
ali contexts, gathering and analyzing the data evaluative premises ar standards . ... A more
also becoming international in the sense of being problems of pollution, of the economics of
that are needed for decision making-difficult straightforward approach is just to say that
at the sarne time more indigenous, more global, developing countries, and of the role of women
though that often is-comprises only one of the evaluation has two arms, only one of which is
and more transnational. By indigenousr we mean in society. We cannot say exactly what the best
two key components in evaluation; absent the engaged in data-gathering. The other arm
that eva[uators in different countries around the responses to these internationalizing develop-
other component, and absent a procedure for collects, da ri fies, and verifies relevant values and
world are developing their own infrastructures to ments will be for evaluators, but we do know that
combining them, we simply Jack anything that standards.
support their endeavors as welt as their own recognizing the developments is the first step
qua/ifies as an evaluation. Consumer Reports
preferred theoretical and methodological ap- toward accommodating to them.
proaches. By global, we mean that developments
SOURCE: Quoted, wilh permission 1 from Michael Scriven, Evaluatian Thesaurus, 4th ed. (Newbury Park, CA: Sage,
1991), pp. 1, 4-5.
SOURCE: Quoted, with permission 1 from Eleanor Chelimsky and William R. Shadish, Evafuation for the 21st Century: A
Handbook (Thousand Oaks, CA: Sage, 1997), pp. xi-xii.

formance is judged (see Exhibit 1-G J. It follows credit for its successes, or overlook shortcom-
tration, until this decade most large corpora- than the prel.im.inary version offered earlier: that a central task of the program evaluator is ings for which it should be accountable. A
tions opposed publicly funded national health Program evaluation is the use of social research to construct a valid description of those areas distinct and precise description of program per-
insurance or govenunental health subsidy pro- procedures to systematically investigate the ef- of program performance that are at issue in a formance is one that is sulliciently definite and
grams for employed persons. But the extremely fectiveness of socialintervention programs that form that permits incisive comparison with the discrinllnating for meaningful variations in
high cost of medical care and its impact on is adapted to their political and organizational applicable criteria. This task presents several levei of performance to be detected. An evalu-
worker incarnes has led to a decided change in environments and designed to inform social challenges, some involving identification of the ation that produces an undulyvague or equivo-
outlook. 11ends in values and self-interests action in ways that improve social conditions. areas of performance at issue and the applicable cal description of program performance may
quite likely will have much to do with the Elaborating on the various distinct compo- criteria that we will address !ater Here we focus also fall short by making it impossible to deter-
nature and scope of the social programs that nents of this definition, in tum, will provide an on the problem of constructing a valid descrip- mine with confidence whether program per-
are initiated and those that are continued, with introductory overview of the nature of program tion of performance that is sulliciently distinct formance actually meets some appropriate
corresponding implications for their assess- evaluation as presented in this book. and precise to permit meaningful assessment. standard.
ment. A valid description of program perfor- Social research procedures and the accom-
Application of Social mance is one that accurately represents what panying standards of methodological quality
AN OVERVIEW OF the program actually accomplishes. As should have been developed and refined over the years
PROGRAM EVALUATION Research Procedures be obvious, it is a serious defect for an evalu- explicitly for the purpose of constructing sound
The concept of evaluation entails, on the ation to fail to describe program performance factual descriptions of social phenomena. ln
With the benefit of some historical context, we one hand, a description of the performance of with a reasonable degree of validity. Doing so particulm; contemporary social science tech-
can attempt a more complete definition of the entity being evaluated and, on tl1e othei; is a misrepresentation of the facts that may niques of systematic observation, measure-
programevaluation, as we wish to use the term, some standards or criteria by which that per- distort a program's accomplishments, deny it ment, sampling, research design, and data
22 EVALUATION Programs, Policies 1 and Evaluations 23

analysis represent rather highly evolved proce- productive contributors to furthering methodo- evaluations that cio so are termed comprehen- sector, for instance, but the applicable criteria
dures for producing valid, reliable, and precise logical development in applied social research. sive evaluations. for assessing performance will generally have
characterizations of social behavim: Because Finally, ourview does not imply that meth- Evaluation methods can be applied to more to do with "the bottom line" of profits and
social prograrns are instances of organized so- odological quality is necessarily the most im- many kinds of prograrns, projects, and endeav- productivity than with arnelioration of social
cial behavior, we talce it as virtually self-evident portant aspect of an evaluation nor that only ors, but the domain of prograrn evaluation problems. Sinillarly, evaluation methods could
that social research procedures offer the best tl1e highest technical standards, without com- orients chiefly to social prograrns and the focus be applied to assess social clubs, professional
approach to the task of describing prograrn promise, are appropriate for evaluation. As of this book is primarily on that type of pro- organizations, and other such prograrns whose
performance in ways thatwill be as credible and Carol Weiss (1972) once observed, social pro- grarn. What we mean by a social prograrn in purposes are to provide certain benefits to their
defensible as possible. Moreover, credibility and grarns are inherently inhospitable environ- tls context is a plarmed, organized, and usu- members. The criteria for assessing these pro-
defensibility are important characteristics of ments for research purposes. The nature of ally ongoing set of activities carried out for the grarns would largely and appropriately relate
the evidence evaluators put forward, both be- prograrn circumstances, and of the particular purpose of improving some social condition. A only to the interests of the members. The goals
cause of the practical importance of evaluation issues the evaluator is called on to address, social prograrn thus is directed at arneliorating of other types of prograrns are generally quite
in most contexts of application and because of frequently necessitates compromises and adap- a social problem ar responding to a social need, different from those of social prograrns and the
the disputatious reception often given to evalu- tations of textbook methodological standards. usually through the provision of some form of criteria appropriate for assessing them will also
ation results when they do not conform to the The challenges to the evaluator, as we see human services. As we are using the term, be different.
expectations of signilicant stalceholders. them, are to match the research procedures to therefore, social prograrns are defined as enti- Our focus on the large and important topic
Regardless of the type of social intervention the evaluation questions and circumstances as ties whose principal reason for existing is to "do of evaluating social prograrns, therefore, carries
under study, therefore, we advocate evaluations well as possible and, whatever procedures are good," that is, to produce social benefits and with it a set of assumptions about the general
that employ social research procedures for used, to apply them at the highest possible improve social conditions. It follows that iliey value frarnework within which appropriate cri-
gathering evidence about prograrn performance standard feasible in those circumstances. are appropriately held accountable within an teria and standards for assessing the various
and analyzing and interpreting that evidence. evaluative frarnework on the basis of their con- aspects of those prograrns will be defined. ln
This commitment to the rules of social re- tribution to the social good. Most social pro- particular, when we describe evaluation as in-
search is at the core of our perspective on The Effectiveness of Social grarns will thus hold themselves accountable vestigating the effectiveness of social prograrns
evaluation and is what we mean by the subtitle for producing positive social effects, at least to we are assUillng that what effectiveness means
Intervention Programs the extent of recognizing the legitimacy of that for such prograrns relates ultimately to their
of tls book, A Systematic Approach. This is
not to say, however, that we believe evaluation expectation. ln addition, of course, many social contribution to improving social conditions. Of
Any prograrn evaluation worthy of the
studies must follow some particular social re- prograrns will be held accountable for such course, there may be arnbiguity and dispute
name must, of course, evaluate; t11at is, some
search style or combination of styles, whether results by those parties who invest in them, about just what contributions a prograrn
assessment must be made of one ar more as-
quantitative ar qualitative, experimental or sponsor them, administer them, ar are legally should be making and the implications for
pects of the prograrn. As indicated above, evalu-
etlmographic, "positivistic" ar "naturalistic." responsible for them, for instance, taxpayers, everyday prograrn operations, which an evalu-
ating something requires that pertinent dimen-
Indeed, one of the principal characteristics of funders, boards of directors, agency heads, and ator will have to resolve before appropriate
sions of its performance ar characteristics be
prograrn evaluation is that its methods cover legislators. criteria can be defined and an evaluation can be
described and then judged against appropriate
the garnut of prevailing social research para- The importance of tls issue is that it has conducted. These matters will be discussed in
standards ar criteria. Prograrn evaluation gen-
digms. Nor does tls commitment to the meth- criticai implications for the question of what greater detail in oilier chapters of tls book.
erally involves assessment of one ar more of
ods of social science mean that we think cur- criteria or standards should be used to assess Most of our discussion, advice, and illustra-
five prograrn domains: (a) the need for the pro-
rent methods are a finished piece of work prograrns when conducting a prograrn evalu- tions, however, assumes iliat it is social pro-
grarn, (b) the design of the prograrn, (c) tl1e
beyond improvement. Evaluators must often ation. Dillerent values frarneworks are appro- grarns that are being evaluated and that the
prograrn implementation and service delivery,
innovate and improvise as they attempt to find priate for different types of progr=. Many of foundation for judgments about how effective
(d) the prograrn impact or outcomes, and (e)
ways to gatl1er credible, defensible evidence the evaluation methods described in this book they are is some articulation of the social good
prograrn efficiency (cost-effectiveness). ln some
about social prograrns. ln fact, evaluators have can be applied to prograrns in the business they are expected to produce.
circumstances, an evaluation of a social prograrn
been, and will likely continue to be, especially may encompass ali iliese prograrn domains;

L
24 EVALUATION Programs, Policies, and Evaluations 25

Adapting Evaluation to the Political and AE important to tailor-making an evalu-


ation plan as the questions to be answered are
Organizational Context of the Program
the reasons why those questions are being
Program evaluation is not a cut-and-dried asked and the use that will be made of the
Evaluation is a rationa[ enterprise that takes issues as the problematic nature of some
activity like puttingup a prefabricated house ar answers. Social programs consist of, and exist
place in a political context. Political consider- programs and the unchallengeability of others,
checking a docurnent with a word processor's within, a swirl oi individual, organizational, ations intrude in three major ways, and the the legitimacy of program goals and program
spelling program. Rathe~ evaluation is a prac- and political decisions dealing with a range of evaluator who fails to recognize their presence strategies, the utility of strategies of incremental
tice in which the initial evaluation plan must issues from the trivia oi ordering paper clips to is in for a series of shocks and frustrations: reform, and even the appropriate role of the
be tailor-made to the particular program circum- threat of termination. ln such a context, an First, the policies and programs with which social scientist in policy and program formation.
stances and then typically requires revision and evaluation must deal with the issues that mat- evaluation deals are the creatures of political Knowing that political constraints and resis-
modilication during its implementation. The te~ provide information that addresses those decisions. They were proposed, defined, de- tances existis nota reason for abandoning eval-
specific form and scope of an evaluation depend issues, develop that information in a way that bated, enacted, and funded through political uation research; rather, it is a precondition for
primarily on its purposes and audience, the is timely and meaningful for the decisionmak- processes, and in implementation they remain usable evaluation research. Only when the
nature oi the program being evaluated, and the ers, and co=unicate it in a form that is usable subject to pressures-both supportive and evaluator has insight into the interests and mo-
political and administrative context within for their purposes. An evaluation might be hostile-that arise out of lhe play of politics. tivations of other actors in the system, into the
Second, because evaluation is undertaken in roles that he himself is consciously or inadver-
which the evaluation is conducted. designed quite differently if it is to provide
arder to feed into decision making, its reports tently playing, the obstacles and opportunities
The evaluation plan is generally organized information about the quality of service as
enter the poltica! arena. There evaluative that impinge upon the evaluative effort, and the
around the questions posed about the program feedback to the program director for purposes evidence of program outcomes has to compete limitations and possibilities for putting the results
by those who request and commission the of incremental prograrn improvement than if it for attention with other factors that carry weight of evaluation to work-onlywith sensitivity to the
evaluation (the evaluation sponsor) and other is to provide such information to an externai in the political process. politics of evaluation research-can the evalua-
pertinent stakeholders. These questions may funder who will use it to decide whether to Third, and perhaps least recognized, evalua- tor be as creative and strategically useful as he
be stipulated in very specific, fixed terms that renew the prograrn's funding. ln ali cases, how- tion itself has a political stance. By its ve.ry nature, should be.
allow little flexibility, as in a detailed contract eve~ itmust be sensitive to the political context it makes implicit political statements about such
for evaluation services, but typically the evalu- within which it is planned and conducted (see
ator must negotiate with the evaluation spon- Exhibit 1-HJ. SOURCE: Quoled, with permission, from Carol H. Weiss, "Where Politics and Evaluation Research Meet," Evalualion
Praclice, 1993, 14{1 ):94, where the original 1973 version was reprinted as one of the classics in the evaluation field.
sors and stalceholders to develop and refine the As a practical matte~ of course, an evalu-
questions. Although these parties presurnably ation must also be tailored to the organiza-
know their own interests and purposes, they tional malceup of the prograrn. The availability
will not necessarily formulate their concems in of adrninistrative cooperation and support; the
ways that the evaluator can use to structure an ways in which program files and data are kept Moreove~ adaptations may be required in the fined by the program itself ar the expectations
evaluation plan. For instance, the initial ques- and access permitted to them; the character of basic questions being addressed in response to of the public that supports it, is to intervene in
tions may be vague, overly general, ar phrased the services provided; the nature, frequency, shifts that occur in the operation of the pro- social conditions in ways that make them bet-
in prograrn jargon that must be translated for duration, and location oi the contact between gram ar the composition and interests oi the ter And if the purpose of these programs is in
more general consurnption. Occasionally, the program and client; and nurnerous other such stakeholders. some way to improve the hurnan condition, the
evaluation questions put forward are essen- matters must be taken into consideration in . purpose of evaluation, in tum, is to improve the
tially pro forma (e.g., is the program effectivel J the evaluation design. ln addition, once an prograrns.
and have not emerged from careful reflection evaluation is launched, it is co=on for
Informing Social Action to
ln particul~ the role of prograrn evaluation
regarding the relevant is sues. ln such cases, the changes and "in-flight" corrections to be re- Improve Social Conditions
is to provide answers. It answers questions
evaluator must probe thoroughly to determine quired. Modilications, perhaps even compro- As indicated, this book is about the evalu- about what the program is doing but, more
what this means to the evaluation sponsor and mises, may be necessary in the types, quantity, a tion of social prograrns ar, more generally, important, about how well it is being dane and
prograrn stalceholders and why they are con- ar quality of the data collected as a result of those prograrns whose mission, whether de- whether it is worth doing. It is undertaken on
cemed. unanticipated practical ar political obstacles.
26 EVALUATION Programs, Policies, and Evaluations 27

the assumption that there is an audience with Programs, like people, have their unique evaluation-related activity is the integration of in any signilicant way. Responsible evaluators
such questions and an interest in the answers. profiles of characteristics but also share charac- the findings of multiple evaluations of a par- try to avoid being drawn into such situations of
The concept of program evaluation presup- teristics with others that make for meaningful ticular type of prograrn into a synthesis that can "ritualistic" evaluation. An early step in plan-
poses more than a merely interested audience, categories and groupings. What is learned from inform policy making and program planning. ning an evaluation, therefore, is a thorough
however It is characteristically designed to pro- an evaluation about one specilic prograrn, say, Some evaluation researchers, therefore, have inquiry into the motivation of the evaluation
duce answers that will be useful and will actu- a druguse prevention program irnplemented at been involved in the activities of systematic sponsors, the intended purposes of the evalu-
ally be used. An evaluation study, therefore, a particular high school, also tells us something research synthesis ar meta-analysis (Lipsey and ation, and the uses to be made of the findings.
primarily addresses the audience 1ar, more ac- about the whole category of similar programs. Wilson, 1993).
curately, audiencesJ with the potential to make Many of the parties involved with social inter- Evaluations can thus inform social action
decisions and talce action on the basis of the vention must make decisions and talce action by providing useful feedback for management
evaluation results. This point is fundamental that relates to categories ar types of programs and adrninistrative purposes; by supportingthe EVALUATION RESEARCH
to evaluation-its purpose is to inform social rather than individual instances. Policymakers, oversight functions of those funders, sponsors, IN PRACTICE
action. program planners, and program sponsars and and authorities to which the prograrn is ac-
In most instances, the main audiences to funders, for instance, must often select, pro- countable; ar by accurately depicting prograrn
which an evaluation is directed are the spon- mote, ar support a particular type of program activities and accomplishments to advocates, We have outlined the general considerations,
sors of the evaluation and other program stake- rather than any one instance. A federal legisla- adversaries, clients, and other stakeholders. purposes, and approaches that shape evalu-
holders. These are the individuals ar groups tive committee may deliberate the merits of Theymay also contribute information for plan- ation research and gulde its application to any
with rather immediate interests in the particu- compensatory education programs, ar a state ning and policy purposes, indica te if innovative program situation. In actual practice, applica-
lar program being evaluated and includes those correctional department may consider institut- approaches to co=unity problems are worth tion of these concepts typically involves some-
with decision-malcing authority over the pro- ing boot camps for juvenile offenders, ar a pursuing, ar demonstrate the utility of some thing of a balancing act between competing
gram ar the capability to influence such deci- philanthropic foundation may decide to pro- principle of professional practice. Evaluation forces. Paramount arnong these is the inherent
sion making. Evaluation findings may assist mote and underwrite programs that provide research may even help shape our general conflict between the requirements of system-
such persons to make go/no-go decisions about visiting nurses to single mothers. The body of understanding of how to bring about planned atic inquiry and data collection associated with
specific program modilications ar, perhaps, evaluation findings for programs of each of social change by testing social science hypothe- evaluation research and the organizational irn-
about initiation ar continuation of entire pro- these types is very pertinent to decisions and ses regarding the effects of certain broad forms peratives of a socialprogram devoted to delivery
grams. They may bear on political, practical, social actions of this sort. Each evaluation of intervention. The co=on denominator is of service and maintenance of essential routine
and resource considerations ar make an im- study, therefore, not only informs the immedi- that evaluation research is intended to be use- activities. The planning phase of evaluation,
pression on the views of individuals with influ- ate stakeholders but potentially informs those ful and used, either directly and immediately which is best acc;omplished in collaboration
ence. They may have direct effects on judg- whose situations require decisions and action ar as an incremental contribution to a cumula- with program personnel and stakeholders, and,
ments of a program's value as part of an about different program concepts. tive body of practical knowledge. especially, the data collection phase necessarily
oversight process that holds the program ac- Indeed, one irnportant form of evaluation These assertions, of course, assume that an place unusual and not altogether welcome de-
countable for results. Ortheymay have indirect research is that which is conducted on dem- evaluation would not be undertaken urtless mands on program personnel and program pro-
effects in shaping the way program issues are onstration programs, that is, social interven- there was an audience interested in receiving cesses. Data collection, for instance, may re-
framed and the nature of the debate about tion projects designed and irnplemented explic- and, at least potentially, using the findings. quire interaction with program files, clients,
them. The evaluation sponsar and other such itly to test the value of an innovative program Unfortunately, there are instances in which staff, and facilities that are disruptive of normal
decisionmakers and stalceholders have a rather concept. In such cases, the findings of the evaluations are commissioned without any in- program processes and distract from and, in
obvious primacy in these matters; however, evaluation are signilicant because of what they tention of using their findings. Evaluations some cases, even compromise the service func-
they are not the only audience potentially in- reveal about the program concept and are used may be conducted only because they are man- tions that are the program's primary obligation.
terested in the evaluation nor are they neces- primarily by those involved in policy making dated by program funders and then ignored Every evaluation plan, therefore, must ne-
sarily the only agents whose actions may be and prograrn development at levels broader when the findings are presented. Or an evalu- gotiate a middle way between optimizing the
influenced by the evaluation. than any one program. Another signilicant ation may be carried out because 11 everyone program circumstances for research purposes
does it" without expectation of using the results and minimizing the disruption caused to nar-
28 EVALUATION Programs, Policies, and Evaluations 29

mal program operation. We use the word nego- 1996 welfare reform legislation has drastically generally make little sense to design a rigorous experimental model, as he leamed and prac-
tiate quite deliberately here, because tbe best altered tbe nature of incarne support for poor impact assessment for a program under consid- ticed it in social psychology, to evaluation re-
approach to the inherent tension between tbe families. A program reconfiguration of tbis eration for significant revision by relevant de- search. Altbough he tempered his position in
requirements of research and tbose of running magnitude clearly requires evaluations of fam- cisionmakers. Of equal importance, howeve~ is some of his !ater writing, it is fair to charac-
a service program is for tbe evaluator to develop ily incarne support programs to be defined dif- tl1e flexibility tbe evaluator brings to tbe evalu- terize him as fitting evaluation research into
the evaluation plan collaboratively witb pro- ferently tban in tbe past witb new outcomes ation task. Knowiog tbe dynarnic nature of tbe scientific research paradigrn (see Exhibit
gram personneL If tbe needs and purposes of and quite different program components at programs, evaluators must be prepared to sub- 1-I).
tbe evaluation are spelled out in detail before issue. stantially modify an evaluation if it becomes Campbell's position was challenged by Lee
tbe research begins, and tbose program person- Priorities and responsibilities more specific apparent tbat tbe original plan is no longer Cronbach, anotber giant in tbe evaluation field.
nel who will be affected (not just tbe aclminis- to tbe organizations implementing a program appropriate to tbe circumstances. This often While acknowledging tbat scientific investiga-
trators) are given an opportunity to react, make can also change in significant ways. For exam- involves difficult issues associated witl1 tbe tion and evaluation may use some of tbe sarne
input, and otberwise help shape tbe data col- ple, a school system relieved by tbe courts from availability of resources for tbe evaluation, tbe logic of inquiry and research procedures, Cron-
lection plan, the result is usually a more work- forced school busing may lose interest in its time lines for producing results, and tbe rela- bach argued tbat tbe purpose of evaluation
able plan and better cooperation from program programs to increase white students' accep- tionships witb tbe program administrators and 9harply differentiates it from scientific research
personnel in tbe face of the inevitable strains tance of attendance in predominantly minority evaluation sponsors, so it is not to be talcen (Cronbach, 1982). In his view, evaluation is
tbe evaluation will place on tbem. schools. Or unanticipated problems witb tbe lightly. Social programs are not research labo- more art tban science and every evaluation
In addition to tbe conflict between evalu- intervention may require modifying tl1e pro- ratories, howeve~ and evaluators must expect should be tailored to meet tbe needs of pro-
ation and program functions, tbere are otber gram and, consequently, tbe evaluation plan as to be buffeted about by forces and events out- gram decisionmakers and stalceholders. Thus,
inherent tensions in tbe practice of evaluation well. For instance, a program to reduce tbe side tbeir contrai. whereas scientific studies strive principally to
tbat warrant co=ent. Here we introduce a absence rates of low-income high school stu- The contrast between tbe image of a re- meet research standards, evaluations should be
few of tbe more notable dile=as tbe evaluator dents by providing comprehensive medical care search laboratory and tbe reality of social pro- dedicated to providing maximally useful infor-
must confront: tbe incompatibility of a fixed might be tbwarted if a large proportion of tbe grams as places to conduct social research leads mation for decisionmalcers given tbe political
evaluation plan with tbe volatility of social eligible students refused tbe services. us directly to anotber of tbe inherent tensions circumstances, program constraints, and avail-
programs; tbe strain between a press for evalu- Somewhat ironically, preliminary findings in evaluation, tbat between a scientific and a able resources (see Exhibit 1-T).
ations to be scientific, on tbe one hand, and from tbe evaluation itself may stimulate pro- pragrnatic perspective on tbe process. One might be inclined to agree witb botb
pragrnatic, on tbe otber; and the competing gram changes tbat render tbe remainder of tbe tbese views-tbat evaluations should meet
approaches to evaluation offered up by a field evaluation plan obsolete. Conside~ for exam- high standards of scientific research quality and
of great diversity and little consensus. ple, a study of tbe impact of an alcohol treat- Scientific Versus Pragmatic be fully dedicated to serving tbe information
ment program tbat included six-montb and Evaluation Postures needs of program decisionmakers. The prob-
one-year follow-ups of tbe clients. When tbe lem, of course, is tbat in practice tbese two
six-montb follow-up revealed very high rates of Perhaps tbe single most influential article goals often are not especially compatible. In
Evaluation and the Volatility in tbe evaluation field was written by tbe late
drunkenness among tbe treatrnent group, tbe particular, social research at a high scientific
of Social Programs program staff markedly modified tbe interven- Donald Campbell and published in 1969. This standard generally requires resources tliat ex-
One of the most challenging aspects of tion. article outlined a perspective tbat Campbell ceed what is available in tbe typical program
program evaluation is tbe continually changing Not ali social programs, of course, trans- advanced over severa! decades: Policy and pro- evaluation context. These resources include
decision-malting milieu of tbe social programs form significantlywhile an evaluation is under gram decisions should emerge from continua! time, because high-quality research cannot be
that are evaluated. In particular, tbe resources, way. Nonetbeless, tbe evaluator must attempt social experimentation tbat tests ways to im- dane quickly whereas program decisions often
priorities, and relative influence of tbe various to anticipate such changes and prepare for prove social conditions. Not only did he hold have to be made on short notice, and funding
sponsors and stalceholders of social programs tbem to tbe extent possible. More important, tbis position in principie, but he contended proportionate to tbe expertise, levei of effort,
are dynamic. These changes are frequently as- perhaps, is to match tbe form of tbe evaluation tbat tbe technology of social research made it and materials required for research to scientific
sociated witb tbe shifts in political context and to tbe program circumstances and prospects at feasible to actually develop tbe "experimenting standards. Moreove~ research witbin tbe scien-
social trends we noted earlieL For example, tbe tbe time tbe evaluation is planned. It would society." Campbell, tbus, sought to extend tbe tific framework may require structuring tbe

L
30 EVALUATION Programs, Policies, and Evaluations 31

undocumented clairns about the way things somewhere in between. Occasionally, the pur-
are. But this statement, in tum, assumes that poses and priority users of an evaluation are
those conclusions meaningfully address as- defined so clearly and explicitly in advance that
pects ofthe situation of concem to the decision- the evaluator has relatively little difficulty in
The United States and other modem nations not these programs are effective, and in which makers; if not, they may be praiseworthy for balancing scientilic and pragrnatic consider-
should be ready for an experimental approach we retain, imitate, modify1 or discard them on their valiclity and creclibility, but still irrelevant. ations. For instance, an evaluation of a dem-
to social reform, an approach in which we try the basis of apparent effectiveness on the ln practice, therefore, the evaluator must onstration project on needle exchange to pre-
out new programs designed to cure specific multiple imperfect criteria available.
struggle to finda workable balance between the vent AIDS among drug adclicts funded by the
social problems, in which we learn whether or
emphasis to be placed on procedures that help National lnstitutes for Health may be clearly
ensure the valiclity of the evaluation finclings stipulated as a contribution to general policy-
SOURCE: Quoted from Donald Campbell, "Reforms as Experiments," American Psycha/ogis~ April 1969, 24:409.
and those that make the finclings tirnely, mean- relevant lmowledge about this intervention ap-
ingful, and useful to the consumers. Where proach that should meet the highest possible
that balance point should be will depend on the scientilic standards. On the other hand, an
purposes of the evaluation, the nature of the evaluator retained as a consultant by a program
program, and the political or decision-making administrator to provide an assessment of a
context. ln many cases, evaluations will justi- problematic unit in a co=un.ity mental
fiably be undertaken that are "good enough" for health center mayunderstand quite clearly that
An evaluative study of a social program is To enlighten, it must do more than amass good answering relevant policy and program ques- the finclings are for the sole purpose of inform-
justified to the extent that it facilitates the work data. Timely communications-generally not tions even though program conclitions or avail- ing certain decisions that administrator must
of the polity. lt therefore is to be judged primarily "final" ones-should distribute information to the able resources prevent them from being the make and, indeed, will not be reported outside
by its contribution to public thinking and to the persons rightfutly concerned, and those hearers best possible desigos from a scientilic stand- the organization.
quality of service provided subsequent to the should take the information into their thinking. point. For example, program sponsors con- Howeve~ many program evaluation situ-
evaluation .... An evaluation pays off to the To speak broadly, an evaluation ought to inform cemed about whether a rehabilitation treat- ations are not so clear-cut. Evaluation may be
extent that it offers ideas pertinent to pending and improve the operaons of the social system. ment for alcoholics is effective may find routinely required as part of funcling or contract
actions and people think more clearly as a result. six-month follow-up interviews showing that arrangements with the presumption that it will
few clients report heavy drinking to be very be generally informative to program managers,
SOURCE: Quoted from Lee J. Cronbach and Associates, Toward Refarm of Program fva/uarion (San Francisco:
useful even though the data Jack the experi- sponsors, and other interested parties. Or it
Jossey-Bass, 1980), pp. 65-G6.
mental contrais that permit this result to be may evolve from a collaboration between a
con.fidently attributed to the influence of the service agency with a need for information for
program. management purposes and a researcher with
inquiry in ways that do not mesh well with the but some contrived and constrained version of What further complicates an already diffi- broader interests in the type of intervention
perspectives of those who must make decisions uncertain relevance to the actual program. cult situation for evaluation planning is that that particular program provides. lndeed, given
about the program. For example, specifying On the other hand, one cannot blithely there is often ambiguity about the identity of the effort and expense required for evaluation,
variables so that they are well defined and clismiss scientilic concems in evaluation. Prop- the ultimate users of the evaluation and which there are probably more instances in which it
measurable under scientilic standards may erly understood, what the scientilic approach of the potential users should be given priority is expected to be multipurpose than cases
fragrnent and, in some regards, trivialize what represents is a very considered attempt to pro- in the desigo. An evaluation generally has vari- where the purpose and user are tightly speci-
the policymakers see as complex and dynamic duce conclusions that are valid and creclible. ous potential audiences, some with very imme- fied. Unfortunately, the trade-offs between util-
facets of the program. Similarly, meeting scien- Even when it falis short of this ideal, which is cliate interests in particular aspects of the pro- ity for program decisionmakers and scientilic
tilic standards for inferring causality, as when inevitable, such input makes a very irnportant gram under investigation, some with broader rigor are such that it is rarely possible to desigo
investigating program outcomes, may require contribution to a decision-making context that interests in the type of intervention the particu- an evaluation that serves both interests well.
such elaborate experimental contrais thatwhat otherwise is rife with seli-interested percep- lar program represents, and others falling Thus, if evaluators choose to emphasize the
is stuclied is no longer the program's services, tions and assertions, ideological biases, and

L
32 EVALUATION Programs, Policies, and Evaluations 33

needs of the immediate consumers, they must Diversity in Evaluation


be prepared for the possibility that the findings
Outlooks and Approaches
will be criticized on methodological grounds by
more remate audiences, for example, applied As the preceding discussion illustrates, the The ideal (never achievable) evaluation theory evaluative activities and the processes and goals
researchers, other evaluators, or sophisticated field of evaluation is not monolithic in concep- would describe and justify why certain evaluation they facilitate; and (e) empirically test propo-
policymalcers or service professionals. This car- tual outlook or methodological approach. On practices lead to particular kinds of results across sitions to identify and address those that conflict
ries with it the risk that the credibility of the the contrary, it is a contentiously diverse field. situations that evaluators confront. lt would (a) with research or other critically appraised knowl-
evaluation will be undermined, perhaps even The fundamental difference represented h.is- clarify the activities, processes, and goals of edge about evaluation.
in the eyes of the immediate consumers to torically by Carnpbell and Cronbach represents evaluation; (b) explicate relationships among
whom it was directed. But if the evaluation but one instance of this diversity. Evaluation
emphasizes methodological quality at the ex- practitioners are drawn .Eram a wide range of SOURCE: Quoted from William R. Shadish, Thomas D. Cook, and Laura C Leviton, Foundations of Program Evafuation:
Theories o( Practice (Newbury Park, CA: Sage, 1991 ), pp. 30-31.
pense of utility, it may satisfy those knowledge- academic disciplines and professions with dif-
able about research standards but be assailed ferent orientations and methods, and this mul-
byprograrn stalceholders as too academic, ivory tidisciplinary mixhas contributed significantly
tower, and even irrelevant to the 11real 11 program to the mult.iplicity of perspectives. Other differ-
issues. ences in outlook are related to the motivations scribes to the view that the field of evaluation .Eram a text can provide only part of the knowl-
Some evaluation theorists charnpion utili- of evaluators and the settings in which they has sufficiently clear boundaries to dist.ingu.ish edge needed to become a capable evaluatm:
zation as the overriding concem and advocate work. The solo practitioner who undertalces it conceptually .Eram what goes on generally in
evaluation that is designed around the specific short-term evaluat.ions on contract with local the policy sciences or procedurally .Eram the WHO CAN DO EVALUATIONS?
information needs of individually identified agencies and the tenured professor with long- "rules" that gu.ide applied social research.
target consumers with whom the evaluator term foundation funding will likely have quite There is probably as much diversity in outlook Systematic evaluation is grounded in social
collaborates very closely (e.g., Patton, 1997). divergent views on their evaluation activities. arnong evaluators about the utility of evalu- science research techniques; hence, most
The authors of review articles in applied re- As the field of evaluation has matured and ation theory as there is about the right way of evaluation specialists have had some social
search joumals who attempt to synthesize become institut.ionalized, interest has devel- doing evaluations. research training. But we should be qu.ick to
available research on the effectiveness of vari- oped in explicating the different postures to- At present, therefore, we must acknowl- point out that there is great heterogeneity in
ous interventions, on the other hand, regularly ward evaluation and the methods preferred by edge that evaluat.ion is at least as much art as the disciplinary and professional training of
deplore the poor methodological quality of leaders in various "carnps." ln particulai; there science, and perhaps should be and always will persons doing evaluations (see Exhibit 1-L).
evaluation studies and urge a higher standard. is a growiog interest in. ident.ifyiog congruent be. lnevitably, the evaluator's task is to crea- Ideally, every evaluator should be familiar with
Some co=entators want to have it both ways elements arnong different perspect.ives to ad- tively weave together many competing con- the full repertoire of social research methods.
and press the view that evaluations should vance what is referred to as 11 evaluation theory" cerns and objectives into a tapestry in which ln practice, we can come dose to this ideal only
strive to have utility to prograrn stalceholders (Shadish, Cook, andLeviton, 1991).Advocates different viewers can find different messages. by continually broadening and deepening our
and contribute to cumulative knowledge about of the evaluat.ion theory movement envision We recogn.ize, too, the difficulty of teaching an techn.ical knowledge by means ali of us know
socialintervention (Lipsey, 1997). Ouroutlook, the development of a theory that will serve as art form, especiallyvia the written word. Teach- about: keeping up with the literature, attending
for the didactic purposes of this book, is that ali the basis for decision makiilg by evaluators as ing evaluation is analogous to training physi- workshops and professional conferences, and
these options are defensible, but not necessar- they proceed with their work (see Exhibit 1-K). cians to be diagnost.icians. Any intelligent per- learn.ing .Eram colleagues. Moreover, it would be
ily equally defensible in any given evaluation Vutually ali experienced evaluators see the son can be taught to understand the results deceptive to suggest that this or any textbook
situation. Th.is, then, presents yet another is- need for better formulated gu.idelines as they .Eram laboratory tests, but a doctor becomes an can teach someone how to do evaluations.
sue for wh.ich the evaluator will be required to face the various decision points that come up astute diagnostician only through practice, ex- There is no substitute for experience. What we
malce a judgment cal! and must attempt to in any evaluation. Also, there is a need for such perience, and attention to the idiosyncrasies do bel.ieve is that this book will provide an
tailor the evaluation design to the particular gu.idelines so that training the fledgling evalu- of each individual case. ln this sense, learn.ing organized conceptual .Erarnework that identifies
purposes and circumstances presented in each ator is not so heavily dependent on trial-and-
application. error experience. But not every evaluator sub-

L
34 EVALUAT!ON Programs, Policies, and Evaluations 35

!11! Program evaluation is the use of social research methods to systematically investigate
Major Professional Responsibility Organizationa/ SeUing Primary Discipline the effectiveness of social intervention programs. It draws on the techniques and
Evaluation 28 College or university 40 Education 22 concepts of social science disciplines and is intended to be useful for improving
Research 19 Private business 12 Psychology 18 programs and informing social action aimed at ameliorating social problems.
Administration 18 Nonprofit organization 11 Evaluatlon 14
Teaching 13 Federal government agency 10 Statistical methods 10 !11! Modem evaluation research grew from pioneering efforts in the l 930s and burgeoned
Consulting 8 State/local government agency 10 Sodology 6 in the postwar years as new methodologies were developed that could be applied to
Student 5 School system 4 Economics and political science 6
Other 9 Other 13 Organizational development 3
the rapidly growing social program arena. The social policy and public administration
Other 21 movements have contributed to the professionalization of the field and to the
sophistication of the consumers of evaluation research.
SOURCE: Adnpted &om Eva]uation Practicc News !October l993)i based on 2 1045 AEA members !lS of June 1993.
!11! The need for program evaluation is uncliminished in the l 990s and may even be
expected to grow. lndeed, contemporary concem over the allocation of scarce re-
sources makes it more essential than ever to evaluate the effectiveness of social
interventions.
the important issues and the options for ad- ations are usually conducted by specialized
dressing them. evaluation staffs. At the other extreme, there !11! Evaluation must be tailored to the political and organizational context of the program
Although knowledge of the concepts and are many evaluation tasks that can be under- to be evaluated. It typically involves assessment of one or more of five program
methods instrumental to good program evalu- stood easily and carried out by persons of mod- domains: (a) the need for the program, (b) the design of the program, (c) the program
ation research is essential for conducting evalu- est expertise and experience. implementation and service delivery, (d) the program impact or outcomes, and (e)
ations, it is important to note that a great deal It is the purpose of this book to provide an program efficiency. Evaluation requires an accurate description of the program
of knowledge about the target problem area introduction to the field for those whose cur- performance or characteristics at issue and assessment of them against relevant
(e.g., crime, health, drug abuse) and the nature, rent positions, professional interests, ar natu- standards or criteria.
range, and results of the interventions that ral curiosity inspire them to want to leam how !11! ln practice, program evaluation presents many challenges to the evaluator. Program
have been used to address that problem are also evaluations are conducted. Studying the book circumstances and activities may change during the course of an evaluation, an
required. This is necessary not only so the is, of course, only a start along the path to appropriate balance must be found between scientific and pragrnatic considerations
evaluator will understand the issues and con- becoming an expert in evaluation. We also aim in the evaluation design, and the wide diversity of perspectives and approaches in
text with which the program deals but so that to provide persons responsible for adrninister- the evaluation field provide little firm guidance about how best to proceed with an
an appropriate evaluation plan can be devel- ing and managing human resource programs evaluation.
oped that reflects the reality of the program and with sufficient understanding of evaluation !11! Most evaluators are trained either in one of the social sciences or in professional
existing lmowledge relevant to such programs. tasks and activities to be able to judge for schools that offer applied social research courses. Highly specialized, technical, or
At the most complexlevel, evaluation activities themselves what kinds of evaluations are ap- complex evaluations may require specialized evaluation staffs. A basic knowledge of
can be so technically complicated, sophisti- propriate to their programs and projects and to the evaluation field, howeve~ is relevant not only to those who will perform
cated in conception, costly, and of such long comprehend the results of evaluation studies of evaluations but also to the consumers of evaluation research.
duration that they require the dedicated partici- their programs. ln brief, we have tried to pro-
pation of highly trained specialists at ease with vide a text that is helpful to those who conduct
the latest in social science theory, program evaluations, those who commission them,
lmowledge, data collection methods, and sta- those who oversee evaluation staffs, and those
tistical techniques. Such highly complex evalu- who are consumers of evaluation research.

L
Formative evalualion Evaluative activities undertaken to furnish information that will guide program improvement.
TAILORING EVALUATIONS
Summalive evaluation Evaluative activities undertaken to render a summary judgment on certain criticai aspects
of the program's performance, for instance, to determine if specific goals and objectives
were met. Evezy evaluation must be tailored to its program. The tasks tbat evaluators undertalrn depend on
Target The unit (individual, family, community, etc.) to which a program intervention is directed. tbe purposes of tbe evaluation, tbe conceptual and organizational stmcture of tbe program, and
Ali such units within the area served by a program comprise its target population. tbe resources available. Formulating an evaluation plan tberefore requires tbe evaluator to fiist
Stakeholders Individuais, groups, ar organizations having a significant interest in how well a program explore tbese aspects of tbe evaluation situation witb tbe evaluation sponsor and such otber
functions, for instance, those with decision-making authority over it, funders and sponsors, stakeholders as policymakers, program personnel, and program participants. Based on tbis recon-
administrators and personnel, and dients ar intended beneficiaries. naissance and negotiation witb tbe key stakeholders, tbe evaluator can tben develop a plan tbat
Evaluation sponsor The person(s), group, ar organization that requests ar requires the evaluation and provides identifies tbe evaluation questions to be answered, tbe metbods to be used to answer tbem, and
the resources to conduct it. tbe relationships to be developed witb tbe stakeholders during tbe course of tbe evaluation.
lndependent evaluation An evaluation in which the evaluator has the primary responsibility for developing the No hard-and-fast guidelines mwct tbe process of investigating tbe evaluation situation and
evaluation plan, conducting the evaluation, and disseminating the results. designing an evaluation-it is necessarily a creative and collaborative endeavoL Nonetbeless,
Parlicipatory or An evaluation organized as ateam project in which the evaluator and representatives of achieving a good fit between tbe evaluation plan and tbe program circumstances usually involves
collaborative evaluation one ar more stakeholder groups work collaboratively in developing the evaluation plan, attention to certain criticai tbemes. lt is essential, for instance, tbat tbe evaluation plan be
conducting the evaluation, or disseminating and using the results.
responsive to tbe purposes of tbe evaluation as understood by tbe evaluation sponsor and otber
Empowerment evaluation A participatory ar collaborative evaluation in which the evaluator's role indudes consultation central stakeholders. An evaluation intended to provide feedback to program decisionmakers so
and facilitation directed toward the development of the capabilities of the participating
tbat tbe program can be improved will take a mfferent approach tban one intended to help fun-
stakeholders to conduct evaluation on their own, lo use it effectively for advocacy and
change, and to have some influence on a program that affects lheir lives.
ders determine if a program should be terminated. ln admtion, tbe evaluation plan must reflect
an understanding of how tbe program is designed and organized so tbat tbe questions asked and
Evaluation questions A set of questions developed by the evaluator, evaluation sponsor, and other stakeholders;
the questions define the issues the evaluation will invesligate and are stated in terms such
tbe data collection arranged will be appropriate to tbe circumstances. Finally, any evaluation, of
that they can be answered using methods available to the evaluator in a way useful to course, will have to be designed witbin tbe constraints of available time, personnel, funding, and
stakeholders. otber such resources.
Needs assessment An evaluative study that answers questions about the social conditions a program is Altbough tbe particulars are &verse, tbe basic program circumstances for which evaluation
intended to address and the need for lhe program. is requested typically represent one of a small number of recognizable variations. Consequently,
Assessment of program An evaluative study that answers questions about the conceptualization and design of a tbe evaluation designs tbat result from tbe tailoring process tend to be adaptations of one or more
theory program. of a set of familiar evaluation approaches or schemes. ln practice, tberefore, tailoring an evalu-
Assessment of program An evaluative study that answers questions about program operations, implementation, and ation is ofren primarily a matter of selecting and adapting tbese schemes to tbe specific circum-
process service delivery. Also known as a process evaluation ar an implementation assessment. stances of tbe program to be evaluated. One set of evaluation approaches is defined around tbe
lmpact assessment An evaluative study that answers questions about program outcomes and impact on the nature of tbe evaluator-stakeholder interaction. Evaluators may function relatively independently
social conditions it is intended to ameliorate, Also known as an impact evaluation ar an or work quite collaboratively witb stakeholders in designing and conducting tbe evaluation.
outcome evaluation. Anotber mstinct set of evaluation approaches is organized around common combinations of
Efficiency assessment An evaluative study that answers questions about program costs in comprison to either evaluation questions and tbe usual metbods for answering tbem. Among tbese are evaluation
the monetary value of its benefits ar ilS effectiveness in terms of the changes brought schemes for assessing social problems and needs, program tbeory, program process or imple-
about in the social conditions it addresses. mentation, program impact or outcome, and program efficiency.

37
38 EVALUAT!ON Tailoring Evaluations 39

ne of the most chalienging aspects of of services, whether the prograrn is having its ioteractingwith prograrn stalceholders ta iden- pects of the evaluation plan they most io.flu-
O evaluation is that there is no 11one size fi.ts
ali" approach. Every evaluation situation has
desired irnpact, and its costs and efficiency. No
evaluation can, nor generaliy should, attempt
tify and clarify the issues, conduct the evalu-
ation, and malce effective use of the evaluation
ence; an overview is provided here.

its unique profile of characteristics, and the to address ali such concems. A central feature findings. This interaction may be highly col-
evaluation design must involve an interplay of an evaluation design, therefore, is a specili- laborative, with the evaluator serving as a con-
sultant ar facilitator to a group of stakeholders
The Purposes of the Evaluation
between the nature of the evaluation situation, cation of the guiding purpose of the evaluation
on the one side, and the evaluator's repertoire and the corresponding questions on which it who talce primary responsibility for planning, Evaluations are initiated for many reasons
of approaches, techniques, and concepts, on will focus. Later in this chapte~ and in more conducting, and using the evaluation. Or the andmay have quite different purposes from one
the other. A good evaluation design is one that detail ia Chapter 3, we discuss the nature of evaluator may take that responsibility but seelc situation to another. They may be intended to
fits the circumstances while yielding credible evaluation questions, how they can be derived, essential guidance and inforrnation from the help management iroprove a prograrn; support
and useful answers to the questions that moti- and some of the factors that io.fluence the stalceholders. ln addition, an evaluation plan advocacy by supporters ar critics; gain knowl-
vate it. This chapter provides an overview of priority they should be given. should iodicate which audiences are to receive edge about prograrn effects; provide input to
the issues and considerations the evaluator which inforrnation at what times, what the decisions about prograrn funding, structure, ar
should take into account when tailoring an Tbe methods and procedw:es the evalu- nature and schedule of written reports and oral adrninistration; respond to political pressures;
evaluation plan to accomplish these purposes. ation will use to answer the questions. An briefings will be, and how broadly findings are ar have any of a number of such purposes
irnportant aspect of the evaluator's distinctive to be dissemioated beyond the evaluation spon- iodividualiy ar ia combioation. One of the first
expertise is knowiog how to obtain useful, sor. The evaluator-stalceholder relationship is deterrnioations the evaluator must malce,
timely, and credible inforrnation about the vari- discussed !ater in this chapter and ia Chap- therefore, is just what those purposes are. This
WHAT ASPECTS OF THE EVALUATION ous dimensions of prograrn performance that ter 12.. is not always a sirnple matter. Some statement
PLAN MUST BE TAILORED? are to be evaluated. A large repertoire of social of the purposes generaliy accom panies the ini-
research techniques and conceptual toais are tial request for an evaluation, but these an-
available for this task. An evaluation design nounced purposes rarely tell the whole story
Evaluation designs may be quite sirnple and must identify the methods that will be used to WHAT CONSIDERATIONS SHOULD and sometimes are only rhetorical. Further-
direct, perhaps addressing only one narrow answer each of the questions at issue and orga- GUIDE EVALUATION PLANNING? more, evaluations may be routioely required in
question such as whetherusing a computerized nize them iota a feasible work plan. Moreove~ a prograrn situation ar sought sirnply because
instructional prograru helps a class of third the methods selected must not only be capable it is presumed to be a good idea without any
graders read better. Or they may be prodigiously of providing meaningful answers to the ques- Many aspects of the prograrn and the circum- distinct articulation of its purposes ar the spon-
complex, as in a national evaluation of the tions but also must be practical while still stances of the evaluation will necessarily shape . sor's intent (see Exhibit 2.-Aj.
operations and effects of a diverse set of pro- providing the degree of scientific rigor appropri- the evaluation design. Some of these iovolve The prospective evaluator deterrnioes the
grams for reduciog substance abuse in multiple ate to the evaluation circumstances. Most of general considerations of almost universal rele- purposes of the evaluation by attemptiog to
urban sites. Fundarnentaliy, howeve~ we can the rest of this book (Chapters 4-11 J is devoted vance to evaluation planning, but others will be establish as firm!y as possible who wants the
view any evaluation as structured around three to consideration of evaluation methods and the specific to the particular situation of each evaluation, what theywant, and whytheywant
issues: circumstances ia which they are applicable. evaluation. Development of the evaluation it. There is no cut-and-dried method for doiog
plan, therefore, must be guided by a careful this, but it is usualiy best approached the way
The questions the evaluation is to answer. Tbe nature of the evaluator-stakebolder analysis of the evaluation context. The more a joumalist would try to dig out a story. That
An endless number of questions might be relationsbip. One of the most irnportant les- sigruncant considerations for that analysis can is, source documents should be exarnioed, key
raised about any social prograrn by a wide range sons from the first severa! decades of experi- be organized into three categories, having to do inforrnants with different vantage poiots on the
of interested parties. There may be concems ence with systematic evaluation is that there is with (a) the purposes of the evaluation, (bj the situation should be interviewed, and pertioent
about such matters as the needs of the target nothiog automatic about the assirnilation and prograrn structure and circumstances, and history and background should be uncovered.
population and whether they are beiog ade- use of evaluation findings by the stal<eholders (cJ the resources available for the evaluation. Although the details will vary greatly, evalu-
quately reached and served, the management presumed ioterested in them. Part of an evalu- Ali these topics will receive !ater attention in ations are generaliy dane for one ar more of the
and operation of the prograrn, the effectiveness ation design, therefore, is a plan for effectively the course of discussion about the specific as- followiog broad reasons (Chelimsky, 1997):
40 EVALUATION Tailorlng Evaluations 41

Our initial meetings with the Bureau of the evaluatian was an administrator in charge of Forma tive evaluation procedures were used to routinely distributed to all project participants.
Community Services administrators praduced contracts for the Department of Carrections, but help design a "stop smoking" hotline for 2, 148 Despite these efforts, an average af /ess than
only vague statements about the reasans for the we were unable to obtain specific information adult smokers in a cancer contrai praject three calls per month was made during the 33
evaluation. They said they wanted some infor- concerning where ar how the evaluation would sponsored by a health maintenance organization months the hot/ine was in operation, abaut a
mation abaut the cost-effectiveness of both New be used. We could only discern that an evalu- (HMO). Phone scripts for use by the hotline 2.4% use rate by the target population. To further
Dawn and Pegasus and also how well each ation of state-run facilities had been mandated, counselors and other aspects of lhe planned assess this disappointing response, comparisons
program was being implemented .... lt gradually but it was not clear by whom. services were discussed with focus groups of were made with similar services around the
became clear that the person most interested in smokers and reviewed in telephone interviews country. This revealed that 1/o-2/o use rates were
with a representative sample of HMO members typical but the other hotlines served much larger
SOURCE: Quoted from Dennis J. Palumbo and Michael A. Hallett, "Conlict Versus Consensus Models in Policy who smoked. Feedback from these informants populations and therefore received many more
Evaluation and lmplementation," Eva/uation and Program Planning, 1993 1 16{1 ):11-23. led to refinement of the scripts, hours of calls. The program sponsors concluded that to be
operation arranged around the times participants successful, the smoker's hotline would have to be
said they were mos! likely to call, and advertising offered to a larger population and be intensively
of the service through newsletters and "quit kits" publicized.
program improvement, accountability, knowl- ings tbat are timely, concrete, and immediately
edge generation, and political ruses ar public useful. Correspondingly, the co=unication SOURCE: Adapted from Russell E. Glasgow, H. Landow1 J. Hollis, S. G. McRae, and P. A. La Chance, "A Stop-Smoking
Telephone Help Une That Nobody Called," American Jouma/ o( Public Hea/rh, February 1993, 83:252-253.
relations. between the evaluator and the respective audi-
ences about the findings may occur regularly
throughout the evaluation and be relatively
Program improvement. The evaluation find- irormal.
ings may be intended to furnish irormation
that will guide program improvement. Such
evaluation is often called formative evaluation Accountability. The use of social resources
jScriven, 1991) because its purpose is to help such as taxpayer dollars by human service pro- The Mammography Quality Standards Act of helped to minimize adverse effects on access.
form ar shape the program to perform better grams is justilied on the grounds that these 1992 required the Food and Drug Administration The FDA inspectors had not closed many facilities
jfor an example, see Exhibit 2-B). The audi- programs malce beneficial contributions to so- (FDA) to administer acode of uniform standards that failed to meet certification standards; rather,
ciety. It follows that persons with significant for mammogram-screening procedures in all the they had given them additional time to carreei
ences for the findings of formative evaluation
states. When the act was passed, Congress was the problems found during inspections and to
typically are the program planners lin the case responsibilities for such social investments will
concerned that access to mammography ser- meet the new quality assurance requirements.
of programs in tbe planning stage) ar program expect programs to manage resources effec-
vices might decrease because providers would Only a relatively small number of facilities had
administrators, oversight boards, ar funders tively and efficiently and actually produce the choose to drop .!hem rather than upgrade terminated their mammography services and
with an interest in opt:i:ntlzing program effec- intended benefits. Evaluation may be con- operations to comply with the new standards. those were generally small-volume providers
tiveness. The irormation desired by these per- ducted, therefore, to determine if these expec- The U.S. General Accounting Office (GAO) was located within 25 miles of another certified fa-
sons may relate to the need for the program, tations are met. Such evaluation is often called asked to assess the early effects of implementing cility. The GAO concluded that the Mammog-
the program concept and design, its implemen- summative evaluation IScriven, 1991) because the act and repor! back to Congress. They found raphy Quality Standards Act was having a
tation, its impact, ar its efficiency. Typically, the its purpose is to render a summary judgrnent that the FDA had taken a gradual approach to positive effect on the quality of mammography
evaluator in this situation will work closely on certain criticai aspects of the program's implementing the act's requirements, which had services, as Congress had intended.
with program management and other stake- performance IExhibit 2-C provides an exam-
holders in designing, conducting, and reporting ple). The findings of summative evaluation are SOURCE: Adapted from U.S. General Accounting Office, Mammography Services: Jnitial lmpact o( New Federal Law
Has Been Positive. Report 10/27/95, GAO/HEHS-96-17 {Washington, OC: General Accounting Office, 1995).
the evaluation. Evaluation for program im- usually intended for decisionmakers with ma-
provement characteristically emphasizes find- jor roles in program oversight, for example, a
42 EYALUATION Tailoring Evaluations 43

funcling agency, governing board, legislative vention. Because evaluations of this sort are
committee, political decisionmake~ ar upper intended to make contributions to the social
management, but may also be of interest to science knowledge base, they are usually con-
critics, constituents, and concemed citizens ducted in a scientific framework using the most Pathological gambling is characterized by a To test their treatment concept, the
outside the formal decision-making chan- rigorous methods feasible. The audienceforthe loss of contrai over gambling impulses, lies about researchers used media advertisements and
nels. Summative evaluation may influence resulting finclings may include the research the extent of gambling, family and job disruption, referrais from health praviders to recruit 40
such significant decisions as program continu- sponsors in cases of demonstration projects ar stealing money, and chasing lasses with pathological gamblers willing to accept
ation, allocation of resources, restmcturing, ar extemally funded research. Beyond that, how- additional gambling. Though recent increases in treatment. These were randomly assigned to the
legal action. For this reason, such evaluation eve~ the audience is generally quite dilluse-all the availability of gambling have led to treatment ar contrai group and measures of
often requires information of sufficient credi- those interested in the particular type of pro- corresponding increases in the prevalence of pathological gambling, perception of contrai,
bility under scientific standards to provide a gram o~ perhaps, the particular methods used pathological gambling, few treatment pragrams desire to gamble, self-efficacy perception, and
confident basis for action and to withstand to study it. Dissemination of the evaluation have been developed to help the victims of this frequency of gambling were taken at various
disorder. Research on the psychology of intervals before and after the treatment period.
criticism aimed at discrediting it. The evalu- finclings in these situations is most likely
gambling has shown that prablem gamblers The results showed significant changes in the
ator may be expected to function relatively through scholarly journals, conferences, and
develop an illusion of contrai such that they treatment group on ali outcome measures with
independently in planning, conducting, and other such professional outlets. These knowl- believe they can employ strategies that will maintenance of the gains at 6- and 12-month
reporting the evaluation with input from, but edge generation studies may tum out to be increase their winnings despite the inherent follow-up. However, the results may have been
no direct decision-making participation by, useful for the development of new public pro- randomness of games of chance. A team of compramised by high attrition-8 of the 20
stalceholders. Similarly, it may be important to grams as program developers draw on social clinicai researchers in Canada hypothesized that gamblers who began treatment and 3 of the 20
avoid premature ar careless conclusions and, science research for program ideas. a treatment based on 11 cognitive correction" of in the contrai group dropped out, a common
therefore, co=unication of the evaluation these erroneous beliefs would be an effective occurrence during intervention for addictive
finclings to the respective audiences may be Political ruses ar public relations. Sometimes, therapy. Because excessive gambling leads to problems. Despite this limitation, the researchers
relatively formal, rely chiefly on written re- the true purpose of the evaluation, at least for financial problems and interpersonal difficulties, concluded that their results were strong enough
ports, and occur primarily at the end of the those who initiate it, has little to do with they combined their cognitive intervention with to demonstrate the effectiveness of their
evaluation. actually obtainiog information about program prablem-solving and social skills training. treatment concept.
performance. It is not unusual, for instance, for
SOURCE: Adapled from Caroline Sylvain, Robert Ladouceur, and Jean-Marie Boisvert, "Cognitive and Behavioral
Knowledge generation. Some evaluations are program adrninistrators ar boards to launch an
Treatment of Pathological Gambling: A Controlled Study, 11 }oumal of Consulting and Clinicai Psycholagy, 1997,
not intended to directly inform decisions re- evaluation because they believe it will be good 65(5):727732.
lated to specific programs in place ar contem- public relations and might impress funders ar
plated but, rathe~ mainly describe the nature political decisionmakers. Occasionally, an
and effects of an intervention for broader pur- evaluation is commissioned to provide a public
poses and audiences. The intervention at issue, context for a decision that has already been cult dile=a. The evaluation must either be evaluator may wish to assume an "evaluation
for instance, might be a demonstration config- made behind the scenes to terminate a pro- guided by the political ar public relations pur- consultant" role and assist the relevant parties
ured expressly to try out a promising concept gram, fire an adrninistrato~ or the like. Or the poses, which may compromise its integrity, ar to clarify the nature of evaluation, identify ap-
such as integrated services for children with evaluation may be a delaying tactic to appease focus on program performance issues that are propriate and realistic expectations, and redi-
mental health problems or monthly visits by critics and defer difficult decisions, rather like of no real interest to those commissioning the rect the effort toward more appropriate uses.
nurses to pregnant women at risk of premature appointing a committee to study a problem evaluation and may even be threatening. ln
births isee Exhibit 2-D for another example). A rather than acting on the problem. either case, the evaluatorwould be well advised
similar situation occurs when an academic re- Vrrtually all evaluations have some ele- to try to avoid such situations. If a !ade of The Program Structure
searcher arranges to study an intervention with ments of political maneuvering and public re- serious intent becomes evident during the ini-
tial exploration of the evaluation context, pro-
and Circumstances
interesting characteristics, for example, an in- lations among their instigating motives, but
novative science curriculum.1 to contribute to when these are the principal purposes, the ceecling with an evaluation plan would not No two programs are identical in their
lmowledge about that particular form of inter- prospective evaluator is presented with a diffi- generally be wise. Instead, the prospective organizational structure and environmental,
44 EVALUATION Talloring Evaluatlons 45

social, and political circwnstances, even when stages and, therefore, different evaluation ap-
they ostensibly provide the "same" service. The proaches must be applied to answer those ques-
particulars of a program's structure and cir- tions (see Exhibit 2.-E). Assessment of a pro-
cwnstances constitute major features of the gram still in the early stages of planningwill be Stage o( Pragram Develapment 1Question to Be Asked fva/uarion Funcrian
evaluation situation to whlch the evaluation a distinctly different endeavor than assessment
1. Assessment of social problems and needs

..
plan must be tailored. Although there is a of a well-established program. Similarly, assess- To what extent are community needs Needs assessment;
and standards met? problem description
myriad of such particulars, three broad catego- ment of an established program for whlch re-
2. Determination of geais What must be dane to meet those Needs assessment;
ries are especially irnportant to evaluators be- structuring is contemplated ar under way will needs and standards? service needs
cause of their pervasive influence on evaluation raise different concems than a program pre-
{ 3. Design of program alternatives What services could be used to Assessment of program
design and irnplementation: swned stable in its basic operations and func- produce the desired changes? logic or theory
tions. 4. Selection of allernative Which of the possib[e program Feasibility study;
The stage of program development- When new programs are initiated, espe- approaches is best? formative evaluation
whether the pmgram being evaluated is new cially innovative ones, evaluation is often re- li 5. Program implementation How should the program be put lmplementation assessment
ar innovative, established but still develop- quested to examine the social needs the pro- f
into operation?
ing ar undergoing restructuring, ar estab-
lished and presumed stable.
gram should address, the program design and l 6. Program operation Is the program operating as Process evaluation;
objectives, the definition of its target popula- planned? program monitoring
The adnnistrative and political context of tion, the expected outcomes, and the means by 7. Program outcomes Is the program having the desired Outcome evaluation
the pmgram, in particulai; the degree of effecls?
whlch it asswnes those outcomes can be at-
consensus, conllict, ar confusion among B. Program efficiency Are program effects attained ata Cost-benefit analysis;
tained. These issues are especially relevant dur-
stal<eholders about the values ar principies resoniilble cost? costeffectiveness analysis
ing the planning phase when the basic design
the program embodies, its rnission and is being formulated and changes can be made SOURCE: Adapted from S. Mark Pancer and Anne Westhues, "A Developmental Stage ApprO:ach to Program Planning
goals, ar its social sigoificance.
relatively easily. The evaluato~ therefore, may and Evaluation/' Eva/uarion Review, 1989, 13{1 ):56-77.
The struct:uie of the pmgram, including function as a planning consultant before the
both its conceptual and orgaillzational program is launched by helping to assess and
malreup. This involves the nat:uie of the improve the program design as it is developed.
program rationale; the c!iversity, scope, and
Assessment of the program conceptualization
character of the services pmvided and of the attempting to count ali persons sleeping m Sometirnes evaluations of new programs
may also be the focal point of an evaluation
target populations for those services; loca- public places such as bus stations, parks, ar are expected to address questions of impact and
tion of servic.e sites and facilities1 admin- alter the planning phase when the program is
in the early stage of irnplementation. Decision- store entrance ways. efficiency, but the unsettled nature of the pro-
istrative arrangements; record-keeping pro-
makers associated with young programs are grams in their beginning years most often
cedures; and so forth.
often open to some amount of reformulation of A program to increase public awareness makes those issues premature. It can easily
the program model and may wish to have it of risk factors in cardiovascular diseases and take a year ar more for a new program to
The influence of these factors will be consid-
assessed to ensure that their approach is as encourage exercise and proper diet attempted establish facilities, acquire and train staff, malce
ered in relation to various specific aspects of
good as it can be. to organize discussion groups among employ- contactwith the target population, and develop
evaluation design discussed throughout the re-
The followiog examples illustrate the role ees of local firrns. The evaluators found that its services to the desired levei. During tltls
mainder of thls book; we provide a brief orien-
of evaluators in the early stages of program tltls approach was largely unsuccessful because period, it may not be realistic to expect much
tation here.
development: workers were reluctant to form discussion irnpact on the social conditions toward whlch
groups. The evaluators suggested a more suc- the program is directed. Formative evaluation
The Stage of Program Development cessful approach using existing organized
A small New England city wanted to aimed at clarifying target population needs,
The !ife of a social program can be thought establish an emergency shelter for homeless groups such as churches, fraternal organiza- improving program operations, and enhancing
of as a developmental progression in whlch persons. To determine how many beds should tions, and clubs. the quality of service delivery, using approaches
dillerent questions are at issue at different be provided, the city funded a nighttime survey,
46 EVALUATION Tailorlng Evaluations 47

such as those discussed !ater in Chapters 4-6, program plans, and whether it is reaching the (Fraker; Martini, and Ohls, 1995). Signilicant challengingto design an evaluation that sirnul-
is likely to be more apt in these cases. appropriate target population. For example, the differences were found: Households receiving taneously addressed effectiveness questions
Although the evaluation of new programs U.S. Departrnent of Agriculture conducts peri- checks purchased less food than those that generated by stalceholders who asserted that
represents an irnportant activity for the field, odic studies of participation in the food stamps received food stamps. The Departrnent of Ag- the purpose of a program for dysfunctional
by far the greater effort goes into assessing program to measure the extent towhich eligible riculture therefore decided to retain food families was to protect the children from abuse
established programs. Evaluating these pro- households are enrolled and to guide outreach stamps. and, therefore, should readily support remov-
grams requires first understanding their social efforts to increase participation ('lli.ppe, 1995). ing them from the home if there is suspicion of
and political history. Most well-established so- Sometirnes, however; evaluation is sought The Administrative and abuse, and those generated by stalceholders
cial programs have sprung from long-standing for established programs primarily because the Political Context of the Program insisting that the purpose was to keep families
ameliorative efforts,. and unless some crisis program status quo has been calied into ques- intact and help them work out their problems.
necessitates consideration of fundamental tion. This may be due to externa! pressures Except possibly for acadernic researchers Each of these perspectives entails different ob-
change, they are constrained to theirtraditional such as poltica! attack, competition, mounting who canduct an evaluation study on their own jectives, and procedures to attain those objec-
forms and approaches. Often there is consider- program costs, ar dramatic changes in the tar- initiative for knowledge generation purposes, tives, that have correspondingly different con-
able opposition from some stakeholders to any get population served. Or it may occur because evaluators are not free to establish their own ceptions of program effectiveness associated
questioning of their fundamental assumptions program sponsors and staff are dissatisfied with definitions of what the program is about, its with them. Although the evaluator rnight at-
ar the ways they have been put into place. The the effectiveness of their interventions and goals and objectives, and what evaluation ques- tempt to design an evaluation that would en-
value of such well-entrenched programs as So- wish to bring about irnprovement. ln either tions should be addressed. The evaluator inter- compass these different perspectives and thus
cial Security pensions, guidance counselors in event, some restructuring may be considered, acts with the evaluation sponsor; program per- inform both sets of stalceholders, such an effort
schools, vocational programs for disabled per- and evaluation is sought to guide that change. sonnel, and other program stal<eholders to would require a careful balance in determining
sons, parole supervision for released convicts, ln circumstances such as these, the evaluation develop this essential background. Somewhat what data to collect and what criteria to apply
and community health education for the pre- may focus on any and ali aspects of the pro- different perspectives from these various to interpret them.
vention of diseases is tal<en for granted. gram. Questions rnight be raised about the groups are to be expected and, in most in- Altematively, the evaluator could plan the
Evaluation of established, stable programs, need for the program, its conceptualization and stances, the evaluator will attempt to develop evaluation from the perspective of one of the
therefore, rarely focuses on assessing the design, its operations and irnplementation, and an evaluation plan that reflects ali signilicant stal<eholders, typically the evaluation sponsor
underlying program conceptualization. It is its irnpact and efficiency. views and canceras ar; at least, is compatible ar some other stalceholder designated by the
more likely that attention will be directed to- The federal food stamps program men- with the prevailing views among the major evaluation sponsm This, of course, will not be
ward such issues as coverage, effective service tioned above, for instance, has been a national parties. greeted with enthusiasm by stalceholders with
delivery, and the irnpact and efficiency of those program for more than two decades. It is in- If signilicant stakeholders are in substan- conflicting perspectives and they may well op-
services. However; if the program is very large tended to increase the quantity and quality of tial conflict about the rnission, goals, probity, pose the evaluation and criticize the evaluatm
and well established, it can be difficult to evalu- food consumed by poor households by provid- procedures, ar presenting issues for the pro- The challenge to the evaluator is to be clear and
ate irnpact and efficiency, especialiy if it is a ing them with food stamps redeemable only by gram, it presents an irnmense difficulty for straightforward about the perspective repre-
full-coverage program that provides services to purchasing approved foods at grocery stores. evaluation design (see Exhibit 2-FJ. The evalu- sented in the evaluation and the reasons for it,
virtualiy the entire eligible population. ln such The Departrnent of Agriculture contemplated ator can attempt to incorporate the conflicting despite the objections. It is irnportant to recog-
cases, the evaluator has limited ability to de- abandoning food stamps and issuing checks perspectives into the design, but this may not nize that it is not necessarily wrong to plan and
velop credible depictions of what things would instead, thereby elirninating the high costs of be an easy task. The evaluation sponsors may conduct an evaluation from the perspective of
be like in the absence of the program as a printing, distributing, and redeeming an ear- be unwilling to embrace the inclusion of issues one stalceholder without giving strong repre-
baseline for assessing its irnpact. Often, evalu- marked currency. To test the effects of cashing and perspectives from groups they view as ad- sentation to conflicting views. Nonetheless,
ation of such programs is directed toward as- out food stamps, it started four experiments versaries. Furthermore, the issues and perspec- evaluators generally solicit input from ali the
sessing the extent to which the program objec- comparing the food consumption in house- tives may be so different that it is difficult to major stal<eholders and attempt to incorporate
tives are explicit and relevant to the interests holds receiving food stamps with the food con- incorporate them in a single evaluation planar their canceras so that the evaluation plan will
ofprogram sponsors, staff, and other stalcehold- sumption of households receiving the sarne to do so may require more time and resources be as comprehensive as possible and the results
ers, whether the program is conforming to dollar amount of benefits in the form of checks than are available. For instance, it would be as useful as possible. Where there are conflict-
48 EVALUAT!ON Tailaring Evaluatians 49

conllicts often involves matters that the evalu- still respects tl1e pluralism of ideologies and
ator can examine and report on in ways that concems represented by the dillerent stalce-
inform all parties. For instance, parents of spe- holders. Additional discussions of this perspec-
cial education children may believe that their tive can be found in Guba and Lincoln (1987,
ln an evaluation of a home arrest program parole, the evaluators made the following children are stigmatized and discriminated 1989, 1994). It may have particular appeal for
using electronic monitoring for offenders on comments about stakeholder views:
against when mainstreamed in regular class- the evaluator working in contexts where the
rooms. Teachers may feel equally strongly that evaluation is highly politicized ar stalceholder
There were numerous conflicting goals that were considered important by different agencies,
including l~wering costs and prison diversion1 contrai and public safety, intermediate
this is not true. A careful observational study values and views about the program are
punishment and increased options for corrections, and treatment and rehabilitation. Different of the interaction of regular and special educa- strongly divergent.
stakeholders emphasized different goals. Some legislators stressed reduced costs, others tion children conducted by the evaluator may Finally, the evaluator must realize tl1at de-
emphasized public safety, and still others were mainly concerned with diverting offenders reveal that there is a problem, but that it occurs spite best efforts to ca=unicate effectively
from prison. Some implementors stressed the need for contrai and discipline for certain outside the dassroom on the playground and and develop an appropriate, responsive evalu-
11
dysfunctional 11 individuais, whereas others focused on rehabilitation and helping offenders during other informal interactions among the ation plan, program stalceholders owe primary
become reintegrated into society. Thus, there was no common ground for enabling "key children. allegiance to their own positions and political
policymakers, managers, and staff" to come to an agreement about which goals should have Where stalceholder conflict is deep and hos- alignments. This means that sponsors of evalu-
priority or about what might constitute program improvement. tile, it may be based on such profound diller- ation and other stalceholders may tum on the
ences in political values ar ideology that no evaluator and harshly criticize the evaluation il
SOURCE: Dennis J. Palumbo and Michael A. Hallett, "Conllct Versus Consensus Models in Policy Evaluation and
matter how comprehensive and ecumenical, an the results contradict the policies and perspec-
lmplementation," Eva/uation and Program Planning, 1993, 16(1 ):1123.
evaluation cannot conjoin them. One school of tives they advocate. Thus, even those evalua-
thought in the evaluation field holds that all tors who do a superb job ofworkingwith stalce-
program situations are of this sort and that it holders and incorporating their views and
ing perspectives, howeve~ it is not inappropri- encourage their input so that the evaluation is the central feature to which the evaluator concems in the evaluation plan should not
ate for an evaluation sponsor to seek informa- design can be as sensitive as possible to the must attend. In this view, the social problems expect to be acdaimed as heroes when the
tion relevant to its perspective ar the evaluator realities of the program and the legitimate can- that programs address, the programs them- results are in. The multiplicity of stalceholder
to conduct such an evaluation even il some ceras of management about misrepresentation selves, and the meaning and importance of perspectives malces it lilcely that no matter how
stalceholder views are given little ar no influ- of what they are doing and why. But the evalu- tl10se programs are all social constructions that the results come out, someone will be unhappy.
ence. ation design should, nonetheless, be developed will inevitably diller for dillerent individuals Evaluators work in a political environment,
Suppose, for instance, that the funding primarily from the perspective of the evalu- and groups. Thus, rather than focus on pro- and it is the nature of such environments that
sponsors for a program to provide job training ation sponsors and the issues that concem gram objectives, decisions, outcomes, and the stakeholders will often react to evidence con-
to the hard-core unemployed have concerns them. The evaluator's primary responsibilities lilce, evaluators are advised to engage directly trary to their interests with a vigorous attempt
about whether a program is "crearning'' the are simply to be dear about the perspective the the diverse claims, canceras, issues, and values to discredit it and those who produced it. It may
cases that are easy to work with, providing evaluation takes, so there is no misunderstand- put forth by the various stakeholders. matter little that everyone agreed in advance on
services that are more weighted toward voca- ing, and to treat the program personnel fairly Guba and Lincoln (1989), the leading pro- tl1e evaluation questions and the plan for an-
tional counseling than job skill training, and and honestly. ponents of this particular construction of the swering them ar that each stalcel10lder group
are inelliciently organized. They might quite Another approach to situations of stake- evaluation enterprise, have argued that the understood that honest results might not favor
appropriately commission an evaluation to ex- holder conllict is for the evaluator to attempt proper role of tl1e evaluator is to facilitate inter- its position. Nonetheless, it is highly advisable
amine these questions. Program managers and to design an evaluation that facilitates better pretive dialogue among the program stalcehold- for the evaluator to give early attention to the
their advocates, on the other hand, may have a understanding among the conflicting parties ers. Correspondingly, the primary purpose of identilication of stalceholders, working out a
sharply conllicting perspective that jus tilies about the aspects of the program at issue. This the evaluation is to facilitate a negotiation strategy for minimizing discord in the evalu-
their selection of clients, training program, and might be dane, for instance, by efforts to darify among the stalceholders from which a more ation due to their dillerent perspectives, and
management practices. A conscientious evalu- the dillerent concems, assumptions, and per- shared construction of the value and social conditioning tl1eir expectations for the nature
atorwill listen to the managers' perspective and spectives of the parties; some portion of such significance of the program can emerge that and significance of the evaluation results.
50 EVALUATION Tailoring Evaiuations 51

The Conceptual and Organizational lyiog prograrn rationale ar agree on any single
Structure of the Program version of it. For instance, the personnel in a
counseling agency under contract to the school
It is a simple truism that if authoritative district to work with children who are having
stakeholders do not have a clear idea about academic problems may be quite articulate in Besides the usual methodological considera- ar individuais with political agendas. Une staff
what a prograrn is supposed to be doing, it will describing their counseling theories, goals for tions involved in conducting credible evalu- easily adopt an "us against them" mentality
be dillicult to evaluate how well it is doing it. clients, and therapeutic techniques. But they ations, the structural features of criminal justice toward outside evaluators. Also, criminal justice
One factor that shapes the evaluation design, may have dilliculty expressing and agreeing on settings impose social, political, and organi- agencies generally exist in highly charged
therefore, is the nature of the prograrn concep- a view of how their focus on improving farnily zational constraints that make multisite eval- political environments. They are the most visible
tualization, that is, the distinctness and explic- co=unication is supposed to translate into uations difficult and risky. To begin, the system components of local government, as well as the
itness of its plan of operation, the logic that better grades. Howeve~ the evaluator needs is extremely decentralized. Police departments1 most expensive, and their actions are frequently
connects its activities to the intended out- for example, can operate within the province of monitored by the media, who historically have
some understanding of their assumptions on
municipalities, counties, campuses, public hous- assumed a watchdog ar adversaria! posture to-
comes, and the rationale provided for why it this matter to plan any assessment of the pro-
ing, mass transit, and the states. The criminal ward lhe system. Finally, the criminal justice
does what it does. This conceptual structure ar gram's overt performance ar outcomes. Corre-
justice system is also highly fragmented. Cities system operates within a context of individual
program theory can itself be a focus of evalu- spondingly, the more explicit and cogent the administer police departments and jails; coun- rights-legal constraint in procedural issues, an
ation. prograrn conceptualization, the less specific as- ties administer sheriffs' and prosecutors' offices, unwillingness to risk injustice in individua] cases,
If there is significant uncertainty about sessment it may require in the evaluation plan jails, and probation agencies; state governments and a stated (though not actually delivered)
whether the prograrn conceptualization is ap- and the easier it will be to identify the prograrn run the prisons. Agencies are embedded in commitment to providing individualized treat-
propriate for the social problem the prograrn functions and effects on which the evaluation disparate political settings, each with its own ment. This translates, for example, into a general
addresses, it may make little sense for the should focus. priorities for taxing and spending. ln addition, aversion to the concept of random ar unbiased
evaluation design to focus on how well those Ata more concrete levei, the organizational criminal justice agencies foster a subculture of assignment, lhe hallmark of the best designs for
concepts are implemented. ln such cases, the structure of the prograrn must also be tal<en secrecy concerning their work that has serious yielding interpretable information about pro-
evaluation activities may be more usefully de- into consideration when planning an evalu- consequences for evaluators, who are readily gram effects.
11 11
voted to assessing and better developing the ation. Such prograrn characteristics as multiple seen as snoops for management the courts,

prograrn plan. services ar target populations, distributed ser-


SOURCE: Adapted from Wesley G. Skogan and Arthur J. Lurigio, Mu/tisite Eva/ualions in Criminal Justice Seltings:
ln the planning stages of a new prograrn, vice sites ar facilities, ar extensive prograrn- Structura/ Obsracles to Success, New Directions for Evaluation, no. 50 (San Francisco: Jossey-Bass, summer 1991 ),
establishing the prograrn design is a major matic collaboration with other organizational pp. 83-96.
activity and its nature and details are usually entities have powerful implications for the na-
easily identified and articulated. The participa- ture and range of evaluation questions to be
tion of an evaluator, howeve~ often helps covered, data collection procedures, resources
sharpen and shape the conceptualization to required for the evaluation, and stakeholder
them are distinct topics of discussion in the interventions are usually relatively straightfor-
mal<e it both more explicit and more useful for groups to involve. Organizational structures
evaluation literature (see Exhibit 2-G; Turpin ward (soup kitchen), the serviceitselfis uncom-
identifying key issues ofprograrn performance. that are large~ more complex, more decentral-
and Sinacore, 1991 ). plicated (hand out meals), and the outcomes
After the planning stage, especially for well- ized, and more geographically dispersed will
Equally important is the nature and struc- are direct (people eat). These features greatly
established prograrns, prograrn personnel ar present greater practical dilliculties than their
ture of the particular intervention or service the simplify the evaluation questions likely to be
sponsors generally find little need ar opportu- simpler counterparts. ln such cases, a tearn of
prograrn provides. The easiest interventions to raised, the data collection required to address
nity for identifying and reviewing basic as- evaluators is often needed, with resources and
evaluate are those that are discrete, one-shot them, and the interpretation of the findings.
sumptions and expectations in any systematic time proportionate to the size and complexity
events (e.g., serving meals to homeless persons) The most dillicult interventions to evalu-
manner Everyday practice and routine operat- of the prograrn. The challenges of evaluation
expected to have relatively immediate observ- ate are those that are diffuse in nature (com-
ing procedures tend to dominate, and person- for complex, multisite prograrns are suffi-
able effects (they are not hungry). The organi- munity organizing), extend over long time pe-
nel may find it dillicult to articulate the under- ciently daunting that planning and conducting
zational activities and delivery systems for such riods (an elementary school math curriculum),
52 EVALUATION Tailoring Evaluations 53

vary widely across applications (eclectic planning, budgeting, and detennining time- all concemed. ln the most severe cases, such in machine-readable databases is usually easier
psychotherapy), ar have expected outcomes lines can be found in Hedrick, Biclanan, and resistance may comprornise the scope ar valid- to process. lncreasingly, agency records are kept
that are long tenn (preschool compensatory Rog (1992), Card, Greeno, and Peterson (1992), ity of the evaluation ar even make it impossible on computers, with duplicate copies obtainable
education) ar indistinct (improved quality of and Fink (1995, chap. 9). to complete. for analysis. Of course, machine data bases may
life). For interventions of this sort, many evalu- Although the available funding is, of An especially important interaction with contain rnissing infonnation that reduces their
ation questions dealing with prograrn process course, one of the critica! resource issues the prograrn involves access to and use of pro- utility, but in most cases, MIS data are valuable
and outcome can arise because of the differen- around which the evaluation must be planned, grarn records, documents, and other such in- in evaluations.
tiated nature of the services and their potential it is important to recognize that the dollar terna! data sources. It is a rare evaluation de- The crucial point here is that the evaluator
effects. Furthennore, the evaluator may have arnount of that funding is not the only resource sign that does not require at least some must view cooperation from prograrn person-
difficulty developingmeasures that cleanly cap- that will cancera the evaluatm Evaluation is a infonnation from prograrn records and many nel, access to prograrn materials, and the na-
ture critica! aspects of prograrn implementa- specialized fonn of inquiry that talces place are based heavily qn those records. Such re- ture, quality, and availability of data from pro-
tion and outcome when they are complex ar largelywithin the operating environment of the cords may be necessary to identify the number gram records as major resource issues when
dilluse. Actual data collection, too, may be prograrn being evaluated. This means, for in- and characteristics of clients served, the type planning an evaluation. The potential for rnis-
challenging if it must take place over extended stance, that pertinent technical expertise must and arnount of services they received, and the understanding and resistance can be lowered
time periods ar involve many different vari- be available if the evaluation is to be dane well. cost of providing those services. The scope, considerably if early discussions with evalu-
ables and observations. All these factors have ln a large evaluation project, a number of pro- completeness, and quality of prograrn records, ation sponsors, prograrn personnel, and other
implications for the particulars of the evalu- ficient evaluators, data collectors, data manag- as well as access to those records, thus are relevant stalceholders spell out the resources
ation plan and, especially, for the effort and ers, analysts, and assistants may be required to frequently major resource issues for an evalu- and support needed for the various aspects of
resources that will be required to complete the do a quality job. Even with generous funding, ation. Infonnation that can be confidently ob- the evaluation. It follows that an important
plan. it will not always be easy to obtain the services tained from prograrn records need not be step in the planning process is to canvass such
of sufficient persons with the requisite exper~ sought in a separate, and almost certainly more resources as thoroughly as possible so that
tise. This is why large, complex evaluation expensive, data collection aclministered by the realistic assumptions can be made about them
The Resources Available projects are often dane through contracts with evaluatm: ln some cases, prograrn personnel in the evaluation design (Hatry, 1994). As early
research finns with appropriate personnel on may be provided to compile such data with, of as possible during planning, therefore, the
for the Evaluation hand. course, appropriate monitoring from the evalu- evaluator should meet with a range of prograrn
It requires resources to conduct a prograrn Another critica! resource for an evaluation ator to ensure their integrity. lndeed, evalu- personnel and discuss their role in the evalu-
evaluation. Some number of person-hours de- is support from prograrn management, staff, ations conducted by evaluators intemal to an ation and issues of access to staff, records,
voted to the evaluation activities and materials, and other closely related stakeholders. For in- organization, and therefore already on the pay- clients, and other pertinent infonnation
equipment, and facilities to support data col- stance, the degree of cooperation from prograrn roll, that rely primarily on prograrn records may sources. It is also wise to detemline what pro-
lection, analysis, and reporting must be avail- personnel on certain aspects of data collection be undertalcen with very little direct funding. grarn records are kept, where and how they are
able whether drawn from the existing resources such as opportunity to observe key prograrn Evaluation sponsors and prograrn managers are kept, and what access will be permitted. It is
of the prograrn ar evaluation sponsor ar sepa- activities can have considerable influence on often quite unaware of what will be required of advisable to actually inspect a sarnple of actual
rately funded. An important aspect of planning how much an evaluation can accomplish. Al- tl1em in the course of an evaluation. They may, prograrn records both to try out the procedures
an evaluation, therefore, is to break down the though these factors cannot be easily repre- for instance, be surprised by an evaluator's for accessing and working with them and to
tasks and timelines so that a detailed estimate sented as dollar values, they are valuable re- request to have "hands on" access to records ar determine their completeness and quality. Be-
can be made of the personnel, materials, and sources for an evaluation. Barriers to access and to have prograrn staff undertalce activities in cause of the heavyworkload demands often put
expenses associated with completion of the lackof cooperation from the prograrn, orworse, support of the evaluation. on prograrn personnel, record keeping is not
steps essential to the plan. The sum total of the active resistance, are very expensive to the Prograrn records vary in how easy it is to always a high priority. A prograrn may, there-
resources required must then, of course, fit evaluation effort. It can talce a considerable use them. Records kept in writing are often fore, have a very complete set of fonns and
within what is available ar some changes in arnount of time and effort to overcome these difficult to use without considerable arnounts procedures, but examining their files will reveal
either the plan ar the resources must be made. obstacles sufficiently to complete the evalu- of processing. ln contrast, a management infor- that they are used inconsistently o~ perhaps,
Useful advice on the practicalities of resource ation, not to mention the associated stress for mation system (MIS) consisting ofrecords lcept hardly at all.
54 EVALUAT!ON
Tailoring Evaluations 55

Alongside adequate funding and coopera- that can be realistically planned. With few ex- primary stalceholders. Every program is neces- Program competitors: Organizations or
tion from program personnel, experienced ceptions, the higher the scientific standard to sarily a social structure in which various indi- groups who compete with the prograrn for
evaluators lmow that one of the most precious be met by the evaluation findings, the greater viduals and groups engage in the roles and available resources. For instance, an educa-
resources is time. The period of time allotted the time, expertise, effort, and program coop- activities that constitute the program: Program tional prograrn providing alternative schools
for completion of the evaluation and the flexi- eration that is required. Evaluations to which managers administe~ staff provide service, par- will attract the attention of the public
bility of those time parameters are essential very limited resources are allocated ofnecessity ticipants receive service, and so forth. ln addi- schools because they see the new schools as
considerations in evaluation planning but are must either focus on a circumscribed issue or tion, every program is a nexus in a set of competitors.
rarely determined by the evaluator's prefer- rely on relatively informal procedures for ob- political and social relationships among those Contextual stakeholders: Organizations,
ences. The decisions about the program that taining pertinent information. with an association or interest in the program, groups, individuals, and other social units
the evaluation is expected to inform follow the It takes careful planning to get the scope of such as relevant policymakers, competiog pro- in the irnmediate environment of a prograrn
scheduling imperatives of the policy process. work for the evaluation in balance with the grams, and advocacy groups. These parties are with interests in what the prograrn is doing
Evaluation results often have to be in the hands funding, program cooperation, time, and other typically involved in, affected by, or interested orwhat happens to it (e.g., other agencies ar
of certain decisionmakers by a certain date to essential resources allocated to the project. in the evaluation, and interaction with them programs, public officials, orcitizens' groups
have any chance of playing a role in a decision; Evaluation sponsors who insist on more work in the jurisdiction in which the program
must be anticipated as part of the evaluation.
after that they may be relatively useless. Such than available resources adequately support, or operates).
Who are the parties typically involved in, ar
constraints often set very tight timelines for evaluators who overpromise what they can ac- affected by, evaluations l Listed below are some Evaluationandresearchcommunity: Evalu-
conductiog an evaluation. Further complicat- complish with those resources, are creating a of the stalceholder groups that often either par- ation professionals who read evaluations
ing the situation is a pervasive underestima- situation that will likely result in shoddy work, ticipate directly or become interested in the and pass judgrnenton their tecbnical quality
tion among evaluation sponsors and decision- unfulfilled promises, ar both. It is generally evaluation process and its results: and credibility and academic aod other re-
malcers of how long it takes to complete an better for an evaluation to answer a limited searchers who work in areas related to a
prograrn.
evaluation. It is not uncommon for evaluation number of important questions well than a
sponsors to request an evaluation that encam- larger number poorly. Because evaluation spon- Policymakers and dedsionmakers: Persons
passes an imposing range of is sues and requires sors and other stakeholders often do not have responsible for deciding whether the pro-
grarn is to be started, continued, discontin- Although other parties might be involved in the
considerable effort and then expect results in a realistic notions of the amount of effort and
ued, expanded, restructured, or curtailed. "politics of evaluation," this !ist represents the
matter of a few months. expertise required to conduct quality evalu- stakeholders who most often pay attention to
The trade-offs here are quite signilicant. ation, it is very easy for misunderstandings and Program sponsors: Organizations that initi-
ate and fund the prograrn. They may also an evaluation and with whom the evaluator
An evaluation can have breadth, depth, and conflicts to develop. The best way to prevent
overlap with policymakers and decision- may internet while the evaluation is being con-
rigor bu t will require proportionate funding and this is to negotiate very explicitly with the
mal<ers. ducted ar when its findings are reported. We
time. Or it can be cheap and quick but will, of evaluation sponsor about the resources to be emphasize may: these stakeholders are poten-
necessity, either deal with a very narrow issue made available to the evaluation and the trade- Evaluationsponsors: Organizations thatini-
tial participants in one way ar another in the
ar be relatively superficial (or bothJ. Ali but the offs associated with the inevitable constraints tiate and fund the evaluation {sometimes
the evaluation sponsors and the pmgrarn evaluation process or potential audiences for
most sophisticated evaluation sponsors usually on resources.
sponsors are the sarne). the evaluation. ln any given case, all these
want evaluations that have breadth, depth, and
groups ar only a few may be involved. But
rigor and are cheap and quick. The result is all Target partidpants: Persons, housebolds, or whatever the assortrnent of individuals and
too often both overburdened evaluators work- THE NATURE OF THE other units who receive the intervention or
groups with signilicaot interests in the evalu-
ing frantically against deadlines with inade- EVALUATOR-STAKEHOLDER services being evaluated.
ation, the evaluator must plan to internet with
quate resources andfrustrated evaluation spon- RELATIONSHIP Program managers: The personnel respon- them in some fashion aod be aware of their
sors perturbed about shortfalls and delays in sible for overseeing and administering the concems. Consideration of the appropriate
receiving the product they have paid fm: An intervention pmgram.
form of internction for at least the major stalce-
especially direct relationship exists between One of the matters requiring early attention in holders thus should be part of evaluation plan-
Program staff: Personnel responsible for de-
the time and technical expertise available for the planning of an evaluation is the nature of livering the prograrn services or in support- ning (see Exhibit 2-H for one point of view on
the evaluation and the methods and procedures the relationship between the evaluator and the ing roles. involving stakeholdersj.
Tailoring Evaluations 57
56 EVALUATION

may want to be involved in an ongoing way circumstances at a given time, and to evaluate
with the planning, implementation, and analy- the lessons leamed" (Millett, 1996, p. 68).
sis of results, either to react step by step as the The evaluator's relationship to the evalu-
evaluator develops the project ar to actually ation sponsor ar another stal<eholder desig-
Based on experience working with school ally ali phases of the evaluation; if possible, participate with the evaluator in each step. nated by the evaluation sponsor is so central to
district staff, one evaluator offers the following schedule regular group meetings. Variations on this form of relationship are typi- the evaluation context and planning process
advice for bolstering evaluation use through lnvo/ve stakeho/ders actively: The essential cal for intemal evaluators who are part of the that a somewhat specialized, and not altogether
stakeholder invo\vement: element of stakeholder involvement is that organization whose program is beingevaluated. systematic, vocabulary has arisen in the evalu-
it be aclive; stakeholders should be asked to ln such cases, the evaluator generally works ation profession to describe various circum-
address design issues, help draft survey closelywith management in planning and con- stances. Some of the major forms of evaluator-
ldentify stakeholders: At the outset, define questions, provide input into the final re- ducting the evaluation, whether management stakeholder relationships recognized in this
the specific stakeholders who will be in-
port, and deliberate about ali important as- of the evaluation unit, the program being evalu- vocabulary are as follows:
volved with emphasis on those closest to
pects oi the project. ated, someone higher up in the organization,
the program and who hold high stakes in it.
Estab/ish a structure: Develop and use a con- ar some combination. Or an evaluator from Independent evaluation. The evaluator tal<es
lnvo/ve stakeholders early: Engage stake- outside the organization may be retained as an
ceptual framework based in content familiar the primary responsibility for developing the
holders in the evaluation processas soon as
to stakeholders that can help keep dialogue evaluation consultant to assist the evaluation evaluation plan, conducting the evaluation,
they have been identified because many
focused. This framework should high\ight sponsors in planning and conducting the evalu- and disseminating the results. The evaluator
criticai decisions that affect the evaluation
key issues within the local setting as topics ation but not take the primaryrole in doing that may initiate and direct the evaluation quite
occur early in the process. for discussion so thatstakeholders can share work. autonomously, as when a social scientist un-
lnvo/ve stakeho/ders continuously: The input concems and ideas, identify information
ln some instances, the evaluation sponsor dertakes an evaluation of an interesting pro-
of key stakeholders should be part of virtu- needs, and interpret evaluation results.
will ask that the evaluator work collaboratively gram for purposes of knowledge generation un-
but stipulate that the collaboration be with a der the researcher's own sponsorship ar with
stakeholder group other than the evaluation research funding that leaves the particulars to
SOURCE: Adapted from Robert A. Reineke, "Stakeholder lnvolvement in Evaluation: Suggestions for Practice,"
fva/uafion Practice, 1991, 12(1 ):39-44.
sponsors themselves. For instance, private the researcher's discretion. More ofren, the
foundations that fund social programs ofren independent evaluator is commissioned by a
want an evaluation to be developed in dose sponsoring agency that stipulates the purposes
interaction with the local stakeholders of the and nature of the evaluation but leaves it to the
but otherwise tal<e full responsibility for plan- program. An especially interesting variant of evaluator to do the detailed planning and con-
The process of considering the relation-
ning, conducting, and reporting the evaluation. this approach is when the evaluation sponsor duct the evaluation. ln such cases, howeve~ the
ships with stakeholders necessarily starts with
For instance, gove=ent agencies and other requires that the evaluation be a collaborative evaluator generally confers with a range of
the evaluation sponsm: The sponsor is the
program funders ofren commission evalu- venture in which the recipients of program stal<eholders to give them some influence in
agent who initiates the evaluation, usually pro-
ations by publishing a request for proposals services take the primary role in planning, set- shaping the evaluation.
vides the funding, and mal<es the decisions
(RFP) orrequest for applications (RFA) towhich ting priorities, collecting information, and in-
about how and when it will be dane and who
evaluators respond with statements of their terpreting the results. Part of the philosophy of Participatory ar collaborative evaluation. This
should do it. Various relationships with the
capability, proposed design, budget, and time the W. K. Kellogg Foundation, for instance, is form of evaluation is organized as a team proj-
evaluation sponsor are possible, and their par-
line, as requested. The evaluation sponsor then to avoid being prescriptive in its approach to ect with the evaluator and representatives of
ticular form will largely depend on the spon-
selects an evaluator from among those re- evaluation of the programs it funds. ln the one ar more stakeholder groups constituting
sor's preferences and whatever negotiation
sponding and establishes a contractual ar- words of the foundation's director of evalu- the team (Greene, 1988; Mark and Shotland,
tal<es place with the evaluatm: A co=on situ-
rangement for the agreed-on work. ation, "We believe that people in the co=u- 1985). The participating stakeholders are di-
ation is one in which the sponsor expects the
Other situations are configured so that the nities and institutions we serve are in the best rectly involved in planning, conducting, and
evaluator to function as an independent profes-
evaluator works more collaboratively with the position to mal<e decisions, to implement the analyzing the evaluation in collaboration with
sional practitioner who will receive guidance
evaluation sponsors. For instance, the sponsors programs that are the best suited for their the evaluatorwhose function might range from
from the sponso~ especially at the beginning,
58 EVALUATION
Tailorlng Evaluatlons 59
team leader or consultant to that of a resource will pursue that sponsor's concems under ex-
person to be called on only as needed. One pert guidance from a professional evaluatm:
particularly well-lmown forrn of participatory This process assures that the perspective of the
evaluation is Patton's 11986, 1997) "utiliza- evaluation sponsor will have priority and, given
tion-focused evaluation." Patton's approach a cornpetent evaluatoi; that the results will Two Canadian evaluators who were invalved in
emphasizes close collaboration with those spe- have a certain credibility stemming from the /nstrument design and data collection:
a participatory evaluation of programs spon-
cific individuais who will use the evaluation evaluator's expertise and a decision-rnaking sored by a community economic development Small groups are reconvened to decide on
findings to ensure that the evaluation is re- process that filters the influence of the self- organization offered the following "blueprint" the final wording and format oi questions
sponsive to their needs and produces inforrna- interests of the stakeholders. for the evaluation process they used: retained at the general meeting.
tion that they can and will actually use. There are other situations, howevei; where Data are collected by the program evaluator
the advantages of independent evaluation are lnitiation phase:
and other interested participants who are
Empowerment evaluation. Various proponents not relevant or are outweighed by the benefits Presentation to the board of administrators given appropriate training.
have articulated a concept of evaluator-stake- of a more participatory process. Direct partici- for their approval.
holder interaction that emphasizes the initia- pation by the evaluation sponsors or one or Data analysis and reporting:
ldentification of interested stakeholders.
tive, advocacy, and self-deterrnination of the more other stakeholder groups can ensure that
Data analysis proceeds in small groups, with
stakeholder group iFetterrnan, Kaftarian, and the evaluation results will address their con- the evaluator participating in ali groups.
Sefecting the topics and questions to be
Wandersman, 1996). In this forrn of evalu- cems and be useful and usable for them. More- addressed:
ation, the evaluator-stakeholder relationship is ove~ it can create a sense of ownership in the Individual reports are drafted for each pro-
lnterested stakeholders meet severa] times gram; the evaluator is responsible for writing
participatory and collaborative, as described evaluation that amplifies the signilicance of its
in small groups, each centering on one of reports in consultation with stakeholders.
above. In addition, howevei; the evaluator's findings and reduces its potential to engender
role includes consultation and facilitation di- resistance. And as the empowerrnent theorists the services offered by the organization, to
brainstorm ideas for questions; guidelines in Strategic p/anning:
rected toward development of the capabilities point out, when stakeholder groups with little
Patton (1986) are followed to explain tasks A series of strategic planning meetings is
of the participating stakeholders to conduct formal power are able to conduct and use an such as focusing evaluation questions.
evaluation on their own, to use it effectively for evaluation, it can alter the balance of power in convened to study the evaluation reports

advocacy and change, and to experience some a program context by enhancing their influence Questions are rephrased clearly, regrouped and decide on follow-up steps.
for each program, and collated into one
sense of contrai over a program that affects and sense of efficacy. It is thus appropriate for The evaluation questions are revised for
document.
their lives. The evaluation process, therefore, the evaluation sponsors and the evaluator to future use on an ongoing basis; program
is not only directed at producing inforrnative give explicit consideration to the question of A general meeting of stakeholders is called workers are expected to coordinate future
and useful findings but also at enhancing the how the evaluation responsibilities are to be to prioritize questions according to their evaluation efforts.

self-development and political influence of the assigned and the arrangements for organizing potential utility and to plan how the evalu-
participants. AB these themes irnply, empower- the evaluator-stakeholder interactions. Where ation results will be utilized once they are
available.
ment evaluation most appropriately involves such deliberation has not already taken place
those stalceholders who otherwise have little or an arrangernent stipulated, it may be con-
power in the program context, often the pro- structive for the evaluator to raise the issue and ~O~RCE: ~dapted from Danielle Papineau and Margaret C. Kiely, "Participatory Evalualion in a Community Organiza-
t1on. Fostenng Stakeholder Empowerment and Utilizalion," Evaluaon and Program Planning, 1996, 19(1 ): 79. .
gram recipients or intended beneficiaries. suggest that it be a matter of discussion during 93

A significant contribution of the participa- the earliest phase of evaluation planning. Ex-
tory and empowerrnent perspectives is to cal! hibit 2-I illustrates what might be in store for
into question what might otherwise be a rou- stalceholders who opt to participate in a collabo-
tine presumption that an independent evalu- rative evaluation. the nature of the decision making, the evalu-
ation is appropriate. There are, of course, many Whether the evaluation is planned and tion of recognizable evaluation concepts and
ator 's role, and, most likely, the focus and methods to a particular program. We thus dis-
situations in which the evaluation sponsor ex- conducted by an independent evaluator or by a character of the evaluation. The resulting proj-
plicitly wants an independent evaluation that team of stakeholders has considerable effect on tinguish the process ofworkingwith stakehold-
ect, nonetheless, should represent an applica- ers, whether as an independent evaluatoi; col-
60 EVALUATION
Tailoring Evaluations 61

laborato~ facilitato~ or resource person, from may be relevant for different stakeholder
the evaluation plan that results from that pro- groups, depending on their interest in the pro-
cess. That plan may be developed and largely grarn and their decision-malcing roles. More-
settled early in the process or may emerge ove~ that information might be communicated
piecemeal as tl1e process develops and evolves, as soon as available or !ater when alI phases of Torres, Preskill, and Piontek (1996) surveyed
and interviewed members of the American lt is important to use varied formats for
but the features of a good plan for the evalu- the evaluation are complete, and it might be
Evaluation Association about their experiences communication. These might include short
ation context and the prograrn at issue can be communicated in writing or verbally and for-
communicating with stakeholders and reporting reports and summaries, verbal presen-
considered separate from the process through mally or informally. Typically, the evaluator
evaluation findings. The respondents identified tations, and opportunities for informal inter-
which tlie planning and iniplementation is should consider multiple communication the following elements of effective commun- actian.
dane. ln the remainder of this chapte~ there- events ranging from informal oral briefings to cation:
fore, we will discuss general planning issues formal written reports. The objective of the The content af the communication should

communication and dissemination plan Ongoing, collaborative communication be tailored to the audience and be easy for
and, when reference is made to the evaluator's
processes were the most successfu/. Peri- them ta understand. Communication
role, assume tl1at can mean either an inde- should be to report the findings of most interest
odic meetings and informal conversations shauld use clear language, graphs and
pendent evaluator or a collaborative tearn. to each stalceholder as soon as possible and
can be used to maintain close contact charts, and vivid, concrete il/ustrations. lt
First, howeve~ one other aspect of evalu- proper and in forms that are easy to understand throughout the evaluation, and interim shauld present contextua[ infarmation
ator-stal<eholder interaction must be ad- and use lsee Exhibit 2-J). memos and draft reports can be used to about the program and the evaluation
dressed: tl1e communication and dissemina- Evaluation conducted for purposes of pro- convey findings as they develop. cover bath positive and negative findings:
tion of the evaluation findings. Even in the grarn iniprovement, for instance, might in- and be specific about recommendations.
most participatory evaluation, there will be clude regular briefings for the evaluation spon-
stalceholders who have not been dilectly in- sor and prograrn managers conducted as soon SOURCE: ~dapted fro~ Rosalie T. Torres, Hallie 5. Presklll, and Mary E. Piontek, Evaluarion Strategies for Communicating
and Reporling: Enhancmg Learning in Drganizations (Thousand Oaks, CA: Sage, 1996), pp. 4-6.
volved who will want to lmow the results. For as each distinct data collection reaches a point
an independent evaluation, of course, the where tentative analysis and interpretation are
evaluation sponsor may be arnong those stalce- possible. These might be relatively informal
holders. For evaluation to be useful, a necessary briefings in which the evaluator presents a be especially controversial. It should be noted
pictorial displays, uses engaging audiovisual
step is that its findings be communicated to verbal SUillillarywith supporting handouts and that tls approach does not necessarily mean
materiais, and personalizes portions of tlie
tl10se wit11 interest in tlie prograrn, especially encourages discussion. Otlier stalceholders being highly secretive about tlie evaluation
story as much as possible through well-chosen
to those witli responsibility for malcing inipor- might receive sinillar interim briefings in writ- findings until the final report is delivered. The
anecdotes and case examples. To have such
tant decisions about the prograrn. It is dillicult ten or verbal format at less frequent intervals. process ofverifying and analyzing tlie informa-
material available when needed, tlie evaluator
to communicate evaluation findings in fine At tlie conclusion of tlie evaluation, a written tion in such formal circumstances might quite
must plan for its development during tlie
detail, and additionally, there is often inherent report might be prepared for the record and for appropriately include soliciting tlie reactions of
course of tlie evaluation. For instance, it may
uncertainty about what information will be of tlie more peripheral stalceholders or might not potential critics and otlier iniportant stalce-
be appropriate to malce audio or video record-
most interest to stalceholders at the time the if no use for it was apparent. Evaluation con- holders to tlie initial SUillillaries of the major
ings of certain events or situations, systemati-
evaluation is completed. It is usually best, ducted for purposes of accountability, on the findings and incorporating tlieir feedback in tlie cally collect anecdotes and case examples from
tl1erefore, to discuss this issue witli the major otlier hand, might properly involve a more final report.
which to select representative instances, and
stal<eholders and develop an organized com- formal communication and dissemination pro- Whatever tlie schedule, form, and audi-
malce otlier such preparations for effective com-
munication and dissemination p]an from the cess throughout. Information might be re- ences for evaluation findings, it is wise to in-
munication. Useful advice for planning effec-
beginning. leased only after it was thoroughly verifi.ed and clude some consideration of tlie communica-
tive commu.nication and dissemination activi-
As a general frarnework, the communica- analyzed and primarily in carefully worded tion media and materiais in the planning.
ties is Iound in Torres, Preskill, and Piontek
tion and dissemination plan should indicate written forra with verbal briefings used only as Evaluation findings, lilce the prograrns tliey (1996; also see Exhibit 2-J).
what information from tlie evaluation is to be a supplement. This higher levei of caution and describe, are rarely sinlple and easily under-
For many evaluations, it is also appropriate
communicated to which stalceholders in what forraality would be justified if the stalces were stood. Communication will often be most ef-
to allow stal<eholders access to tlie database on
forra and at which time. Dillerent information high or tlie evaluation results were expected to fective if it malces good use of graphical and
which tlie evaluation was based, at tlie sarne
62 EVALUATION Talloring Evaluations

time sa.feguarding the privacy of sources from tions are the obvious starting point for defining Chapter 3 is devoted entirely to discussing the
methods to address each type of evaluation
whom the data were obtained. Making a the questions around which the evaluation will forra they should take, how they are generated,
question. Evaluators use these schemes to or-
dataset public signals to stakeholders that the be desigued but usually should not be taken as and how they are winnowed organized and
. '
mtegrated to provide the structure for the
' g:imze their thinking about how to approach
evaluators have nothing to hide and allows final for purposes of evaluation planning. Often different prograrn evaluation situations. For
them to try altemative modes of analysis to the questions presented at this stage are too evaluation design. For present purposes, wewill
planning purposes, an evaluator will typically
verify that the evaluator's findings were prop- general ar abstract to function well as a basis assume that an appropriate set of evaluation
select the general evaluation approach that cor-
erly drawn. We believe that malng data pub- for evaluation planning. Or the questions, as questions has been identified and consider
responds to the types of questions to be an-
licly available should be a routine procedure in worded, may be beyond the capability of the some of the broader implications of their char-
swered in an evaluation, then tailor the particu-
evaluations of large-scale or very important evaluator to answer within the operative con- acter for tailoring and planning the evaluation.
lars to the specifics of the questions and the
programs. straints on time, resources, available informa- In particul~ the evaluation questions to be
program situation. To complete our discussion
tion, and organizational ar political arrange- answered for a given program will, of necessity,
of tailoring evaluations, therefore we must
be very specilic to the idiosyncratic nature of
ments. introduce the co=on evaluation ~pproaches
EVALUATION QUESTIONS AND Any initial description of what the evalu- that program. They will ask such things as
ar schemes and review the circumstances in
ation sponsors have in mind, therefore, must "How many of the households that fall below
EVALUATION METHODS which they are most applicable.
usually be further explored, refined, and aug- the federal poverty line in the Fairview School
The co=on conceptual and methodo-
mented to obtain a set of meaningful, appropri- District need a.fterschool care for school-aged
logical frarneworks in evaluation correspond to
A program evaluation is essentially an informa- ate evaluation questions around which the children between 3 and 7 p.m. on weekdaysl"
the types of frequent evaluation questions as
tion-gathering and -interpreting endeavor that evaluation can actually be planned. In addition, ''What proportion of the juveniles on probation follows: '
attempts to answer a specilied set of questions it is usually useful for the evaluator to analyze have at least three contacts with their probation
about a program's performance and effective- the program independently and derive evalu- officer per month for the full six-month proba-
ness. An important step in designing an evalu- ation questions thatmay not otherwise arise so tionary periodl" and ''What nutritional bene- Needs assessment: answers questions about
ation, therefore, is determining the questions that they too can be considered during the fits does the meals-on-wheels program at the the social eonditions a program is intended
the evaluation must answeL This is some times planning process. However accomplished, a senior citizens' center have for the housebound to address and the need for the program.
dane in a very perfunctory manne~ but we thorough effort must be made to generate a set frail elderly in its catchment areal" Beyond the Assessment of program theory: answers
advocate that it be given studious and detailed of candidate evaluation questions that covers specifics, howeve~ evaluation questions fall questions about program conceptoalization
attention. A carefully constructed set of evalu- all the issues of potential relevance to the con- into recognizable types according to the pro- anddesign.
ation questions gives structure to the evalu- cems of the evaluation sponso~ the decision- gram issues they address. Five such types are
Assessment of program process (ar process
ation, leads to ppropriate and thoughtful plan- makers who will use the findings, and other readily distinguished:
evaluation): answers questions about pro-
ning, and serves as a basis for informative signilicant stakeholders. This approach allows gram operations, implernentation aod ser-
discussions about who is interested in the an- the evaluation design to be responsive to the Questions about the need for program ser- vice delivery. '
swers and how they will be used. Indeed, con- needs of decisionmakers and offers the poten- vices
Impact assessment {impact evaluation ar
structing such questions and planning how to tial to involve stakeholders as collaborators in Questions about program eonceptualization outcome evaluation}: answers questions
answer them is the primary way in which an the evaluation process. It is relatively easy to ar design about program outcomes aod impact
evaluation is tailored to the unique circum- generate questions, howeve~ so the initial set Questions about program operations and Efficiency assessment: answers questions
stances associated with each program that resulting from a diligent effort will likely be too service delivery
about program eost and eost-effectiveness.
comes under scrutiny. large for the evaluation to answer them all. The
Questions about program outcomes
Generally, the evaluation sponsor puts for- evaluato~ evaluation sponso~ and other key
ward some initial evaluation questions when stakeholders must therefore impose priorities Questions about program cost and effi-
ciency These forms of evaluation are discussed in
proposing ar commissioning an evaluation o~ to select a workable number dealing with the detail inlater chapters of this volume (Chapters
in the case of a competition to select an evalu- most important issues. 4-11). Here we will only provide some guidance
ato~ as part of the RFP or RFA that goes out to Because the evaluation questions to be ad- Evaluators have developed relatively dis-
regarding the circurnstances for which each is
prospective evaluators. Those initial declara- dressed are so pivotal to evaluation planning, tinct conceptual frarneworks and associated most appropriate.
Tailoring Evaluations 65
64 EVALUATION

to the actual needs of the target participants


Needs Assessment and to provide guidance for improvement.
Needs assessments may talce the form of
The primary rationale for initiating ar
finding out the needs of a potential target popu- A representative sample of 1,260 homeless more often reported needs for help with drinking
maintaining a social program is a presenting ar
lation as they perceive them. For example, men and women were interviewed in New York problems, drug problems, /earning how to
incipient social problem-by which we mean
homeless persons may be queried about the City's municipal shelters for single adults to handle money, getting veterans benefits
socially recognized deficiencies in the social determine their perception of their needs. The problems with the police, getting along bette;
kinds of services forwhich they feel the greatest
conditions-that legitimate social agents en- interview covered 20 items, each indicating with other people, and finding a place to live.
need (e.g., see Exhibit 2-K). Alternatively, the
deavor to remedy. The impetus for a new pro- need for help in a particular area. Most Women more frequently reported needs for help
objective of needs assessment may be to de-
gram to increase literacy, for example, is likely respondents identified multiple needs, averaging w1th health and medical problems and learning
scribe conditions in such a way that the ser-
to be recognition that a significant number of 6.3. The need for help in finding a place to live self-protection skills. The evaluators pointed out
vices needed to alleviate them can be inferred.
persons in a given population are deficient in and having a steady incarne were the most that for programs to be truly responsive to these
Needs assessments may be conducted commonly cited needs overall, closely followed multiple needs, they must have the capacity to
reading skills. Sirnilarly, an ongoing program
through surveys of knowledgeable inforrnants, by the need for help in finding a job and deliver or broker access to a comprehensive
may be justified by the persistence of a social
such as personnel of service agencies ar poten- improving job skills. Compared to women, men range of services.
problem: Driver education in high schools re-
tial service recipients, that focus on perceived
ceives public support because of the continuing
problems and needs, services desired, and
high rates of automobile accidents among ado-
shortcomings of existing services. They may
lescent drivers. also analyze demographic and social indicatar
If there is no significant problem ar no
data from such sources as the U. S. Census ar
perceived need for intervention, there is gener-
data Iram local agencies that describe the avail-
ally no basis for affinning the value of a pro-
ability of services and pattems of current use.
grarn that purports to address this nonproblem. it. Put another way, every social program is so~abl~, ~easible, ethical, and otherwise appro-
The resulting descriptions of social problems, based on some planar blueprint that represents pnate it is. Assessment of program theory is
One important form of evaluation, therefore,
service utilization, and perceived needs must the way it is "supposed to wark" according to most essential to prograrns during their early
assesses the nature, magnitude, and distribu-
then be assessed against some set of social those who understand its history, purposes stages, for exarnple, when they are new ar in
tion of a social problem; the extent to which
norrns ar some view of desired conditions held and activities the best. This plan is rarely writ'. pilot testing ar even earlierwhen they are in the
there is a need for intervention to address it;
by social agents ar those experiencing the prob- ten out in complete detail, and may not be planning stage. Howeve~ it is also applicable to
and the implications of these circumstances for
lems to evaluate their magnitude, seriausness, written out at all, but exists nonetheless as a established prograrns, especially when ques-
the conceptualization and design of the inter-
and actionable implications. Chapter 4 of this shared conceptualization among the principal tions arise about how well matched their ser-
vention. These diagnostic activities are often
book discusses the various aspects of needs stalreholders. Because this program plan con- vices are to the social needs they are attempting
referred to as needs assessment in the evalu-
ation field but overlap what is called social assessment in detail. sists essentially of a set of assumptions and to meet. The sponsars of this forrn of evalu-
epidemiology and social indicatars research in expectations about how the program should ation are generally those attempting to launch
other fields (McIGllip, 1987; Reviere et al., conduct its business and attain its goals, wewill a new program, such as the funding agency ar
1996; Sariano, 1995; Witkin and Altschuld, Assessment of Program Theory refer to it as the prograrn theory {discussed adrninistratars, ar those seeking assurance that
1995). Needs assessment is often used as afirst more fully in the next chapter). If this theory is the conceptualization and design of a program
Given a recagnized problem and need for are appropriate for its purposes. Exhibit 2-L, for
step in designing and planning a new program faulty, the intervention will fail no matter how
intervention, it does not follow that any pro- elegantly it is conceived ar howwell it is imple- exarnple, describes an exarnination of the con-
ar restructuring an established prograrn to pro-
grarn, willy-nilly, will be appropriate forthe job. ceptual foundation for farnily preservation pro-
vide information about what services are mented (Chen, 1990; Weiss, 1972).
The conceptualization and design of the pro- Assessment of the program theory in- grarns, which indicated that they had little
needed and how they might best be delivered
grarn must reflect valid assumptions about the prospect for success.
to those who need them. Needs assessment is volves, first, representing it in explicit and de-
nature of the target problem and represent a Evaluation of program theory rests on the
also often important for established, stable pro- tailed written ar graphical forrn. Then, various
well-founded and feasible approach to resolving presumption that the need for the program and
grams to examine whether they are responsive approaches can be used to examine how rea-
Tailoring Evaluations 67
66 EVALUATION

Assessment of Program Process of prograrn resources, and other such matters


(Exhibit 2-M provides an example).
Given a plausible theory about how to fa- ln a typical process evaluation, criteria are
vorably intervene to arneliorate accurately di- developed for the prograrn functions viewed as
program managers, and operating-level staff on agnosed social problems, a prograrn must still
As part of an evaluability assessment (see criticai in two ways. These may be configured
four key dimensions: (a) program goals, (b) as- be implemented well to have a reasonable pros-
Chapter 5), evaluators working under contract in the form of a "blueprint" of the intended
pects of the child welfare system that affect the
to the U.5. Department of Health and Human pect of actually affecting the target problem. prograrn design that depicts the functions, ac-
programs, (c) the target population, and (d) the
Services reviewed the design of family Many prograrns are not implemented and exe- tivities, and service transactions that the pro-
characteristics that distinguish FPPs from other
preservation programs (FPPs). FPPs are tim': cuted according to their intended design. A grarn is supposed to accomplish in, perhaps,
home-based services. Based on their own
limited, intensive home-based services to fami- prograrn may simply be poorly managed ar be flowchart form (a version of prograrn theory;
ana\ysis and discussions with an expert advisory
lies in crisis that are intended to prevent the
committee, the evaluators concluded that as cur- compromised by poltica! interference. Some- see Chapter 3). Dr the criteria may be stated in
placement of children in foster care. The eva~
rently designed, family preservation programs times personnel are not available ar facilities the form of specific administrative ar service
uators held discussions with the staff of federal
could not achieve the policymakers' primary are in disrepair; sometimes project staff cannot objectives, for example, to emoll 20 new clients
and national private sector agencies about the
goal of preventing placement in foster care. The carry out the prograrn dueto Jack of motivation each week, to provide a minimum of ten ses-
definition of FPPs, reviewed available literature,
major flaw found in the program design was the ar expertise. Often the prograrn design is not sions of service to each client within the first
obtained descriptions of state and local
practical difficulty of identifying children at well structured, leaving much roam for inter- three months, to have 90% of the clients receive
programs, and made site visits to four programs.
"imminent risk 11 of placement; this meant that
From this information they developed "models" pretation, ar the original prograrn plan may not the full term of service, and to mal<e educa-
programs could not consistently target families
of how the programs were supposed to operate be transmitted well to staff so that prograrn tional presentations to one co=unity group
and then obtained the views of policymakers, with children truly at risk of placement.
activities drift over time. Possibly, the intended each week. Such criteria can be developed in
f 1 fE 1 !' "inHandbook prograrn participants do not exist in the num- various ways: They may simply be stipulated
SOURCE: Adapted from jo~eph S. Wholey,h"AlssesHsin~ t~e ;easib~i~ ~~~:~~~:~ ~:~s;~nc~:C~~ J~~'ey-Bass, 1994), bers reqllired, cannot be identified precisely, ar by prograrn administrators, they may be man-
of Practica/ Program Evaluauon, ed. J. S. W o ey, . . a ry, an
pp. 29-31. Wholey's account, in tum, is based on Kaye and Bell {1993). are not cooperative. For example, some pro- dated by prograrn funders, they may be derived
grarns to serve children with congenital heart from studies and reports of other prograrns ar
conditions found it so clifficult and costly to follow the specification of some model pro-
!acate potential clients that insufficient funds grarn, they may result from a process of reflec-
remained for providing the intended treat- tion and goal setting by prograrn personnel ar
involves making that theory explicit so there
adequate diagnosis of the problem the prograrn ments. other stakeholders, ar they may be drawn from
will be little uncertainty about significant as-
is to address has already been established ar can A central and widely used form of evalu- accepted principies of organizational effective-
pects of the prograrn concept and intended
confidently be assumed. Also, because analysis ation, therefore, assesses the fidelity and effec- ness ar professional practice.
implementation. Stakeholder disagreement
of prograrn design reqllires a dose collaboration tiveness of prograrn implementation. Imple- Wlth the criticai prograrn functions and
over such matters raises uncertainty and may
arnong the evaluator, prograrn designers and mentation assessment evaluates program corresponding performance criteria identified,
also create a politically fluid situation in which
managers, and other key stakeholders'. it is process, the activities and operations of the the other component of process evaluation is
fundamental prograrn changes are possible.
most readily accomplished when ali parues are prograrn. For this reason, it is co=only called the definition and operationalization of perfor-
These conditions are sufficiently similar to
willing to be fully engaged in the process and process evaluation ar, when it is an ongoing mance measures that describe prograrn accom-
those of a new prograrn still under design to
can establish constructive working relation- function, program monitoring. Process evalu- plishments in relation to the respective criteria.
permit methods for assessing prograrn theory
ships. Somewhat paradoxically, however, many ation addresses questions related to how well Thus, data collection procedures might be put
to be potentially helpful. ln this case, however,
of the tecbn.iques associated with the evalu- the prograrn is functioning. It might include in place to determine the number of new pa-
the context of application may be distinguished
ation of prograrn theory are applicable in si~ assessment of how congruous the services are tients emolled each week, the percentage who
more by hostility arnong stakeholder groups
ations of stakeholder conflict, for example, dis- with the goals of the prograrn, whether services complete a full term of service, the number of
than cooperative working arrangements, pos-
agreement about prograrn goals and objectives, are delivered as intended to appropriate recipi- presentations to co=unity groups, and the
ing special challenges (and hazards) to the
appropriate priorities, and the nature of pro- ents, howwell service delivery is organized, the like. Prograrn performance can then be as-
grarn activities. Assessment of prograrn theory evaluatm: effectiveness of prograrn management, the use sessed by comparing what is found on these
Tailoring Evaluations 69
68 EVALUAT!ON
what was paid for and deemed desirable was failed to produce the expected effects. On the
actually accomplished. other hand, when program effects are found,
Process evaluation of some variant is the process evaluation helps confirrn that they re-
assessment approach most frequently applied sulted from program activities, ratherthan spu-
transaction expected between welfare workers to social programs. It is used both as a free- rious sources, and identify those aspects of
Work Pays is a state-\evel we\fare reform standing evaluation and in conjunction with service most instrumental to producing the
and their c\ients under the new po\icy was
demonstration program in California designed to
exceedingly rare. ln more than 80% of their impact evaluation as part of a more compre- effects so that program managers know where
establish incentives to work and disincentives for
interviews with clients, workers did not provide hensive evaluation. AB a freestanding evalu- to concentrate their efforts.
staying on the AFDC welfare program. ~he
and interpret information about the new po\icy. ation, it yields quality assurance information. As a general evaluation approach, process
program administrators recogn1zed that t~ realize
Most workers continued their routine patterns of That is, it assesses the extent to which a pro- evaluation is widely applicable to social pro-
the policymakers' intent, the workers 1n local
collecting and verifying eligibility information and gram is implemented as intended and operat- grams. For stable programs that have estab-
we\fare offices would have to inform their clients
providing scripted recitations of we\fare rules.
about the new policy and present this ing up to the standards established for it. When lished operating procedures, personnel, and fa-
However the evaluators also found that the
information in a positive, individualized way that the program model employed is one of estab- cilities, process evaluation may provide
workers had been given only minimal information
would reinforce clients' understanding of their
about the Work Pays program and no additional lished effectiveness, a demonstration that the summative information relevant to both pro-
ob\igations and choices about work and welfare. model is well implemented is presumptive evi- gram accountability and knowledge genera-
time ar resources for educating their large
An implementation assessment wa.s the~efore dence that the expected outcomes are produced tion. For new programs or those in flux, process
caseloads about the changes. These findings
conducted in which researchers 1nterv1ewed as well. When the program is new, a process evaluation may constitute a formative evalu-
demonstrated that welfare reform was not fully
we\fare workers about the Work Pays program evaluation provides invaluable feedback to ad- ation that provides useful feedback for program
implemented at the street levei in Ca\ifornia and
and observed a number of meetings with clients.
revealed some of the reasons why it was not. ministrators and other stakeholders about the managers attempting to improve program op-
This information revealed that the type of
progress that has been made operationalizing erations. ln either case, however; process evalu-
1 d Karin MacDonald "On the Front Unes ofWelfare Delivery: the program theory. From a management per- ation requires a well-defined, consensual pro-
SOURCE AdaptedfromMarciaK.Meyers,Bonn1eGaser,an . dM ' ~ 1998 17(1)1-22
Are Workers lmplementing Policy Reforms?" foumal o( Policy Analys1s an anagemen , spective, process evaluation provides the feed- gram theory that stipulates the "program as
back that allows a program to be managed for intended." If program managers and other per-
high performance (Wholey and Hatry, 1992), tinent stakeholders cannot delineate the pro-
and the associated data collection and reporting gram model that is supposed to be imple-
ter the program. Managers who develop repu- of key indicators may be institutionalized in the mented, or cannot agree on the intended
measures with the criterion for desired perfor- form of an MIS to provide routine, ongoing clientele, services, and procedures, then there
tations for wasting funds, using staff resources
mance on that program function.
inappropriately, and being inefficient in oth~r performance feedback. is no defined process for the evaluator to ob-
Although process evaluation is often dane
regards frequently jeopardize not onl~ therr ln its other co=on application, process serve and assess. ln this case, the evaluator may
as a one-shot evaluation study, say, for one evaluation is an indispensable adjunct to im- adopt the techniques for assessing program
own positions but the futures of therr pro-
cohort of program clients, it should be apparent pact evaluation. The information about pro- theory and work with program managers,
that similar procedures can be used routinely grams. .
Also, program monitoring informanon gram outcomes that impact evaluation pro- evaluation sponsors, and other stalceholders to
as a management too!. When set up to provide vides is incomplete and ambiguous without better define the program conceptualization.
systems give program administrators a power-
periodic performance data for key. program knowledge of the program activities and ser- Process evaluation and its variants are de-
ful too! for documenting for program sponsors
functions on a continuous bas1s, this form of vices that produced those outcomes. When no scribed in greater depth in Chapter 6 of this
and stakeholders the operational effectiveness
assessment is generally known as program impact is found, process evaluation has signili- volume. It is important to note here, howeve~
of the organization, justifyingthe ways staff are
monitoring or performance monitoring. ~her_e cant diagnostic value by indicating whether that although it is widely used as a freestancling
deployed, requesting further support, and de-
are many good reasons for programs to msn- evaluation approach to give evaluation spon-
fending the program's performance co'."pared this result occurred because ofimplementation
tute monitoring schemes. For instance, mom- sors, program managers, and other stakehold-
with its competitors. Routinely collectmg and failme, that is, the intended services were not
toring provides a way for program managers to provided hence the expected benefits could not ers an assessment of how well the program is
reporting program performance information,
ensure that the day-to-day operations of a pro- have occurred, or theory failme, that is, the implemented, it is not a substitute for impact
therefore, makes the program accountable and
gram are conducted appropriately and .e~ program was implemented as intended but evaluation. Process evaluation does not address
provides evidence to funders and sponsors that
ciently and thus helps them properly adnurus-
70 EVALUATION
Tailoring Evaluations 71

the question ofwhether a program produces the pact of the intervention uncontarninated by the
intended outcomes and benefits for its recipi- influence of the other processes and events that
ents. Even in cases where the program model also affect the conditions the program attempts
is lmown to be effective in other applications to change. To conduct an impact assessment,
Taiwan is a high-density island country with a process evaluation established that the program
and process evaluation demonstrates that it is the evaluator needs a plan for collecting data
garbage problem. Garbage accumulation has was implemented as planned.
well implemented, there remains a possibillty that will permit a persuasive demonstration increased exponentially in recent years, 26 riv-
that circumstances are sufficiently dillerent in that observed changes are a function of the lhe impact assessment was conducted by
ers are polluted by garbage, and the number obtaining records of the dailyvolume oi garbage
the program at issue to keep it from being intervention and cannot readily be accounted of landfill sites is increasingly limited. Conse- for Nei-fu and the similar, adjacent suburb of
effective there. for in other ways. This requires a careful speci- quently, in 1993 a demonstration garbage Nan-kan for a period beginning four months
fication of the outcome variables on which reduction program (GRD) was launched in prior to the program onset and continuing four
program effects rnay occw; developrnent of Nei-fu, a suburb of Taipei, and evaluated for its months alter. Analysis showed no reduction in
measures for those variables, and a research impact on the amount of waste produced.
IIllpactAssessillent design that not only establishes the status of Garbage is collected daily in Taiwan and the plan
the volume of garbage collected in Nei-fu during
the program period relative to the preprogram
An impact assessment, sometimes called program recipients on those measures but also of the GRD was to disrupt this routine by volume ar that in the comparison community.
an irnpact evaluation or an outcome evalu- estimates what their status would be had they suspending Tuesday collections. The theory was The evidence indicated that residents simply
that requiring residents to st~re garbage one day
ation, gauges the extent to which a program not received the intervention. Much of the saved their customary volume of Tuesday gar-
a week in their homes, which are ili equipped for bage and disposed of it on Wednesday, with no
produces the intended irnprovements in the cornplexity of irnpact assessment is associated
that function, would create sufficient incon- carryover effects on the volume for the re-
social conditions it addresses. The evaluation with obtaining a valid estimate of the latter venience and unpleasantness to raise aware-
questions around which impact assessment is status, known as the counterfactual because it mainder of each week. lnterviews with residents
ness of the garbage problem. As a resul~ it was revealed that the program theory was wrong
organized relate to such matters as whether the describes a condition contrary to what actually expected that residents would make efforts to -they did not report the inconvenience ar
desired program outcomes were attained, happened to program recipients. Specific im- reduce the volume of garbage they produced. A unpleasantness expected to be associated with
whether the program was effective in producing pact assessment designs vary considerably. storing garbage in their homes.
change in the social conditions targeted, and Sometimes it is possible to use classic experi-
whether program irnpact included unintended mental designs in which contrai and experi- SOURCE: Adapte.d from Huey-Tsyh Chen, Juju C. S. Wang, and Lung-Ho Un, "Evaluating the Process and Outcome of
side effects. These questions assume a set of mental groups are constructed by random as- a Garbage Reduct1on Program ln Taiwan," Evalualion Review, 1997, 21 (1 ):27-42.

operationally defined objectives and criteria of signment and receive dillerent interventions.
success. The objectives may be social-behav- For practical reasons, howeve~ it is often nec-
ioral ones, such as lowering functional illiter- essary to ernploy statistical approaches to
acy or nutritional deficiencies among children; isolating program effects rather than true evaluation, it is often referred to as black box tended effects. On the other hand, impact as-
theyrnay be co=unity related, such as reduc- experiments. Thus, nomandomized quasi- ! evaluation because the evaluator may Jearn sessment is characteristically very dernanding
ing the frequency of certain crimes; or they may experiments and othernonexperimentalrneth- 1! what the program effects are but does not lmow of expertise, time, and resources and is often
be physical, such as decreasing water pollution ods are co=only employed in impact assess- ! anything about the program processes that very difficult to set up properly within the
!
or the amount of litter on city streets. Irnpact ments. With proper safeguards and appropriate produced those effects-the program is a blacl< constraints of routine program operation. For
assessrnents are essential when there is an qualifications, such nonexperirnental designs box into which the evaluation cannot (or does these reasons, a full-blown impact assessrnent
interest in deterrnining if a program is effective may provide reasonable estimates of effects not) see.
is not to be undertal<en lightly. It is generally
in its efforts to ameliorate a target problem, (Exhibit 2-N describes such a situation). Deterrnining when an impact assessment appropriate only when there is an irnportant
comparing the effectiveness of different pro- AB mentioned.above, irnpact assessment is 1 is appropriate, and what evaluation design to purpose to be served by learning about program
grams, or in testing the utility of new efforts to often combined with process evaluation so that use when it is, present considerable challenge effects. This may be because the program con-
address a particular co=unity problem. a linlcage can be made between program irnple- to the evaluatOL On the one hand, evaluation cept is innovative and promising ar in ciicum-
Irnpact assessrnent has the basic airn of mentation and the program outcornes ob- 1 sponsors often believe that they need an irnpact stances where identifiable decisionrnakers
producing an estimate of the net effects of an
intervention-that is, an estirnate of the irn-
served. When an irnpact assessment is con-
ducted without any semblance of a process
! evaluation and, indeed, it is the only way to
determine if the program is having the in-
have a high likelihood of actually using evi-
dence about program irnpact as a basis for

L
72 EVALUATION
Tailoring Evaluations 73

signilicant action. Such conditions may occm; Efficiency Assessment


for instance, with a demonstration project set
up to test a program concept that, if effective, Unless programs have a demonstrable im-
will be disseminated to other sites. Or impact pact, it is hard to defend irnplementing or
assessment may be called for in high-account- maintaining them-hence the need for irnpact
lf provided with supportive services, persons
ability contexts where, for instance, continued assessments. But knowledge of impact alone is combined with the direct cosi of the community
with mental disabilities can often be maintained program itself. Costs for wards where patients
funding for a program depends on its ability to often insufficient; program results must also be in community settings rather than state mental
demonstrate impact. judged against their costs. This is especially resided 90 ar more days were gathered Iram the
hospitais. But is such community treatment
If the need for outcome information is true in the present political climate as the Ohio Department of Mental Health budget data
more costly than residential hospital carel A and subdivided into categories that corres-
sufficient to justify the expense and effort of an resources for supporting social programs are team of researchers in Ohio compared the costs
ponded as closely as possible to those tabulated
impact assessment, there is still a question of curtailed and competition among programs for of a community program that provides h>using
for the community program participants. Mental
whether the program circumstances are suit- funds is intensilied. Some programs may not subsidies and case management for state-
health care comprised the largest component of
able for such an evaluation. For instance, it be supportable because of high costs relative to certified severely mentally disabled clients with
service cost for both program and hospital
malces little sense to establish the irnpact of a their impact. ln the face of budget problems, lhe costs of residential patients at the regional clients. Overall, however, the total cosi for all
program that is not well structured OI cannot for example, some universities have termi- psychiatric hospital. Program clients were
services was estimated at $1,730 per month for
be adequately described. Even if positive effects interviewed monthly for more than two years to
nated their student counseling programs be- the most intensive version of community pro-
are found under such circumstances, ambigu- determine their consumption of mental health gram services and about $6,250 per month for
cause costs are high and benefits slight. Other
services, medical and dental services, housing
ity remains about what program features initiatives may be expanded, retained, OI ter- residentia/ hospital care. Community care, there-
services, and other personal consumption. lnfor-
caused them or how they would be replicated minated on the basis of their comparative fore, was much less costly than hospital care, not
mation on the cost of those services was ob- more costly.
elsewhere. Impact assessment, therefore, is costs. For instance, findings about the irnpact tained from the respective service providers and
most appropriate for mature, stable programs of institutional versus co=unity care for ado-
with a well-defined program model and a clear lescent offenders suggest that co=unity pro-
S~URCE: Adapted from Geo~ge C Gal~ter, Timothy F. Champney, and Yolonda Williams, "Costs of Caring for Persons
use for the results that justifies the effort re- grams are preferable because of their markedly W1th long-Term Mental lllness rn Alternat1ve Residentia[ Settings,1' Eva/uation and Program Planning, 19 ( ): _ .
941 17 3 239 348
quired to conduct this form of evaluation. The lower costs.
most useful impact assessment results are for Talcing account of the relationship between
well-structured, well-documented programs costs and effectiveness requires efficiency as-
believed to have important effects that must be sessments IExlllbit 2-0 provides an example}.
established to support important decisions This form of evaluation builds on process and cost-effectiveness analyses. Cost-benefit analy-
fying the program models that produce the
about the program or the program model. impact assessment. If it is established that a sis studies the relationship between program strongest results with a given amount of fund-
Chapters 7-1 O of this volume discuss im- program is well implemented and produces the costs and outcomes, with both costs and out-
ing, and determining the degree of political
pact assessment and the various ways in which desired outcomes, questions re]ated to effi- comes expressed in monetary terms. Cost- support stalceholders provide to a program.
it can be designed and conducted. Although ali ciency become relevant. Typical questions of effectiveness analysis examines the relation-
Lilce impact assessment, elliciency assess-
such designs are demanding, some are easier to this sort include "Is a program producing suffi- ship between program costs and outcomes in
ment is most appropriate for mature, stable
irnplement than others in typical program cir- cient benefits in relation to the costs incurred l" terms of the costs perunit ofoutcome achieved.
programs with a well-structured and well-docu-
cumstances. For impact assessment, therefore, and "Does it produce a particular benefit at a Efficiency assessment can be tricky and argu-
mented program model. ln addition, as men-
mucl1 of the tailoring that must be dane for lower cost per unit of outcome than other able because it requires malcing assumptions
tioned above, efficiency analysis builds on both
application to a particularprogram is determin- interventions OI delivery systems designed to about the dollarvalue ofprogram-related activi- process and im.pact assessment, so it is irnpor-
ing just which design to use, how to configure achieve the sarne goall" The techniques for ties and, sometimes, irnputing monetary value tant that the nature and magnitude of program
it, and what problems are associated with any answering these types of questions are found in to program benefits. Nevertheless, such esti- effects be determined in advance of, OI parallel
compromises. two closely related approaches: cost-benefit and mates are often essential for decisions about to, efficiency assessment. Given the specialized
the allocation of resources to programs, identi- expertise required to conduct efficiency assess-
74 EVALUATION
Tailoring Eva!uations 75
ments, it is also apparent that it should be must engage, the main alteroatives that might
undertalcen onlywhen there is a clear need and be contemplated, and the primary consider-
an identified user for the information. With the ations involved in making choices that tailor
high leve! of concero about prograrn costs in an evaluation plan to the prograrn circum-
A systematic approach to eva/uation p/anning
many contexts, however, this may not be an stances (see Exhibit 2-P for a similar perspec-
developed by lhe Research and Evaluation ne~d oth~; ;esources including money, ma-
unusual cir=stance. tive from "dowo under"J. terial, facrhhes, transportation, and the like.
Support Services Uni! of the New South Wales
The procedures used for efficiency assess- It should be evident that there is a certain 7. What data need to be collected? Data col-
Department of Education is organized around
ment are not as demanding of resources and logic in the relationships arnong the various the following ten questions: lection activities should be planned in rela-
prograrn cooperation as are those of irnpact prograrn issues and the evaluation approaches tion to the major issues/questions identified
evaluation, but they are ctuite technical and surveyed in this chapter ln that logic, the social in Step 4. They should be specific with
1. What is the program to be eva/uated? A
rectuire a high levei of expertise. Chapter 11 conditions, target population, and associated regard to from whom lhe data are to be
common problem for the evaluator is defin-
collected, how they will be collected / and
discusses efficiency assessment methods in service needs a prograrn is intended to address ing what constitutes the "program" for
what information must be covered.
more detail. must first be identified and assessed. With that eval~ation. Some educationa/ programs
8. How will the data be analyzed? Before data
assessment in hand, the evaluator can ask if consist of a set of initiatives that may not be
are collected, consideration should be given
the basic conceptualization of the prograrn (the clo~ely integrated. ln general, a program is
STITCHING IT ALL TOGETHER defrned as "the set of operations, actions or
lo what ultimately will be dane with them.
prograrn as intendedJ represents a reasonable
means for addressing those needs. Deficiencies activities designed to produce certain de- The answers will influence decisions both
about the information to be collected and
sired effects or outcomes."
This chapter has reviewed the major consider- in this domain must be remedied by reconcep- lhe form in which it will be collected.
2. Why is the program being evaluated? Evalu-
ations involved in tailoring an evaluation so tualizing the prograrn. If the prograrn theory is ation may focus on information needs re-
9. Whatwill be the reporUng procedure? ln this
that there is an appropriate fit between the reasonable, the next ctuestion in this evaluation step, decisions should be made about to
lated to what should be dane, what can be
evaluation plan and the circumstances of the logic is whether the prograrn is actually irnple- whom lhe reports will be provided and the
done, what is being done, or what has been
prograrn to be evaluated. Howeve~ it supplies mented and operationalized as intended, that dane. appropriate ways to do this (e.g., written
report, group discussion, newspapers). A
few directly prescriptive injunctions that tell is, as stipulated by the theory. Shortcomings in 3. How are peop/e to be prepared for the
further consideration is who should be
the prospective evaluator justwhat approach to irnplementation generally must be solved by eva/uaUon?Thought should be given to who
as~d to respond to the evaluation report.
talce, what options to select, and how to go managerial initiatives. If the prograrn is irnple- is likely to feel threatened by the evaluation,
ThlS step is important both for maintaining
about putting the elements of an evaluation mented as intended, then it is meaningful to whose acceptance of the evaluation is es-
support for the evaluation and for ensuring
plan together Designing an evaluation is nota ask if it has the intended effects-this is irnpact sential, and what might be dane to provide
that the report is communicating effectively
reassurance and gain acceptance.
mechanical activity that can be accomplished evaluation. A prograrn that does not have the to lhe groups who need to know lhe results.
4. What are the main issues/quesUons with
by applying a set of rules. A good evaluation intended irnpact is either not irnplemented as 1O. How will the report be imp/emented? Atten-
which the evaluaUon is to dea/? ln this step
plan is heavily contextualized by the political intended or the prograrn theory on which the tion should be given to identifying who is to
the evaluator expands on the decisions
situation, the nature of the prograrn, the inter- prograrn's operational plan is based must be '."ade at Step 2 and develops a list of major
be responsible for making recommenda-
ests of the stakeholders, and many other such faulty. If the intended effects are produced, it is 1ssues or questions that define the evalu-
tions on the basis of the evaluation results
specilic features of the prograrn landscape. It then especially germane for the evaluation to ation's focus. and who is responsible for implementing
thus rectuires the evaluator to malce many judg- inctuire into the costs associated with attaining lhe recommendations. lt should not be
5. Who will do what? The responsibilities of
ment calls based on a careful reconnaissance of those effects and, especially, the efficiencywith automatically assumed that the evaluation
participants in the evaluation should be
team will be the ones formulating recom-
that landscape. Moreove~ experienced evalua- which theywere attained. Prograrns with costs understood and agreed on before the proj-
ect is started. mendations; the evaluation may inform rele-
tors will disagree arnong themselves on what judged to be disproportionately large relative to vant decisionmakers who assume that task.
"call" should be made for many aspects of an their benefits, or to altemate ways of attaining 6. What are the resources for the evafuation? ln
evaluation plan for a given prograrn situation. those benefits, may need to find more cost- addition to workers, the evaluation may

What we have tried to do in this chapte~ there- effective approaches or be replaced with more
fore, is identify the major issues the evaluator cost-effective alteroatives. SOURCE: Adapled from Linda J. Lee and John F Sam son "A
and Program Planning, 1990, 13(2):157-164. . p , Practrcal Approach lo Program Evaluation," fva/uation
76 EVALUATION
Tailoring Evaluations 77

Fl.'l The evaluation questions that are identified during planning d th h d f


answering them ali fali . , an e met o s ar
. ' gener Y mto one ar more recognizab!e categories havmg to d0
Wlth ia) the need for services, lb) program conceptualization and d 1)
Fl.'l Every evaluation must be tailored to the circumstances of the program being impleme t ti ldJ esrgn, e program
n a on, program outcomes, ar ie) program efficiency Eval t l
evaluated so that the evaluation design will be capable of yielding credible and useful developed relatively distinct conceptual and methodo]ooi 1 . h ufa ors iave
thes diff . o.ca approac es ar each of
answers to the specilic questions at issue while still being sufficiently practical to
actually implement within the resources available.
e erent categones ! issues that are referred to by such terms as needs
as~essment. process .evaluatlon, and impact assessment. ln practice, much of evalu-
Fl.'l One important influence on an evaluation plan is the purpose the evaluation is at10n pla:onn;-g consrsts of identifyillg the evaluation approach corresponding to the
intended to serve. Evaluations generally are initiated to either provide feedback for type of quest10ns to be answered il an evaluation then tailoring th u h
program situation. ' e spec ics to t e
program improvement to program managers and sponsors, establish accountability
to decisionmakers with responsibility to ensure that the program is effective, ar
contribute to knowledge about some form of social intervention. The overall purpose
of tl1e evaluation necessarily shapes its focus, scope, and construction.
Fl.'l Another important factor in planning an evaluation is the nature of the program
structure and circumstances. The evaluation design must be responsive to how new
ar open to change the program is, the degree of consensus ar conflict among
stakeholders about the nature and mission of the program, the values and concepts
inherent in the program rationale and design, and the way in which the program is
organized and administered.
Fl.'l Evaluation planning must also acco=odate to the inevitable limitations on the
resources available for the evaluation effort. The critical resources include not on!y
funding but a!so the amount of time allowed for completion of the work, pertinent
technical expertise, program and stakeholder cooperation, and access to important
records and program material. A balance must generally be found between what is
most desirable Iram an evaluation standpoint and what is feasible in terms of
available resources.
Fl.'l The evaluation design itse!f can be structured around three issues: la) the questions
the evaluation is to answer, lb) the methods and procedures to be used to answer
those questions, and ic) the nature of the evaluator-stakeholder interactions during
the course of the evaluation.
Fl.'l Deciding on the appropriate relationship between the evaluator and the evaluation
sponsor, as well as other major stakeholders, is an often neglected, but criticai aspect
of an evaluation plan. An independent evaluation, in which the evaluator takes
primary responsibility for designing and conducting the evaluation, is often expected.
ln some circumstances, 11owevelj a more participatory ar collaborative interaction
with stalceholders may be desirable, with the evaluation conducted as ateam project.
ln the latter case, the evaluation may be designed to help develop the capabilities of
the participating stalceholders in ways that enllance their skills ar political influence.

-,
i '
1

Evaluation questions
~:e~u:s~~~:ti~;~n~e~:!iPd,::e~ihte
h

that they can be answere 1n a way use


:~:i~t~~n::~~~~:;i,~a~~i~;:;~~t;:e;~~;~;::as~~h
1 tor eva\uation sponsor, and other stakeholders;
IDENTIFYING ISSUES AND
Program goal
evaluator.
A statement, usually general and abs~c~, of a desired state toward which a program is
.
FORMULATING QUESTIONS
directed. Compare with program ob1ectives.
Program objectives Specific, operationalized statements detailing the desired accompllshments of a progra~.
Performance criterion . a d1mens1on
The standard against wh1ch . . of program pe rformance fs compared so that 1t The previous chapter presented an overview of the many considerations that go into tailoring
can be evaluated. an evaluation. Although a1l those matters are important to evaluation design, the essence of the
Utilization of evaluation . f 1 tion by decisionmakers and other evaluation enterprise is generating credible answers to questions about the performance of aso-
The use of the concepts and find1ngds o an eva ua t leve! ar at broader funding ar policy
stakeholders whether at the day-to- ay managemen cial program. Good evaluation questions must address program issues that are meaningfuJ in rela-
tion to the nature of the program and also of concern to key stakeholders. They must be answer-
leveis. h Jb fts
able with the research techniques available to the evaluator, and they must be formulated so that
The set of assumptions about the manner n; ~e r the program has adopted to achfeve
Program theory 1 h' h the program relates to t e sacia ene 1
it is expected to produce and the strategy an c ics the criteria by which the corresponding program performance wil1 be iudged are explicit or can
its geais and objectives. h be determined in a straightforward way.
Jmpact theory . d ectations inherent in a program about the nature of t e A set of appropriate evaluation questions, therefore, is the hub around which evaluation re-
The beliefs, assumpt1ons, an exp dh 't results in the intended improvement
h b ht b ut by program action an ow 1 d ff volves. It follows that a careful, explicit formulation of those questions greatly facilitates both the
c ange. 1 rougd' a o Pr0 gram impact theory 1s causa 1th eory.. lt describes a cause-an -e ect
.
in sacia can ltions.certain program act1v1t1es design of the evaluation and the subsequent use of its findings. Evaluation questions may take
sequence in which . .
are th e instigating causes and certain sacia1
benefits are the effects they eventually produce. . ..
various forms, some of which are more useful and meaningfuJ than others for stakeholders and
program decisionmakers. Furthermore, some forms of the evaluation questions are more amena-
The assumptions and expectat10~ a. ~u.t ~
Service utilization plan . b h w the target population will make 1n1t1al contact
h the completion of the intended services. ble to the evaluator's task of providing credible answers, and some address criticai program effec-
with the program and be ~ngage1 wti'1t ttlt ~~~~cribes the sequence of events through tiveness issues more dll:ectly than others. Careful development and formulation of the questions
1 'ts 1 st form a service utl 1za on P a .
n~ic~I~~ ~tended clients are expected to interact with the intended services. that the evaluation wil1 be designed to answer are, therefore, crucial steps in conducting a pro-
gram evaluation.
The assumptions and expectations a ~ut w a d th program that will produce the intended
Organizational plan w . b h t the program must do to bring about the
This chapter discusses practical ways in which effective evaluation questions can be fash-
transactions between t~e target popu atlo~ an ~ational plan is articulated from the
changes in social cond1tlons. The program s organ b th the functions and activities ioned from input by stakeholders and analysis by the evaluator. An essential procedure for this
. f management and encampasses o purpose is identification of the decisionmakers who wil1 use the evaluation results, what informa-
perspect1ve o program ,
the program is expected to penorm an d th e human , financial , and physica\ resources
tion they need, and how they expect to use it. The evaluator's own analysis of the program is
required for that performance.
also important. One approach that is particularly useful for this purpose is articulation of the pro-
Program process theory . ' nizational plan and its service utilization plan lnto
The combinat1on of the program s or.ga d
an averall description of the assumpttons an expec 1
tafons about how the program is gram theory, a detailed account of how and why the program is supposed to work. Consideration
supposed to operate. . of program theory focuses attention on criticai events and premises that may be candidates for in-
quiry in the evaluation.
lmplementation failure 1 rf m the activities specified in the program design
The program does not adequate yfpe b.r . g bout the intended social improvements.
that are assumed to be necessary or. rmgmt a ugh service ar the wrong service is
lt includes situations in which no service, no ena , 1 .
d \. d ar the service varies excessively across the target popu atlon.
e 1vere , d th diate
The program is implemente as p anne t ~ the ultlmate social benefits intended ar both.
Theory failure d 1 d b t its services do not pro uce e tmme
effects on the participants that are expec e ar

79
80 EVALUATION
Identifying lssues and Formulating Questions 8!

P rogram evaluation is fundamentally an en-


deavor that gathers and interprets infor-
mation about program performance to answer
able about evaluation and will already have
dane the necessary background work and for-
mulated a complete and workable set of ques-
questions relevant to decision making 01; at tions to whlch the evaluation should attend.
least, of appreciable interest to one or more Such situations are not typical, howeveL More There are different kinds of inquiry across
2. Constructing standards. How well should lhe
program stakeholders. A criticai phase in an often, the evaluation sponsors and program pract1ce areas, such as that which is found in law
evaluand perform?
evaluation, therefore, is the identification and stakeholders are not especially expert at evalu- medicine, and science. Common to each kind 0 f
3, Measuring performance and comparing with
formulation of the questions the evaluation is ation o~ if so, have not dane all the groundwork inq~iry is a general pattern of reasoning ar basic
standards. How well did lhe evaluand per-
log1c thal guides and informs lhe praclice.... form?
to address. One might assume that this step needed to focus the evaluation. Thls means
Evaluation is one kind of inquiry, and it, too, has
would be very straightforward, indeed, that the that the evaluatorwill rarely be presented at the 4. Synthesizing.and integrating data into a iudg-
a basic logic ar general patlern of reasoning [lhat
questions would be stipulated routinely as part outset with a finished list of every issue the ment of mentor worth. What is the merit or
has been pul forlh by Michael Scriven] .... This
of the process of commissioning the evalu- evaluation should address for the results to be worlh of the evaluand?
general logic of evaluation is as follows:
ation. As described in Chapter 2, howeve~ it is useful, interpretable, and complete. Nor will
... To evaluate anything means to assess the
rare for final, workable evaluation questions to the issues and questions that are put forward 1. Establishing criteria of merit On what dimen-
merit ar worth of something against criteria and
be specilied by the evaluation sponsor at the generally be formulated in a manner that per- sions musl lhe evaluand [thing being evalu-
aled] do well? standards. The basic logic explicated by Scriven
beginning of an evaluation. Nor can the evalu- mits ready translation into research design.
reflects what it means when we use the term to
ator usually step up and define the focal ques- Thus, although the specilication of the evafuate.
tions unilaterally on the basis of hls or her evaluation questions must involve input from
professional expertise. That maneuver would stakeholders, the evaluator's role is also cru- SOURCE: Quoted from Deborah M. Fournier E5t.ablishin Eva/uat' Co I
Working Lagic, New Directions for Evaluation 'no 68 (SangF
. ive) nc us1ons: A D1suncuon Between General and
increase the risk that the evaluation would not cial. The stakeholders will be the experts on the , . ranc1sco: asseyBass, 1995), p. 16.
be responsive to stakeholder canceras, would practical and political issues facing the pro-
not be useful or use d, and would be attacked as gram, but generally the evaluatorwill know the
irrelevant or inappropriate. most about how to analyze a program and focus acceptable to them. Such a procedure also can
Their principal role is to focus the evaluation
To ensure that the ev aluation will attend to an evaluation. The evaluato~ therefore, must safeguard against !ater misunderstanding of
on those areas of program performance at issue
the matters of greatest cancero to the pertinent be prepared to raise issues for consideration what the evaluation was supposed to accom-
plish. for key decisionmakers and stal<eholders and t
facilitate development of a design for data co~
decisionrnal<ers and stakeholders, therefore, that otherwise might be overlooked, identify
the initial evaluation questions are best formu- aspects of program operations and outcomes The remainder of this chapter examines
l~ction that will provide meaningfui informa-
lated through interaction and negotiation with that might warrant inquiry, and work with the two ~ost important topics related to speci-
tion about howwell the program is performing.
those decisionmal<ers and stakeholders. This, stakeholders to translate their canceras into fying the 1ssues and questions that will guide
-'.' good evaluation question, therefore must
of course, helps direct the evaluation toward questions of a form that evaluation research an evaluation: (a) how to formulate evaluation
1dentify a distinct dimension of progra.:n per-
the most relevant practical issues. Of equal can attempt to answeL questions in such a way that they can be ad-
formance that is at issue and do so in such a
importance, howeve~ is that such interaction ln all but the most routine situations, it is dressed using the research procedures available
way that the quality of the performance can be
engages key stakeholders in the process of de- generally wise for the evaluator to construct a to the evaluato~ and (bJ how to determine the
credibly assessed. Such assessment in tum
fining the evaluation in a detailed and personal written statement of the specilic questions that specific questions on whlch the evaluation . ,
reqwres an accurate description of the nature
'
way that increases the likelihood that they will will guide the evaluation design. This provides should focus.
of the performance and some standard by
understand, appreciate, and make effective use a reference to consult while designing the
which it can be evaluated (see Exhibit 3-AJ.
of the findings when they become available. evaluation and selecting research procedures WHAT MAKES A GOOD Thus a good evaluation question must specify
Although stakeholder input is criticai, the that can be very useful. Perhaps more impor- EVALUATION QUESTION? some measurable or observable dimension of
evaluator should not depend only on the deci- tant, this written statement can be discussed
program performance in reference to the crite-
sionmal<ers and stalreholders to identify the with the evaluation sponsor and key stakehold-
rion bywhlch that performance is to be judged.
issues the evaluation will address. Sometimes ers to ensure that it encampasses their can- The form evaluation questions should take is Bach of these different aspects warrents further
the evaluation sponsors are very koowledge- ceras and defines a focus for the evaluation shaped by the functions they must perform. discussion.
ldentifying lssues and Pormulating Questions 83

82 EVALUAT!ON
service literature. Questions involving certain lation of the various components of prograrn
might initially ask, ''Are our education and prograco performance dimensions would be as- theory, described !ater in this chapte~ generally
Dimensions of outreach services successful in informing the sessed as more appropriate if they were consis- provide a very effective approach to developing
Program Performance public about the risk of AIDS?" ~ prac~ce, tent with experience in similar prograrns ar this detailed understanding of the prograrn.
howeve~ those services may cons1st of little studies of those prograrns. For instance, the
Good evaluation questions will identify as-
more than occasional presentations by pro- sponsor of an evaluation of a prograrn for juve- Evaluation Questions
pects of performance dimensions that are rele- nile delinquents might initially ask if the pro-
graco staff at Rotary Clubs and health fairs. Must Be Answerable
vant to the expectations stakeholders hold for
With this rather modest levei of activity, it may grarn increases the self-esteem of the delin-
the prograco and represent domains in which quents, in the belief that increased self-esteem It is rather obvious that the evaluation
not be realistic to expect the public at large to
the prograco can realistically hope to have ac-
receive much AIDS information tluough this is a problem for these juveni!es and iroprove- questions around which an evaluation plan is
complishments, but where effectiveness can- ments in self-esteem will lead to betrer behav-
channel much less for that information to developed should be answerable. Questions
not necessarily be taken for granted. It would im Examination of the applicable social science that cannot be answered may be intriguing to
lower the risk of AIDS in the community. If a
hardly be fair to ask if a low-income housing
question about this service is deemed im~or research, howeve~ will reveal that juvenile de- philosophers but serve poorly the needs of
weatherization prograco reduced the preva- linquents do not generally have problems with evaluators and the decisionmakers who intend
tant for the evaluation, a better versrnn nnght
lence of drug dealing in a neighborhood. Nor self-esteem and, moreove11 that increases in to use the evaluation results. What is not so
be something such as "Do our education and
would it generally be useful to ask if the pr.o- self-esteern are not generally associated with obvious, perhaps, is how easy it is to formulate
outreach services raise awareness of AIDS is-
graco got a good deal on its purchase of file an unanswerable evaluation question without
sues arnongthe audiences addressed?" and "Do reductions in delinquency. Inlight of this infor-
cabinets for the office. Furthermore, the evalu- mation, the evaluator and the evaluation spon- realizing it. This may occur because the terms
those audiences represent community !eaders
ation questions must invalve performance sor may well agree that the question of the used in the question, although seemingly com-
who are likely to inf!uence the opinions of
diroensions that are sufficiently specific, con- prograrn's iropact on self-esteem is not appro- monsensical, are actually arnbiguous ar vague
sigriificant others?"
crete, and practical that meaningful informa- priate after ali. when the tiroe comes for a definitive interpre-
There are two complementary ways for an
tion can be obtained about therr status. An The foundation for formulating appropri- tation ("Does this prograrn enhance familyval-
evaluator in collaboration with pertinent
evaluator would have great difficulty determin- ate and realistic evaluation questions is de- ues?"). Or sensible-sounding questions may
stakeholders, to assess how appropriate ai:'d
ing if an adult literacy prograco iroproved a
realistic a candidate evaluation questrnn is. tailed and complete prograco description. Early invoke issues forwhich there are so few observ-
community's competitiveness in the global
The first is to examine the question in the in the process, the evaluator should become able indicators that little can be learned about
economy ar if the counselors in a drug preven-
context of the actual prograrn activities related thoroughly acquainted with the prograco-how them (''Are the case managers sensitive to the
tion prograrn were sufficiently caring in their it is structured, what activities take place, the social circumstances of their clients?"). Also,
to it. In the exarople above, for instance, the
relations with clients. Jow-key nature of the education and outreach roles and tasks of the various personnel, the some questions Jack sufficient indication of the
services were clearly not up to the task of nature of the participants, and the assumptions relevant criteria to be answered ("Is this pro-
Evaluation Questions Must Be "informing the public about the risk of AIDS" inherent in its principal functions. The stake- gram successful?"). Finally, some questions
Reasonable and Appropriate and there would be little point in having the holder groups with whom the evaluator col- may be answerable but to do so would require
evaluation atrempt to determine if this was laborates (especially prograrn managers and more expertise, data, ar resources than are
Prograrn advocates sometimes put. forw":"d
accomplished. The evaluator and relevant staff) will also, of course, have knowledge about available to the evaluation ("Do the prenatal
grandiose goals (e.g., improve the quality of life
stakeholders should identify and scrutinize the the prograrn. Evaluation questions that are in- services this prograrn provides to high-risk
for childreri), expect unrealistically large. ef-
prograrn components, activities, and personnel spired by dose consideration of actual prograrn women increase the chances that their children
fects ar believe the prograco has accomplish-
assigruoents that relate to prograrn perfor- activities and assumptions will almost auto- will complete college?").
men~s that are disproportionate to its actual For an evaluation question to be answer-
mance and formulate the evaluation question matically be appropriate and realistic. And a
capabilities. Good evaluation questions deal
in a way that is reasonable given those charac- clear understanding of the prograrn operation able, it must be possible to identify in advance
with performance dimensions that are appro-
teristics. and its rationale is a necessary prerequisite for some evidence ar "observables" that can realis-
priate and realistic for the prograco. This
Another form of review for candidate entering the social science and social service tically be obtained and will be credible as the
means that the evaluator must often workwith
evaluation questions is to analyze them in literature to find relevant concepts and analo- basis for an answer This generally means de-
relevant stakeholders to scale down and focus
relationship to the experience and findings ~e gous situations. The investigation and articu- veloping questions that involve measurable
the evaluation questions. The manager of a
ported in applicable social science and social
community health prograrn, for instance,
84 EVALUATION Identifylng Issues and Formuiatlng Questions 85
performance dimensions, that are sufficiently dren in the catchment area should be in
unarnbiguous so that explicit, noncontrover- the prograrn); and
sial definitions can be given for each of their e. Have the evaluation sponsors and other
terms, and for which the relevant standards or pertinent stalceholders lwho should be
cri teria are specilied or obvious. The best way The standards by which program performance
involved in the whole process) agree that may be judged in an evaluation include: Ethical or moral values; social justice, equity
for an evaluator to test whether a candidate this would, indeed, answer the question.
question is answerable in these terms is to Past performance; historical data
lhe needs ar wants of the target
determine whether a realistically attainable, If such conditions can be met and, in addition, Targets set by program managers
population
specilic evaluation finding can be imagined the resources are available to collect, analyze, Expert opinion
such that the relevant decisionmakers and Stated program goals and objectives
and report the applicable data, then the evalu-
stalceholders agree that it constitutes a mean- ation question can be considered answerable. Professional standards Preintervention baseline leveis for the target
ingful answei: population
Customary practice; norms for other
Suppose, for instance, that a proposed Conditions expected in the absence of the
programs
evaluation question for a compensatory educa- program (counterfactual)
tion prograrn lilce Head Start is, "P.e we reach-
Criteria for Program Performance Legal requirements
Cost or relative cost
ing the children most in need of this prograrnl" Beginning a study with a reasonable, an-
To affirm that this is an answerable question, swerable question or set of questions, of course,
the evaluator should be able to is standard in the social sciences lalthough
often frarned as hypotheses). What distin- numbers and arnounts can be evaluated on A considerable complication for the evalu-
a. Define the group of children at issue 1e.g., guishes evaluation questions is that they have those performance dimensions. ator is that there are many forms in which the
those in census tract such and such, age to do with performance and are associated, at One implication of this distinctive feature applicable performance criteria may appear for
five years or less, living in households least implicitly, with some criteria by which of evaluation is that good evaluation questions various dimensions of prograrn performance
with annual income under 150% of the that performance can be judged. Identifying the will, when possible, convey the performance isee Exhibit 3-B), and indeed, it is not always
federal poverty levei); relevant criteria was mentioned above as part standard that is applicable as well as the per- possible to establish an explicit, consensual
b. Identify the specilic measurable charac- ofwhat makes an evaluation question answer- formance dimension that is at issue. Thus, performance standard in advance of collecting
teristics and cutoff values that represent able. Howeve~ this is such an important and evaluation questions should be much lilce this: data and reporting results. Nonetheless, to the
the greatest need ie.g., annual income distinctive aspect of evaluation questions that "Is at least 75% of the prograrn clientele appro- extent that the formulation of the initial evalu-
belowthe federal poverty levei, single par- it warrants separate discussion. priate for services l" lby some explicit definition ation questions includes explicit criteria on
ent in the household with educational When prograrn managers or evaluation of appropriate) or ''Do the majority of those which key stalceholders agree, evaluation plan-
sponsors ask such thiogs as "P.e we targeting who receive the employment services get jobs ning is made easier and the potential for dis-
attainment of less than high school);
the right client populationl" or "Do our ser- within 30 days of the conclusion of training agreement over the interpretation of the evalu-
c. Give an exarnple of the evaluation finding vices benefit the recipients?" they are not only that they keep at least three months?" ln addi- ation results is reduced. It is worth noting that
that rnight result ie.g., 60% of the chil- asking for a description of the prograrn's per- tion, the performance standards represented in
dren currently served fali in the high- the criterion issue cannot be avoided. An evalu-
formance with regard to serving appropriate these questions should have some defensible, ation thatonly describes prograrn performance,
need category; 75% of the high-need chil- clients and providing services that yield bene- though possibly indirect, relationship to the
dren in the catchment area are not and does not attempt to assess it, is not truly
fits. They are also asking if that performance is social needs tl1e prograrn addresses. There an evaluation iby definition; see Exhibit 3-A)
enrolled in the prograrn); good enough according to some standard or must be some reason why attaining that stan- and, at most, only pushes the issue of setting
d. Stipulate the evaluative criteria (e.g., to judgrnent. There is lilcely little doubt that at dard is meaningful, and the strongest rationale criteria and judging performance onto the con-
be satisfactory, at least 90% of the chil- least a few of the "right client population" is that it represents a levei of performance sumer of the information.
dren in the prograrn should be high need receive services ar that some recipients receive sufficient for that prograrn function to contrib- With these considerations in mind, we
and at least 50% of the high-need chil- some benefit from services. But is it enoughl ute effectively to the overall prograrn purpose tum attention to the various lcinds of perfor-
Some criterion levei must be set by which the of improving the target social conditions. mance criteria that appear in evaluation studies
86 EVALUATION Identlfylng lssues and Formulating Questions 87

and may be relevant to formulation of useful cally a certain arnount of arbitrariness in these leveis ar, perhaps more appropriate in such each category, stated in summary form, are as
evaluation questions. Perhaps the most com- criterion leveis, but if they are administratively circumstances, to identify broad criterion follows.
mon criteria are those based on prograrn goals stipulated ar can be established through stalce- ranges that can be accepted to distinguish, say,
and objectives. ln this case, certain desirable holder consensos and are reasonable, they are high, medium, and low performance.
Questions about the need for prograrn services:
accomplishments are identified as the prograrn quite serviceable in the formulation of evalu- When the performance dirnension in an
airns by prograrn officials and sponsors. Often ation questions and the interpretation of the evaluation question involves outcome ar im-
What are the nature and magnitude of the
these statements of goals and objectives are not subsequent findings. However, it is not gener- pact issues, establishing a criterion levei can be problern to be addressed?
very specific with regard to the nature ar levei ally wise for the evaluator to press for such particularly difficult. Prograrn stalceholders and
of prograrn performance they represent. One of evaluators alike may have little idea about how What are the characteristics of the popula-
specific statements of target performance leveis
tion in need?
the goals of a shelter for battered women, for if the prograrn does not have them ar cannot much change on a given outcome variable (e.g.,
instance, m.ight be to "empower them to talce readily and confidently develop them. Setting a scale of attitude toward drug use) is large and What are the needs of the populationl
contrai of their own lives." Although reflecting such targets on a highly arbitrary basis only how much is small. By default, these judg- What services are needed?
commendable values, this statement may leave creates a situation in which they are arbitrarily ments are often made on the basis of statistical Howmucb service is needed, overwhat time
the evaluator uncertain of the tangible mani- revised when the evaluation results are in. criteria. For instance, any statistically signifi- period?
festations of such empowerment ar what levei ln some, albeit rare, instances there are cant improvement in an outcome dimension
What service delivery arrangements are
of empowerment constitutes attainment of established professional standards that can be may be viewed as an indication of prograrn
needed to provide those services to the popu-
this goal. Considerable discussion with stalce- invoked as prograrn performance criteria. This success. This is a poor practice for reasons that lationl
holders may be necessary to translate such is particularly likely in medical and health pro- will be more fully exarnined la ter in this volume
statements into mutually acceptable term.inol- grarns where various practice guidelines and when irnpact evaluation is discussed. Statisti-
Questions about prograrn conceptualization ar
ogy that describes the intended outcomes more managed care standards have developed and cal criteria have no intrinsic relationship to the
design:
concretely, identifies the observable indicators may be relevant for setting desirable perfor- practical significance of a change on an out-
that correspond to those outcomes, and speci- mance leveis for prograrns. Much more com- come dirnension and can be misleading. Thus,
as much as possible, the evaluator should at- What clientele should be served?
fies the levei of accomplishment on each that mon, however, is the situation where there are
would be considered a success in accomplish- no established criteria ar even arbitrary admin- tempt to determine and specify in practical What services should be pmvided?
ing this goal. istrative objectives to invoke. A typical situ- terms what "success" levei is appropriate for What are the best delivery systerns for the
Some prograrn objectives, on the other ation is one in which the performance dirnen- judging the nature and magnitude of the pro- se.ivices?
hand, may be very specific. These often come sion itself is clearly recognized, but there is grarn effects. How can the pmgrarn identify, recruit, and
in the form of administrative objectives arnbiguity about the criterion for good perfor- sustain the intended clientele?
adopted as targets for routine prograrn func- mance on that dimension. For instance, rele-
How should the pmgrarn be organized?
tions. The target leveis may be set according to vant stakeholders may agree that the prograrn
past experience ar the experience of comparable should have a low drop-out rate, a high propor- Typical Evaluation Questions What resources are necessary and appropri-
ate for the pmgrarn?
prograrns, judgrnent of what is reasonable and tion completing service, a high levei of client
desirable, ar maybe only on a "best guess" satisfaction, and the like, but only nebulous As should be evident froin the discussions
basis. Exarnples of administrative objectives ideas as to what levei constitutes "low" ar above, well-formulated evaluation questions Questions about prograrn operations and ser-
may be to establish intalce for 90% of the refer- "high" on the respective dirnensions. Some- are very concrete and specific to the prograrn at vice delivery:
rals within 30 days, to have 75% of the clients tirnes the evaluator can make use of prior ex- issue and the circumstances of the prospective
complete the full term of service, to have 85% perience ar find information in the evaluation evaluation. It follows that the variety of ques- Are adrninistrative and service objectives
"good" ar "outstanding" ratings on the client and prograrn literature that provides a reason- tions that m.ight be re]evant to some social beiogmet?
satisfaction questionnaire, to provide at least able basis for setting a criterion levei. Another prograrn ar another is enormous. As noted in Are the intended services being delivered to
three appropriate services to each person under approach is to collect judgrnent ratings from Chapter 2, however, evaluation questions typi- the intended persons?
case management, and the like. There is typi- relevant stalceholders to establish the criterion cally deal with one of five general prograrn Are there needy but unserved persons the
issues. Some of the more common questions in program is not reacbing?
88 EVALUATION Identifylng Issues and Formulating Questlons 89

Once in service, do sufficient numbers of ator can identify the criticai evaluation ques-
client.s complete service? tions.
Are the client.s satisfied with the services l
Are administrative, organizational, and per- An alterschool program located in an Is the program wef/ designed?
sonnel functions handled welll economically depressed area uses the facilities of
DETERMINING THE a local elementary school to provide free Question: Are the planned educational activi-
Questions about program outcomes:
QUESTIONS ON WHICH THE alterschool care from 3:30 to 6:00 for the chil- ties the best ones for this clientele and the pur-
EVALUATION SHOULD FOCUS dren oi the neighborhood. The program's goals poses of enhancing their performance in school?
are to provide a safe, supervised environment for
Are the outcome goals and objectives being Standard: There should be indications in the
latchkey children and to enhance their school
achievedl educational research literature to show that these
Occasionally, the evaluator is also the evalu- performance through academic enrichment ac-
activities have the potential to be effective. ln
Do the services have beneficial effect.s on the tivities. The following are examples of the ques-
ation sponsor and primary stalreholder in a addition, experienced teachers for the relevant
recipients? tions that an evaluation might be designed to
program.. For instance, an academic researcher grade leveis should endorse these activities.
answer for the stakeholders in this program:
Do the services have adverse side effect.s on who heads a university counseling clinic may
Question: Is there a sufficient number oi staff
the recipient.sl have an innovative program concept, imple- Is there a need for the program? positions in the program?
Are some recipient.s affected more by the ment it in the university clinic, and then con-
services than others? duct an evaluation. lt is far more typical, how- Question: How many latchkey children reside Standard: The staff-student ratio should ex-
eve~ for persons other than the evaluator to be within a radius oi 1.5 miles oi the school? Latch- ceed the state standards for licensed child care
Is the problern ar situation the services are facilities.
intended to address made betterl the ones who have responsibility for the pro- key children are defined as those oi elementary
gram, initiate the evaluation, and use the find- school age who are without adult supervision
during some period after school at least once a Is the program
ings. ln such circumstances, the evaluation is implemented effective/y?
Questions about program cost and elliciency: week during the school year.
a project of the sponsor and other involved
stalceholders; the evaluator is only the instru- Standard: There should be at least 100 such Question: What is the attendance rate for
Are resources used efficientlyl
ment for accomplishing that project. Corre- children in the defined neighborhood. The enrolled children?
m Is the cost reasonable in relation to the spondingly, it is only fitting that the evaluation planned enrollment for the program is 60, which
magnitude of the benefit.sl Standard: Ali enrolled children should either
give central attention to the issues and ques- should yield enough children in attendance on
be in attendance every alternoon for which they
m Would alternative approaches yield equiva- tions of the evaluation sponsor and the other any given day for efficient staffing, and it is as-
are scheduled ar excused with parental permis-
lent benefit.s at less costl principal stalreholders. ln the discussion that sumed that some eligible children will not enroll
sion.
for various reasons.
follows, therefore, we first examine the matter
Question: Is the program providing regular
These families of evaluation questions are of obtaining appropriate input from the evalu- Question: What proportion of the children
support for school homework and related tasksl
not mutually exclusive, of course. Questions in ation sponsor and relevant stalceholders prior enrolled in lhe program are actually latchkey
more than one category, and maybe in ali the to and during the design stage of the evaluation. children? Standard: There should be an average oi 45
categories, could be relevant to a program for Howeve~ it is rarely appropriate for the minutes of supervised study time for completion
Standard: At least 75% of the enrolled children
of homework and reading each afternoon, and
which an evaluation was being planned. To evaluator to rely only on input from the evalu- should meet the definition for latchkey children.
ali the attending children should participate.
develop an appropriate evaluation plan, the ation sponsor and stalceholders to determine This is an administrative target that reflects the
many possible questions that rnight be asked the questions on which the evaluation should program's intent that a large majority oi the
Does the program have
about the program must be narrowed down to focus. Because of their dose familiarity with enrollees be latchkey children while recognizing
the intended outcomes?
those that are most relevant to the program the program, stalceholders may overlook criti- that other children will be attracted to, and ap-
context and the inforrnation needs of the key cai, but relatively routine, aspects of program propriate for, the program even though not meet- Question: Is there improvement in the atti-
ing that definition. tudes oi the enrolled children toward school?
stalceholders isee Exhibit 3-C for an example of performance. Also, the experience and knowl-
evaluation questions for an actual program). edge of the evaluator may yield distinctive in-
We tum now to a discussion of how the evalu- sights into program issues and their inter-
(cantinued)
90 EVALUATION Identifying Issues and Formulating Questlons 91

Standard: At leasl 80% of lhe children should Is the program cost-effective?


show measurable improvement in their attitudes The Joint Program was initiated to improve the ation, an evaluation steering committee was set
Question: Whal is the cosi per child for run-
toward school between the beginning and end accessibility of health and social services for the up with representatives of the different types of
ning this program beyond the fixed expenses
of the school year. Norms for similar students homeless population oi Montreal through coordi- agencies involved in the program and which, in
associated with the regular operalion of the
show that their altitudes tend to gel worse each nated activities involving provincial, regional, and turn, coordinated with two other stakeholder com-
year of elementary school; the program objective school facilityl
municipal authorities and more than 20 nonprofit miltees charged with program responsibililies.
is to reverse this trend, even if the improvement Standard: Costs per child should be near or and public agencies. The services developed Even though ai/ the stakeholders shared a com-
is only slight. below the average for similar programs run in through the program included walk-in and referrai mon cause to which they were firmly committed
other school districts in the state. services, mobile drop-in centers, an outreach -the we/fare of lhe homeless-they had quite
Question: Is there an improvement in the
team in a community health center, medical and varied perspectives on the evaluation. Some of
academic performance of the enrolled children Question: Would the program be equally ef-
nursing care in shelters, and case management. these were described by the eva/uators as
in their regular school workl fective and less costly il staffed by community
To ensure stakeholder participation in the evalu- follows:
volunteers (except the director) rather than paid
Standard: The average term grades on aca-
demic subjects should be at least a half letter paraprofessionals?
The most glaring imbalance was in the various agencies' different organizational cultures,
grade better than they would have been had the Standard: The annual cost of a volunteer- which /ed them to experience their participation in the evaluation very differently. Some of
children not participated in the program. based program, including recruiting, training, and the service agencies involved in the Joint Program and its evaluation were front-line public
supporting the volunteers, would have to be ai organizations that were accustomed to viewing their actions in terms of a mandate with a
least 20% less than the cost oi the current pro- target clientele. They were familiar with the evaluation process, both as an administrative
gram with no loss of effectiveness to justify the procedure and a measurement of accountability. Among the nonprofit agencies, however,
effort associated with making such a change. some relative newcomers who had been innovators in the area of community-based
intervention were hoping the evaluation would recognize the strengths of their approach
and make useful suggestions for improvement. Other nonprofil groups were offshoots oi
religious or charitable organizations that had been invo/ved with the home/ess for a very
long time. For those groups the evaluation (and the logica/, planning-based program itself)
We therefore discuss the different components was a procedure completely outside of anything in their experience. They perceived the
relations that are important for identifying rele-
of prograrn theory, how the evaluator can de- evaluators as outsiders meddling in a reality that they had managed to deal with up until
vant evaluation questions. Generally, there-
now, under very difficult conditions. Their primary concern was the client. More than the
fore, it is desirable for the evaluator to make a scribe and represent it, and how it can be used
public agencies, they probably saw the evaluation as a waste of time, money, and energy.
relatively independent analysis of the prograrn diagnostically to identify those prograrn func-
Most oi the day centers involved in lhe program fell into this category. They were the ones
for the purpose of identifying areas of prograrn tions that relate most directly to its effective- who were asked to take part in a process with which they were unfamiliar, alongside their
performance that may be pertinent for investi- ness. counterparts in the public sector who were much better versed in research procedures.
gation. (p. 471)
The second topic addressed in the discus-
sion that follows, therefore, is how the evalu- SOURCE: Quoled, wi!h permission, from Cline Mercier, "Participation ln Stakeholder-Based Evalualion: A Case Study/'
ator can analyze a prograrn in a way that will
Representing the Concerns Eva/uarion and Program P/anning, 1997, 20(4):467-475.

uncoverpotentially irnportant evaluation ques- of the Evaluation Sponsor


tions for consideration in designing the evalu- and Major Stakeholders
ation. An especially useful too! for this purpose
is the concept ofprogram theory, a depiction of In planning and conducting an evaluation,
evaluators usually find themselves confronted and sometimes conflicting views on the pro- be affected by the outcome {see Exhibit 3-D for
the sigoilicant assurnptions and expectations
with multiple stakeholders who hold different grarn ar its evaluation and whose interests will an illustration). Ar. the planning stage of an
on which the prograrn depends for its success.
92 EVALUATION Identifylng lssues and Formulating Questions 93

evaluation, the evaluator usualiy attempts to and one or more of the stakeholders regarding A snowbaJJ sampling approach is often team. Fewer such contacts may be needed by
identify ali the stakeholders with an important what issues the evaluation should address. It is helpful in identifying the various stakeholder evaluation teams that already represent many
point of view on what questions should be especialiy important, therefore, that there be groups and persons involved in relationships stakeholders and more by those on which few
addressed in the evaluation, set priorities fui] and franlc co=unication between the with the program. As each such representative or no stalceholders are represented. In cases
among those viewpoints, and integrate as evaluator and the pertinent stakeholder groups is identified and contacted, the evaluator asks where the evaluation has not initialiy been
many of the relevant concems as possible into from the earliest possible point in the planning for nominations of other persons or groups who organized as a collaborative endeavor with
the evaluation plan. process. Along with obtaining critical input have a significant interest in the program or are stakeholders, the evaluator may wish to con-
The starting point, of course, is with the from the stakeholders about the program and likely to have useful information about it. sider configuring such an arrangement to en-
evaluation sponsors. Those who have comrnis- the evaluation, this exchange should empha- Those representatives, in tum, are asked the sure engagement by key stakeholders and fui]
sioned and funded the evaluation rightfully size realistic, shared understanding ofwhat the sarne question. When this process no longer representation of their views in the evaluation
have priority in defining the issues it should evaluation will and will not do, and why. Most produces consequential new nominations, the design and implementation. Similarly, various
address. Sometimes evaluation sponsors have essentialiy, the evaluator should strive to en- evaluator can be reasonably assured that ali participatory arrangements might be made
stipulated the evaluation questions and meth- sure that the key stalceholders understand, and major stakeholders have been identified. through stakeholder advisory boards, steering
ods completely and want the evaluator only to find acceptable, the nature of the evaluation If the evaluation is structured as an explic- comrnittees, or simply involvement of key
manage the practical details. ln such circum- process, the type of information the evaluation itly collaborative or participatory endeavor so stalceholder representatives in regular consul-
stances, the evaluator should assess which, if will produce, what it might mean if the results that certain stakeholders are directly involved tation with the evaluatm More information
any, stalceholder perspectives are excluded and come out one way or anothe~ and what ambi- in designing and conducting the evaluation (as about the procedures and benefits of such ap-
whether they are sufficiently clistinct and im- guities or unanswered questions may remain. described in Chapter 2), they will, of course, proaches can be found in Fetterman, Kaftarian ,
portant that their omission compromises the have a firsthand role in shaping the evaluation and Wandersman (1996), Greene (1988), Mark
evaluation. If so, the evaluator must then de- Obtaining Input questions. Similarly, an intemal evaluator who and Shotland (1985), and Patton (1997).
cide whether to conduct the evaluation under From Stakeholders is part of the organization that administers the Outside of organized arrangements,
the specified constraints, reporting the limita- program will likely receive forthright counsel evaluators generaliy obtain stakeholder views
tions and biases along with the results, or The major stakeholders, by definition, from program personnel. Even when such about the important evaluation issues through
attempt to negotiate an arrangement whereby have a significant interest in the program and stalceholder involvement is built into the way personal or telephone interviews. Because
tl1e evaluation is broadened to include adcli- the evaluation. It is thus generaliy straightfor- the evaluation is organized, howeve~ this ar- these early contacts with stakeholders are pri-
tional perspectives. ward to identify them and obtain their views rangement is usualiy not sufficient to represem marily for orientation and reconnaissance,
More often, howeve~ the evaluation spon- about the issues and questions to which the tl1e full range of pertinent stakeholder perspec- such interviews are typicaliy unstructured o~
sors' initial ~pecilications are not so con- evaluation should attend. The evaluation tives. There may be important stakeholder perhaps, semistructured around a smali set of
strained or nonnegotiable that the concems of sponso~ program administrators (who may groups that are not involved in the partici- themes of interest to the evaluatm Input from
otl1er stalceholders cannot be considered. ln also be the evaluation sponsorJ, and intended patory structure but have clistinct and signif- some number of individuals representing one
this situation, the evaluator typicaliy makes program beneficiaries are virtualiy always ma- icant perspectives on the program and the or more stalcelmlder groups might also be ob-
tl1e best attempt possible within the con- jor stakeholders. Identification of other impor- evaluation. Moreove~ there may be a range of tained through focus groups (Kruege~ 1988).
straints of the situation to consult fullywith ali tant stakelmlders can usualiy be accomplished viewpoints among the members of those Focus groups have the particular advantages of
stalceholders, set reasonable priorities, and de- by analyzing the network of relationships sur- groups that are represented in a participatory efficiency in getting information from a num-
velop an evaluation plan that will enhance the rounding a program. The most revealing rela- evaluation process so that a broader samp- ber of people and the facilitative effect of group
information available about the respective con- tionships involve the flow of money to or from ling of opinion is needed than that brought by interaction in stimulating ideas and observa-
cems of all parties. the program, political influence on and by the the designated participant on the evaluation tions. They also may have some disadvantages
Given the usual multiplicity of program program, those whose actions affect or are af- team. for this purpose, notably the potential for con-
stalceholders and their perspectives, and de- fected by the program, and the set of direct Generaliy, therefore, formulating respon- flict in politicaliy volatile situations and the
spite an evaluator's efforts to be inclusive, there interactions between the program and its vari- sive evaluation questions requires some discus- lack of confidentiality in group settings. In
is considerable inherent potential for misun- ous boards, patrons, collaborators, competi- sion with members of stakeholder groups who some cases, therefore, stalceholder informants
derstandings to develop between the evaluator tors, clients, and the like. are not directly represented on the evaluation may speakmore frankly about the program and

....
94 EVALUATION Identlfylng Issues and Formulating Questions 95

the evaluation one-on-one with the evaluator use is likely to be made of the results. The
than they will in a focus group. evaluation may be desired by prograrn manag-
The evaluator will rarely be able to obtain ers to determine if the prograrn is effective, to
input from every member of every stakeholder fi.nd ways to improve it, ar to 11prove" its value Four techniques are particularly helpful for tive. For example, the statement "to begin three
group, nor will that ordinarily be necessary to to potential funders, donors, critics, ar the like. writing useful objectives: (a) using strong verbs, prenatal classes for pregnant women by sub-
identify the major issues and questions with Sometimes the evaluation is politically moti- (b) stating only one purpose ar aim, (e) speci- contracting with City Memorial Hospital" con-
which the evaluation should be concemed. A vated only, for example, as a stalling tactic for fying a single end-product ar result, and (d) spec- tains two results, namely, the three classes and
ifying the expected time for achievement the subcontract. lt is better to state these objec-
modest number of carefully selected stake- a controversial prograrn. Whatever the reasons,
(Kirschner Associates, 1975). tives separately, particularly since one is a higher-
holder informants who are representative of they provide an irnportant starting point for 11
A strong 11 verb is an action-oriented verb that arder objective (to begin three prenatal classes)
significant groups ar distinctly positioned in determining what questions will be most irn- describes an observable ar measurable behavior which depends partly on fulfillment of a lower-
relation to the prograrn is typically sufficient to portant for the evaluation to answer and for that will occur. For example, "to increase the use arder objective (to establish a subcontract).
identify the principal issues. When the evalu- whom. of health education materiais" is an action- A clearly written objective must have both a
ator no longer hears new themes in discus- oriented statement involving behavior which can single aim and a single end-product ar result. For
sions with diverse stakeholders, the most sig- What are the program goals and obiectivesi be observed, ln contrast, "to promote greater example, the statement "to establish communi-
nificant prevailing issues have probably all been Inevitably, whether a prograrn achieves certain use of health education materiais" is a weaker cation with the Health Systems Agency" indi-
discovered. of the goals and objectives ascribed to it will be and less specific statement. The term "promote" cates the aim but not the desired end-product or
pivotal questions for the evaluation to answei: is subject to many interpretations. Examples of result What constitutes evidence of communication
The distinction between goals and objectives is action-oriented, strong verbs include: "to write," -telephone calls, meetings, reports? Failure to
Tupics for Discussion
criticai. Goals are typically stated by prograrns "to meet," "to find," "to increase," and "to sign." specify a clear end-product makes it extremely
With Stakeholders
Examples of weaker, nonspecific verbs include: difficult for assessment to take place.
in broad and rather abstract terms. For evalu-
"to understand," "to encourage," "to enhance," Those involved in writing and evaluating ob-
As mentioned in the previous chapte~ the ation purposes, such goal statements must be
and "to promote." jectives need to keep two questions in mind.
issues identified by the evaluation sponsor refined and restated in terms that can be mea-
A second useful suggestion for writing a clear First, would anyone reading the objective, with
when the evaluation is requested usually need sured. For example, a prograrn for the homeless objective is to state only a single aim ar purpose. ar without knowledge of the program, find the
further discussion with the sponsor and other may have as its goal "the reduction of home- Most programs will, of course, have multiple sarne purpose as the one intended? Second,
stalceholders to clarify what they mean to the lessness" in its urban catchment area. Al- objectives, but within each objective only a what visible, measurable, or tangible results are
various parties and what sort of information though easily understood, such a goal is too single purpose should be delineated. An objec- presentas evidence that the objective has been
would usefully bear on them. This endeavor vague to support agreement that it has ar has tive that states two ar more purposes ar desired met? Purpose ar aim describes what will be
may then lead to refinement and revision of the not been met. Is a 11reduction of homelessness" outcomes may well require different imple- dane; end-product or result describes evidence
questions the evaluation will address. The top- 5%, 10%, ar 100%? Does it refer to only those mentation and assessment strategies, making that will exist when it has been dane. This is as-
ics that should be addressed in these discus- who are homeless ar also to those who are achievement of the objective difficult to de- surance thatyou "know one when you see one."
sions will depend in large part on the particu- marginally housed and at irnminent risk of termine. For example, the statement ''to begin Finally, it is useful to specify the time of
lars of the evaluation situation. We will review three prenatal classes for Pregnant women and expected achievement of the objective. The
homelessnessl For evaluation purposes, these
provide outreach transportation services to statement ''to establish a walk-in clinic as soon
some of the general topics that are often rele- broad goals must be translated into concrete
accommodate twenty-five women per class" as possible" is nota useful objective because of
vant. statements that specify the condition to be creates difficulties. This objective contains two the vagueness of "as soon as possible. 11 lt is far
dealt with together with one ar more measur- aims-to provide prenatal classes and to provide more useful to specify a target date, or in cases
Why is an evaluation neededi It is usually able criteria of success. Evaluators generally outreach services. lf one aim is accomplished where some uncertainty exists about some
worthwhile for the evaluator to probe the rea- refer to these more specific statements of mea- but not the other, to what extent has the specific date, a range of target dates-for
sons an evaluation is desired with the evalu- surable attainments as obiectives. Exhibit 3-E objective been met? example, "sometime between March 1 and
ation sponsor and other stakeholders. The presents helpful mies for specifying objectives. Specifying a single end-product or result is a March 30"-is also useful.
evaluation may be motivated by an externai An irnportant task for the evaluato~ there- third technique contributing to a useful objec-
requirement, in which case it is irnportant to fore, is to collaborate with the evaluation spon-
know the nature of that requirement and what sors, prograrn managers, and other relevant SOURCE: Adapted, with permission, from Stephen M. Shortell and William C Richardson, Healrh Program Evaluafion
(St. louis, MO: CV. Mosby, 1978), pp. 26-27.
Identifylng Issues and Formulating Questions 97
96 EVALUATION

stakeholders to identify the program goals and What are the most important questions for the
transform overly hroad, amhiguous, or ideal- evaluation to answer! With an understanding
ized representations of them into ele ar, explicit, of why an evaluation is desired and by whom,
and a careful specification of the program ob- An evaluation manager for a social services development of new action plans for the
concrete statements of objectives. The more
jectives that key stakeholders agree are central organization summarized his observations operational manager.
closely the objectives describe situations that
to the evaluation, attention can be given to about the use of evaluation findings by program 3. Utilization of evaluation research invo/ves
can be directly and reliably observed, the more
decisionmakers as follows: political activity. lt is based on a recognition
likely it is that a meaningful evaluation will formulating the questions the evaluation will
and focus on who in the organization has
result. Furthermore, it is essential that the be designed to answet We echo Patton's (1997)
1. The utilization of evaluation ar research what authority to make x, y, ar z happen. To
evaluator; evaluation sponsors, and other per- view that the delineation of priority evaluation does not take care of itself. Evaluation re- change programs ar organizations as a re-
tinent stakeholders achieve a workable agree- questions should be organized around a con- ports are inanimate objects, and it takes sult of some evaluation requires support
ment on which program objectives are most cept that generally concems evaluators very human interest and personal action to use from the highest leveis of management.
central to the evaluation and the criteria to be much: utilization. Evaluation results are rarely and implement evaluation findings and rec- 4. Ongoing systems to engender evaluation
used in assessingwhether those objectives have intended by evaluators or evaluation sponsors ommendations. The implications of evalu- use are necessary to legitimate.and formal-
been met. For instance, if one stated objective to be "]mowledge for knowledge's sake."Rather; ation must be transferred from the written ize lhe organizational learning process.
of a job training program is to maintain a low they are intended to be useful, and to be used, page to the agenda of program managers. Otherwise, utilization can become a per-
by those with responsibility for making deci- 2. Utilization ofevaluation, through which pro- sonalized issue and evaluation advocates
drop-out rate, the key stakeholders should
sions about the program, whether at the day- gram lessons are identified, usually de- just another self-serving group vying for
agree to its importance before it is accepted as
to-day management levei or at broader funding mands changed behaviors or policies. This power and contrai.
one of the focal issues around which the evalu- requires the shifting of priorities and the
ation will be designed. or policy leveis (see Exhibit 3-F for an evalu-
If consensus about an appropriate criterion ation manager's view of this process).
SOURCE: Quoted, with permission, from Anthony Dibella, ''The Research Manager's Role in Encouraging Evaluation
is weak, or not attained at ali, it may be wise Unfortunately, the experience of evaluators Use," Evaluation Practice, 1990, 11 (2):119.
for the evaluator to employ multiple criteria is replete with instances of evaluation findings
that reflect the interests of the various stake- that were virtually ignored by those to whom
holders concemed with a particular objective. they were reported. There are numerous rea-
If consensus is wealc or nonexistent about sons why this may happen, many of which are whom the evaluation results are directed are actions or decisions are potentially affected.
which objectives are important, one solution is notwithin the contrai of the evaluatot Program not initially altogether clear in their own minds The evaluator wants to come as dose as possi-
to include ali those put forward by the various circumstances may change, for instance, be- about what information they need for what ble to understanding, in an explicit, detailed
stalceholders and, perhaps, additional objec- tween the initiation of the evaluation and its purposes. fashion, who specificallywill use the evaluation
tives drawn from current viewpoints and theo- completion in ways that make the evaluation With these considerations in mind, we ad- and what specifically they will use it for For
ries in the relevant substantive field (Chen, results irrelevant when they are delivered. But vocate that the development of evaluation instance, the administrator and board of direc-
1990). For example, the sponsors of a job train- !ade of utilization may also occur because the questions involve backward mapping, which tors of the program may intend to use the
ing program may be interested solely in the evaluation does not actually provide informa- starts with a specification of the desired end- evaluation results to set administrative priori-
frequency and duration of postprogram em- tion useful to the decisionmakers for the deci- point then worlcs backward to determine what ties for the next fiscal yeru: Or the legislative
ployment. But the evaluator may propose that sions they must make. Moreover; this can hap- must be dane to get there (Elmore, 1980). committee that oversees a program area may
stability of living arrangements, competence in pen rather innocently as well as through Taking this approach, the essential discussion desire the evaluation as input to their delib-
handling finances, and efforts to obtain addi- ineptness. It may well be, for instance, that an with the evaluation sponsor and other key erations about continued funding for the pro-
tional education be examined as program out- evaluation plan looks like it will produce rele- stalceholders must establish who will use the gram. Or the program monitors in the govem-
comes because these lifestyle features also may vant information but, when that information evaluation results and for what purposes. Note ment agency that has initiated the program
undergo positive change with increased em- is generated, it is notas useful as the recipients that the question is not who isinterested in the may want to know if it represents a successful
ployment and job-related skills. expected. It may also happen that those to evaluation findings. Although relevant, that model that should be disseminated to other
question does not probe the matter of what sites.
98 EVALUAT!ON Identifying Issues and Formulating Questlons 99

In each case, the evaluator should work Analysis of Program


with the respective evaluation users to describe
Assumptions and Theory
the range of potential decisions or actions that
they might consider taking and the form and Evaluation is about assessing how the pro- Service Utilization Plan
nature of information that they would find gram is performing, whether at some global
pertinent in their deliberation. To press this levei or with regard to specific functions and
exercise to the greatest levei of specilicity, the aspects. Most evaluation questions, therefore, Target Target lnteractlon wlth dellvery system
evaluator might even generate dummy infor- are variations on the theme of "Is what's sup- Population
lmpact Theory
mation of the sort that the evaluation might posed to be happening actually happeningl" for
produce, for example, "20% of the clients who example, "Are the intended target participants
complete the program relapse within 30 days," being reached l" 'l\re the services adequately Service Proximal Distal
and discuss with the prospective users what delivered l" or 'l\re the goals being met l" A very Arena }ro!ram-t!rget servlle tlnsact!onsl r-+ Outcomes H Outcomes

this would mean to them and how they would useful analysis of a program for purposes of 1 t 1 t 1 t
use such information. identifying relevant and irnportant evaluation
A careful specification of the intended use questions is to delineate in some detail just Program Program facllltles, personnel, actlvltles
of the evaluation results and the nature of the what it is that is supposed to be happening in
information that is expected to be useful leads a program. The evaluator can construct a rep-
directly to the formulation of questions the resentation, a conceptual model, of how the
Program's Organizatlonal Plan
evaluation must attempt to answer (e.g., "What program is expected to work and the connec-
proportion of the clients who complete the tions presumed between its various activities
program relapse during the first monthl") and and functions and the social benefits it is in-
provides a contextwithin which to set priorities tended to produce. This representation of the
something to be dane or of the method of doing ture, although ali show co=on elements.
forwhich questions are most irnportant. At this program assumptions and expectations can
it." It is generally this sense of the word that Consequently, we will describe representations
juncture, consideration must also be given to then be used to identify those aspects of the
evaluators mean when they refer to program of severa! separate components of program the-
matters of timing. It may be that some ques- program most essential to effective perfor-
theory. It might altematively be called the pro- ory that we have found useful in our own eval-
tions must be answered before others can be mance. These, in tum, raise evaluation-related
gram conceptualization o~ perhaps, the pro- uation activities and that illusttate themes found
asked, ar users may need answers to some questions about whether the key assumptions
gram plan, blueprint, or design. io most variations of this type of analysis.
questions before others because of their own and expectations are reasonable and appropri-
Evaluators have long recognized the irnpor- For this purpose, we depict the typical so-
timetable for decision making. The irnportant ate and, if so, whether the program is enacting
tance of program theory as a basis for formu- cial program as centering on a set of program-
questions can then be organized into related them in an effective manneL
lating and prioritizing evaluation questions, target ttansactions, those points of direct con-
groups, combined and integrated as appropri- What we are describing here is an explica-
designing evaluation research, and interpreting tact between program operations and the target
ate, sequenced in appropriate time lines, and tion of the program theory, the set of assump-
evaluation findings (Bickman, 19 8 7; Chen and population the program serves that occur in
worked into final form in consultation with the tions about the relationships between the strat-
Rossi, 1980; Weiss, 1972; Wholey, 1979). It is, some service arena (see Exhibit 3-G). These
designated users. With this in hand, developing egy and tactics the program has adopted and
howeve~ described and used under various dif- might involve counseling sessions for women
the evaluation plan is largely a matter ofwork- the social benefits it is expected to produce.
ferent names, for example, logic model, pro- with eating disorders in therapists' offices, rec-
ing backward to determine what measures, Theory has a rather grandiose sound to it and
gram model, outcome line, cause map, action reational activities for high-risk youths at a
observations, procedures, and the like must be few program directors would claim that they
theory, and so forth. Moreove~ there is no co=unity cente~ educational presentations
undertalcen to provide answers to the irnpor- were working from any distinct theory. Arnong
general consensus about how best to depict or to local citizens' groups, nutrition posters io a
tant questions in the form that the users re- the dictionary definitions of theory, howeve~
represent program theory, and many different clinic, informational pamphlets about empow-
quire by the time they are needed. we find "a particular conception or view of
versions can be found in the evaluation litera- erment zones and tax law mailed to potential
100 EVALUATION Identifying Issues nnd Formulnting Questlons 101

investors, delivery of meals to the front doors by the program, the intended social benefits
of elderly persons, ar any such point of service will not be achieved.
contact. On one side of this program-target To instigate the change process posited in
transaction, we have the program as an organi- the program's irnpact theory, the program must Program theory
zational entity, with its various facilities, per- first provide the intended services to the target
sonnel, resources, activities, and so forth. On population. If we view the program from the Process lheory
the other side, we have the target participants perspective of the target population, attention
in their lifespaces with their various situations focuses on the points of service delivery and Organizational plan Service utilization plan lmpac:t theory
and behaviors, including their circumstances their accessibility, whether the services are ac- Haw to garner, configure, and deploy How the intended target populatian How the intended intervention for
and experiences in relation to the service deliv- tually delivered to the intended targets, and the resources, and organize program receives the intended amount af the the spedfied target papulation brings
activities se that the inlended service intended intervention through inter~ about Lhe desired social benefits.
ery system that provides them with points of extent to w hich targets complete the full se- delivery system is developed and action with the program's service
contact with the program. quence of services. Every program works within maintained. delivery system.
For purposes of explicating and analyzing a framework of assumptions and expectations
program theory, this sirnple scheme highlights about how to reach the target population, pro-
three different, but interrelated, theory-compo- vide and sequence service contacts with those
nents, each of which focuses attention on an designated as clients, and conclude the rela- The program, of course, must be organized that working relations should be maintained
irnportant facet of program performance. Most tionship when services are no longer needed ar in such a way that it can, indeed, actually with regard to referrals from the Department of
irnportant are the program-target transactions, appropriate. These assumptions and expecta- provide the intended services, which, in tum, Public Health, and so forth.
for they constitute the means by which the tions constitute an irnportant part of program are expected to produce the desired benefits. Adequate resources and effective organiza-
program expects to bring about its intended theory that we will cal! the program's service The third component of program theory, there- tion, in this scheme, are the factors that malce
effects. These transactions are thus operation- utilization plan. fore, has to do with the nature of the program it possible to develop and maintain a service
alizations of the program's impact theory, the ln sirnplest forrn, a service utilization plan resources, personnel, administration, and gen- delivery system that enables utilization of the
assumptions about the change process actu- proposes that if the intended targets experience eral organization. It might be called the pro- services so that the target population receives
ated ar facilitated by the program and the irn- particular encounters and opportunities pro- gram's organizational plan. It can generally be the intervention. Program organization and the
proved conditions expected to result from in- vided by the program's service delivery system, represented as a set of propositions: If the service delivery system it supports are the parts
ducing that change. This irnpact theory may be they will receive the intended services. For a program has such and such resources, facilities, of the program most directly under the contrai
as sirnple as presuming that exposure to infor- program to increase awareness of AIDS risk, for personnel, and so on, is organized and admin- of program admioistrators and staff. These two
mation about the negative effects of clrug abuse instance, the service utilization plan may be istered in such and such a manne~ and engages aspects together are often referred to as pro-
will motivate high school students to abstain sirnply that appropriate persons will read infor- in such and such activities and functions, then gram process, and correspondingly, the as-
oras complexas the ways in which an eighth- mative posters if they are put up in subway cars. a viable organization will result with the capa- sumptions and expectations on which program
grade science curriculum will lead to deeper A multifaceted AIDS prevention program, on bility of developing and/or maintaining the in- process is based may be called the program's
understanding of natural phenomena. It may the other hand, may be organized on the as- tended service delivery system and correspond- process theory.
be as informal as the co=onsense presump- sumptions that if high-risk clrug abusers in ing service utilization. Elements of programs' The intervention the program irnplements
tion that providing hot meals to elderly persons specified neighborhoods encounter outreach organizational theories include such presump- as a result of its organizational and service
improves their nutrition ar as formal as classi- workers and are referred to clinics, and if street- tions as that case managers should have mas- delivery activities, in tum, is the means by
cal conditioning theory adapted to treating front clinics are available nearby, and if cli.ents ter's degrees in social work and at least five which the program expects to bring about the
phobias. Whatever its nature, howeve~ an irn- receive encouragement .from case managers to years' experience, that at least 20 case manag- desired changes in the target population ar
pact theory of some sort constitutes the essence maintain continuing program contact, and if ers should be employed, that the agency should social conditions. Thus, ali three theory com-
of a social program. If the assumptions embod- they receive testing and information at the have an advisory board that represents local ponents are closely interrelated and, collec-
ied in that theory about how desired changes clinics, then high-risk clrug abusers will have business owners, that there should be an ad- tively, can be viewed as constituting the overall
are brought about by program action are faulty, received the preventive service package the pro- ministrative coordinator assigned to each site, program theory (Exhibit 3-H gives a surnmary
or if they are valid but not well operationalized gram intends to delivei:
102 EVALUATION Identlfying Issues and Formulatlng Questions 103

of the theory components). With this overview, persons prepare for and find jobs that pay a
we tum now to a more detailed discussion of living wage.
each of these theory components with particu- The simplest program impact theory,
lar attention to howthe evaluator can construct therefore, is generally the basic "two step" in
a workable representation of program theory
and use it to analyze the program and generate
potentially important evaluation questions.

The Program Impact Theory


which services change some intermediate con-
dition such as motivation or employability
that, in tum, helps ameliorate the social con-
ditions of cancera, for example, by reducing
alcohol abuse or unemployment (Lipsey and
Pollard, 1989). More complexprogram theories
Mass
Media
Campaign
- Dietary
Knowledge
&Awareness
'---+
Healthier
Diet

may have more steps along the path between

- -
Training Knowledge
The central premise of any social program program and social benefit and, perhaps, in- & Motivation
Regular
Program on Useof Reduced
is that the services it delivers to the target volve more than one distinct path. Exhibit 3-I About i--.. Hearing
Hearing Protective
population induce some change that improves illustrates causal diagrams for severa! different Protection Safety Loss
Behavior Devices
social conditions. The program impact theory program impact theories. The distinctive fea-
is the set of assumptions embodied in the tures of any representation of program impact
program about how its services actuate or fa- theory are that each element is either a cause
cilitate the intended change. Program impact or an effect and that the causal linkages be- Employment
theory, therefore, is causal theory: It describes tween those elements show a chain of events Skills
a cause-and-effect sequence in which certain that begins with program actions and endswith
program activities are the instigating causes change in the social conditions the program Job Job Search Better Improved
and certain social benefits are the effects they ultimately intends to improve. Tralning r-1>1 & lnterview Job Economic
Skills Status
eventually produce. Evaluators, therefore, typi- Depiction of the program's impact theory
cally represent program impact theory in the has considerable power as a framework for
form of a causal diagram showing the pattem analyzing a program and generating significant Motivatlon
to Work
of cause-and-effect linkages presumed to con- evaluation questions. First, the process of mak-
nect the program activities with the expected ing that theory explicit brings a sharp focus to
outcomes (Chen, 1990; Lipsey, 1993; Martin the nature, range, and sequence of program
and Kettne~ 1996). Because programs rarely outcomes that are reasonable to expect and lmproved
exercise complete, direct contrai over the social may be appropriate for the evaluator to inves- Math Knowledge
conditions they are expected to improve, they tigate. Every event following the instigating for Students Practical
must generally work indirectly by attempting program activity in the causal diagram repre- Life Skills
to alter some criticai but manageable aspect of senting a program's impact theory is an out- Computer-Alded lmproved
the situation, which, in tum, is expected to lead come. Those following directly from the insti- Math lnstruction Computer
to more far-reaching improvements. For in- gating program activities are the most direct in High School Skllls for
Students
stance, a program cannot make it impossible outcomes, often called proximal or immediate
for people to abuse alcohol, but it can attempt outcomes, whereas those further down the Reduced Costs
to change their attitudes and motivation to- chain constitute the more distal or ultimate Lower Teacher-
Pupll Ratios; to Educatlonal
ward alcohol in ways that help them avoid outcomes. Program impact theory highlights Fewer Math System
abuse. Similarly, a program may not be able, at the dependence of the more distal, and gener- Teachers Needed
a stroke, to eliminate poverty in a target popu- ally more important, outcomes on successful
lation, but it may be able to help unemployed attainment of the more proximal ones. For a
104 EVALUAT!ON
Identifylng Issues and Formulating Questions 105

full understancling of program impact, there- assumed that if the program does its part by
fore, it may be important for the evaluation to implementing the campaign in such a way that
examine the proximal outcomes even when knowledge and awareness are heightened, the
they are not themselves the accomplishments intended social benefits will follow. These as-
forwhich the program will be held accountable. sumptions have been referred to as the pro- Proximal or Distal or
Program
A second, and related, contribution of pro- gram's "conceptual theory" (Chen, 1990; see
Actions lmmediate -- Ultimate
gram impact theory to forrnulation of key also Exhibit 3-Jl, although, again, we would Outcome Outcome
evaluation questions is the distinction it re- prefer conceptual hypothesis. This hypothesis
. . ...
veals between two rather different sets of as- is the part of the impact theory that assumes
sumptions inherent in the program. The first that success in changing the targeted aspect of
set of assumptions represents the expectation the problem (proximal outcomes) will, in tum, . :
that the program actions will have the intended result in the desired social benefits (distal out- :
effects on the proximal ar immediate out- comes).
. :

....:. .. .
comes. For instance, a mass media campaign This aspect of impact theory, of course,
about AIDS must assume that the public ser-
vice anoouncements, billboards, and other pro-
draws the evaluator's attention to another set
of linl<ages that might bear investigation and
...
motion it does (program actions) will-result in helps forrnulate questions aboutwhether, given Action Conceptual
heightened awareness and lmowledge of the proximal outcomeA, distal outcome B actually Hypothesis Hypothesis
risk of unsafe sex practices (proximal out- follows. The evaluator, therefore, might find it
comes). This set of assumptions thus links important not only to ask if awareness and
program actions to the immediate outcomes lmowledge of AIDS risk increased, but if such
expected to follow from them and has been increases were further associated with changed
referred to as the program's "action theory" behavior and reduced incidence of AIDS. This Example:
(Chen, 1990; see also Exhibit 3-Jl. Because it is line of analysis helps identify the full set of
only one linl< in the impact theory, however, we outcome variables potentially relevant to an
would prefer to call it the action hypothesis. impact evaluation.
Articulating the action hypothesis allows the Of course, the intended outcomes may not lmproved
Protein - Decreased --
evaluator to identify the important evaluation be achieved by a program and the eval uator will Physical
Supplements Malnutrition
questions that 'relate to it, particularly with generally want to be in a position to explain Growth
regard to whether the intended actions were
implemented and the expected proximal effects
why unfavorable results occurred. One of the
useful applications of program impact theory is
. .
.
.
. . .
achieved. for identification of the various points in the
The second set of assumptions inherent in anticipated chain of events where things may .. .
program impact theory connects the proximal
outcomes with the distal ones. ln the mass
not happen as expected. At a minimum,
evaluators maywish to distinguishimplemen-
.
.
media campaign, for instance, it is expected tation failure from theory failure. . .......:
that if lmowledge and awareness are height-
ened (proximal outcomes), appropriate safe-sex
Implementation failure is the failure of the
program to adequately deliver the services ar
....
behavior will follow with corresponcling de- perforrn the actions that are supposed to start Action Conceptual
creases in AIDS transmission (distal out- off the change process expected to lead to im- Hypothesis Hypothesis
comes). This part of the process is completely proved social conditions (Exhibit 3-K provides
out of the contrai of the program; it is only an example). Obviously, if the program is not SOURCE: Adapted from Huey-Tsyh Chen, Theory-Driven Evaluations (Newbury Park, CA: Sage, 1990).

.,,'i.'-......._ ___ _
Identifying lssues and Formulating Questlons 107
106 EVALUATION

implemented, ar implementation is incom- this situation closely, it fust requires moving


plete ar weak, we would not expect it to be very one's hand to properly manipulate the switch,
successful in producing the intended out- then having the switch activate a flow of cur-
comes, either the most immediate proximal rent that causes the light to illuminate. Our
was implemented but not all the lessons outcomes ar the ultima te outcomes to which it assumptions about the relationship between
A team of administrators and staff from a
Southern school district worked with evaluators
were covered. Severa! teachers were trained aspires. Evaluatars gather information on this moving our hand and the position of the switch
to teach the LST course and a portion oi it aspect of program performance thrnugh assess- are the action hypothesis. We can fail to change
to plana program to reduce problem behaviors-
was taught in health classes. ments of program process, including attention the position of the switch because we do not
including drug and alcohol use, delinquent
During the second school year, cooperative to both service utilization and program organi- move our hands at all, ar not the way we
behavior, pregnancy, nonattendance, and
learning was implemented by more than
misconduct-in a middle school with a large zational issues. intended (implementation failureJ, ar because
half the teachers but in lewer than half their
proportion of high-risk students. A program lessons. The CED course was provided to Programs can also fail when the intended we move them exactly the way we intend but
theory was developed on the basis of research most oi the eighth graders with about half program activities are implemented but those somehow those moves are not successful in
showing that improvements in social bonding, the intended number of sessions. Seventh- activities do not actually have the intended flipping the switch (failure of the action hy-
social competency skills, and school success grade high-risk students received the SPS effects. ln the example of the mass media cam- pothesis). If the switch is flipped, our assump-
should result in a reduction in problem behav- course but eighth graders got neither the
paign on AIDS risk, the campaign may be tion that the light will come on represents our
iors. Funding was obtained to support a four-year LST nor VP modules. About one-third of the
high-risk students received tutoring, but the implemented just as planned but may not be conceptual hypothesis. We can quite success-
program with the following components:
average was only five sessions for the year. widely noticecl and, therefare, not result in any fullymanipulate the switch and still get no light
/nstructional improvement: Cooperative ln the third year, all the program compo- heightened awareness ar greater Imowledge of if that part of the impact theory is in erro~ for
learning techniques were to be used school- nents except tutoring and mentoring were AIDS risk. This is one form of theory failure, example, the circuit is broken ar the switch is
wide. ln addition, for high-risk students, one- incorporated into a single life focus course in particulru; a failure of the action hypothe- not hooked up to a circuit (failure of the con-
on-one tutoring was to be provided by com- adapted to each grade levei. Not all the sis-the program services do not bring about ceptual hypothesisJ.
munity volunteers. intended material was covered, however, so
the immediate outcomes that are expected. The concept of program impact theory, the
Mentoring: High-risk students were to be some students did not get some of the
paired with teachers who volunteered as components and, in other cases, received Another form of theory failure involves the distinctions between proximal and distal out-
"academic godparents" to tutor them, moni- fewer lessons than intended. The mentoring conceptual hypothesis. This form of theory comes, and the related distinctions between the
tor their progress, and share in recreational component improved from the previous failure occurs when the program implements program's action and conceptual hypotheses,
activities. year, but the high-risk tutoring component the intended services and, indeed, achieves the therefore, can alert the evaluatar to different
Social competency promotion: The Botvin deteriorated. Cooperative learning was im-
expected proximal outcomes, but those, in aspects of program performance that may be
life skills training (LST) and Manning's cog- plemented more fully but still at only about
two-thirds the intended levei.
1 tum, do not lead to the expected distal out- appropriate to assess. It is for this reason that
nitive self-instruction (CSI) curriculum were
to be implemented schoolwide. These were During the final year of the program, imple- 1 comes. Thus, a mass media campaign may be articulation of program impact theory during
augmented with a social problem solving mentation of a few of the components im-
1 hugely successful in raising awareness and the planning stage of an evaluation is an im-
(SPS) course for seventh graders, a violence proved but, in general, the overall levei of lmowledge about AIDS risk and how to reduce portant form of analysis for the evaluatar to
prevention (VP) curriculum for eighth grad- the program declined. 1 it, but people may not translate that lmowledge undertake. That exercise almost always yields
ers, and a career and educational decision into changed sexual behaviar (one of the distal very relevant evaluation questions regarding
ln summary, the program was never im-
skills (CED) class for sixth and eighth graders.
plemented according to the initial intentions of outcomes expected), and consequently, there whether key program actions were imple-
Assessment of the implementation of this mu~ the team that developed it. The outcome eval- will be no reduction in the incidence of AIDS mented as intended and, if so, whether they
ticomponent program revealed the following: uation examine d change in measures of problem (the social benefit the program ultimately produced the expected effects. Of course, the
The first school year was largely a start-up
behavior and antisocial attitudes, positive school hopes to produce). Exhibit 3-L provides an ex- evaluatar must also identify those key program
adjustment, and school attendance. Not sur- ample of theory failure. activities in some detail so that appropriate
period. A group of teachers was trained to
use cooperative learning techniques, but prisingly, the resu\ts showed no reductions on Although it simplifies considerably, we questions can be raised about program imple-
only 13 actually used them. The CED course any of these variables. might liken the causal sequence embodied in mentation. This is where program process the-
program impact theory to the assumption that ory, encompassing service utilization and pro-
SOURCE: Adapted from Denise C Gottfredson, Carolyn M. Flnk, Stacy Skroban, and Gary D. Gottfredson, "Making llipping a switch turns on a light. If we analyze gram arganization, can be helpful.
Prevention Work," in Establishing Prevencive Services, eds. R. P. Weissberg, T. P. Gullotta, R. L Hamplon, B. A. Ryan, and
G. R. Adams (Thousand Oaks, CA: Sage, 1997), pp. 219-252.
108 EVALUATION Identlfying Issues and Formulating Questions 109

The Progrmn Service some appropriate point prior to first program


Utilization Plan contact through a point where there is no
longer any contact. Exhibit 3-M shows an ex-
Mental health services for children are often Project began sooner, were more individualized, Whereas program impact theory describes ample of a simple service utilization flowchart
underfunded, fragmented, and limited in variety. had more variety, lasted longer with fewer the chain of events leading from program-target for a hypothetical aftercare program for released
The five-year, $80 million Fort Bragg Demon- dropouts, showed greater continuity and more transactions to the intended improvements in psychiatric patients. One of the desirable fea-
stration Project was designed to test an inno- parent involvement, and represented better social conditions, the service utilization plan tures of such charts is the identification of
vative alternative to traditional mental health matches between treatment and needs as describes the sequence of events through whlch possible situations in whlch the program tar-
systems for children. Developed around the judged by parents. The condusion of the clients engage in those transactions. The ser- gets are not engaged with the program as in-
1
concept of a "continuum of care,' the Demon- evaluators was that "the Demonstration was
vice utilization plan is the set of assumptions tended. For instance, for the co=unity after-
stration Projectwas organized to deliver needed executed with sufficient fidelity to provide an
and expectations about how the targets will care program in Exhibit 3-M, we see that
services on an individualized basis at all levels of excellent test of the program theory-the
make initial contact with the program and be fonnerly hospitalized psychiatric patients in
severity using case management and interdis- continuum of care. 11
lhe impact evaluation examined parents'
engaged with it through the completion of the the target population may not receive the
ciplinary treatment teams to integrate and
caordinate care. This variant of managed care satisfaction, treatment costs, and mental health intended services. Its distinctive theme is that planned visit from a social worker or referrais
was expected to result in improved treatment outcomes with the following results: it describes the program-target transaction to community agencies and, as a consequence,
outcomes and Jower cost of care per client. from the perspective of the targets and their may receive no service at ali. The size of this
Parents were more satisfied with the ser-
lhe Demonstration Project was set up for the experience and hlstory of engagement with the group will be a function of how vigorously the
vices from the Demonstration Project than
42,000 military dependents under age 18 in the in the comparison sites.
program. An explicit, even if relatively infor- program contacts potentially eligible cases and
vicinity of the Fort Bragg military base in North The costs per treated child were substan- mal, service utilization plan pulls into focus the establishes case management for them. Simi-
C'arolina. For evaluation purposes, two compar- tially higher in the Demonstration Project, criticai assumptions about how and why the larly, a service utilization flowchart can hlgh-
ison sites were selected-Fort Campbell, Ken- not lower as expected. intended recipients of service will actually be- light such issues as insufficient referrais from
tucky, and Fort Stewart, Georgia. Dependent Mental health data collected on 984 chil- come engaged with the program and follow gateway agencies, program dropouts, early ter-
children in those areas received mental health dren and families within 30 days after entry through to the point of receiving sufficient minations prior to receiving the full-service
care under a conventional health insurance plan into the system and in two follow-up waves services to initiate the change process repre- package intended, and other such issues of
in which parents used independent practitioners six months apart showed essentially no dif-
ferences in clinica! outcomes between the
sented in the program impact theory. ln the incomplete service or service not offered or not
or agencies and were reimbursed, subject to
deductibles, by CHAMPUS, the military insur- Demonstration and comparison sites. Of example of a mass media campaign to reduce delivered. Of course, at the sarne time, it por-
ance provider. 116 distinct comparisons representing gen- AIDS risk, the service utilization plan would trays the pattem of outreach, intake, receipt of
lhe evaluators identified criticai implemen- eral and individualized measures reported describe how persons at risk for AIDS will service, and exit from service that represents
tation and outcome issues with the aid of a
by children, parents, therapists, and trained encounter the co=unications disseminated the program's scheme for making services
raters, 101 show no significant difference, 7
carefully developed program theory description. by the program and engage them sufficiently available to the targets.
favored the comparison, and 8 favored the
To assess implementation, the program-as-
Demonstration.
for their message to be received. O~ for another As a too! for program analysis and formu-
implemented was compared with the program- example, the service utilization plan for a ]ation of evaluation questions, articulating the
as-planned. The results showed that, as intended, ln short, the continuum of care concept was neighborhood afterschool program for latchkey service utilization plan contributes an impor-
the demonstration had implemented a single well implemented but did not produce the children would describe how parents are ex- tant perspective on how the program is de-
point of entry to services for the target popu- effects that were expected on the basis of the
pected to learn of the program and enroll their signed and what assumptions are made about
lation, provided a comprehensive range of program theory. Or, as the evaluators put it,
children as well as how the children are ex- the ways in whlch the target population is
services, and established case management and "Commonly accepted wisdom about what is a
better quality system of care is called into pected to get to the program regular!y and expected to engage the program services. That
treatment teams to coordinate services. More-
over, relative to the comparison sites, the ser- question.11 return home again afterward. perspective facilitates the identification of im-
vices the children received in the Demonstration A program's service utilization plan can be portant questions of program performance re-
usefully depicted in a flowchart that tracks the lated to whether the appropriate target popula-
SOURCE: Adapted from Leonard Bickman, "lmplications of a Children 1s Mental Health Managed Care Demonstralion various paths program targets can follow from tion is being served and in sufficient numbers,
Evalualfon," Journal o{ Mental Hea/th Administration, 1996, 23(1):107-118.
110 EVALUAT!ON ldentlfying lssues and Formulating Questions 111

what barriers there may be to entry into the consistentwith the schemes we have presented
program, the extent to which full and appropri- for irnpact theory and the service utilization
ate service is completed by an acceptable pro- plan, it is desirable to adopt a form that centers
portion of those beginning service, and whether on those target-program service transactions
Program desirable follow-up contact is made following that constitute the major points of contact
Patients plan service completion. An evaluator who has between the program and the target population.
released made the effort to explicate the program's ser- The first element of the organizational plan,
from hospital vice utilization plan and analyze its irnplica- then, will be a description of the program's
tions for program performance will be able to objectives with regard to the particular services

---------------,. raise many important issues for consideration


in developing the questions around which the
evaluation will be designed.
it will provide: what those services are, how
much is to be provided, to whom, and on what
schedule. The next element of the organiza-
tional plan might then describe the resources
Not vlslted
bysoclal ------, The Program 's Organizational Plan and prior functions necessary to engage in
those critica! service activities. For instance,
worker 1
1 The organizational arrangements and pro- sufficient personnel with appropriate creden-
1 gram operations that constitute the routine tials and skills will be required as will logistical
1
1 functions of the program are based on a set of support, proper facilities and equipment, fund-
1 beliefs, assumptions, and expectations about ing, supervision, clerical support, and so forth.
1 what the program must do to bring about the As with the other portions of program the-
Not referred 1
intended target-program service transactions. ory, it is often useful to describe a program's
to community
services
-----1 1
These beliefs, assumptions, and expectations
are what we cal! the program's organizational
organizational plan with a chart or diagram.
Exhibit 3-N presents an example in that form
1 plan. The organizational plan is articulated that describes the major organizational compo-
1
1 from the perspective of program management nents of the hypothetical aftercare program for
1 and encampasses both the functions and activi- psychiatric patients whose service utilization
1 ties the program is expected to perform and the scheme is depicted in Exhibit 3-M. A rather
Does not
1
1 human, financial, and physical resources re- co=on way of depicting the organizational
receive i------1 quired for that performance. Central to this plan of a program is in terms of inputs, repre-
community scheme are the program services, those specific senting the resources and constraints applica-
services
activities that constitute the program's role in ble to the program, and activities, indicating
the target-program transactions expected to the services the program is expected to provide.
lead to social benefits. Howeve~ it also must When included in a full logic model, these
include those functions that provide essential schemes typically represent receipt of services
preconditions and ongoing support to the orga- (service utilization) as program outputs, which,
Readmitted nization's ability to provide its prirnary ser- in tum, are related to the desired outcomes.
to hospital vices, for instance, fund-raising, personnel Exhibit 3-0 shows such a scheme, drawn from
management, facilities acquisition and main- a widely used workbook prepared by the United
tenance, political liaison, and the like. Way of America.
1 1 There are many ways the evaluator may Naturally, a description of program organi-
L---------------~----------- depict the program's organizational plan. Tu be zation and articulation of the underlying as-
112 EVALUATION Identlfying Issues and Pormulating Questions 113

Outcomes
Inputs Acriviries Outputs lnirial lntermediate longer Term
Case Contact Agency pravides Program providl!S Pregnant leens Teens are Teens a11aw Teens de!iver Bables achleve
Monitor hospital releases MSW program parenting cfosses on allend program. knowledgeable proper nutritfan healthy babfes. appropriate 12-manth
manager, parltlme RN prenata~through-
Assign social workers abaut prenatal and health m!!estones for
Jnstructor, natlonally infant nutrftlon, nutritian and guJdellnes, phys!C<JI, motor,
Arrange appointments
cerfilied educatfon deve!opment, safety, health gulde!fnes. verbal, and social
Revlew eligibllity manuais, videos, and and care!aklng develapment.
Establish intake ather teachfng toais. delivered ln h!gh
schoals tw!ce a weelc
for one hour to teen
mothers from three
months prior, to one
Case Management year after, del!very of
a chlld.
Administration lnterview cllents
Establish priorities lnterview families Agency and hlgh TeE!ns are Teens provide proper care,
school ldentify preg knowledgeable foedlng. and saciai Jnteractlon to
Allocate resources Assess service needs Target nant leens ta partlcJ. about proper theJr babfes,
Train staff Refer to services Population pale ln program. care, feedJng.
Monitor program Make initial appointment and social
Develop referrai Assis! with transportation lnteractfon with

network Follow up with provlder fnfants.


Follow up with client SOURCE: Adaped from United Way of America, Measuring Program Outcomes: A Practical Approach. Alexandria, VA:
Ongoing assessment Author, 1996, p. 42. Used by permission, United Way of America.

Casa closing schoolwork for every child in every family as- be raised about specific managerial functions
Monitor readiness signed to their caseload. A related question is or the effectiveness of overall program admin-
for discharge
Feedback to hospital whether those functions and activities are op- istration. Similarly, the nature and relationship
Feedback to family timally configured for their purposes, represent of governing and advisory boards may be of
Transition plan for client appropriate standards of professional practice, interest as well as relations with other agencies,
Ciosa case are adequately supervised, and so forth. superordinate organizational structures, politi-
Another set of important organizational cal entities, and so on.
questions relates to program resources. An
evaluation may encompass questions about How to Articulate
whether the personnel are sullicient in num- Program Theory
sumptions, expectations, and intentions reveal instance, relates to whether the program is bers or credentials for their assigned tasks, the
many aspects of the program related to how actually implementing the functions and ac- adequacy of program funding, availability of the Program theory in the detail indicated
effectively it performs its various functions. tivities that are intended and in the intended requisite facilities and equipment, and many above is rarely written down in the documen-
Thus, the evaluator can use the organizational way. Questions on this topic can be rather other such matters. Still another category of tation the p10gram has on hand for the evalu-
planto generate evaluation questions that may general o~ more often, quite specific and de- organizationally important issues has to do ator to examine, although the broad outlines
be relevant to the design and planning of the tailed, for instance, whether the case managers with the administration, management, and will likely appear in statements of the pro-
evaluation. One major category of issues, for are contacting the teachers about the child's govemance of the program. Questions might gram's mission and services or in funding pro-
114 EVALUATION ldentifylng lssues and Pormuiat!ng Questlons 115

posals and anoual reports. Typically, then, the there are important areas of the program con- swer can be much more challenging. Once
and research literature about that type of pro-
evaluator must articulate the program theory gram and sources of expertise and experience ceptualization that are vague, undetermined, articulated, most of the questions about the
on the basis of an initial investigation of the among the members of the professions in- or inconsistent. ln such cases, it may be appro- program that arise during the planning process
program being evaluated. Once articulated in a volved, for example, social work, nursing, psy- priate for the evaluation itself to encompass a are lil<ely to seem interesting to some stalce-
forro endorsed by key program stalceholders, chiatry, or teaching. systematic assessment of the program theory holder or anothei; or to the evaluators them-
the program theory can itself be an object of The greatest difficulty the evaluator will aimed at identifying wealrnesses and assisting selves. Rarely will resources be available to
evaluation. That is, an important evaluation encounter is that the various components of program personnel in clarifying and refining address them all, however At this juncture, it
question may relate to how well conceptualized program theory often are implicit rather than their understancling of what the program is especially iroportant for the evaluator to
the program is, and the evalu ator may conduct explicit and may be in the forro of tacit knowl- should be doing and why (Chapter 5, on assess- focus on the purpose of the evaluation and the
an explicit assessment of the program theory. edge that is so routinized in the program con- ing program theory, describes how this might expected uses to be made of its findings. There
be dane). is little point to investing time and effort in
'".:h.ipter 5 of this volume describes in some text that it is rarely thought about ar cliscussed.
detail how program theory can be evaluated. The evaluator attempting to describe the pro- developing inforroation that is of little use to
Because program theory must first be articu- gram theory, therefore, must generally draw it COLLATING EVALUATION QUESTIONS any stal<eholder
lated before it can be evaluated, Chapter 5 also out piecemeal from program inforroants, avail- AND SETTING PRIORITIES That said, we must caution against an
describes the ways the evaluator can reveal and able documents, and the professional and re- overly narrow interpretation of what inforroa-
express program theory. When program theory search literature, and then attempt to synthe- tion is useful. Evaluation utilization stuclies
is forroulated for the purpose of analyzing a size the pieces into a coherent whole. This The evaluator who thoroughly explores stal<e- have shown that practical, instrumental use,
program to identify pertinent evaluation ques- exercise must involve consderable interaction holder concems and conducts an analysis of for example, for program decision mal<ing, is
tions, as cliscussed in this chaptei; the sarne with program stal<eholders, especially admin- program issues guided by carefully developed only one of the contributions evaluation infor-
procedures are applicable. Because a full discus- istrators, who should provide both critical in- descriptions of program theory will tum up mation mal<es (Leviton and Hughes, 1981;
sion is provided in Chapter 5, we will mention put and feedback on each iteration the evalu- many questions that the evaluation might ad- Rich, 1977; Weiss, 1988). Equally important in
only a few general points here. ator produces. A useful way to approach this dress. The task at this point becomes one of many cases are conceptual and persuasive
It is, for instance, important to recognize task is to draw figures and charts such as those organizing those questions accorcling to clis- uses-the contribution of evaluation finclings
that articulation of program theory should be shown in Exhibits 3-I, 3-M, 3-N, and 3-0 for tinct themes and setting priorities among to the way in which a program and the social
mainly a process of cliscovery and not one of the various components of program theory, them. problems to which it responds are understood
invention. The evaluator is rarely the authori- then go over them in detail with program in- Organization is generally rather straight- and debated. Evaluations often identify issues,
tative voice with regard to how the program is forroants to obtain feedback for refinement. forward. Evaluation questions tend to cluster frame analysis, and sharpen the focus of cliscus-
expected to work. The understanclings of those It is wise to avoid evaluation jargon in this arouod clifferent program fuoctions (e.g., re- sion in ways that are influential to the decision-
persons who originate, plan, administei; and task. Most program administrators will have cruitm.ent, services, outcomesJ and, as noted mal<ing process even when there is no clirect
staff a social program are the primary source of little notion of what is meant by "program earliei; arouod clifferent evaluation issues connection evident between any evaluation
inforroation on this matter The evaluatoi; of theory" if asked outright and are lilrely to as- (need, design, iroplementation, iropact, effi- fincling and any specific program decision. A
course, may play a large and creative role in sume it means something more formal and ciency). ln adclition, evaluation questions tend fuller cliscussion of this issue is presented in
interpreting and organizing that inforroation. abstract than it does in this context. On the to show a natural hierarchical structure with Chapter 12; our purpose here is only to point
Moreovei; few programs are so unique that they other hand, inquiries about how the program manyvery specific questions (e.g., ''Are elderly out the possibility that some evaluation ques-
bear no resemblance to at least some other works, whatvarious personnel do and why, and homebound persons in the public housing proj- tions may be important to answer even though
programs whose funders, administrators, staff, other such questions at a practical levei gener- ect aware of the programl") nested uoder no immediate use or user is evident.
and so forth can be consulted by the evaluator ally lead to fruitful and often lengthy cliscus- broader questions (''Are we reaching our target With the priority evaluation questions for
for adclitional perspectives on how such pro- sions that can be very inforroative. It does population?"). a program decided on through some reasonable
grams should work. There may also be perti- sometimes happen, nonetheless, that the effort Setting priorities to determine which ques- process, the evaluator is ready to design that
nent inforroation available from professional to explicate program theory will reveal that tions the evaluation should be designed to an- substantial part of the evaluation that will be
116 EVALUAT!ON
Jdentifying lssues and Formulatlng Questions 117

devoted to ttying to answer them. Most of the program theory or plan for addressing that ll!l When these various procedures have generated a full set of candidate evaluation
remainder of this book discusses the ap- need, the implementation of the program plan questions, the evaluator must organize them into related clusters and draw on
proaches, methods, and considerations related and the associated program process, the impact stalceholder input and professional judgment to set priorities among them. With the
to that task. That discussion is organized to or outcome of the program implementation on priority evaluation questions for a program deterrnined, the evaluator is then ready
follow the natural logical progression of evalu- the social need, and the efficiency with whlch to design the part of the evaluation that will be devoted to answering them.
ation questions and thus addresses, in tum, the program attains its outcomes.
how to assess the need for a program, the

ll!l A critical phase in evaluation planning is the identification and formulation of the
questions the evaluation will address. Those questions focus the evaluation on the
areas of program performance most at issue for key stakeholders and guide the design
so that it that will provide meaningful information about program performance.
Good evaluation questions, therefore, must identify clear, observable dimensions of
program performance that are relevant to the program's goals and represent domains
in which the program can realistically be expected to have accomplishments.
~ What most distinguishes evaluation questions, howeve~ is that they involve criteria
by whlch the identified dimensions of program performance can be judged. If the
formulation of the evaluation questions can include performance standards on which
key stalceholders agree, evaluation planning will be easier and the potential for
disagreement over the interpretation of the results will be reduced.
18 To ensure that the matters of greatest significance are covered in the evaluation
design, the evaluation questions are best formulated through interaction and nego-
tiation with the evaluation sponsors and other stalceholders representative of signifi-
cant groups or distinctly positioned in relation to program decision malcing.
ll!l Although stalceholder input is critical, the evaluator must also be prepared to identify
program issues that rnight warrant inquiry. This requires that the evaluator conduct
a somewhat independent analysis of the assumptions and expectations on which the
program is based.
ll!l One useful way to reveal aspects of program performance that may be important to
assess in an evaluation is to malce the program theory explicit. Program theory
describes the assumptions inherent in a program about the activities it undertalces
and how those relate to the social benefits it is expected to produce. It encampasses
impact theory, whlch links program actions to the intended outcomes, and process
theory, whlch describes a program's organizational plan and scheme for ensuring
utilization of its services by the target population.
Needs assessment An evaluative study that answers questions about the social conditions a
program is intended to address and the need for the program. Needs ASSESSING THE NEED
assessment may also be used to determine whether there is a need for a new
program and to compare ar prioritize needs within and across program areas. FORAPROGRAM
Key informants Persons whose personal ar professional position gives them a perspective on
the nature and scope of a social problem ar a target population and whose
views are obtained during a needs assessment. Previous chapters provided an overview of evaluation and an orientation to the criticai
Survey Systematic collection of information from a defined population, usually by themes in tailoring an evaluation to program ccumstances and formulating the specific ques-
means of interviews ar questionnaires administered to a sample of units in tions an evaluation wil1 be designed to answer. Beginning with this chapter, we tum to fuller dis-
the population. cussion of the various methods and approaches evaluators use to address different categories of
evaluation questions.
Focus group A small panei of persons selected for their knowledge ar perspective on a
topic of interest that is convened to discuss the topic with the assistance of a
The category of evaluation questions that is logically most fundamental to program evalu-
ation has to do with the nature of the social problem the program is ex]iected to ameliorate and
facilitator. The discussion is usually recorded and used to identify important
themes ar to construct descriptive summaries of views and experiences on
the needs of the population experiencing that problem. These questions follow from the assump-
tion that the pwpose of social programs is to bring about improvement in problematic social con-
the focal topic.
ditions and that they are accountable to those who fund and support them for making a good
Social indicator Periodic measurements designed to track the course of a social condition
faith effort to do so.
over time. Needs assessment, in general, is a systematic approach to identifying social problems, deter-
lncidence The number of new cases of a particular problem ar condition that arise in mining their extent, ond accurately defining the target population to be served and the nature of
a specified area during a specified period of time. their service needs. From a program evaluation perspective, needs assessment is the meons by
The number of existing cases with a particular condition in a specified area at
which on evaluator determines if. indeed, there is a need for a program and, if so, what program
Prevalence
a specified time.
services are most appropriate to that need. Such an assessment is criticai to the effective design
of new programs. However, it is equally relevont to established programs because there are many
Population at ris.k The individuais ar units in a specified area with characteristics judged to
circumstances in which it cannot merely be assumed that the program is needed or that the ser-
indicate that they have a significant probability of having ar developing a vices it provides are well suited to the nature of the need.
particular condition. What makes the assessment of the need for a program so fundamental, of course, is that a
Population in need The individuais ar units in a specified area that currently manifest a particular program cannot be effective at ameliorating a social problem if there is no problem to begin with
problematic condition. or if the program services do not actually relate to the problem. The concepts ond procedures an
The extent to which the criteria used to identify a target population result in evaluator con use to conduct this criticai investigation of the nature and extent of the need for a
Sensitivity
the inclusion of individuais ar units that actually have ar will develop the program are discussed in this chapter.
condition to which the program is directed.

Specificity The extent to which the criteria used to identify the target population result
in the exclusion of individuais ar units who do not have ar will not deveio
the condition to which the program is directed.

Rate The occurrence ar existence of a particular condition expressed as a


proportion of units in the relevant population (e.g., deaths per 1,000 adults).
A s we described in Chapter 1, a fundamen-
tal premise of program evaluation within
the human service domain is that effective
programs are instruments for improving social
conditions. Indeed, bringing about such im-
provement is the primary mission and reason
119
120 EVALUATION Assessing lhe Need for a Program 121

for being of social programs (which is not to say erally referred to as needs assessment. Its pur- dition. Thus, the post-World War II attention the program is meant to address as well as
that they are not also influenced by other pose is to determine if there is a need ar prob- to mental illness was heavily influenced by the precise information about the corresponding
political and organizational imperatives). lem and, if so, what its nature, depth, and scope efforts of a single congressman; federal pro- program targets and the context in which the
Whether a program addresses a signilicant so- are. ln addition, needs assessment often en- grams for mental retardation received a major intervention operates ar will opera te. Here are
cial need in a plausible way and does so in a campasses the process of comparing and priori- boost during Tohn E Kennedy's presidency be- a few examples of what can happen when ade-
manner that is responsive to the circumstances tizing needs according to how serious, ne- cause he had a siblingwith mental retardation, quate diagnostic procedures are ignored:
of those in need, therefore, are essential ques- glected, ar salient they are. improved automobile safety can be credited to
tions for evaluating the effectiveness of a social Within the context of program evaluation, a considerable degree to Ralph Nader's advo- The problem of high unemployment
program. however, the primary focus of needs assess- cacy; and efforts to contra! illegal and improper rates in inner-city neighborhoods frequently
Answering these questions for a given pro- ment is not on human needs broadly defined delivery of health and welfare services have has been defined as reflecting the paucity of
gram first requires a description of the social but, rather, on social conditions deemed unsat- most often come about because of exposs in employment opportunities in those neighbor-
problem the program intends to ameliorate. isfactory through some process of social judg- the mass media and the activities of interest hoods. Programs have therefore been estab-
With that description in hand, the evaluator ment and presumed remediable by social pro- and pressure groups, including the organized lished that provided substantial incentives to
can ask if the program theory embodies a valid grams. The essential tasks for the program efforts of those in need themselves. businesses for locatiog in inner-city neighbor-
conceptualization of the problem and an appro- evaluator as needs assessor are to identify the Nevertheless, evaluators do contribute sig- hoods. Subsequent experiences often found
priate means of remedying it. If that question decisionmakers and claimants who constitute nificantly to efforts to improve the human and that most of the workers these businesses hired
is answered in the affirmative, attention can the primary stal<eholders in the program do- social condition, though not by mobilizing the carne from outside the neighborhood that was
tum to whether the program is actually imple- main of interest, describe the "problem" that disaffected, storming the barricades, ar shoot- supposed to be helped.
mented in line with the program theory and, if canceras them in a manner that is as careful, ing from the lip. Rather, they contribute in
so, whether the intended improvements in the objective, and meaningful to both groups as mundane but essential ways by applying their
repertoire of research techniques to systemati- After a social intervention designed to
social conditions actually result and at what possible, and help draw out the implications of
cally describe the nature of social problems, prevent criminal behavior by adolescents was
cost. Tims, the logic of program evaluation that diagnosis for structuring effective i::nter-
gauge the appropriateness of proposed and es- put in place in a Midwestem suburb, it was
builds upward from careful description of the vention, whether through new ar ongoing pro-
discovered that there was a very low rate of
social problem the program is expected to ame- grams. tablished intervention programs, and assess
the effectiveness of those programs for improv- juvenile crime in the co=unity. The program
liorate.
ing social conditions. plarmers had assumed that because juvenile
Thomy issues in this domain revolve
delinquency was a serious problem nationally,
around deciding just what is meant by a need This chapter focuses on the role of evalua-
THE ROLE OF EVALUATORS IN tors in diagnosing social problems through sys- it was a problem in their co=unity as well.
in contrast, say, to a want ar desire, and what
ideals ar expectations should provide the DIAGNOSING SOCIAL CONDITIONS tematic and reproducible procedures in ways
benchmarks for distinguishing a need (cf. that can be related to the design and evaluation Plarmeis of many of the urban renewal
AND SERVICE NEEDS
McIGllip, 1998; Scriven, 1991). We will not of intervention programs. The importance of projects undertaken during the l 96Ds assumed
attempt to resolve these issues here, if indeed the resulting diagnostic information carmot be that persons living in what the plarmers re-
they can be resolved, but will be content with ln the grand scheme of things, evaluators' con- overstated. Speculation, impressionistic obser- garded as dilapidated buildings also viewed
the notion that a need is a social construction tributions to the identification and alleviation vations, political pressure, and even deliber- their housing as defective and would therefore
negotiated between a set of social agents with of social problems are modest cmnpared with ately biased information may spur policymak- support the demolition of their homes and
responsibility for social programs and policy the weightier actions of political bodies, advo- ers, plarmers, and funding organizations to accept relocation to replacement housing. In
and a set of claimants and their advocates who cacy groups, investigative reporters, and sundry initiate action, support ongoing programs, ar city after city, however, residents of urban re-
assert that a problem exists that warrants in- charismatic figures. The impetus for attending withdraw support from programs. But if sound newal areas vigorously opposed these projects.
tervention. to social problems most often comes from po- judgrnent is to be reached about such matters,
The family of procedures used by evalua- litical and moral leaders and co=unity advo- it is essential to have an adequate under- Media programs designed to encourage
tors and other social researchers to systemati- cates who have a stake, either personally ar standing of the nature and scope of the problem people to seek physical examinations to detect
cally describe and diagnose social needs is gen- professionally, in dealing with a particular con- early signs of cancer had the effect of swamping
122 EVALUATION Assessing the Need for a Program 123

health centers with more clients than they ln ali of these examples, a good needs assess-
could handle. The media effort stimulated mentwould have provided information leading
many hypochondriacal persons without cancer to a valid description of the problem that would
symptoms to believe they were experiencing have prevented programs from implementing
waming signs. inappropriate ar unneeded services. ln some
cases, unnecessary programs were designed be- The widespread use of needs assessment (NA) assessments of whole systems, the dissemination
cause the problem clid not exist. ln others, the as a systematic, rational means of determining of kits of materiais and survey instruments
ln an effort to improve the clinical iden- intervention was not effective because the tar- goals and priorities for program planning and (Witkin, 1977), the spread of NA to city planning
tification of AIDS, community physicians were evaluation in the United States dates from 1965, and the private sector, and some development of
get population clid not desire the services pro-
provided with literature about the details of with the passage of the Elementary and Second- theoretical perspectives.
vided, was incorrectly identified, OI was un- ary Education Act (ESEA, PL 89-10). ln the next With the passage oi the Omnibus Budget
cliagnosing the syndrome among high-risk pa- likely OI unable to act in the way the program
tients using blood tests. Only after the materi- 15 years over 35 titles in the 54 largest grants- Reconciliation Act oi October 1981, about 90
expected. in-aid programs in health, education, and social percent oi the legislation that induded mandates
als had been clisseminated was it recogoized Ali social programs rest on a set of assump- services required app/icants for categorical and for NA was eliminated. ln the following year, there
that few physicians take sex histories as a tions and representations of the nature of the competitive grants to document their needs. was an abrupt drop in NAs, especially in local
routine practice and thus they were unlikely to problem they address and the characteristics, Although satisfying granting agencies was not education agencies. Although applications for
know which of their patients were high risk. needs, and responses of the target population the only reason for needs assessment, the social categorical grants such as ESEA Chapter 1
Consequently, the only way they could make they intend to serve. Any evaluation of a plan legislation was a powerful stimulus to the (compensatory education) still required evidence
use of their new knowledge was by testing ali development of models and the conduct of of need, those NAs often consisted merely of
for a new program, a change in an existing
their patients. The result was an excessive program, OI the effectiveness of an ongoing exemplary studies. The period of 1966-1981 was reporting demographic data and test scores of
amount of testing, at high cost and some risk program must necessarily engage those as- characterized by the conduct of large-scale needs the students to be served.
to the patients. sumptions and representations. Of course, the
SOURCE: Quated, with permission, from Belle Ruth Wilkin, "Needs Assessment Since 1981: The Sta te of the Practice,"
problem cliagnosis and target population de- Eva/uation Practice, 1994, 15(1 ):17.
A birth contrai project was expanded to scription may already be well and convincingly
reduce the reportedly high rate of abortion in a established, in which case the evaluator can
move forward with that as a given. Or the
large urban cente~ but the program failed to
nature of the evaluation task may be stipulated
attract many adclitional participants. Sub-
in such a way that the need for the program and target problem and population that shape the ment may be virtually mandated. For example,
sequently, it was found that most of the in-
the nature of that need is not a matter for nature of a program. Where there is any ambi- the 197 4 community mental health legislation
tended urban clients were already being ade-
independent investigation. lndeed, program guity, it may be advisable for the evaluatOI to called for perioclic community mental health
quately served and a high proportion practiced
personnel and sponsors often believe they WOik with key stakeholders to forrnulate those needs assessments, and the 1987 McKinney
contraception. The high abortion rate was
know the social problems and target population assumptions explicitly so that they may serve Act inclicated that states and local communi-
caused mainly by young women who carne to
needs so well that further inquiry is a waste of as touchstones for assessing the adequacy of ties should use needs assessments as the basis
the city from rural areas to have abortions.
time. Such situations must be approached cau- the program design and theory. Often it will for planning programs for the homeless. The
tiously. As the examples above show, it is re- also be useful for the evaluatOI to conduct at role of the federal gove=ent in fostering
The problem of criminal use of hand- markably easy for a program to be based on least some minimal independem investigation needs assessment for educational programs is
guns has led to legislative proposals to forbid faulty assumptions, either through insufficient of the nature of the program's target problem described in Exhibit 4-A.
the sale of such guns to persons convicted of initial problem cliagnosis, changes in the prob- and population. For new program initiatives, OI It should be noted that needs assessment
felony offenses. Howeve~ most criminals do lem ar target population since the program was established programs whose utility has been is not always dane with reference to a specific
not purchase their guns from legitimate gun initiated, OI selective exposure ar stereotypes called into question, it may be essential to social program OI program proposal. The tech-
dealers, nOI do dealers have reliable ways of that lead to clistorted views. conduct a thorough assessment of the social niques of needs assessmen t are also used as
ascertainingwhether purchasers have criminal ln ali instances, therefore, the evaluator need and target population to be served by the planning toais and decision aids for policymalc-
recOids. should scrutinize the assumptions about the program at issue. ln other cases, aneeds assess- ers who must prioritize among competing
Assesslng lhe Need for a Program 125
124 EVALUATION

McIGllip (1987), Reviere et al. (1996), Sariano determine how many are poor when a deHni-
(1995), and Witkin and Altschuld (1995). tion is given. Nor do they establish poverty as
As the examples and co=entary above a social problem; they only characterize a situ-
indicate, needs assessment has a number of ation that individuals and social agents may
1. Jdentification of users and uses. The users of 3. Need idenUficaUon. Here problems of the facets and applications relevant to program view as problematic. Moreove~ both the defini-
the analysis are those who will act on the target population(s) and possible solutions evaluation. The next sections discuss the tion of poverty and the goals of programs to
basis of the results and the audiences who are described. Usually, more than one evaluator's role in identifying social problems, improve the lot of the poor vary over time,
may be affected by it. The involvement of source of information is used. ldentification analyzing their location and scope, defining the between co=unities, and among stakehold-
both groups will usually facilitate the analysis should include information on expectations
targets of proposed interventions, and describ- ers. Initiatives to reduce poverty, therefore, may
and implementation of its recommenda- for outcomes; on current outcomes; and on
ing the nature of the associated service needs. range from increasing employment opportuni-
tions. Knowing the uses of the need analysis the efficacy, feasibility, and utilization of so-
ties and reducing barriers to economic mobility
helps the researcher focus on the problems lutions. Social indicators, surveys, commu-
nity forums, and direct observation are fre-
to simply lowering the expectations of those
and solutions that can be entertained, but
quently used. DEFINING SOCIAL PROBLEMS persons with low incarne.
also may limit the problems and solutions
4. Need assessment Once problems and solu- Defining a social problem and specifying
identified in Step 3, below.
2. Oescription of the target popu/ation and ser- tions have been identified, this information the goals of intervention are thus ultimately
vice environment. Geographic dispersion, is integrated to produce recommendations Proposals for policy changes, new or modified political processes that do not follow simply
transportation, demographic characteristics for action. Both quantitative and qua\itative programs, or evaluation of existing programs from the inherent characteristics of the situ-
(including strengths) of the target popula- integration algorithms can be used. The generally arise out of the dissatisfaction of one ation. This circumstance is illustrated nicely,
tion, eligibility restrictions, and service ca- more explicit and open the process, the or more groups of stal<eholders with the effec- for instance, in an analysis of legislation de-
pacity are important. Social indicators are greater the likelihood that results will be tiveness of existing policies and programs or signed to reduce adolescentpregnancythatwas
often used to describe the target population accepted and implemented. realization that a new social problem is emerg- conduced by the U.S. General Accounting Of-
either directly or by projection. Resource 5. Communication. Finally, the results of the
ing. Either case assumes that a social problem fice (GAO, 1986). The GAO found that none of
inventaries detailing services available can need analysis must be communicated to
has been identified, a matter that is not as the pending legislative proposals defined the
identify gaps in services and complementary decisionmakers, users, and other relevant
audiences. The effortthat goes into this com- straightforward as it may seem. Indeed, the problemas involving the fathers of the children
and competing programs. Comparison of
munication should equal that given the other question of what defines a social problem has in question; every one addressed adolescent
those who use services with the target popu-
steps of the need analysis. occupied spiritual leaders, philosophers, and pregnancy as an issue of young mothers. Al-
lation can reveal unmet needs ar barriers to
solution implementation.
social scientists for centuries. For our pmposes, though this view of the problem of adolescent
the key point is that social problems are not pregnancy may lead to effective programs,
SOURCE: Adapted from Jack McKillip, "Need Analysis: Process and Techniques," in Handbook o( Applied Social themselves objective phenomena. Rathe~ they clearly there are altemative definitions that
Research Melhods, eds. L Bickman and D. J. Rog (Thousand Oaks, CA: Sage, 1998), pp. 261-284. are social constructions that emerge from the include the adolescent fathers.
interests of the parties involved as they relate Indeed, the social definition of a problem
to observed conditions. ln this sense, co=u- is so central to the political response that the
nity members, together with the stakeholders preamble to proposed legislation usually shows
purpose from the assessment of the need for a involved in a particular issue, literally create some effort to specify the conditions for which
needs and claims. For instance, a regional
particular program, whether existing or pro- the social reality that constitutes a recogillzed the proposal is designed as a remedy. For exam-
United Way or a metropolitan city council
posed, the methods for these broader needs social problem (Miller and Holstein, 1993; ple, two contending legislative proposals may
might commission a needs assessment to help
assessments are much the sarne, and they also Spector and IGtsuse, 1977). both be addressed to the issue of homeless
them determine how funds should be allocated
are generally conducted by evaluation re- It is generally agreed, for example, that persons, but one may identify the homeless as
across various service areas. Or a state depart-
searchers. Exhibit 4-B provides an overview of poverty is a social problem. The observable needy persons who have no kin on whom to be
ment of mental health might assess co=u-
the general steps in a needs assessment. Useful facts are the statistics on the distribution of dependent, whereas the other defines home-
nity needs for different mental health services
book-length discussion of needs assessment incarne and assets. Howeve~ those statistics do lessness as the Jack of access to conventional
to distribute resources optimally among its ser-
applications and techniques can be found in not define poverty; they merely pennit one to shelter The fust definition centers attention
vice units. Although different in scope and
126 EVALUAT!ON Assessing the Need for a Program 127

primarily on the social isolation of potential stakeholders about their perceptions and as- a skeptic that child abuse exists. But specifying ever information already exists on a given so-
clients; the second focuses on housing arrange- sumptions about the nature of the social con- the size of the problem and where it is located cial problem. Here we stress both collate and
ments. The ameliorative actions that are justi- ditions the program addresses. This, then, pro- geographically and socially requires detailed assess-unevaluated information can be as bad
fied in terms of these definitions will likely be vides the evaluator with some basis for knowledge about the population of abused chil- as no information at all.
different as well. The first definition, for in- analyzing the structure and goals of the pro- dren, the characteristics of the perpetrators, For some social issues, existing data
stance, would support programs that attempt gram and assessing its design. and the distribution of the problem throughout sources may be of sufficient quality to be used
to reconcile homeless persons with alienated Also, an important role evaluators may the political jurisdiction in question. For a prob- with confidence. For example, accurate and
relatives; the second, subsidized housing pro- play at this stage is to provide policymakers and lem like child abuse, which is not generally trustworthy information can usually be ob-
grams. program managers with a critique of the prob- public behavio~ this can be difficult. Such so- tained about issues on which measures are
It is usually informative, therefore, for an lem definition inherent in their policies and cial problems are mostly "invisible," so that routinely collected either by the Current Popu-
evaluator to determine what the problem a programs and propose altemative definitions only imprecise estimates of their rates ofoccur- lation Survey or the decennial U.S. Census.
program addresses is thought to be in its par- that may be more serviceable. For example, rence are possible. ln such cases, it is often Moreove~ through the census tract coding, that
ticular political context. To investigate this, the evaluators could point out that a definition of necessary to use data from several sources and information can be disaggregated to state and
evaluator might, for instance, study the im- the problem of teenage pregnancies as primar- use different approaches to estimating rates of local leveis. AI; an illustration, Exhibit 4-D
plicit or explicit definitions that appear in pol- ily one of illegitimate births ignores the large occurrence (e.g., Ards, 1989). describes the use of vital statistics records and
icy and program proposals. Revealing informa- number of births that occur to married teenag- It is also generally important to have at census data to assess the nature and magnitude
tion may also be found in legislative ers and suggest program implications that fol- least reasonably representative samples to esti- of the problem of poor birth outcomes in a
proceedings, including committee hearings low from that definition. mate rates of occurrence. It can be especially Florida county. This needs assessment was
and floor debates, journals of opinion, news- misleading to draw estimates from at-risk aimed at estimating child and maternal health
paper and magazine editorials, and other populations, such as those found in service needs so that appropriate services could be
sources in which discussions of the problem programs, when general population estimates planned. Even when such direct information
apperu: The operative definition of the problem SPECIFYING THE EXTENT OF are needed to determine the extent of a prob- about the problem of interest is not available
a particular program addresses can usually be THE PROBLEM: WHEN, WHERE, lem. Estimation of the rate of spousal abuse from existing records, indirect estimates may
found in program documents, newspaper ac- during pregnancy based on women in shelters, be possible if the empirical relationships be-
AND HOWBIG?
counts of its launch, proposals for funding it, for instance, results in considerable overesti- tween available information and problem indi-
and the like. Such materials may explicitly mation of the frequency of occurrence in the cators are known (e.g., Ciarlo et al., 1992).
describe the nature of the problem and the The design and funding of a social prograrn general population of pregnant women. An es- ln addition to the decennial census, data
program's plan of attack, as in funding propos- sbould be geared to the size, distribution, and timate from a more representative sample still available in many of the statistical series rou-
als, or implicitly define the problem through density of the problem it addresses. ln assess- indicates that battering of pregnant women is tinely collected by federal agencies are often
the assumptions that underlie statements ing, say, emergency shelters for homeless per- a serious problem, but places the extent of the trustworthy. There are, unfortunately, excep-
about program activities, successes, and plans. sons in a co=unity, it makes a very signifi- problem in a realistic perspective (see Exhibit tions. For example, it is widely acknowledged
This inquiry will almost certainly tum up cant difference whether the total homeless 4-C). that the U.S. Census undercounts thenumbers
information that will be useful for a prelimi- population is 350 or 3,500. It also makes a big of African Americans and Hispanics and, to a
nary description of the social need to which the difference whether the problem is located pri- considerable extent, the number of homeless.
program is presumably designed to respond. AI; marily in poor neighborhoods or affluent ones Using Existing Data Sources For the nation as a whole, these undercounts
such, it can guide a more probing needs assess- and how many of the homeless suffer from to Develop Estimates are relatively small and for many purposes can
ment, both with regard to how the problem is mental illoess, chronic alcoholism, and physi- be ignored. For jurisdictions with large popula-
defined and what altemative perspectives cal disabilities. Through their knowledge of existing re- tions of African Americans and Hispanics,
might be applicable. If the evaluation circum- It is much easier to establish that a prob- search and data sources and their under- howeve~ these undercounts may result in sig-
stances do not permit further systematic inves- lem exists than to develop valid estimates of its standing of which designs and methods lead to nificant misestimation of the size of certain
tigation, this information can nonetheless be density and distribution. Identifying a handful conclusive results, evaluation researchers are target populations relevant to assessing the
the basis for a thoroughgoing discussion with of battered children may be enough to convince in a good position to collate and assess what- need for social programs.
128 EVALUATION Assessing the Need for a Program 129

Ali women are at risk of battering; however, lation, McFarlane and associates randomly The Healthy Start lnitiative in Florida, a series
Low birth rate babies. There was a higher
of legislative measures intended to improve
pregnancy places a woman at increased risk for sampled and interviewed 290 healthy pregnant incidence for ado]escents and women over
severe injury and adverse health consequences, women from public and private clinics in a large
pregnancy and birth outcomes within the state age 35.
both for herself and her unborn infant. Local and metropolilan area with a populalion exceeding
provides for the establishment of community~
Very /ow birth weightbirths. The overall rate
based prenatal and infant health care coalitions
exploratory studies have found as many as three mlllion. The 290 black, white, and Latina was twice that for the whole state and ex-
composed of health care providers, represen~
40/a-60/o of battered women to have been women ranged in age from 18 to 43 years; most ceeded state goals for both African Ameri-
ta tives of state and local government, community can and white mothers.
abused during pregnancy. Among 542 women were married, and 80/o were al least fve months
alliances, maternal and child health organiza-
in a Dallas shelter, for example, 42% had been pregnant. Nine questions relating to abuse were Adolescent pregnancy. The proportion of
tions, and consumers of family planning, prenatal
battered when pregnant. Most of the women asked of the women, for example, whether they births to leens was over Wice the state
care, and primary care services. Each coalition is
reported that the vio\ence became more acute were in a relationship with a male partner who average; the rate for African American leens
required to conduct a needs assessment within
during the pregnancy and the child's infancy. ln had hit, slapped, kicked or otherwise physically was more than twice lhat for white teens of
its service delivery area and develop a service the sarne age.
another study, inlerviews of 270 battered women hurl them during the current pregnancy and, if delivery plan. The needs assessment of the
across the United States found that 44% had yes, had the abuse increased. Of the 290 women, Age of mother. The infant mortality and Jow
Gadsden Citizens for Healthy Babies, lnc., birth rates were highest among children bom
been abused during pregnancy. 8/o reported battering during the currenl representing a small, primarily rural, majority to mothers 16-18 years ofage.
But most reports on battering_ during preg- pregnancy (one out of every welve women African American county in north Florida, used
nancy have been secured from samples of interviewed). An additional 15% reported bat- Education of mother. Mothers with less than
existing data from the State of Florida vital sta-
high school education were slightly more
battered women, usually women in shelters. To tering before the current pregnancy. The fre- tistics records and the U.S. Census of Population
likely to have low birth weight newborns but
establish the prevalence of battering during quency of battering did not vary as a function of and Housing to estimate the magnitude and almost eight times more likely to have new-
pregnancy in a representative obslelric popu- demographic variables. distribution of child and maternal health borns identifed as high risk on infant screen-
problems in the county. ing measures.
SOURCE: Adapted from J. McFarlane, "Battering During Pregnancy: Tip of an Iceberg Revealed/' Women and Heaft.h, First, pregnancy outcomes and related
Based on these findings, three groups were
1989, 15(3):69-84. maternal characteristics within the county were
identified with high risk for poor birth outcomes:
investigated using data from the Florida Vital
Statistics volumes, which repor! birth and death Mothers less than 19 years of age
information collected annually within the state. ln Mothers with less than a high school
particular, the following indicators were education
examined:
African American mothers
Because many federal programs are tied to dents. These plans are quite controversial, lnfant mortality. The county's rate was far U.S. Census data on CD-ROM discs available
the size of particular populations, such under- however, and Congress is considering legisla- higher than national ar state rates. from the Bureau of Census were then used to
counting can also translate into substantial tion that would instruct the Bureau of the Fetal mortality. The overall rate for the identify the number of women of childbearing
lasses of federal funds for those jurisdictions. Census to conduct a "complete census" and county was higher than the state goal and age in each of these risk categories, the
This circumstance has led to a stream of law- abandon any plans to adjust census retums on the rate for African American mothers was proportions who were in various low-income
suits by cities and states, and to advocacy of the basis of sampling. higher than for white mothers. strata, and their geographical concentrations
statistical adjustments of the undercount. Al- As this example illustrates, it is often dilli- Neonatal morta/ity. The rates were higher within the county according to census tract and
than the state goal for white mothers but zip code. This information was used by the
though many statisticians regard such adjust- cult to separate technical decisions from politi- below for African American mothers. coalition to identify the major problem areas in
ments as a sound approach, adjustments have cal interests in diagnosing social problems. Ad- Postneonatal mortality. The rates were the county, set goals, and plan services.
been rejected in the 1980 and 1990 censuses. justing for the undercount of African American below state goals.
ln planning for the 2000 census, the Bureau of and, particularly, Hispanic persons would in-
the Census has formulated a plan for statistical crease the amount of resources allocated to
SOURCE: Adapled from E.. Wa~!e.r Terrie, "Assessing Child and Maternal Health: The First Step ln the Design of
adjustments based on sampling nonrespon- them under many federal programs. At the Co~munny-Based. lntervenllons, rn Needs Assessment: A CreaYe and Praccaf Cuide for Social Scientists, eds. R.
Revrere, 5. Berkowrtz, C C Carler, and C G. Ferguson {Washington, DC: Taylor & Francis, 1996), pp. 121-146.
130 EVALUATION
Assessing the Need for a Program 131
same time, howeve~ it would modify the dis- pie, the Current Population Survey of the Bu-
tribution of congressional seats from state to reau of the Census collects data annually on
state and, within states, result in the need to the chatacteristics of the U.S. population using
redraw electoral boundaries. Thus, although a latge household sample. The data include
social reseatchers advise assessing the quality measures of the composition of households, Since 1973 the Bureau of Justice Statistics in to the interview, whether reported to the police
of data in terms of their measurement proper- individual and household incarne, and house- the Department of Justice has conducted an ar not The table below charts the ten-year trends
ties, the poltica! irnplications of estimates of hold members' age, sex, and race. The regular annual survey oi a national sample oi households in crimes with persons or households as their
social problems and the distribution of the Survey of Incarne and Program Patticipation that asks if each person in the household has victims.
affected population may play a latge role in the provides data on the extent to which the U.S. been the victim of a crime during the year prior
procedures used to collect the data. population patticipates in various social pro-
When sources ate used whose validity is gratnS: unemployment benefits, Aid to Fami- Victimizatian Rates per 1,DDD Persans Age 12 and O/der or per 1,DDD Househalds
not as widely recognized as that of the census, lies With Dependent Children, food stamps, 1987 1988 1989 1990 7997 1992 1993 1994 1995 7996
it is always necessary to examine catefully how job training programs, and so on. The National Crimes of violenceil 43.7 44.1 44.4 44.1 48.0 49.3 49.9 51.8 46.6 42.0
Property crimesb 298.4 295.3 295.3
the data were collected. A good rule of thumb Crime Survey compiles annual data on crime 276.5 282.3 325.3 318.9 310.2 290.5 266.3
is to anticipate that, on any issue, different data victimization from a national survey of house- NOTE: lhe 1987-1992 figures incorporate a statistical adjustment for a change in survey methodology.
sources will provide disparate ar even contra- holds (see Exhibit 4-E). a. Rape, robbery, assault
b. Burglary, theft, auto thefL
dictory estimates. For needs assessment pur- These regulatly occurring measures, called
poses, sometimes data on the same topic col- social indicators, can provide irnportant infor-
lected by opposing stakeholders can be mation for assessing social problems and needs SOURCE: ~.S. Oepartment of Justi~e, Bureau of Justice Statlslics, Criminal Victimizalion in the United States: 197392 Trends (Wush
lngton, DC. U.S. Oepartmenl of Justice, July 1994). U.S. Department of Justice Bureau of Justice Statistfcs Criminal Victimizaton
especially useful. For example, both the Coali- in severa! ways. First, when properly analyzed, 1996 (Washington, OC: U.S. Department o Justice, November 1997). ' '

tion Against Handguns and the National Rifle the data can often be used to estimate the size
Association (NRA) have sponsored sample sur- and distribution of the social problem whose
veys of the U.S. population conceming ap- course is being tracked over time. Second, the
proval ar disapproval of gun control legislation. trends shown can be used to alert decisionmak- by the Current Population Survey ate without Unfortunately, the social indicators cur-
Although their reports differed widely in their ers to whether certain social conditions are controversy; many believe that theyunderesti- rently available ate limited in their coverage of
conclusions-the one finding popular support irnproving, remaining the same, ar deteriorat- mate the current levei of poverty, whereas oth- social problems, focusing mainly on issues of
for gun contrai measures and the other the ing. Finally, the social indicator trends can be ers believe the opposite (Ruggles, 1990). poverty and employment, criminal victirniza-
opposite-close inspection of the data showed used to provide a first, if crude, estimate of the Considerable effort is currently going into tion, national program patticipation, and
that many of the specilic findings were neatly effects of social programs that have been in the collection of social indicator data on poor household composition. For many social prob-
identical in the two surveys (Wright, Rossi, and place. For example, the Survey of Incarne and households in an effort to judge whether their lems, no social indicators exist ar those that do
Daly, 1983). Bothsurveys found thatguns were Program Patticipation can be used to estimate circumstances have worsened ar irnproved af- support analysis of national trends but cannot
owned by about half of U.S. households, for the coverage of such national programs as food ter the radical reforms in welfate enacted in the be broken down to provide useful indicators of
instance. Findings on which different surveys stamps or job training. Personal Responsibility and Work Opportunity local trends.
substantially agree can be regatded as having Sirnilatly, the proportions of U.S. house- Reconciliation Act of 1996. Special surveys,
greater credibility. holds at ar below the poverty levei can be concentrating on the well-being of children, ate
followed from yeat to yeat over the post-World being conducted by the Urban Institute and the
Estimating Problem Parameters
Using Social Indicators Wat II yeats through data obtained by the Cur- Manpower Development Reseatch Corpora- Through Social Research
to Identify TJ:ends rent Population Survey. The question whether tion. ln addition, the Bureau of the Census has ln many instances, no existing data source
there was more or less poverty in the 1980s extended the Survey of Incarne and Program will provide estimates of the extent and distri-
On some topics, existing data sources will than in the preceding decades can thus be Patticipation to constitute a panei of house- bution of a problem of interest. For example,
provide periodic measures that can be used to answered by referring to this social indicato~ holds repeatedly interviewed before and after there ate no ready sources of information about
chart historical trends in the society. For exam- This is not to say that the trend data provided the welfate reforms were instituted. household pesticide misuse that would indi-
132 EVALUATION Assessing the Need for a Program 133

cate whether it is a problem, say, in households clients, but others do not keep records of high
with children. ln other instances, good infor- quality or do not keep records at ali. When an
mation about a problem may be available for a agency's clients include ali the persons mani-
national or regional sample that cannot be festing the problem in question and records are Most sample surveys are based on the encountered in the block searches and found not
disaggregated to a relevant local levei. The Na- faithfully kept, then the evaluator need not assumption that ali persons can be enumerated
to rent, own, ar be a member of a household
tional Survey of Household Drug Use, for in- search any furthet Unfortunately, these condi- and surveyed in their dwellings, an assumption
renting or owning a conventional dwel/ing unit.
stance, uses a nationally representative sample tions do not occur often. that fails by definition in any study of the Conventional dwelling units included apart-
to track the nature and extent of substance Itwould be temptingto tryto estimate, say, homeless. The strategy devised for the Chicago ments, houses, rooms in hotels or other struc-
study therefore departed from the traditional tures, and mobile homes.
abuse. Howeve~ the number of respondents the extent of drug abuse by extrapolating from
survey in that persons were sampled from
from most states is not large enough to provide the records of persons treated in drug abuse ln the street surveys, teams of interviewers
non-dwelling units and interviews were con-
good state-level estimates of drug abuse, and no clinics. To the extent that the drug-using com- acconipanied by off-duty Chicago policemen:
ducted at times when the separation between the searched ali places on each sampled biock to
valid city-level estimates can be derived at ali. munity is fully covered by existing clinics, such homed and homeless was at a maximum. Two
When pertinent data are nonexistent or estirnates may be quite accurate. Howeve~ if which they could obtain access, including ai~
complementary samples were taken: (1) a prob- night businesses, alleys, hallways, roofs and
insufficient, the evaluator must consider col- drug abuse clinics did cover ali or most of the ability sample of persons spending the night in basements, abandoned buildings, and parked
lecting new data. There are severa! ways of drug-abusing population, drug abuse treatrnent shelters provided for homeless persons, and (2) cars and trucks. AH persons encountered in the
malcing estimates of the extent and distribu- programs might not be problematic. Hence, to a complete enumeration ofpersons encountered street searches were awakened if necessary and
tion of social problems, ranging in increasing the extent that a problem is being adequately between the hours of midnight and 6 a.m. in a interviewed to determine whether or not they
degrees of effort from relying on "expert" testi- handled by existing programs, data from such thorough search of non-dwelling-unit places in a were homeless. ln lhe shelter samples, ali persons
mony to conducting large-scale sample sur- prograrns may be useful and accurate, but that probability sample of Chicago census blocks. spending the night in such places were assumed
veys. Decisions about the kind of research ef- is not the situation in which data are usually Taken together, the shelter and street surveys to be homeless. Once identified, homeless
constitute an unbiased sample of the homeless
fort to undertake must be based in part on the needed. ln the case of drug abuse clinics, of persons were interviewed to obtain data on their
of Chicago.
funds available and in part on how important course, it is doubtful that ali drug abusers are employment and residence histories as well as
A person was classified as homeless at the their sociodemographic characteristics. Ali co-
it is to have precise estimates. li, for legislative in fact served by the clinics. (The different time of the survey if that person ~as a resident
or program design purposes, it is criticai to prevalence estimates obtained from a served operating respondents were paid $5.00.
of a shelter for homeless persons or was
lmow the number of malnourished infants in population and a sarnple survey of the general
a poli ti cal jurisdiction, a carefully planned population are illustrated in the example of 50.URCE: Adapted from P. H. Rossi, Oown and Out in America: The Origins o{ Hamelessness (Chicago University of
health interview survey may be necessary. ln battered pregnant women in Exhibit 4-C.J Chicago Press1 1989). '
contrast, if the need is simply to determine
whether there is any malnutrition among in- Surveys and Censuses
fants, input from knowledgeable informants
may be ali that is required. This section de- When it is necessary to get very accurate
scribes the various procedures that can be used information on the extent and distribution of to estimate the size and composition of the serious medical conditions existed among the
to determine the size of a social problem and a problem and there are no existing credible homeless population of Chicago. The survey homeless populations in urban centers, there
its geographical and social distribution. data, the evaluator may need to undertake covered both persons in emergency shelters and was virtually no precise, reliable information
original research using sample surveys or com- homeless persons who did not use shelters. on either the size of the homeless population
Agency Records plete enumerations. Because they come in a Surveying the latter involved searching Chi- or the extent of medical problems in that popu-
variety of sizes and degrees of technical com- cago streets in the middle of the night. The lation. Hence, the foundations funded a re-
Records of organizations that provide ser- plexity, either of these techniques can involve survey was undertalcen because the Robert search proj ect to collect the missing informa-
vices to the population in question are infor- considerable effort and skill, not to mention a Wood Johnson Foundation and the Pew Memo- tion. Although many regard the effort as less
mation sources that may be useful for estimat- substantial commitrnent of resources. rial Tiust were planning a program for increas- than satisfactory, this research stimulated fur-
ingthe extent of a socialproblem (Hatry, 1994). To illustrate one extreme, Exhibit 4-F de- ing the access of homeless persons to medical ther efforts to count the homeless and improve
Some agencies keep excellent records on their scribes a needs assessment survey undertalcen care. Although there was arnple evidence that data collection procedures. For example, the

L
134 EVALUATION Assessing the Need for a Program 135

1,100 residents of Los Angeles County to as- Conside~ for example, the task of estimat-
certain the extent of public knowledge concern- ing the number of homeless persons in a com-
ing the effectiveness of clifferent AIDS preven- munity. Although well-placed key informants
tion behaviors. For mass media educational may have experience with some subset of that
To gauge the extent of knowledge about how assurances of safety. Condom use, despite programs aimed at increasing awareness of population, it will be difficult for them to ex-
to avoid HIV infection, a sample of Los Angeles reported problems with breakage, leakage, and ways to prevent AIDS, a survey such as this trapolate from that experience to an estimate
County residents was interviewed on the tele- misuse, was rated as very effective by 42/o of the
identifies both the extent and the nature of the of the size of the total population. lndeed, it can
phone. They were asked to rate the effectiveness respondents and as somewhat effective by
another 50/o. Respondents were much less cer-
gaps in public knowledge. be shown that selected informants' guesses
of four methods that "some people use to avoid
tain about the effectiveness of spermicidal Many survey organizations have the capa- about the numbers of homeless in their locali-
getting AIDS through sexual activity" (see table).
Their highest rating was for monogamous sex agents, regardless of whether they were used in bility to plan, carry out, and analyze sample ties vary widely and tend to be overestimates,
between HIV-negative people, although 12% fel! conjunction with an alterna tive method. surveys for needs assessment. ln addition, it is sometimes quite large overestimates (see Ex-
that even in these circumstances there were no often possible to add questions to regularly hibit 4-HJ.
conducted studies in which a number of orga- On the grounds that key informants' re-
Percentage Distribulion o( Rangs o( the Effecliveness of Different Prevention Methods
nizations buy "time, /1 thereby reducing costs. ports of the extent of a problem are better than
Very Somewhat Not at Ali
Whatever the approach, it must be recognized no information at all, evaluators may wish to
Prevention Melhod Effective Effective EffecrJve Don't Know that designing and implementing sample sur- conduct a key informant surveywhen no other
Monogamous sex between HIV-negative individuais 73 14 12 veys can be a complicated endeavor requiring research is possible ar when available funds are
Using a condom alone 42 50 7 quite specific skills. For discussion of the vari- insufficient to support a better approach. Given
Using a diaphragm wilh spermicide 9 35 50 6
ous aspects of sample survey methodology, see those circumstances, evaluators must talce care
Using spermicide alone 7 32 53 6
Fowler (1993), Henry (1990), Rossi, Wright, to ensure that the key informant survey is of
SOURCE: Adapted from D. E. Kanouse et ai., AJDS-Related Knowledge, Attitudes, Befiefs, and Behaviors in Los Angeles and Anderson (1983), and Sudman and Brad- the highest quality. The researchers should
County R-4054-LACH (Santa Manica, CA: RAND, 1991 ). burn (1982.J. choose the persons to be surveyed very care-
fully, attempting to ensure that they have the
Key Informant Surveys necessary expertise, that they are questioned in
a careful manne~ and that any qualilications
1990 census gave special attention to counting mates of 66% in shelter beds and 14% sleeping Perhaps the easiest approach to obtaining they may have about their reports are obtained
the homeless. One learning experience from on the streets. Given the costs of shelter care estimates of the extent of a social problem is to (Averch, 1994).
subsequent research is the need to take into and the hostility of residents to persons sleep- askkey informants, those persons whose posi-
account the clifferences in the ways the home- ing on the streets in their neighborhoods, accu- tion ar experience should give them some per-
less spend their time from co=unity to com- rate estimates were worth obtaining for both spective on the magnitude and distribution of
munity and the extent to which they are found program planning and political reasons. Forecasting Needs
the problem. Unfortunately, such reports are
in shelters, indoor settings that provide meals, Usually, howeve~ needs assessment re- generally not especially accurate. Although key Both in formulating policies and programs
on the streets, and in other outside areas. search is not as elaborate as that described in informants can often provide very useful infor- and evaluating them, it is often important to
Although time-consuming and costly, such Exhibit 4-F. ln many cases, conventional sam- mation about the characteristics of certain tar- be able to estimate what the magnimde of a
extensive efforts are sometimes requiied for ple surveys can provide adequate information. get populations and the nature of service needs, social problem is likely to be in the future. A
diagnostic purposes. To illustrate, Burnam and If, for example, reliable information is required as we will discuss !ater in this chapte~ few are problem that is serious now may become more
Koegel (198 8) made a strenuous effort to obtain about the number and distribution of children likely to have a vantage point ar information ar less serious in !ater years, and program plan-
a representative sample and found that 44% of needing child care so that new facilities can be sources that permit very good estimation of the ning must attempt to tal<e such trends into
Los Angeles' homeless spent the night before planned, it will usually be feasible to obtain it number of persons affected by a social condi- account. Yet the forecasting of future trends can
being interviewed in a shelter bed and 26% from sample surveys conducted on the tele- -s..I'
tion ar the demographic and geographical dis- be quite risky, all the more soas the time hori-
slept on the street. An earlier study, with a
poorer quality sample, had resulted in esti-
phone. To illustrate, Exhibit 4-G describes a
telephone survey conducted with more than
~- tribution of those persons. zon lengthens.
Assesslng the Need for a Program 137
J 36 EVALUATION
are technical activities that require specialized compassed the large number of individuais who
lmowledge and procedures. lived in single rooms, usually paying for them
daily ar weeldy and without the protection of
!eases ar other contractual arrangements. (For
11 researchers asked eight service providers in the
To ascertain how c\ose 11 expert estimates of further cliscussion of the technical and policy
Skid Row area-shelter operators, social agency
the number of homeless persons in downtown
officials, and the like-to estimate the total DEFINING AND IDENTIFYING THE issues in homeless research, see Can; 1991.)
Los Angeles carne to actual counts of home\ess As we will cliscuss shortly, the targets of an
persons "on the streets,
11
in shelters, or in
homeless population in that 50-block area. The TARGETS OF INTERVENTIONS
estimates abtained were as follows: interve.ntion need not be persons ar groups of
single-room occupancy (SRO) hotels, a team of
persons: They can also be organizations ar
Provider 5: 10,000
Correctly defining and identifying the targets "conditions." Here the sarne point about shifts
Provider 1: 6,000 to 10,000 for intervention is crucial to the success of in target definition applies. The target of a
Provider 6: 2,000 to 15,000
Provider 2: 200,000 social programs from the very early stage when social intervention might be defective housing,
Provider 7: 8,000 to 10,000
Provider 3: 30,000 stalceholders begin to converge in their defini- which, at fust, might be defined in terms of
Provider 8: 25,000
Provider 4: 10,000
tion of a social problem to the extended period serious violations of the builcling code and !ater
number, based an shelter, SRO, and street over which the program is operated. Specifying come to also include, for instance, builclings
Clearly, the estimates were all aver the map. those targets is complicated by the fact that the that are not earthquake proof.
Two providers (4 and S) carne fairly dose to ':hat counts.
definition and corresponding estimates of the Although the definition and estimated
the researchers estimated as the most lrkely
size of the population may shift over this pe- number of targets for a program are always in
[ Th Ch f, ce o( Misery Los Angeles' Skid Row riod. AB a new social problem emerges ar be- flux to some degree, they are critically impor-
SOURCE: Adapted from Hamilton, ~abino~itz, aNnd ~lsc~~er, :~'atye Eas~~~~~gA~geles: Com~unity Redevelopment
Area in Transition-Housing and Social Serwces ee s o en comes increasinglyvisible, one definition ofthe tant at two points in time. First, in the prede-
Agency, July 1987). targets of an intervention may be adopted; as sign phase of laying out policy and program
stalceholders plan and eventually implement a altematives, it is important to clearly identify
program initiative, howeve~ that definition the intended targets. Obviously, this is neces-
may well be modified ar abandoned. sary so that the size and character of the social
We are not arguing against the use of fore-
There are a number of technical and prac- AB an illustration, during the early 1980s problem as a basis for realistic planning can be
casts in a needs assessment. Rathe~ we wish to
tical difficulties in forecasting that derive in the problem of homelessness became ex- estimated. ln addition, how the targets are
warn against accepting forecasts uncritically
part from the necessary assumption that the tremely salient. Initially, the homeless were defined will affect the potential effectiveness of
without a thorough examination of how they
future wiJl be much like the past. For example, identified as those individuais who lived in a policy ar program because that definition will
were produced. Moreove~ such criticai exami- streets and alleyways ar in shacks they con- shape the program emphasis and approach. For
at fust blush a projection of the number of
nation may itself involve some dilliculty. For
persons in the population aged 18 to 30 a structed for themselves. As advocates of the these reasons, new legislation and program
simple extrapolations of existing trends, the
decade from now seems easy to construct from homeless became increasingly active, howeve~ proposals orclinarily contain statements about
asswnptions on which a forecast is based may
current data-it is almost completely deter- the targets of interventions began to also in- who ar what the targets are to be.
be relatively few and easily ascertained. Even if
nlined by the present age structure of the popu- clude persons who spe.nt periods of time sleep- Second, the issue of target definition is
the asswnptions are lmown, howeve~ it may
lation. Howeve~ had demographers made fore- ing in shelters (sensibly so, because many per- criticai during the course of designing the spe-
not be easy to determine whether they are cific content and structure of a program. Al-
casts ten years ago for central Africa, they sons who sleep in shelters also sleep out on the
reasonable. For sophisticated projections such though targets may be initially defined in leg-
would have been substantially off the mark streets, and vice versa). Then, as programs
as those developed from multiple-equation,
because of the unanticipated and tragic impact began to emerge, some of them took the view islation ar program proposals, in practice,
computer-based models, examjnjng the as-
of the AIDS epidemie, which is most prevalent that persons who had no regular place to live programs often have to limit the targets to
swnptions may require the skills of an ad-
among young adults. Projections with longer but moved in for brief periods with various which they actually direct an intervention to
vanced programmer and the expertise of an some portion of those defined targets. This
time horizons would be even more problematic relatives, frie.nds, and sometimes strangers
experienced statistician. ln any event, it must
because they would have to talre into account should be included. For some stakeholders and often happens simply because the funds pro-
be recognized that all but the simplest forecasts
trends in fertility as well as mortality. programs, the homeless population also en- vided are insufficie.nt to cover all the originally
Assessing the Need for a Program 139
138 EVALUATION
if tl1ere is a strong relationship between hous- 4-I, which is extracted from a landmark article
intended targets ar because the nature of tbe organizational-level target for an educational ing quality and healtb, investing in tbe physical tbat greatly influenced ilie development of ilie
prograrn is such tbat it cannot be appropriately intervention might be elementary schools (kin- improvement of housing to indirectly promote "poverty line" concept, a definition of poverty
provided to ali tbose in tbe initially defined dergarten to eightb grade) witb at least 300 householders' well-being may be justified, but iliat is still employed today, adjusted to ilie
pupils in which at least 30% of tbe pupils if tbe correlation is low ar zero, tl1e investrnent current value of ilie dollai: Note iliat Orshan-
target population.
Sometimes the reverse is true and, in op- qualify for tbe federal free lunch prograrn. is likely to be ineffective for tbat purpose. sky's (1969) article dealt boili wiili technical
eration, the prograrn is directed at a broader ar issues, such as ilie availability of appropriate
larger group tban originally planned. For exam- Direct and Indirect Targets data, and wiili substantive issues, such as try-
ple, healtb clinics set up in shelters for tbe ing to arrive at a definition iliat would satisfy
Targets may also be regarded as direct ar . Specifying Targets
homeless were initially intended to provide stakeho!ders and social scientists. It is a tribute
services for persons sleeping tbere and in tbe indirect depending on whetber services are de- At first glance, specification of tbe size and to her skill in balancing iliose objectives iliat
streets. Howeve~ when poor persons housed in livered to tbem immediately (directly) ar even- distribution of target populations may seem ilie poverty-level concept she developed is still
the sarne neighborhoods learoed of tbe clinics, tually (indirectly). Most prograrns specify direct simple. Altbough target definitions may be in use more ilian two decades !atei: The Or-
some of tbem began to use tbem too and were targets. This is clearly tbe case in medical easy to write, howeve~ it is often difficult to shansky measures have been heavily criticized,
interventions, for example, where persons with employ such definitions in tbe more precise howeve~ and severa! alternatives have been
accepted as patients.
a given affliction directly receive medical treat- work of needs assessment and prograrn design. proposed (Ruggles, 1990).
ments. Howeveif in some cases, either for eco- There are few human and social problems tbat Benchmarlcs for ilie size of major problem
nomic ar feasibility reasons, planners may de- can be easily and convincingly described in populations, such as ilie poo~ and tbe defini-
What Is a Target? sign prograrns to affect a target population terms of simple, unarnbiguous characteristics tion underlying ilieir identification have irn-
The targets of social prograrns are usually indirectly by acting on an intermediary popu- of tbe individuais experiencing tbat problem. portant consequences for ilie way governmen-
individuais. But tbey also may be groups (fami- lation ar condition tbat will, in tum, have an Take a single illustration: What is tbe popu- tal and private resources are allocated. For
lies, work tearns, organizations), geographi- impact on tbe intended target population. In a lation of persons wiili cancer in a given com- example, redera! officials use ilie poverty-level
cally and politically related areas (such as com- rural development project, for example, influ- munityl The answer depends, first, 011 wheilier approach just discussed in determining how
munities), ar physical units (houses, road ential farmers were selected from small com- one counts only permanent residents ar in- much to appropriate for food starnp, medical
systems, factories). Whatever tbe target, it is munities for intensive training prograrns. The cludes temporary ones as well (a decision iliat care, and housing assistance prograrns.
imperative at tbe outset of a need~ assessment intention was tbat afterward tbeywould return would be especially irnportant in any co=u-
to define the units in question clearly. to tbeir co=unities and co=unicate tbeir nitywiili a large number of vacationers such as Target Boundaries
In the case of individuais, targets are usu- new knowledge to atber farmers, tbe indirect Orlando, Florida). Second, are "recovered"
ally identified in terms of social and demo- targets of tbe prograrn. Sinlarly, a project that cases to be counted, ar are iliose wiiliout a Adequate target specification establishes
graphic characteristics, location, ar tbeir prob- identifies substandard dwelling units as its di- relapse fo~ say, five years to be eliminated from boundaries, tbat is, rules determining who ar
lems, difficulties, and conditions. Thus, targets rect targets may be intended to benefit (indi- ilie estimatel Third, is having cancer to be what is included and excluded when ilie speci-
of an educational prograrn may be designated rectly) tbe current and future occupants of defined only as diagnosed cases ar does it also fication is applied. One risk in specifying target
as clilldren aged 10 to 14 who are between one tbose dwellings. include iliose persons whose cancer had not yet populations is to make a definition too broad
and tbree years below tbeir normal grade in When targets are defined as indirect, tbe been detectedl Finally, ilie estimate must take ar overinclusive. For example, specifying tliat a
school. Or targets of a maternal and infant care prograrn's elfectiveness depends to a large ex- into account ilie purpose for which it is being crinlinal is anyone who has violated any law ar
prograrn may be pregnant women and motbers tent on whetber tbe patbways leading from used. If it is to be used in designing a special administrative regulation is useless; only saints
of infants with annual incarnes less tban 150% immediate to ultimate targets are correctly nursing-home prograrn, for instance, persons have not at one time ar anotl1er violated some
identified in tbe prograrn tbeory. The elfective- witb skin cancer should not be included be- law ar regulation, wittingly ar oilierwise. This
of tbe poverty line.
When aggregates (groups ar organizations) ness of tbe project tbat used influential farm- cause tbeir condition rarely requires inpatient definition of crinlinal is too inclusive, lumping
are targets, tbey are often defined in terras of ers, for instance, depended heavily on tbe abil- services. togetber in one category trivial and serians
tbe characteristics of tbe individuais tbat con- ity and motivation of tbose farmers to An illustration of tbe considerations iliat olfenses and infrequent violators witb habitual
stitute tbem: tbeir informal and formal collec- co=unicate tbeir knowiedge persuasively to go into specifying targets is provided in Exhibit felons.
tive properties and their shared problems. An otber farmers in tbeir co=unities. Sinlarly,
140 EVALUATION Assessing the Need for a Program 141

Counting the poor is an exercise in the art of Moreover, when we looked at the poor At the Social Security Administration, we sex of the head, the number of children under
the possible. For deciding who is poor, prayers distributed demographically, by camparison decided that we would develop two measures 18, the number of adults, and whether or not the
are more relevant than calculation because with the total population, we made some of need, and state, on the basis of the incarne household lives on a farm. The threshold is
poverty, like beauty, lies in the eye of the unusual discoveries. For example, the percen- sample of the Current Population Survey, how defined as an attempt to 11 specify the minimum
beholder.... To say who is poor is to use all sorts tage of the families classified as poor who had many and what kinds of families these measures money incarne that could support an average
of value judgments. The concept has to be no children was higher than that for the delineated. lt was not the Social Security family of given composition at the lowest levei
limited by the purpose which is to be served by population as a whole, and to make it even more Administration that labeled the poverty line. lt cansistent with the standards of living prevailing
the definition. There is no particular reason to unrealistic, the percentage of the poor families remained for the Office of Ecanomic Oppor- in this country. lt is based on the amOunt needed
count the poor un[ess you are going to do with four children or more was actually less than tunity and the Council of Ecanomic Advisers to by families of different size and type to purchase
something about them. When it comes to de- the representation of such families in the select the lower of the two measures and decide a nutritionally adequate diet on the assumption
fining poverty, you can only be more subjective population. We did not think this was correct, so they would use it as the working toai. The best that no more than a third of the family incarne
ar less so. You cannot be nonsubjective. we tried to vary the poverty line-the necessary you can say for the measure is that at a time is used for food." The two thresholds were
minimum of resources-with the size and when it seemed useful, it was there. lt is in- developed from food consumption surveys con-
Defining the lssue composition of the family. teresting that few outside the Social Security ducted by the Department of Agriculture ....
Administration ever wanted to talk about the These revealed that the average expenditure for
We wanted to be sure that every family ar Setting the Benchmark higher measure. Everybody wanted only to talk food by ali families was about one-third of
consumer unit had its fair chance to be num- about the lower one, labeled the "poverty line," incarne.
bered among those who would be cansidered A cancept which can help influence public which yielded roughly the sarne number of An assumption was made that the poor
as needing attention. lndeed, it was precisely to thinking must be socially and politically credible. people in poverty as the original $3,000 would have the sarne flexibility in allocating
ensure consideration of the needs of large We need benchmarks to distinguish the measure did, except that fewer families with incarne as the rest of the population but that,
families as well as small, and of young people as population group that we want to worry about. more children were substituted for a larger obviously, their margin for choice would be less.
well as old, that we refined the initial standard A benchmark should neither select a group so number of older families without children. The amount allocated to food from the average
developed by the Council of Economic Advisers. small, in relation to ali the population, that it expenditure was cut to the minimum that the
Their standard said that any family of two or hardly seems to deserve a general program, nor Thresholds of Poverty Agriculture Department said cauld still provide
more with less than $3,000 annual incarne, and so large that a solution to the problem appears American families with an adequate diet. We
any single person living alone with less than impossible. For example, in the 1930s, President We have developed two poverty thresholds, used the low-cost plan to characterize the
$1,500, would be considered poor for purposes Roosevelt said, "1 see before me one-third of a corresponding to what we call the "poor" and near-poor and for the poor an even /ower one,
of antipoverty program planning-but not for nation ill-clothed, ill-housed, and ill-fed." This 11
"near-poor. These thresholds are set separately the ecanomy food plan.
program eligibility. This original standard led to fraction is now part of our history. No matter for 124 different kinds of families, based on the
the odd result that an elderly cauple with $2,900 how we get our numbers today, if more than a
incarne for the year would be considered poor, third of the population is called poor, it will lose SOURCE: Quoted, with permission, from Mollie Orshansky, "Perspectives on Poverty: How Poverty Is Measured,"
but a family with a husband, wife, and four little value as a public reference point. Monthly Labor Review, 1969, 92(2):37-38.
children with $3, 100 incarne would not be.

Definitions may also prove too restrictive, rehabilitate released felons decided to include small proportion would be eligible given this may eliminate those most in need of rehabili-
or underinclusive, sometimes to the point that only those who have never been drug or alcohol exclusion. ln addition, because persons with tation as targets of the proposed intervention.
almost no one falis into the target population. abusers. The prevalence of substance abuse is long arrest and conviction histories are more ln addition to specifying appropriate
Suppose that the designers of a program to so great among released prisoners that only a likely to be substance abusers, this definition boundaries, useful target definitions must be
EVALUATION Assessing the Need for a Program 143
142

feasible to apply. A specilication that hinges on ]ower risks of damage and injury. From the Useful Concepts in tions and minar accidents. They are more in-
federal perspective, the target population would terested in prevalence when dealingwith prob-
a characteristic that is clillicult to observe ar for
be viewed as ali those areas in which 100-year
Target Definition
which existing data contain no measures-for lems that cannot be eradicated quickly but
example, a definition of the targets of a job floods may occUL Because the federal govem- Understanding the nature of a social prob- require long-term management and treatrnent
trainingprogram as persons who hold favorable ment must be concemed with ali the flood lem and estimating the size and characteristics efforts, including chronic diseases such as can-
attitudes toward accepting job training-may plains in the United States, their perspective of a target population are prerequisite to docu- cer and clinically observable long-term ill-
be virtualiy impossible to put into practice. recognizes that such a flood may occur some- menting the need for a program. Delivering nesses such as severe malnutrition.
Overly complex definitions requiring much de- where as often as once every few days. 1lue to service to a target population, howeve~ requires The concepts of incidence and prevalence
tailed information are similarly clillicult to ap- their name, howeve~ 100-year floods occur in that the definition of the target population have been adapted to the study of social prob-
ply. The data required to select targets defined any one place only once in every century (on permit targets to be distinguished from nontar- lems. In studying the impact of crime on vic-
as 11farmer members of producers' cooperatives average). From the local perspective, therefore, get units in a relatively unambiguous and effi- tims, for instance, the criticai measure is the
who have planted barley for at least two seasons a given f!ood plain may not be viewed as a cient manner as part of the program's normal incidence ofvictimization: the numbers of new
and have an adolescent son" would be clillicult, reasonable target at ali and local govemments operatiog procedures. To be effective, a program cases (ar persons victimized) per interval of
if not impossible, to gatheL Moreove~ in gen- may object strongly to the burdens of a program must not only know what its target population time in a given area. Similarly, in programs
eral the more criteria a definition has, the targeted on their flood plains. is but also be able to readily direct its services aimed at lowering drunlcen-driver accidents,
smalier the number of units that can qualify for Differences in perspective can arise in pro- to that population and screen out individuais the incidence of accidents involving a drunl<en
inclusion in the target population. (The farm- gram design as well. The planners of programs who are not part of that population. This sec- driver in a specilied area and period of time may
ers satisfying the criteria just given would be a concemed with improving the quality of hous- tion discusses a number of concepts that un- be the best measure of the need for interven-
smali group indeed.) Complex specilications ing available to poor persons may have a con- derlie appropriate target definition and selec- tion. But for chronic conditions such as low
are therefore kin to narrow ones and carry the ception of housing quality much different from tion. educational attainment, criminality, ar poverty,
those of the people who will live in those dwell- prevalence is generaliy the appropriate mea-
sarne risks.
ings. Their definition of what constitutes the Incidence and Prevalence sure. In the case of poverty, for instance, preva-
target population of substandard housing for lence may be defined as the number of poor
Varying Perspectives
renewal, therefore, may result in a great outcry A useful distinction is the difference be- individuais ar families in a co=unity at a
on Target Specification
from residents of those dwellings who find tween incidence and prevalence. Incidence re- given time, regardless of when they became
Another issue in the definition of target them adequate. fers to the number of new cases of a particular pOOL
populations arises from the dillering perspec- A!though needs assessment cannot estab- problem that are identified ar arise in a speci- For other social problems, it is often un-
tives of professionals, politicians, and the other lish which perspective on program targets is fied geographical or otherwise defined area dur- clear whether one should define target popula-
stal<eholders involved-including, of course, "correct," it can help eliminate conflicts that ing a specilied period of time. Prevalence refers tions in terms of prevalence ar incidence. ln
the potential recipients of services. During ali might arise from groups tali<ingpast each otheL to the number of existing cases in a particular dealing with the problem of unemployment, it
phases of intervention, beginning with the This is accomplished by investigating the per- area at a specilied time. These concepts are is important to know its prevalence, the num-
emergence of a social problem, there can be spectives of ali the signilicant stalceholders on derived from the field of public health where bers ar proportions of the total population un-
differences in opinion as to the exact parame- target definition and helping ensure that nane generaliy they are sharply distinguished. For employed ata particular time. If the concem is
ters of the target population. is left out of the decisionprocess through which example, the incidence of influenza during a with providing financial support for the unem-
Discrepancies may exist, for instance, be- the program focus is detennined. lnfo~ation particular month is defined as the number of ployed, howeve~ it is not clear whether the
tween the views of legislators at dillerent leveis collected aboutneeds from varyingperspectives newcases reported during the month; its preva- definition should refer to persons who are un-
of govemment. At the federal levei, Congress may ]ead to a reconceptualization of the target lence during that month is the number of employed at a particular time ar those who
may p]an to alieviate the financial burden on population ar of the prospective intervention, afflicted people at any given time, regardless of become unemployed in a given period. The
the govemment for natural disasters by encour- ar even indicate the advisability of abandoning when they were first stricken. ln the health principie involved centers on the issue of
aging states to invest in such disaster-mitigat- the program (especialiy if the different perspec- secto~ project planners generaliy are interested whether one is concemed with detecting and
ing measures as improved !and-use manage- tives tum out to be contradictory and intensely in incidence when dealing with disorders of treatiognew cases as they appear orwith exist-
ment of f!ood plains and building codes that held by the various stal<eholders). short duration, such as upper-respiratory infec- ing cases whatever their time of origin.
144 EVALUATION Assessing lhe Need for a Program 145

Population at Risk have the relevaot condition, that is, false posi- persons will not use such facilities. Thus, there per 1,000 of a population je.g., 133 newunem-
tives. A prograrn that selects its clientele with may be a need but not a demaod. ployed persons per 1,000 population).
Another public health concept, population poor sensitivity overlooks maoy who need aod Some needs assessments undertaken to Rates or percentages are especialiy critical
at risk, is helpful in specifying targets, particu- qualify for service. A prograrn that selects with estimate the extent of a problem aod serve as in identifying the characteristics of the target
larly in projects that are preventive in character poor specificity uses its resources to serve maoy tl1e basis for designing prograrns are actualiy population. For exarnple, in describing the
Population at risk refers to that group of per- who do not need or qualify for service. Idealiy, at-risk assessments ar demand assessments ac- population of crime victims, it is importaot to
sons or units that has a sigoilicaot probability both high sensitivity aod high specificity are cording to the definitions just offered. Such have estimates by sex aod age group. Although
of developing a given condition. Thus, the desired in defining a target population aod assessments may do duty for true needs assess- alrnost every age group is subject to some kind
population at risk in fertility contrai prograrns selecting individuals for a prograrn. ments either because it is technicaliy infeasible of crime victiniization, young people are much
is usualiy defined as women of childbearing to measure need ar because it is impractical to more likely to be the victims of robbery aod
age. Similarly, projects designed to mitigate the implement a prograrn that deals only with the assault, whereas older persons are more likely
effects of typhoons aod hurricaoes may define Need and Demand at-need population. For exarnple, although only to be the victims of burglary aod larceny; men
targets as co=unities located in the typical sexualiy active females may require farnily are considerably less likely thao women to be
paths of such storms aod, hence, at risk of Whereas a population at risk includes ali plaoning information, the target population for the victims of sexual abuse; aod so on. The
experiencing a disaster those with a high probability of having or ac- most such prograrns is those women assumed ability to estimate targets by various charac-
A population at risk cao be defined only in quiring a given condition, a population in need to be at risk, generaliy defined by ao age spao teristics aliows a prograrn to be planned aod
probabilistic terms. Women of childbearing age is a group of potential targets who currently such as 15 to 50, because it would be difficult developed in ways that maximize opportunities
may be the population at risk in a fertility maoifest the condition. A population in need to identify aod designate only those who are to include the most appropriate participaots
contrai project, but a given womao may or may cao usualiy be defined rather exactly; that is, sexualiy active. Similarly, whereas the in-need aod to tailor the prograrn to the particular
not conceive a child within a given period of one cao identify a precise criterion for including group for ao evening educational prograrn may characteristics of sizable groups.
time. ln this instaoce, specifying the popula- a unit arnong targets je.g., a screening tech- be ali nonliterate adults, only those who are Estimates of target populations aod their
tion at risk simply in terms of age results niqueJ. For instaoce, there are reliable aod valid willing or who cao be persuaded to participate characteristics may be made at several leveis of
unavoidably in overinclusion; that is, the defi- tests for determining ao individual's degree of cao be considered the target population jari "at disaggregation. For exarnple, illiteracy rates,
nition includes maoy women as targets who literacy. These tests cao be used to specify a demaod" definition). Clearly, the distinctions calculated by dividing the number of function-
may not be in need of farnily plaoning efforts target population of functionaliy illiterate per- between populations at risk, in need, aod at aliy illiterate persons in various age groups by
because they are not sexualiy active or are sons. For projects directed at alieviating pov- demaod are importaot for estimating the scope the total number of persons in each group,
otherwise incapable of getting pregnaot. erty, one may define the population in need as of a problem, aoticipating the size of the target aliow one to estimate the target populations
families whose incarne, adjusted for farnily population, aod subsequently designing, im- that cao be reached by tailoring a project to
size, is below a certain minimum. plementing, aod evaluating the prograrn. specific age cohorts. More powerful statistical
Sensitivity and Specificity
Just because individuals constitute a popu- techniques may usefully be employed to talce
The sensitivity of a criterion for target iden- ]ation in need by some criteria representing the into account additional sociodemographic vari-
tification refers to the likelihood of correctly social construction of need used in prograrn or Rates ables simultaoeously.
selecting those targets who should be in a policy context, howeve~ does not meao that ln most cases, it is not only traditional but
prograrn in contrast to those who might also be they necessarilywaot the prograrn or service at ln addition to estimating the size of a prob- also useful to specify rates by age aod sex. ln
selected by the criterion but not be appropriate issue. Desiring service or being willing to par- lem group, it is generaliy also importaot to co=unities in which there are marked sub-
for the prograrn. If the prograrn is designed to ticipate in a prograrn defines demand for ser- lmow the proportion of a population with a cultura! differences, racial, ethnic, aod religious
serve those with a specified condition, sensitiv- vice, a concept that usualiy only partialiy over- particular problem. Maoy times it is critical to groups also become importaot denominators
ity is the ability of a screening or selection laps the applicable criteria of need. Co=unity be able to express incidence or prevalence as a for the disaggregation of characteristics. Other
procedure to identify true positives, that is, leaders aod service providers, for instaoce, may rate to compare areas or problem groups. Thus, variables useful in identifying characteristics of
t110se who actualiy have the condition. Speci- quite reasonably define a "need" for overnight the number of new cases of unemployment or the target population include socioeconomic
ficity, on the other haod, refers to correctly shelter arnong homeless persons sleeping on underemployment during a given period in ao status, geographic location, aod residential mo-
excluding those persons or units who do not the streets but may find that some of these area lincidencej might be described per 100 or bility. jSee Exhibit 4-J for ao exarnple of crime
146 EVALUATION Assessing the Need for a Program 147

offer such courses; thus, part of the problem have based service detivery on a recognition of
tums out to be that opportunities to learn the complexity of their ctients' needs.
foreign languages are insufficient. Similarly,
the fact that many primary school children of
VicLimization per 1,000 Persons Age 12 or O/der: 1996
low socioeconomic backgrounds appear to be
Qualitative Methods
Characteristic Afl Viofenl Rape; Aggravated
tired and listless in class may be explained with for Describing Needs
of Victim Crimes Sexual Assault Robbery Assault Simpfe Assauft
a finding that many regularly do not eat break- Qualitative research can be especially use-
Sex
Male 49.9 0.4 7.2 11.6 30.B fast, which, in tum, reflects their families' eco- ful for obtaining detailed, textured knowledge
Female 34.6 2.3 3.4 6.2 22.7 nomic problems. Of course, different stake- of the specific needs in question. Such research
Age holders are likely to have different views about
12-15 95.0 26 10.0 15.6 66.B can range in complexity from interviews of a
16-19 102.7 4.9 12.0 25.3 60.4 tl1e nature and source of the problem so it is few persons or group discussions to elaborate
20-24 74.3 2.1 10.0 15.9 46.4 important that the full range of perspectives be ethnographic research such as that employed
25-34 51.l 1.B 7.1 9.B 32.4 represented (see Exhibit 4-K for an exaniple of by anthropologists. As an exaniple of the utility
3549 32.B 1.3 3.B 7.4 20.3
diverse stakeholder perspectives). of such research, qualitative data on the struc-
50-64 15.7 0.1 l.B 3.B 10.0
65+ 4.9 o.o 1.1 O.B 3.0 Cultural factors or perceptions and attribu- ture of popular betiefs can contribute substan-
Race tions that characterize a target population may tially to the effective design of educational carn-
White 40.9 1.3 4.2 B.2 27.2 be especially relevant to the effectiveness of a
Black 52.3 l .B 11.4 13.4 25.6 paigns. What, for instance, are the trade-offs
Hispanic 44.0 1.2 B.4 10.6 23.9 prograrn's outreach to members of the target people believe exist between the pleasures of
Olher 33.2 2.1 7.4 7.2 16.6 population and the way in which it detivers its cigarette smoking and the resulting health
service. A thorough needs assessment on pov- risks? A good educational prograrn must be
SOURCE: U.S. Department of Justice, Bureau of Justice Statislics, Criminal Viclimizatian 1996 (Washington, DC: U.S.
erty in Appalachian mountain communities, adapted to those perceptions.
Department of Justice, November 1997).
for instance, should reflect the sensitivities of Carefully and sensitively conducted quali-
the target population about their self-suffi- tative studies are particularly important for
ciency and independence. Prograrns that are uncovering process information of this sort.
construed as charity or that give what are per- Thus, ethnographic studies of disciplinary
victimization rates disaggregated by sex, age, presumed to be responsive to a given problem ceived as handouts are likely to be shurmed by problems within high schools may not only
and race.) or need. To be effective, a prograrn may need to needy but proud families. provide some indication of how widespread
adapt its services to the local nature of the Another important dimension of service disciplinary problems are but also suggest why
problem and the distinctive circumstances of needs may involve difficulties some members some schools have fewer disciplinary problems
DESCRIBING THE NATURE
the persons in need. This, in tum, requires of the target population have in using services. than others. The findings on how schools differ
OF SERVICE NEEDS information about the way in which the prob- This may result from transportation problems, might have implications for the ways prograrns
lem is experienced by those in need, their per- limited service hours, lack of child care, or a are designed. Or consider the qualitative re-
As described above, a central function of needs ceptions and attributions about relevant ser- host of similar such obstacles. The dillerence search on household energy consumption that
assessment research is to develop estimates of vices and prograrns, and the barriers and between a prograrn with an effective service revealed the fact that few householders had any
the extent and distribution of a given problem difficulties they encounter in attempting to delivery to needy persons and an ineffective one information about the energy consumption
and the associated target population. Howeve~ access services. is often chiefly a matter of how much attention characteristics of their appliances. Not know-
it is also often important for such research to A needs assessment might, for instance, is paid to overcoming these barriers. )oh train- ing how they consumed energy, these house-
yield useful descriptive information about the probe into the matterof why the problem exists ing prograrns that provide child care to the holders could not very well develop effective
specific character of the need within that popu- and what other problems are tinked with it. For participants, nutrition prograrns that detiver strategies for reducing their consumption.
lation. This is important because it is often not example, a search for information on how meals to the homes of elderly persons, and A popular and useful technique for obtain-
sufficient for a social prograrn to merely detiver many high school students study a non-English community health clinics that are open during ing rich information about a social problem is
some standard services in some standard way language may reveal that many schools do not evening hours all illustrate approaches that the focus group approach made popular in the
148 EVALUAT!ON Assessing the Need for a Program 149

A focus group protocol is a list of topics ar and for how long, and other services re-
Telephone interviews were conducted in open-ended questions to be covered in a focus ceived (10 minutes).
We don't have training ar equipment to do
three rural Colorado communities to identify group session that is used to guide the group
high-tech care. lntroduce idea of barriers to services-ask
health service problems related to cancer. ln each discussion. The protocol should (a) cover !opies
participants for their views on what have
community the study participants included in a logical, developmental arder so that they
Community influentials: been the most important barriers to receipt
(a) health care providers (physicians, nurses, build on one another; (b) raise open-ended issues
of family support services (probe regarding
public health personnel), (b) community influ- that are engaging and relevant to the participants
People are on waiting lists for services for transportation, treatment by agency person-
entials (teachers, librarians, directors ofcommunM and that invite the group to make a collective
several months. nel, regulations, waiting Jists); have they dis-
ity agencies, business leaders), and (c) patients response; and (c) carve out manageable
11 continued any services ar been unab/e to get
ar family members of patients who had a cancer There are not enough professionals orvo[unM chunks" of topics to be examined one ata time
ones they want? (30 minutes).
experience. While there was general agreement teers here. in a delimited period. For example, the following
about problems with availability and access to is a protocol for use in a focus group with Probe for reasons behind their choices of
services, each stakeholder group had somewhat
There is inadequate provider knowledge low-income women to explore the barriers to most important barriers (20 minutes).
about specialized services. receiving family support services:
different, perceptions of the nature of the Ask for ideas on what could be done to
problems: lntroduction-greetings; explain purpose of overcome barriers in the future-what would
Patients and family members:
the session; fill out name cards; introduce make it ar would have made it easier to enter
Physicians and health care providers: observers, ground rules and how the focus and remain in the service loop? (30 minutes).
A time ar two we have had no doctor-here.
group works (10 minutes).
Regional facilities only accept paying pa- Debrief and wrap up-moderator summary,
We have a doctor here now but his patients
Participant introductions-give first names clarifications, and additional comments ar
tients ar close down. have no money and 1 hear he's going to
only; where participants live, age of children; questions (1 O minutes).
The remoteness of the community creates a leave.
which family support services are received
lack of services. We need treatment locally.
Physician shortage exists because of low SOURCE: Adapted from Susan Berkowitz, "Using Qualitative and Mixed-Method Approaches, 11 in Needs Assessment'
1was on a waiting [jst for three weeks before
A Creative and Practical Cuide for Social Scientists, eds. R. Revlere, S. Berkowitz, C C Carter, and C G. Ferguson
salaries, large workloads, and difficult pa- the mammography van got here. (Washington, DC: Taylor & Francis, 1996), pp. 121-146.
tients.

SOURCE: Adapted from Holly W. Halvorson, Donna K. Pike, Frank M. Reed, Maureen W. McClatchey, and Carol A.
11
Gosselink, "Using Qualitative Methods to Evaluate Health Service Delivery ln Three Rural Colorado Communities, group). A range of other group tecbniques for participants, care must be taken in = g
Evaluation & lhe Health Professions, 1993, 16(4):434447. eliciting information for needs assessment can some of these stakeholders in the sarne focus
be found in Witkin and Altschuld (1995). group.
Appropriate participants in focus groups Any use of key informants in needs assess-
would generally include various knowledgeable ment must, therefore, involve a careful selec-
past several presidential elections when small 1994; Kruege~ 1988). With a careful selection co=unity leaders, directors of service agen- tion of the persons or groups whose perceptions
paneis of voters were assembled to provide and grouping of individuais, a modest number cies, the line personnel in those agencies who are going to be talcen into account. A useful
perceptions of how the campaign rhetoric was of focus groups can provide a wealth of descrip- deal firsthand with clients, representatives of approach to identifying key informants for a
being received (Morgan, 1988). Focus groups tive information about the nature and nuances advocacy groups, persons experiencing the so- needs assessment is snowball sampling. This
bring together selected knowledgeable persons of a social problem and the service needs of cial problem or service needs directly, and other tecbnique requires that an initial set of appro-
for a discussion of a particular topic or theme those who experience it (Exhibit 4-L provides a such stakeholders. Of course, for these inter- priate informants be located through some rea-
under the supervision of a facilitator (Dean, helpful protocol for a needs assessment focus actions to be productive and comfortable for the sonable means and surveyed. They are then
. 150 EVALUATION Assessing lhe Need for a Program 151

Because of the distinctive advantages of nature of the problem (e.g., Mitra, 1994). The
qualitative and quantitative approaches, a use- second stage, estimation, builds on this infor-
ful and frequently used strategy is to conduct mation to design a more quantitati.ve assess-
As efforts to help the homeless move beyond and women interviewed in New York City shel- needs assessment in two stages. The initial, ment that provides reliable estimates of the
the provision of temporary shelter, it is important ters revealed that they had multiple needs no! exploratory stage uses qualitative research ap- extent and distribution of the problem.
to understand homeless individuais' perspectives easily met by a single service. The percentage proaches to obtain rich information on the
on their needs for assistance. Responses from a reporting a need for help on each oi 20 items was
representative sample of 1,260 homeless men as follows:
Finding a place lo live 87.1 Problems with drugs 18.7

Having a steady incarne 71.0 learning to gt along better with other people 18.5

Finding a job 63.3 Nerves and emotional problems 17.9


!\"! Within evaluation research, needs assessment attempts to answers questions about
lmproving my job skills 57.0 Learning how to protect myself 17.6 the social conditions a program is intended to address and the need for the program,
Learning how to get what 1have Learning how to read and fill out forms 17.3 or to determine whether a new program is needed. More generally, it may be used to
coming from agencies 45.4 15.0 identify, compare, and prioritize needs within and across program areas.
legal problems
Getting on public assistance 42.1 13.0 !\"! Adequate diagnosis of social problems and identification of the target population for
Drinking problems
Health and medical problems 41.7 12.4 interventions are prerequisites to the design and operation of effective programs.
Getting around town
Learning how to manage money 40.2 9.6
Nonetheless, it must be recognized that social problems cannot be objectively defined
Getting veteran's benefits
but, rathe~ are social constructions that emerge from the interests and poltica!
Getting along wilh my family 22.8 5.1
Problems with the police actions of the parties involved.
Getting on 551/SSD 20.8
!\"! To specify the size and distribution of a problem, evaluators may gather and analyze
SOURCE: Adapted from Daniel B. Herman, Elmer L Struen!ng, and Susan M. Barrow, "Self-Reported Needs for Help data from existing sources, such as the U.S. Census, ar use ongoing social indicators
Among Homeless Men and Women, 11 Evafuation and Program Pfanning, 1994, 17(3):249-256. to identify trends. Because the needed information often cannot be obtained from
such sources, howeve~ evaluators frequently conduct their own research on a social
problem. Useful approaches include studies of agency records, surveys, censuses, and
key informant surveys. Each of these has its uses and limitations; for example, key
informant surveys may be relatively easy to conduct but of doubtful reliability; agency
asked to identify other informants whom they ment consists of a program's current clientele records generally represent persons in need of services but may be incomplete;
believe are lmowledgeable about the matter at o~ in the case of anewprogram, representatives surveys and censuses can provide valid, represen- tative information but can also be
issue. These other informants are then con- of its potential clientele. This group, of course, expensive and technically demanding.
tacted and asked, in tum, to identify still oth- is especially knowledgeable about the charac- ii!i Forecasts for future needs are often very relevant to needs assessment but are complex
ers. When this process no longer produces rele- teristics of the problem and the associated and technical activities ordinarily performed by specialists. ln using forecasts,
vant new names, it is likely that most of those needs as they are experienced by those whose evaluators must talce care to examine the assumptions on which the forecasts are
whowould qualify as key informants have been lives are most affected by the problem. Al- based.
identilied. Because those persons aclive and though they are not necessarily in the best
!\"! Appropriate definitions and accurate information about the numbers and charac-
involved in any matter of public interest in a position to report on how widespread the prob-
teristics of the targets of interventions are crucial throughout the intervention
co=unity tend to lmow of each othe~ snow- lem is, they are the key witnesses with regard
process, from initial planning through all the stages of program implementation.
ball sampling works especially well for key to how seriously the problem affects individu-
Targets may be individuais, groups, geographical areas, or physical units, and they
informant smveys about social problems. ais and what climensions of it are most press-
may be defined as direct ar indirect objects of an intervention.
An especially useful group of informants ing. Exhibit 4-M illustrates the unique perspec-
that should not be overlooked in a needs assess- tive of potential service beneficiaries.
152 EVALUAT!ON

!l'l Good target specilications establish appropriate boundaries, so that an intervention


correctly addresses the target population, and are feasible to apply. ln defining targets,
care must be taken to allow for the varying perspectives of dillerent stakeholders.
U seful concepts in target definition include incidence and prevalence, population at
risk, sensitivity and specilicity, need and demand, and rates.
!l'l For purposes of prograrn planning or evaluation, it is important to have detailed
information about the local nature of a social problem and the distinctive circum-
stances of tl1ose in need of prograrn services. Such information is usually best
obtained through qualitative methods such as ethnographic smdies or focus groups
with selected representatives of various stakeholders and observers.
EXPRESSING AND ASSESSING
PROGRAM THEORY
Program theory The set of assumptions about the manner in which the program relates to the Maria Cuomo, former govemor of New Turk, once descrbed bis mother's rules for success
social benefits it is expected to produce and the strategy and tactics the as {a} figuIB out wlwt you want to do and (b} do it. These are pretty much the same rules that
program has adopted to achieve its goals and objectives. Within program social programs must follow if they are to be effective. ln the last chapter, we mscussed how the
theory we can distinguish impact theory, relating to the nature of the change evaluator could assess the need for a program. Given an identified need, program decisiomnakers
in social conditions brought about by program action, and process theory, must {a} conceptualize a program capable of alleviating that need and {b} implement it. ln this
which depicts the program's organizational plan and service utilization plan chapter, we review the concepts and procedmes an evaluator can apply to the task of assessing
(see Chapter 3 for fuller descriptions). the quality of the program conceptualization. ln the next chapter, we descrbe the ways in which
An explicitly stated version of program theory that is spelled out in some the quality of the program implementation can be assessed.
Articulated program
detail as part of a program's documentation and identity oras a result of The social problems that programs adch:ess are often so complex and mfficult that bringing
theory
efforts by the evaluator and stakeholders to formulate the theory. about even mm:ginal improvement may pose formidable challenges. The foundation on which
evezy program rests is some conception of what must be dane to bring about the intended social
lmplicit program theory Assumptions and expectations inherent in a program's services and practices
benefits, whether that conception is expressed in a detailed program plan and rationale ar is only
that have not been fully articulated and recorded.
implieit in the program's structure and activities. That conception is what we have referred to as
Evaluability assessment Negotiation and investigation undertaken jointly by the evaluator, the the program theory.
evaluation sponsor, and possibly other stakeholders to determine if a program A program's theozy can be a good one, in which case it represents the "know-how" necessary
meets the preconditions for evaluation and, if so, how the evaluation should for the program to attain the desired results, ar it can be a poor one that would not produce the
be designed to ensure maximum utility. intended effects even if implemented well. One aspect of evaluating a program, therefore, is to
Black box evaluation Evaluation of program outcomes without the benefit of an articulated program assess how good the program theory is-in particular, how well it is formulated and whether it
theoiy to provide insight into what is presumed to be causing those outcomes presents a plausible and feasible plan for improving the target social conmtions. For program
and why. theozy to be assessed, however, it must fiist be expressed clearly and completely enough to stand
for review. This chapter descrbes how evaluators can tease out the theozy implicit in a program
and then, after it has been made explicit, assess how good it is.

n Chapter 3, we advocated that evaluators ation questions. This advice was presenteei in
l analyze a program's critical assumptions
and expectations about the way in which it is
the context of planning an evaluation and set-
ting priorities for the issues it would address.
intended to improve social conditions as an aid ln this chapte~ we return to the topic of pro-
to identification of potentially important evalu- gram theory, notas a framework for identifying
155
156 EVALUAT!ON Expressing and Assessing Program Theory 157

important evaluation questions, but as a con- criminal behavio~ early sexual activity, ar teen and expectations that constitute the rationale This led to the view that a qualitative assess-
stituent part of the program itself pregnancy, are addressed by programs that pro- for the way the program is structured and op- ment of whether minimal preconditions for
Every program embodies a conception of vide some mbc of counseling and educational erated. It is rare for a program to be able to evaluation were met should precede most
the structure, functions, and procedures appro- services. Although rarely made explicit during immediately provide the evaluatorwith a state- evaluation efforts. Wholey and bis colleagues
priate to attain its goals. This conception con- planning, these programs are based on the ment of its program theory in a sufficiently termed the process evaluability assessment
stitutes the "logic" ar plan of the program, assumption that people will change their prob- explicit and detailed form to allow meaningful (see Exhibit 5-A).
which we have called program theory. The lem behavior if given information and inter- assessment. Although always implicit in pro- Evaluability assessment generally involves
program theory explains why the program does personal support for doing so. Such theories gram structure and operations, a full descrip- three primary activities: (a) description of the
what it does and provides the rationale for may seem reasonable in the general case, but tion of the program theory is seldom written "program model" with particular attention to
expecting that doing things that way will experience and research provide ample evi- down and available in program documents. defining the program goals and objectives,
achieve the desired results. dence that many such behaviors are very resis- Moreove~ when some write-up of program the- (b) assessment of how well defined and
Evaluators and other informed observers tant to change despite knowledge by the par- ory is available, it is often in material that has evaluable that program model is, and (c) iden-
recognize tha t there is little basis for presuming ticipants about how to change and strong been prepared for funding proposals ar public tification of stakeholder interest in evaluation
that program theory is universally sound and encouragement from loved ones to doso. Thus, relations purposes and may not correspond and the likely use of the findings. Evaluators
thus warrants little cancera. There are many the theory that education and supportive coun- well with actual program practice. conducting evaluability assessments operate
poorly designed social programs in operation seling will reduce deviant behavior may not be Assessment of program theory, therefore, much like program ethnographers. They seek
with faults that reflect deficiencies in their a sound basis for program design. almost always requires that the evaluator first to describe and understand the program
underlying conception of how the desired social The rationale and conceptualization on draw on program sources to synthesize and through interviews and observations that will
benefits can be attained. This circumstance which a program is based, therefore, should be articulate the theory in a form amenable to reveal its "social reality" as viewed by program
stems in large part from the fact that careful, subject to critica! scrutinywithin an evaluation analysis. Accordingly, the discussion in this personnel and other significant stakeholders.
explicit conceptualization of program objec- just as any other important aspect of the pro- chapter is organized around two themes: The evaluator begins with the conception of the
tives and how they are supposed to be achieved gram. If the program's goals and objectives do (a) how the evaluator can explicate and express program presented in documents and official
is often not given sufficient attention during not relate in a reasonable way to the social program theory in a form that will be repre- information, but then tries to see the program
planning for new programs. Sometimes the conditions the program is intended to improve, sentative of key stakeholders' actual under- through the eyes of those closest to it. The
political context within which programs origi- ar the assumptions and expectations embodied standing of the prograrn and workabie for pur- intent is to end up with a description of the
nate does not permit extensive planning, but in a program's functioning do not represent a poses of evaluation, and (b) how the quality of program as it exists and an understanding of
even when that is not the case, conventional credible approach to bringing about that im- the articulated program theory can then be the program issues that really matter to the
practices for designing programs are not very provement, there is little prospect that the evaluated. various parties involved. Although this process
probing with regard to the nature and plausi- program will be effective. Evaluations of pro- clearly involves considerable judgment and dis-
bility of the underlying theory. The human gram process, impact, and efficiency thus ride cretion on the part of the evaluator, various
service professions operate with repertoires of on the presumption that the program theory is practitioners have attempted to codify its pro-
established modes and types of intervention sound. Accordingly, evaluators must often THE EVALUABILITY cedures so that evaluability assessments will be
associated with their respective specialty areas. make some assessment of the quality of a ASSESSMENT PERSPECTIVE reproducible by other evaluators (see Rutrnan,
As a result, program design is often principally
a matter of configuring a variation of familiar
program's theory. Evaluating program theory,
howeve~ is not an easy task and certainly does
----------------- 1980; Smith, 1989; Wholey, 1994).
A common outcome of evaluability assess-
"off the shell" services into a package that not lend itself to structured and formalistic One of the earliest systematic attempts to de- ments is that program managers and sponsors
seems appropriate for a social problem without procedures. It is an important task, nonethe- scribe and assess program theory arose from recognize the need to modify their programs.
a dose analysis of the match between those less, and one the evaluator must be prepared to the experiences of an evaluation research group The evaluability assessment may reveal faults
services and the nature of the problem. undertake in many situations. at the Urban Institute in the 1970s (Wholey, in a program's delivery system, that the pro-
For example, many social problems that The first step in assessing program theory 19 79). They found it often difficult, sometimes gram's target population is not well defined, ar
involve deviant behavior arnong the target is to articulate it, that is, produce an explicit impossible, to undertal<e evaluations of public that the intervention itself needs to be recon-
population, such as alcohol and drug abuse, description of the conceptions, assumptions, programs and began to analyze the obstacles. ceptualized. Or there may be few prograrn
158 EVALUATION Bxpresslng and Assesslng Program Theory 159

lf evaluators and intended users fail to agree These four problems, which characterize Evaluators from the Urban Jnstitute worked Analysis and synthesis of lhe resulting data
on program goals, objectives, information many public and private programs, can be with managers and policymakers in the yielded a logic model that presented program
priorities, and intended uses of program reduced and often overcome by a qualitative Appalachian Regional Commission (ARC) on the activities, program objectives, and the assumed
performance information, those designing evaluation process, eva/uability assessmen~ that design of their health and child development causal links between them. The measurability and
evaluations may focus on answering questions documents the breadth oi lhe four problems and program. ln this evaluability assessmen~ the plausibility of program objectives were than
evaluators analyzed and new program designs more likely
that are not relevant to policy and management helps programs-and subsequent program
decisions. Jf program goa\s and objectives are evaluation work-to meet the following criteria: to lead to demonstrably effective performance
unrealistic because insufficient resources have Reviewed existing data on each of the 13 were presented. These included both an overall
been applied to criticai program activities, the Program goals, objectives, important side state ARC-funded health and child develop- ARC program model and a series of individual
program has been poorly implemented, ar effects, and priority information needs are ment programs; models, each concerned with an identified
administrators Jack knowledge of how to achieve well defined. Made visits to five states and then selected objective of the program.
program goals and objectives, the more fruitful two states to participate in evaluation design ln reviewing lhe report, ARC staff were asked
Program goals and objectives are plausible.
course may be for those in charge of the program and implementation; to choose explicitly among a/ternative courses of
to change program resources, activities, ar Relevant performance data can be obtained. action. lhe review process consisted of a series
objectives before formal evaluation efforts are Reviewed documentation related to con- of intensive discussions in which ARC and Urban
lhe intended users of the evaluation results gressional, commission, state, and project
undertaken. Jf relevant data are unavailable and lnstitute staff focused on one objective and
have agreed on how they will use the infor- objectives and activities (including the
cannot be obtained at reasonable cost, program model at a time. ln each session, the
mation. authorizing legislation, congressional hear-
subsequent evaluation work is likely to be evaluators and staff attempted to reach
inconclusive. lf policymakers ar managers are Evaluability assessment is a process for ings and committee reports, state planning agreement on the validity oi the flow models
unable ar unwilling to use the evaluation clarifying program designs, exploring program documents, project grant applications, ARC presented, the importance of the respective
information to change the program, even the reality, and-il necessary-helping redesign contract reports, local planning documents, objective, and lhe extent to which any oi the
most conclusive evaluations are likely to produce programs to ensure that they meet these four project materiais, and research projects); information options ought to be pursued.
"information in search of a user." Unless these criteria. Evaluability assessment not only shows lnterviewed approximately 75 people on ARC ended up adopting revised project
problems can be overcome, the evaluation will whether a program can be meaningfully congressional staffs and in commission designs and deciding to systematically monitor
probably not contribute to improved program evaluated (any program can be evaluated) but headquarters, state ARC and health and lhe performance of ali their health and child
performance. also whether evaluation is likely to contribute to child development staffs, local planning development projects and to evaluate the
improved program performance. units, and local projects; effectiveness of the 11 innovative 11 ones. lwelve of
lhe 13 ARC states have since adopted the
SOURCE: Quoted, with perrnission 1 frorn Joseph S. Wholey, "Assessing the Feasibility and Likely Usefulness of Participated in workshops with approxi- performance monitoring system. Representatives
Evaluation," in Handbook of Practica/ Prugram Evaluation, eds. J. S. Wholey, H. P. Hatry, and K. E. Newcomer {San mately 60 additional health and child devel- of those states report that project designs are
Francisco: Jossey-Bass, 1994), p. 16. opment practitioners, ARC state personnel, now much more clearly articulated and that they
and outside analysts. believe lhe projects themselves have improved.

SOURCE: Adapted from Joseph S. Wholey, 11 Using Eva[uation to Improve Program Performance,'' in Evafuation Research
objectives that stakeholders agree 011 ar 110 The aim of evaluability assessme11t is thus and Practice: Comparative and lnternational Perspectives, eds. R. A. Levine, M. A. Solomon, G.-M. Hellslern, and H.
Wollmann (Beverly Hills, CA: Sage, 1981), pp. 92-106.
feasible perform311ce in.clicators for the objec- to create aclimate favorable to evaluatio11 work
tives. ln. such cases, the evaluability assess- 311d 311 agreed-011 nnder~t311din.g of the 11ature
me11t has nncovered problems with the pro- 311d objectives of the program thatwillfacilitate
gram design, which program m311agers must evaluatio11 design. As such, it C3Il be in.tegral to (see Chapters 2 311d 3). To illustrate the typical Evaluability assessmem requires program
correct before 311y meaningful perform311ce the approach the evaluator employs to tailor 311 procedure, Exhibit 5-B presents 311 example of stakeholders to articulate the program design
evaluatio11 C311 be nndertaken. evaluatio11 311d formulate evaluatio11 questio11s 311 evaluability assessme11t. 311d logic (the program model); howeve~ it c311
160 EVALUATION Expressing and Assessing Program Theory 161

also be carried out for tbe purposes of describ- witb marital difficulties. Altbough it may be planJ. A brief review of tbese basic tbeory com- Witb tbis review as background, we tum to
ing and assessing program tbeory (Wholey, reasonable to assume tbat discussing marital ponents may be helpful at tbis point. consideration of tbe concepts and procedures
1987). Indeed, tbe evaluability assessment ap- problems witb a trained professional would be Program impact tbeory delineates tbe an evaluator can use to extract and articulate
proach represents tbe most fully developed set helpful, tbe way in which such interactlon cause-and-effect sequence tbrough which tbe program tbeory as a prerequisite for assessing it.
of concepts and procedures available in tbe translates into improvements in tbe marital program is expected to bring about change in
evaluation literature for describing and assess- relationship is not described by an explicit tbe- tbe social conditions it addresses. An agricul-
ing a program's conceptualization of what it is ory nor would different counselors necessarily tura! extension program to increase tbe use of
What Is a Program for
supposed to be doing and why. We tum now to agree about tbat process. disease-resistant seeds among com growers, for
a more detailed discussion of procedures for When a program's tbeory is implicit ratber instance, may consist of distribution of educa- Purposes of Program Theory?
identifying and evaluating program tbeory, tban articulated, tbe evaluator must extract tional materials, promotional talks witb farm- A crucial early step in articulating program
drawing heavily on tbe writings associated witb and describe it tbrough some appropriate ers' groups, and supply of seeds at discount tbeory is to define tbe boundaries of the pro-
tbe practice of evaluability assessment. means before it can be analyzed and assessed. prices. The presumptions tbat information, gram at issue (Smith, 1989). A human service
The first topics we must discuss, tberefore, are persuasion, and financial incentives will influ- agency may have many programs and provide
how program tbeory can be elicited if it is not ence farmers' motivation (tbe action hypotbe- multiple services; a regional program may have
ELICITING AND EXPRESSING already fully articulated, how it might most sisJ and tbat increased motivation will lead many agencies and sites. Depictiog program
PROGRAM THEORY usefully be expressed, and how it can be vali- tbem to use tbe new seeds (conceptual hy- tbeory tbus almost always requires a clear defi-
dated to ensure tbat it is an accurate repre- pothesisJ constitute tbe program's impact nition of which components, activities, objec-
sentation of a program's actual working as- tbeory. tives, and target populations are encompassed
Sometimes, tbough not often, a program's tbe- sumptions. Program process tbeory provides an ac- in tbe program at issue. There is usually no one
ory is spelled out in some detail in program The evaluation literature presents diverse count of how tbe program intends to bring correct definition of a program for tbis purpose
documents and well understood by staff and ways of defining and depicting program tbeory about tbe desired interactions witb tbe target and tbe boundaries tbat tbe evaluator applies
stalceholders. In tbis case, we might say tbe and tbus muddies tbe waters for anyone want- population and provide tbe planned services. will depend, in large part, on tbe scope of tbe
program is based on an articulated theory ing to see clearly to tbe bottom of tbis issue The service utilization plan describes how tbe evaluation sponsor's concems and tbe program
(Weiss, 1997). This is most !ikely to occur (Weiss, 1997). As indicated in Chapter 3, we target population will be engaged witb tbe pro- domains to which tbey apply.
when tbe original planning and design of tbe view program tbeory as a relatively detailed gram from initial contact to completion of tbe One way to circumscribe tbe program en-
program are tbeory based. For instance, tbe description of tbe relationships between pro- intended services. For tbe agricultura! exten- tity at issue is to work from tbe perspective of
design and delivery of a school-based drug use gram resources, program activities, and pro- sion program outlined above, tbe service utili- tbe decisionmakers expected to act on tbe
prevention program tbat features role-playing gram outcomes tbat shows how tbe program is zation plan would lay out tbe sequence of inter- evaluation findings and tbe nature of tbe deci-
of refusal behavior in peer groups may be de- supposed to work and, in particul~ how it is actions tbe target farmers are expected to have sions tbey are expected to malte. What consti-
rived directly from social learning tbeory and supposed to bring about the intended out- witb tbe educational materials, tbe agricultura! tutes tbe program for which tbeory is to be
its implications for peer influences on adoles- comes. The objective for an articulation of extension agents, and tbe suppliers of tbe dis- articulated, tben, should at minimum repre-
cent behavim program tbeory is to depict tbe "program as ease-resistant seeds. The organizational plan sent tbe relevant jurisdiction of tbose decision-
ln many cases, however:, programs involve intended," tbat is, tbe actual expectations held would describe the program activities and re- makers and tbe organizational structures and
services and practices tbat are viewed as rea- by program decisionmakers about what tbe sources and how tbey are to be organized and activities about which decisions are Wcely to be
sonable for tbe purposes of tbe program but tbe program is supposed to do and what results are managed. To effectively promote tbe new seeds, made. If tbe evaluation sponsor is tbe director
underlying assumptions and explanations of expected to follow. for instance, tbe agricultura! extension pro- of a single local co=unity mental healtb
just how tbey are presumed to accomplish For expository purposes, we highlighted gram would have to prepare and distribute agency, for instance, the decisions at issue and,
tbose purposes has not been fully articulated two components of a complete program tbeory: educational materials, train agents and arrange hence, tbe boundaries of the program to be
and recorded. In tbese cases, we might say tbat program impact tbeory {consisting of an action promotional opportunities for tbem, acquire assessed may be defined primarily around one
tl1e program has animplicit theory o~ as Weiss hypotbesis and a conceptual hypotbesisJ and sufficient quantities of tbe new seeds, and or- of tbe distinct combinations of service packages
(1997) put it, a tacit theory. This might be tbe program process tbeory (consisting of tbe ser- ganize distribution and the cost subsidy and target patients adrninistered by tbat
case for a counseling program to assist couples vice utilization plan and tbe organizational tbrough, say, local co=ercial suppliers. agency, such as outpatient counseling for eating
162 EVALUATION Expressing and Assesslng Program Theory 163

clisorders. When the evaluation sponsor is the Procedures for Explicating


state clirector of mental health, howeve~ the Program Theory
relevant program boundaries may be defined
around effectiveness questions that relate to For a program in the planning stage, theory Adapted Work Services (AWS) was initiated at cognitively from working in a sheltered environ-
outpatient counseling services statewide, that might be derived from prior practice and re- the Rochelle Center in Nashville, Tennessee, to ment and their family members would benefit
is, the outpatient counseling component of ali search. For an existing program, howeve~ the provide low-stress, paid work and social from being occasionally relieved of the burden of
the local mental health agencies in the state. appropriate task is to describe the theory that interaction to patients in the early stages of caring for them. The evaluator described the
Because program theory deals mainlywith is actualiy emboclied in the program structure Alzheimer's disease. lt was based on the belief procedures for farmulating a program process
and operation. To accomplish this, the evalu- that the patients would benefit emotionally and theory for this program as follows:
means-ends relations, the most critical aspect
of defining program boundaries is to ensure ator must internet with the program stakehold-
The creation of the operational model of the AWS program involved using Post-it notes and
that they encompass ali the important activi- ers to draw out their implicit program theory,
butcher paper to provide a wallsize depiction of the program. The first session involved only
ties, events, and resources linked to one ar that is, the theory represented in their actions
the researcher and the pragram director. lhe first question asked was, 11 What happens when
more outcomes recognized as central to the and assumptions. a prospective participant calls the center for information?" lhe response was recorded on
endeavoL This involves a form of backward a Post-it note and placed on the butcher paper. The next step was then identified, and this
mapping in which the point of departure is a The Implicit Theory of Program too was recorded and placed on the butcher paper. The process repeated itself until all
set of well-defined program objectives relating Personnel and Other Stakeholders (known) activities were identified and placed on the paper. Once the program director could
to the social benefits the program intends to nat identify any more activities, the Post-it notes were combined into clusters. lhe clusters
produce. From there, ali the activities and re- The most straightforward approach to de- were discussed until potential component labels began to emerge. Since this exercise was
sources under the relevant organizational aus- veloping a description of a program's theory is the product of only two people, the work was left in an unused roam for two weeks so that
pices that are presumed to contribute to attain- to obtain it from program personnel and other the executive director and all other members of the management team could react to the
pertinent stakeholders. The general procedure work. They were to identify missing, incorrect, ar misplaced activities as well as comment
ing those objectives are identified as part of the
for this involves successive iteration. Draft de- on the proposed components. After severa! feedback sessions from the staff members and
program. From this perspective, the eating clis-
scriptions of the program theory are generated, discussions with the executive director, the work was typed and prepared for presentation
orders program at either the local ar state leve!
usualiy by the evaluato~ and cliscussed with to the Advisory Baard. lhe board members were able to reflect on the content, provide
would be defined as the set of activities orga- further discussion, and suggest additional changes. Severa! times during monthly board
nized by the respective mental health agency koowledgeable stakeholder informants to get
meetings, the executive director asked that the model be revisited for planning purposes.
that have an identifiable role in attempting to feedback. The draft is then refined on the basis This helped further clarify the activities as well as sharpen the group's thinking about the
alieviate eating clisorders for the eligible popu- of their input and shown again to appropriate program.
lation in the respective juriscliction. stakeholders. This process continues until the
Note, howeve~ that although these ap- stakeholders find little to criticize in the de- SOURCE: Quoted, with permission, from Doris C. Quinn, "Formative Evaluation of Adapted Work Services for
proaches to defining a program for purposes of scription. The theory description developed in Alzheimer's Disease Victims: A Framework for Practical Evaluation in Health Care" {do cloral diss., Vanderbilt University,
this fashion may involve iropact theory ar pro- 1996), pp. 4647.
articulating program theory are straightforward
in concept, they can be problematic in practice. cess theory ar any components ar combination
Not only can programs be complex, with cross- deemed relevant to the evaluation purposes.
cutting resources, activities, and goals, but the Exhibit 5-C presents one evaluator's account program functions and circumstances. Each of there will likely be pertinent information in the
characteristics described above as linchpins for of how a program process theory was formu- these warrants some discussion. authorizing legislation ar legislative history
program definition can themselves be clifficult lated. and there may be accompanying regulations
to establish. Thus, in this matte~ as with so The primary sources of information for Documents. Some written description of the and guidelines. Descriptive information for
many other aspects of evaluation, the evaluator developing and clifferentiating descriptions of program ar crucial aspects of it will almost programs is often found in conjunction with
must be prepared to negotiate a program defi- prograrn theory are ia) review of program docu- always be available. Howeve~ the form, variety, fund-raising and fiscal accountability. For ex-
nition agreeable to the evaluation sponsor and ments; lbJ interviews with program personnel, and amount of such documenta tion will vary ample, grants, grant applications, contract
key stakeholders and be flexible about moclifi- stakeholders, and other selected informants; considerably depencling on the nature of the documents, budget justifications, auclit re-
cations as the evaluation progresses. and icJ site visits and observation of various program. For programs with legislative origins, ports, and annual financial reports may provide
164 EYALUATION Expressing and Assesslng Program Theory 165

infonnation about program goals and charac- evaluation. Thus, although program docu- serves. Surprisingly often this group is over- work group. A work group consists of program
teristics. Similarly, documents delineating for- ments can be very illUIIlinatiog to an evaluator looked by evaluators fonnulating program de- managers, staff, and representatives of stal<e-
mal commitments with other agencies ar attemptiog to understand and describe a pro- scriptions and depictiog program theory, per- holders who are knowledgeable about program
groups, for instance, interagency agreements to gram, their original context and purposes must haps because they are generally less accessible details and internet extensively with the evalu-
provide certain services, often include descrip- be tal<en into account when they are inter- than program personnel and other such stal<e- ator to fashion a vilid and useful rendition of
tive infonnation about clients, services, and preted. holders. Nonetheless, representatives of the program theory. The policy group, on the other
program objectives. Intemal documents in- target population will generally have a unique hand, is composed of upper-level adminis-
volving formal commitments, such as mission Interviews. The most important sources of perspective on the program, sometimes one at trators, policymal<ers, and significant stal<e-
orvision statements, manuals ofoperating pro- infonnation describing a program and contrib- variance from that of other infonnants. They holder representatives in decision-mal<ing roles
cedures, contracts with clients, job descrip- utiog to the articulation of program theory are can be especially helpful to the process of for- whose feedback and endorsement are impor-
tions, and other such material, my also be those persons with firsthand knowledge and mulatiog the service utilization plan by report- tant to the acceptability and credibility of the
infonnative. experience of the program. Generally, the best ing on the nature of their contact and access to theory description. This group is convened
ln addition, many programs prepare and way for the evaluator to internet with these the program. As the recipients of a program's periodically for briefing and discussion as the
distribute promotional material This may in- infonnants is through face-to-face discussion, attempts to bring about change, their stories work progresses.
clude flyers, brochures, newsletters, reports of individually ar in small groups. Whereas writ- are also frequently illUIIlinating for purposes of
program accomplishments, and listings in ser- ten surveys and questions might be useful for articulating the program's impact theory. Observation. Although program documents
vice and agency directories. Finally, any avail- some limited purposes, that approach lacl<s the Finally, of course, the evaluator should ob- and stakeholder interviews will usually prove
able management reports, organization charts, flexibility to tailor the line of discussion to the tain input from representatives of major stal<e- very informative, an evaluator is wise not to
flowcharts, program monitoring documents, ar expertise of the individual, probe and explore holders outside of the circle of persons directly rely exclusively on them for describing the pro-
evaluation studies relatiog to the program are issues in depth, and engage the infonnant in involved with the program. This might include gram and the theory it embodies. Documents
good prospects for providing important descrip- careful reflection. infonnants from other agencies, advocacy and infonnants both have inherent limitations
tive infonnation. Even though ali these various Central among the infonnants whose in- groups, co=urrity leaders, professional resulting from their partisan role in relation-
sources of written. information are not neces- put is needed to properly depict program theory groups, and the like who have some interest in ship to the program and the particular purposes
sarily available ar useful to the evaluato~ thor- are the various members of the program staff the program and some awareness of its pur- and vantage points of their accounts. Based on
ough canvassing will generally tum up enough in positions to know about key aspects of the poses and activities. Many infonnants from experience with evaluability assessment,
documentation to permit creation of a first program. Program managers and adminis- these groups will not possess detailed lmowl- Wholey (1994) reco=ends that evaluators
approximation to a program theory descrip- trators, of course, are especially relevant be- edge of the program but may, nonetheless, pro- "explore program reility" firsthand through
tion. cause of their positions of oversight and respon- vide useful insights about the perception of the site visits and direct observation. ln particul~
Although program documents can be infor- sibility. Line personnel should not be neglected program's purposes in the infonned co=u- evaluators should observe what they can of the
ma tive for the evaluator attempting to describe when selectiog infonnants, howevei: They nity, its relationships with other agencies and program resources and routine operations so
the program and articulate the components of often have the most detailed firsthand knowl- programs, and how it relates to social condi- that they may make independent input to the
program theory, their limitations must be kept edge of how things actually work and generally tions and social needs recognized in that com- fonnulation of program theory and so that they
in mind. Ali program documents are prepared are the personnel most closely in contact with munity. can be assured that the input from other
for some purpose, and that purpose will rarely the target population the program serves. Because theory description is worked out sources is reilistic with relation to the program
be to present program theory in a vilid and Unique vantage points are also held by the chiefly with stal<eholders, evaluators experi- capability.
straightforward mannei: The most descriptive program sponsors, funders, and policymakers, enced with evaluability assessment recom- The articulation of program theory neces-
documents are usually written to persuade who often have a broaderview of the objectives mend that one ar more stal<eholder groups be sarily and appropriately represents the program
some outside partyto support the program and, and goals of a program and its significance to organized to facilitate interaction for this pur- as intended more than the program as it actu-
naturally, have a self-serving bias. Others may the co=urrity than program personnel. pose (e.g., Smith, 1989; Wholey, 1994). For ally is. Program managers and policymal<ers
describe an official ar historical view of the Critical sources of infonnation related to example, Wholey (1994) reports that in many will generally think of the idealized programas
program that does not coincide well with the the various components of program theory are evaluability assessments it has proven useful the "real" one with various shortfalls from that
program reility as it exists at the time of an members of the target population the program to organize two groups, a policy group and a ideal as glitches that do not represent what the
166 EVALUATION Expressing and Assessing Program Theory 167

program is really about. Those further away evaluator should provide that feedback to the
from the day-to-day operations, on the other pertioent stakeholders.
hand, may be unaware of such shortfalls and
will naturally describe what they presume the Collating information from different sources.
Theoretica/ sensitivity is the ability to sensitivity are creative and imaginative in
program to be even if in actuality it does not Evaluators typically handle the ioformation recognize what is important in data and to give character, it is important to keep a balance
quite live up to that image. gleaned from program documents, interviews, it meaning. lt helps to formulate theory that is between that which is created by the researcher
Some discrepancy between program theory and observations with some form of informal faithful to lhe reality oi the phenomena under and the real. You can doso by (a) asking, what is
and program reality is therefore natural. In- content analysis. Surnrnaries or transcripts are study. Theoretical sensitivity has two sources. rea/Jy going on here? (b) maintaining an attitude
deed, examination of the nature and magni- made from the source material and then re- First, it comes from being well grounded in the of skepticism toward any categories or
tude of that discrepancy is the task of process viewed so that ambiguous or incomplete por- technical literature as well as from professional hypotheses brought to or arising early in the
or implementation evaluation, as discussed in tions can be clarified with appropriate iofor- and personal experience. You bring this complex research, and validating them repeatedly with the
the next chapteL Howeve~ if the discrepancy is mants. The evaluator next extracts the knowledge into the research situation. However, data themselves; and (e) by following the data
so great that the program theory describes ac- pertioent ioformation from each document in theoretical sensitivity is also acquired during the collection and analytic procedures as discussed
tivities and accomplishments that the program the form of thematic notes or excerpts and sorts research process through continuai interactions in this book. Good science (good theory) is
with the data-through your co/Jection and produced through this interplay of creativeness
clearly cannot attain given its actual nature and them according to the aspect of the program to
analyses of the data. While many of the analytic and the skil/s acquired through training.
resources, then the theory is overblown and which they relate, such as goals, services, cli-
techniques that one uses to develop theoretical
cannot be realistically held up as a depiction of ents, personnel, program components, and re-
what is supposed to happen in the program sources. The ioformation in each category is
SOURCE: Quoted, with per~ission, from Anselm Strauss and Juliet Corbin, Basics of Qua/itarive Research: Grounded
context. For instance, suppose that a job train- then used, along with other available informa- Theory Procedures and Techn1ques (Newbuiy Park, CA: Sage, 1990), pp. 46-47.
ing program's service utilization plan calls for tion, to depict program theory in whatever
monthly contacts between each client and a representational form is preferred, for instance,
case manageL li the program resources are a chart or graphic. Discussions of the general
insullicient to support case managers, and nature of this process can be found in Boyatzis
nane are employed by the program, this part of (1998), Miles and Huberman (1994), Patton ofthe program; these are necessarily anintegral How could you tell if those results actually
the theory is fanciful and should be revised to (1990), and Strauss and Corbin (1990). Exhibit part of program theory, especially impact the- occurred?" This approach attempts to keep the
more realistically depict what the program 5-D reveals somethiog of what the evaluator ory. The goals and objectives that must be discussion concrete and specific rather than
might actually be able to accomplish. must bring to this process. represented in program theory, howeve~ are not abstract and general as is typically the case if
The purpose of supplementing the ac- necessarily the sarne as those identified in pro- program goals are asked about directly (see
counts from program documents and stake- gram mission statements ar in response to Exhibit 5-E).
Tbpics for Attention During Document
holders with direct observation, therefore, is questions about goals to stalceholders. To be Given a set of relatively concrete and real-
Review, Interviews, and Observations
not for the evaluator to verify that the program meaningful withio an evaluation context, pro- istic goal statements, they must be integrated
actually lives up to the intentions represented Above, we reviewed the common sources gram goals must identify a state of affairs that iota the descriptions of program theory in a
in its various theory components, but to ascer- of ioformation useful to the task of articulating could realistically be attained as a result of meaningful way. Withio the context of the
tain that those intentions are generally realis- program theory but gave little attention to what program actions; that is, there must be some division of program theory into impact theory
tic. When the program reality falls well short of ioformation the evaluator might attempt to reasonable connection between what the pro- and process theory used here, the first distinc-
the design envisioned by the key stakeholders, obtain from those sources. ln this section, we gram does and what it intends to accomplish. tion to be made is between goals appropriate to
and that shortfall is readily apparent, there is tum attention to that matteL Smith (1989) suggests that the evaluator use a these different theory components. The ulti-
little poiot in pursuing assessment of the the- line of questioning that does not ask about mate goal of any social program should always
ory or the details of how well the program Program goals and obiectives. Perhaps the most goals directly but, rathe~ about consequences. be a specifiable improvement in the social con-
implements the theory. Program redesign or important matter to be deterrnioed from pro- For instance, in a review of major program ditions the program addresses. Thus, the goals
reconceptualization is more ia arde~ and the gram sources relates to the goals and objectives activities, the evaluator might ask about each, and objectives that describe the outcome of the
"Why do it? What are the expected results? change process the program aims to bring about
EVALUATION Expresslng and Assesslng Program Theory 169
168

tion plan, for instance, could be "all persons distinct program component, its functions, and
released from mental institutions are contacted lhe particular activities and operations associ-
and offered services" or "80% of lhe clients are ated wilh lhose functions. For this pUipose, it
retained in service for lhe full ten-week dura- is usually most instructive to view lhe program
The Illinois Cooperative Extension Service's Tasks and activities: tion." These statements describe program ac- as a process ralher lhan as an entity and de-
teleconferencing network (TeleNet) was initiated complisbments related to service delivery in sctibe it wilh verbs ralher lhan nouns (Weick,
to provide information and educational What tasks do you perform with the
program?
terms ofwhat happens to members of lhe target 1982). An organization chart reveals little
assistance to public officials 1 civic organizations,
How does each of these tasks cantribute to population, but do not address lhe benefits of about how a program actually operates to
planning groups, and the general public on
locally identified issues such as county jails, accomplishing the objectives? those services for lhose persons. Similarly, lhe achieve its objectives; lhat information appears
collective bargaining, and financial and person- What problems, if any, do you face in goals and objectives related to lhe program's in a description of what lhe program does.
nel management. ln an evaluability assessment, performing these tasksl organizational plan would deal wilh perform- Thus, an essential part of describing program
Midge Smith developed the following interview To what extent do you feel you reach ing certain program functions 1 for instance, "to lheory is to identify all lhe important program
questions for program staff to help formulate the target audience? prepare curricular materials" or "to offer liter- functions lhat must be performed for lhe pro-
program goals and objectives and the program acy classes four times a week" gram to operate as intendecl.
activities and resources associated with them: Resources: One other consideration is important for Program functions include such operations
What resources are used ar are available lhe evaluator who is attempting to ascertain as "assess client need, 1111 complete intal<e," "as-
Coais and objectives: at the local levei to carry out lhe different lhe various program goals and objectives and sign case manage:r;." "recruit referra! agencies,"
program activities? organize lhem into a description of program "train field workers," and lhe like. Viewed from
What do you think theTeleNet programs are
trying to accomplishl How adequa te are these resources? lheory. The inquiry should attend to possible lhe clients' perspective as part of a description
What changes ar differences, if any, is this side effects and unintended outcomes lhat may of a service utilization plan, these functions
program making with regard to participants, Performance indicators: be important for understanding lhe programas appear in such forms as "receive referral for
county advisors, the community, and the well as to lhe intended effects. Thus, a program services," "contacted by case manage:r;" 11par-
What are some of the indicators of success
county?
that the evaluation might try and measure? "accomplisbment" may be to have some im- ticipate in group counseling sessions, /1 and so
What negative effects, if any, might the When could they be measured? pact lhat was not desired and may not be forth. Each such function, whelher represented
program have orbe having? (lf some Are there any questions or concerns about
are mentioned, ask: What do you
desirable. A mandatory job training program from lhe program or lhe client perspective, will
the program operation or results that you
think could/should be done to avoid for women on welfare, for instance, may have consist of various specific activities and will be
would like to see addressed by an
these negative effects?)
evaluation?
lhe effect of increasing lhe number of children associated wilh certain program personnel or
in substandard child care arrangements (Ex- components and resources. Full description of
hibit 5-F provides anolher example). Allhough lhe program functions, lherefore, also entails
SQURCE: Adapted from Midge F. Smith, Evaluability Assessmenl: A Practical Approach (Norwell, MA: Kluwer, such unintended effects cannot be said to be some levei of description of lhe constiruent
1989), p. 91. activities and lhe program components and
program goals, lhey follow from program activi-
ties in the sarne way as goal attainment and resources lhat support lhose activities (an ex-
should be represented in program lheory as ample appears in Exhibit 5-G).
in social conditions relate to program impact lhose related to program activities and service possible outcomes.
theory. Also associated wilh impact lheory are delivery. These, in tum, are relevant to program Logic ar sequence linking program functions,
any intermediate objectives lhat represent process theory. For instance, 11 to provide case Program functions, components, and activities. activities, and components. A critical aspect of
steps along lhe palhway leading from program management 11 is a service objective but not an Program process lheory mainly represents dis- program lheory is how lhe various steps and
services, on one end, to the improved social outcome goal because it desc