A Pragmatic Model for a Comprehensive Public School Research and Evaluation System

Author(s): William J. Webster and Robert L. Mendro

Source: The Journal of Educational Research, Vol. 69, No. 4 (Dec., 1975), pp. 156-159
Published by: Taylor & Francis, Ltd.
Stable URL: http://www.jstor.org/stable/27536859
Accessed: 07-03-2017 21:16 UTC

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms

Taylor & Francis, Ltd. is collaborating with JSTOR to digitize, preserve and extend access to The Journal
of Educational Research

This content downloaded from 132.174.250.76 on Tue, 07 Mar 2017 21:16:27 UTC
All use subject to http://about.jstor.org/terms
 156

A Pragmatic Model for a Comprehensive

Public School Research and
Evaluation System
WILLIAM J. WEBSTER
ROBERT L. MENDRO
Dallas Independent School District

ABSTRACT

A model for research and evaluation in a large urban school district is presented. The relationship between research, evaluation,
planning, and development is discussed in light of major functions including context, input, process, and product evaluation, as well
as applied and basic research. The role of these functions in supplying the information base necessary for educational planning and
decision-making is highlighted.

IN RECENT YEARS there has been increased emphasis
on evaluation in the public schools. Many districts have es
tablished comprehensive systems of performance monitor
ing based on the continuous assessment of program and
project outcomes (1, 5, 6,14). Typically, they are struc
tured so as to provide information to decision-makers, but
are largely limited to evaluation rather than research func
tions (4). Consequently, the information provided by these
systems is of limited utility.
The restricted scope of these systems undoubtedly is
due, in part, to the strained finances of many school sys
tems and to the critical lack of competent personnel
trained to conduct and implement systematic research pro
grams in the public schools. However, a significant factor
influencing the implementation of such limited systems is
the lack of information detailing the integration of evalua
tion and research operations in a real world environment.
More directly, the fundamental works underlying the cur
rent renaissance of educational evaluation have failed to
emphasize the critical role of research in objective decision
making (7, 9, 10, 11).
Evaluation alone, regardless of its comprehensiveness,
cannot supply sufficient information for educational de
cision-making. Rather, systematic evaluation programs
must be carefully integrated with applied and basic re
search programs to provide the data essential for educa
tional renewal.
This paper provides a model for a comprehensive evalu
ation and research system highlighting the interrelation
ships between evaluation and research and indicating the
role of both in the process of developing and implementing
educational programs. The CIPP model, proposed by Stuf
flebeam et al. (11), was chosen as a reasonable general pur
pose evaluation model for purposes of modification into
the more comprehensive combined model.
Figure 1 presents a flowchart outlining the essential de
tails of the revised system. The flowchart depicts an opera
tional research and evaluation system which begins with
yearly context evaluation, moves through input, process,
and interim product evaluation stages, and completes the
annual cycle with summative product evaluation and ap
plied research information. The process outlined in Figure
1 is discussed below.
A prerequisite to improvement must be a knowledge of
existing performance levels. Thus, the backbone of any re
newal system must be a comprehensive context evaluation
program. Stufflebeam, et ah, define context evaluation as
the provision of baseline information that defines the en
vironment of interest, describes desired and actual condi
tions pertaining to the environment, identifies unmet needs
and unused opportunities, and diagnoses the problems that
prevent needs from being met and opportunities from being
used. An adequate context evaluation system is founded on
a longitudinal data base and provides periodic reports on
such variables as student dropout, attendance, achievement
levels, drug usage, demographic and vocational patterns,
community socioeconomic status and dominant value pat
terns, and teacher academic and demographic characteris
tics. Thus, a context evaluation system provides the basis
for formulating change objectives by identifying needs
and, in some cases, outlining practical constraints in iden
tified problem areas.
Once the context evaluation system has identified needs,
decision-makers must set priorities for those needs and fo
cus on reducing the discrepancy between desired and exist
ing conditions by establishing goals for those needs that
receive highest priority. It is at this point in time that in
put evaluation information is used. Stufflebeam, et al, de
fine input evaluation as the provision of information for
determining methods of resource utilization for accom
plishing program goals. In a functioning evaluation system,
there are four sources of input information:
1. previous product evaluation information
2. basic research information
3. applied research information
4. non-research and evaluation information.
Product evaluation information concerns the extent to
which specific project or program goals are achieved. When

This content downloaded from 132.174.250.76 on Tue, 07 Mar 2017 21:16:27 UTC
All use subject to http://about.jstor.org/terms
 WEBSTER-MENDRO 157

product evaluation information is available, relative to a The development of this evaluation design must necessarily
given program with goals similar to those identified in re involve continuous interaction between program personnel
sponse to context evaluation information, that information and the evaluator in order ultimately to produce maximally
provides useful input to decision-makers in determining effective information. Obviously, the evaluator must be in
the probability that the program would reduce the identi dependent of program management to ensure the optimum
fied discrepancy between desired and existing conditions. objectivity of evaluation results.
Basic research information pertains to information Once the program implementation phase is entered, the
about fundamental relationships that affect student learn evaluator's role becomes one of providing continuous for
ing. Before making a decision to implement a given pro mative evaluation reports relative to program implementa
gram, decision-makers should be apprised of the extent to tion. These reports fall primarily into two categories, pro
which that program is or is not consistent with the princi cess evaluation and interim product evaluation. Process
ples established by basic research in learning. evaluation has three major objectives: (1) the detection or
Applied research information concerns the interaction prediction of defects in procedural design or its implemen
between student characteristics, teacher characteristics, tation during program implementation stages, (2) the pro
and instructional systems. Applied research differs from vision of information for programmed decisions, and (3)
basic research in that the information provided is more the maintenance of a record of the implementation pro
closely related to specific decisions in an applied educa cedure as it occurs (11). Thus, process evaluation informa
tional setting. Decision makers need information relative tion keeps program management informed of the extent
to the types of students (e.g., high anxiety versus low anxi to which program implementation conforms to specifica
ety) that function best in given instructional systems im tions and from an evaluation standpoint, guards against the
plemented by specific types of teachers. Applied research evaluation of a fictitious event.
must be implemented in the public schools by public Interim product evaluation provides periodic feedback
school research departments. Basic research should probably to program management relative to the attainment of spe
be implemented in the universities in conjunction with the cific sub-objectives during the implementation phase. Thus,
public schools, because many local boards of education are process and interim product evaluation reports inform pro
probably too impatient for results and are unlikely to wait gram management as to implementation and goal attain
for a long-term, systematic, longitudinal basic research pro ment levels while program adjustments are still feasible.
gram to pay off. Upon completion of a given cycle of program implemen
Finally, non-research and evaluation information must tation, a summative product evaluation report is prepared.
enter into most educational decisions. Such information This report generally addresses three areas of concern:
as capabilities of staff members, costs, political feasibility 1. the extent to which program objectives were achieved
of program implementation, and existing facilities must relative to some specific set of criteria
be taken into account.
2. the extent to which system objectives were achieved
After the collection of relevant input information, feed
relative to alternative instructional strategies
ing a pre?minary program planning stage, decision makers
determine whether or not sufficient resources are available 3. the cost-effectiveness of the program relative to alter
to make the desired instructional changes. Quite often suf native instructional strategies.
ficient resources are not available and some compromise is It should be obvious that information relative to these three
necessary. In most cases, the lack of resources is not lim areas of concern must be interpreted in light of process and
ited to the realm of cost and political feasibility, but rather interim product evaluation information. Without informa
stems from an insufficient base of research information.
tion about program implementation, product evaluation
Thus, educators are often in the position of having suffi information is of little use. A detailed discussion of an oper
cient material resources but insufficient information re
ational project evaluation process can be found in Webster
sources.
(12).
If sufficient material resources are not available, the sys Most evaluation systems stop at the provision of product
tem may have to exist for some period of time in a state ofevaluation information. These data generally bear upon the
enlightened persistence. Periodic context evaluation will performance of different groups of students under varying
continue to highlight the extent of discrepancy between treatment configurations. Unfortunately, as a result, most
that which is desired and that which exists. If the problem product evaluation reports have generally focused on the
results from insufficient information resources, programs search for single best treatments for all learners, i.e., main
are often implemented without sufficient support data and effects. In order to provide needed information for educa
an information base is built through a series of systematic tional decision-makers, applied research studies involving
evaluation and applied research studies. the systematic investigation of aptitude-treatment and
If sufficient material and information resources are availtrait-trait interactions must be undertaken. Such studies
able, or if sufficient material and minimal information re would be expected to provide important information rela
sources are available, the program planning phase is entered.tive to replicable relationships between student, teacher,
The evaluator's role in program planning involves making and program characteristics (2, 3, 8, 13).
all relevant, available input information available to pro Once context, input, process, and product evaluation in
gram planners and ensuring that stated program objectives formation, as well as applied research data, are available,
are measurable. Out of the program planning sessions, the non-research and evaluation information once more is
evaluator develops a detailed program evaluation design brought to bear on the decision-making process. It would
specifying the criteria by which the program will be judged.be naive to expect educational decisions to be made purely

This content downloaded from 132.174.250.76 on Tue, 07 Mar 2017 21:16:27 UTC
All use subject to http://about.jstor.org/terms
 158 THE JOURNAL OF EDUCATIONAL RESEARCH

Figure 1.-Flowchart for an Integrated Research and Evaluation System

Interim
Product Process
Enlightened ?^
Persistence ' Evaluation Evaluation

Operational
Yearly Preliminary ! Program Program
Context w*\ Need? )-^^ Program ^[ Resources > Planning Implementation
Evaluation Planning

t i:
Input
Program
Completed

Product
Previous Basic Applied Evaluation
Product Research Research Non - R&E
Evaluation Info. Info. Info.

Li. . u Applied
Research

Expand
Additional
Context
Evaluation
.-'Fate V,
of
^v Program, '' ^i Non - R&E ?
Info. |

Discontinue

Research and Evaluation Function

Non-Research and Evaluation Function

on the basis of research and evaluation data. Once again, inadditional context evaluation information must be exam
formation such as the absolute program costs, capabilities ined to determine if similar needs exist in other settings.
of program staff members, political feasibility of program If such needs are demonstrated, then the program may be
implementation, and existing facilities and resources, must expanded to other settings and the program planning stage
be considered by decision makers in rendering a final entered to extend the program implementation. Other set
decision. tings eligible for program expansion may include entire
In determining the fate of a given program, four primary schools or specific sub-populations (e.g., highly motivated
choices are available to decision makers. First, they can students) as indicated by applied research data. If such
choose to continue the program in its current setting. If needs do not exist, the program is continued with the
this alternative is chosen, the summative product evaluation original target population or a reduced target population
report and the applied research data become the context based on the results of the applied research studies. The
evaluation information for the next implementation phase extent of continued evaluation to be implemented under
and program implementation commences. This alternative either the expansion or continuation alternative is deter
generally occurs when decisions are to be made on the basis mined by decision-makers under advisement from evalua
of longitudinal studies, i.e., where it is expected that results tion personnel.
will not be in evidence after a relatively short implementa A fourth alternative involves program revision. Much
tion period. program revision should be accomplished on the basis of
A second alternative is to discontinue the program. This process and interim product evaluation reports. Often,
is usually done after product evaluation studies demon however, summative product evaluation and applied re
strate the failure of the program to meet its objectives or search reports highlight weaknesses in portions of programs
in those cases where the program is simply not cost-effec that would otherwise appear to be functional. In this in
tive. (Failure to meet objectives is often a necessary but stance, the summative product evaluation and applied re
not sufficient condition for program discontinuation.) search reports become the context evaluation information
Once a program is discontinued, the system returns to con for the next program planning cycle.
text evaluation and once again applies the needs assessment Figure 1 outlined a model for an operational research
and orientation phases. and evaluation system. However, data, no matter how com
If the product evaluation and applied research informa prehensive, are of limited use if not disseminated to deci
tion are favorable, and it is practically and politically feasi sion-makers. Table 1 outlines the information requirements
ble, the program may be expanded to serve additional stu of decision-makers at different management levels.
dents. Prior to making the decision to expand the program,

This content downloaded from 132.174.250.76 on Tue, 07 Mar 2017 21:16:27 UTC
All use subject to http://about.jstor.org/terms
 WEBSTER-MENDRO 159
Table 1.-Information Requirements of Various Management
Levels that Should be Met Through Systematic Research and
Evaluation Programs
Management Levels Information Requirements
Board of Education (public) Product evaluation information (gen
erally summative) on the effects of
district projects and programs.
Context evaluation information on
the general state of the educational
environment.
Upper Management Product evaluation information (gen
erally summative) on the effects of
district projects and programs.
Context evaluation on the general
state of the educational environment.
Basic research information (mostly
literature reviews) on selected funda
mental relationships in learning.
Input evaluation information rela
tive to alternative strategies for meet
ing specific needs.
Applied research information on the
interactions between student charac
teristics, teacher characteristics, and
program success in all district pro
grams.
Project Management Product evaluation information of a
formative and summative nature on
the effects of specific projects.
Context evaluation information on
the baseline state of the project
environment.
Process evaluation information on a
continuing basis relative to the ex
tent of program implementation.
Applied research information on the
interaction between student charac
teristics, teacher characteristics, and
program success.
Basic research information on select
ed fundamental relationships in
learning.
Building Management Product evaluation information (gen
erally summative) on the effects of
district projects and programs.
Context evaluation information on
the general state of building teachers
and students and the sponsoring
community.
Input evaluation information relative
to alternative strategies for meeting
specific needs.
Formative and summative product
evaluation and applied research infor
mation on specific projects imple
mented in the building.
Teachers Product evaluation information of a
formative and summative nature on
the effects of specific programs in
which the teacher is involved.
Context evaluation information rela
tive to student entry level.
Process evaluation information on
degree of program implementation
This content downloaded from 132.174.250.76 on Tue, 07 Mar 2017 21:16:27 UTC
All use subject to http://about.jstor.org/terms
for specific programs in which the
teacher is involved.
Applied research information rela
tive to the types of instructional
strategies that are most effective with
different learner characteristics and
specific validated instruments and
instructions for their use in student
diagnosis.
Parents/Students Feedback on individual student
performance.
The establishment of effective research and evaluation
systems in the public schools is essential for educational re
newal. To assume that public school systems should be pas
sive recipients of externally developed products, often un
accompanied by sufficient empirical evidence of success or
accompanied by empirical support based on restricted stu
dent or teacher populations, is ludicrous. Often adjustments
must be made in externally developed programs to meet
local requirements, or, where externally developed programs
do not exist, strategies must be locally developed for meet
ing specific needs. Evaluation, in the absence of applied and
basic research data, is not capable of providing the necessary
information for meeting the needs of diverse student
populations.
REFERENCES
1. Barker, L. W., "Research and Evaluation in the Louisville Pub
He Schools," Journal of Research and Development in Educa
tion, 5:79-97",1972.
2. Bracht, G. H., "The Relationship of Treatment Tasks, Person
ological Variables, and Dependent Variables to Aptitude
Treatment Interactions," unpublished doctoral dissertation,
University of Colorado, 1969.
3. Cronbach, L. J.; Snow, R., Individual Differences in Learning
Ability as a Function of Instructional Variables, Final Report,
U.S. Office of Education (Contract No. OEC 4-6-061269
1217), Stanford University School of Education, March, 1969.
4. Dziuban, C. D.; Moser, R. P., "Some Functional Dimensions of
Large City Public School Research," Journal of Educational
Research, 66:471-478, 1973.
5. Jacobs, J., "A Model for Program Development and Evaluation,"
Theory into Practice, 13:15-21,1974.
6. Merriman, H. O., "DOPI CIPP," paper presented to the Greater
Cities Research Council, Washington, D.C, November, 1969.
7. Pro vus, M., Discrepancy Evaluation, McCutcheon Publishing
Company, Berkeley, California, 1971.
8. Salomon, G. S., "Heuristic Models for the Generation of Apti
tude Treatment Interaction Hypotheses," Review of Educa
tional Research, 42:327-343,1972.
9. Scriven, M., "The Methodology of Evaluation" in Worthen and
Sanders (eds.), Educational Evaluation: Theory and Practice,
Charles Jones Publishing Company, Worthington, Ohio, 1973.
10. Stake, R. E., "The Countenance of Educational Evaluation,"
Teachers College Record, 68:523-540,1967.
11. Stufflebeam, D. L., et al, Educational Evaluation and Decision
Making, Peacock Publishers, Inc., Itasca, Illinois, 1971.
12. Webster, W. J., "The Dallas Approach to Systematic Research
and Evaluation in Development" Abstracts of Research and
Evaluation Reports, 1972-73, Dallas Independent School Dis
trict, August, 1973, pp. 165-199.
13. Webster, W. J.; Mendro, R. L., "The Investigation of Aptitude
Treatment Interactions as an Integral Part of Program Evalua
tion," Journal of Experimental Education, 43: 86-91, Fall 1974.
14. Webster, W. J.; Schuhmacher, C. C, "A Unified Strategy for
System-wide Research and Evaluation," Educational Tech
nology, 13:68-72,1973.