T 2000-2001 E R W C T P: HE Valuation of THE Eading and Riting For Ritical Hinking Roject

THE 2000-2001 EVALUATION OF
THE READING AND WRITING FOR

CRITICAL THINKING PROJECT
SEPTEMBER 5, 2001
SUBMITTED BY:
AMERICAN INSTITUTES FOR RESEARCH
1000 THOMAS JEFFERSON STREET, N.W.
SUITE 400
WASHINGTON, D.C. 20007
SUBMITTED TO:
OPEN SOCIETY INSTITUTE
400 WEST 59TH STREET
NEW YORK, NY 10019
2000-2001 RWCT Evaluation
TABLE OF CONTENTS
I. Introduction ............................................................................................. 1
II. The RWCT Model for Professional Development .................................. 2
III. Study Design and Data Collection .......................................................... 2

Conceptual Model ................................................................................... 3
Instrumentation........................................................................................ 4
Measuring Critical Thinking .......................................................... 5
Measuring Teaching Practices ....................................................... 6
Sampling.................................................................................................. 6
Data Collection........................................................................................ 9
Study Limitations .................................................................................... 10
IV. Evaluation Results ................................................................................... 11

Question 1 ...................................................................................... 11
Summary of Findings ........................................................ 11
Detailed Findings............................................................... 12
Question 2 ...................................................................................... 17
Question 3 ...................................................................................... 25
V. Conclusion ........................................................................................... 33
Appendix A
Appendix B
Appendix C
Appendix D
American Institutes for Research

LIST OF TABLES
1 Sampling Frame ...................................................................................... 7

2 Comparisons of RWCT and Non-RWCT Teachers ................................ 8
3 Comparisons of RWCT and Control-Group Pupils................................. 8
4 Differences in Critical Thinking Scores Between RWCT and
Control Group Pupils ........................................................................ 12
5 Differences in Critical Thinking Scores Between RWCT Pupils
and Pupils Whose Teachers Did Not Participate in
Professional Development Activities in the Past Three Years.......... 13
6 Differences in Critical Thinking Scores Between RWCT Pupils
and Pupils Whose Teachers Participated in Other Content-
or Instruction-Based Professional Development Activities
in the Past Three Years ..................................................................... 14
7 Proportional Reduction in Critical Thinking Score Differences ............. 15
8 Proportional Reductions in Critical Thinking Score Differences
Related to the Integration of Critical Thinking
in Classroom Practices, by Country .................................................. 16
9 Differences between RWCT and Control-Group Teachers in
Scores Related to the Integration of Critical Thinking Principles..... 19
10 Differences between RWCT Teachers and Control-Group Teachers
Who Participated in Other Content- or Instruction-Based
Professional Development Activities in the Past Three Years
in Scores Related to Integrating Control Thinking Principles
Into Teaching Practices ..................................................................... 20
11 Differences between RWCT and Control-Group Classes in
Classroom Communication Patterns ................................................. 22
12 Differences in Authentic Pedagogy and Other Classroom Observation
Scales between RWCT and Control-Group Teachers....................... 23
13 Differences in Observed Classroom Activities in RWCT and
Control-Group Classrooms ............................................................... 24
14 Differences between RWCT and Control-Group Pupils (Cohort 1)
in What They Enjoy Most about the Work They Do in Class .......... 27
15 Differences between RWCT and Control-Group Pupils (Cohort 2)
in What They Enjoy Most about the Course That Data
Collectors Observed .......................................................................... 27
16 Differences between RWCT and Control-Group Teachers in
Views About Teaching and Learning................................................ 28
17 Views among RWCT Teachers about RWCT......................................... 29
18 Opinions among RWCT Teachers about Changes in Pupil
Behavior Since Teachers Began Participating in RWCT.................. 29
19 Most Frequent RWCT Teachers’ Descriptions about Themselves
as Teachers Before They Began to Participate in RWCT ................. 30
20 Most Frequent RWCT Teachers’ Descriptions about
as Teachers After They Began to Participate in RWCT ................... 30

LIST OF TABLES (CONTINUED)
21 Most Frequently Reported Changes in How Teachers Prepare for

Lessons Since Beginning their Participation in RWCT .................... 31
22 RWCT Teachers’ Opinions about How RWCT Instructional
Practices Differ from Previous Instructional Practices ..................... 31
23 RWCT Teachers’ Opinions about the Most Positive Effects
of Applying RWCT Strategies in Teaching ...................................... 32
24 RWCT Teachers’ Opinions about the Most Negative Effects
of Applying RWCT Strategies in Teaching ...................................... 32

LIST OF EXHIBITS
1 Project Timeline ...................................................................................... 3
2 Conceptual Model of the Professional Development Process for

RWCT Teachers and Pupils .............................................................. 4
3 Critical Thinking among RWCT Pupils, based on Years

Teacher Participation in RWCT..................................................... 16
4 Integration Scores among RWCT Teachers, Based on Years

of Participation in RWCT .............................................................. 21

THE 2000-2001 EVALUATION OF THE READING

AND WRITING FOR CRITICAL THINKING PROJECT
I. INTRODUCTION
In April 2000, the Open Society Institute (OSI) commissioned the American Institutes for Research
(AIR) to conduct an evaluation of the Reading and Writing for Critical Thinking (RWCT) project.
AIR was asked to examine three research questions:
1. To what extent do pupils whose teachers participate in RWCT1 have higher critical thinking
skills than pupils in a non-RWCT control group?
2. To what extent do RWCT teachers maintain the fidelity of the RWCT model in their teaching
practices?
3. To what extent do RWCT teachers’ and pupils’ attitudes about teaching and learning differ
from those of teachers and pupils in control groups?
The evaluation sought differences among pupils and teachers in RWCT classrooms and in control-
group classrooms. Findings from the evaluation indicate that participation in RWCT is positively
and significantly associated with each of these aspects of teaching and learning:
Pupil critical thinking skills:
• RWCT pupils on the whole demonstrate higher critical thinking skills than pupils in non-
RWCT control groups, differences that are largely attributable to classroom communication
patterns and teachers’ integration of critical thinking principles into teaching practices;
Teaching practices:
• RWCT teachers maintain the fidelity of the RWCT model by spending less classroom time
than their peers lecturing, engaging pupils more frequently in small group activities, and
encouraging more pupil-led classroom discussions; and
• RWCT teachers spend more time than their peers promoting “authentic pedagogy,” including
higher-order thinking, substantive conversation, connections to the world outside of the
classroom, and an appreciation for multiple approaches to problem solving.
Teacher and pupil attitudes:
• On affective scales, RWCT teachers strongly support the project and are more likely than
control-group teachers to enjoy their jobs; and
• RWCT pupils are more likely than pupils in control groups to enjoy classroom activities in
which they can be active participants in the learning process.
This report describes these findings in detail. It begins with a description of the RWCT project and
the design of this study and data collection procedures, and it continues with a discussion of findings
as they relate to the evaluation questions.
1
These pupils are referred to throughout this report as RWCT pupils and their teachers as RWCT teachers.
1 American Institutes for Research

II. The RWCT Model of Professional Development

RWCT is a professional development project for educators, the purpose of which is to provide
participants with strategies for interactive methods of teaching that prepare pupils for citizenship in
open societies. In its first three years, 1997-2000, more than 12,200 teachers completed RWCT
training workshops in 24 countries throughout Eastern Europe and Central Asia, and many thousands
more participated in short-term RWCT seminars and mini-courses. When a country first join RWCT,
four educators who have volunteered through the International Reading Association travel to the
host country to train 20-40 teachers to use RWCT strategies. These volunteers offer a series of four
workshops over the course of 12-15 months. During that time, participating teachers practice the
curriculum and adapt RWCT strategies based on individual circumstances. Between workshops,
RWCT participants meet monthly with colleagues to discuss progress in mastering RWCT strategies.
They also receive feedback from peers and RWCT volunteers who observe their classroom teaching.
After completing the entire RWCT course, first-year participants are expected to become trainers for
future generations of RWCT teachers.
RWCT training focuses on process rather than content. It is designed to be applicable to educators at
every grade level, from primary school through post-secondary education, and is not limited to
specific subject areas. Teachers learn strategies to help pupils use self-reflection to solve problems
and to engage actively in the educational process. They are then supposed to incorporate these
strategies into their instructional practices—encouraging pupils to examine the implications of their
ideas, exposing those ideas to polite skepticism, balancing ideas against opposing points of view,
constructing supporting belief systems to substantiate the ideas, and taking a stand based on those
structures.
III. STUDY DESIGN AND DATA COLLECTION

The RWCT evaluation took place between April 2000 and May 2001. It was conceived as a
collaboration between the American Institutes for Research, RWCT country coordinators, and in-
country data collection teams.2 Responsibilities were divided as follows:
• AIR was responsible for all aspects of study design and administration, instrument development,
selecting data collectors, data-collection training, data analysis, and reporting;
• Country coordinators were responsible for identifying potential data collectors and populations of
appropriate RWCT and control-group teachers for the study, translating and back-translating
survey instruments, and piloting survey instruments; and
• In-country data collectors were then responsible for collecting survey and classroom observation
data and scoring open-ended survey items.
The timeline for conducting the evaluation is shown as Exhibit 1.
2
A complete list of advisors and international project staff is included in Appendix D.

EXHIBIT 1: PROJECT TIMELINE
• Refining the conceptual framework and developing an instrumentation plan and sampling
frame (April 2000);
• Identifying and selecting in-country data collectors (April – May 2000);
• Developing and refining survey instruments, classroom observation protocols, and pupil
critical thinking activities (April – October 2000);
• Piloting draft survey instruments in U.S. schools (May 2000);
• Translating and back-translating survey instruments in five languages (July – December
2000);
• Piloting draft survey instruments in evaluation countries (September – October 2000);
• Training in-country staff in data collection and scoring procedures (February 2001);
• Piloting and refining data collection procedures (February 2001);
• Conducting full-scale data collection (February 2001);
• Conducting quality control procedures in the United States and participating countries
(February – March 2001); and
• Analyzing data and producing the evaluation report (March – May 2001).
Conceptual Model
Guiding the evaluation is a conceptual model of how professional development activities for teachers
can lead to changes in pupil outcomes.3 According to the model, which is illustrated in Exhibit 2,
professional development directly influences teachers’ knowledge and skills. Teachers change their
attitudes about teaching and learning and alter their teaching behaviors based on their reaction to the
professional development experiences. They may also change their underlying beliefs about teaching
and learning.
Ultimately, changes in teachers’ instructional practices affect pupils’ attitudes, behavior, and
psychological functioning. These changes can also influence pupils’ achievement, knowledge, and
skills. Finally, changes in one teacher’s instructional practices can affect programmatic structures at
the school level. School-level changes, such as changes in teaching requirements, pupil assessments,
and enhanced schoolwide professional development opportunities, increase the likelihood that the
benefits of instructional change will be institutionalized for teachers and pupils in the long run.
3
Kutner, M. et al. (1997). Evaluating Professional Development: A Framework for Adult Education. Washington,
DC: U.S. Department of Education, Division of Adult Education and Literacy.

EXHIBIT 2: CONCEPTUAL MODEL OF THE PROFESSIONAL

DEVELOPMENT PROCESS FOR RWCT TEACHERS AND PUPILS
Professional Development A
(RWCT Workshops)
D
Teachers: Reactions/Attitudes B Programs: Instructional arrangements
Knowledge and skills Program processes
Behavior Instructional practice
Psychological functioning* Pupil assessment
Learner supports
C
Pupils: Knowledge and skills
(including critical thinking skills)
Behavior
Psychological functioning*
*
Not included in Kutner, et al.
Instrumentation
This evaluation examines the three research questions by measuring the direct link in the conceptual
framework between professional development and teachers; the direct link between teachers and
pupils; and the indirect link between professional development and pupils. This is accomplished
through a set of linked evaluation instruments, including classroom observations and teacher and
pupil surveys, that compare teaching and learning in RWCT and control-group classrooms. The
triangulation of these instruments provides multiple opportunities to test the reliability of information
related to teaching and learning. The scope of the evaluation made it impossible to undertake a
systematic examination of the associations between RWCT and program-level changes and between
program-level changes and pupil outcomes, but these associations are described anecdotally in a
supplemental impact and institutionalization study.
AIR drew on multiple sources to develop the data collection instruments. Among them were surveys
of teaching practices developed for AIR’s evaluation of the federal Eisenhower Professional
Development Program in the United States; classroom observation protocols developed by
Newmann, Secada, and Wehlage to evaluate the alignment of instruction and assessment4; and the
rubrics developed by Crawford and Mathews for use in RWCT classrooms. Instruments developed
by other RWCT participants were also used, and new questions were designed at AIR to measure 1)
pupils’ critical thinking skills; and 2) the extent to which teachers understand and implement the
RWCT strategies that promote critical thinking.
4
Fred Newmann, Walter Secada, and Gary Wehlage, A Guide to Authentic Instruction and Assessment: Vision,
Standards, and Scoring. Madison, WI: University of Wisconsin Center on Organization and Restructuring of
Schools, 1995.

Measuring Critical Thinking
Three open-ended items were developed to measure pupils’ critical thinking skills. The items were
designed to be provocative and engaging for pupils and to be culturally sensitive and cross-nationally
consistent. Pupils in different grades answered two short essay questions during the first 25-30
minutes of their 45-minute surveys. AIR used a definition of critical thinking from the RWCT
guidebooks5 as the basis for the critical thinking construct. Specifically, pupils would be evaluated
on their ability to do the following:
• Form original opinions,

• Choose rationally between competing ideas,
• Solve problems, and
• Debate ideas responsibly.
Two critical thinking questions were developed for each of two pupil surveys. One question
identified a controversial issue (“working alone on a test” for one survey and “immigration/
emigration” for another) and asked pupils to provide arguments on both sides of the issue and make a
decision about which position has more merit. Pupils’ written responses were scored for the number
of reasons provided for agreeing and disagreeing with the topic—a breadth dimension of critical
thinking—and the level of justification that they provided for their answers—a depth dimension of
critical thinking. They also received credit for taking a position on the topic and providing
justifications for that position.
The second question asked pupils to make a business decision about locating a snack kiosk in a small
town. Pupils were again asked to explain why they selected one location and not others and were
scored based on the breadth and depth of their responses. All pupils were asked the same question,
but the expectations for pupil responses differed by grade level. Again, pupils’ responses were
scored on the breadth and depth of their responses. They received credit for the number of reasons
that they provided in support of their positions and the level of justification that they provided for
their answers.
AIR subjected all instrumentation to thorough internal reviews, and RWCT country coordinators
reviewed the questions to ensure that they were culturally sensitive and cross-nationally consistent.
Questions were then field tested in classrooms in the United States and evaluation countries and
refined further. Lastly, scoring rubrics were designed to measure differences in teacher and pupil
outcomes, and prototypical answers at each score level were created to assist in the training of data
collection staff.
Questions developed for pupil surveys are included in Appendix A, and the rubrics developed to
score pupil responses are included in Appendix B.
5
Jeannie L. Steele, Kurtis S. Meredith, and Charles Temple, RWCT Guidebook IV: Further Strategies for Promoting
Critical Thinking, 1998, pp.1-2.

Measuring Teaching Practices
AIR measured the extent to which RWCT teachers employ practices that promote critical thinking
through survey questions to teachers and pupils and with classroom observations. The teacher survey
included an open-ended question about teachers’ integration of critical thinking principles into
teaching practices. This question, adapted from a previous RWCT evaluation, asked teachers to map
out how they would plan and execute a classroom lesson. Teachers received credit on a five-point
scale based on the number of ways that critical thinking dimensions were incorporated into the lesson
plan and the extent to which those dimensions were perceived as a minor or major part of the
classroom activity (see Appendix B). These dimensions were again based on the critical thinking
construct from Volume Four of the RWCT guidebooks6: providing opportunities for pupils to form
original opinions, choose rationally between competing ideas, solve problems, and debate ideas
responsibly. Teachers also received credit for enabling pupils to practice two social dimensions of
critical thinking and active learning:
• Valuing the ability to work cooperatively with others to construct meaning; and
• Appreciating different points of view and recognizing the ways people’s background can
influence their attitudes and perceptions.
Teaching practices were also measured directly through classroom observations and indirectly
through teacher and pupil reports of classroom activities. Observation scales included measures of
teacher-pupil communication patterns, classroom activities, and eight additional scales:
• Activities that promote higher-order thinking;

• Substantive conversation;
• Connections to the world beyond the classroom;
• Organization of information;
• Approaches to solving problems;
• Recognition of different solutions to problems;
• Classroom organization; and
• Teacher wait time after asking questions.
Questions developed for the teacher survey are included in Appendix A, and the rubric developed to
score teacher responses are included in Appendix B.
Sampling
This evaluation was conducted in four target countries: the Czech Republic, Kyrgyzstan, Latvia, and
Macedonia. Countries were selected in collaboration with OSI and RWCT co-directors based on the
following criteria:
• At least two years of teacher groups that had completed the entire set of RWCT courses;
• A single RWCT team that was implementing the project within the country;
• Representation of the geographic diversity of RWCT, including a country in Eastern Europe,
Central Asia, the Baltics, and the Balkans;
6
Ibid.

• Willingness on the part of country coordinators to facilitate the logistical aspects of the
evaluation; and
• No logistical or political problems in the country during the first few years of the project that
would have prevented teachers from experiencing the complete set of RWCT workshops and
follow-up activities.
Within each country, the primary sampling unit was two cohorts of RWCT teachers. Cohort 1
consisted of fourth or fifth year pupils and their teachers; Cohort 2 consisted of seventh or eighth
year pupils and their teachers. These grades provided an opportunity to examine RWCT outcomes for
pupils at two different levels.
Power analyses determined that it was necessary to select at least 36 RWCT teachers and 36 control-
group teachers per country to measure moderate RWCT effects. The sampling frame is provided in
Table 1.
TABLE 1: SAMPLING FRAME
Number of RWCT Number of Control-

Teachers per group Teachers per
Country Country
Cohort 1 18 18
Cohort 2 18 18
Total 36 36
AIR selected RWCT teachers at random from the population of eligible teachers in each country (i.e.,
those who were teaching pupils in Cohort 1 or Cohort 2 during the 2000-2001 school year). Eighty-
eight percent of teachers who were identified through the sampling process participated in the study
(i.e., agreed to classroom visits and completed teacher surveys). Alternates to replace non-
respondents were drawn at random from the remaining pool of eligible teachers. During the quality
control process, data from one teacher and the associated three pupils were excluded from the
evaluation due to the inappropriateness of the initial selection of that teacher—very young pupils in a
special school for the arts.
RWCT country coordinators identified control-group teachers to match RWCT teachers in multiple
ways. The goal was to identify schools that were similar geographically and demographically to the
schools from which RWCT teachers were selected and then to identify teachers who were teaching
similar classes and had similar educational backgrounds and work experience. In reality, once
schools were identified, data collectors were often routed to teachers whom the school principals had
selected.
With limited information about control-group teachers, country coordinators often felt compelled to
accept principal recommendations as a pre-requisite for site visits. They therefore caution that there
might be somewhat of an imbalance in the selection of control-group teachers, many of whom were
picked selectively by school principals who had an incentive to identify the best Cohort 1 or Cohort 2
teachers or teachers with the best pupils in their schools.
In fact, there were very few demographic and experiential differences between the RWCT and
control-group teachers who participated in the evaluation (see Table 2). The only statistically
significant difference between the two groups is that on average RWCT teachers are from smaller

towns than control-group teachers. Educational background was not included in the table because it
was calculated on a country-by-country basis, but in these instances, too, differences were not
statistically significant.
TABLE 2: COMPARISONS OF RWCT AND NON-RWCT TEACHERS
RWCT Teachers Control-Group Teachers

(N=146) (N=139)
Females 90 percent 85 percent
Average age 40 years old 42 years old
Percentage of teachers whose first language was not the same 2.7 percent 2.8 percent
as the class observed
Average urbanicity * Town of approx. 10,000 Town of approx. 34,000
Average years with the class that was observed 2.5 years 2.6 years
Average hours per week with the class that was observed 6.8 hours per week 8.0 hours per week
Average years working as a teacher 15.0 years 16.6 years
Average years working at that school 9.6 years 9.0 years
* Statistically significant difference at the 0.05 level.
Table 3 compares background characteristics of the pupils who participated in this evaluation. It
shows that there was also only one statistically significant demographic difference between the
RWCT and control-group pupils who participated in the evaluation. Control-group pupils in Cohort
2 were a tenth of a year older on average than RWCT pupils.
TABLE 3: COMPARISONS OF RWCT AND CONTROL-GROUP PUPILS
Cohort 1 Cohort 2
Control-
RWCT Pupils Control-group RWCT Pupils group Pupils
(N=210) Pupils (N=207) (N=228) (N=206)
Percent girls 52 percent 50 percent 58 percent 53 percent
Average age 10.6 years 10.4 years 13.4 years * 13.5 years *
Percentage of pupils whose first 4.3 percent 2.9 percent 2.2 percent 3.9 percent
language was not the same as the
class observed
Kinds of reading materials in house 3.2 kinds 3.2 kinds 3.4 kinds 3.4 kinds
(i.e., regular newspaper, dictionary,
encyclopedia, magazines)
Kinds of electronic devices in house 3.3 kinds 3.3 kinds 3.3 kinds 3.3 kinds
(i.e., record player, cassette player,
calculator, color television)
Kinds of high-level luxury items in 1.0 kind 1.0 kind 1.0 kind 1.0 kind
house (i.e., computer, automobile)
The educational backgrounds of pupils’ fathers, mothers, and oldest siblings were calculated on a
country-by-country basis. The only statistically significant differences between RWCT and control
groups was in Macedonia, where mothers of RWCT pupils were more educated than mothers of non-
RWCT pupils (2.0 and 1.4 respectively on a four-point scale), and older siblings of control-group
pupils were more educated than those of RWCT pupils (3.6 and 2.6 respectively on a six-point scale).

Data Collection
Teams of five to seven in-country data collectors conducted data collection activities in each country.
Data collectors, who were initially recommended by in-country RWCT and Soros Foundation staffs,
were selected by AIR through an application process. Applicants who had previously worked for or
participated in the RWCT project were disqualified. Otherwise, candidates with the most interest,
professional experience working in schools, and experience conducting educational research were
chosen to conduct classroom observations, administer teacher and pupil surveys, and score open-
ended survey items.
All data collectors participated in a three-day intensive workshop conducted by AIR staff from
February 3-5, 2001, in which they became familiar with data collection instruments and procedures
and practiced scoring classroom observations and open-ended survey items. Data collectors were
required to achieve a high degree of inter-rater reliability before completing the workshop and
beginning the data collection process. Twenty-four of 26 data collectors completed the workshop
requirements successfully.
The Czech and Latvian teams piloted data collection procedures between February 7-9, and all data
collectors received slightly revised procedural instructions on February 10. Data collectors in all four
countries then conducted full-scale data collection between February 12 and March 2. Each data
collector conducted approximately 10-15 visits to RWCT and control-group classrooms—double
blind to the extent possible7 and with a random distribution of RWCT and control group classes—
and observed classroom activities for at least 45 minutes per class. Data collectors then asked the
teacher to complete a 45-minute survey and administered a 45-minute survey to three pupils whom
they selected at random from the class. When data collectors were surveying fourth- and fifth- grade
pupils from Cohort 1 classrooms, they read all of the questions aloud.
Data collectors scored open-ended survey items during the week of March 5. RWCT country
coordinators then conducted random quality control checks during the week of March 12 to ensure
that surveys were scored accurately and that data were entered correctly into spreadsheets. They also
made random calls to teachers who participated in the study to ensure that data collectors had
actually visited their classes and that there were no problems with the data-collection process. Data
were then sent electronically to AIR during the week of March 19, where they were cleaned and
underwent a series of validity checks. AIR staff then analyzed survey and classroom observation
data during April and May.
In April, it was discovered that two data collectors in Latvia had only visited RWCT classes and that
one data collector had visited nine control classes and only two RWCT classes, thereby creating a
potentially large “rater stringability effect.”8 To remedy the situation, it was necessary to exclude the
7
Data collectors were not informed prospectively whether they were visiting classrooms with RWCT or control-
group teachers, and teachers were not supposed to know that data collectors were visiting their classrooms explicitly
to evaluate the RWCT project. In some circumstances, however, school principals demanded to know the explicit
purpose of the classroom visits before they would permit the visits to take place. It is therefore assumed that some
teachers were told of the purpose of classroom visits before they took place.
8
Under normal scoring conditions, the effect of a rater’s natural scoring biases is minimized when that bias is
distributed randomly across the scores of intervention and control groups. If, however, a rater receives a non-
random allocation of items to score in which the intervention or control group predominates, that rater’s bias
becomes more pronounced and can have an undue effect on scoring outcomes, or a rater stringability effect.

classroom observation data from all three data collectors and to re-score the open-ended survey data
with a more even distribution of RWCT and control-group surveys among raters. After reviewing
scoring procedures on May 14-15 and working again with an AIR staff member to achieve a high
level of inter-rater reliability on open-ended survey items, four members of the original Latvian data-
collection team re-scored the problematic survey data, and data were returned to AIR for re-analysis.
Study Limitations
The data collected for this evaluation provide extensive information about RWCT, pupil critical
thinking skills, instructional practices, and teacher and pupil attitudes and behavior. The scope of the
evaluation, however, made it impossible to examine how RWCT pupils compare to control-group
pupils on traditional academic measures of pupil outcomes. An extensive effort was made to collect
national-level pupil achievement data from all 24 RWCT countries, but no such data were available
publicly. It was also not possible to establish a causal relationship between RWCT participation and
teacher and pupil outcomes since data for this evaluation were only collected at a single point in
time. Quasi-longitudinal analyses were conducted with teachers who joined RWCT in the first,
second, third, and fourth years of the project, making it possible to infer causality in some
circumstances, but these inferences cannot be considered conclusive.

IV. EVALUATION RESULTS

This section of the report describes the results of the RWCT evaluation based on the three research
questions. It begins with pupil critical thinking skills, continues with instructional practices, and
concludes with attitudinal changes among RWCT teachers and pupils as a consequence of teachers’
participation in the project.
Evaluation Question 1: To what extent do RWCT pupils have higher critical

thinking skills than their control-group peers?
Summary of Findings
One of the cornerstones of RWCT teaching strategies is the “ERR” framework, a process by which
teachers guide pupils to “evoke” prior knowledge, “realize” meaning by synthesizing new
information with their prior knowledge, and “reflect” on the implications of that meaning for
themselves and their understanding of the world. RWCT Guidebook I,9 for example, describes how a
third-grade teacher might use the framework to teach a lesson about sea turtles. Under this scenario,
a teacher would ask pupils to write down everything they know about sea turtles (to evoke prior
knowledge); read the story and mark in the margins in an article about sea turtles whether these
expectations were confirmed or denied (to realize meaning through an active cognitive process); and
discuss in pairs what knowledge was confirmed, what new information was encountered, and what
new questions the article had raised (to reflect on the information). It is through this process that
RWCT teaching strategies are supposed to help pupils refine their critical thinking skills. The
question is whether such strategies are successful at fostering critical thinking.
Evaluation results indicate that RWCT pupils, on the whole, demonstrate a higher level of critical
thinking than pupils in control groups (see p.12), even compared to pupils whose teachers have
participated in other content- or instruction-based professional development activities in the past
three years (see p.14). Differences between RWCT pupils and pupils in control groups are not a
consequence of pupil characteristics or teacher professional histories but largely a consequence of
teachers’ integration of critical thinking principles into teaching practices (see p.14).
It is not clear, however, how long it takes for pupils to benefit from their teachers’ RWCT
participation or how those benefits change over time. Among RWCT pupils, there are no statistically
significant patterns in pupils’ critical thinking scores based on the number of years an RWCT teacher
has participated in the project (see p.16). There are also no statistically significant differences in
critical thinking scores between RWCT pupils whose teachers were trained by international
volunteers and those whose teachers were trained by in-country teacher educators (see p.16).
9
Jeannie L. Steele, Kurtis S. Meredith, and Charles Temple, RWCT Guidebook I: A Framework for Critical
Thinking across the Curriculum, 1998, pp.12-14.

Detailed Findings
At the aggregate level, RWCT pupils score higher than pupils in control groups on critical thinking
assessments.
The overarching question for this evaluation is whether RWCT pupils score higher than pupils in
control groups on critical thinking assessments that measure pupils’ ability to form original opinions,
chose rationally among competing ideas, solve problems, and debate ideas responsibly. Table 4
indicates that RWCT pupils overall scored higher than pupils in the control group by statistically
significant differences. On 25-point critical thinking scales, RWCT pupils in Cohort 1 scored an
average 2.2 points higher than control-group pupils, and RWCT pupils in Cohort 2 scored an average
of 1.7 points higher. Both of these differences are statistically significant.
Results, however, varied by country. The most substantial country-level differences were in
Macedonia, where differences were positive and statistically significant for both Cohort 1 and Cohort
2. Cohort 1 RWCT pupils in Macedonia, for example, scored almost five points higher on average
than did their non-RWCT counterparts. Differences were also positive in the Czech Republic and
Kyrgyzstan and with statistical significance for Cohort 1 in the Czech Republic and Cohort 2 in
Kyrgyzstan.
In Latvia, however, RWCT pupils scored lower than pupils in the control group, although not by
statistically significant differences in either cohort. When comparing pupils in Latvia to pupils in
other countries, data reveal two important differences. First, RWCT pupils in Latvia scored lower on
critical thinking assessments than RWCT pupils in the other three countries. Second, pupils in the
Latvian control groups scored higher than control groups in the other three countries.
TABLE 4: DIFFERENCES IN CRITICAL THINKING SCORES

BETWEEN RWCT AND CONTROL-GROUP PUPILS
Czech
Overall Republic Kyrgyzstan Latvia Macedonia
Score (SD) Score (SD) Score (SD) Score (SD) Score (SD)
Cohort 1
RWCT Pupils 13.90 (5.07) 15.04 (4.73) 14.20 (5.41) 11.89 (4.18) 14.58 (5.37)
Control-group Pupils 11.71 (5.08) 11.57 (4.68) 12.25 (4.56) 13.38 (4.15) 9.63 (6.22)
Difference 2.19 * 3.47 * 1.95 -1.49 4.95 *
Sample Size 417 108 102 102 105
Cohort 2
RWCT Pupils 13.49 (4.74) 14.21 (4.18) 14.05 (4.81) 11.92 (3.85) 14.02 (5.70)
Control-group Pupils 11.77 (5.01) 12.17 (4.54) 11.57 (4.77) 12.56 (4.55) 10.87 (5.97)
Difference 1.72 * 2.04 2.48 * -0.64 3.15 *
Sample Size 434 101 114 111 108
One possible explanation for the first difference, lower critical thinking scores among RWCT pupils
in Latvia compared to RWCT pupils in other countries, is that RWCT has been operating in Latvia
one year less. This suggests a possible “generational effect,” whereby a country must have the
project in place for a specific amount of time for pupil-level effects to be observed. This hypothesis,
however, cannot be tested within the scope of this evaluation. It would require the collection of new

data in Latvia in February 2002 to determine whether the patterns in pupil critical thinking scores
were more like those in the Czech Republic, Kyrgyzstan, and Macedonia in February 2001.
A possible explanation for the second difference, that control-group pupils in Latvia have higher
critical thinking scores than control-group pupils in other countries, is a potential “saturation” of in-
service professional development activities for teachers in that country. First, a larger percentage of
teachers in the Latvian sample participated in in-service professional development activities than
teachers in other countries. Second, teachers in Latvia participated in an average 2.2 professional
development activities in the past three years—approximately the same number as in the Czech
Republic but nearly twice as many as teachers in Macedonia and more than three times as many as in
Kyrgyzstan (see Appendix C, Table I).10 The implication is that Latvian teachers may be learning
about critical thinking strategies in other professional development settings. The plausibility of this
hypothesis is bolstered by the fact that the range of critical thinking scores in Latvia for each cohort
and measured separately for RWCT and non-RWCT pupils is generally smaller than in the other
evaluation countries (see Table 4).
To test this hypothesis, analyses were conducted to determine how RWCT pupils compare in critical
thinking scores to pupils in control groups whose teachers did not participate in professional
development activities in the past three years. Even under these circumstances, however, RWCT
pupils in Latvia score lower on critical thinking assessments than do pupils in control groups, albeit
again by a non-significant amount (see Table 5 and C,II). The negative differences in critical
thinking scores between RWCT pupils and control-group pupils in Latvia therefore cannot be
explained by teachers’ in-service professional development activities in the past three years.
These findings suggest that higher critical thinking scores among pupils in the Latvian control groups
may be due to other professional development experiences, such as pre-service teacher education and
professional development opportunities prior to the last three years that promote critical thinking and
active learning strategies. Since the early 1990s, educational policymakers in Latvia have worked to
incorporate activities that promote critical thinking and active learning into classroom instruction. In
1998, Latvia even enacted standards for primary education that included critical thinking as one of
the main components of teaching practices. Future studies will have to collect information about pre-
service teacher education and earlier professional development experiences to test whether they
explain higher critical thinking scores among control-group pupils in Latvia.
TABLE 5: DIFFERENCES IN CRITICAL THINKING SCORES BETWEEN RWCT PUPILS AND PUPILS
WHOSE TEACHERS DID NOT PARTICIPATE IN PROFESSIONAL DEVELOPMENT ACTIVITIES IN
THE PAST THREE YEARS
Differences in Critical Thinking Scores

Czech
Cohort 1 2.51 * 2.21 2.47 * -0.33 5.45 *
Cohort 2 2.32 * 2.38 2.40 * -2.58 4.19 *
10
Hereafter, tables referenced in Appendix C will be noted as “(C,I)” for the first table, “(C,II)” for the second table,
etc.

At the aggregate level, RWCT pupils in both cohorts scored higher on critical thinking assessments
than pupils whose teachers participated in other content- or instruction-based professional
development activities over the past three years but only with statistical significance for RWCT
pupils in Cohort 1.
Table 5 also demonstrates that RWCT pupils overall score higher on critical thinking assessments
than pupils whose teachers have not participated in professional development activities. The next
question is how RWCT pupils’ scores compare against a higher standard: pupils whose teacher had
participated recently in other professional development activities. Table 6 indicates that RWCT
pupils in both cohorts score higher than pupils whose teachers participated in instruction- and
content-based professional development activities in the past three years (also see C,III). These
differences, however, were only statistically significant for pupils in Cohort 1.
At the country level, RWCT pupils in the Czech Republic (Cohort 1) scored significantly higher than
pupils in the control group while RWCT pupils in Macedonia (Cohort 2) scored significantly lower
than pupils in the control group whose teachers had participated in content-based professional
development activities over the past three years. These findings indicate that other content-based
professional development interventions for secondary school teachers in Macedonia, where they
exist, may be even more powerful in improving pupils’ critical thinking scores than RWCT.
TABLE 6: DIFFERENCES IN CRITICAL THINKING SCORES BETWEEN RWCT PUPILS AND PUPILS
WHOSE TEACHERS PARTICIPATED IN OTHER CONTENT- OR INSTRUCTION-BASED
PROFESSIONAL DEVELOPMENT ACTIVITIES IN THE PAST THREE YEARS
Differences between RWCT and Control Group

Czech
Cohort 1
Difference (Pupils whose teachers had content-based PD) 1.31 * 2.30 * -2.80 -0.41 2.36
Difference (Pupils whose teachers had instruction-based PD) 1.28 * 2.18 * -2.13 -0.49 2.15
Cohort 2
Difference (Pupils whose teachers had content-based PD) 0.57 1.57 0.22 -0.64 -3.65 *
Difference (Pupils whose teachers had instruction-based PD) 0.43 0.83 0.22 0.70 -0.77
RWCT teaching strategies, which include integration of critical thinking principles into teaching
practices and facilitating pupil-to-pupil interaction, explain much of the difference in overall critical
thinking scores between RWCT and control-group pupils.
Aggregate results in Table 6 indicate that a teacher’s participation in RWCT makes more of a
difference in critical thinking scores than other content- or instruction-based in-service professional
development activities, particularly for pupils in Cohort 1. What is it about RWCT classrooms,
though, that leads to higher critical thinking scores?
To answer this question, AIR conducted a series of analyses to determine the extent to which
differences in critical thinking scores are mediated by other variables. Table 7 illustrates how
various combinations of variables effect the significant relationship between teachers’ RWCT
participation and pupils’ critical thinking skills (also see C,IV). Columns labeled “proportional

reduction” show the percent of the difference in critical thinking scores that can be explained by
other variables. These results indicate that pupil background characteristics and the specific subject
taught in each class have little bearing on why RWCT pupils outperform pupils in control groups on
critical thinking assessments. Teacher characteristics, particularly the number of years a person has
been a teacher, explain some of the score differential for pupils in Cohort 2, but this proportional
reduction is also modest compared to others in the table.
Classroom communication patterns (for Cohort 2) and teachers’ integration of critical thinking
principles into teaching practices (for both cohorts) are the variables that explain much of the positive
and statistically significant differences in pupil critical thinking scores.11 And within the group of
classroom communication patterns, the facilitation of pupil-to-pupil interaction has the most
substantial effect on critical thinking differences (see C,IV). Given that RWCT teachers integrate
more critical thinking principles into teaching practices than teachers in the control group and also
spend more class time facilitating pupil-to-pupil interaction (findings that are discussed later in the
report), the proportional reduction statistics in Table 7 provide strong evidence that teachers’ use of
RWCT strategies has a positive effect on pupils’ critical thinking skills.
TABLE 7: PROPORTIONAL REDUCTIONS IN CRITICAL THINKING SCORE DIFFERENCES
Cohort 1 Cohort 2
Variable Categories Proportional Proportional
Reduction Reduction
Pupil Background Characteristics (i.e., gender, age, language minority status, 0.07 0.03
reading materials at home, electronic devices at home, luxury items at home,
urbanicity)
Teacher Characteristics (i.e., age, years working with the same class, hours per 0.00 0.19
week with the same class, years as a teacher, years at the same school)
Class Subject (i.e., general, humanities, social science, physical science, fine arts) 0.00 0.02
Classroom Communication Patterns (i.e., teacher to pupil, teacher to pupil to 0.00 0.51
teacher, pupil to pupil, teacher to pupil to pupil to teacher)
Integration of Critical Thinking Principles into Teaching Practices 0.43 0.52
The role of teacher integration of critical thinking principles in pupils’ critical thinking scores,
however, varies by country (see Table 8 and C,V). The association is the strongest in Kyrgyzstan,
where teachers’ integration of critical thinking principles into teaching practices explains 127 percent
of the difference in Cohort 1 pupil scores. This means that Cohort 1 control-group pupils in
Kyrgyzstan would have actually scored higher on critical thinking assessments than RWCT pupils if
not for RWCT teachers’ integration of critical thinking principles into their teaching practices. In
Latvia, however, teachers’ integration of critical thinking principles has almost no effect on RWCT
pupils’ critical thinking scores.
11
Proportional reductions can sum to more than 100 percent because analyses do not take into consideration the
inter-correlations among combinations of variables.

TABLE 8: PROPORTIONAL REDUCTIONS IN CRITICAL THINKING SCORE

DIFFERENCES RELATED TO THE INTEGRATION OF CRITICAL THINKING
PRINCIPLES IN CLASSROOM PRACTICES, BY COUNTRY
Proportional Reductions for Differences in “Integration” Scores

Czech
Cohort 1 0.43 0.37 1.27 0.00 0.08
Cohort 2 0.52 0.06 0.84 0.05 0.56
The weak association between teaching practices and critical thinking scores may thus explain the
negative differences in critical thinking scores between RWCT pupils and control-group pupils in
Latvia, but it does not explain the cause of that association. Table 5 indicates that saturation of
professional development activities in Latvian schools is not a sufficient answer. Determining
whether the difference is due to the maturity of RWCT in Latvia as compared to the other evaluation
countries cannot be answered in the absence of follow-up data.
At the aggregate level, there are no statistically significant patterns in pupils’ critical thinking scores
based on the number of years an RWCT teacher has participated in the project or whether RWCT
teachers were trained by international or in-country teacher educators.
Analyses were also conducted just using RWCT pupil data to determine whether pupils performed
higher on critical thinking assessments based on length of teacher involvement in the project.
Exhibit 3 illustrates overall results (also see C,VI). This histogram indicates that there are no
statistically significant associations between the number of years teachers have been participating in
RWCT and pupils’ critical thinking scores for either cohort. For pupils in Cohort 1, there is a dip in
critical thinking scores with third-year teachers (from 14.35 to 13.05 points) before they increase
again with fourth-year teachers (to 15.00 points). In contrast, scores increase for pupils in Cohort 2
with more teacher exposure for first- and second-year teachers (from 12.93 to 14.08 points), but they
then decline for third- and fourth-year teachers (to 13.67 and 12.44 points respectively).
EXHIBIT 3: CRITICAL THINKING AMONG RWCT PUPILS, BASED ON YEARS OF TEACHER

PARTICIPATION IN RWCT
25
20
Overall scale
14.35 15
13.59 14.08 13.67
15 13.04 12.93 12.44
10
0
1 2 3 4 1 2 3 4
Cohort 1 Years Cohort 2 Years

At the country level, the only statistically significant correlation between years of teacher
participation in RWCT and pupil critical thinking scores is for Cohort 1 pupils in the Czech
Republic. In this instance, however, mean critical thinking scores are 3.6 points lower for pupils of
second-year teachers than pupils of first-year teachers.
AIR also examined the extent to which pupils whose teachers were trained in RWCT strategies by
international volunteers are at more or less of an advantage compared to those whose teachers were
trained by in-country teacher educators. Findings indicate that Cohort 1 pupils whose teachers were
trained by international volunteers scored 1.2 points higher than pupils whose teachers were trained
by in-country teacher educators (see C,VII); however, the reverse was true for pupils from Cohort 2
(a negative 1.1 point difference). In neither instance, however, were differences in source of training
statistically significant. There are also no statistically significant differences in scores at the country
level.
Question 2: To what extent do RWCT teachers maintain the fidelity of the

RWCT model in their teaching practices?
Summary of Findings
Measuring the extent to which teachers who have participated in RWCT maintain the fidelity of the
RWCT model in their teaching practices is important for two reasons. First, investigating teacher
practices provides insight into the effects of professional development activities (see Exhibit 2, Box
A, in the conceptual model) on teacher knowledge, attitudes, skills, and behavior (Box B). Are the
RWCT workshops clear enough so that teachers understand key critical thinking concepts? Do
teachers believe RWCT concepts to be important enough to incorporate into their teaching practices?
Second, maintaining the fidelity of RWCT is important because classroom communication patterns
and teachers’ willingness to integrate critical thinking principles into teaching practices have
appreciable effects on pupils’ critical thinking scores (see Table 7).
Aggregate-level findings indicate that RWCT teachers develop lesson plans that integrate more
critical thinking principles into teaching practices than peers in control groups (see p.18). These
differences continue to be statistically significant even when comparing RWCT teachers to control-
group teachers who have participated in content-based professional development or instruction-based
professional development activities in the past three years (see p.19). Furthermore, differences in
how teachers integrate critical thinking principles into their teaching practices are not a consequence
of teacher background characteristics or the subjects of the classes that data collectors observed for
this evaluation (see p.19). By all accounts, differences appear to be a consequence of teachers’
participation in the RWCT project.
Results also demonstrate that RWCT teachers’ integration of critical thinking principles into teaching
practices increases over time (see p.20). Teachers integrate more critical thinking principles into
their teaching with each year of RWCT participation. It is therefore not clear why pupils who work
with more mature RWCT teachers do not also score systematically higher on critical thinking
assessments than pupils who work with teachers who are more recently exposed to the RWCT
curriculum.

Teachers who were trained in RWCT strategies by international volunteers are also likely to integrate
critical thinking principles into teaching practices more than teachers who were trained by in-country
teacher educators (see p.21). This difference, however, cannot be disentangled from the fact that
teachers trained by international volunteers have participated in RWCT longer than teachers trained
by in-country teacher educators.
In addition the integration of critical thinking principles, there is substantial evidence that RWCT
teachers maintain the integrity of the project in other ways. At the aggregate level, for example,
RWCT teachers outperform teachers in the control group on all observed measures of effective
instructional practices. For instance,
• RWCT teachers spend more time promoting classroom communication patterns that generate
classroom discussion and pupil interaction and less time lecturing and asking questions to
individual pupils (see p.21);
• RWCT teachers score higher on scales that measure “authentic pedagogy” (i.e., promoting
higher-order thinking, substantive conversation, connections to the world outside the classroom,
the organization of information, and the consideration of alternative solutions), classroom
organization, and teacher wait time after asking questions (see p.22); and
• RWCT teachers spend less time lecturing, leading discussions, and conducting routine
administrative work, and more time facilitating pupil-led discussions, and small-group work than
teachers in the control group (see p.23).
Detailed Findings
RWCT teachers are more likely than their peers in control groups to develop lesson plans that
integrate critical thinking principles into teaching practices.
The RWCT approach to teaching deconstructs traditional teacher-pupil relationships and re-
establishes the educational process to help pupils take more control of their learning. Teachers
relinquish their former roles as instructors and assume new roles as mentors and facilitators. Rather
than reporting information and asking pupils to memorize and recite facts, teachers learn to help
pupils form original opinions, choose rationally between competing ideas, solve problems, and
debate ideas responsibly. They also learn to help pupils work cooperatively with others to construct
new ideas, appreciate different points of view, and recognize the ways people’s background can
influence their attitudes and perceptions. These principles are consistent with progressive views of
effective instructional practices and resonate with all but the most traditional views of education. But
do RWCT workshops provide willing teachers with enough guidance and support to accept this
philosophy and actually incorporate critical thinking principles into their own teaching practices?
Evaluation results indicate that they do.
On a survey question that asked teachers to describe the different phases of their lesson planning,
RWCT teachers were likely to incorporate more critical thinking principles into their teaching
practices and at a deeper level than teachers in the control group (see Table 9). RWCT teachers
scored 1.1 points higher on a five-point “integration scale” than teachers in the control group. Score
differentials are statistically significant as well for teachers at the country level in the Czech
Republic, Kyrgyzstan, and Macedonia but not for teachers in Latvia. The scores of Latvian RWCT

teachers are lower than those of RWCT teachers in other countries, and the scores of the Latvian
control group are again higher than control groups in other countries.
TABLE 9: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP TEACHERS

IN SCORES RELATED TO THE INTEGRATION OF CRITICAL THINKING PRINCIPLES
Czech
Score (SD) Score (SD) Score (SD) Score (SD) Score (SD)
RWCT Teachers 2.67 (1.09) 2.75 (1.14) 3.29 (0.65) 2.05 (1.05) 2.62 (1.11)
Control Group 1.54 (1.03) 1.61 (1.35) 1.45 (0.71) 1.68 (1.01) 1.42 (0.90)
Difference 1.13 * 1.14 * 1.84 * 0.37 1.20 *
Sample Size 177 66 71 70 70
* Statistically significant correlation at the 0.05 level.
One could again ask whether the non-significant differences in Latvia are due to the fact that the
program is one year younger than the RWCT programs in the three other evaluation countries. The
data, however, do not suggest that this is the case. RWCT teachers in Latvia only score 0.05 points
higher on the integration scale than control-group teachers who did not participate in professional
development activities in the past three years (C,VIII). RWCT teachers in other countries scored at
least 1.4 points higher, and in all of these countries the differences were statistically significant. The
only way to ensure that smaller score differentials in Latvia are not due to the relative youth of
Latvia’s RWCT project compared to the other three countries, however, would be to collect follow-
up data in Latvia in February 2002 to compare teacher and pupil outcomes for RWCT projects that
are at similar stages of development.
Differences in teachers’ integration of critical thinking principles into teaching practices appear to
be a consequence of RWCT participation and not participation in professional development activities
in general, teacher background characteristics, or subjects taught.
Similar to pupil-level analyses, analyses of teacher integration scores considered whether positive
differences for RWCT teachers are a consequence of mediating variables. For example, how do the
integration scores of RWCT teachers compare to teachers in the control group who have had other
professional development experiences? Table 10 indicates that RWCT teachers are still likely to
integrate more critical thinking principles into classroom practices than control-group teachers who
have participated in other content- or instruction-based professional development activities in the
past three years (also see C,IX). Differences continue to be statistically significant among teachers
overall but not at the country level.

TABLE 10: DIFFERENCES BETWEEN RWCT TEACHERS AND CONTROL-GROUP TEACHERS

WHO PARTICIPATED IN OTHER CONTENT- OR INSTRUCTION-BASED PROFESSIONAL
DEVELOPMENT ACTIVITIES IN THE PAST THREE YEARS IN SCORES RELATED TO INTEGRATING
CRITICAL THINKING PRINCIPLES INTO TEACHING PRACTICES
Differences between RWCT and Control Group

Czech
Teachers with content-based PD 0.68 * 0.65 -0.38 0.23 0.73
Teachers with instruction-based PD 0.63 * 0.71 0.29 0.26 0.46
* Statistically significant differences at the 0.05 level.
AIR again sought to determine the extent to which the difference in integration scores were a
consequence of variables such as teacher characteristics or class subjects that teachers were observed
teaching. Examination of proportional reductions (see C,X) indicate that they do not. These results
indicate that neither background characteristics nor class subject have measurable effects on the score
differentials between RWCT and control-group teachers. Country-by-country results were similarly
trivial, including those related to teacher education variables that were conducted on a country-level
basis.
The integration of critical thinking principles into teaching practices increases with each year of
teacher participation in RWCT.
Analyses were also conducted to determine whether participation in RWCT increases teachers’
integration of critical thinking principles in teaching practices over time. Exhibit 4 illustrates that
they do. Correlations between years of RWCT participation and scores on the integration scales at
the aggregate level are positive and statistically significant (ρ=0.32, see C,XI). At the aggregate
level, teachers’ integration scores increase with each year of RWCT participation. Length of RWCT
participation, however, does not translate into greater gains in pupil critical thinking scores (see
Exhibit 3). Annual gains in integration scores are statistically significant in Kyrgyzstan (ρ=0.33) but
not in Macedonia (ρ=0.12), the only other country with data for teachers who have more than two
years of RWCT experience.

EXHIBIT 4: INTEGRATION SCORES AMONG RWCT TEACHERS, BASED ON YEARS OF

PARTICIPATION IN RWCT
4
3.38
3.1
2.83
3
Overall scale
2.28
2
0
1 2 3 4
Integration Score *
Years of Participation
Teachers who were trained in RWCT strategies by international volunteers are likely to integrate
critical thinking principles into teaching practices more than teachers who were trained by in-
country teacher educators—a difference, however, that cannot be separated from the fact that
teachers trained by international volunteers have participated in RWCT longer than teachers trained
by in-country teacher educators.
AIR conducted a final set of analyses with integration scores to determine the extent to which scores
differed between RWCT teachers who were trained by international volunteers and those who were
trained subsequently by in-country teacher educators. Results indicate that RWCT teachers trained
by international volunteers score 0.8 points higher on integration scales than teachers trained by in-
country teacher educators (see C,XII). This difference is statistically significant, as is the country-
level difference of 0.8 points in Kyrgyzstan. It is impossible, however, to separate this finding from
the fact that these teachers have participated in RWCT at least one year longer than other teachers.
Future studies will have to control for the years of RWCT participation to make useful conclusions
about the relative effect of international and domestic RWCT instructors on teacher and pupil
outcomes.
RWCT teachers spend more time than teachers in the control group promoting classroom
communication patterns that generate classroom discussion and pupil interaction and less time
lecturing and asking questions to individual pupils.
Analysis of integration scores is one of three ways that this evaluation examined how teachers
incorporate RWCT strategies into their teaching practices. Classroom observations and closed-ended
survey questions were two others. In classroom observations, for example, data collectors recorded
the relative percentage of time that teachers and pupils engaged in various communication patterns.
These patterns included the following:
• Teacher to pupil interaction: One-way flow of information from teachers to pupils (e.g., lectures
or demonstrations);

• Teacher to pupil to teacher interaction: Two-way interaction in which a teacher may ask a
question, call on a pupil, and then respond to the pupil’s answer;
• Pupil to pupil interaction: Two-way interaction in which pupils communicate with each other;
and
• Teacher to pupil to pupil to teacher interaction: Two-way interaction in which a teacher

introduces a topic or a question and then pupils work on the issue together before the teacher
speaks again to the entire class.
Table 11 displays differences in classroom communication patterns between RWCT and control-
group classes (also see C,XIII). A positive number indicates that RWCT teachers spent a greater
percent of time using a particular communication pattern than teachers in the control group, while a
negative number indicates the opposite. At the aggregate level, RWCT teachers spent significantly
more time than teachers in the control group promoting classroom communication patterns that
facilitate classroom discussion and pupil interaction and less time lecturing and asking questions to
individual pupils. This is further evidence that RWCT teachers are maintaining the fidelity of the
RWCT model in their instructional practices. The directionality of these patterns is similar for all
four countries, but not all differences are statistically significant.
TABLE 11: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP CLASSES

IN CLASSROOM COMMUNICATION PATTERNS
Difference between RWCT and Control Group Classes in

Percent Time Spent Using Different Communication Patterns
Czech
Teacher – Pupil -15.59 * -17.48 * -15.42 * -15.02 * -10.47
Teacher – Pupil – Teacher -14.30 * -1.88 -25.16 * -18.20 * -10.16
Pupil – Pupil 14.86 * 15.52 * 24.78 * 5.13 15.18 *
Teacher – Pupil – Pupil – Teacher 7.16 * 3.85 15.81 * 0.12 8.04 *
RWCT teachers scored higher than teachers in the control group on scales that measure “authentic
pedagogy” (i.e., promoting higher-order thinking, substantive conversation, connections to the world
outside the classroom, the organization of information, and the consideration of alternative
solutions), classroom organization, and teacher wait time after asking questions.
Classroom observation protocols were also designed to measure various classroom processes. The
first is a set of “authentic pedagogy” scales created and validated by Fred Newmann, Walter Secada,
and Gary Wehlage at the University of Wisconsin’s Center on Organization and Restructuring of
Schools; and the second set is based on Alan Crawford and Sam Mathews’ RWCT teacher scoring
rubrics. Table 12 indicates that RWCT teachers at the aggregate and country levels score higher on
almost all of these scales than non-RWCT teachers (also see C,XIV). The only exceptions are for
Latvia, where the differences in three scales still favored RWCT but were not statistically significant.

These statistics suggest substantial differences in classroom processes between RWCT and control-
group classes. For example, the composite “authentic pedagogy” score12 indicates that RWCT
classes overall scored 1.3 standard deviations higher than control-group classes on classroom
observations that have been used extensively to evaluate the alignment between instruction and
assessment. As a more specific example, RWCT classes scored an average 3.77 on the higher-order
thinking scale compared to 2.41 for control-group classes—a statistically significant difference of
difference of 1.36 points.
A score of “2” indicates that pupils in the class are primarily engaged in lower-order thinking (i.e.,
receive or recite factual information, or to employ rules and algorithms through repetitive routines),
although at some point they perform higher-order thinking as a minor diversion within the lesson. In
contrast, a score of “4” indicates that many pupils in the class are engaging in higher-order thinking
(i.e., performing operations that require them to combine facts and ideas to synthesize, generalize,
explain, hypothesize, or arrive at some conclusion or interpretation) for a substantial portion of the
lesson. More descriptions about differences in classroom observation scores are provided in Table
XV in Appendix C.
TABLE 12: DIFFERENCES IN AUTHENTIC PEDAGOGY AND OTHER CLASSROOM

OBSERVATION SCALES BETWEEN RWCT AND CONTROL-GROUP TEACHERS
Difference between RWCT and Control-Group Classes in Scores on
Classroom Observation Scales
Czech
Scale Overall Republic Kyrgyzstan Latvia Macedonia
Overall “Authentic Pedagogy” Score Standard deviations 1.33 * 1.16 * 1.64 * 0.75 * 1.55 *
Higher-order Thinking Skills 5-point scale 1.36 * 1.29 * 1.62 * 0.78 * 1.56 *
Substantive Conversation 5-point scale 1.28 * 1.08 * 1.80 * 0.76 * 1.24 *
World beyond the Classroom 5-point scale 0.80 * 0.42 1.31 * 0.44 0.75 *
Organization of Information 3-point scale 0.80 * 0.69 * 0.88 * 0.46 * 1.05 *
Consideration of Alternatives 3-point scale 0.79 * 0.71 * 0.87 * 0.38 1.13 *
Encouragement of Alternative Answers 3-point scale 0.73 * 0.57 * 1.04 * 0.63 * 0.64 *
Classroom Organization 3-point scale 0.61 * 0.66 * 0.64 * 0.62 * 0.53 *
Teacher Wait Time after Asking 3-point scale 0.55 * 0.43 * 0.82 * 0.27 0.58 *
Questions
RWCT teachers spend less time lecturing, leading discussions, and conducting routine administrative
work, and more time facilitating pupil-led discussions, and small-group work than teachers in the
control group.
Finally, classroom observation protocols recorded the amount of time that teachers engaged classes
in various classroom activities. Table 13 summarizes the differences between RWCT and control-
group classes in the percent of time spent in each of these activities (also see C,XVI). RWCT
teachers overall spent an average 10.6 percent less time lecturing, 5.3 percent less time leading
discussions, 2.6 percent less time involved in routine administrative work, and 7.9 percent more time
12
This composite score is based on a psychometric model created using LISREL 8.0. It combines five scales from
the Authentic Pedagogy framework: promotion of higher-order thinking skills, substantive conversation,
connections to the world beyond the classroom, organization of information, and consideration of alternatives.

facilitating pupil-led discussions and 13.0 percent more time facilitating small-group activities. At
the country level, differences in percent time lecturing were statistically significant for every country.
The most substantial country-level difference was for RWCT teachers in Kyrgyzstan, who spent 21.4
percent more time facilitating small group work than teachers in the control group.
TABLE 13: DIFFERENCES IN OBSERVED CLASSROOM ACTIVITIES

IN RWCT AND CONTROL-GROUP CLASSROOMS
Differences between RWCT and Control Group in Percent

Time Spent on Classroom Activities
Czech
Lecture -10.57 * -8.36 * -10.77 * -8.53 * -14.27 *
Demonstrations -2.18 -3.77 0.00 -1.61 -2.62
Teacher-led Discussion -5.34 * 4.09 -9.90 * 0.10 -13.65 *
Pupil-led Discussion 7.89 * 0.76 12.88 * 1.05 12.53 *
Small Group Work 12.95 * 9.16 * 21.41 * 13.85 6.96
Work Individually -1.96 -1.65 -7.41 -8.62 5.92
Routine Administrative Work -2.60 * 1.68 -5.90 * -0.17 -0.48
Individual Projects 1.07 -0.53 0.37 0.68 4.22 *
Group Projects 1.99 1.82 0.59 5.94 * 1.94
These patterns are largely consistent with teacher and Cohort 2 pupil reports of the most common
classroom activities throughout the school year (see C,XVII and C,XVIII). Even though the
magnitude of the score differentials varies between the classroom observation protocols and the
surveys as well as between the teacher and pupil surveys, and even though teacher and pupil
questions were worded somewhat differently, the directionality of the differences between RWCT
and control-group classrooms is virtually identical at the aggregate level for all three sets of
instruments. The aggregate results indicate the following:
• RWCT teachers spend less time than control-group teachers:

Lecturing;
Conducting demonstrations;
Leading discussions; and
Having pupils work individually.
• At the same time, RWCT teachers spend more time than teachers in the control groups:
Facilitating pupil-led discussions;
Facilitating small-group work; and
Facilitating group projects.

Question 3: To what extent do RWCT teachers and pupils change their

attitudes about teaching and learning after teachers join the project?
Summary of Findings
The last topic of investigation is the extent to which RWCT changes teachers and pupils attitudes
about teaching and learning. This question emerges from the conceptual framework that has driven
the design of this evaluation (see Exhibit 2). It anticipates that teachers will change their attitudes
and behavior (Box B) based on their reactions to professional development interventions (Box A).
Data presented previously have already determined that RWCT teachers structure classroom
activities quite differently from teachers in control groups and that RWCT pupils consequently score
higher on critical thinking assessments than non-RWCT peers. Do affective measures among RWCT
teachers and pupils vary as well? This section examines the answers to this question. In addition, it
examines RWCT teachers’ impressions of the RWCT project overall, including perceptions about
changes in teaching practices and pupil outcomes. Please also refer to the companion piece to this
report, The 2000-2001 Reading and Writing for Critical Thinking Project Impact and
Institutionalization Study,13 for more analysis of teacher attitudes about RWCT based on participant
interviews.
Findings indicate that there were very few differences between RWCT and control-group pupils in
their general views about learning. One important difference was that RWCT pupils in both cohorts
were more likely to report enjoying activities that involve active pupil involvement in the educational
process (see p.26), a value that is closely aligned with the RWCT framework. They were
significantly more likely than their peers to respond to open-ended questions and that they like to
conduct experiments, write reports, complete homework assignments, and “find my own answers.”
In teacher responses, there was only one statistically significant difference between RWCT and
control-group teachers in their views about teaching and learning: RWCT respondents tend to enjoy
their jobs as teachers more than their colleagues in the control group (see p.26).
In terms of teachers’ perceptions of the RWCT project, teachers believe that their participation in the
project has improved their teaching, pupil learning, and pupil behavior; and they have thoroughly
enjoyed their participation in RWCT (see p.28). Teachers report that since joining RWCT they
spend more time planning their lessons and have changed their planning to consider strategies that
enable pupils to express their opinions, engage in group work, be exposed to a variety of teaching
methods, use supplementary materials, and use their experiences to enhance how they learn new
information (see p.30).
AIR used open-ended questions to ask teachers about the most positive and negative effects of
applying RWCT strategies in teaching. Nearly half of RWCT teachers wrote that enabling pupils to
participate more in class and to guide their own learning are the most positive effects of applying
RWCT strategies in teaching (see p.31). At the same time, the most common responses about the
most negative effects were that using RWCT strategies increases the class time required to complete
course requirements and that RWCT can require that teachers use supplemental course materials (see
p.32).
13
American Institutes for Research, The 2000-2001 Impact and Institutionalization Study of the Reading and
Writing for Critical Thinking Project, Washington, DC, 2001.

The last question in the teacher survey asked RWCT to recommend future changes to the project.
There was very little overlap in teacher responses. The most common recommendation, shared by 11
percent of RWCT teachers, is that RWCT develop more specific classroom curricula based on
RWCT methods (see p.32). However, almost the same percentage of respondents recommended that
RWCT change nothing.
Detailed Findings
In questions about teaching and learning, RWCT pupils in both cohorts were more likely than pupils
in control groups to enjoy activities in which they participate actively in the learning process, and
RWCT teachers enjoy their jobs as teachers more than teachers in the control group.
Pupils in both cohorts were asked open-ended questions about what they enjoy most about the work
they do in class. Since most pupils in Cohort 2 have classes each day with multiple teachers, “class”
for this cohort was defined as the one that was observed earlier on the day that they completed the
pupil surveys. Differences between RWCT and control-group pupils are summarized in Tables 14
and 15 (also see C,XIX and CXX). The aggregate results were consistent for pupils in both cohorts:
RWCT pupils were more likely than pupils in the control groups to say that what they enjoy most
about their class are activities in which they participate actively in the learning process. Examples
include “I like conducting experiments,” “I like writing reports,” “I like finding my own answers,”
and “I like homework.” RWCT pupils in Cohort 2 were in fact twice as likely as pupils in the control
group to say that they enjoy activities that promote active learning.
Table 14 indicates that there were no statistically significant differences at the country level between
RWCT and control-group pupils in Cohort 1. There were, however, statistically significant
differences among pupils in Cohort 2 (see Table 15). Twenty-six percent more RWCT pupils than
control-group pupils in Latvia and 41 percent more RWCT pupils in Macedonia, for example, wrote
that they most enjoy activities related to active learning than pupils in their respective control groups.
At the same time, because pupils were limited to discussing the one thing they enjoy most about
class, 28 percent fewer RWCT pupils than control-group pupils in Macedonia wrote that what they
enjoy most about class is learning generally. This included responses such as “I like to learn new
things,” “I like classroom discussions,” “I like to learn from the teacher,” “I like to learn from fellow
pupils,” and “I like [X] subject.” The difference between learning and active learning is the level of
pupil involvement in the educational process. The other statistically significant difference at the
country level was in the Czech Republic, where four percent more RWCT pupils wrote that they
enjoyed “everything” about school than pupils in the control group. There were no statistically
significant differences between RWCT and control-group pupils in Kyrgyzstan.

TABLE 14: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP PUPILS (COHORT 1)

IN WHAT THEY ENJOY MOST ABOUT THE WORK THEY DO IN CLASS
Differences between RWCT and Control-Group Pupils in the

Percent of Respondents Who Report Enjoying Different Activities
Overall Czech Republic Kyrgyzstan Latvia Macedonia
Learning generally -8.33 -14.58 15.69 -9.03 -24.67
Activities that involve active learning 12.43 * 21.25 -7.84 22.45 16.34
Everything 1.82 7.92 0.00 -2.31 2.52
Social activities 1.82 2.92 3.92 -4.86 6.69
Succeeding 1.35 -4.58 0.00 6.94 2.19
Extracurricular Activities -2.94 -3.33 -5.88 -2.31 1.75
Nothing -2.43 -1.67 -1.96 -2.08 -4.82
Working alone -0.50 2.08 -3.92 -0.23 0.00
Chores 2.38 0.00 1.96 7.41 0.00
TABLE 15: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP PUPILS (COHORT 2)

IN WHAT THEY ENJOY MOST ABOUT THE COURSE THAT DATA COLLECTORS OBSERVED
Differences between RWCT and Control-Group Pupils in the

Percent of Respondents Who Report Enjoying Different Activities
Czech
Learning generally -3.07 -6.41 -20.14 1.03 -27.78 *
Activities that involve active learning 18.60 -1.22 -5.36 25.96 * 40.74 *
Social activities 2.46 7.04 -1.59 6.35 -1.85
Everything -0.17 4.17 * 1.49 3.64 -7.41
Working with the Teacher -2.36 -0.51 7.24 0.00 0.00
Nothing -3.24 -9.24 2.58 -4.30 3.70
Succeeding -1.21 4.17 0.50 -5.88 -1.85
Extracurricular Activities -2.04 0.00 0.00 -8.59 0.00
It is easy 0.25 2.48 0.50 -0.37 0.00
Surveys also asked teachers their opinions about different aspects of teaching and learning, questions
that were provided to teachers in forced-choice formats. Table 16 displays the differences between
RWCT and control-group teachers on four-point agree/disagree scales (also see C,XXI). These
results indicate that there was only one statistically significant difference between RWCT and
control-group teachers: RWCT teachers on average enjoy their jobs as teachers more than their
colleagues. No differences were statistically significant at the country level.

TABLE 16: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP

TEACHERS IN VIEWS ABOUT TEACHING AND LEARNING
Differences between RWCT and Control-Group

Teachers on Four-Point Agree-Disagree Scales
Czech
I enjoy my job as a teacher 0.12 * -0.03 0.20 0.16 0.18
If I could begin my career again, I would not choose teaching -0.04 0.40 -0.50 0.04 -0.14
I decide what to teach this class -0.05 0.17 -0.04 -0.16 -0.21
I select outside texts for this class -0.05 0.00 -0.08 -0.11 -0.02
I decide how to teach class curriculum 0.00 0.08 0.02 0.06 -0.16
I rarely share ideas with colleagues -0.19 0.13 -0.49 -0.17 -0.22
I discuss, work, or share ideas about teaching with other teachers in
my school -0.02 -0.33 -0.03 0.10 0.16
I discuss, work, or share ideas about teaching with teachers from other
schools 0.10 0.01 -0.04 0.19 0.17
Despite my best efforts, it is impossible for me to teach all my pupils
to learn -0.07 -0.08 -0.25 0.00 -0.01
I am optimistic about the future of education in my country 0.17 0.10 0.34 -0.01 0.23
It is bad to change classroom practices based on student suggestions -0.17 -0.09 -0.13 -0.08 -0.36
RWCT teachers have enjoyed their participation in RWCT, believe that their participation in the
project has improved their teaching and pupil learning, and believe that their participation in RWCT
has had a positive effect on pupil behavior.
Other sets of forced choice questions asked teachers about various aspects of RWCT (see Table 17),
including how their participation in RWCT has affected pupil behavior (see Table 18). The results
indicate that RWCT teachers overall and at the country level have enjoyed their participation in
RWCT, believe that RWCT has helped their teaching and pupil learning, would recommend RWCT
to colleagues, and think that RWCT should be taught broadly to teachers in their country. They do
not believe that RWCT detracts from other teaching responsibilities or that pupils who are exposed to
RWCT learn less than their peers. RWCT teachers also believe that RWCT has increased pupil
involvement and cooperation in class, access to and retention of information, and enthusiasm for
learning new information. RWCT teachers also believe that RWCT has helped teachers improve
their working relationships with pupils.

TABLE 17: VIEWS AMONG RWCT TEACHERS ABOUT RWCT
Czech
Agreemnt N Agreemnt N Agreemnt N Agreemnt N Agreemnt N
Enjoyed workshops 3.86 146 3.69 32 3.92 38 3.82 39 3.97 37
Has helped my teaching 3.75 145 3.69 32 3.73 37 3.72 39 3.84 37
Has helped pupil learning 3.56 141 3.37 30 3.78 37 3.38 37 3.68 37
Would recommend to colleagues 3.81 145 3.66 32 3.87 38 3.79 38 3.92 37
Should be taught broadly 3.78 142 3.69 32 3.86 37 3.69 36 3.86 37
Detracts from other teaching responsibilities 1.46 142 1.47 32 1.22 37 1.67 36 1.49 37
Decreases content learned in class 1.73 143 1.84 32 1.38 37 1.81 37 1.89 37
1 = Disagree Strongly; 2 = Disagree Somewhat; 3 = Agree Somewhat; 4 = Agree Strongly.
TABLE 18: OPINIONS AMONG RWCT TEACHERS ABOUT CHANGES

IN PUPIL BEHAVIOR SINCE TEACHERS BEGAN PARTICIPATING IN RWCT
Czech
Effect N Effect N Effect N Effect N Effect N
Pupil involvement 4.54 138 4.13 30 4.83 35 4.41 39 4.76 34
Pupil cooperation 4.48 140 4.43 30 4.43 35 4.36 39 4.69 36
Access to/Retention of information 4.29 140 3.93 30 4.34 35 4.10 39 4.75 36
Enthusiasm for new information 4.36 139 4.07 30 4.49 35 4.05 38 4.81 36
Relationship with me, the teacher 4.26 140 3.70 30 4.54 35 3.92 39 4.81 36
1 = Became Much Worse; 2 = Became Somewhat Worse; 3 = No Major Differences; 4 = Became Somewhat Better; 5 = Became Much Better
RWCT teachers believe that their participation in RWCT has had a positive effect on their teaching.
The teacher survey also asked RWCT teachers to reflect on specific differences in their teaching
behavior as a result of their participation in RWCT. The survey first asked teachers to list three
descriptors about themselves as teachers before they joined RWCT and the three descriptors about
themselves after joining RWCT. Results were then organized topically, and the most frequent
responses are summarized in Tables 19 and 20 (see C,XXII and XXIII for more extended lists of
responses). More than 10 percent of RWCT teachers wrote that prior to joining RWCT they used
traditional teaching methods (e.g., lecturing, recitation, and drill-type questions to individual pupils),
were rigid in their instruction, were bored, and were not confident in their teaching. In contrast,
RWCT has helped teachers learn how to use non-traditional teaching methods (e.g., more pupil
interaction, small-group work, investigative problem-solving activities), work with pupils’ based on
pupils’ individual needs, become flexible in instruction, be receptive to multiple answers to
problems, respect pupil opinions, enjoy teaching, and be excited and confident about their work.
Although individual country-level patterns varied somewhat, they were largely consistent with
overall trends.

TABLE 19: MOST FREQUENT RWCT TEACHERS’ DESCRIPTIONS

ABOUT THEMSELVES AS TEACHERS BEFORE THEY BEGAN TO PARTICIPATE IN RWCT
Percent of RWCT Respondents

Czech
Used traditional teaching methods 44.52 50.00 34.21 33.33 62.16
Was rigid 31.51 15.63 28.95 28.21 51.35
Was bored 15.07 3.13 26.32 10.26 18.92
Was not confident 13.70 6.25 10.53 20.51 16.22
TABLE 20: MOST FREQUENT RWCT TEACHERS’ DESCRIPTIONS

ABOUT THEMSELVES AS TEACHERS AFTER THEY BEGAN TO PARTICIPATE IN RWCT

Czech
Use non-traditional teaching methods 28.08 25.00 21.05 28.21 37.84
Work with all pupils—pupil centered 22.60 25.00 18.42 15.38 32.43
Am flexible 21.23 3.13 13.16 17.95 48.65
Am open to all answers 19.86 25.00 34.21 12.82 8.11
Focus on pupil needs 19.18 21.88 26.32 10.26 18.92
Respect pupil opinions 18.49 9.38 36.84 2.56 24.32
Enjoy teaching 18.49 9.38 23.68 20.51 18.92
Am excited 15.07 3.13 26.32 10.26 18.92
Am confident 13.01 6.25 10.53 20.51 13.51
Since joining RWCT, teachers report that they spend more time planning their lessons and have
changed their planning to consider strategies that enable pupils to express their opinions, engage in
group work, be exposed to a variety of teaching methods, use supplementary materials, and use their
experiences to enhance how they learn new information.
The next question on the teacher survey asked RWCT teachers how they had changed their lesson
planning as a result of their participation in RWCT. Open-ended responses were grouped and coded,
and the most frequent responses are summarized in Table 21 (also see C,XXIV). The most frequent
response overall was that teachers were spending more time planning their lessons than they spent
prior to joining RWCT. In addition, teachers spent more time thinking about how to implement
RWCT methodologies (e.g., how to enable pupils to express their opinions, how to incorporate more
group work into classes, how to use a variety of teaching methods to teach pupils about a particular
topic or concept, how to incorporate different aids such as manipulables or texts to supplement the
required curriculum, and how to use pupils’ experiences to help pupils learn new information).

TABLE 21: MOST FREQUENTLY REPORTED CHANGES IN HOW TEACHERS

PREPARE FOR LESSONS SINCE BEGINNING THEIR PARTICIPATION IN RWCT

Czech
Use more time to plan lessons 25.34 36.26 24.91 19.51 22.49
Plan opportunities for pupils to express opinions 20.55 21.88 36.84 20.51 2.70
Plan more group work 19.86 9.38 26.32 30.77 10.81
Consider a variety of teaching methods 19.86 6.25 18.42 25.64 27.03
Use more aids and supplementary texts 18.49 31.25 15.79 10.26 18.92
Think about pupils' experiences 11.64 18.75 13.16 12.82 2.70
RWCT teachers were then asked to describe how their teaching practices had changed since they
began participating in RWCT. Open-ended responses were again categorized and coded, and the
most frequent responses are summarized in Table 22 (see also C,XXV). More than one-third of
RWCT teachers reported that one of the biggest changes in teaching practices was the emphasis on
pupil expression and opinions. Other frequent responses include the fact that RWCT instruction
creates more interesting classes, systematizes creative teaching practices, and emphasizes pupil
interaction and group work. Ten percent of teachers overall, however, reported that their teaching
had not changed very much since they began their participation in RWCT. This statistic was more
than twice as high in Latvia, where 23 percent of RWCT teachers reported that RWCT had not
changed their teaching practices very much.
TABLE 22: RWCT TEACHERS’ OPINIONS ABOUT HOW RWCT INSTRUCTIONAL

PRACTICES DIFFER FROM PREVIOUS INSTRUCTIONAL PRACTICES

Czech
Emphasizes pupil opinions and expression 36.30 50.00 39.47 28.21 29.73
Creates more interesting classes 20.55 15.63 26.32 20.51 18.92
Is systematic about creative teaching 13.01 9.38 5.26 12.82 24.32
Emphasizes pupil interaction and group work 12.33 15.63 15.79 10.26 8.11
Not very different 10.27 12.50 2.63 23.08 2.70
Nearly half of RWCT teachers wrote that RWCT enabling pupils to participate more in class and to
guide their own learning are the most positive effects of applying RWCT strategies in teaching.
The teacher survey also asked RWCT teachers to identify the three most positive and negative effects
of applying RWCT strategies in teaching. The most frequent responses to this open-ended question
are summarized in Table 23 (also see C,XXVI). Nearly half of RWCT teachers said that RWCT
enables pupils to participate more in class (49 percent) and guide their own learning (47 percent).
Other common responses were that RWCT increases the speed and quality of pupil learning, it
improves teacher/pupil relationships, and that it improves pupil behavior. These patterns were again
generally consistent cross-nationally.

TABLE 23: RWCT TEACHERS’ OPINIONS ABOUT THE MOST POSITIVE EFFECTS
OF APPLYING RWCT STRATEGIES IN TEACHING

Czech
Enables pupils to participate more in class 48.63 37.50 52.63 56.41 45.95
Enables pupils to guide their own learning 46.58 40.63 28.95 58.97 56.76
Increases the speed and quality of learning 15.75 15.63 18.42 10.26 18.92
Improves teacher/pupil relationships 15.07 9.38 10.53 5.13 35.14
Improves pupil behavior 11.64 0.00 10.53 10.26 24.32
At the same time, the most common responses about the most negative effects of RWCT on teaching
were that using RWCT strategies increases the class time required to complete course requirements
and that RWCT can require that teachers use supplemental course materials.
A parallel question asked RWCT teachers to list the three most negative effects of applying RWCT
strategies in teaching. Table 24 indicates that teachers did not answer this question with the same
amount of unanimity as the question about positive effects (also see C,XXVII). The most frequent
response, that RWCT increases the class time to complete course requirements, was cited by one of
five respondents. The next most frequent responses were that RWCT often requires that teachers use
additional classroom materials, that it makes pupil behavior worse, and that it makes it difficult to
teach the required curriculum. Teachers in the Czech Republic and Latvia generally had longer lists
of negative effects than those in Kyrgyzstan and Macedonia.
TABLE 24: RWCT TEACHERS’ OPINIONS ABOUT THE MOST NEGATIVE EFFECTS

Czech
Increases class time to complete requirements 21.23 25.00 15.79 30.77 13.51
Requires additional materials 12.33 0.00 10.53 20.51 16.22
Makes pupil behavior worse 10.27 18.75 2.63 20.51 0.00
Makes it difficult to teach required curriculum 10.27 25.00 5.26 12.82 0.00
The most common recommendation about future changes to the project, suggested by 11 percent of
RWCT teachers, was to develop more specific classroom curricula based on RWCT methods.
The final question in the teacher survey asked teachers to recommend changes to RWCT. The most
common recommendation was that RWCT develop more concrete, subject-specific materials that
exemplify RWCT methods (see C,XXVIII). The only other recommendation cited by 10 percent or
more of respondents overall was that RWCT not change anything in its workshops or subsequent
follow-up activities.

V. CONCLUSION
The purpose of this evaluation was to examine three key research questions about the consequences
of RWCT on participating teachers and their pupils as they related to a conceptual model of the
professional development process. That model predicts that an effective professional development
intervention for teachers would have a variety of direct effects on teachers and indirect effects on
pupils. For example, it would have a direct effect on teachers’ skills, and teachers’ reactions to the
intervention would have an effect on their attitudes and behavior. Changes in teachers’ skills,
attitudes, and behavior would then affect pupils’ attitudes and behavior.
This evaluation cannot lead to conclusive statements about effects since data were only collected at
one point in time. Comparisons between RWCT teachers/pupils and their peers in control groups,
however, strongly support the suggestion that RWCT has had a substantial positive effect on teaching
and learning. At the aggregate level, teachers have a high regard for the project and are enthusiastic
about the project and teaching overall. RWCT teachers also believe that their participation in RWCT
has changed their teaching practices and their pupils’ learning outcomes.
In fact, RWCT teachers exhibit substantially different teaching behaviors from teachers in control
groups. They are more likely to engage pupils actively in class activities and provide opportunities
for pupils to create knowledge for themselves by helping them to synthesize prior experiences and
new information. Pupils in turn have responded earnestly to new classroom processes and have
become more interested in their own active learning. These changes have translated into more
enhanced critical thinking skills.
The evaluation, however, indicated that changes for teachers and pupils as a consequence of RWCT
took place at different times. Quasi-longitudinal analyses of teachers, for example, suggest that
teachers continue to increase their appreciation for critical thinking and integrate more critical
thinking principles into teaching practices with each year of teacher participation in RWCT. In
contrast, data indicate that pupils did not achieve higher gains in critical thinking skills based on the
number of years their teachers had participated in RWCT. It would be informative in a future study
to track the progress of RWCT pupils’ critical thinking skills based on their own exposure to RWCT
teaching strategies over time.
The positive associations among teachers’ participation in RWCT, teaching practices, and learning
outcomes, however, were not consistent in all four countries. Differences between Latvian RWCT
and control-group teachers in their appreciation of critical thinking and integration of RWCT
principles into teaching practices, for example, were not statistically significant. In addition,
differences in pupil critical thinking scores in Latvia, although not statistically significant, favored
pupils in the control group.
Hypotheses about the anomalous Latvian findings were posed earlier in this report. The first is that
Latvia has one year less experience with RWCT than the other three countries included in this
evaluation. Second, Latvia’s saturation with in-service professional development opportunities may
reduce the differences in teaching and learning outcomes between RWCT and control groups. Third,
pre-service teacher education in Latvia may already promote critical thinking and active learning
strategies to the point that in-service professional development projects such as RWCT may not
result in measurably different teacher and pupil outcomes. Each of these hypotheses should be tested
in future studies.

Other findings related to RWCT in Latvia, however, indicate important outcomes for the project even
after one less year of work than in the other three countries. RWCT teachers are just as excited as
their colleagues in other countries about their participation in RWCT, and their teaching practices are
significantly different from control-group teachers—albeit many at a lower level than the differences
in other countries. These include less time spent in lecturing to classes, leading discussions, and
asking questions to individual pupils, and more time promoting small group work, higher-order
thinking, substantive conversation, lesson plans that promote critical thinking, multiple “correct”
answers to problems, and classroom organization that facilitates pupil interaction. Furthermore,
RWCT pupils in Latvia are significantly more likely than their non-RWCT peers to enjoy activities
that promote active learning.

APPENDIX A
ID#_________________________
PUPIL SURVEY, 4TH/5TH FORM
Cross-National Study of Teaching and Learning
Prepared by:
February 2001
The questions you are about answer will help us learn more about what it is like to be a pupil in your class
at school. Please answer these questions the best that you can. All of your answers will be kept a secret.
We will not tell anyone what you say so you can be honest and tell the truth. There are no right and
wrong answers. We just want to know what you think.
© 2001 American Institutes for Research

ID#_________________________
Pupil Survey, 4th and 5th Form
Section I: Problem Solving

1. Your teacher tells you that working by yourself while taking a test is a class rule. Think about this
rule and answer the following questions. Why can this rule be good? Why can this rule be bad?
Decide whether you think this rule is more good than bad. Then explain why you think that way. Use
as many examples as you can, and be as detailed as possible. Remember that you have 15 minutes to
answer this question.
This rule can be good because...
(Please write on the back of this page if you need more space)
This rule can be bad because...
Decide whether you think this rule is more good than bad, and explain why you think that way:
Appendix A 1 American Institutes for Research

ID#_________________________
2. Pretend that you want to open a snack kiosk in a city. Place an “X” on the map to indicate where you
would open the kiosk. Then, explain why you have chosen this location and why you think it is better
than other places on the map. Use as many examples as you can, and be as detailed as possible.
Remember that you have 15 minutes to answer this question.
Why did you choose this place for your kiosk? _______________________________________________
Why do you think this place is better than others on the map?
ID#_________________________
Section II: How Your Teacher Teaches

3. Please check the best answer for each of the following questions:
a) How often does your teacher teach a lesson d) How often do you work in small groups with
while pupils listen? other students?
___ Never ___ Never
___ Sometimes ___ Sometimes
___ Most of the time ___ Most of the time
___ Always ___ Always
b) How often does your teacher demonstrate e) How often do pupils work on projects or
something to the class? tasks that they pick for themselves?
___ Never ___ Never
c) How often does your teacher lead a discussion f) How often does your teacher use pupils’
in which the whole class participates? suggestions in deciding what to teach?
___ Never ___ Never
Section III: Beliefs about School and Learning

4. What do you think are the three most important reasons that children should go to school?
a) _______________________________________________________________________________
b) _______________________________________________________________________________
c) _______________________________________________________________________________
5. What do you enjoy most about the work that you do in your class?
ID#_________________________
Section IV: Background Information
6. Are you a boy or a girl? __________
7. How old are you? __________
8. What language do you speak in classes at your school? ______________________________
9. What is language do you speak with your family at home? ______________________________
10. Circle the number that indicates whether your family has these at home:
Do not
Have have
A place set aside for you to study…………. 1 2
Daily newspaper…………………………… 1 2
Dictionary………………………………….. 1 2
Encyclopedia ……. 1 2
Magazines…………………………………. 1 2
Record player……………………………… 1 2
Tape recorder or cassette player…………… 1 2
Calculator………………………………….. 1 2
Color television……………………………. 1 2
Computer…………………………………… 1 2
Automobile………………………………… 1 2
11. About how many books are there in your home?
a) Few (0-10)
b) Enough to fill one shelf (11-25)
c) Enough to fill one bookcase (26-100)
d) Enough to fill several bookcases (more than 100)
12. Which of the following best describes your father’s and mother’s education?
Father’s Education Mother’s Education

___ Finished elementary school ___ Finished elementary school
___ Finished middle school ___ Finished middle school
___ Finished high school ___ Finished high school
___ Went to college ___ Went to college
___ Finished college ___ Finished college

ID#_________________________
13. Which of the following best describes your oldest brother or sister’s education?
___ I do not have an older brother or sister

___ Is attending or finished elementary school
___ Is attending or finished middle school
___ Is attending or finished high school
___ Attended but did not complete college
___ Is attending college
___ Finished college
Closing
Thank you again for helping us to answer these questions and allowing us to observe your class. The
results from this survey will help us learn more about how to make schools better. Your thoughts are very
valuable in this process. Please return the survey now.

ID#_________________________
PUPIL SURVEY, 7TH/8TH FORM
Prepared by:
February 2001
This survey is designed to learn more about classroom practices as well as your experiences and thoughts.
Please answer questions as they relate to the course that was observed earlier today. We really appreciate
your participation. All individual responses will remain confidential.

ID#_________________________
Section I: Problem Solving

1. Some countries in Eastern Europe and Central Asia have experienced an increase in emigration (i.e.,
people moving to other countries) over the last 10 years. Emigration can have both positive and
negative effects on a country. Think about emigration and write an essay that explains how
emigration can benefit a country and harm a country. Then, decide whether you think emigration is
more beneficial or harmful to a country, and explain why you think that way. Make sure to answer all
parts of the question. Use as many examples as you can, and be as detailed as possible. Remember
that you have 15 minutes to answer these questions.

ID#_________________________
2. Suppose you want to open a snack kiosk in a city. Place an “X” on the map to indicate where you
would open the kiosk. Then, explain why you have chose this location and why you think it is better
than other places on the map. Use as many examples as you can, and be as detailed as possible.
Remember that you have 15 minutes to answer this question.

ID#_________________________
Section II: Classroom Instruction
NOTE: FOR THIS SECTION, PLEASE ANSWER QUESTIONS BASED ON THE COURSE THAT WAS OBSERVED
EARLIER TODAY.
3. a. Indicate how often your teacher does each of the following:
Less than At least At least At least

once a once a once a once a
Teacher activities month month week class
a. Lectures to the class……………………………………………….. 1 2 3 4
b. Provides demonstrations to the class (including lab demos.)……... 1 2 3 4
c. Leads whole class discussions………………………………...…... 1 2 3 4
d. Listens to class-led discussions……………………………………. 1 2 3 4
e. Has pupils work in small groups…...……………………………… 1 2 3 4
f. Has pupils work individually…...…………………………….….... 1 2 3 4
g. Helps pupils with their experiments, projects, or other hands-on
experiences…….………………………………………………….. 1 2 3 4
b. Please circle the letters associated with the three teacher activities listed in 3a that best describe your class.
4. a. Indicate how often you do each of the following:

once a once a once a once
Pupil activities month month week a class
a. Listen/take notes/observe demonstrations in whole-class settings… 1 2 3 4
b. Participate in discussions with peers………………….………….... 1 2 3 4
c. Participate in group discussions with the teacher………………….. 1 2 3 4
d. Do lab or field work, or other experiments or hands-on work…….. 1 2 3 4
e. Read silently….…...……………………………………...……….. 1 2 3 4
f. Read orally……….………………………………………...……… 1 2 3 4
g. Write essays or reports…………………………………………….. 1 2 3 4
h. Make presentations to the class....…………………………………. 1 2 3 4
i. Generate your own projects……………………………………….. 1 2 3 4
j. Complete worksheets related to a specific topic………………...… 1 2 3 4
k. Work on independent, long-term (at least one week long) projects.. 1 2 3 4
l. Work on problems with no single best solution…………………… 1 2 3 4
m. Debate ideas or otherwise explain your reasoning….…………...... 1 2 3 4
n. Complete tests or quizzes……………………………………...…... 1 2 3 4
o. Use hands-on models or manipulatives to solve problems…..……. 1 2 3 4
b. Please circle the letters associated with the three pupil activities listed in 4a that best describe your class.

ID#_________________________
5. What does your teacher use to evaluate your achievements? (Circle one for each line.)
Not Minor Moderate Very

used importance importance important
a. Multiple choice, true/false, or short answer…………………….. 1 2 3 4
b. Essay tests.……………………………………………………… 1 2 3 4
c. Performance on experiments, projects, or other hands-on
experiences……………………………………………………… 1 2 3 4
d. Teacher opinions of your work in class.……….……………….. 1 2 3 4
e. Oral reports……………………………………………………… 1 2 3 4
f. Written reports…………………………………………….…….. 1 2 3 4
g. Good classroom behavior……………………………………….. 1 2 3 4
h. Bad classroom behavior………………………………………… 1 2 3 4
i. Evaluations from other students………………………………… 1 2 3 4
j. Your impressions of your own work.…………………………… 1 2 3 4
6. How often does your teacher use pupils’ suggestions in deciding what to teach?
___ Never ___ Sometimes ___ Most of the Time ___ Always
Section III: Beliefs about School and Learning

7. The following is a list of reasons people often give for attending school. How important do you think
each of these reasons is? (Circle one number on each line.)
Strongly Somewhat Somewhat Strongly

School is important to… disagree disagree agree agree
a. Train for a good job….……………………….…………………… 1 2 3 4
b. Learn how to make one’s own decisions…………………………. 1 2 3 4
c. Chance to meet good friends………………………………….…… 1 2 3 4
d. Learn how to be sociable and get along with people……………… 1 2 3 4
e. Increase understanding of the world and oneself…………………. 1 2 3 4
f. Develop interest in good books, music, and art..………………….. 1 2 3 4
g. Prepare for higher education……………………………………… 1 2 3 4

ID#_________________________
8. What do you enjoy most about the course that was observed earlier today?
(Please use the back of this page if necessary)
Section IV: Background Information

9. Are you male or female? __________
10. How old are you? __________
11. Where do you live? Small village (less than 5,000 people) ___ Village (5,000 - 10,000) ___
Town (10,000 - 50,000) ___ Small City (50,000 - 100,000) ___ Large City (more than 100,000)___
12. What language do you spoke in classes at your school? ______________________________
13. What is the language that you speak with your family at home? ______________________________
14. Circle the number that indicates whether your family has these at home:
Do not
Have have
A place set aside for you to study… 1 2
Daily newspaper…………………….……. 1 2
Dictionary…………………………….…… 1 2
Encyclopedia or other reference books.….. 1 2
Magazines…………………………….….. 1 2
Record player…………………………….. 1 2
Tape recorder or cassette player…………. 1 2
Calculator………………………………… 1 2
Color television………………………….. 1 2
Computer…………………………………. 1 2
Automobile……………………………….. 1 2

ID#_________________________
15. About how many books are there in your home?
a. Few (0-10)
b. Enough to fill one shelf (11-25)
c. Enough to fill one bookcase (26-100)
d. Enough to fill several bookcases (more than 100)
16. Which of the following best describes your father’s and mother’s education?
Father’s Education Mother’s Education

___ Finished elementary school ___ Finished elementary school
___ Finished middle school ___ Finished middle school
___ Finished high school ___ Finished high school
___ Went to college ___ Went to college
___ Finished college ___ Finished college
17. Which of the following best describes your oldest brother or sister’s education?
___ I do not have an older brother or sister

___ Finished elementary school
___ Is attending or finished middle school
___ Is attending or finished high school
___ Attended but did not finish college
___ Is attending or finished college
Closing
Thank you again for completing this survey and allowing us to observe your class. The results from this
survey will be used to learn more about effective teaching and learning strategies. Your thoughts are very
valuable in this process. Please return the survey now.

ID#_________________________
TEACHER SURVEY
Prepared by:
February 2001
The survey that you are about to complete is designed to help us learn more about teaching practices. It
asks questions about instructional methods and your experiences as a teacher. We appreciate your
participation. All individual responses will remain confidential.

ID#_________________________
Teacher Survey
Section I: Views about schooling and teaching

1. What do you consider to be the attributes of an excellent pupil? Please explain your answer, and
provide as much detail as possible.
2. Please indicate your level of agreement with the following statements. (Circle one for each line.)
disagree disagree agree agree
a. I enjoy my job as a teacher……………………………………… 1 2 3 4
b. If I could begin my career again, I would not choose teaching… 1 2 3 4
c. I decide what to teach this class……………………………….... 1 2 3 4
d. I select outside texts for this class……………………………..… 1 2 3 4
e. I decide how to teach class curriculum………………..………… 1 2 3 4
f. I am rarely able to share ideas with colleagues…………….…… 1 2 3 4
g. I discuss, work, or share ideas about teaching with other teachers
in my school……………..….………………………………….… 1 2 3 4
h. I discuss, work, or share ideas about teaching with teachers from
other schools ….………………………………………………… 1 2 3 4
i. Despite my best efforts, it is impossible for me to teach all my
pupils to learn………….………………………………………… 1 2 3 4
j. I am optimistic about the future of education in my country….… 1 2 3 4
k. It is bad to change classroom practices based on student
suggestions……………………………………………….……… 1 2 3 4

ID#_________________________
Teacher Survey
Section II: Teaching activities

3. We would like to know more about your instructional practices. Please think about a text you
have used recently for the group of pupils that was observed today. Tell us the name of the text,
and answer the following questions.
a. Text: _____________________________________________________________________________
b. Before assigning this text, I would…
c. To help pupils identify important ideas in this text I would…
d. While pupils are reading this text, I would…
e. After pupils read this text I would…

ID#_________________________
Teacher Survey
Section III: Instructional practices

4. a. Indicate how often you do each of the following in your class(es):

Teacher activities month month week class
a. Lecture to the class………………………………………………… 1 2 3 4
b. Provide demonstrations to the class (including lab demos)………. 1 2 3 4
c. Lead whole class discussions, in which you do most of the talking. 1 2 3 4
d. Listen to class-led discussions, in which the students do most of
the talking………………………………………………………….. 1 2 3 4
e. Have pupils work in small groups…...…………………………….. 1 2 3 4
f. Have pupils work individually…...………...……………………... 1 2 3 4
g. Help pupils with their individual experiments, projects, or other
hands-on experiences.……………………………………………. . 1 2 3 4
h. Help pupils with group experiments, projects, or other hands-on
experiences………..……………………………………….………. 1 2 3 4
b. Please circle the letters associated with the three teacher activities listed in 4a that best describe your class.
5. Indicate how often pupils do each of the following in your class(es):

Pupil activities month month week class
a. Listen and take notes in whole-class settings……………………… 1 2 3 4
b. Observe demonstrations in whole-class settings………………….. 1 2 3 4
c. Engage in discussions with peers………………………………….. 1 2 3 4
d. Engage in individual discussions with the teacher………………... 1 2 3 4
e. Engage in group discussions with the teacher……………………... 1 2 3 4
f. Do lab or field work, or other experiments or hands-on work.……. 1 2 3 4
g. Read silently….…...……………………………………………….. 1 2 3 4
h. Read orally……….………………………………………………... 1 2 3 4
i. Write essays or reports…………………………………………….. 1 2 3 4
j. Make presentations to the class....………………………………… 1 2 3 4
k. Work on or review homework in class...………….….…………… 1 2 3 4
l. Generate their own projects……………………………………….. 1 2 3 4
m. Work on paper and pencil exercises related to a specific topic..….. 1 2 3 4
n. Work on independent, long-term (at least one week long) projects. 1 2 3 4
o. Work on problems with no single best solution………………….... 1 2 3 4
p. Debate ideas or otherwise explain their reasoning….………….…. 1 2 3 4
q. Complete tests or quizzes………………………………………….. 1 2 3 4
r. Use hands-on models or manipulatives to solve problems…..……. 1 2 3 4
b. Please circle the letter associated with the three pupil activities listed in 5a that best describes your class.

ID#_________________________
Teacher Survey
6. How much do you use the following in evaluating pupils’ achievement? (Circle one for each line.)
Not Minor Moderate Very

Assessment strategies used importance importance important
a. Objective tests (e.g., multiple choice, true/false, short answer)…... 1 2 3 4
b. Performance on experiments, projects, or other hands-on
experiences..……………….…………….……………….………. 1 2 3 4
c. Systematic observation of pupils.……….…………….…………... 1 2 3 4
d. Oral reports……..……………………….…………….……...…… 1 2 3 4
e. Written reports……………………………………………………. 1 2 3 4
f. Peer evaluation …………………………………………………… 1 2 3 4
g. Self evaluation …………………………………………………… 1 2 3 4
h. Good classroom behavior………………………………………… 1 2 3 4
i. Bad classroom behavior…………………………………………... 1 2 3 4
Section IV: Background information

7. Are you male or female? ________
8. How old are you? ________
9. Where is your school located? Small village (less than 5,000 people) ___ Village (5,000 – 10,000) ___
Town (10,000 - 50,000) ___ City (50,000 - 100,000) ___ Large City (more than 100,000)___
10. What level of pupils do you teach (please refer to the class that was observed today)?
Primary school ____ Lower secondary school ____ Upper secondary school ____
11. What age pupils do you teach (please refer to the class that was observed today)? ____
12. a) How many years have you taught the pupils in the class that was observed earlier today? ____
b) How many hours per week, on average, do you teach the pupils in the class we observed
today? ____
13. What subject area(s) do you teach? Literature ____ History ____ Geography ____
Psychology ____ Foreign language ____ Mathematics ____ Chemistry ____ Physics____
Biology ____ Primary education ____ Other (please specify) ________________________
14. How many years have you been a classroom teacher? ____

ID#_________________________
Teacher Survey
15. How many years have you taught at your current school? ____
16. What is the highest level of schooling that you have completed?
Less than secondary education ____ Secondary degree only ____
Some postsecondary education ____ Baccalaureate degree ____
Some post-baccalaureate education ____ Post-baccalaureate degree ____
17. What is your first language? ______________________
18. What is the primary language used for teaching in your school? _____________________
19. Have you participated in professional development activities sponsored by an outside agency (e.g.,
foundations, non-government organizations, other private organizations) in the last three years?
Yes ____ No ____ (If your answer is “No,” please skip to question 21.)
20. Please name all of the professional development programs in which you have participated:
a. ______________________________________________________________________
b. ______________________________________________________________________
c. ______________________________________________________________________
d. ______________________________________________________________________
(If one of the answers to this question is the “Reading and Writing
for Critical Thinking” project, please skip to question 25.)
21. Are you familiar with the “Reading and Writing for Critical Thinking” (RWCT) project?
Yes ____ No ____ (If the answer is “No,” please skip to the “Closing” at the end of this survey)
22. How are you familiar with RWCT? (Please check all that apply.)
I have seen it advertised ____ I know people who have participated ____
I applied to participate myself ____
23. How familiar are you with RWCT practices?
Not familiar ____ Somewhat familiar ____ Very familiar ____

ID#_________________________
Teacher Survey
24. To what extent have you tried to adopt RWCT practices into your own teaching?
Not at all ____ Sometimes ____ As often as possible ____ Always ____
(Please skip to the “Closing” at the end of this survey.)
Section V: Experiences with RWCT

25. What year did you begin your participation with RWCT? ________
26. How many RWCT workshops have you attended? _______
27. In what other ways do you stay involved with RWCT? (Check all that apply.)
I attend ongoing meetings ____ I communicate with RWCT coordinators ____
I communicate with RWCT participants ____ I work as a RWCT teacher trainer ____
I write for RWCT publications ____ I attend monthly meetings ____
28. How did you find out about RWCT? (Check all that apply.)
I saw advertisements ____ I knew others who had participated ____
Someone at school informed me about it____ Other (please specify) ________________________
29. Why did you first participate in RWCT? (Check all that apply.)
I was curious ____ I was asked by my school ____ I was required by my school ____
Other (please specify) _____________________________________________________
30. Please indicate your level of agreement with the following statements. (Circle one for each line.)

disagree disagree agree agree
a. I have enjoyed my participation in RWCT workshops………………….. 1 2 3 4
b. RWCT techniques have helped me improve my teaching………………. 1 2 3 4
c. RWCT techniques have improved my pupils’ learning……………….… 1 2 3 4
d. I would recommend RWCT workshops to my colleagues……………… 1 2 3 4
e. RWCT principles should be taught broadly to teachers in my country…. 1 2 3 4
f. Use of RWCT techniques detract from other teaching 1 2 3 4
responsibilities…………………………………………………………..
g. Pupils learn less course material when I use RWCT ideas……………… 1 2 3 4

ID#_________________________
Teacher Survey
31. Think about your teaching before and after attending the RWCT workshops. List at least three
descriptors about yourself as a teacher:
a. before the RWCT training b. after the RWCT training
________________________________ ________________________________
________________________________ ________________________________
________________________________ ________________________________
32. What (if any) changes have you made in the way you prepare for lessons as a result of the RWCT
training? Please describe the changes:
33. How much, in your view, are the RWCT strategies different from your previous instructional
practices?

ID#_________________________
Teacher Survey
34. In your opinion, what are three positive and negative effects of applying RWCT strategies in
teaching?
a. positive b. negative
________________________________ ________________________________
________________________________ ________________________________
________________________________ ________________________________
35. If you could make a change in the RWCT program, what would it be?
36. Think about any changes you may have noticed in your pupils’ behavior since you started applying
RWCT strategies in teaching. How have your pupils’ behavior changed in terms of the following?
It has It has There have It has It has

become become been no become become
much somewhat major somewhat much
worse worse differences better better
a. Individual involvement during the lesson…………….… 1 2 3 4 5
b. Cooperation with other pupils………………………….. 1 2 3 4 5
c. Access to, and retention of, the information presented.… 1 2 3 4 5
d. Their enthusiasm for expanding the acquired knowledge 1 2 3 4 5
e. Their relationship with me, the teacher………….……… 1 2 3 4 5
Closing
Thank you again for completing this survey and allowing us to observe your class. The results from this
survey will be used to learn more about effective teaching strategies. Your thoughts are very valuable in
this process. Please return the survey now.

Classroom Observation Protocol, American Institutes for Research ID#_________________________
CLASSROOM OBSERVATION PROTOCOL
Prepared by:
February 2001
Appendix A American Institutes for Research

BACKGROUND INFORMATION
School or institution ____________________________________ Date ___________________
Teacher: First name ___________________ Last name _____________________
Time beginning ___________________ Time ending __________________
Primary school ____ Secondary school ____
Age of pupils ____ Number of pupils ____ (Number Male ____ Number Female ____)
Language of instruction _________________________
Primary language of most pupils _______________________
Subject area _________________________
Activity(ies) observed: 1) Lecture ___ 2) Discussion ___ 3) Small group ___ 4) Experiment ___ 5) Other ___
(Separate on answer sheet by columns: “1,3,4” for example)
Text(s) used ________________________________________________________________________
Other instructional materials _________________________________________________________
Communication pattern (percent time):
Teacher to pupil
Teacher to pupil to teacher
Pupil to pupil
Teacher to pupil to pupil to teacher
Total (should equal 100%)
This rubric is to be used for classroom observations for the RWCT evaluation. Observers should take brief
notes as unobtrusively as possible for one class period and then answer the questions as soon as possible after
the actual observation. Please also try to acquire a copy of the lesson plan for the class that day.
The ten categories of interest are:
1. Higher order thinking 6. Teacher interaction with pupils

2. Deep knowledge 7. Classroom organization
3. Substantive conversation 8. Teacher wait time
4. Connections to the world beyond the classroom 9. Classroom instruction
5. Lesson plan
Detailed information about the first four categories is included in the next few pages. The others categories
should be self-explanatory.

CLASSROOM OBSERVATION PROTOCOL
From Newmann, Secada, and Wehlage:
1. Higher order thinking 5 4 3 2 1
5 = Almost all pupils, almost all of the time, are performing HOT.
4 = Pupils are engaged in at least one major activity during the lesson in which they perform HOT
operations. This activity occupies a substantial portion of the lesson and many pupils are performing
HOT.
3 = Pupils are primarily engaged in routine LOT operations during a good share of the lesson. There is at
least one significant question or activity in which some pupils perform some HOT operations.
2 = Pupils are primarily engaged in LOT, but at some point they perform HOT as a minor diversion within
the lesson.
1 = Pupils are engaged only in LOT operations; i.e., they either receive, or recite, or participate in routine
practice, and in no activities during the lesson do pupils go beyond LOT.
2. Deep knowledge 5 4 3 2 1
5 = Knowledge is very deep because during the lesson almost all pupils do most of the following: sustain a
focus on a significant topic; or demonstrate their understanding of the problematic nature of
information and/or ideas; or demonstrate complex understanding by arriving at a reasoned, supported
conclusion; or explain how they solved a complex problem. In general, pupils’ reasoning,
explanations and arguments demonstrate fullness and complexity of understanding.
4 = Knowledge is relatively deep because either the teacher or the pupils provide information, arguments
or reasoning that demonstrate the complexity of an important idea. During the lesson many pupils do
at least one of the following: sustain a focus on a significant topic for a period of time; or demonstrate
their understanding of the problematic nature of information and/or ideas; or demonstrate
understanding by arriving at a reasoned, supported conclusion; or explain how they solved a relatively
complex problem.
3 = Knowledge is treated unevenly during instruction; i.e., deep understanding of something is countered
by superficial understanding of other ideas. At least one significant idea may be presented in depth
and its significance grasped, but in general the focus is not sustained.
2 = Knowledge remains superficial and fragmented; while some key concepts and ideas are mentioned or
covered, only a superficial acquaintance or understanding of these complex ideas is evident.
1 = Knowledge is very thin because it does not deal with significant topics or ideas; the teacher and pupils
are involved in the coverage of simple information which they are to remember.

3. Substantive conversation 5 4 3 2 1
Substantive conversation has three features:
a) The talk is about subject matter in the discipline and includes higher order thinking, such as making
distinctions, applying ideas, forming generalizations, or raising questions; not just reporting of
experiences, facts, definitions, or procedures.
b) The conversation involves sharing of ideas. Sharing is best illustrated when participants explain
themselves or ask questions in complete sentences, and when they respond directly to comments of
previous speakers.
c) The dialogue builds coherently on participants’ ideas to promote improved collective understanding of a
theme or topic (which does not necessarily require an explicit summary statement).
5 = All three features of substantive conversation occur, with at least one example of sustained
conversation, and almost all pupils participate.
4 = All three features of substantive conversation occur, with at least one example of sustained
conversation, and many pupils participate.
3 = Features “b” (sharing) and/or “c” (coherent promotion of collective understanding) occur and involve
at least one example of sustained conversation (i.e., at least 3 consecutive interchanges).
2 = Features “b” and/or “c” occur briefly and involve at least one example of two consecutive interchanges.
1 = Virtually no features or substantive conversation occur during the lesson.

4. Connections to the world beyond the classroom 5 4 3 2 1
5 = Pupils study or work on a topic, problem, or issue that the teacher and pupils see as connected to their
personal experiences or actual contemporary public situations. Pupils recognize the connections
between classroom knowledge and situations outside the classroom. They explore these connections
in ways that create personal meaning and significance for the knowledge. This meaning and
significance is strong enough to lead pupils to become involved in an effort to influence a larger
audience beyond their classroom in one of the following ways: by communicating knowledge to others
(including within the school), advocating solutions to social problems, providing assistance to people,
or creating performances or products with utilitarian or aesthetic value.
4 = Pupils study or work on a topic, problem, or issue that the teacher and pupils see as connected to their
personal experiences or actual contemporary public situations. Pupils recognize the connections
between classroom knowledge and situations outside the classroom. They explore these connections
in ways that create personal meaning and significance for the knowledge. However, there is no effort
to use the knowledge in ways that go beyond the classroom to actually influence a larger audience.
3 = Pupils study a topic, problem, or issue that the teacher succeeds in connecting to pupils’ actual
experiences or to contemporary public situation. Pupils recognize some connections between
classroom knowledge and situations outside the classroom, but they do not explore the implications of
these connections, which remain abstract or hypothetical. There is no effort to actually influence a
larger audience.
2 = Pupils encounter a topic, problem, or issue that the teacher tries to connect to pupils’ experiences or to
contemporary public situations; i.e., the teacher informs pupils that there is potential value in the
knowledge being studied because it relates to the world beyond the classroom. For example, pupils
are told that understanding Middle East history is important for contemporary politicians trying to
bring peace to the region; however, the connection is unspecified and there is no evidence that pupils
make the connection.
1 = The lesson topic and activities have no clear connection to anything beyond themselves; the teacher
offers no justification beyond the need to perform well in school.

5. Lesson Plan
Please acquire today’s lesson plan from the classroom teacher and evaluate the primary activity in it based on
the following information:
a) Organization of information 3 2 1
Consider the extent to which the task asks pupils to organize, interpret, evaluate, or synthesize complex
information, rather than to retrieve or to reproduce isolated fragments of knowledge or to repeatedly apply
previously learned algorithms and procedures.
3 (High) = The task should call for interpretation of nuances of a topic that go deeper than
surface exposure or familiarity and draw on external sources of information in ways
that reflect unique ideas and present potentially divergent ideas as well as pupil
experiences to analyze data.
2 (Moderate) = Pupils are asked to gather information that indicate some selectivity and organizing
beyond mechanical copying, but are not asked for interpretation, evaluation, or
synthesis.
1 (Low) = Pupils are asked to do mechanical copying and are not asked for interpretation,
evaluation, or synthesis.
b) Consideration of alternatives 3 2 1
To what extent does success in the task require consideration of alternative solutions, strategies,
perspectives and points of view?
3 (High) = The task should clearly involve pupils in generating and considering alternatives,
either through explicit presentation of the alternative perspectives and
interpretations or through an activity that cannot be successfully completed
without examination of alternatives implicit in the work. It is not necessary that
pupils’ final conclusions include listing or weighing of alternatives, but this could
be an impressive indicator that it was an expectation of the task.
2 (Moderate) = The task involves some consideration of alternatives, even if pupils are not asked
to generate them. It can include opportunities to examine alternatives, even if it
does not require pupils to examine alternatives to complete the task.
1 (Low) = Pupils do not need to consider alternatives to complete the task.

Adapted from Alan Crawford and Sam Mathews:
6. Teacher interaction with pupils 3 2 1
3 = Teacher encourages widely different responses in classroom discussion.

2 = Teacher encourages some divergent responses in classroom discussion.
1 = Teacher requires formal recitation of response.
7. Classroom organization 3 2 1
3 = Pupils move around the classroom to work collaboratively on classroom assignments.

2 = Pupils are seated in one or more groups so that they may work collaboratively on classroom
assignments.
1 = Pupils are seated at desks, facing the teacher, and work independently on classroom assignments.
8. Teacher wait time 3 2 1
3 = Teacher frequently provides wait time for pupils to answer questions.

2 = Teacher sometimes provides wait time for pupils to answer questions.
1 = Teachers rarely provide wait time for pupils to answer questions.
Adapted from the Evaluation of the Eisenhower Professional Development Program State and Local
Activities:
9. Classroom instruction
Please indicate the approximate number of minutes during the period that the teacher spent on the following
activities. Please also make sure that you are accounting for all minutes during the class period.
Approximate
Activities number of minutes
a. Lecturing to the class…………………………………………………….….…………… _________
b. Providing demonstrations to the class (including lab demonstrations)….….…………… _________
c. Leading whole class discussions, in which the teacher does most of the talking……….. _________
d. Listening to class-led discussions, in which the pupils do most of the talking…………. _________
e. Having pupils work in small groups……….…...………………………………….……. _________
f. Having pupils work individually……….…...…………………………………………… _________
g. Performing routine administrative tasks (e.g., taking attendance, making
announcements, classroom management, etc.)…………………………………………... _________
h. Helping pupils with their individual experiments, projects, or other hands-on
experiences………………………………………………………………………………. _________
i. Helping pupils with group experiments, projects, or other hands-on experiences……… _________
j. Other: (please specify)___________________________________________________ _________
TOTAL _________

APPENDIX B
4th and 5th Year Pupil Survey Scoring Rubric
4TH – 5TH YEAR PUPIL SURVEY: SCORING RUBRIC FOR OPEN-

ENDED ITEMS
QUESTION #1: GOOD/BAD RULE
This rule can be good because…
Breadth (aB)
blank No response
0 Unrelated/Off-topic/Circular reasoning/Merely states “good” or “bad”
1 One related response
2 More than one related response
Depth (aD)
blank No response
0 Unrelated/Off-topic/No link to prompt/Circular reasoning
1 Link to prompt but no justification for idea
2 Link to prompt and some justification for idea
3 Link to prompt and thorough justification for idea (for example, discusses experiences)
This rule can be bad because…
Breadth (bB)
blank No response
0 Unrelated/Off-topic/Circular reasoning/Merely states “good” or “bad”
Depth (bD)
blank No response
Decide whether you think this rule is more good than bad, and explain why you think that way:
Breadth (cB)
blank No response
0 Unrelated/Off-topic/Non-response
1 Merely states that the rule is more good or bad
2 States that the rule is good or bad and provides related explanation
Appendix B 1 American Institutes for Research

Depth (cD)
blank No response
QUESTION #2: KIOSK

Decision to place the kiosk in a particular location…
Breadth (aB)
blank No response
Depth (aD)
blank No response
Decision about why this place is better than other places…
Breadth (bB)
blank No response
0 Unrelated/Off-topic/No comparison made
Depth (bD)
blank No response
0 Unrelated/Off-topic/No link to prompt/Circular reasoning/No comparison made

7TH – 8TH YEAR PUPIL SURVEY: SCORING RUBRIC FOR OPEN-

ENDED ITEMS
QUESTION #1: IMMIGRATION/EMIGRATION
Emigration/immigration can be beneficial because…
Breadth (aB)
blank No response
0 Unrelated/Off-topic/Circular reasoning/Merely states “beneficial” or “harmful”
Depth (aD)
blank No response
0 Unrelated/Off-topic response/No link to prompt/Circular reasoning
Emigration/immigration can be harmful because…
Breadth (bB)
blank No response
0 Unrelated/Off-topic/Circular reasoning/Merely states “beneficial” or “harmful”
Depth (bD)
blank No response
3 Link to prompt and thorough justification for idea (e.g., discusses experiences)
Decide whether you think emigration is more beneficial or harmful, and explain why you think that way:
Breadth (cB)
blank No response
1 Merely states that emigration is more “beneficial” or “harmful”
2 States that emigration/immigration is beneficial or harmful and provides related explanation

Depth (cD)
blank No response
QUESTION #2: KIOSK
Decision to place the kiosk in a particular location…
Breadth (aB)
blank No response
Depth (aD)
blank No response
3 Link to prompt and thorough justification for idea (e.g., discusses experiences)
Decision not to place the kiosk elsewhere…
Breadth (bB)
blank No response
0 Unrelated/Off-topic/ No comparison made
Depth (bD)
blank No response
0 Unrelated/Off-topic/No link to prompt/Circular reasoning/No comparison made

Teacher Survey Scoring Rubric
TEACHER SURVEY: SCORING RUBRIC FOR OPEN-ENDED ITEMS

QUESTION #1: EXCELLENT PUPIL
Points
0 Only provides answers such as “obedient,” “answers questions correctly,”

“smart,” “works hard,” and “pays attention.”
Lists CT Discusses CT Discusses CT

characteristic* characteristic* characteristic in detail*
1 1 characteristic
2 2 characteristics 1 characteristic
3 3 characteristics 2 characteristics 1 characteristic
4 4+ characteristics 3+ characteristics 2+ characteristics
*
Examples of critical thinking:
a) Forms original opinions; e) Works cooperatively with others to construct meaning;

b) Chooses rationally among competing ideas; f) Appreciates different points of view; and
c) Solves problems; g) Recognizes the ways people’s backgrounds can influence
d) Debates ideas responsibly; attitudes and perceptions.
QUESTION #3: TEACHING ACTIVITIES
Points
0 Only provides answers such as “obedient,” “answers questions correctly,”

“smart,” “works hard,” and “pays attention.”
CT is a minor part CT is a major part

Just mentions CT* of the activity* of the activity*
1 1 characteristic
2 2 characteristics 1 characteristic
3 3 characteristics 2 characteristics 1 characteristic
4 4+ characteristics 3+ characteristics 2+ characteristics
*
Examples of critical thinking:
a) Forms original opinions; e) Works cooperatively with others to construct meaning;

b) Chooses rationally among competing ideas; f) Appreciates different points of view; and
c) Solves problems; g) Recognizes the ways people’s backgrounds can influence
d) Debates ideas responsibly; attitudes and perceptions.

APPENDIX C
APPENDIX C: STATISTICAL METHODS AND DETAILED TABLES

OF RWCT EVALUATION FINDINGS
A variety of software packages and statistical techniques were used to analyze the data for this
evaluation report. The confirmatory factor analysis capabilities of LISREL 8 and a National
Institutes for Mental Health program called Factor Score were used to develop preliminary
psychometric models of variables such as breadth and depth of critical thinking as well as an overall
measure of critical thinking. These programs were also used to create the composite “authentic
pedagogy” score that is highlighted in Tables 12 and C,XIV.
Analyses were then conducted using SAS 8.0, a statistical software package that is not only able to
produce descriptive statistics and regression analysis but also to account for complex survey designs
such as the one used in this study (i.e., the selection of pupils nested in classrooms). AIR used the
advanced regression capabilities of SAS to account for the complex sampling design and the fact that
pupils are nested in classrooms. There was no accounting, however, for the fact that classrooms are
nested within individual countries in the aggregate analyses. Finally, proportional reductions in
regression analysis were computed in Microsoft Excel.
Following is a list of statistical techniques used to produce the tables and charts in this report:
• Descriptive analysis, including means and frequency distributions: Exhibits 3 (and C,VI) and
4 (and C,XI), and Tables 17, 18, 19 (C,XXII), 20 (C,XXIII), 21 (C,XXIV), 22 (C,XXV), 23
(C,XXVI), 24 (C,XXVII), and (C,XXVIII);
• Student’s t-test: Tables 2, 3, 9, (C,VIII), 10 (and C,IX), (C,XII), 11 (C,XIII), 12 (C,XIV), 13

(C,XVI), (C,XVII), (C,XVIII), and 16 (C,XXI);
• T-test that use a Bonferroni adjustment to account for the fact that the pupil samples are
nested within classrooms and not random: Tables 4, 5 (and C,II), 6 (and C,III), (C,VII), 14
(C,XIX), and 15 (C,XX);
• Regression analysis that accounts for complex survey designs (i.e., SAS’s “Survey
Regression” procedure) and the calculation of proportional reductions: Tables 7 (and C,IV) 8
(C,V), and (CX); and
• Correlations: Exhibits 3 (and C,VI) and 4 (and C,XI).
Appendix C 1 American Institutes for Research

TABLE I: TEACHER PARTICIPATION IN PROFESSIONAL

DEVELOPMENT ACTIVITIES IN THE PAST THREE YEARS
Percentage of
Mean Number of
Teachers
PD Activities per N
Participating in
Teacher
PD Activities
Overall 1.59 68.54 285
Czech Republic 2.35 80.00 70
Kyrgyzstan 0.64 52.05 72
Latvia 2.20 88.64 71
Macedonia 1.19 66.07 72
TABLE II: DIFFERENCES IN CRITICAL THINKING SCORES BETWEEN RWCT PUPILS AND
PUPILS WHOSE TEACHERS DID NOT PARTICIPATE IN PROFESSIONAL DEVELOPMENT
ACTIVITIES IN THE PAST THREE YEARS
Czech
Score N Score N Score N Score N Score N
Cohort 1
RWCT pupils 13.90 210 15.04 48 14.20 51 11.89 54 14.58 57
Control pupils whose teachers had no PD 11.39 84 12.83 6 11.72 54 12.22 9 9.13 15
Difference 2.51 * 2.21 2.47 * -0.33 5.45 *
Cohort 2
RWCT pupils 13.49 228 14.21 48 14.05 63 11.92 63 14.02 54
Control pupils whose teachers had no PD 11.17 117 11.83 18 11.65 51 14.50 6 9.83 42
Difference 2.32 * 2.38 2.40 * -2.58 4.19 *
TABLE III: DIFFERENCES IN CRITICAL THINKING SCORES BETWEEN RWCT PUPILS AND
PUPILS WHOSE TEACHERS PARTICIPATED IN OTHER CONTENT- OR INSTRUCTION-BASED
PROFESSIONAL DEVELOPMENT ACTIVITIES IN THE PAST THREE YEARS
Czech
Cohort 1
RWCT pupils 13.90 210 15.04 48 14.20 51 11.89 54 14.58 57
Control pupils whose teachers had content-based PD 12.59 162 12.74 78 17.00 3 12.30 63 12.22 18
Difference 1.31 * 2.30 * -2.80 -0.41 2.36
RWCT pupils 13.90 210 15.04 48 14.20 51 11.89 54 14.58 57
Control pupils whose teachers had instruction-based PD 12.62 210 12.86 72 16.33 3 12.38 63 12.43 72
Difference 1.28 * 2.18 * -2.13 -0.49 2.15
Cohort 2
RWCT pupils 13.49 228 14.21 48 14.05 63 11.92 63 14.02 54
Control pupils whose teachers had content-based PD 12.92 161 12.64 56 13.83 6 12.56 90 17.67 9
Difference 0.57 1.57 0.22 -0.64 -3.65 *
RWCT pupils 13.49 228 14.21 48 14.05 63 11.92 63 14.02 54
Control pupils whose teachers had instruction-based PD 13.06 173 13.38 68 13.83 12 11.22 54 14.79 39
Difference 0.43 0.83 0.22 0.70 -0.77

TABLE IV: PROPORTIONAL REDUCTIONS IN CRITICAL THINKING SCORE DIFFERENCES
Cohort 1 (N=417) Cohort 2 (N=434)

Difference in Propor- Difference in Propor-
RWCT/Control- tional RWCT/Control- tional
Variable Categories Variables group Score Reduction group Score Reduction
RWCT 2.19 1.72
RWCT + Background Gender (Boys) 2.20 0.00 1.63 0.05

Characteristics Pupil Age 2.10 0.04 1.75 -0.02
Different Language 2.18 0.00 1.75 -0.02
Reading Materials at Home 2.23 -0.02 1.70 0.01
Electronic Devices at Home 2.21 -0.01 1.70 0.01
Luxury Items at Home 2.15 0.02 1.72 0.00
Urbanicity 2.20 0.00 1.72 0.00
Combination of 2.04 0.07 1.67 0.03
Background Characteristics
RWCT + Teacher Teacher Age 2.35 -0.07 1.65 0.04
Characteristics Years with Class 2.18 0.00 1.73 -0.01
Hours per Week with Class 2.13 0.03 1.62 0.06
Interaction between Years 2.14 0.02 1.65 0.04
and Hours with Class
Years as Teacher 2.34 -0.07 1.48 0.14
Years at School 2.20 0.00 1.73 -0.01
Combination of Teacher 2.51 -0.15 1.40 0.19
Characteristics
RWCT + Class Subject General 2.15 0.02 1.71 0.01
Humanities 2.20 0.00 1.63 0.05
Social Science 2.19 0.00 1.72 0.00
Physical Science 2.23 -0.02 1.64 0.05
Fine Arts 2.12 0.03 1.72 0.00
Combination of Class Types 2.18 0.00 1.69 0.02
RWCT+ Classroom Teacher-Pupil 2.32 -0.06 1.31 0.24
Communication Pattern Teacher-Pupil-Teacher 2.16 0.01 1.69 0.02
Pupil-Pupil 2.02 0.08 0.96 0.44
Teacher-Pupil-Pupil-Teacher 2.18 0.00 1.46 0.15
Combination of 2.21 -0.01 0.85 0.51
Communication Patterns
RWCT + Integration of Integrating CT Principles 1.25 0.43 0.83 0.52
Critical Thinking (CT)

TABLE V: PROPORTIONAL REDUCTIONS IN CRITICAL THINKING

SCORE DIFFERENCES BY COUNTRY
Overall Czech Kyrgyzstan Latvia Macedonia

Republic
Difference Difference Difference Difference Difference

in RWCT/ Propor- in RWCT/ Propor- in RWCT/ Propor- in RWCT/ Propor- in RWCT/ Propor-
Control- tional Control- tional Control- tional Control- tional Control- tional
group Reduc- group Redu- group Reduc- group Reduc- group Reduc-
Score tion Score tion Score tion Score tion Score tion
RWCT for Cohort 1 2.19 3.47 1.95 -1.49 4.95
RWCT +
Integrating CT Principles 1.25 0.43 2.19 0.37 -0.52 1.27 -1.53 -0.02 4.56 0.08
Sample Size 417 108 102 102 105
RWCT for Cohort 2 1.72 2.04 2.48 -0.64 3.15
RWCT +
Integrating CT Principles 0.83 0.52 1.91 0.06 0.40 0.84 -0.61 0.05 1.37 0.56
Sample Size 434 101 114 111 108
TABLE VI: PUPIL CRITICAL THINKING SCORES BY NUMBER

OF YEARS OF RWCT TEACHER INVOLVEMENT

Cohort 1
1 year + 13.59 93 18.00 9 15.67 6 11.77 48 14.77 30
2 years 14.35 81 14.40 42 14.92 24 12.83 6 13.56 9
3 years 13.04 24 12.33 15 14.22 9
4 years 15 15 14.50 6 15.33 9
Absolute value of 0.04 .30 * 0.14 0.08 0.01
correlation
Cohort 2
1 year + 12.93 90 12.33 9 13.33 3 11.94 51 14.96 27
2 years 14.08 87 14.89 36 13.73 30 11.83 12 15.00 9
3 years 13.67 42 11.67 3 14.48 27 12.33 12
4 years 12.44 9 14.00 3 11.67 6
Absolute value of 0.04 0.09 0.07 0.01 0.22
correlation
+
First year teachers include those who joined RWCT in the fall of 2000.
* Statistically significant correlation at the 0.05 level.

TABLE VII: DIFFERENCES IN PUPIL CRITICAL THINKING SCORES

BY SOURCE OF RWCT TEACHER TRAINING
Czech
Overall Republic + Kyrgyzstan Latvia + Macedonia
Cohort 1
International volunteers 15.00 15 0 14.50 6 0 15.33 9
In-country teacher educators 13.83 198 15.04 51 14.16 45 11.89 54 14.44 48
Difference 1.17 0.34 0.90
Cohort 2
In-country teacher 13.52 219 14.21 48 14.05 60 11.92 63 14.31 48
educators
Difference -1.08 -0.05 -2.64
+
Randomized samples of RWCT teachers did not
include first-year teachers in these countries.
TABLE VIII: DIFFERENCES BETWEEN RWCT TEACHERS AND CONTROL-GROUP TEACHERS

WHO HAD NOT PARTICIPATED IN PROFESSIONAL DEVELOPMENT ACTIVITIES OVER THE PAST
THREE YEARS IN SCORES RELATED TO INTEGRATING CRITICAL THINKING PRINCIPLES INTO
TEACHING PRACTICES

Integrating CT Principles
RWCT teachers 2.67 142 2.75 28 3.29 38 2.05 39 2.62 37
Teachers with no PD 1.41 64 1.13 8 1.50 34 2.00 4 1.22 18
Difference 1.26 * 1.63 * 1.79 * 0.05 1.40 *
TABLE IX: DIFFERENCES BETWEEN RWCT TEACHERS AND CONTROL-GROUP TEACHERS

WHO PARTICIPATED IN OTHER CONTENT- OR INSTRUCTION-BASED PROFESSIONAL
DEVELOPMENT ACTIVITIES IN THE PAST THREE YEARS IN SCORES RELATED TO INTEGRATING
CRITICAL THINKING PRINCIPLES INTO TEACHING PRACTICES
Czech
Integrating CT Principles
RWCT teachers 2.67 142 2.75 28 3.29 38 2.05 39 2.62 37
Teachers with content-based PD 1.99 105 2.10 42 3.67 3 1.82 51 1.89 9
Difference 0.68 * 0.65 -0.38 0.23 0.73
RWCT teachers 2.67 142 2.75 28 3.29 38 2.05 39 2.62 37
Teachers with instruction-based PD 2.04 127 2.04 46 3.00 5 1.79 39 2.16 37
Difference 0.63 * 0.71 0.29 0.26 0.46

TABLE X: PROPORTIONAL REDUCTIONS IN SCORES RELATED TO INTEGRATING CRITICAL

THINKING PRINCIPLES INTO TEACHING PRACTICES
Integration of CT Principles
(N=285)
Difference in
RWCT/Control Proportional
Variable Categories Variables -group Score Reduction
RWCT 1.13
RWCT + Male 1.12 0.01

Background Teacher Age 1.11 0.01
Characteristics Urbanicity 1.14 -0.01
Years as Teacher 1.10 0.02
Years at School 1.15 -0.02
Combination of Background 1.16 -0.03
Characteristics
RWCT + Class Primary Education 1.12 0.00
Subject Humanities 1.11 0.02
Social Sciences 1.12 0.00
Science or Math 1.14 -0.01
Fine Arts 1.13 0.00
Combination of Classes Observed 1.18 -0.05
TABLE XI: CHANGES IN SCORES RELATED TO INTEGRATING CRITICAL THINKING PRINCIPLES

INTO TEACHING PRACTICES BASED ON YEARS OF RWCT INVOLVEMENT
Czech
Integrating Critical Thinking Principles
1 year + 2.28 60 3.20 5 3.00 3 2.00 33 2.42 19
2 years 2.83 53 2.65 23 3.17 18 2.33 6 3.00 6
3 years 3.10 21 3.36 14 2.57 7
4 years 3.38 8 4.00 3 3.00 5
Correlation 0.32 * -0.19 0.33 * 0.12 0.16
+
First year teachers include those who joined RWCT in the fall of 2000.
TABLE XII: DIFFERENCES IN SCORES RELATED TO INTEGRATING CRITICAL THINKING

PRINCIPLES INTO TEACHING PRACTICES BY SOURCE OF RWCT TEACHER TRAINING
Czech
Overall Republic + Kyrgyzstan Latvia + Macedonia
Integration CT Principles
In-country teacher educators 2.63 134 2.75 28 3.23 35 2.05 39 2.56 32
Difference 0.75 * 0.77 * 0.44
+
Randomized samples of RWCT teachers did not include first-year teachers in these countries.

TABLE XIII: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP CLASSES

IN CLASSROOM COMMUNICATION PATTERNS

% Time (SD) % Time (SD) % Time (SD) % Time (SD) % Time (SD)
Teacher – Pupil RWCT Teachers 14.01 (17.04) 25.13 (21.63) 10.46 (7.45) 6.23 (11.41) 16.24 (19.47)
Control Group 29.60 (27.06) 42.61 (30.63) 25.88 (22.46) 21.25 (20.08) 26.71 (28.98)
Difference -15.59 * -17.48 * -15.42 * -15.02 * -10.47
Sample Size 185 70 72 71 72
Teacher – Pupil - RWCT Teachers 31.49 (27.94) 38.56 (29.38) 34.90 (24.22) 16.33 (24.75) 37.84 (28.35)
Teacher Control Group 45.78 (31.03) 40.45 (28.88) 60.06 (27.29) 34.53 (27.37) 48.00 (35.13)
Difference -14.30 * -1.88 -25.16 * -18.20 * -10.16
Sample Size 185 70 72 71 72
Pupil – Pupil RWCT Teachers 26.39 (23.82) 28.28 (26.17) 30.01 (17.84) 15.13 (24.37) 32.89 (23.30)
Control Group 11.53 (19.64) 12.76 (17.64) 5.24 (10.21) 10.00 (15.08) 17.71 (28.91)
Difference 14.86 * 15.52 * 24.78 * 5.13 15.18 *
Sample Size 185 70 72 71 72
Teacher – Pupil - RWCT Teachers 12.98 (16.37) 8.03 (11.44) 24.63 (19.75) 5.90 (9.59) 12.76 (15.77)
Pupil –Teacher Control Group 5.82 (12.01) 4.18 (8.73) 8.82 (17.47) 5.78 (6.49) 4.71 (12.42)
Difference 7.16 * 3.85 15.81 * 0.12 8.04 *
Sample Size 185 70 72 71 72

TABLE XIV: DIFFERENCES IN AUTHENTIC PEDAGOGY AND OTHER CLASSROOM

OBSERVATION SCALES BETWEEN RWCT AND CONTROL-GROUP TEACHERS
Czech
% Time N % Time N % Time N % Time N % Time N
Overall “Authentic RWCT Teachers 0.61 124 0.39 32 0.96 38 0.68 17 0.42 37
Pedagogy” Score Control Group -0.72 129 -0.77 37 -0.68 34 -0.07 23 -1.13 35
Difference 1.33 * 1.16 * 1.64 * 0.75 * 1.55 *
Higher-order Thinking RWCT Teachers 3.77 124 3.69 32 3.97 38 3.82 17 3.62 37
Skills (5-point scale) Control Group 2.41 129 2.39 38 2.35 34 3.04 23 2.06 34
Difference 1.36 * 1.29 * 1.62 * 0.78 * 1.56 *
Substantive Conversation RWCT Teachers 3.52 124 3.16 32 3.97 38 3.59 17 3.32 37
(5-point scale) Control Group 2.24 129 2.08 38 2.18 34 2.83 23 2.09 34
Difference 1.28 * 1.08 * 1.80 * 0.76 * 1.24 *
World beyond the RWCT Teachers 2.99 111 2.42 19 3.37 38 3.18 17 2.81 37
Classroom (5-point scale) Control Group 2.19 112 2.00 21 2.06 34 2.74 23 2.06 34
Difference 0.80 * 0.42 1.31 * 0.44 0.75 *
Organization of RWCT Teachers 2.57 116 2.42 31 2.79 38 2.41 17 2.53 30
Information (3-point scale) Control Group 1.77 115 1.73 33 1.91 34 1.96 23 1.48 25
Difference 0.80 * 0.69 * 0.88 * 0.46 * 1.05 *
Consideration of RWCT Teachers 2.44 115 2.39 28 2.58 38 2.29 17 2.41 32
Alternatives (3-point scale) Control Group 1.65 114 1.69 32 1.71 34 1.91 23 1.28 25
Difference 0.79 * 0.71 * 0.87 * 0.38 1.13 *
Encouragement of RWCT Teachers 2.42 119 2.28 29 2.68 38 2.59 17 2.17 35
Alternative Answers Control Group 1.69 128 1.70 37 1.65 34 1.96 23 1.53 34
(3-point scale) Difference 0.73 * 0.57 * 1.04 * 0.63 * 0.64 *
Classroom Organization RWCT Teachers 2.06 124 2.19 32 2.05 38 2.06 17 1.97 37
Difference 0.61 * 0.66 * 0.64 * 0.62 * 0.53 *
Wait Time RWCT Teachers 2.55 123 2.32 31 2.79 38 2.71 17 2.43 37
Difference 0.55 * 0.43 * 0.82 * 0.27 0.58 *

TABLE XV: DESCRIPTIONS OF LOWER AND UPPER CLASSROOM

OBSERVATION RATINGS BASED ON AVERAGE SCORES
Score Description
Higher-order thinking skills
Pupils are primarily engaged in lower-order thinking (i.e., receive or recite factual information, or to employ rules and
2 algorithms through repetitive routines), but at some point they perform higher-order thinking as a minor diversion
within the lesson.
4 Pupils are engaged in at least one major activity during the lesson in which they engage in higher order thinking (i.e.,
perform operations that require them to combine facts and ideas to synthesize, generalize, explain, hypothesize, or
arrive at some conclusion or interpretation). This activity occupies a substantial portion of the lesson, and many pupils
are performing these activities.
Substantive conversation
2 The following activities occur briefly during the lesson and involve at least one instance of two consecutive
interchanges among participants in the discussion:
• The conversation involves sharing of ideas; and/or
• The dialogue builds coherently on participants’ ideas to promote improved collective understanding of a theme or
topic.
4 All three of the following activities occur with at least one example of sustained conversation, and many pupils
participate:
• The talk is about subject matter in the discipline and includes higher order thinking, such as making distinctions,
applying ideas, forming generalizations, or raising questions;
• The conversation involves sharing of ideas; and
• The dialogue builds coherently on participants’ ideas to promote improved collective understanding of a theme or
topic.
World beyond the classroom

2 Pupils encounter a topic, problem, or issue that the teacher tries to connect to pupils’ experiences or to contemporary
public situations; i.e., the teacher informs pupils that there is potential value in the knowledge being studied because it
relates to the world beyond the classroom. For example, pupils are told that understanding Middle East history is
important for contemporary politicians trying to bring peace to the region; however, the connection is unspecified and
there is no evidence that pupils make the connection.
4 Pupils study a topic, problem, or issue that the teacher succeeds in connecting to pupils’ actual experiences or to
contemporary public situation. Pupils recognize some connections between classroom knowledge and situations
outside the classroom, but they do not explore the implications of these connections, which remain abstract or
hypothetical. There is no effort to actually influence a larger audience.
Organization of Information
1 Pupils are asked to do mechanical copying and are not asked for interpretation, evaluation, or synthesis.
3 The task should call for interpretation of nuances of a topic that go deeper than surface exposure or familiarity and
draw on external sources of information in ways that reflect unique ideas and present potentially divergent ideas as
well as pupil experiences to analyze data.
Consideration of Alternatives
1 Pupils do not need to consider alternatives to complete the task.
3 The task should clearly involve pupils in generating and considering alternatives, either through explicit presentation of
the alternative perspectives and interpretations or through an activity that cannot be successfully completed without
examination of alternatives implicit in the work. It is not necessary that pupils’ final conclusions include listing or
weighing of alternatives, but this could be an impressive indicator that it was an expectation of the task.
Encouragement of Alternative Answers

1 Teacher requires formal recitation of response.
3 Teacher encourages widely different responses in classroom discussion.
Classroom organization
1 Pupils are seated at desks, facing the teacher, and work independently on classroom assignments.
3 Pupils move around the classroom to work collaboratively on classroom assignments.
Wait time
1 Teachers rarely provide wait time for pupils to answer questions.
3 Teacher frequently provides wait time for pupils to answer questions.

TABLE XVI: DIFFERENCES IN OBSERVED CLASSROOM ACTIVITIES

IN RWCT AND CONTROL-GROUP CLASSROOMS
Czech
% Time N % Time N % Time N % Time N % Time N
Lecture RWCT Teachers 7.81 124 9.18 32 7.47 38 5.97 17 7.82 37
Control Group 18.38 129 17.54 38 18.24 34 14.50 23 22.09 34
Difference -10.57 * -8.36 * -10.77 * -8.53 * -14.27 *
Demonstrations RWCT Teachers 6.30 124 8.83 32 2.27 38 4.35 17 9.16 37
Control Group 8.48 129 12.6 38 2.27 34 5.96 23 11.78 34
Difference -2.18 -3.77 0.00 -1.61 -2.62
Teacher-led RWCT Teachers 14.25 124 12.94 32 11.35 38 21.64 17 14.97 37
Discussion Control Group 19.59 129 8.85 38 21.25 34 21.54 23 28.62 34
Difference -5.34 * 4.09 -9.90 * 0.10 -13.65 *
Pupil-led RWCT Teachers 15.04 124 7.86 32 21.19 38 10.62 17 16.95 37
Difference 7.89 * 0.76 12.88 * 1.05 12.53 *
Small Group Work RWCT Teachers 19.99 124 14.1 32 25.36 38 22.21 17 18.53 37
Control Group 7.04 129 4.94 38 3.95 34 8.36 23 11.57 34
Difference 12.95 * 9.16 * 21.41 * 13.85 6.96
Work Individually RWCT Teachers 18.14 124 13.06 32 19.89 38 16.25 17 21.63 37
Control Group 20.10 129 14.71 38 27.30 34 24.87 23 15.71 34
Difference -1.96 -1.65 -7.41 -8.62 5.92
Routine Admin. RWCT Teachers 4.05 124 8.94 32 2.70 38 5.13 17 0.70 37
Work Control Group 6.65 129 10.62 38 8.60 34 5.30 23 1.18 34
Difference -2.60 * 1.68 -5.90 * -0.17 -0.48
Individual Projects RWCT Teachers 4.90 124 5.53 32 3.06 38 5.76 17 5.85 37
Control Group 3.83 129 6.06 38 2.69 34 5.08 23 1.63 34
Difference 1.07 -0.53 0.37 0.68 4.22 *
Group Projects RWCT Teachers 4.63 124 6.98 32 2.33 38 6.42 17 4.13 37
Control Group 2.64 129 5.16 38 1.74 34 0.48 23 2.19 34
Difference 1.99 1.82 0.59 5.94 * 1.94

TABLE XVII: DIFFERENCES IN REPORTS OF THE THREE MOST COMMON CLASSROOM

ACTIVITIES IDENTIFIED BY RWCT AND CONTROL-GROUP TEACHERS

% Rspndnts N % Rspndnts N % Rspndnts N % Rspndnts N % Rspndnts N
Control Group 13.67 139 18.42 38 14.71 34 6.25 32 14.29 35
Difference -2.71 3.46 -12.08 6.57 -6.18
Demonstrations RWCT Teachers 30.14 146 28.13 32 50 38 12.82 39 29.73 37
Control Group 33.81 139 34.21 38 64.71 34 15.63 32 20 35
Difference -3.67 -6.08 -14.71 -2.81 9.73
Teacher-led RWCT Teachers 15.75 146 25 32 13.16 38 10.26 39 16.22 37
Difference -24.54 * -22.37 -42.72 * -24.12 * -6.64
Difference 22.62 * 11.18 40.71 * 2.08 34.21 *
Control Group 30.94 139 55.26 38 20.59 34 25 32 20 35
Difference 32.76 * 13.49 60.99 * 33.97 * 25.95 *
Control Group 38.85 139 47.37 38 58.82 34 50 32 0 35
Difference -1.18 12.01 -16.71 -19.23 21.62 *
Control Group 20.14 139 31.58 38 20.59 34 18.75 32 8.57 35
Difference -5.07 -12.83 -10.06 -11.06 15.75
Group Projects RWCT Teachers 19.86 146 21.88 32 18.42 38 15.38 39 24.32 37
Control Group 10.79 139 15.79 38 14.71 34 6.25 32 5.71 35
Difference 9.07 * 6.09 3.71 9.13 18.61 *

TABLE XVIII: DIFFERENCES IN REPORTS OF THE MOST COMMON CLASSROOM ACTIVITIES

IDENTIFIED BY RWCT AND CONTROL-GROUP PUPILS (COHORT 2)

Control Group 72.82 206 90.57 53 56.86 51 68.75 48 74.07 54
Difference -20.62 * -9.32 -23.53 -16.37 -25.93 *
Demonstrations RWCT Teachers 12.72 228 6.25 48 15.87 63 9.52 63 18.52 54
Control Group 19.90 206 16.98 53 27.45 51 20.83 48 14.81 54
Difference -7.18 -10.73 -11.58 -11.31 3.70
Teacher-led RWCT Teachers 50.88 228 50.00 48 80.95 63 14.29 63 59.26 54
Difference -4.95 10.38 41.33 -37.80 * -5.56
Difference 14.92 * 3.14 16.43 13.89 25.93 *
Control Group 21.36 206 13.21 53 23.53 51 14.58 48 33.33 54
Difference 19.43 * 15.96 20.92 31.45 * 7.41
Control Group 47.09 206 67.92 53 45.10 51 52.08 48 24.07 54
Difference -6.30 -13.76 -14.94 3.47 0.00
Control Group 30.10 206 39.62 53 33.33 51 16.67 48 29.63 54
Difference 1.92 6.21 -1.59 2.38 5.56

TABLE XIX: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP PUPILS (COHORT 1)

IN WHAT THEY ENJOY MOST ABOUT THE WORK THEY DO IN CLASS
Czech
Learning generally RWCT Pupils 41.43 210 35.42 48 60.78 51 38.89 54 31.58 57
Control Group 49.76 207 50.00 60 45.10 51 47.92 48 56.25 48
Difference -8.33 -14.58 15.69 -9.03 -24.67
Activities that involve RWCT Pupils 42.38 210 56.25 48 25.49 51 53.70 54 35.09 57
active learning Control Group 29.95 207 35.00 60 33.33 51 31.25 48 18.75 48
Difference 12.43* 21.25 -7.84 22.45 16.34
Everything RWCT Pupils 8.10 210 14.58 48 7.84 51 1.85 54 8.77 57
Control Group 6.28 207 6.67 60 7.84 51 4.17 48 6.25 48
Difference 1.82 7.92 0.00 -2.31 2.52
Social activities RWCT Pupils 8.10 210 14.58 48 3.92 51 5.56 54 8.77 57
Control Group 6.28 207 11.67 60 0.00 51 10.42 48 2.08 48
Difference 1.82 2.92 3.92 -4.86 6.69
Succeeding RWCT Pupils 7.14 210 2.08 48 3.92 51 11.11 54 10.53 57
Control Group 5.80 207 6.67 60 3.92 51 4.17 48 8.33 48
Difference 1.35 -4.58 0.00 6.94 2.19
Extracurricular RWCT Pupils 2.86 210 8.33 48 0.00 51 1.85 54 1.75 57
Activities Control Group 5.80 207 11.67 60 5.88 51 4.17 48 0.00 48
Difference -2.94 -3.33 -5.88 -2.31 1.75
Nothing RWCT Pupils 0.95 210 0.00 48 0.00 51 0.00 54 3.51 57
Control Group 3.38 207 1.67 60 1.96 51 2.08 48 8.33 48
Difference -2.43 -1.67 -1.96 -2.08 -4.82
Working alone RWCT Pupils 1.43 210 2.08 48 1.96 51 1.85 54 0.00 57
Control Group 1.93 207 0.00 60 5.88 51 2.08 48 0.00 48
Difference -0.50 2.08 -3.92 -0.23 0.00
Chores RWCT Pupils 2.38 210 0.00 48 1.96 51 7.41 54 0.00 57
Control Group 0.00 207 0.00 60 0.00 51 0.00 48 0.00 48
Difference 2.38 0.00 1.96 7.41 0.00

TABLE XX: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP PUPILS (COHORT 2)

IN WHAT THEY ENJOY MOST ABOUT THE COURSE THAT DATA COLLECTORS OBSERVED
Czech

Learning generally RWCT Pupils 46.93 228 52.08 48 35.42 63 53.97 63 24.07 54
Control Group 50.00 206 58.49 53 55.56 51 52.94 48 51.85 54
Difference -3.07 -6.41 -20.14 1.03 -27.78*
Activities that involve RWCT Pupils 39.47 228 27.08 48 37.50 63 39.68 63 46.30 54
active learning Control Group 20.87 206 28.30 53 42.86 51 13.73 48 5.56 54
Difference 18.60* -1.22 -5.36 25.96* 40.74*
Social activities RWCT Pupils 8.77 228 14.58 48 0.00 63 6.35 63 14.81 54
Control Group 6.31 206 7.55 53 1.59 51 0.00 48 16.67 54
Difference 2.46 7.04 -1.59 6.35 -1.85
Everything RWCT Pupils 6.14 228 4.17 48 6.25 63 9.52 63 5.56 54
Control Group 6.31 206 0.00 53 4.76 51 5.88 48 12.96 54
Difference -0.17 4.17* 1.49 3.64 -7.41
Working with the RWCT Pupils 3.95 228 14.58 48 10.42 63 0.00 63 0.00 54
Teacher Control Group 6.31 206 15.09 53 3.17 51 0.00 48 0.00 54
Difference -2.36 -0.51 7.24 0.00 0.00
Nothing RWCT Pupils 3.07 228 2.08 48 4.17 63 1.59 63 7.41 54
Control Group 6.31 206 11.32 53 1.59 51 5.88 48 3.70 54
Difference -3.24 -9.24 2.58 -4.30 3.70
Succeeding RWCT Pupils 2.19 228 4.17 48 2.08 63 0.00 63 3.70 54
Control Group 3.40 206 0.00 53 1.59 51 5.88 48 5.56 54
Difference -1.21 4.17 0.50 -5.88 -1.85
Extracurricular RWCT Pupils 0.88 228 0.00 48 0.00 63 3.17 63 0.00 54
Activities Control Group 2.91 206 0.00 53 0.00 51 11.76 48 0.00 54
Difference -2.04 0.00 0.00 -8.59 0.00
It is easy RWCT Pupils 2.19 228 6.25 48 2.08 63 1.59 63 0.00 54
Control Group 1.94 206 3.77 53 1.59 51 1.96 48 0.00 54
Difference 0.25 2.48 0.50 -0.37 0.00

TABLE XXI: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP

TEACHERS IN VIEWS ABOUT TEACHING AND LEARNING
Czech
Agreemnt N Agreemnt N Agreemnt N Agreemnt N Agreemnt N
Enjoy my job as a teacher RWCT Teachers 3.75 146 3.63 32 3.82 38 3.62 39 3.92 37
Control Group 3.62 138 3.66 38 3.62 34 3.45 31 3.74 35
Difference 0.12* -0.03 0.20 0.16 0.18
Would not choose RWCT Teachers 2.08 144 2.19 31 2.00 37 2.23 39 1.92 37
teaching again Control Group 2.12 139 1.79 38 2.50 34 2.19 32 2.06 35
Difference -0.04 0.40 -0.50 0.04 -0.14
Decide what to teach RWCT Teachers 3.10 146 3.06 32 3.61 38 3.00 39 2.70 37
Control Group 3.14 138 2.89 38 3.65 34 3.16 31 2.91 35
Difference -0.05 0.17 -0.04 -0.16 -0.21
Select outside texts RWCT Teachers 3.68 145 3.66 32 3.66 38 3.67 39 3.75 36
Control Group 3.73 139 3.66 38 3.74 34 3.78 32 3.77 35
Difference -0.05 0.00 -0.08 -0.11 -0.02
Decide how to teach the RWCT Teachers 3.53 141 3.68 28 3.61 38 3.66 38 3.22 37
Curriculum Control Group 3.54 138 3.59 37 3.59 34 3.59 32 3.37 35
Difference 0.00 0.08 0.02 0.06 -0.16
I rarely share ideas with RWCT Teachers 1.96 145 2.19 32 2.16 38 1.77 39 1.75 36
Colleagues Control Group 2.15 139 2.05 38 2.65 34 1.94 32 1.97 35
Difference -0.19 0.13 -0.49 -0.17 -0.22
I share ideas about RWCT Teachers 3.39 146 3.06 32 3.26 38 3.49 39 3.70 37
teaching with colleagues Control Group 3.41 138 3.39 38 3.29 34 3.39 31 3.54 35
in school Difference -0.02 -0.33 -0.03 0.10 0.16
I share ideas about RWCT Teachers 2.68 146 2.41 32 1.58 38 3.26 39 3.46 37
teaching with colleagues Control Group 2.58 139 2.39 38 1.62 34 3.06 32 3.29 35
outside school Difference 0.10 0.01 -0.04 0.19 0.17
Impossible to teach all RWCT Teachers 2.88 144 2.84 32 2.75 36 3.15 39 2.76 37
pupils to learn Control Group 2.96 139 2.92 38 3.00 34 3.16 32 2.77 35
Difference -0.07 -0.08 -0.25 0.00 -0.01
Optimistic about the RWCT Teachers 3.05 145 2.81 32 3.19 37 2.77 39 3.41 37
future of education in my Control Group 2.88 138 2.71 38 2.85 33 2.78 32 3.17 35
Country Difference 0.17 0.10 0.34 -0.01 0.23
Bad to change practices RWCT Teachers 2.16 146 2.13 32 2.39 38 1.92 39 2.22 37
based on pupil suggestions Control Group 2.33 139 2.21 38 2.53 34 2.00 32 2.57 35
Difference -0.17 -0.09 -0.13 -0.08 -0.36
1 = Disagree Strongly; 2 = Disagree Somewhat; 3 = Agree Somewhat; 4 = Agree Strongly.

TABLE XXII: RWCT TEACHERS’ DESCRIPTIONS ABOUT THEMSELVES

AS TEACHERS BEFORE THEY BEGAN TO PARTICIPATE IN RWCT
Czech
Used traditional teaching methods 44.52 146 50.00 32 34.21 38 33.33 39 62.16 37
Was rigid 31.51 146 15.63 32 28.95 38 28.21 39 51.35 37
Was bored 15.07 146 3.13 32 26.32 38 10.26 39 18.92 37
Was not confident 13.70 146 6.25 32 10.53 38 20.51 39 16.22 37
Used teacher-centered activities 8.90 146 12.50 32 21.05 38 2.56 39 0.00 37
Focused on the curriculum 7.53 146 9.38 32 10.53 38 7.69 39 2.70 37
Was confident 5.48 146 0.00 32 13.16 38 2.56 39 5.41 37
Had less knowledge 5.48 146 3.13 32 7.89 38 10.26 39 0.00 37
Only accepted one correct answer 5.48 146 0.00 32 13.16 38 5.13 39 2.70 37
Was a strong teacher/more effective 5.48 146 3.13 32 0.00 38 10.26 39 8.11 37
Respected pupil opinions 3.42 146 3.13 32 2.63 38 7.69 39 0.00 37
Used non-traditional teaching methods 3.42 146 3.13 32 0.00 38 7.69 39 2.70 37
Enjoyed teaching 2.74 146 3.13 32 0.00 38 5.13 39 2.70 37
Was creative in my approach 2.74 146 9.38 32 0.00 38 2.56 39 0.00 37
Did not respect pupil opinions 2.05 146 3.13 32 5.26 38 0.00 39 0.00 37
Was flexible 2.05 146 0.00 32 0.00 38 5.13 39 2.70 37
Was dissatisfied with teaching practices 2.05 146 0.00 32 5.26 38 2.56 39 0.00 37
Was open to all answers 2.05 146 3.13 32 0.00 38 5.13 39 0.00 37
Was excited 1.37 146 0.00 32 0.00 38 5.13 39 0.00 37
Facilitated group work 1.37 146 6.25 32 0.00 38 0.00 39 0.00 37

TABLE XXIII: RWCT TEACHERS’ DESCRIPTIONS ABOUT THEMSELVES

AS TEACHERS AFTER THEY BEGAN TO PARTICIPATE IN RWCT
Czech
%
% Rspndnts N % Rspndnts N % Rspndnts N % Rspndnts N Rspndnts N
Use non-traditional teaching methods 28.08 146 25.00 32 21.05 38 28.21 39 37.84 37
Work with all pupils—pupil centered 22.60 146 25.00 32 18.42 38 15.38 39 32.43 37
Am flexible 21.23 146 3.13 32 13.16 38 17.95 39 48.65 37
Am open to all answers 19.86 146 25.00 32 34.21 38 12.82 39 8.11 37
Focus on pupil needs 19.18 146 21.88 32 26.32 38 10.26 39 18.92 37
Respect pupil opinions 18.49 146 9.38 32 36.84 38 2.56 39 24.32 37
Enjoy teaching 18.49 146 9.38 32 23.68 38 20.51 39 18.92 37
Am excited 15.07 146 3.13 32 26.32 38 10.26 39 18.92 37
Am confident 13.01 146 6.25 32 10.53 38 20.51 39 13.51 37
Am creative in my approach 6.85 146 9.38 32 10.53 38 7.69 39 0.00 37
Am a strong teacher/more effective 5.48 146 6.25 32 2.63 38 12.82 39 0.00 37
Am more motivated to make
lessons interesting 5.48 146 9.38 32 0.00 38 10.26 39 2.70 37
Am content 3.42 146 0.00 32 0.00 38 7.69 39 5.41 37
Facilitate group work 3.42 146 12.50 32 2.63 38 0.00 39 0.00 37
Use RWCT strategies 2.74 146 9.38 32 2.63 38 0.00 39 0.00 37
Am interested in new methods 2.05 146 0.00 32 2.63 38 5.13 39 0.00 37

TABLE XXIV: SELF-REPORTED CHANGES IN HOW TEACHERS PREPARE

FOR LESSONS SINCE BEGINNING THEIR PARTICIPATION IN RWCT
Czech
Use more time to plan lessons 25.34 146 36.26 32 24.91 38 19.51 39 22.49 37
Plan opportunities for pupils to express
opinions 20.55 146 21.88 32 36.84 38 20.51 39 2.70 37
Plan more group work 19.86 146 9.38 32 26.32 38 30.77 39 10.81 37
Consider a variety of teaching methods 19.86 146 6.25 32 18.42 38 25.64 39 27.03 37
Use more aids and supplementary texts 18.49 146 31.25 32 15.79 38 10.26 39 18.92 37
Think about pupils' experiences 11.64 146 18.75 32 13.16 38 12.82 39 2.70 37
Ensure that classes are more exciting 8.90 146 3.13 32 10.53 38 7.69 39 13.51 37
Think about applications of information
and outcomes 8.22 146 6.25 32 10.53 38 7.69 39 8.11 37
Enjoy planning more 7.53 146 9.38 32 10.53 38 5.13 39 5.41 37
Am more free and creative 6.85 146 6.25 32 7.89 38 2.56 39 10.81 37
Combine RWCT with other practices 6.16 146 6.25 32 10.53 38 5.13 39 2.70 37
Focus more on pupil activities 5.48 146 6.25 32 7.89 38 2.56 39 5.41 37
Consider longer term projects and
connections between lessons 4.79 146 9.38 32 5.26 38 5.13 39 0.00 37
Emphasize individual work 4.79 146 3.13 32 2.63 38 5.13 39 8.11 37
Use pupil self-evaluation 4.79 146 0.00 32 2.63 38 12.82 39 2.70 37
Plan for the teacher to speak less 4.11 146 3.13 32 7.89 38 5.13 39 0.00 37
There is no change 3.42 146 9.38 32 0.00 38 2.56 39 2.70 37
Integrate critical thinking into lesson 3.42 146 3.13 32 2.63 38 2.56 39 5.41 37
Increased planning time doesn't allow
for RWCT in every class 2.74 146 3.13 32 0.00 38 5.13 39 2.70 37
Incorporate RWCT into lesson planning 2.74 146 3.13 32 0.00 38 2.56 39 5.41 37
Consider classroom layout in planning 2.74 146 0.00 32 0.00 38 0.00 39 10.81 37
Planning is easier 2.05 146 0.00 32 5.26 38 2.56 39 0.00 37

TABLE XXV: RWCT TEACHERS’ OPINIONS ABOUT HOW RWCT INSTRUCTIONAL

PRACTICES DIFFER FROM PREVIOUS INSTRUCTIONAL PRACTICES
Overall Czech Latvia Macedonia

Republic Kyrgyzstan
Emphasizes pupil opinions and expression 36.30 146 50.00 32 39.47 38 28.21 39 29.73 37
Creates more interesting classes 20.55 146 15.63 32 26.32 38 20.51 39 18.92 37
Is systematic about creative teaching 13.01 146 9.38 32 5.26 38 12.82 39 24.32 37
Emphasizes pupil interaction and group work 12.33 146 15.63 32 15.79 38 10.26 39 8.11 37
Not very different 10.27 146 12.50 32 2.63 38 23.08 39 2.70 37
Is more efficient: learn more in less time 7.53 146 6.25 32 5.26 38 5.13 39 13.51 37
Uses literature and writing 6.16 146 15.63 32 5.26 38 5.13 39 0.00 37
Encourages more pupils to participate in class 5.48 146 6.25 32 5.26 38 5.13 39 5.41 37
Creates closer pupil/teacher relationships 5.48 146 6.25 32 10.53 38 5.13 39 0.00 37
Makes teaching more demanding but creative 4.79 146 6.25 32 0.00 38 12.82 39 0.00 37
Increases critical thinking 4.79 146 3.13 32 7.89 38 5.13 39 2.70 37
Differs in how teachers present information 4.11 146 0.00 32 2.63 38 0.00 39 13.51 37
Increases tolerance of other views 2.74 146 3.13 32 2.63 38 0.00 39 5.41 37
Makes presenting information more difficult 2.74 146 6.25 32 5.26 38 0.00 39 0.00 37
Makes pupils more responsible for their work 2.05 146 6.25 32 0.00 38 2.56 39 0.00 37
Uses different methods to evaluate pupil
performance 1.37 146 3.13 32 0.00 38 2.56 39 0.00 37
Takes more time to present information 1.37 146 6.25 32 0.00 38 0.00 39 0.00 37
TABLE XXVI: RWCT TEACHERS’ OPINIONS ABOUT THE MOST POSITIVE EFFECTS
Czech
Enables pupils to participate more in class 48.63 146 37.50 32 52.63 38 56.41 39 45.95 37
Enables pupils to guide their own learning 46.58 146 40.63 32 28.95 38 58.97 39 56.76 37
Increases the speed and quality of learning 15.75 146 15.63 32 18.42 38 10.26 39 18.92 37
Improves teacher/pupil relationships 15.07 146 9.38 32 10.53 38 5.13 39 35.14 37
Improves pupil behavior 11.64 146 0.00 32 10.53 38 10.26 39 24.32 37
Allows for individual expression of opinion 8.90 146 25.00 32 0.00 38 7.69 39 5.41 37
Motivates pupils to learn 8.90 146 18.75 32 15.79 38 2.56 39 0.00 37
Allows for a range of teaching methods 7.53 146 9.38 32 15.79 38 2.56 39 2.70 37
Increases creative and critical thinking 6.16 146 9.38 32 5.26 38 7.69 39 2.70 37
Assists in teaching the required curriculum 5.48 146 0.00 32 5.26 38 12.82 39 2.70 37
Assists in evaluating pupil performance 4.79 146 0.00 32 0.00 38 2.56 39 16.22 37
Makes class more interesting 4.79 146 9.38 32 5.26 38 5.13 39 0.00 37
Builds confidence (teachers and pupils) 3.42 146 3.13 32 0.00 38 10.26 39 0.00 37
Improves pupil communication 3.42 146 3.13 32 5.26 38 5.13 39 0.00 37
Improves teacher motivation 2.74 146 9.38 32 2.63 38 0.00 39 0.00 37
Take responsibility for work (pupils) 2.74 146 0.00 32 5.26 38 5.13 39 0.00 37
Allows for pupil self-assessment 1.37 146 6.25 32 0.00 38 0.00 39 0.00 37

TABLE XXVII: RWCT TEACHERS’ OPINIONS ABOUT THE MOST NEGATIVE EFFECTS
Czech
Increases class time to complete requirements 21.23 146 25.00 32 15.79 38 30.77 39 13.51 37
Requires additional materials 12.33 146 0.00 32 10.53 38 20.51 39 16.22 37
Makes pupil behavior worse 10.27 146 18.75 32 2.63 38 20.51 39 0.00 37
Makes it difficult to teach required curriculum 10.27 146 25.00 32 5.26 38 12.82 39 0.00 37
Increases teacher work load 8.22 146 12.50 32 0.00 38 20.51 39 0.00 37
Decreases the speed/quality of pupil learning 4.79 146 6.25 32 7.89 38 5.13 39 0.00 37
Makes it difficult to evaluate pupil performance 3.42 146 0.00 32 7.89 38 5.13 39 0.00 37
Makes it difficult to discuss all pupils’ work 2.05 146 9.38 32 0.00 38 0.00 39 0.00 37
Makes teacher/pupil relationships worse 1.37 146 0.00 32 0.00 38 0.00 39 5.41 37
Decreases pupil motivation 1.37 146 6.25 32 0.00 38 0.00 39 0.00 37
TABLE XXVIII: RWCT TEACHERS’ RECOMMENDATIONS

FOR MAKING FUTURE CHANGES TO RWCT
Czech
Develop concrete curricula based on RWCT methods 10.96 146 9.38 32 18.42 38 15.38 39 0.00 37
Do not change anything 10.27 146 3.13 32 13.16 38 7.69 39 16.22 37
Meet periodically with teachers and conduct classroom
observations 8.90 146 12.50 32 7.89 38 15.38 39 0.00 37
Provide post-workshop reviews 6.85 146 15.63 32 5.26 38 7.69 39 0.00 37
Increase text books and support materials 4.79 146 0.00 32 5.26 38 10.26 39 2.70 37
Expand RWCT with new methods and ideas 4.11 146 6.25 32 2.63 38 2.56 39 5.41 37
Develop ways to introduce RWCT to more traditional
teachers 3.42 146 3.13 32 2.63 38 5.13 39 2.70 37
Make RWCT mandatory 2.74 146 6.25 32 2.63 38 0.00 39 2.70 37
Increase class time/Reduce number of pupils per class 2.74 146 0.00 32 0.00 38 0.00 39 10.81 37
Develop strategies for grading/evaluating pupils 2.05 146 0.00 32 7.89 38 0.00 39 0.00 37
Reduce amount of preparation and paperwork for
RWCT instruction 2.05 146 3.13 32 2.63 38 0.00 39 2.70 37
Facilitate more dynamic workshops in less time 2.05 146 0.00 32 0.00 38 7.69 39 0.00 37

APPENDIX D
APPENDIX D: STUDY STAFF AND ADVISORS

AIR would like to thank colleagues from the Open Society Institute (OSI), the International
Reading Association (IRA), and national RWCT project offices for their hard work, insight, and
support throughout the evaluation process. We would also like to thank Holly Baker, whose editorial
insight improved the quality of this report. We are indebted to the graduate students, teachers,
principals, and university faculty who collected and scored data for this evaluation and to the
teachers and pupils who invited data collectors into their classrooms and completed survey forms.
The claims in this report, however, are solely the responsibility of the American Institutes for
Research.
IRA/OSI ADVISORY PANEL

Alan Crawford
Sam Matthews
Alison Preece
Iveta Silova
Wendy Saul
COUNTRY COORDINATORS AND RWCT PROJECT STAFF

IN EVALUATION COUNTRIES
Irena Freimane
Hana Koštálová
Spomenka LazarevskaIrina Nizovskaya
Nurbek Teleshaliyev
DATA COLLECTORS
Aisuluu Bedelbaeva Indra Odina
Satkyn Beketaeva Simona Palcevska
Kristina Bocanova Dace Penke
Boge Bozinovski Larisa Petre
Saltanat Builasheva Jana Petrova
Tereza Foltynova Barbora Pintyrova
Magdalena Grillova Elvira Ramcilovik
Valeri Hardin Marite Seile
Tomas Jacko Sanita Sumane
Natasa Jasimovska Atina Tasevska
Aigul Khakimova Iveta Teibe
Jolana Langrova Elizabeta Tomeska
Larisa Nagda Vineta Trumsina
Appendix D American Institutes for Research

T 2000-2001 E R W C T P: HE Valuation of THE Eading and Riting For Ritical Hinking Roject

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

T 2000-2001 E R W C T P: HE Valuation of THE Eading and Riting For Ritical Hinking Roject

Transféré par

Droits d'auteur :

Formats disponibles

THE 2000-2001 EVALUATION OF

THE READING AND WRITING FOR

II. The RWCT Model for Professional Development .................................. 2

III. Study Design and Data Collection .......................................................... 2

IV. Evaluation Results ................................................................................... 11

American Institutes for Research

1 Sampling Frame ...................................................................................... 7

American Institutes for Research

LIST OF TABLES (CONTINUED)

21 Most Frequently Reported Changes in How Teachers Prepare for

American Institutes for Research

1 Project Timeline ...................................................................................... 3

2 Conceptual Model of the Professional Development Process for

3 Critical Thinking among RWCT Pupils, based on Years

4 Integration Scores among RWCT Teachers, Based on Years

American Institutes for Research

THE 2000-2001 EVALUATION OF THE READING

Pupil critical thinking skills:

Teacher and pupil attitudes:

1 American Institutes for Research

II. The RWCT Model of Professional Development

III. STUDY DESIGN AND DATA COLLECTION

The timeline for conducting the evaluation is shown as Exhibit 1.

2 American Institutes for Research

EXHIBIT 1: PROJECT TIMELINE

3 American Institutes for Research

EXHIBIT 2: CONCEPTUAL MODEL OF THE PROFESSIONAL

4 American Institutes for Research

Measuring Critical Thinking

• Form original opinions,

5 American Institutes for Research

Measuring Teaching Practices

• Activities that promote higher-order thinking;

6 American Institutes for Research

TABLE 1: SAMPLING FRAME

Number of RWCT Number of Control-

7 American Institutes for Research

TABLE 2: COMPARISONS OF RWCT AND NON-RWCT TEACHERS

RWCT Teachers Control-Group Teachers

TABLE 3: COMPARISONS OF RWCT AND CONTROL-GROUP PUPILS

8 American Institutes for Research

9 American Institutes for Research

10 American Institutes for Research

IV. EVALUATION RESULTS

Evaluation Question 1: To what extent do RWCT pupils have higher critical

11 American Institutes for Research

TABLE 4: DIFFERENCES IN CRITICAL THINKING SCORES

12 American Institutes for Research

Differences in Critical Thinking Scores

13 American Institutes for Research

Differences between RWCT and Control Group

14 American Institutes for Research

TABLE 7: PROPORTIONAL REDUCTIONS IN CRITICAL THINKING SCORE DIFFERENCES

15 American Institutes for Research

TABLE 8: PROPORTIONAL REDUCTIONS IN CRITICAL THINKING SCORE

Proportional Reductions for Differences in “Integration” Scores

EXHIBIT 3: CRITICAL THINKING AMONG RWCT PUPILS, BASED ON YEARS OF TEACHER

16 American Institutes for Research

Question 2: To what extent do RWCT teachers maintain the fidelity of the

17 American Institutes for Research

18 American Institutes for Research

TABLE 9: DIFFERENCES BETWEEN RWCT AND CONTROL-GROUP TEACHERS