B.Becker Istation Program Eval

Istation Program Evaluation
Istation and Reading Success

A Program Evaluation of Istation at Ruth Barron Elementary, Pflugerville ISD
Evaluated by Bianca Becker
Spring 2014
14850 Harris Ridge Blvd., Pflugerville, TX 78660
PHONE
FAX
WEB
(512) 5944300
(512) 5944305
http://cms.pflugervilleisd.net//site/Default.aspx?
PageID=1311
TABLE OF CONTENTS
Section I. Executive Summary. 5
Introduction 5
Focus of Evaluation 5
Description of the Evaluation Study.. 6
Results 6
Student Data.. 6
Student Assessment Performance Scores- Eduphoria..... 6
Student Progress Monitoring Data and Frequency of Usage- Istation/Staff Survey... 6
Staff Data.. 7
Staff Survey..... 7
Staff Interview..... 7
Interpretation and Discussion. 8
Interpretation of Student Results... 8
Interpretation of Staff Results... 8
Discussion. 8
Conclusions and Recommendations.. 9
Recommendations for Staff Training... 9
Recommendations for School Wide Systems for Istation Implementation & Monitoring. 10
Section II. Introduction... 11

Purpose of Evaluation.. 11
Stakeholders..... 11
Overview of Report Contents... 11

Section III. Focus of Evaluation.... 12
Description of Program 12
Related Literature..... 13
Reliability and Validity of Istation Reading Program.... 13
Relationship of Istations ISIP and Other Developmentally Appropriate Reading Assessments... 13
Evaluative Questions 13
Section IV. Description of the Evaluation Study...... 15

Evaluation Design and Procedures... 15
Goals... 15
Participants... 15
Staff. 15
Students... 16
Data Collection and Analysis... 18
Evaluation Limitations. 19
Section V. Results..... 20
Student Results. 20
Student Assessment Performance Scores - Eduphoria... 20
Student Progress Monitoring Data & Frequency of Usage Istation/Staff Survey... 22
Staff Results..... 24
Staff Survey Results........ 24
Teacher Use and Familiarity with Istation Reading Program Tools and Resources..... 24
Istation for Student Intervention.... 26
Effectiveness and Importance of Istation Reading Program. 27
Staff Interview Results... 30
Staff Knowledge and Use of Istation Reading Program... 30

Student Academic Performance Prior to & After Istation Use... 30
Istation Use by Students- Opportunities and Frequency. 31
Effectiveness of Istation Reading Program... 32
Section VI. Interpretation and Discussion...... 33

Interpretation of Student Results.. 33
Interpretation of Staff Results.. 34
Discussion.... 35
Section VII. Conclusions and Recommendations..... 36

Conclusions from the Evaluation. 36
Recommendations for Integration and Monitoring of Istation Program...... 36
Design & Development of Istation Program Training for Staff..... 36
Development of School Wide Systems for Istation Implementation and Monitoring.... 37
References... 38
SECTION I. EXECUTIVE SUMMARY

INTRODUCTION
Using a decision-making model, the purpose of this formative program evaluation is to assess the effectiveness
and impact of the computer-based reading program named Istation, on 3rd-5th grade students reading performance scores
at Ruth Barron Elementary in Pflugerville ISD. This evaluation will serve as an effectiveness and needs assessment tool of
the current resource that is part of the campus intervention and progress-monitoring model. The information collected
from this evaluation is important data to take into account as campus staff develop differentiated instruction to meet
student needs, prepare guided reading groups, and get ready for STAAR testing. Guided by Ruth Barrons mission and
campus improvement goals, the Istation Reading program evaluation goals include:
Intermediate Goal (Process)- 3rd through 5th grade students at Ruth Barron Elementary School use the Istation
reading program based on their level of intervention in order to target key reading areas in order to improve their
reading comprehension scores.
Outcome Goal (Product)- Ruth Barron Elementary School students will show an increase in reading performance
scores, level, and overall reading comprehension.
The stakeholders include Jos Medina (Ruth Barron Elementary Principal), Cynthia Ward, Adrian Tristan (Ruth
Barron Elementary Assistant Principals), Ruth Barron Elementary faculty, and Ruth Barron Elementary students.
FOCUS OF EVALUATION
This evaluation will focus on one of Ruth Barrons campus improvement goals related to student intervention
systems and improvement of academic performance scores; by establishing educational systems that allow for the RTI
process to function at high levels of efficiency, every RBES student will show improvement in the areas of ELA/SLA,
Writing, Math, and Science based on data from common assessments during the school year, resulting in successful
performance on state assessments. Student and campus needs in the area of reading comprehension and analysis will
frame perspectives for considering how the Istation Reading Program is being implemented in 3 rd- 5th grade classrooms,
and how its use has impacted student reading abilities and academic performance. Evaluative questions will include:
1. Is the Istation reading program effective in growing students reading scores and reading comprehension?
a. What aspects of the program are effective?
b. Has there been any documented growth in the students' reading skills/comprehension? (Use campus
determined data sources)
c. Have students demonstrated an increase in reading levels? (DRA/EDL)

2.
In what ways do teachers and staff integrate Istation into the curriculum?
a. What opportunities are being provided for students to use the program?
b. How often are students using the program?
c. Does the exposure time to the program affect students growth in reading?
DESCRIPTION OF THE EVALUATION STUDY

A decision-making model was used because it aims to provide an analytic and rational basis for program decisionmaking, in which an emphasis is placed on systematic data collection and reporting in order to promote effective program
management. The decision-making model utilizes a mixed-methods design to help generate a formative evaluation. A
decision-focuses approach to evaluation is based on a cycle of planning, structuring, implementing and reviewing and
revising decisions, each examined through a different aspect of evaluation context, input, process and product evaluation
(Robinson, 2002). This evaluation sought to identify the process of how faculty are providing opportunities for students to
use the Istation program, the frequency of students program use, and the impact of Istation Reading Program on students
reading level and reading comprehension scores. Participants included 26 staff members and 316 students in grades 3
through 5. The use of multiple sources, including student performance scores in assessments, student progress monitoring
data, and staff surveys and interviews, strengthens crosschecking and triangulation. Despite the fact that all 3 rd-5th grade
staff members and students were invited to participate, low response rates on the faculty surveys and the discrepancy with
the number of students assessed in each benchmark assessment, and the number of students enrolled in the designated
grade levels could impact the perceived credibility of the results.
RESULTS
STUDENT DATA
STUDENT ASSESSMENT PERFORMANCE SCORES EDUPHORIA
Performance scores for 316 students were analyzed using Eduphoria, the school districts data program, to assess
students performance across four different assessments. The assessments consisted of District Assessment 1 (DA1), Mock
STAAR, District Assessment 2 (DA2), and Spring Common Assessment 1 (SprCA1). Based on evaluation of student
assessment performance, it is unclear to determine whether the impact on their scores is derived from the use of Istation or
from a multitude of interventions that are in place at the campus and other factors. Never the less, students in grades 3- 5
have demonstrated a steady increase between some of the assessments administered.
STUDENT PROGRESS MONITORING DATA AND FREQUENCY OF USAGE ISTATION/ STAFF SURVEY
This section addresses students frequency and duration of use, opportunities to engage in the Istation program,
and ISIP score distribution. About 29% of staff members surveyed stated that their students use Istation on a monthly
basis. 53% of teachers surveyed reported that their students use the Istation program for 20-30 minutes. 39% of teachers

surveyed reported scheduling Istation use into their Daily 5 routine. Of the 69 dual language 3 rd grade students assessed
during BOY and MOY benchmarks, 58 demonstrated an increase in their ISIP score and 11 students showed a decrease.
44 monolingual 3rd grade students were assessed and of those 32 increased, 11 decreased and 1 student maintained the
same ISIP score. In 4th grade 46 dual language students were assessed and of those students 39 increased and 7 decreased
their score. 50 4th grade monolingual students were assessed and of those students 39 increased their score and 11
decreased. Of the 51 5th grade dual language students that were assessed, 35 demonstrated an increase in their score and
16 decreased their ISIP score. Of the remaining 43 5 th grade monolingual students, 32 increased and 11 decreased their
score. It is clear based on these data categories that students growth in the area of reading and their performance in
subject-related assessments is impacted by the amount of exposure they have with Istation, and of their progress on its
ISIP evaluation.
STAFF DATA
STAFF SURVEY
All 19 teachers and 4 of the 7 interventionists (those that service general education students) were sent a link to a
staff survey via email. Of the total 23 staff members emailed only 17 replied (74% return rate). The staff survey consisted
of 17 questions which were formatted as likert scales, yes and no questions, multiple choice, and open-ended response.
The first question provided demographic information that illustrated the distribution of staff members by position (see
Figure 14). Results of the staff survey are displayed in the following three sections: 1) Teacher Use and Familiarity With
Istation Reading Program Tools and Resources, 2) Istation for Student Intervention, and 3) Effectiveness and Importance
of Istation Reading Program.
Responses to the Teacher Use and Familiarity With Istation Reading Program Tools and Resources section
indicated that 59% of staff members are familiar with the Istation Reading program and the same percent have explored
its resources. 46% of staff members surveyed found the ISIP Summary Report to be the most helpful tool when evaluating
student data. Of the staff members surveyed 48% dont use any of the resources located in the teacher database.
Responses to the Istation for Student Intervention section indicated that teachers differ in their implementation of
the program and vary in their use for the data collected from its evaluations. Several teachers use the Istation data to
develop and differentiate student guided reading groups and monitor student progress. The majority of teachers expressed
that the data collected through Istation and its teacher resources are helpful, but a couple of the members surveyed
expressed not knowing about them.
Responses to the Effectiveness and Importance of Istation Reading Program section indicated that 59% of
teachers felt that Istation had impacted their students reading comprehension and 76% had seen growth in their students
ISIP score since the programs implementation. 59% of teachers felt that Istation was moderately effective on their
students comprehension scores. 47% of staff members think that the ISIP score is not as accurate as DRA data, and 76%
of staff thinks Istation is relatively important to Ruth Barrons campus goals and student population.
STAFF INTERVIEW
Seven teachers agreed to be interviewed, of which six were actually interviewed. Each individual interview
consisted of open-ended retrospective questions and lasted for approximately 40-60 minutes. The interview questions
asked the staff members to reflect upon their personal knowledge and use of the Istation Reading program, student reading
academic performance prior to and after the use of Istation program, opportunities for and frequency of students use of
program, and perceived effectiveness of Istation Reading program. Staff members varied in their methods of
implementation for Istation and in their perception of Istations impact on students reading performance and its
significance to Ruth Barron Elementary and its student population. The majority of staff members mainly used the ISIP
Student Report for progress monitoring and to develop their reading intervention groups and curriculum. In addition staff
members expressed that they had little to no knowledge of the programs resources for instruction and therefore did not
use them. All participants interviewed expressed the need for training in regards to Istation, and felt that there was no
follow-up on the use of the program from the administrative and intervention team.
INTERPREATION AND DISCUSSION

INTERPREATION OF STUDENT RESULTS
Student performance scores in District Assessment 1 (DA1), Mock STAAR, District Assessment 2 (DA2), and
Spring Common Assessment 1 (SprCA1) show a variation in each grade level from one assessment to the next. Both
English and Spanish test taking students in grades 3-5 have shown a gradual increase in the amount of students meeting
standard for each assessment. Based on this data, students growth in the area of reading and their performance in subjectrelated assessments is impacted by the amount of exposure they have with Istation, and of their progress on its ISIP
evaluation. Students need to know that the ISIP program is a tool that helps them work on the skills they need. It provides
exposure to reading passages and skill-based games that help them increase their reading fluency, vocabulary, and
comprehension. This in turn serves as an intervention that can help close the gap in student achievement in standardized
assessments.
INTERPREATION OF STAFF RESULTS
In order for staff members to feel that Istation is important to Ruth Barrons campus goals and student population,
the program must be part of classroom expectations. This means that just as word walls, CAF, Daily 5 rotations, and
math centers are part of the routines established in the classroom setting, Istation needs to be given the same priority. Its a
tool to be used by students in order to enhance reading skills. In addition is provides crucial information that can be used
to supplement or further define DRA scores, generate intervention groups, target skills, and help make language of
assessment decisions.
DISCUSSION
A synthesis of the results of this evaluation indicates that the majority of staff members and students in grades 3-5
are not using the Istation Reading program to its fullest potential. This is of great concern because Istation is a resource

that is adopted at the district level and provides valuable data, documentation, and tools for student reading achievement.
There is disconnect at the campus and district levels that limit the way this intervention is utilized and implemented.
Similarly in line with Ruth Barrons mission and goals, the Istation program evaluation goals included:
Intermediate Goal (Process)- 3rd through 5th grade students at Ruth Barron Elementary School use the Istation reading
program based on their level of intervention in order to target key reading areas in order to improve their reading
comprehension scores.
Outcome Goal (Product)- Ruth Barron Elementary School students will show an increase in reading performance scores,
level, and overall reading comprehension.
The lack of staff training and school wide systems for program implementation indicate that the administration and
instructional/intervention teams need to work collaboratively to prepare and roll out the necessary training and adoption
policy that will enable teachers and students in grades 3-5 to adopt and implement Istation successfully. Only then,
teachers and students will use the program with fidelity and thus improve their reading performance.
CONCLUSIONS AND RECOMMENDATIONS

The majority of staff members and students do not seem to be implementing and using the Istation Reading program
successfully. This lack of teacher and student program buy-in leads to the ineffective use of tools and resources that have the potential
to impact students reading skills. Recommendations for Ruth Barron Elementarys integration and use of Istation consist of two
components, including 1) the design and development of Istation program training for staff, and 2) the development of school wide
systems for Istation implementation and monitoring.
RECOMMENDATIONS FOR STAFF TRAINING
The administrative, intervention, and technology teams should attempt to develop professional development
training for teachers in order to inform them of the ins and outs of the Istation Reading program. The training should focus
on the use and implementation of the programs various components. By offering training to staff members, the schools
leadership team will be ensuring that staff members in grades 3-5 will successfully implement the program in their
classrooms and use it for student monitoring and intervention. Some topics to discuss in the training are:
Access to student data
Unpacking and Evaluation of student data (being able to read data shown in reports)
Exploration of Data Tools and Teacher Database Resources
Modeling of how to use the data gathered and resources as part of reading intervention
Student log-in from home
As illustrated by the data gathered in this evaluation, the majority of staff members and students do not appear to
be utilizing the Istation Reading program in a uniform and effective manner. The inconsistency of student and teacher use
of the program has limited the results of how the program has impacted student academic performance, and the potential

support that Istation can provide students in the area of reading. The majority of teachers have expressed that they are
unaware of many of the tools and resources provided by Istation, and/or have failed to utilize them.
In addition, there is a lack of a systematic implementation of the Istation program across grade levels. In third
grade teachers are required to progress monitor their students on a monthly basis using ISIP. In fourth and fifth grade
teachers are only expected to use the program during BOY and MOY benchmarks. There is also an inconsistency with the
amount of time students use the program. Limited computer access narrows down the number of students that can use
Istation on a frequent basis. Tier 2 and 3 students tend to have the most exposure to the program. But within this group it
seems that they only access Istation during their pull out time or during before or after school programs. If students are
arriving on time in the morning and attending all afternoon sessions, then there is potential for Istation impact on reading
development. But if students dont attend their morning or afternoon sessions, they are only using the program once a
week or for a 15 minutes a day during Daily 5 stations. In grades 4 and 5 there is a drop in the amount of time students use
Istation. The majority of teachers felt that Istation has little value in the campus intervention program because it is a
component that is part of a long list of systems already implemented by the campus.
RECOMMENDATIONS FOR SCHOOL WIDE SYSTEMS FOR ISTATION IMPLEMENTATION AND MONITORING
In order to ensure the success of Istations implementation, Ruth Barron Elementary needs to set up a system that
assists staff and students in adopting the program. A suggestion for increasing student buy-in is developing a system that
is similar to the Accelerated Reader (AR) program established on campus. There can be monthly check-ins to monitor
student progress. These check-ins can provide a reward via current school behavior management systems to students that
show continuous use and progress. Also, the program needs to have a consistent implementation across grade levels. If
one grade level is using the program for monthly progress monitoring and the others arent, the other grade levels are
missing out on student interaction with Istation. All grades should use the program on a consistent basis. In addition,
students in grades 4-5 need to be able to access the program in both English and Spanish. Students in the dual language
program need to be able to uses the resources in both languages. Once teachers have been trained, they can in turn deliver
the program and login information to their students and parents.
10
SECTION II. INTRODUCTION

PURPOSE OF EVALUATION
Using a decision-making model, the purpose of this formative program evaluation is to assess the effectiveness
and impact of the computer-based reading program named Istation, on 3rd-5th grade students reading performance scores
at Ruth Barron Elementary in Pflugerville ISD. This evaluation will serve as an effectiveness and needs assessment tool of
the current resource that is part of the campus intervention and progress-monitoring model. The information collected
from this evaluation is important data to take into account as campus staff develop differentiated instruction to meet
student needs, prepare guided reading groups, and get ready for STAAR testing.
STAKEHOLDERS
Jos Medina (Ruth Barron Elementary Principal)

Cynthia Ward, Adrian Tristan (Ruth Barron Elementary Assistant Principals)
Ruth Barron Elementary faculty
Ruth Barron Elementary students.
OVERVIEW OF REPORT CONTENTS

Section III of this report includes a descriptive overview of Ruth Barron Elementarys student population and of
the Istation Reading program. Literature related to the implementation of Istation in schools and of its ISIP component is
addressed to support results and the discussion of recommendations. Section IV outlines how the evaluation was guided
by questions that align with the schools mission and goals in order to explore how the implementation of Istation will
help improve student reading comprehension and performance scores. Mixed-methods and a variety of data sources are
used to address different perspectives and to allow for crosschecking and triangulation. Data sources include student
performance scores in assessments, student progress monitoring data, and staff surveys and interviews. Section V presents
the results thematically for each data source, both in narrative and graphic formats. Section VI provides an interpretation
and discussion of the implications for all of the data sources in a holistic assessment. Section VII includes a conclusion to
the program evaluation report and a detailed overview of recommendations for future consideration. References and
appendixes are included at the end of the report for further reference.
11
SECTION III. FOCUS OF THE EVALUATION

DESCRIPTION OF THE PROGRAM
Ruth Barron Elementary is a Title 1 campus located in Pflugerville, Texas. The campus is comprised of 87%
economically disadvantaged students with 73% of that population being at-risk. Students attending this school are
primarily of Hispanic and African American ethnic background. 1 District and campus initiatives have set in place
programs for intervention in order to strengthen students reading performance and to serve as an RTI (Response to
Intervention) measure. One of the main components used with 3 rd through 5th grade students is a program called Istation.
Students at Ruth Barron Elementary use the program according to the level of intervention they are classified under. Tier 1
students are assessed monthly using Istation, Tier 2-3 students use the program on a daily basis. Istation is a fully
integrated reading intervention program that helps meet the campus strategic goals and mission as outlined in Figure 1.
Istation delivers individualized instruction complete with age-appropriate content for pre-K through high school
students. Every lesson is supported with data-rich benchmark and continuous progress monitoring assessments through
Istation's proprietary ISIP technology. Thanks to ISIP, Istation's Indicators of Progress assessment technology, the
curriculum is adjusted automatically to every child's individual ability in all of the skill areas from struggling to
advanced.2 Teachers and staff members at Ruth Barron utilize the data provided by Istation to develop differentiated
instruction to meet student needs, prepare guided reading groups, and get ready for STAAR testing.
1 Descriptive and demographic information regarding Ruth Barron Elementary School is located at http://cms.pflugervilleisd.net/Page/1534
2 Descriptive information regarding Istation is located at http://www.istation.com/About
12

FIGURE 1. RUTH BARRON ELEMENTARY MISSION AND GOALS3
Ruth Barron Elementary

School
Mission:
Mission:
The
mission of
of Ruth
Barron Elementary
Elementary is
is to
to cultivate
cultivate a
expectations through
The mission
Ruth Barron
a culture
culture of
of high
high expectations
through a
a rigorous
rigorous curriculum,
curriculum, relevant
relevant learning
learning experiences,
experiences, embracing
embracing our
our diversity,
diversity, and
and
creating
strong relationships
relationships with
with our
learning community
community that
result in
creating strong
our learning
that will
will result
in the
the success
success of
of every
every child.
child. Simply
Simply put,
put, "lo
"lo que
que se
se dice,
dice, se
se hace."
hace." This
This translates
translates to
to "that
"that which
which you
you say,
say,
you
do."
you do."
Goals:
Goals:
1.
1. RBES
RBES will
will continue
continue to
to strengthen
strengthen our
our systematic
systematic approach
approach to
to providing
providing high
high quality
quality core
core instruction
instruction in
in every
every classroom,
classroom, in
in 2013
2013 -- 2014,
2014, in
in order
order to
to allow
allow students
students to
to reach
reach their
their
full
potential
and
will
result
in
a
15
point
increase
on
all
state
assessment
measures.
full potential and will result in a 15 point increase on all state assessment measures.
2.
2. By
By establishing
establishing educational
educational systems
systems that
that allow
allow for
for the
the RTI
RTI process
process to
to function
function at
at high
high levels
levels of
of efficiency,
efficiency, every
every RBES
RBES student
student will
will show
show improvement
improvement in
in the
the areas
areas of
of ELA/SLA,
ELA/SLA,
Writing,
Writing, Math,
Math, and
and Science
Science based
based on
on data
data from
from common
common assessments
assessments during
during the
the school
school year,
year, resulting
resulting in
in successful
successful performance
performance on
on state
state assessments.
assessments.
RELATED LITERATURE
Response to Intervention programs are a widely adopted intervention method that allow schools to target students
who need additional instructional support. Amongst their leveled tiers, there are many programs that fall under the Tier 2
category. Ruth Barron Elementary uses Istation as part of their Tier 2 intervention for reading. Students engage in the
program in hopes of targeting the skills they need in reading, and to strengthen their reading comprehension. In order to
assess the Istations quality and effectiveness on student reading performance, two main issues must be examined: 1) the
reliability and validity of the Istation reading program and 2) the relationship of Istations Indicators of Progress (ISIP)
with other widely accepted developmentally appropriate reading assessments and predictors.
RELIABILITY AND VALIDITY OF ISTATION READING PROGRAM
During the 200809 school year, a validity and reliability study using Istations computer adaptive reading
assessment program was conducted in five elementary schools from a north Texas school district. Data was examined for
internal consistency, testretest reliability, concurrent validity with external measures, including DIBELS, TPRI, AND
ITBS, and predictive validity with TAKS. Results show moderate to strong evidence of reliability and validity with
regards to phonemic awareness, alphabetic knowledge, vocabulary, and reading comprehension (Mathes, 2009).
In addition, a study was conducted in Wedgewood Elementary School in East Baton Rouge Louisiana. The school
began using the assessment as part of their reading intervention program with all students. The reading coach and teachers
of Wedgewood used Istation as their RTI program. As early as December 2010, the teachers began seeing great
improvement in their students (Bugbee, 2011).
3 Ruth Barron Elementary School mission and goals are detailed in the 2013-2014 Campus Strategic Improvement Plan document,
which is located at http://cms.pflugervilleisd.net/Page/1534
13
RELATIONSHIP OF ISTATIONS ISIP AND OTHER DEVELOPMENTALLY APPROPRIATE READING ASSESSMENTS
In 2012 a study conducted by Braden Hoelzle compared the relationship of the Istations Indicators of
Progress (ISIP) and the Measures of Academic Progress (MAP) reading assessments with the Developmental Reading
Assessment, 2nd Edition (DRA2) using single predictor regression. Using a series of regression analyses, a conversion
formula and table were created to convert an ISIP score into an estimated DRA2 score. Using data gathered from K-5th
grade students in a large suburban district in Texas, the predicted (R) values derived from regression analyses between
the ISIP and the DRA2 and the MAP and DRA2 were compared. In all grades with comparable data, the ISIP was
moderately to highly correlated with the DRA2, and the ISIP was more highly correlated with the DRA2 than was the
MAP (Hoelzle, 2012).
EVALUATIVE QUESTIONS
1. Is the Istation reading program effective in growing students reading scores and reading comprehension?
a. What aspects of the program are effective?
b. Has there been any documented growth in the students' reading skills/comprehension? (Use campus
determined data sources)
c. Have students demonstrated an increase in reading levels? (DRA/EDL)
2.
In what ways do teachers and staff integrate Istation into the curriculum?
What opportunities are being provided for students to use the program?
How often are students using the program?
Does the exposure time to the program affect students growth in reading?
a.
b.
c.
14
SECTION IV. DESCRIPTION OF THE EVALUATION STUDY

EVALUATION DESIGN AND FEATURES
Using a decision-making model, this formative program evaluation used a mixed-methods design to assess
effectiveness and impact of the computer-based reading program named Istation, on 3rd-5th grade students reading
performance scores at Ruth Barron Elementary in Pflugerville ISD. A decision-making evaluation model was used
because it aims to provide an analytic and rational basis for program decision-making, in which an emphasis is placed on
systematic data collection and reporting in order to promote effective program management. A decision-focuses approach
to evaluation is based on a cycle of planning, structuring, implementing and reviewing and revising decisions, each
examined through a different aspect of evaluation context, input, process and product evaluation (Robinson, 2002). This
evaluation sought to assess the effectiveness and impact the Istation reading program has on students reading
performance scores by including data collection and analysis of multiple sources. These include students performance
scores on four assessments (District Assessment 1 (DA1), Mock STAAR, District Assessment 2 (DA2), and Spring
Common Assessment 1(SprCA1)), Istation teacher surveys, and teacher interviews.
GOALS
The goals for this evaluation seek to first identify the process of how faculty are providing opportunities for
students to use the Istation program, how often students are using the program, has there been an increase in students
reading level and reading comprehension scores, and lastly, how does the frequency of the use of the Istation program
impact students reading performance. The following goals serve to frame the data collection and analysis procedures for
this evaluation:
PARTICIPANTS
A targeted group of members from the campus were invited to participate in the evaluation. Third through fifth
grade staff (teachers and interventionists), and 3 rd-5th grade students were selected.
STAFF
Figure 2-3 shows the distribution of the 19 teachers, and 7 interventionists who participated in this study. All 26
members were working with 3rd- 5th grade students during the 2013-2014 school year. All 19 teachers had their student
performance data analyzed and a volunteer sample participated in a survey and interview.
15
FIGURE 2. TEACHER DISTRIBUTION BY GRADE LEVEL
RBES Teacher Distribution

5th Grade
4th Grade
3rd Grade
FIGURE 3. INTERVENTIONIST DISTRIBUTION BY GRADE LEVEL AND ASSIGNMENT
RBES Interventionists
SPED Interventionist
4th & 5th Grade Interventionist
3rd Grade Interventionist

0
0.5
1.5
2.5
STUDENTS
Student participants were selected based on their grade level. Participants came from a sample of 316 students
classified as 3rd through 5th grade students enrolled in either monolingual or dual language classes during the 2013-2014
school year. All 316 students had their performance scores, Istation login frequency records, and progress monitored.
16

Figures 4-6 show the distribution of the 316 students who participated in this study. Students were distributed based on 3
categories. Figure 4 shows the students distribution based on grade level. Figure 5 shows students distribution based on
ethnicity, and Figure 6 shows student distribution based on special indicators (i.e. Limited English Proficient (LEP),
Special Education (SpEd), etc.).
FIGURE 4. DISTRIBUTION OF STUDENTS BY GRADE LEVEL
RBES Student Distribution by Grade Level
FIGURE 5. DISTRIBUTION OF STUDENTS BY ETHNICITY
RBES Student Distribution by Ethnicity
17
FIGURE 6. DISTRIBUTION OF STUDENTS BY SPECIAL INDICATORS
RBES Student Distribution By Special Indicators
DATA COLLECTION AND ANALYSIS

A variety of data sets were collected to cross-check and triangulate findings in order to provide a holistic look at
effectiveness of the Istation reading program in increasing students reading performance scores. Student distribution and
assessment performance scores data were obtained from the district ITCCS system and Eduphoria. Student Istation
progress monitoring data and usage frequency were collected from teacher surveys, interviews, and Istation system. The
teacher survey was an online form that consisted of 17 questions formatted as likert scales, yes and no questions, multiple
choice, and open-ended response. Teacher interviews were conducted on a volunteer basis. The group of teachers
interviewed sampled teachers from 3rd-5th grade from both monolingual and dual language classrooms. Staff member
interviews occurred once for approximately 40-60 minutes, which was audio-recorded. Table 1 indicates details regarding
data collection and analysis.
TABLE 1. DATA SOURCES AND ANALYSIS PROCEDURES

Data Source
Student Assessment Performance
Scores (Eduphoria)
Student Progress Monitoring Data &
Sample Size
Analysis Procedure
Connection to Evaluative
Question (s)
All 3rd- 5th Grade Students
Content Analysis
1b, 1c, 2c
All 3rd- 5th Grade Students
Content Analysis
1a, 1b, 1c, 2b, 2c

18

Frequency of Usage (Teacher
Survey/Interview & Istation)
Staff Survey
Staff Interview
All 3rd 5th Grade Teachers

Volunteer Sample
One-Way Tables and Cross-Tabulation

Retrospective Interview
1a, 1b, 1c, 2a, 2b, 2c

1a, 1b, 1c, 2a, 2b, 2c
EVALUATION LIMITATIONS
The use of multiple data sources, including analysis of student assessment performance scores, progress
monitoring and usage frequency data, teacher survey, and interview strengthens the discussion and subsequent
recommendations. Despite the fact that all 3rd- 5th grade teachers and interventionists were invited to participate, low
response rates on the faculty surveys could impact the perceived credibility of the results. In addition, the limited amount
of usage time generated by some students in all grade levels can skew the results of the level of efficiency of the Istation
program increasing student reading performance scores. In addition, there is some discrepancy with the number of
students assessed in each benchmark assessment and the number of students enrolled in the classrooms. This may be due
to students withdrawing from school during the school year, answer documents not being scanned within the assessment
window, or scores not being counted. The absence of student assessment scores can skew the data to show that there is a
higher number of students failing or passing. Another factor that can limit this evaluation is the absence of teacher training
or information of how to use the program, and no correlation with other assessment pieces. As of now campus teachers
use Istation to perform progress monitoring of their students and document the ISIP score at the beginning, middle, and
end of the year. Theres no follow up of how to correlate the scores students obtain in the program with student academic
progress or assessment performance. Istation is used as a resource in isolation.
19
SECTION V. RESULTS
STUDENT DATA RESULTS
STUDENT ASSESSMENT PERFORMANCE SCORES- EDUPHORIA
Performance scores for 316 students were analyzed using Eduphoria, the school districts data program, to assess
students performance across four different assessments. The assessments consisted of District Assessment 1 (DA1), Mock
STAAR, District Assessment 2 (DA2), and Spring Common Assessment 1 (SprCA1). Data collected is reflected on
figures 7-9. The data was distributed by each grade levels performance on each assessment and the language it was taken
in.
pa
n.
-S
Sp
rC
A1
-E
ng
.
Sp
rC
A1
pa
n.
-S
DA
2
-E
ng
.
DA
2
pa
n.
-S
ST
AA
R
-E
ng
.
M
oc
k
M
oc
k
ST
AA
R
pa
n.
-S
DA
1
DA
1
-E
ng
.
FIGURE 7. 3RD GRADE STUDENTS ASSESSMENT PERFORMANCE SCORES
Figure 7 shows the performance of 3rd grade students in both monolingual and dual language classrooms.
Students scores were categorized based on language of assessment. All students in 3rd grade who are enrolled in dual
language classrooms took the assessment in Spanish in alignment with the models framework. The students in
monolingual classrooms were administered the English equivalent of the exam.
20
Sp
an
.
2Sp
rC
A
Sp
rC
A
1-
En
g
n.
Sp
a
2-
2E
ng
.
DA
kS
TA
A
M
oc
DA
-S
pa
R
R
kS
TA
A
oc
M
n.
.
-E
ng
n.
Sp
a
1DA
DA
1-
En
g
FIGURE 8. 4TH GRADE STUDENTS ASSESSMENT PERFORMANCE SCORES
Figure 8 shows the performance of 4th grade students in both monolingual and dual language classrooms. In fourth
grade the dual language framework follows a 50/50 model. This means that some students took the assessments in English
only, or in both English and Spanish.
FIGURE 9. 5TH GRADE STUDENTS ASSESSMENT PERFORMANCE SCORES
21
Sp
an
.
1Sp
rC
A
1Sp
rC
A
Sp
a
2DA
En
g
n.
.
2DA
-S
pa
oc
kS
TA
A
R
oc
kS
TA
A
En
g
n.
.
-E
ng
n.
Sp
a
M
DA
DA
1-
1-
En
g
Figure 9 shows the performance of 5th grade students in both monolingual and dual language classrooms. Just like
in 4 grade, students in 5th grade follow a 50/50 model that allows them to take assessments in either English only, or in
both English and Spanish.
th
STUDENT PROGRESS MONITORING DATA & FREQUENCY OF USAGE- ISTATION/STAFF SURVEY
Students progress and frequency of Istation use was monitored and assessed via Istation logins and a staff survey.
Figures 10-13 show the distribution of the data based on frequency of use, duration of use, opportunities to engage in the
program, and student ISIP scores distribution.
FIGURE 10. FREQUENCY OF STUDENT ISTATION USE
6%
29%
65%
Daily Basis
3-4 Times a Week
At Least Once a
Month
Never
22

Figure 10 shows the amount of time students spent using Istation in grades 3-5.
FIGURE 11. DURATION OF STUDENT ISTATION USE
12%
18%
18%
53%
10-15 minutes
20-30 minutes
40-45 minutes
Other
Figure 11 shows the duration of students time on the Istation reading program. 53% of teachers surveyed reported
that their students use the Istation program for 20-30 minutes. 17% of teachers reported that students engage in the
program for 10-15 minutes, and 18% reported that their students use Istation for 40-45 minutes.
FIGURE 12. STUDENT OPPORTUNITIES FOR ISTATION USE
Opportunities for Istation Use
39%
43%
Computer Lab (with

teacher of record 1x a
week)
During CAFE
Instruction
Other
Computer Lab (with

interventionists 5x a
week)
During Daily 5
18%
Figure 12 illustrates the opportunities offered to students for them to engage in the Istation program.
FIGURE 13. STUDENT ISIP SCORE DISTRIBUTION
23
5th Grade Monolingual
5th Grade Dual
4th Grade Monolingual
4th Grade Dual
3rd Grade Monolingual
3rd Grade Dual
Figure 13 shows the number of students that have demonstrated a steady increase in their ISIP score based on
their beginning and middle of the year (BOY/MOY) progress monitoring. Scores for students who were assessed at both
benchmark points were taken into account. Students who withdrew before the middle of the year assessment or enrolled
halfway through the year were not included in the data.
STAFF DATA RESULTS

STAFF SURVEY RESULTS
All 19 teachers and 4 of the 7 interventionists (those that service general education students) were sent a link to a
staff survey via email. Of the total 23 staff members emailed only 17 replied (74% return rate). The staff survey consisted
of 17 questions which were formatted as likert scales, yes and no questions, multiple choice, and open-ended response.
The first question provided demographic information that illustrated the distribution of staff members by position (see
Figure 14). Results of the staff survey are displayed in the following three sections: 1) Teacher Use and Familiarity With
Istation Reading Program Tools and Resources, 2) Istation for Student Intervention, and 3) Effectiveness and Importance
of Istation Reading Program.
FIGURE 14. STAFF PARTICIPANTS BY POSITION
24
12%
12%
35%
18%
24%
3rd Grade
Teacher
4th Grade
Teacher
5th Grade
Teacher
2nd-3rd Grade
Interventionist
4th-5th Grade
Interventionist
TEACHER USE AND FAMILIARITY WITH ISTATION READING PROGRAM TOOLS AND RESOURCES
Four of the 17 questions on the staff survey were related to the topic of Istation tools and resources, and to the
teacher use of these tools. These questions were based on teachers familiarity with the program and the awareness and
implementation of the Istation tools and resources in their instruction/intervention. Question 2 assessed the general
familiarity that teachers have with the reading program resulting in 89% of staff members reporting they were familiar
with the reading program (see Figure 15). Question 7-9 assessed staff members knowledge and use of the Istation
Reading program tools and resources. 59% of teachers were aware of the tools, but less than half actually used the
resources provided in their instruction. The majority of teachers found the ISIP Summary Report to be the most helpful
tool and very few of the teachers surveyed actually explored the resource database. 10 out of 17 teachers reported not
using any of those tools at all (See Figures 16-18).
FIGURE 15. HOW FAMILIAR ARE YOU WITH THE ISTATION READING PROGRAM?
6% 12%
24%
59%
Not Familiar- 1
Very Little- 2
Somewhat
Familiar- 3
Familiar-4
Very Familiar- 5
25

FIGURE 16. HAVE YOU HAD AN OPPORTUNITY TO UNPACK THE ISTATION PROGRAM AND EXPLORE ITS
RESOURCES?
12%
29%
59%
Yes
Yes, but I have
only used the
summary reports.
No
FIGURE 17. WHAT ISTATION TOOLS DO YOU FIND MOST HELPFUL WHEN EVALUATING STUDENT DATA?
None
Teacher Resources
ISIP Distribution
ISIP Tier Movement
ISIP Skill Growth
ISIP Summary Report
FIGURE 18. WHAT RESOURCES DO YOU USE FROM THE ISTATION TEACHER RESOURCE DATABASE?
26
None at all
Alternative Stories and Poems
Passages
Books- Istation Reading/ Reading en Espaol
Cycle Based Interventions
Skill Based Interventions
ISIP Espaol- Lecciones
ISIP Advanced Reading Interventions
ISIP Early Reading Interventions
0
10
ISTATION FOR STUDENT INTERVENTION
Questions 3 and 10 focused on teachers implementation of Istation Reading program as an intervention tool for
students. Both questions consisted of an open-ended opportunity to allow teachers to use their words to describe how they
integrate the reading program into their intervention protocols. Question 3 asked: Do any of your students use Istation as
part of their reading interventions? How so? Responses for this question varied from reading centers, before and after
school programs, to daily use:
My students have the Istation as part of their Daily 5. If they finish they can then go to the Istation.
Students below reading level will use the Istation 2 to 3 times a week.
I have a dual-language class; therefore my entire class receives intervention through Istation.
Only at BOY, MOY, EOY time.
Yes, the Tier 3 interventionists utilize Istation to intervene and help students improve their reading skills.
Yes, this program is offered as a tool during morning and after school tutoring.
My Tier 3 students do not have time to work on Istation during their pullout time because of strict programs we are
implementing. However, I do discuss the importance and give out rewards for them meeting their weekly usage.
We have 4th grade students that are going to the library and using the computers for Istation on a daily bases for 2030 minutes.
Tier 2-3 students are staying afterschool and using Istation in the computer lab 4-5 days a week for 45 minutes.
My Tier 3 group is pulled out during the day by an Interventionist to use the computer in the library and do Istation
once a week.
27

Question 10 asked: How do you use the Istation student data to plan for reading instruction and interventions? Staff
members responses vary from using the data extensively, to not knowing of its availability to teachers. Some of the
responses included:
Not really. I mainly use the ISIP score as a progress monitor and to see in what areas students need work, but
groups are arranged by DRA and/or based on Common Assessment scores.
I have based many LOA (Language of Assessment) decisions on tier movement and overall growth throughout the
year.
I check their score on the report and see if theyre on level. Then, I create guided reading groups for instruction.
The data helps me form my small groups and target certain skills (fluency, vocabulary, comprehension, etc.) It helps
me form groups and pick skills to focus on.
I look at the DRA correlation and compare it to the DRA level at which I tested the student. I use the tiered report to
aid in forming reading groups and the individual intervention recommendations when planning lessons.
I have not incorporated this much in my data driven instruction except for BOY, MOY, and EOY.
I still have to figure out how to read the data.
Didnt know it was available.
We gathered the ISIP scores to form our Tier 3 groups for our intensive intervention.
I use the graph to show student growth.
I dont. I use DRA.
EFFECTIVENESS AND IMPORTANCE OF ISTATION READING PROGRAM
Five of the 17 questions on the staff survey addressed the importance of the Istation Reading program at Ruth
Barron Elementary. Figures 19- 23 show the data collected for each of these questions.
FIGURE 19. DO YOU THINK THAT THE ISTATION READING PROGRAM HAD AN IMPACT ON YOUR STUDENTS
READING COMPREHENSION?
35%
6%
59%
Yes
No
Not Sure
Figure 19 illustrates question 12s assessment of staff members perceived impact of Istation on students reading
comprehension. 59% of teachers feel that the Istation program has made an impact on their students, but 35% are still
unsure.
28

FIGURE 20. AFTER USING THE ISTATION READING PROGRAM ON A CONSTANT BASIS, HAVE STUDENTS
DEMONSTRATED A STEADY INCREASE IN THEIR ISIP SCORE?
12%
12%
Yes
76%
No
Not Sure
Question 13 evaluated whether teachers had seen an increase in their students ISIP score during their use of the
program. The data is represented in Figure 20.
FIGURE 21. WHEN CONSIDERING TIER 2 AND 3 STUDENTS, HOW EFFECTIVE DO YOU THINK ISTATION
HAS BEEN IN DEVELOPING STUDENTS READING SKILLS AND INCREASING THEIR COMPREHENSION
SCORES?
10
4
2
1.00
2.0
3.0
4.0
5.0
Figure 21 represents question 14s responses. This question used a likert scale to assess staff members perceived
level of Istations effectiveness on student reading skills and development. On the scale 1 represents not effective and 5
29

represents highly effective. 10 out of 17 teachers found the program to have some effect on students reading skills and
comprehension. Only 5 staff members felt that the program had a moderate to high level of impact.
FIGURE 22. COMPARED TO DRA DATA, HOW ACCURATE AND VALID TO YOU CONSIDER THE ISIP DATA TO
BE?
8
5
3
1
1.0
2.0
3.0
4.0
5.00
Question 16 used a likert scale to assess staff members perceived level of Istations accuracy compared to DRA
data. On the scale 1 represents not accurate and 5 represents highly accurate. The majority of teachers, 16 out of 17, find
Istation to be little to non-accurate when compared to the data obtained by DRA. (See Figure 22)
FIGURE 23. HOW IMPORTANT DO YOU CONSIDER THE ISTATION READING PROGRAM TO BE FOR OUR
STUDENT POPULATION AND CAMPUS?
8
0
1.0
2.0
3.0
4.0
5.0
Question 17 used a likert scale to assess staff members perceived level of Istations importance to students and to
the campus. On the scale 1 represents not important at all and 5 represents essential. 13 staff members felt that the
30

program has somewhat of an importance, but dont see it as crucial to student success in the area of reading and to Ruth
Barron's systems. The data is represented in Figure 23.
STAFF INTERVIEW RESULTS
Seven teachers agreed to be interviewed, of which six were actually interviewed. Each individual interview
consisted of open-ended retrospective questions and lasted for approximately 40-60 minutes. The format of the questions
generated followed four categories. In each category there was a lead-off question to initiate dialogue, and then there were
follow- up questions to clarify and deepen discussion. Within the four categories identified, there were covert categories
addressed in order for data analysis. The interview questions asked the staff members to reflect upon their personal
knowledge and use of the Istation Reading program, student reading academic performance prior to and after the use of
Istation program, opportunities and frequency of students use of program, and perceived effectiveness of Istation Reading
program.
STAFF KNOWLEDGE AND USE OF ISTATION READING PROGRAM
Self- reported use of the Istation Reading program and its resources varied among the six interviewees, however,
the majority of them indicated that they used the ISIP Summary Report to progress monitor and provide reading
interventions for their students. Three of the six interviewees reported that they felt comfortable in analyzing the data
provided by Istation and used it to update their reading groups on a frequent basis. The other three interviewees reported
that they only reviewed the ISIP data 3 times a year, during beginning (BOY), middle (MOY), and end of the year (EOY)
assessment window. Three of the six interviewees reported that they only use the English version of Istation despite
having Spanish speaking students in their class. This is due to district guidelines. The three interviewees expressed
concern in regard to this policy. Participants C and D reported that they began to use the Spanish Istation component to
help students who were low in reading comprehension in Spanish.
In regards to the tools and resources provided by Istation, 2 out of 6 participants indicated that they were aware of
all of the resources provided and used them actively as part of their interventions and progress monitoring. Both
participants A and D reported using the ISIP Summary Report and Skill Growth Report to determine the areas of need for
their students. Based on the data presented in this reports, they would then use the ISIP Early Reading Interventions and
Skill Based Interventions in their guided reading groups. Participant A used the books provided by Istation to have
students work on fluency and comprehension. In addition, Participant A used the Alternative Poems to help supplement
resources during their review for poetry. Participant A reported using the ISIP Tier Movement report to track students
progress and threshold jumps.
3 out of 6 participants reported that they didnt use any other report besides the ISIP Summary or any of the
resources because they didnt know about it. Participant F stated, I knew about some of the tools we could use because
it was briefly discussed at the beginning of the year, but I didnt dig for them on the site.
STUDENT ACADEMIC PERFORMANCE PRIOR TO & AFTER ISTATION USE
When asked about students reading performance prior to the implementation of Istation, all six interviewees
reported to having varied results. At the beginning of the year Participant A reported having 3 students out of 11 on or
31

above level according to DRA (Diagnostic Reading Assessment) standards. 5 students were one year behind and 3
students were more than 1 year behind. Participant B had 9 out of 16 students on level. Of those sixteen, 4 were one year
behind, and 3 students were more than one year behind in their reading levels. Participant C stated that in Spanish 1there
was only 1 student on level and in English there were 0 students on reading level. In Spanish 9 students were below level
by one year, and 5 were below level more than one year. Participant D reported that 13 out of 17 students were below
level. 6 students were 1 year or less below their reading level, and 7 were below level by two years. Participant E stated
that most of the students in the class were between a 2 nd and 3rd grade reading level. 5 students were on level and the
remaining 13 were at least 1 year behind. Participant F only DRA tested the students that did not meet standard on the
STAAR test. Eight students were tested and of those 5 were on reading level, and 3 were below level by 1 year.
All six participants were asked if the ISIP score provided by Istation at the first benchmark given matched the
DRA scores reported. Five out of six participants stated that the data did not correlate. Participant A stated that, the DRA
score was higher, but the ISIP score was lower. The scores did not start matching up until the second assessment.
Participant C stated that the class was split in half. Half of the students DRA and ISIP scores matched and the rest did not.
According to Participant C, students were getting higher scores on ISIP. Participant D was the only participant that
stated that the data matched for both the DRA and ISIP assessments.
For District Assessment 1 all six participants reported that their students did not do well. Participant A said that,
the highest score was a 60 and the lowest was 21. The average for Participant As class was in the 40s. Participant A did
not think that data from DA1 compared to that of common and weekly assessments. Weekly tests are shorter and easier,
so scores were better. Common assessments targeted specific skills and the district assessment was the first time my
students saw 3 passages back to back, said Participant A. Participant B reported that students had a 42% average. Only 1
student in the class passed the test. District Assessment 1 was much harder. I had more students passing each time we
took a common assessment, said Participant B. Participant C stated that students did not do well in DA1, but the data
matched common assessments because at the beginning of the year students were still learning 4 th grade content.
Participant F only had 3 students pass DA1. These students scores were between 70-80%. Participant F reported that 9
out of 17 students passed DA1. The number of students who passed matched the same numbers in the common
assessments administered at the beginning of the year.
The six participants interviewed reported that their reading interventions consisted of guided reading groups that
followed the Ruth Barron protocol, but the students who used Istation as part of their intervention varied. Participant A
stated that, 4 students qualified for Tier 3 intervention at the beginning of the year, so they used Istation twice a week
with interventionists and everyone else used it once a week during computer lab time. Participant B didnt have any
students qualify for Tier 3 interventions, but the Tier 1 and 2 students were serviced in class. They used Istation for
monthly progress monitoring and at the computer lab once every other week because it was alternated with Think
Through Math. Participants C, E, and F reported that Istation was only used for progress monitoring at BOY, MOY, and
EOY. Only their Tier 3 students used Istation on a weekly basis with the interventionists. Participant D used Istation for
progress monitoring 3 times a year, but also had students use it on a daily basis during acceleration block.
All six teachers saw a significant growth in their students performance scores after implementing Istation.
Students improved their scores by the second major district assessment, Mock STAAR. Scores increased between 15 to 40
points.
ISTATION USE BY STUDENTS- OPPORTUNITES AND FREQUENCY
32

The six interviewees reported varying degrees of student access to the Istation program. All participants reported
that their entire class accessed Istation at least once a week during their designated computer lab time. Participants C and
E stated that their students used Istation in class during Daily 5 centers but the time they spent on the computer was
limited to 15 minutes because they only had 2 computers available and the iPads dont work with Istation. Participant F
reported that students didnt use the program in class, but instead went to the library or the computer lab during their
designated time. Participants A, B, and D reported to letting students use Istation during Daily 5 centers and provided each
student at least 20 minutes of access. Participant D had students at the Tier 2 and 3 levels come in the morning during
acceleration block and work on Istation. This was done on a daily basis and students had access for 15-20 minutes.
Participant D would review students progress during their conferences and discuss what needed to be worked on.
Of all six participants 4 reported that their students used Istation at home. Participant A knew of 1 student,
Participant B reported 2 students, Participant D reported 2 students, and Participant E knew that 1 student used the
program at home. All six participants stated that the biggest deterrent for students to use Istation at home was the need to
download the program onto the computer. Most parents dont know how to download Istation or find it complicated,
said participant F.
EFFECTIVENESS OF ISTATION READING PROGRAM
Four of six participants reported that Istation was an effective program to implement. All six staff members
interviewed stated that it was difficult to attribute all student success to Istation because there were many intervention
programs and reading strategies being implemented at the campus, therefore it is difficult to figure out what worked best.
Five of the six participants reported that Istation provided targeted instruction for students and provided support in the
areas they needed to work on. Participant C reported that having access to both English and Spanish versions of Istation
would have been more beneficial. One out of six participants reported that the ISIP Score data was comparable to that of
DRA. All other participants felt that DRA levels provided more information in regards to students reading
comprehension and had more validity. All six participants expressed the need for staff training in regards to the program,
and the implementation of a uniform system at the campus level. Also, all participants expressed that both teachers and
students need to know the importance of this program and make it part of their intervention routine.
33
SECTION VI. INTERPRETATION AND DISCUSSION

INTERPRETATION OF STUDENT RESULTS
Student performance scores in District Assessment 1 (DA1), Mock STAAR, District Assessment 2 (DA2), and
Spring Common Assessment 1 (SprCA1) show a variation in each grade level from one assessment to the next. In Figure
7 it was evident that there is a larger number of third grade students who took the assessments in Spanish that dont meet
the standard score in each assessment vs. the number of students who took the assessments in English and didnt meet the
standard. But, despite this difference in passing rates, both English and Spanish test taking students have shown a gradual
increase in the amount of students meeting standard for each assessment. In the Mock STAAR assessment only 1 student
obtained commended performance, and the student was assessed in Spanish.
Figure 8 showed that the numbers of fourth grade students who took the assessments in English and did not meet
standard are larger than the number of students who took the assessments in Spanish and did not meet standard in 3 out of
4 assessments taken. The number of students who were assessed in Spanish and met standard is greater than that of
students assessed in English and who met standard in 3 out of 4 assessments. The only exception was on the Spring
Common Assessment 1. The students assessed in English showed a steady increase in the amount of students meeting
standard each assessment. Students assessed in Spanish followed the same trend with the exception of District Assessment
2. In this assessment students showed a drop in passing scores, but the group made gains on the following assessment,
Spring Common Assessment 1.
Data in Figure 9 illustrates the trends found in 5th grade performance scores. In this grade level the number of
students who took the assessment in English and did not meet standard in 4 out of 4 assessments was greater than the
number of students who took the assessment in Spanish. But despite this distribution of numbers, in 3 out of 4
assessments the number of students being assessed in English that met standard was greater than the number of students
assessed in Spanish who met standard. Students assessed in Spanish have had mixed outcomes in their performance data.
There was a significant drop between DA1 and Mock STAAR with the number of students assessed in Spanish that met
standard. A low increase was gained by DA2, and the greatest leap of passing scores of students assessed in Spanish was
found between DA2 and Spring CA 1.
Each grade level has show some drops and gains throughout their assessment history, but it is unclear whether the
impact on their scores is derived from the use of Istation or from a multitude of interventions that are in place at the
campus and other factors.
The second component that was examined was students progress and frequency of Istation use. Figures 10-13
showed the distribution of the data based on frequency of use, duration of use, opportunities to engage in the program, and
students ISIP score distribution. Figure 10 showed that the majority of students in grades 3-5 use the Istation reading
program 3-4 times a week. About 29% of staff members surveyed stated that their students use Istation on a monthly
basis. There arent any students that use the program on a daily basis. A huge factor that impacts the results shown in this
figure is the access that students have to computers. The majority of students use the program when they attend their
designated computer lab time.
Figure 11 showed that 53% of teachers surveyed reported that their students use the Istation program for 20-30
minutes. 17% of teachers reported that students engage in the program for 10-15 minutes, and 18% reported that their
students use Istation for 40-45 minutes. The range for the duration of students use of Istation is from 10-45 minutes. 10Istation and Reading Success
34

30 minutes on the program reflects the students that use the program 3-4 times a week, and 30-45 minutes reflects the data
for students who use Istation once a month during their computer lab time.
Figure 12 illustrated that the majority of students use the Istation program once a week with their teacher of
record during their computer lab time. This is because the availability of computers in the classroom limits the
opportunities for students to engage in the Istation program. Despite having access to only 2 computers in the classroom,
there is a substantial amount of students that use the program during their Daily 5 centers. 39% of teachers surveyed
reported scheduling Istation use into their Daily 5 routine. These students have to use Istation in rotations. Unfortunately,
this means that their time on the program is limited. The Istation program is not supported on iOS tablets therefore
students are confined to using it on the desktop computers located in the classroom and the computer lab. When there is
only 1 opportunity a week to attend the computer lab, and time on Istation in the classroom is limited to 15-20 minutes,
its evident that students arent maximizing their use of Istation.
In figure 13 we learned that of the 69 dual language 3 rd grade students assessed during BOY and MOY
benchmarks, 58 demonstrated an increase and 11 students decreased in their ISIP score. 44 monolingual 3 rd grade students
were assessed and of those 32 increased, 11students decreased and 1 student maintained the same ISIP score. In 4 th grade
46 dual language students were assessed and of those 39 students increased and 7 decreased their score. 50 4 th grade
monolingual students were assessed and of those students 39 increased their score and 11 decreased. Of the 51 5 th grade
dual language students that were assessed, 35 demonstrated an increase in their score and 16 decreased their ISIP score.
Of the remaining 43 5th grade monolingual students 32 increased and 11 decreased their score. As the data was analyzed it
was evident that there were some students that withdrew or enrolled in the middle of the year. These students scores
skewed the data slightly, but overall the majority of students across all grade levels and in monolingual and dual language
programs demonstrated an increase in their ISIP score from their BOY to MOY assessment. A defining factor for many of
the students was the amount of time they spent using the Istation program.
It is clear based on these data categories that students growth in the area of reading and their performance in
subject-related assessments is impacted by the amount of exposure they have with Istation, and of their progress on its
ISIP evaluation. Students need to know that the ISIP program is a tool that helps them work on the skills they need. It
provides exposure to reading passages and skill-based games that help them increase their reading fluency, vocabulary,
and comprehension. This in turn serves as an intervention that can help close the gap in student achievement in
standardized assessments.
INTERPRETATION OF STAFF RESULTS

All staff members that were surveyed and interviewed used Istation as part of their interventions. The number of
teachers that were knowledgeable about he programs tools and resources varied as did the frequency of student use of
Istation in their classes. 59% of staff members are familiar with the Istation Reading program and the same percent have
explored its resources. 46% of staff members found the ISIP Summary Report to be the most helpful tool when evaluating
student data. Of the staff members surveyed, 48% dont use any of the resources located in the teacher database.
Responses to the use of Istation as an intervention tool indicated that teachers differ in their implementation of the
program and vary in their use for the data collected from its evaluations. Several teachers use the Istation data to develop
and differentiate student guided reading groups and monitor student progress. Of the teachers surveyed and interviewed 7
reported using the Istation program with fidelity and providing multiple opportunities for students to access the program
besides their designated computer lab time. The majority of teachers expressed that the data collected through Istation and
its teacher resources are helpful, but a couple of the members surveyed expressed not knowing about them.
35

59% of teachers felt that Istation had impacted their students reading comprehension and 76% had seen growth in
their students ISIP score since the programs implementation. The majority of teachers felt that Istation was moderately
effective on their students comprehension scores, but 47% of staff members think that the ISIP score is not as accurate as
DRA data. These staff members feel that Istation is a helpful tool for intervention, but not as an assessment.
The relatively low percentage of teacher buy-in for the Istation program is due to the lack of training and structure
provided by the campus administration and instructional/intervention teams. Staff members were expected to implement
Istation without knowing much about the program or how to interpret the data that was collected by its assessments. In
addition, the majority of staff did not use the resources Istation has for instruction because teachers were never informed
of them and taught how to use them. Third grade was expected to use the program monthly for progress monitoring, but
4th and 5th grades were only asked to monitor 3 times a year (BOY, MOY, EOY). Obviously the data gathered by each
grade level will vary and so will the impact it has on the types of interventions students receive. It appears that the only
grade level that has had a continuous and loyal adoption to Istation is third grade. Students are using the program for
reading centers, intervention, and as progress monitoring. The amount of time students in this grade level use Istation is
higher than that of 4th and 5th grade students. This will impact their exposure to reading skills such as fluency, vocabulary,
and comprehension, and thus impacting their progress in reading.
In order for staff members to feel that Istation is important to Ruth Barrons campus goals and student population,
the program must be part of classroom expectations. This means that just as word walls, CAF, Daily 5 rotations, and
math centers are part of the routines established in the classroom setting, Istation needs to be given the same priority. Its a
tool to be used by students in order to enhance reading skills. In addition it provides crucial information that can be used
to supplement or further define DRA scores, generate intervention groups, target skills, and help make language of
assessment decisions.
DISCUSSION
A synthesis of the results of this evaluation indicates that the majority of staff members and students in grades 3-5
are not using the Istation Reading program to its fullest potential. This is of great concern because Istation is a resource
that is adopted at the district level and provides valuable data, documentation, and tools for student reading achievement.
There is disconnect at the campus and district levels that limit the way this intervention is utilized and implemented.
Similarly in line with Ruth Barrons mission and goals, the Istation program evaluation goals included:
The lack of staff training and school wide systems for program implementation indicate that the administration and
instructional/intervention teams need to work collaboratively to prepare and roll out the necessary training and adoption
policy that will enable teachers and students in in grades 3-5 to adopt and implement Istation successfully. Only then,
teachers and students will use the program with fidelity and thus improve their reading performance.
36
SECTION VII. CONCLUSIONS AND RECOMMENDATIONS

CONCLUSIONS FROM THE EVALUATION
week or for 15 minutes a day during Daily 5 stations. For 4th- 5th grade the amount of time students are exposed to Istation
decreases exponentially. With the exception of one classroom in these two grade levels, everyone else seems to rarely use
the program or finds little value in it. The majority of teachers felt that Istation was another component in a long list of
interventions implemented by the campus. Given the lack of training and implementation of a uniform system for Istation
use, both students and teachers have not used the program effectively.
RECOMMENDATIONS FOR INTEGRATION AND MONITORING OF ISTATION PROGRAM

Recommendations for Ruth Barron Elementarys integration and use of Istation consist of two components, including 1) the
design and development of Istation program training for staff, and 2) the development of school wide systems for Istation
implementation and monitoring.
DESIGN AND DEVELOPMENT OF ISTATION PROGRAM TRAINING FOR STAFF
The administrative, intervention, and technology teams should attempt to develop professional development
training for teachers in order to inform them of the ins and outs of the Istation Reading program. The training should focus
on the use and implementation of the programs various components. By offering training to staff members, the schools
leadership team will be ensuring that staff members in grades 3-5 will successfully implement the program in their
classrooms and use it for student monitoring and intervention. Some topics to discuss in the training are:
37
Access to student data

Unpacking and Evaluation of student data (being able to read data shown in reports)
Exploration of Data Tools and Teacher Database Resources
Modeling of how to use the data gathered and resources as part of reading intervention
Student log-in from home
week or for a 15 minutes a day during Daily 5 stations. In grades 4 and 5 there is a drop in the amount of time students use
Istation. The majority of teachers felt that Istation has little value in the campus intervention program because it is a
component that is part of a long list of systems already implemented by the campus.
DEVELOPMENT OF SCHOOL WIDE SYSTEMS FOR ISTATION IMPLEMENTATION & MONITORING
In order to ensure the success of Istations implementation, Ruth Barron Elementary needs to set up a system that
assists staff and students in adopting the program. A suggestion for increasing student buy-in is developing a system that
is similar to the Accelerated Reader (AR) program established on campus. There can be monthly check-ins to monitor
student progress. These check-ins can provide a reward via current school behavior management systems to students that
show continuous use and progress. Also, the program needs to have a consistent implementation across grade levels. If
one grade level is using the program for monthly progress monitoring and the others arent, the other grade levels are
missing out on student interaction with Istation. All grades should use the program on a consistent basis. In addition,
students in grades 4-5 need to be able to access the program in both English and Spanish. Students in the dual language
program need to be able to uses the resources in both languages. Once teachers have been trained, they can in turn deliver
the program and login information to their students and parents.
38
REFERENCES
Hoelzle, H. (2012). Predicting Student Performance on the Developmental Reading Assessment, 2 nd Edition: An
Independent Comparison of Two Different Tests. Retrieved from
http://www.istation.com/Content/downloads/whitepapers/DRA_study.pdf
Mathes, P. (2009). Istation Indicators of Progress Early Reading Reliability and Validity Evidence. Retrieved from
http://www.istation.com/Content/downloads/studies/isip_rr.pdf
Bugbee, A.C. (2011). The Effectiveness of Istation in a School: East Baton Rouge Parish School System. Retrieved from
http://www3.istation.com/Content/downloads/whitepapers/East_Baton_Rouge.pdf
Robinson, B. (2002). COLLIT Project: A Background note from Bernadette Robinson. Retrieved from
http://www.fivehokies.com/Evaluation/Evaluation%20Approaches/Management%20Oriented/CIPP%20Approach%20to
%20Evalaution.pdf
Descriptive and demographic information regarding Ruth Barron Elementary School is located at
http://cms.pflugervilleisd.net/Page/1534
Descriptive information regarding Istation is located at http://www.istation.com/About
Ruth Barron Elementary School mission and goals are detailed in the 2013-2014 Campus Strategic Improvement Plan
document, which is located at http://cms.pflugervilleisd.net/Page/1534
39

B.Becker Istation Program Eval

Transféré par

Informations du document

Copyright

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

B.Becker Istation Program Eval

Transféré par

Droits d'auteur :

Istation Program Evaluation

Istation and Reading Success

14850 Harris Ridge Blvd., Pflugerville, TX 78660

Istation Program Evaluation

Section II. Introduction... 11

Istation and Reading Success

Istation Program Evaluation

Section IV. Description of the Evaluation Study...... 15

Istation Program Evaluation

Section VI. Interpretation and Discussion...... 33

Section VII. Conclusions and Recommendations..... 36

Istation and Reading Success

Istation Program Evaluation

SECTION I. EXECUTIVE SUMMARY

Istation Program Evaluation

DESCRIPTION OF THE EVALUATION STUDY

Istation Program Evaluation

Istation and Reading Success

Istation Program Evaluation

INTERPREATION AND DISCUSSION

INTERPREATION OF STAFF RESULTS

Istation and Reading Success

Istation Program Evaluation

CONCLUSIONS AND RECOMMENDATIONS

RECOMMENDATIONS FOR STAFF TRAINING

Istation and Reading Success

Istation Program Evaluation

Istation and Reading Success

Istation Program Evaluation

SECTION II. INTRODUCTION

Jos Medina (Ruth Barron Elementary Principal)

OVERVIEW OF REPORT CONTENTS

Istation and Reading Success

Istation Program Evaluation

SECTION III. FOCUS OF THE EVALUATION

2 Descriptive information regarding Istation is located at http://www.istation.com/About

Istation and Reading Success

Istation Program Evaluation

Ruth Barron Elementary

RELIABILITY AND VALIDITY OF ISTATION READING PROGRAM

Istation Program Evaluation

RELATIONSHIP OF ISTATIONS ISIP AND OTHER DEVELOPMENTALLY APPROPRIATE READING ASSESSMENTS

Istation and Reading Success

Istation Program Evaluation

SECTION IV. DESCRIPTION OF THE EVALUATION STUDY

Istation and Reading Success

Istation Program Evaluation

FIGURE 2. TEACHER DISTRIBUTION BY GRADE LEVEL

RBES Teacher Distribution

FIGURE 3. INTERVENTIONIST DISTRIBUTION BY GRADE LEVEL AND ASSIGNMENT

4th & 5th Grade Interventionist

3rd Grade Interventionist

Istation Program Evaluation

FIGURE 4. DISTRIBUTION OF STUDENTS BY GRADE LEVEL

RBES Student Distribution by Grade Level

FIGURE 5. DISTRIBUTION OF STUDENTS BY ETHNICITY

RBES Student Distribution by Ethnicity

Istation and Reading Success

Istation Program Evaluation

FIGURE 6. DISTRIBUTION OF STUDENTS BY SPECIAL INDICATORS

RBES Student Distribution By Special Indicators

DATA COLLECTION AND ANALYSIS

TABLE 1. DATA SOURCES AND ANALYSIS PROCEDURES

All 3rd- 5th Grade Students

All 3rd- 5th Grade Students

1a, 1b, 1c, 2b, 2c

Istation Program Evaluation