Evaluation of RAZ Kids Online Reading Program

Evaluated By: Debbie Turner A Report Submitted To: Big Creek Elementary Cumming, Georgia May, 2012

Executive Summary Schools across the nation have been faced with the costly challenge of leaving no child behind. In 2001, the federal government passed the No Child Left Behind Act. One of the cornerstones of NCLB is Annual Yearly Progress measures. Students in each state must participate in assessments which measure annual growth. Schools are mandated to meet requirements on annual measures set forth by the state and federal government. Schools that do not meet AYP in the same subject for two or more consecutive years are placed in Needs Improvement status with escalating consequences for each successive year not meeting standards. AYP requires schools to meet standards in three areas. One of these critical areas is reading. The goal of NCLB is that 100 percent of all public school students achieve proficiency in reading by the year 2014. This is a very hefty goal. Keeping the lofty goals of AYP in mind, school administrators are struggling to come up with ways to meet these ever increasing goals. At the same time, states are facing cutbacks due to the current economic crisis. Funding for programs has been significantly cut. Administrators are forced to take a look at their current programs to ensure that needs are being met, and goals are being attained. To meet the demands of APY, teachers at Big Creek have stressed the need for supplemental reading programs to assist them in meeting the wide range of developmental levels of their students. With the ever increasing goals of AYP, it is crucial that programs are effective and at the same time, affordable. In an effort to supplement to the current curriculum and improve reading skills, Big Creek purchased a license to use the RAZ Kids reading program in August, 2011. Teachers received training on using the software in August, 2011 during preplanning. The program was fully implemented by September, 2011. A formal evaluation was requested to determine the efficacy of the program and whether or not the program merits continuous funding. The focus of this evaluation was to determine teacher and student attitudes toward the program and impact on student achievement in reading. The following aspects of the program were examined: 1) the teachers level of satisfaction with the program; 2) the students level of motivation in using the program; and 3) student achievement outcomes in reading. Specific evaluation questions included the following: 1. Does the program have an impact on students reading text level? 2. Are teachers satisfied with the overall program? 3. Does the program increase student motivation and self-confidence in reading? Data for the evaluation were collected through 1) a report on student reading level change through the RAZ on line teacher tool, 2) an attitude survey completed by classroom teachers, and 3) a motivation survey completed by students.

Based on the current data, the average achievement in reading in all grade levels improved. This data was based on the average of the reading level changes based as reported within the RAZ program. Data shows that students were also highly motivated to use the program. The data shows that students feel more confident in their reading ability. The data shows that the students think the program is fun to use. The data also shows that teachers were satisfied with the program. Teachers felt that students did not have easy access to computers at school to utilize the program. While they were satisfied with the programs ease of use and effectiveness, they were not satisfied with the teacher assessments within the RAZ program. This evaluation revealed that RAZ is effective in improving students achievement in reading. The evaluation also revealed that students were highly motivated to read when using the RAZ program. While achievement and motivation improved, teachers felt that the teacher/classroom assessments within the program were difficult to use and the limited number of computers in the classroom is an obstacle in using the program effectively. Introduction: To supplement the current reading curriculum, Big Creek purchased a subscription to RAZ Kids in August, 2011. RAZ Kids is an online reading library resource which provides teachers with resources to assist students with reading practice. The Raz-Kids program is available through the website only and is not a program that has to be downloaded, but rather requires a subscription fee for access. The website allows students to practice listening and reading books on their level, as well as taking comprehension quizzes on the books they read. Teachers are able to track student progress and pull various reports for analyzing and review of the class or individual students. Raz-kids provides students with practice and a way for teachers to monitor students growth and use of the program. Raz-Kids is used as a resource for students in and out of the classroom. Teachers and students are able to login to the site by using their assigned username and password. Teachers assign each student a password for increased privacy. Teachers input student names into a class roster so each student has their own personal reading page and points.. Teachers also assign students reading levels based on student reading level assessments. In addition to assigning reading levels, teachers may also assign specific reading assignments to individual students. The goal of the program is to provide teachers with instructional tools that are easy to use and meet the reading needs of all learners. The RAZ program is designed to address the foundational components of reading. The focus of this evaluation was to determine if the program is worthy of continuous funding and use as a supplemental reading program. The program is designed to provide guidance in reading at differentiate reading levels. The program is also designed to be kid friendly and provide monitoring of student growth. Based on the programs design, the program was evaluated based on impact of student achievement in reading, student motivation, and teacher satisfaction with the program.

Several methods of evaluation were utilized to gather data during the evaluation. Teachers were surveyed regarding their level of satisfaction with the program. Students were surveyed regarding their level of motivation and self-confidence in reading while using the program. A document analysis was performed on reports gathered from the online teacher tool within RAZ to gather data on the changes in reading level of the selected evaluation participants. Description of the Program Evaluated When school administrators purchased the RAZ program in August, staff development was planned during the pre-planning period before the school year began. The program is a computer based, on line program. All certified staff received training on the use of the components of the RAZ program. All certified staff was expected to implement RAZ into their current reading instruction. The program was expected to be used on a daily basis in reading centers within the classroom, and as a homework assignment on any given number of days during the week. Teachers were expected to use the program for a minimum of one hour per week. Teachers created class rosters within the program. When school began, students in grades 1 - 5 were assessed using running records to determine their beginning reading levels. Students were then given preprinted log in cards so they could easily access the program at school, and at home. Students were given instructions on using the program. Teachers created student specific assignments within the RAZ program. Students listened to stories, practiced fluency, recorded themselves reading, and took reading comprehension quizzes on the stories that they read. Students could log on to the program at school, or at home. Teachers assigned a number of minutes per day that students were expected to practice reading on RAZ for homework. This also gave parents an opportunity to become familiar with the program. Teachers monitored student progress, and use of the program, throughout the year by accessing the online teacher tool within RAZ. Teachers could adjust student assignments based on individual needs of the students. Program Objectives: After reviewing data from current state assessments, Smart Goals were made by administrators and staff to ensure student growth in reading. The goals are based on standards set forth by the state of Georgia. These goals were used when reviewing supplemental reading programs to determine whether the program could be effective in meeting these goals. Based on the Smart Goals, objectives for the program were determined to be as follows: 1. Student reading skills and achievement will be increased through guided reading, instruction and practice. 2. Students motivation in reading will be increased. 3. Teachers would provide ongoing monitoring of student performance to identify student need and growth.

Program Components: The RAZ Kids program consists of several teacher and student components. The Teachers Corner within the program allows the teacher to leave the class a message, assign activities to students, check assignment progress, complete a running record and practice recordings. The site allows teachers to see what progress students are making in their reading level. The report tab allows teachers to see student activity, assignment progress, reading level progress and student reading rate. These reports can be pulled for individual students. The class report can also be run for assignment progress, class activity, running record progress and students that have completed running records. When running the reports teachers can see what books students have listened to, read, and completed a comprehension quizzes on. The report also analyzes the missed areas in comprehension. This area for teachers is important so that teachers can manage student reading and review any rules for student online reading when necessary. Within the student component of the program, students are able to listen, read and take comprehension quizzes on the assignment page. They are able to do the same in the bookroom; however, students do not receive points towards the next reading level when reading bookroom books. When on the assignment page or in the bookroom, students can listen to the story with animations and music for engagement. They are able to read stories and get assistance on specific words that are not normally on their reading level. If the teacher has assigned or the student chooses, they can record their reading through the microphone. The teacher is also able to pull the student reading rate from the reports section for individual students. In addition to listening and reading the online books, students are able take comprehension quizzes online. If the student needs help reading the question, they can click on the word and the program reads it to them as well as reading any possible answer choices. When the student chooses an answer the program responds with a check or X mark for correct or incorrect. The teacher can go back through the assessment to see what areas students need improvement. The program also offers a motivational component for students called RAZ Rockets. RAZ Rockets is a gaming feature within the program. RAZ Rockets is a system of rewards and incentives. Students earn stars for every book read and listened to, and quizzes and assignments successfully completed. Students use earned stars to buy items from the Raz Rocket Catalog to personalize their rocket. In order to use the RAZ Kids program, there are certain technology requirements. These requirements include: Internet Access: Windows- Internet Explorer 7.0 or Fire Fox 3.0; Mac-Safari/ Fire Fox 3.0 Windows XP/Vista or Mac OS 10.4 Adobe Reader or Mac Built-In PDF Reader Enable Cookies Flash 9+

Resolution: 1024 x 768 Headset: includes headphones, microphone, speakers Evaluation Method

The focus of this evaluation was to determine the satisfaction with the overall program and to determine if the program should continue to be funded for use as a supplemental reading program. The RAZ program is designed to increase student achievement in reading through guided practice, motivate students to read, and provide teachers with a system of being able to monitor student progress. Based on the design and goals of the program, the evaluation was based on students attitude towards reading, teachers attitude toward effectiveness of the program, and the impact on student achievement. The evaluation uses both qualitative and quantitative methods to evaluate the program. The Data was collected through 1) a survey of student attitudes 2) a survey of teacher attitudes, and 3) a data report of student reading level change. Participants: The general population of Big Creek includes 609 students and 50 classroom teachers in grades K-5. Because Kindergarten students are emerging readers, only students and teachers in grades 1-5 participated in the evaluation. Grades 1-5 consist of 529 students and 46 teachers. Because of the relatively small number of teachers, all teachers participated in the survey of teacher attitudes. Ten percent or 52 students were chosen to participate in the attitudes survey, as well as the achievement data collection. The students that were chosen to participate in the evaluation were randomly chosen. To randomly choose the students for the evaluation, the school bookkeeper was asked to randomly select 52 student ID numbers in grades 1-5. These ID numbers were then entered into an Infinite Campus program to gather student information and identify student participants. Procedures: After defining the purpose and scope of the evaluation, specifying the evaluation questions, and developing the evaluation design and data collection plan, surveys were constructed that were to be delivered to the teachers and the students. The attitude survey for teachers was constructed using SurveyMonkey.com. The link to the survey was sent to all teachers in grades 1-5 via the school email. Teachers were asked to complete the online survey within two weeks. During the first week of the evaluation, individual student reading level change reports on students that were selected to be participants in the evaluation were created and printed using the online teacher tool within the RAZ program. The selected students were then given the attitude surveys. These surveys were hard copy surveys. Classroom teachers of the selected students were asked to distribute the surveys. The teachers were asked to collect and return the surveys to a designated box in the teacher workroom within one week.

Data Sources: The data collection methods consisted of three sources. A student attitude survey, a teacher attitude survey, and individual reading level change reports within RAZ were used to collect data for the purpose of this evaluation. Students attitude towards reading while using the program was evaluated using a motivation survey (See Appendix 3). The survey consisted of 5 Likert items with 3 ordinal variables. The ordinal values were yes, no and maybe. The survey was very basic to ensure that students could effectively understand and evaluate the program. This survey ranked student attitude and motivation towards the program. The survey also evaluated student satisfaction with the materials within the program, student willingness to use the program, and student confidence in reading ability while using the program. Teacher attitude toward the program was evaluated using a survey created at SurveyMonkey.com (See Appendix 2). The survey was designed to measure overall perception and attitude toward the program. The survey consisted of 6 Likert items with 4 ordinal variables. The ordinal variables were very satisfied, satisfied, unsatisfied, very unsatisfied. The Likert items focused on teachers overall impression with the program and components within the program. Participants were asked to rate each item according to the scale. They were instructed that all responses would be anonymous. Student achievement was measured using the reading level progress report within the RAZ program (See Appendix 1). The report measures reading level change over a period of time. Reading levels within RAZ correlate to Lexile levels which are the same levels used on state yearly assessment and performance standards. Reading level change reports were created, and printed, for each participant. The reading level change report for each participant was analyzed to determine the number of reading level changes that occurred for each participant. The sum of these figures was used to calculate the average reading level change of the participants since the program began. Results: The results are described in relation to the overall goals of the evaluation. The results are designed to answer the overarching questions within the scope of the evaluation. Evaluation question 1: Does the program have an impact on students reading text level? Student growth and proficiency based on the reading level change reports from the sampling of the population is summarized in the table below.

The reading level changes of the 52 participants were used to find the mean (average), the median, and the mode of the number of reading level changes.The average reading level change based on the sampling of the population was an increase of three reading levels since the implementation of the program in August, 2011. Based on the results of the reading level change report, the program did demonstrate an impact on reading text level. Evaluation question 2: Are teachers satisfied with the overall program? All of the certified classroom teachers were asked to complete the survey regarding attitude and overall impression of the RAZ program (See Appendix). The survey was distributed to 46 teachers within the school. Of the 46 teachers, 32 teachers responded to the survey. The survey results, along with the survey questions are outlined in the table below. Very Satisfied 14 Satisfied Unsatisfied Very Unsatisfied 0

Question:
The RAZ Reading program is an excellent source for supplementing reading instruction. Students are given sufficient opportunities to access computers to use RAZ The components of the RAZ program are effective and easy to use. RAZ offers sufficient assessment tools. I have seen an impact on student reading ability. The on line teacher tool within RAZ is beneficial in monitoring student progress.

18

12

20

28 0 17 24

4 13 13 6

0 19 2 2

0 0 0 0

TOTALS:

83

66

43

Percentage:

43%

34%

22%

Based on the Likert items outlined in the survey, 43% of responses to the items indicated that participants were very satisfied with the program, 34% of the responses indicated satisfaction with the program, and 22% of the responses indicated unsatisfaction with the program. There were no responses that indicated participants were very unsatisfied with the program. Based on the responses, participants were least satisfied with the assessment tools within the program and the inability for students to access computers to use the program during the school day. The results of the survey indicate that the participants were satisfied with the overall program. Evaluation question 3: Does the program increase student motivation and selfconfidence in reading? Fifty two students were selected to participate in the survey of attitudes. The survey was distributed to these students by the classroom teachers. Of the 52 surveys that were distributed, 48 surveys were returned. The chart below outlines the questions and results of the survey. Questions:
I have fun reading on RAZ. I feel that I am a good reader when using RAZ. I like the books on RAZ I would like to read on RAZ more often. RAZ is a good reading program for kids.

Yes 45 37 41 38 46 207

No 0 0 2 1 0 3

Maybe 3 11 5 9 2 30

Total:

The results of the survey indicate that students have a very positive attitude about using the RAZ program. Ninety percent of the participants indicated that they would like to use RAZ more often. Eighty five percent of the students surveyed indicated that they like the books on RAZ. Seventy seven percent indicated that they feel increased confidence in their reading ability when using the program. Overall, the results of the survey indicate that RAZ does increase student motivation and self - confidence. Discussion The purpose of this evaluation was to determine if the RAZ program was to determine the satisfaction with the overall program and to determine if the program should continue to be funded for use as a supplemental reading program. The results of the evaluation revealed that reading achievement was impacted, teachers were satisfied with the program, and students were motivated to use the program.

The results revealed that there was an increase in reading achievement. After tabulating the reading level changes in all of the participants in the evaluation, it was found that there was an average of three levels of reading change. While it is unclear if the change is a result of the program itself, or reading instruction within the classroom, participants did show an increase in reading achievement. The results also revealed that teachers surveyed were very satisfied with the program. Forty three percent of the participants rated their satisfaction level as very satisfied on Likert items. The results also revealed that students were also very motivated to use the program. The surveyed students felt more self-confidence in their ability to read. While the results showed that teachers were generally satisfied with the program, the results also revealed that teachers were not satisfied with the assessment tools within the program. While teachers felt that the program was effective and easy to use, they felt that students did not have adequate opportunities to access computers to use the program. Based on the results of the collected data, the program could be used more effectively if students had access to more computers. Based on the results of the evaluation, the RAZ program should be continued to be funded and implemented as a supplemental reading program. Specific recommendations for future use of the program include: Adding additional computers for student access to the program Additional staff development on the use of assessment tools within the RAZ program Evaluation of data on state assessments to determine the impact of the program on reading scores In summary, all participants involved in the evaluation were satisfied with the program, and reading achievement was impacted. Students were motivated to read using the program. Because the results showed positive teacher and student attitude towards the program, and increased reading achievement, the program should be included in the school wide literacy initiative.

INVOICE Turner Evaluation Services

1209 Peach St. Atlanta, Ga. 30345
DATE: MAY 2, 2012

BILL TO

Big Creek Elementary 2200 South Parkway Atlanta, Ga. 30345

ORDER DATE

ORDER NUMBER

JOB

3/28/2012
ITEM # DESCRIPTION

RAZ Kids Evaluation

AMOUNT

Personnel

Debbie Turner 10 days x $500

5000.00

Travel

5 trips to evaluation location/Big Creek : 23 miles one way @ .55 per mile / 46 miles round trip x 5 = 230 miles x .55 126.50

Per Diem

$25 per day x 5 days

125.00

Office Supplies

Printing supplies, copies, report binding

100.00

Total >>>>>>

$5351.50

Thank you for your business!

Appendices

(Appendix 1)

Sample Reading Level Change Report

(Appendix 2)

Teacher Attitude Survey

(Appendix 3)

Directions: Please read the questions below. Please tell how you feel about each question by circling YES, No, or Maybe. When you have answered all of the questions, return the completed sheet to your teacher. I have fun reading on RAZ Kids YES NO MAYBE

I feel that I am a good reader when using RAZ. I like the books on RAZ I would like to read on RAZ more often. RAZ is a good reading program for kids. YES NO MAYBE YES NO MAYBE

YES

NO

MAYBE

YES

NO

MAYBE