0 évaluation0% ont trouvé ce document utile (0 vote)

475 vues28 pagesFRIT 8435 Evaluation Project. My group and I evaluated the iPass Math program that is used at Blakeney Elementary.

Feb 17, 2013

© Attribution Non-Commercial (BY-NC)

DOCX, PDF, TXT ou lisez en ligne sur Scribd

FRIT 8435 Evaluation Project. My group and I evaluated the iPass Math program that is used at Blakeney Elementary.

Attribution Non-Commercial (BY-NC)

0 évaluation0% ont trouvé ce document utile (0 vote)

475 vues28 pagesFRIT 8435 Evaluation Project. My group and I evaluated the iPass Math program that is used at Blakeney Elementary.

Attribution Non-Commercial (BY-NC)

Vous êtes sur la page 1sur 28

Table of Contents

I. Executive Summary -------------------------------------------------------------- page 3 II. Introduction ------------------------------------------------------------------------- page Purpose of the evaluation ---------------------------------------------------page Audiences for the evaluation----------------------------------------------- page Limitations of the evaluation-------------------------------------------------page Overview of report content --------------------------------------------------page 6 7 7 7 8

III. Focus of the Evaluation ---------------------------------------------------------- page 8 Description of the evaluation object -------------------------------------- page 8 Evaluative questions used to focus the study ------------------------- page 10 Information needed to complete the evaluation ----------------------- page 10 IV. Brief Overview of Evaluation Plan and Procedures ---------------------- page 11 V. Presentation of Evaluation Results ------------------------------------------ page 11 Summary of evaluation findings ------------------------------------------- page 11 Interpretation of evaluation findings -------------------------------------- page 15 VI. Conclusions and Recommendations ----------------------------------------- page Criteria and standards used to judge evaluation object --------------page Judgments about evaluation object ---------------------------------------page Recommendations ------------------------------------------------------------page 16 16 18 19

I. EXECUTIVE SUMMARY

Purpose The purpose of this evaluation is to determine the impact of the Blakeney Elementary iPass Program on participating students and their CRCT scores. The evaluation should help determine if this computer based, web enabled program is making a significant impact and if the school should continue using it. Currently the iPass program is used as a remedial tool for those students who have exhibited difficulties in learning mathematics in the past and students who are not working on their grade level. It is available for all students in grades 3 through 5, but quite often the students working on iPass are the ones who scored below proficient on state tests. Depending on the individual teacher, students are exposed to the program about 5 days a week for forty-five minutes a day. Students work in a designated area of the classroom or math computer lab. The main focus of this evaluation is to find out if the iPass program is effective in increasing CRCT scores for the students of Blakeney Elementary School.

Information Collection During the evaluation process, a variety of quantitative data was collected. Within the iPass program there is the ability to track students and collect data. The data collected to assess for this evaluation was based on the 2011-2012 school year. Key characteristics of data criteria would be: Diagnostic tests that identify any gaps of knowledge and understanding for each individual child Grade level score in math for the year entering program

Progress of students over the evaluation time period CRCT scores of individual students before using iPass CRCT scores after using iPass Interviews were also conducted to collect information to judge the effectiveness of the iPass program. Teachers and administrators at Blakeney Elementary School were interviewed. During these interviews teachers were asked to list the positive and negative aspects of iPass. This was done to help determine the human aspect or attitude across many interviewees. Teachers were also asked to describe the effectiveness of the program for individual student success in math: Is iPass a helpful tool in bringing up math scores in your classroom? Students were also interviewed to find out what they thought about iPass and how the program could be improved.

Findings After pulling data and viewing student performance we found that: 50% of students entering the program were not on grade level in math, therefore some type of remediation was required. 83% of students who used iPass increased more than two grade levels in math over the period studied. When comparing CRCT scores of students before and after using iPASS, data shows a significant increase in performance. Pretest or diagnoses of individual students shows inconsistent gaps of knowledge and understanding which means that an individual plan of remediation was needed.

The interview data collected shows that teachers value iPass as a key remediation tool for those students who are behind in grade level and performance. The list of pros outweighs the list of cons significantly. Some of the positives points made were: The iPass is very user friendly and allows students to excel where traditional methods have failed. The interactive technology that the iPass program offers will not allow students to cheat to move beyond their ability. Teachers are able to monitor students' progress as they move through the program. Students are allowed to start at their own level and work at their own pace. This allows students to take more ownership of their learning. The negative points made by those interviewed were helpful in determining recommendations made later in the report. Some of these points were: When students miss school they miss remediation. Time lost cannot be made up. There is a limited amount of technology or computers available for student use, therefore limiting the effectiveness of the program. There needs to be some type of reward system The quantitative and qualitative data found helped in determining the effectiveness of the program and provided detailed information in answering the evaluation questions.

Judgment Increasing CRCT scores and insuring that students are learning is the bottom line. Those students that get behind or do not master the content will need some form of remediation. The evaluation shows that the use of iPass is working and does provide a

5

scaffolding approach that is designed individually for each child. Evidence shows that students are increasing in grade level math and that CRCT scores are going up.

Recommendations To help students achieve higher scores on standardized tests and bring up scores in the classroom, the following recommendations are given. 1. For school systems to provide more computer equipment in the regular education classroom to accommodate more students in the regular math class to be able to use the iPASS Program. 2. For the school system to offer the iPASS Program to younger students when they first start exhibiting signs of struggling in Math. Using iPass as early intervention before a student falls behind could prevent the student from failure on quarterly benchmarks and state standardized tests. 3. For school systems to schedule longer periods of time in the computer classroom for students to participate in the iPass program. 4. For schools to create and implement some type of reward system for students.

II. INTRODUCTION

The iPass Math Program is an internet-delivered software that was adopted by the Burke County School System in 2011. Since the programs adoption, it has been used as an intervention tool for students who are struggling in math or who are not on grade level in the area of math. Participants in the program receive a web-based

instruction that is aligned with the State and National standards. The curriculum is designed for students in grades 3-8, as well as high school students in support classes.

Purpose of the evaluation The purpose of the evaluation is to look at the relationship between a students success in iPass and their CRCT Mathematics score. The evaluation is intended to find out if the use of iPass is helping the schools Math Program and whether or not it is truly beneficial to students. Are students that use iPass also increasing their grade level performance? Finding the answer to these questions will help determine the programs effectiveness and hopefully provide recommendations that would help improve its implementation. The best method used to find these answers was to pull existing data collected in years past and to interview the stakeholders to determine a clear perspective viewed by teachers, administrators, students and parents.

Audiences for the evaluation The primary audience would be administrators, teachers, and parents. Because they are directly responsible for student success, teachers can view the data collected and determine where improvements may be needed. Administrators could also determine if the iPass program is working and if it should be continued or changed.

Limitations of the evaluation Because of time constraints a lengthy evaluation with a large control group was not possible. Because of the nature of the school year, the evaluators examined the 2010-11 fifth grade students iPass reports and CRCT Math scores. A full years

information could be compared from the beginning to the end of the school year. The sample group of students were selected at random, but includes E.I.P (Early Intervention Program) students and regular education students. Current students enrolled in Blakeney Elementary School were not evaluated. A true snapshot of information has to be collected from beginning to end of a school year.

Overview of the report content The remainder of the report is broken down into sections that will go into more detail. Section III will pinpoint the evaluation questions and give information needed to complete the evaluation. The body of the report will be discussed in section V where the findings are presented. You will find in section VI, the judging criteria that are presented to help draw a conclusion. Ultimately, recommendations that surfaced during the evaluation process will be presented. All detailed information, graphs or other support material will be located in the appendices.

Description of the Evaluation Object iPass stands for Individual Prescription for Achieving State Standards, which is an Internet-delivered software from iLearn, that is administered in grades 3rd, 4th, and 5th grade Math at Blakeney Elementary School (BES). The iPass program is a fully automated, web-based math curriculum and instruction that covers all the core concepts necessary for success in elementary grades, middle grades as well as Algebra or Integrated Math in High School. The system provides individualized instruction that is

used to meet the identified needs of each student. The two most important features of iPass are: 1 It is prescriptive at a very detailed level - that is, instruction is matched to the specific concept and skill deficits of individual students. 2 It is mastery-based, which means students achieve mastery of each standard as a prerequisite for progress through the program. It is a policy of the Burke County Board of Education that each school in the system will have a math program in place for remedial purposes for students who have shown difficulty in mathematics or students who did not meet standards on the CRCT. The program was adopted at Blakeney Elementary School at the beginning of the 20112012 school year. The program is available to all students at BES, but is mainly used to meet the needs of students who struggle or are not currently working on grade level in the area of Mathematics. Signs of a student struggling are: Low CRCT scores and/or low grades in math class. BES students are provided a scheduled time to report to the iPass computer lab on a daily basis. The classes that are required to participate report to the lab with their Math teacher. The computer lab contains twenty-five desktop computers that have access to the licensed program. The computer lab is monitored by a lab teacher and the classroom math teacher. Students that are required to do iPass are students that are currently part of the Early Intervention Program (E.I.P.), but the program is not limited to only this group of students. All E.I.P. classes are required by administration to report to the computer lab and to work with the iPass program on a daily basis. A schedule of the lab times is created by the Instructional Coordinator at

the beginning of the year. When students report to the lab they have forty-five minutes to work in the program. Evaluation Questions The following questions were used to focus the evaluation. Is there a relationship between the amount of success in iPass and improved CRCT scores? What impact has the iPASS program had on student achievement?

Information Needed to Complete the Evaluation To conduct this evaluation, several pieces of information were collected and analyzed to help provide a good snapshot of how to answer the evaluation questions. The quantitative and qualitative data collected were: iPass Unit Progress Report iPass Placement Test Grade and Unit Starting Point CRCT Scores in 4th grade and 5th grade Interview questions from teachers that use iPass Interview questions from students that used iPass These reports provided the information needed to evaluate the iPass program. Additional information was collected within the teacher and administrator interviews which gave them an opportunity to express their thoughts of the program.

10

The evaluation site was Blakeney Elementary School in Burke County. Data was collected from a random sample of students using the iPASS program. The evaluation plan is to examine any change in students math performance over a period of time. To determine this we obtained and reviewed the end of the year CRCT scores and iPass results throughout the 2011-2012 school year. Grade Level Performance Reports were also used to compare the success of the students achievement in mathematical performance. Other methods for collecting information were benchmarks scores, interviews with teachers and administrators and student surveys. Because the iPass program was adopted in the summer of 2011 there is only one years worth of data through which to look. Data was generated through the iPass program that described the progress of students from their beginning grade level to ending grade level (Appendix A). Other helpful data was the placement test grade and unit starting point report (Appendix B). This report identifies where students begin their remediation. To determine student gains or losses on the CRCT a table was created (Appendix C) to show two years worth of scores. Five Blakeney Elementary teachers were interviewed (Appendix D).

All students involved in the evaluation were surveyed (Appendix E). Throughout this process a number of quantitative and qualitative data was accessed in the efforts to answer the evaluation questions. The evaluation plan was administered by first collecting the data from iPass reports and CRCT scores for the students involved in the evaluation. Students involved

11

in the iPass Program were interviewed along with teachers that are responsible for making sure students are using the program.

Summary of evaluation findings Several conclusions were drawn after data was collected through interviews, surveys, observation, and data from CRCT Scores and iPASS reports. Eighty-three percent of a random sample of students receiving instruction from the iPASS program increased significantly on the CRCT scores. In a class of 12, 95% of the students increased over 15 points on the CRCT. Below is a chart of the before and after CRCT scores and the gains the student made after receiving instruction in the iPASS program. Table 1: Student 2010-2011 Math CRCT Score (4th grade) 2011-2012 Math CRCT Score (5th grade) 833 797 811 873 819 819 776 811 873 833 + 37 -4 + 20 + 77 + 28 + 28 - 24 + 44 + 35 + 18

Student A Student B Student C Student D Student E Student F Student G Student H Student I Student J

796 801 791 796 791 791 800 767 838 815

12

Student K Student L

807 835

821 850

+ 14 + 15

These scores were also compared to the grade level performance data collected. In this student group thirty-seven percent of 5th grade students started in the iPASS program at a first grade level. Another thirty-seven percent of the 5th graders started out below grade level. Only twenty-five percent of the random sample started the program on grade level.

Students were interviewed who participated in the iPASS program. Ninety percent of students interviewed stated that the iPASS program had increased their understanding in math and increased their grades in math. This was asked in yes/no form questions and was followed by asking students to explain their answer in detail. The students who answered no stated that they had a strong dislike for math and were not interested in it. The students who answered yes commented on how the iPASS program walked them through difficult problems and how working in an individualized setting help them better understand. Students also commented that in the iPASS environment they were not afraid to ask questions when they didnt understand a problem. Eighty percent of the students said the iPASS program helped them understand math and increased their math grades. For example, one student commented that she was always scared of math because of the lack of confidence she had in her ability to perform in the math area. After completing the iPASS program, the student says that she has gained confidence and finds that math is not so scary any more. She reported that her grades have come up significantly.

13

Teachers were asked if they could see the iPASS program positively affecting the students. One hundred percent of the teachers said yes they could see the positive impact on students. After interviewing several teachers, it became evident that teachers could see a difference the program was making in the performance of their students. Some teachers commented about the increased confidence the students portrayed as they tackled math problems.

Observations Students were also asked to rate their experience in the iPASS program. This was done so that other aspects of the program would be taken into consideration. Eighty percent of the students rated the iPASS program excellent, ten percent rated the iPASS program as good, and ten percent rated the iPASS program as below average. This same question was asked to the teachers in the iPASS classroom. One hundred percent rated the iPASS program as excellent. Students were given the opportunity to discuss on the survey and in the interviews what they thought could improve the iPASS program. Some of the responses were to make the program more fun by having games to play as an incentive when they pass a unit or chapter. The students stated that would be cool to make the program more like a game based program with incentives. Teachers were also asked how the program could be improved. Some of the responses were to provide some incentives for the students to encourage them to want to work, especially the students who dont like math to begin with. The teachers stated that the program was great, but sometimes the students need some rewards or

14

incentives to motivate them to keep working hard. These improvements will be addressed in the recommendations section of the report.

Interpretation of evaluation findings The evaluation indicates that the program is in fact helping students learn and excel in math. Because students are provided with hands-on learning experiences, they are able to comprehend the math skills that they have missed in a regular education class. The iPASS program provides students with an individualized instructional setting to learn and catch back up to grade level performance. The data from the CRCT scores show a significant increase in the scores after participating in the iPASS program. In the evaluation procedure the question was posed, Did students who reached grade level standards in iPASS achieve a minimum score of 800 on the CRCT? According to the data collected from the CRCT Test Scores from the school year 2011-2012, eighty-three percent met the standards on the CRCT. This eighty-three percent had an average increase of 31.6 points on the CRCT. In relation to the evaluation question, Is there a relationship between the amount of success in iPass and improved CRCT scores? this data proves a positive relationship between the iPASS program and improved scores. In question 2, Did participating students exceed standards by achieving a score of 850 or higher on the CRCT? the data collected shows that 41% of students exceeded the standards on CRCT after work in the iPASS program. Data collected shows that after working in the iPASS program for a school year, students can catch up to grade level performance as well as exceed the standards in grade level math. According to the data and teacher/student surveys, the iPASS program is an

15

individualized program tailored to the needs of the students. This program is offered in a different classroom setting, making it more productive for a student to focus on one on one instruction. According to student surveys, eighty-three percent contribute their new understanding, confidence, and higher achievement to the instruction offered in the iPASS program. The teachers in the iPASS classroom also see the connection between the iPASS program and higher levels of achievement for their students. This is beneficial to the students because they can get the one on one instruction they need to achieve their highest potential in the area of math. Not only does the program teach the students in the areas of math in which they are behind, it also builds confidence in the students ability to solve mathematical problems.

Criteria and Standards Used to Judge Evaluation Object During the early stages of planning for this evaluation, we established two evaluation questions along with simple criteria and standards. The table below shows this information. Evaluation Question 1. Is there a relationship between the amount of success in iPass and improved CRCT Criteria Standard

Score of 800 on CRCT and student advancement to or through grade level performance.

16

scores? 2. What impact has the iPASS program had on student achievement? CRCT Score, mastery of Georgia Performance Standards, positive progression throughout the school year 800 on CRCT, promotion to the next grade, passing scores on benchmark test

Question one asks, Is there a relationship between the amount of success in iPass and improved CRCT scores? To apply the criteria designed to help answer this question, reports had to be created or collected. One of these tables would be the level achieved in iPass (appendix A). This data explains on what grade level the student entered the iPass program and how they progressed across the given time period. Also in the Placement Test Grade and Unit Starting Point table (Appendix B), it pinpoints the starting point and how many times a student took the placement test. These two pieces of data helped identify the achievement levels of the students. Appendix C was used to identify the CRCT scores for the past two years. In the area of math in the 5th grade, students are required to make an 800 or above on the CRCT. This score shows that the student has mastered grade level objectives and is on target for that grade level. This was part of the standard used to address question one. Of the 12 students, their compared CRCT mean score from 2011 to 2012 increased by 24 points. Although some students digressed, 83% of the students increased their score while only 17% did not. Six of the twelve increased their score above 800 with a total of 10 students with a score above 800. The data shows a

17

correlation between progress through the iPass program and the success on the CRCT. IPass has helped students increase their CRCT scores. Question 2 asked, What impact has the iPASS program had on student achievement? Achievement is a word that can describe many things. In this case achievement is considered passing of the CRCT, showing positive scores on benchmark tests, and mastery of the Georgia Performance Standards all resulting in promotion to the next grade level. While some of the information collected came from many of the same sources as question one, we had to rely on teacher interviews to gain insight on the benchmark scores. According to those interviewed, scores on benchmark tests varied. Many times students did not do well on these tests. There was no solid or quantitative data available for benchmarks, so this criteria did not help as we originally had hoped. Promotion data was available: Of the 12 students, 10 were promoted to the next grade level. According to the data collected, there is a positive correlation between iPass and student achievement within the classroom.

Judgements about Evaluation Object - Strengths and Weaknesses Based on the information evaluated by the iPass program, this evaluation study concludes: Strengths iPass is an effective tool to remediate students in the area of math. Program is organized and well managed by personnel Designated labs are staffed and equipped to implement the remediation tool Program is available to all students in the school, not just those who may be behind.

18

There is a designated time period and schedule followed by all teachers. iPass is personally prescribed for each student. Mastery of each standard must be achieved before moving to the next level. Teachers can generate ongoing formative assessment data and feedback on students performance. There is a vertical alignment of iPass used through all grade levels. Weaknesses This evaluation looked at a small sample group. Data from larger groups would give a better perspective of the true effectiveness of iPass. Data was collected only from the 5th graders during the 2011-2012 school year. It would be beneficial to evaluate data from a five year span. Limited space or labs for the implementation Designated lab times are not flexible. There are no built-in incentives for students from within the program. There are limited external incentives for students

Recommendations To help students achieve higher scores on standardized tests and bring up scores in the classroom, the following recommendations are given. 1. For school systems to provide more computer equipment in the regular education classroom to accommodate more students in the regular math class to be able to use the iPASS Program. If more updated computers were placed in the classroom setting, students would not have to have a scheduled time to go to the lab, but instead could access the program in the classroom. With so many classes being

19

scheduled for the computer labs, it is difficult for teachers to schedule individual students a time to go to the lab for the purpose of using iPass. If the computers in the classroom were more up-to-date, then the teacher could come up with his/her own schedule to use within the classroom. 2. For the school system to offer the iPASS Program to younger students when they first start exhibiting signs of struggling in math. Using iPass as early intervention before a student falls behind could prevent the student from failing quarterly benchmarks and state standardized tests. 3. For school systems to schedule longer periods of time in the computer classroom for students to participate in the iPass program. In order for the iPass

Program to be most effective, students need to participate in the program on a daily basis. When the computer lab schedule is made, many times it does not take into account transition times between classes and this can shorten the time drastically and cause students to exit the program prematurely. 4. For schools to create and implement some type of reward system for students. To help motivate students to work diligently in the program some type of incentive program could be established for those students who are expected to use the program on a daily basis. When students were interviewed they mentioned that they liked computer programs that let them play games, but iPass does not allow them to play games. Since this is something the creators of iPass would have to correct an idea of how teachers and computer lab teachers could motivate students is by creating a daily incentive for those students who make progress in the program.

20

VII. APPENDICES

1. iPass Unit Progress Report 2. Placement Test Grade and Unit Starting Point 3. CRCT Data of Random Sample Class- 4th Grade and 5th Grade Scores 4. Teacher Interview Questions 5. Student Interview Questions 6. Work Sheet for Summarizing the Evaluation Plan pg.22 pg. 23 pg. 24 pg. 25 pg. 26 pg. 27

21

The table below shows a class of 5th grade E.I.P. students who were involved in the iPass program for the 2011-2012 school year. The bar graph shows what grade level they began in iPass and what grade was mastered by the end of the school year. Most of the students started out the program at the 1st or 2nd grade level based on the results of the iPass placement test. A few of the students were not enrolled at BES the entire year. The students who entered and withdrew are Wallace M. and Gary S.

22

When students first encounter the Internet based iPASS, they use a pre-assessment tool to determine the gaps in their education. After the pre-assessment, the program places them at the appropriate skill level or more specifically at the unit most appropriate to their skill level. Below you will find the scores of the placement test and where the program identified the gaps in each students math understanding. The program automatically assesses the gaps based on answers and understandings of students. The iPass program then places them in the unit/grade level in which he or she will start the program. Unit A begins with first grade standards and progresses from there. The majority of the students in this class started at the first grade level.

23

Appendix C- CRCT Scores in 4th grade and 5th grade The table below gives the Mathematics CRCT data for students at the end of 4th grade and 5th grade. The students did not participate in the iPass program while in 4th grade, but did in 5th grade. The last column shows the increase or decrease in scores from the fourth to fifth grade. Student 2010-2011 Math CRCT Score (4th grade) 796 801 791 796 791 791 800 767 838 815 807 835 2011-2012 Math CRCT Score (5th grade) 833 797 811 873 819 819 776 811 873 833 821 850

Charles A. Diana B. Amya C. Monique C. Tyquavious C. Kelondra H. Brandon J. Christian L. Eric L. JaNiyah R. Pablo R. Seven W.

+ 37 -4 + 20 + 77 + 28 + 28 - 24 + 44 + 35 + 18 + 14 + 15

24

Appendix D: Teacher Interview Questions Interview Questions iPass Math Intervention Program

Do you have any examples you could share of how the iPass program has been effective with students?

What would you say to another school district that is looking into purchasing the iPass program for their county/school? How would you rate the iPASS Program on the following scale? Excellent Good Average Below Average Bad

8.

9.

How has the iPass program helped on benchmark testing throughout the school year.

10.

Please provide any additional information about the iPass program that may not have been asked in the above questions.

25

Interview Questions iPass Math Intervention Program

1. Do you think you did better in math this year than last year? Yes No Explain your answer:_________________________________

2. Do you think iPass helped you do better in math? Yes No Explain your answer:___________________________________

3. Rate your experience with the iPASS Program. Excellent Good Average Below Average

Bad

26

Evaluation Questions Is there a relationship between the amount of success in iPASS and improved CRCT scores? Information Required Design Information Source Method for Collecting Info Obtain & review end of year CRCT & iPASS results Gathering existing data Survey & possible interview Gathering existing data Survey & possible interviews

Time Series

Time Series

Evaluation Questions

Sampling

Is there a relationship between the amount of success in iPASS and improved CRCT scores?

Information Collection Procedures Collect CRCT and iPASS data on 12, 5th grade student from Spring 2011 Condition CRCT Confidential records

Schedule

Analysis Procedures Interview with teacher, Statistical analysis of CRCT scores and iPASS reports Statistical analysis of progress with the data from the iPASS reports and grade level performance

End of last year grade level. Beginning grade level Current grade level CRCT scores 2011 grade level Current Grade level CRCT Scores

Students at Blakeney Elementary School who began iPASS below grade level

Obtain & review end of year CRCT and iPASS results, collect & review benchmarks in Math. Report Card/Progress Reports

27

Evaluation Questions Is there a relationship between the amount of success in iPASS & improved CRCT scores? What impact has the iPASS program had on student achievement?

Interpretation Procedures Did students that reached grade level standards in iPASS achieve a minimum score of 800 on the CRCT?

Audience

Content

Format

Schedule

Teachers, 2011 CRCT scores administrators, from Blakeney parents, Elementary School students iPASS student reports

Did participating students exceed standards by achieving a score of 850 or higher on the CRCT? Did participating students rate iPASS with high scores as a contributing factor in higher achievement?

Teachers, Quarterly administrators, benchmark test parents, students Advancement through iPASS program

Report to school administrators to discuss results, Interpretation of results, & possible changes. Meet with clients to report results and receive their input. Report to school Administrators to discuss the results and possible changes. Meet with clients to report results and receive their input.

28