Evaluation Plan

I. BACKGROUND

The Saturn Learning Technology Initiative(SLTI) is a program that works to provide 1:1

technology devices for students in order to promote digital literacy in the classroom. This

program also provides professional development opportunities and support during the

implementation period. The five goals of the Saturn Learning Technology Initiative are to

provide equity, integrate with Saturn’s Learning Results, sustainability, promoting teacher

professional development, and economic development. The SLTI has provided over 75,000

tablets and devices to students in grades 7-12. This program is needed to provide devices

and professional development to teachers to promote digital literacy and career readiness

skills.

II. EVALUATION PURPOSE

The purpose of the evaluation is to find areas of effectiveness and areas that need

improvement. Teacher and student surveys should be conducted to find out how the devices

are being utilized and how the professional development is being carried through into the

classroom. This information can help validate the programs’ effectiveness and gather

information on weaknesses of the program to promote improvements.

III. USE OF EVALUATION FINDINGS

A variety of stakeholders may be interested in your evaluation findings. The primary

stakeholders are those responsible for managing and implementing the program. These

1
 stakeholders can use findings as a way to find areas of improvement and areas that are

working well and need to be sustained. Other stakeholders, such as the people you are

serving, funders of the program, and others in the community, will also have an interest in

your evaluation findings.

Audience Evaluation findings of Utilization of evaluation findings

Teachers, support staff
interest
- Program effectiveness on ways
technology is being implemented.
- Program effectiveness on student
performance
-Improvement to current lessons by hearing other ways to
implement technology effectively and see what is
currently working in regards to student performance
results.

Administrators, School IT - Program effectiveness on ways - Modify and continue professional development
Specialists technology is being implemented. including examples of what is effective. Including
- Program effectiveness on student additional professional development for areas of
performance. weaknesses.
- Program effectiveness on
professional development.
Funders Overall data that shows Determination of future funding
effectiveness of program in regards
to program’s goals.

Evaluation Questions and Anticipated Data Sources

The first evaluation question is, “What are the impacts of devices on instruction?” This

question is important as if the devices are not being properly and effectively utilized, then the

whole Saturn Learning Technology Institute’s 1:1 program is rendered useless. This question

also can help gather information on whether the professional development was implanted

effectively and what aspects of professional development need to be addressed. To answer this

question, data can be gathered through checklists and surveys of teachers and students.

The second evaluation question is, “In what ways have the devices had an impact on

student achievement?” This question helps gather data related to the desired outcome of

increased student achievement. The use of end of course state testing scores as well as student

growth shown through these tests can help answer this evaluation question. Classroom

2
observations can help answer this question to see if the students are showing achievement in the

classroom while using the devices as well.

The third evaluation question is, “To what extent is the program helping to develop

digital literacy?” Developing digital literacy is an outcome of the program and is a crucial skill

for 21st century learners. Schools can develop tests for digital literacy skills, or students could be

asked questions about their technology usage. This feedback can be useful in making

adjustments to the implementation of the devices in order or maximize digital literacy skills.

Student Classroom Participant Surveys

Projected/Proposed Data Achievement Observations Interviews (Student/
Source Data Teacher/Other)

Evaluation Question
What are the impacts of devices on
instruction? X X X
In what ways have the devices had an
impact on student achievement? X X

To what extent is the program helping

to develop digital literacy? X X

IV. DATA COLLECTION

Data will be collected using surveys, interviews, classroom observations, and accessing student

achievement data. For gathering evidence on the impacts of devices on instruction, surveys will

utilize a pre and post test format where feelings regarding technology use will be collected before

professional development and afterwards from teachers. Surveys will also be collected from students

and administrators on the types of instruction that is being used. Classroom observations should also

occur twice a year to gather information on the how devices are being used during instruction. Finally

interviews can help gather additional narrative on how devices are being used to gather information

that is not seen during observations and surveys.

3
 Data from standardized tests will be used to see gains in academic content areas and overall

student achievement. This data would be collected from the state wide tests in the Spring. Focus on

the growth from the previous year would be important. Through interviews, teachers, students, and

administrators can share information about the change in student achievement in their classes.

To see the impact on digital learning, a pre and post test should be given to students to measure

the knowledge of digital literacy skills. This test should be given 3 times a year. Once, to gain a

baseline data, then in the middle of the year to give feedback to teachers, then at the end of the year

for all stakeholders to see gains or room for improvements. These will serve as the survey data.

Interviews will also be collected to gather opinions on how the devices are being utilized.

Data Collection Plan

Indicator Data Sources/Tools Collection

Increased instruction Classroom Observations & From Who When How

effectiveness. Anonymous surveys
Administrator Throughout year Observational
Checklist

Student Achievement Growth State Assessments State End of school Online test
Department of year
Education
Improved Digital Literacy Pre/Post Test Teacher/ School 3 times a year Online test
ITS

V. DATA ANALYSIS

The use of qualitative and quantitative data will be used to evaluate this program. Quantitative

data such as the percentage of teachers seen utilizing technology during observations, and the increase

of achievement test scores. The use of box plots and percentages would help display the data easily.

Specifically, to show findings of the effects on student achievement, two different perspectives will

be important. The level of achievement, or score, on the end of year assessments will be looked at

compared to the school average from the previous year. In addition, the growth percentile compared

to other middle schools who are not using the program will be utilized to gain further insight in how

4
the program effects a growth in student achievement data.

To analyze quantitative data collected from the digital learning pre and post tests, comparing the

pre and post test score averages should tell if there has been sufficient change. Showing this

information in a box plot can help show the percentiles of data collected and help indicate a shift in

the data.

Much of the evaluation will involve more descriptive statistics. From classroom observation

checklists, evaluators can compile a list of best practices that were seen which should be shared with

teachers and all stakeholders. A breakdown report of these checklist can include the percentage of

times that teachers were utilizing the technology and the percentage of times that students were

utilizing the technology.

Evaluators will also look at survey data to find common strengths and weaknesses of program.

The narrative information gained during surveys and interviews help the evaluators see the full

picture of the program. Information that is repeated consistently should be reported as a narrative.