Online Assessments

Evaluation Plan

Prepared by:
Susan Hussey
ECI 502: NC State University

March 2015

1. Introduction
Chatham County Schools has a 1:1 program at each of our high schools since 2009-2010.
Students use these devices on a daily basis in their classes, however when it comes to end of the
course assessments, students are given a paper and pencil test. The district wishes to
standardize testing at all high schools for equitable comparison of results, reduce burdens on
staff, increase security of test content, and facilitate improved efficiency of data collection. As of
the spring of 2014-2015, all state mandated end-of-course and NC Final Exams will be available
for online testing. Online assessments expedite the scoring process for several assessments that
contain constructed response questions that currently requiring shipping student answer sheets
to an outside vendor for scoring. Online assessments also reduce opportunities for human error
involved with counting and distributing testing materials and also allow opportunities for
combining testing settings to reduce strain on schools related to locating spaces and staff to
administer assessments. To view a summary and more information on implementing online
assessments in North Carolina, visit this article by the State Educational Technology Directors
Association (SETDA).

2. Overview of Evaluation
This project will evaluate the pros and cons of online testing and determine if the LEA will
mandate that all assessments be given online to align with the LEAs Technology Plan. The LEAs
goal is to create a strong instructional technology program to facilitate highly engaging 21 st
Century classrooms.
There are 3 high schools in the district that are a part of the 1:1 program. With pre-established
access to laptops, the district wants to move forward with online state assessments. As of the
spring of 2014-2015, all current North Carolina end-of-course and NC Final Exams will be
available in an online format. For the past few years, students in grades 5 and grades 8 have
been taking the online science end-of-grade test with technological success as have chosen
subjects at two of the three high schools.
The district will work with school administration to utilize technology facilitators and testing
coordinators to gather data regarding classroom use of laptops for classroom assessments.
District administration will analyze testing results of paper/pencil assessments versus online
assessments to take note of any noticeable differences in results and will contact other local
school districts to inquire about their online assessments. Surveys will be administered to school
administration, teachers, and students to determine their opinions regarding online testing. After
data is gathered and analyzed, course enrollments will be reviewed to determine which
subject(s) will be available for testing at all locations. Staff development will be provided to
teachers to aide in the creation of online classroom assessments and programs to prepare
students for online state assessments.
The goal of the evaluation will determine the appropriateness of online testing. The desired
outcome will be that there is no discernible difference in the type of assessment and student
results and that the district will start requiring some (if not all) assessments online at all high
schools.

Logic Model

3. Evaluation Design
Evaluation Questions

To what extent was the staff development useful and appropriate?

To what extent did the increased bandwidth occur within all schools by the date necessary
for online testing?
To what extent do online assessments decrease the amount of time needed to gather
testing data?
What are attitudes towards online assessments among the students and staff?
To what extent did online testing affect student proficiency?
To what extent does the staff and community support the use of online assessments? Are
the opinions in support or opposition to the initiative?

Stakeholder Needs
District administration will use the evaluation findings to make district-wide decisions
regarding online testing (potentially by subject area) for all county high schools.
District administration will need to analyze the data to determine if 1) the schools have
the needed bandwidth to complete testing and 2) identify and hopefully alleviate any
concerns from staff or students about transitioning to online assessments.
Findings from the evaluation will be used to address any concerns or known issues prior to
any additional online assessments (beyond what is required by NCDPI) to provide a
positive and successful test administration.
Opinions of staff and students are valuable information but more credible information
would be facts, figures, and statistics that exist which can provide more concrete evidence
to support or not support the initiative.

Evaluation Approach
The CIPP model will be used in the program evaluation. Input from stakeholders will be
evaluated as plans are put into place for the immediate future testing windows. The
process and outcomes will be evaluated to determine if online testing was successful in
regards to technology in the schools and if there was any discernible difference in student
test results. The success or challenges of the implementation will be reviewed and any
changes that need to be made can be corrected prior to the next testing window. The high
schools will serve as the initial phase of implementation that will eventually filter down to
the middle and elementary-aged students as NCDPI makes more state assessments
available in the online format for those grade levels.

4. Data Collection
Data Collection Methods
Data will be collected through staff and student surveys. School testing and technology
facilitators will work with classroom teachers to collect data in regards to the use of the
computers for classroom assessments. Central office staff will also be collecting technology and
testing data from nearby districts.

Table 1: Evaluation Questions and Associated Data Collection Methods

Evaluation Question
1. To what degree was the staff development
useful & appropriate?
2. Did the increased bandwidth occur within
all schools by the date necessary for online
testing?
3. To what extent do online assessments
decrease the amount of time needed to
gather testing data?
4. What are attitudes towards online
assessments among the students and
staff?
5. To what extent did online testing affect
student proficiency?

6. To what extent does the staff and

community support the use of online
assessments? Are the opinions in support
or opposition to the initiative?

Data Collection Method

Staff surveys upon completion of staff
development then follow survey after 3 months
to determine use of skills learned in the
classroom
Reports will be provided by the technology
department at the central office to monitor
peaks or lags in network usage versus capacity
Testing department will provide results to
schools immediately upon receipt from NCDPI
and school staff will determine if turn around
time is adequate or faster
Student and staff surveys will be provided preand post-assessment to determine any changes
in attitude after actual administration
Data will be disaggregated and analyzed for the
online tested subjects in comparison to results
for the same subjects the previous testing cycle
which was administered using paper
assessments
Staff and community survey will be given to
determine support of the online transition and
to help identify any issues or concerns relative
to this transition.

5. Data Analysis and Interpretation

Indicators and Standards
Table 2. Indicators and Success
Evaluation Question

Criteria or Indicator

1. To what degree was the

staff development useful &
appropriate?

Survey

2. Did the increased bandwidth

occur within all schools by the
date necessary for online
testing?

Technology usage report

3. To what extent do online

assessments decrease the

Time report of shipment

of assessments and return
of results

Standards
(What Constitutes Success?)

Implementation of strategies
Increased use of online
assessment tools for classroom
tests
Decreased speed of delivery did
not occur
No technology failure when trying
to access assessments
Grades and report cards are
available to students without any

amount of time needed to

gather testing data?

Time required by staff

from schools to deliver,
count, package materials
to be returned to central
office for scoring

delay due to waiting for

assessment scores

4. What are attitudes towards

online assessments among the
students and staff?

Student and staff survey

Same questions are asked and for

those with negative responses,
90% will now report positively

5. To what extent did online

testing affect student
proficiency?

Data comparison using

public NCDPI data reports

No discernible difference in paper

versus online proficiency in the
same subject area
Minimal technical issues and fewer
irregularities involving mismatched
testing materials

6. To what extent does the

staff and community
support the use of online
assessments? Are the
opinions in support or
opposition to the initiative?

Staff and community

survey

At least 75% of survey participants

will respond in favor of the
initiative
Areas of concern will be addressed
and resolutions communicated to
the staff and community

Analysis

Quantitative analysis of results and qualitative analysis of testing experience