Vous êtes sur la page 1sur 28

Evaluation Plan for LiveLesson Training

EDCI 577 Project 3

Amy Keyser
Purdue University

Identification of Evaluation Goals and Scope

Connections Education is a private-sector corporation that operates
online elementary and high schools called Connections Academy in over 25
states. Students learn their academic subjects through learning modules
developed by Connections Education in conjunction with Pearson Learning.
One of the ways that students interact with their teachers is through
LiveLesson, which is a virtual classroom administered through Adobe
Connect (What Is Connections Academy, 2015). Teachers use this medium to
teach synchronous lessons to groups of students or individual students.
They also use the medium to record lessons for students to watch at a later
time. As part of the onboarding process, newly hired teachers get training
on how to use LiveLesson. At Ohio Connections Academy, this training
consists of an hour long seminar during the first two days of teacher
meetings and an online course that they take independently. The online
course has three modules: Basics of LiveLesson, Connections Education
LiveLesson policies, and Advanced LiveLesson. If all three modules are
taken without interruption, the online portion takes about three hours. For a
table of objectives in the modules, see Appendix A. All of the objectives will
be needed to successfully utilize LiveLesson.
Teachers are expected to host LiveLesson sessions at least twice per
week for their entire class of students. LiveLesson sessions start the
second week of school so it is important that teachers learn to use it quickly.
Due to internet safety issues and copyright concerns, LiveLesson is really
the only way that teachers can connect with their students visually, since
YouTube and similar media use are forbidden for teachers to use with their
students at Connections Academies.
The audience for this training is new hires. Each year about 20 new
teachers are hired at Ohio Connections Academy. All new hires are certified
teachers in their content fields. About half the teachers are new to the field.
The rest come from brick and mortar schools. A very small number (1 or 2)
per year come from other virtual schools. Unless they are former students of
a Connections Academy or parents of a student at a Connections Academy,
they will be unfamiliar with LiveLesson. However, some of them may be
familiar with Adobe Connect and Webinar/Webcast type trainings.
The stakeholders are many. The teachers themselves want to be adept
at using the technology before they try to use it in front of others. The
students, especially visual learners, will benefit from the visual nature of
LiveLesson as well as the synchronous interaction with their teachers.
Parents will want to be sure their child has competent teachers; being unable
to run a LiveLesson session might reduce their confidence in their choice of
schooling for their students. There are also principals, superintendents,
board members and sponsors who want to see that technology is being used

appropriately so that student learning is maximized. The school board has

requested this evaluation as one aspect of a broader goal of improving the
onboarding process.

Description of Process
Level 1 Reaction surveys and Level 2 Learning assessments will be given
immediately upon completing the seminar and online course. Level 3
evaluations will take place one month and three months after the initial
course. This will give participants time to implement the things they learned
in the courses. Participants will do a self-evaluation and a supervisor will also
do an evaluation of participants LiveLesson to see if they are utilizing the
tools they learned in the training. In this context, it is difficult to measure the
results of the training. However, results will be measured by looking at
trends in LiveLesson attendance. If attendance trends upward, it will be
inferred that participants are using LiveLesson effectively. If attendance is
trending downward, the training should be evaluated to determine if
teachers need to learn other things to make the LiveLessons more effective
and engagement. See Appendix B for a Gantt Chart of the timing of each
Instruments & Instrument Descriptions
Evaluation will be done at all 4 levels of Kirkpatricks Four Levels
(Kirkpatrick & Kirkpatrick, 2006, p. 21). The purpose of the evaluation will be
to gain information on how to improve future training programs
(Kirkpatrick & Kirkpatrick, 2006, p. 21). According to one new hire, it would
have been better to have the online training before the seminar. The timing
of both will be measured in the reaction sheet to see if this is more than one
persons opinion.
Immediately following the seminar, a smile sheet will be given to
measure reaction, Level 1 of Kirkpatricks model (Kirkpatrick & Kirkpatrick,
2006, p. 21). Participants will be asked to fill it out before leaving the room.
This should ensure a 100% response rate as recommended by Kirkpatrick
(Kirkpatrick & Kirkpatrick, 2006, p. 35). The reaction sheet will be anonymous
and will measure how the participants felt about the training. The reaction
sheet will have questions that use a 3 level evaluation (too soft, just right,
too loud) and 5 point Likert scale as well as a section for comments.
Although it is recommended that reaction sheets evaluate content
(Kirkpatrick & Kirkpatrick, 2006, p.120), questions about the content will be
omitted since this session is an overview. It is not expected that all the
participants questions about LiveLesson will be answered at this time.
See Appendix C for the survey.

A reaction assessment will also be embedded at the end of the

online portion of the training. This survey will also include questions rated on
a Likert scale and a place for comments. Participants will not be able to mark
the training complete until the survey is completed, which will also ensure a
100% response rate.
See Appendix D for the survey questions. The actual survey will be
completed electronically.
It is also important to determine whether the participants learned
anything, which is Level 2 of Kirkpatricks model (Kirkpatrick & Kirkpatrick,
2006, p. 21). Since the participants will have little knowledge of
LiveLesson from a teachers point of view, it will not be necessary to give a
pretest to the participants. Also, since all the teachers need to be able to
use LiveLesson by the end of the first week of school, a control group will
not be practical (Kirkpatrick & Kirkpatrick, 2006, p. 46). Since this is a small
group who will be receiving a lot of new information during that day and the
seminar is really an overview, it would not be advisable to take a test
because they will not have learned enough information. At the end of the
seminar, the presenters will ask the participants questions to ascertain their
learning. This question and answer period will serve a dual purpose of
evaluating learning and clearing up any misconceptions participants may
have (Kirkpatrick & Kirkpatrick, 2006, p. 236).
See Appendix E for the questions that the presenter will ask.
At the end of the online modules, a multiple choice assessment will be
embedded into each module. Since using LiveLesson is a skill, a
performance test might be preferred, but since those would need to be
analyzed by a person instead of a computer, it is not practical to do so at the
beginning of a school year (Kirkpatrick & Kirkpatrick, 2006, p. 49). The
results will be tabulated by the learning management system for analysis.
See Appendix F for a list of sample questions. Due to the proprietary nature
of LiveLesson, more specific questions may not be stated here.
Since behavior takes time to take place (Kirkpatrick & Kirkpatrick,
2006, p. 53), participants will analyze their application of learning about six
weeks after the initial training and then two months after that. In addition,
the lead teachers will watch the LiveLessons of the new teachers and
analyze them for competent usage of the components of LiveLesson. It is
anticipated that these evaluations will not only produce data to evaluate the
training, but will also serve as a further teaching tool/reminder of the
capabilities of LiveLesson (Kirkpatrick & Kirkpatrick, 2006, p. 236). In an

attempt to determine whether the behaviors resulted from the training or

from other avenues, such as asking other teachers or experimentation,
participants will also be asked where they learned to use each component.
See Appendices G and H for the evaluation instruments.
The goal of LiveLesson sessions is to help students learn material
and feel engaged with their teachers, classmates and curriculum. There are
many factors that contribute to engagement, so it will be difficult to analyze
whether the LiveLesson training is having its desired effect. One measure
of this would be attendance in LiveLesson. Attendance is not mandatory
so students will likely only attend if they find the LiveLesson sessions
helpful. If there is a trend of greater or at least stable attendance over time,
this would indicate that teachers are using LiveLesson effectively and,
consequently, that the training was effective. The evaluation team will
collect the attendance numbers for the teachers who participated in the
training. These numbers are automatically recorded
Data Collection
Data will be collected from participants, trainers and lead teachers.
Also, data will be collected from the LMS. Table 1 describes the kinds of
data being collected, by whom and when.

Task Who
Seminar Participants

Start Date

er of



Reaction Sheet* Seminar Trainer


Analysis of Learning* Seminar Trainer


Analysis of reaction Evaluation

sheets* Team
Online Learning Module Participants









Lead Teachers


Analysis of online
Self- Evaluation of
Evaluation of Practices*

Self- Evaluation of Participants


7 7-Dec

Evaluation of Practices* Lead Teachers
Data analysis of behavior Evaluation
evaluations Team
Analysis of Attendance* Evaluation

Final Report Delivered Evaluation
*Data will be collected during these tasks.






Analysis Process
Level I
Reaction Sheet for Seminar. Table 2 gives guidelines for the
interpretation of the data, based upon the average response to each
question, which in turn will be averaged for the category (Kirkpatrick &
Kirkpatrick, 2006, p. 159).
Table 2
3 3.6
4.2 5

The participants did not feel that this part was done well.
There were some good things about this area, but it could be
Participants had a positive reaction to this area of the training.
The participants felt there was little to change in this area of the

Results of Data Collection.

The average score for the instructor was 4.2.

The average score for the materials was 1.9. Most commented that the
screen was too small to see the screenshots in the PowerPoint. Also, many
felt that the handouts were irrelevant.
The average score for the session being a good use of time was 3.3.
More than half (14/20) the participants wished they had the online training
The average score for I learned a lot in this session was 2.3.

Comments: I felt the instructor assumed that I knew something about this
LiveLesson and I didnt. He didnt make it clear at the beginning what we
were learning about. I had to figure it out as the session continued.

I couldnt read the screen shots.

It would have been nice to have a computer to do things he was telling us
could be done as he was telling us about them.
Recommendations. The participants felt that the instructor was clear,
but the visuals were not. In the future, a larger screen should be used so
that participants can see the screenshots clearly. Some of the participants
felt that they should have done the online session first or had computers
with them while doing this session. These things should be considered if
logistics could be overcome. This year, the participants did not yet have log
in information or computers so they would not have been able to try the
system out first. Perhaps the face to face training could take place after
most participants had a chance to do the online training.
Reaction Sheet for Online Modules. Since most of these questions
have only three responses, the interpretation will be different. The negative
answer will be given a score of -1, the neutral answer 0 and the positive
answer will be scored at 1. If a question receives an average score that is
negative, that aspect will need to be improved.
Results of Data Collection. There were no significant technical glitches
with only 2 of the 20 reporting 1 or 2 glitches. No question received an
average score that was negative. 12/20 felt that the online training should
have been done first. Four people felt that the seminar was unnecessary and
three were not sure.
Comments from the free response:
I would have liked to learn more about best practices for using LiveLesson.
Should we just teach lessons like in a classroom or use it to review key
points? How long should LiveLesson sessions be?
I liked the self-paced nature of the training.
I felt overwhelmed with training this week. I wish I could have less training
this week and more time to practice with LiveLesson.
I wish other teachers would talk about how they prepare for LiveLesson
and what they do there.
Recommendations. The online training was done well, from a reaction
standpoint. After the participants took the online training, fewer of them felt
that it should have been done before the seminar. Based on this and the
responses to the seminar, perhaps the seminar should be adjusted to give
new teachers a better idea of what LiveLesson is and what they are

expected to learn from the seminar (just an overview/introduction to the

Level 2
Learning Data Collection for Seminar. The presenter will write up a
narrative summary of the session, including the responses to the questions
to give to the evaluation team.
Narrative Summary. By the end of the session, the group was able to
answer all the questions correctly. They did appear to have a deer in the
headlights look. Some of the questions during the seminar indicated that
they did not understand that it was synchronous instruction. They also
wondered how to do specific things, which I assured them they would learn
in the online module.
Learning Data Collection for Online Modules. There is a target score
of 80% for each participant. If the average score is below 80% or if a
question is missed by more than half the participants, the online portion will
need to be evaluated to determine how to better align the instruction and
Results of Data Collection. 18/20 participants scored an 80% or higher
on the online assessment. The other two scores 74% and 78% respectively.
The only question that more than half the participants answered incorrectly
was about break out rooms.
Recommendations. The session and module produced the desired
learning. The only area that needs to be further evaluated is the break out
room question and training segment.
Level 3
Data Collection. The training will be considered successful if the average
uses of that feature are at least weekly by self-report and used twice during
the lead teacher observation period. Further analysis will need to be done if
more than half the participants learned to do something on the list through a
method other than the training (or if they do not know how to do something
on the list).
Results. Both the self-report and lead teacher evaluations indicate that
all teachers are doing the skills learned in the Basic LiveLesson. However,
more than half the participants are not doing the skills learned in the
advanced section of LiveLesson. Most report learning the skills during the
online portion of the LiveLesson, but some reported that other teachers
taught them the skills, especially the skills they were doing from the
advanced module.. No one reported learning any of the skills at the seminar.

As a result of the low scores for the advanced skills, the evaluation team
decided to interview a sample of teachers (5 of the 20) to see why they were
doing the basic skills and not the advanced skills. Most reported that they
forgot that those things were possible or they did not think they were
necessary. When they learned that they should be doing something, like
logging attendance, they asked another teacher about it.
Recommendations. The surveys and interviews indicate there may be
some information overload at the beginning of the school year. In order to
make sure teachers are fully utilizing LiveLesson, it is recommended that the
advanced portion be taken later into the school year, when teachers have
become comfortable with the basics and are better equipped to make things
more complicated.
Level 4
Data collection. Attendance information is recorded automatically by the
software and can be downloaded into a spreadsheet. If the attendance to
LiveLesson sessions are decreasing on average as time goes by, the
evaluators will need to determine how to adjust the training so that teachers
use LiveLesson more effectively. The school calendar and policy will also
be considered when analyzing these trends. For example, if a mandatory
LiveLesson policy is implemented for certain students, those students will
not be included in the final count.
Results. For most teachers, attendance was high at the beginning and
began to dip. However, after a few weeks it began to increase. Chart 1
shows a sample of two classes.
Chart 1

Attendance: September - December


1 2 3

4 5 6 7

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Recommendations. There is a slightly positive trend in the data which

may indicate that the teachers are using LiveLesson sessions well. It
would be beneficial to survey students to see what draws them to
LiveLesson and what causes them to miss.

Executive Summary
To: Members of the School Board
From: Evaluation Team
As a part of your ongoing process to improve the onboarding process, you
asked us to do an evaluation of the LiveLesson training for newly hired
teachers to determine ways that this training could be improved. Since
LiveLesson is one of the main ways that teachers communicate instruction
to students, it is important that they are well trained in this medium.
Currently, new teachers attend a one hour seminar that introduces
LiveLesson and then take an online course that consists of three modules:
Basic LiveLesson, Policies for LiveLesson and Advanced LiveLesson.
The evaluation was based on Kirkpatricks four levels of evaluation: reaction
to the training, learning from the training, transfer of learning to actual skills
and final results (Kirkpatrick & Kirkpatrick, 2006, p. 21).
After the seminar and online training, participants were surveyed about their
reactions and given a traditional test to assess their learning. Sample
surveys and tests are included in this document. One month and three
months later, participants evaluated their own use of LiveLesson and lead
teachers evaluated the participants use of LiveLesson. Finally, the
students attendance in LiveLesson sessions was analyzed.
The way the seminar was done, some participants were left confused and
there was some problems with the technology and the participants ability to
see the screen clearly. One recommendation of this committee is that the
teachers take this seminar with computers available to them. Perhaps they
could even take the seminar via LiveLesson so that they can really
understand the purpose of LiveLesson and how it will look to their
students. This would also eliminate the visual issues since each teacher
would be in front of his or her own screen.
Based on the tests and the self-reports of skills learned, it appears that there
is too much information in the initial module. Participants learned the basics
of LiveLesson, but they were not implementing many of the advanced
features even three months later. We recommend that the advanced
training be postponed until the teachers become proficient with the basic
Student attendance starts high and then begins to decrease. However,
especially between months 1 and 3, the attendance levels off and then

begins a slow increase. This indicates to us that the training is effective and
students are finding it beneficial to attend the LiveLessons.
In summary, our recommendations include
1) Teaching the seminar in LiveLesson.
2) Postponing the advanced session until later in the school year.
We think that these two simple changes could cause your teachers to use
LiveLesson even more effectively so that more students benefit from the
LiveLesson experience.
We have enjoyed working with your teachers. We look forward to working
with you and them in the future.

Amy Keyser

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs:
The Four Levels (Third ed.). San Francisco: Berrett-Koehler Publishers, Inc.
What is Connections Academy? (2015). Retrieved from
http://www.connectionsacademy.com/online-school on September 18,


A: LiveLesson module objectives

B: Gantt Chart of Tasks
C: Reaction sheet to seminar
D: Reaction survey to online learning
E: Questions at the end of the seminar
F: Assessments to evaluate learning in online modules
G: Self-Evaluation of LiveLesson Behaviors
H: Lead Teacher Evaluation of LiveLesson Behavior

Appendix A: Online LiveLesson module objectives

Module 1: Basics of LiveLesson

Introduction to LiveLesson
LiveLesson Server Clusters
Accessing Your LiveLesson
Pod Demonstration
Uploaded Content
Meeting Room Management,
Microphone, and Layouts
LiveLesson Room Maintenance

Module 2: Connections
Educations Policies for

Copyright, YouTube and Fair Use

Policies in LiveLesson
LiveLesson Access, Offline
Recordings and Room Settings

Module 3: LiveLesson Advanced LiveLesson Breakout Rooms

and Quizzes
LiveLesson Meeting Room Data
Recording and Editing
LiveLesson Sessions
Recording Presentations with
Adobe Presenter

Contents retrieved from:

641457&idSection=691112&self=True&menu=on&resizeWindow=true# on
October 1, 2015.

Appendix B: Gantt Chart of Tasks

Reaction Sheet
Analysis of Learning
Analysis of reaction sheet
Online Learning Module
Analysis of online Learning
Self Evaluation of Practices
Evaluation of Practices
Self Evaluation of Practices
Evaluation of Practices
Data analysis of behavior evaluations
Analysis of Attendance
Final Report Delivered

Appendix C: Reaction sheet to seminar

Directions: Circle the appropriate response, where applicable. All responses
should be kept anonymous. For the numbered choices, 1 means that you
strongly disagree and 5 means that you strongly agree.
The length of the course was

too short. too long. just


The room was

too cold.

too hot. just

What could have been done to make this session more comfortable?
The instructor spoke

too soft.

too loud. just

The instructor seemed prepared.

1 2 3 4 5

The instructor seemed knowledgeable about


1 2 3 4 5

The instructor clearly explained how to use


1 2 3 4 5

I felt comfortable asking the instructor


1 2 3 4 5

The instructor made me feel like is easy to

use LiveLesson.

1 2 3 4 5

What more could the instructor have done to help you learn how to use

The visuals were easy to see

1 2 3 4 5

The technology ran smoothly.

1 2 3 4 5

I think I will use the handouts in the future.

1 2 3 4 5

How can the materials used in the session be improved?

This session was a good use of my time.

I wish the online session had been held first.
I learned a lot in this session.

1 2 3 4 5


1 2 3 4 5

What other comments do you have about this session?

Appendix D: Reaction survey to online learning

Directions: Answer all the questions in this survey. When you are done, hit
Submit. You will not be able to complete the course until you submit the
The technology ran smoothly (all
There were no glitches.
links worked, the videos played,
There were 1 or 2 glitches.
There were more than 2 glitches.
I could not get any links to work.
The text increased my
understanding of how to use

Not really
Very much

The videos increased my

understanding of how to use

Not really
Very much

The screen shots increased my

understanding of how to use

Not really
Very much

The Practice sections increased my

understanding of how to use

Not really
Very much

The Activities increased my

understanding of how to use

Not really
Very much

The length of the modules were

Too short.
Just right.
Too long.
Too much: I feel overwhelmed.
Just right: I think I can lead a
Too little: I still have too many
Boring. I could barely stay awake.
Interesting. I was fully engaged!

The amount of information covered


The way the information was

delivered was
I wish this online training had been
done before the seminar.

I do not feel the online training was

I do not feel the Face to Face session
was necessary.
Which topics would do you wish you
would have learned more about?
What did you like about the
What could be done to improve the

Not sure
Not sure
Free response
Free response
Free response


Appendix E: Questions at the end of the seminar

What is LiveLesson?
How often will you use LiveLesson?
What are some best practices when using LiveLesson?
Are there any other questions about LiveLesson?

Appendix F: Assessments to evaluate learning in online modules

Module 1: Sample questions with responses
1) It is good practice to have all your materials loaded into LiveLesson
before you start a lesson?
a) No, because you will make the screen too crowded.
b) No, because you will run out of memory.
c) Yes, because uploading the materials while teaching may slow the
bandwidth down and cause a poor viewing experience for students.
d) Yes, because you cant upload anything when the lesson is live.
2) Which is the best pod to use if you wanted to find out if your students
know prerequisite material?
a) share pod
b) poll pod
c) video pod
d) chat pod
3) Who can access your LiveLesson room by clicking on your DataView?
Check all that apply.
a) parents of students
b) your students
c) other teachers
d) anyone with a computer
4) Which type of file can NOT be uploaded to LiveLesson?
a) pptx
b) docx
c) jpg
d) swf
5) When you end a meeting, all your pods are deleted.
a) True
b) False.
Module 2: Sample questions with responses
1) You can never upload copyrighted material into LiveLesson.
a) True
b) False
2) You can never upload YouTube videos into LiveLesson.
a) True
b) False

Module 3: Sample questions with responses

1) What is the default status of participants in a break out room?
a) attendee
b) presenter
c) host
2) You should log LiveLesson attendance.
a) True
b) False
3) You can edit LiveLesson recordings.
a) True
b) False

Appendix G: Self-Evaluation of LiveLesson Behaviors

Directions: Mark the appropriate box for each statement
Every Weekl Occasiona
I provide a link to my LiveLesson
I use a share pod in your LiveLesson
I use a poll pod in your LiveLesson.
I use a Fileshare in your LiveLesson.
I use a video pod in myLiveLesson.
I use a chat pod in myLiveLesson.
I upload content (PowerPoints, pictures,
etc.) to my LiveLesson
I have trouble getting my microphone
to work in LiveLesson.
I record my LiveLesson sessions.
I use more than one layout for
I enable audio rights for attendees.
I enable drawing rights for attendees.
I delete pods from LiveLesson.
I use breakout rooms in LiveLesson.
I edit my LiveLesson recordings.
I log attendance from my LiveLesson.


What would you like to learn more about in terms of using LiveLesson?
Would you like to receive more training in LiveLesson? If so, how would
you like to receive it. Number your responses with 1 being the best way:
___ A LiveLesson session.
___ A LiveLesson recording.
___ An online training like the original training.
___ A seminar.
___ one on one training with another teacher.
__ I would not like to do more training on LiveLesson.

Tell where you learned to do each thing.

I dont
how to
do it.
I provide a link to my
LiveLesson room
I use a share pod in
your LiveLesson
I use a poll pod in
your LiveLesson.
I use a Fileshare in
your LiveLesson.
I use a video pod in
I use a chat pod in
I upload content
pictures, etc.) to my
I have trouble getting
my microphone to
work in LiveLesson.
I record my
I use more than one
layout for
I enable audio rights
for attendees.
I enable drawing
rights for attendees.
I delete pods from
I use breakout rooms
in LiveLesson.
I edit my
I log attendance from
my LiveLesson.


Asked a


Appendix H: Lead Teacher Evaluation of LiveLesson Behavior

Lead Teachers: Attend LiveLesson or view three LiveLesson recordings.
Mark the number of times you see each thing being done.
The teacher used a share pod in
The teacher used a poll pod in
The teacher used a Fileshare in
The teacher used a video pod in
The teacher used a chat pod in
The teacher uploaded content
(PowerPoints, pictures, etc.) to
The teacher had trouble getting the
microphone to work in LiveLesson.
The teacher recorded LiveLesson
The teacher use more than one layout
for LiveLesson.
The teacher enabled audio rights for
The teacher enabled drawing rights for
What things did the teacher do well?
What are some things the teacher could improve upon?