FORMATIVE EVALUATION

Table of Contents
Introduction.......................................................................................... 3
Alpha Test............................................................................................... 3
Beta Test.................................................................................................. 5
Assessment Questions...................................................................... 7
Evaluation.............................................................................................. 9
Evaluation Results.............................................................................15

INTRODUCTION
This Formative Evaluation report details the process by which the Band Orientation Program for Almost Elton
John and the Rock Band was evaluated and tested. This review includes data on a the Alpha and Beta tests
for the Program, detailing the Objectives of the tests, the audience, the criteria, the process, and results and
feedback.

ALPHA TEST
Objective
The objective of the Alpha test was to test basic functionality of the course and the medium of the course.
This included testing for usability of the slides, continuity of content, navigation, audio and video quality,
and other technical issues.

Audience
Alpha testers were comprised of:
A Subject Matter Expert (SME)
An instructional design expert
Three instructional design graduate students

Criteria
All Alpha testers were asked to evaluate the program for clear statement of objectives and instructional
goals, content quality and accuracy, content continuity, relevance, and scope. The testers were asked to provide feedback in the form of a written report.

Process
The Alpha testers were instructed to take the program by following a weblink provided by email. The testers
were provided with a password to access the program, and informed that running time for the program was
approximately 1 hour.
3

Results and Feedback

Based on the comments from the Alpha testers, most were impressed with the overall look and layout of the
slides and graphics. The text was consistent and easy to read, and the voiceover was clear and easy to understand.
The suggested revisions revolved largely around technical aspects of Captivate, grammatical and spelling
errors, some content accuracy, and the functionality of the assessment section. Also, the videos required
some editing to remove some noise and false starts from the footage. The overall assessment, though, was
positive.
Below are excerpts from the reports of two of the Alpha testers:
Subject Matter Expert
We need to edit some of the clips. Misspelling: ADVISE should be spelled with a C ADVICE. LEATHERHEADS
is misspelled. ID Thief should be IDENTITY THIEF; DESERT SYMPHONY Palm is not a part of their name. I
need to get you a current Stage Plot. The dos and donts are hysterical!!!
Maybe trim the interviews? They go on a bit long. I was getting bored listening to myself! Brevity is soul
of wit! Rehearsals are paid at $25 per rehearsal SLIDE 20. Travel Tips: Some of the onscreen writing didnt
show on the second screen. Liquids, gels are allowed in 3 oz containers SLIDE 25
Instructional Design Professional
On the initial loading screen: you can speed things up by going to Preferences -> Project -> Start and End,
checking Preloader, and setting Preloader % to something like 15. This tells Captivate to start running the
project as soon as it has loaded 15% of the content, instead of waiting until 100% is loaded. For users with
slow connections, this can save quite a bit of time. You can tweak other Project settings to improve performance as well.
I noticed at least one slide, I think the first Introduction slide, advanced to the next slide by itself when I had
to leave the tutorial for a while. You can prevent this by including an invisible click box on each slide with
the option Pause project til user clicks. You can see examples of this in the Arkansas files.
Some typos...Introduction slide -- replace its with its; Mission statement slide -- commas go inside quotation marks; Vision slide -- period goes inside quotation marks
Obviously, the videos need some edits to remove his flubs and false starts. It would be good to have the contract PDF open in a new window rather than the same window as Captivate. Some users might have trouble
navigating back to the tutorial from the PDF. On the assessment questions, I was only able to attempt each
question once as I went through -- probably intentional?
Question 3 -- response J is cut off. This seems like a weak question because there are too many options.
Question 8 -- response D is cut off I answered 73% of the questions correctly and this was a failing score -probably intentional. Clearly Im not ready for the Rocket Band yet!

BETA TEST
Objective
The objective of the Beta test was to further check for functionality of the course, as well as to assess aspects
of the program related to subject matter, auxiliary information, interface, navigation, and pedagogy.

Audience
The Beta testers were comprised of:
Three Instructional design graduate students
Two Current members of the Rocket Band (who are the intended audience for the program)

Asseessment Criteria and Instruments

All Beta testers like the Alpha testers were asked to evaluate the program for clear statement of objectives and instructional goals, content quality and accuracy, content continuity, relevance, and scope. Additionally, they were to evaluate the program relevance of the subject matter, auxiliary information, interface
design and functionality, navigation, and pedagogy.
The instrument used for assessment was a Google Form survey, which was linked to the program.

Procedures
Beta testers were instructed to go through the entire program. On the final slide, the testers were to follow a
hyperlink to the Google Form. The survey asked the testers to rate the program on a scale of 1 to 5 in terms
of its level of effectiveness in a number of areas (subject matter, auxiliary information, interface, navigation,
and pedagogy). For each area, the testers was given the opportunity to give feedback in the form of written
comments.

Limitations of Beta Test

There are a number of important limitations that must pointed out pertaining to the Beta test.
To start, we were unable to find any new band members to go through the program. While the feedback
from existing band members is certainly useful, the other intended audience is band members who are not
already acquainted with the normal operating procedures of the organization.
Another limitation was in the format of the survey. It was mentioned by a number of the Beta testers that
the survey was too long, and that certain questions seemed irrelevant or confusing. While there were fields
available for comment feedback, we did not receive as much comment feedback as we would have liked.

Results Feedback
Most of the feedback indicated that the testers found the program engaging, informative, functional, aesthetically attractive, and entertaining. Testers indicated that the program appeared to be an interesting and
unique way to understand the subject matter and to learn the various learning objectives.
Criticisms of the program included largely technical matters, such as slide duration, design anomolies, and
issues with the assessment questions. There were also comments about the Google Form Evaluation Survey
itself, which had two of the ratings headings switched around. Overall, the feedback again was very positive.
Comment Feedback on Subject Matter
I am not really an expert on this subject. -intro slide was a bit quick -code of conduct video bits...there was
a ghostly image and slide bar in the middle...a remnant of something that had been removed? -assessment
section, contracts...reresponsibility instead of responsibility. I learned a lot from the basic info in your presentation. Since Im not a musician I cant reliably comment on tech jargon. I think language and layout and
design fit your target audience well. -did u switch places on the good/acceptable scale? from first set of?s
to next sets of ?s? i consider good better than acceptable on the poor to excellent scale... It was absolutely
great. The interviews were informative but I felt a sense of wanting them to be expedited.
Comment Feedback on Auxiliary Information
i didnt check out the links but they looked to be related to the subject matter. the poor/needs work/good/
acceptable/excellent arrangement still bothers me...think good/acceptable should be reversed, (isnt good
better than acceptable?) but thats just me. I only put needs work because I am unsure where your help features were. Is cc considered a help feature, is the toc considered a help feature?
Comment Feedback on Interface
N/A
Comment Feedback on Navigation
i did not try to bookmark anything or restart the training and just went straight thru.... but i loved the simple
straightforward navigation bar you used. it looked good and was easy to use.
Comment Feedback on Pedagogy
If I was a musician, hired to tour with the band, this module would give me a good foundation to function
appropriately in the band. Good tips and clear goals. Excellent use of photography without being distracting. Very professional layout. I answered the quiz at the end and it said that I failed showing that I didnt even
get one answer correct - I am sure that it was an error. Not sure why it would not let me start again. I did not
hit submit after the last question because I did not see it clearly. I suppose that this is why I FAILED. GREAT
presentation - makes me very proud to be a member of the Rocket Band!!!

ASSESSMENT QUESTIONS
Please answer the following questions. (The correct answers are in red)
1. The mission statement of Almost Elton John and the Rocket Band is:

A. Be the best Elton John Tribute in the United States

B. Celebrate the songs and career of Elton John

C. Help introduce people to music of the 1970s

D. Play with symphony orchestras
2. Members of the Rocket Band:

A. Are all hired guns and are there to just play another gig

B. Have equal billing and prominence in the Almost Elton John show

C. Arent very sociable because they work so hard

D. Are like a family
3. Which of the following are acceptable behaviors for members of the band? Please select all that apply.

Get to the hotel and spend quiet time alone.

Show up at least 10 minutes early to all band engagements.

Its okay to be difficult to deal with, as long as youre a really good musician.

Show up to the performance in blue jeans and make faces at the audience.

Ask the band leader if you can have a solo number in the show.

Ask the band leader how you can help improve a certain part of the show.

Make silly jokes during a van ride.

Pass out personal business cards during the intermission at a performance.

While in flight to engagement, invite other people to a concert.

Demand that you have your own hotel room because you have 20 years of experience in the industry.

You have to miss a rehearsal, but you called two weeks ahead of time to let the band leader know.

You have to miss a concert that you are under contract to perform because a better paying offer came through, but
you called two weeks ahead of time to let the band leader know.
4. True or False:
Craig views his band as a group of highly-trained professionals who are in the group to simply do a job, nothing more.
5. Complete the following sentence:
Craig is interested in creating ____________________ in the lives of those around him.

A. Careers

B. Opportunities

C. Purpose and Meaning

D. Cheesecake
6. Contracts are:

A. Important because they obligate you to certain responsibilities with the band

B. Important because all bands have contracts

C. Negotiable and can be broken at any time with no fear of recompense or termination

D. Optional
7. During air travel, it is recommended that you:

A. Meet the band at the flight gate
7

B. Meet the band at the security gate

C. Meet the band at curbside dropoff
D. Meet the band at the Delta Sky Priority Desk

8. Showing up to soundcheck is:

A. Highly recommended, but not mandatory.

B. Completely optional and you can stay at the hotel if you like.

C. Mandatory for all members of the band.

D. Not necessary if youre a backup singer because you dont have equipment to setup.
9. Craig believes in __________________ the people he works with.

A. Challenging

B. Simply employing

C. Scrutinizing

D. Investing in
10. If you have special dietary needs, please make sure they are included in:

A. The band schedule and itinerary

B. Your per diem

C. The production rider

D. Your personal contract
11. If you get an email from Craig requiring a time-sensitive reply:

A. Its okay to call another band member

B. Wait about a week to get back to him

C. Reply as soon as possible.

D. Dont respond
12. True or False: the ideal employee will make the band look good, sound good, and feel good.

EVALUATION
A live form of the Evaluation Survey can be found here.

10

11

12

13

14

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

EVALUATION RESULTS
Pat Strawser

Edit this form

A live form of the Evaluation Survey can be found here.

5 responses
Publish analytics

Summary
Learning objectives were clear [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

0%

Excellent

100%

Content structure was easy to follow [1 SUBJECT MATTER]

1 of 20

4/23/14 7:20 PM

15

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

0%

Acceptable

0%

Good

40%

Excellent

60%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Content was accurate [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

20%

Excellent

80%

Language, style, and grammar was correct [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

2 of 20

4/23/14 7:20 PM

16

Band Orientation Evaluation Form - Google Forms

Acceptable

0%

Good

0%

Excellent

100%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Reading level was appropriate for the audience [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

0%

Excellent

100%

Cultural bias language [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

20%

3 of 20

4/23/14 7:20 PM

17

Band Orientation Evaluation Form - Google Forms

Excellent

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

80%

Cultural bias reference [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

20%

Excellent

80%

Correct use of technical terms and jargon [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

20%

Excellent

80%

4 of 20

4/23/14 7:20 PM

18

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Spelling and punctuation is correct [1 SUBJECT MATTER]

Poor

0%

Needs Work

0%

Acceptable

0%

Good

25%

Excellent

75%

Subject Matter Comments

I am not really an expert on this subject.

-intro slide was a bit quick -code of conduct

video bits...there was a ghostly image and slide bar in the middle...a remnant of something
that had been removed? -assessment section, contracts..."reresponsibility" instead of
responsibility. I learned a lot from the basic info in your presentation. Since I'm not a
musician I can't reliably comment on tech jargon. I think language and layout and design fit
your target audience well. -did u switch places on the good/acceptable scale? from first set
of?s to next sets of ?s? i consider good better than acceptable on the poor to excellent
scale...

It was absolutely great. The interviews were informative but I felt a sense of

wanting them to be expedited.

Introductory slides are clear, engaging, and informative [2 AUXILIARY

INFORMATION]

5 of 20

4/23/14 7:20 PM

19

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Directions are clear [2 AUXILIARY INFORMATION]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

Help features are accessible and easy to use [2 AUXILIARY

INFORMATION]

6 of 20

4/23/14 7:20 PM

20

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

20%

Good

20%

Acceptable

0%

Excellent

60%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Assessment/quiz questions are reasonable and appropriate for scope of

the training [2 AUXILIARY INFORMATION]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

Auxiliary Information Comments

i didn't check out the links but they looked to be related to the subject matter. the

7 of 20

4/23/14 7:20 PM

21

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

poor/needs work/good/acceptable/excellent arrangement still bothers me...think

good/acceptable should be reversed, (isn't good better than acceptable?) but that's just
me.

I only put needs work because I am unsure where your help features were. Is cc

considered a help feature, is the toc considered a help feature?

Slide displays are clear and easy to read [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

Presentation modes are appropriate and engaging (i.e. video clips, audio
clips, etc.) [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

20%

8 of 20

4/23/14 7:20 PM

22

Band Orientation Evaluation Form - Google Forms

Acceptable

20%

Excellent

60%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Text/fonts are clear and easy to understand [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

Assessment/quiz questions are reasonable and appropriate for scope of

the training [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

9 of 20

4/23/14 7:20 PM

23

Band Orientation Evaluation Form - Google Forms

Excellent

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

100%

Animation and graphics are smooth and professional [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

Audio is clear and of good quality [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

10 of 20

4/23/14 7:20 PM

24

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Video is clear and of good quality [3 INTERFACE]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

Interface Comments
It is easy to navigate the slides within the training [4 NAVIGATION]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

11 of 20

4/23/14 7:20 PM

25

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Navigation icons are clear and consistent [4 NAVIGATION]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

The training can be restarted with no difficulty [4 NAVIGATION]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

40%

Excellent

60%

Bookmarking within the training is clear and easy to implement [4

NAVIGATION]

12 of 20

4/23/14 7:20 PM

26

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

0%

Good

0%

Acceptable

20%

Excellent

80%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Navigation Comments
i did not try to bookmark anything or restart the training and just went straight thru.... but i
loved the simple straightforward navigation bar you used. it looked good and was easy to
use.

The methods used to present the content are appropriate for the
audience [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

13 of 20

4/23/14 7:20 PM

27

Band Orientation Evaluation Form - Google Forms

Excellent

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

100%

The training is interactive and engaging [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

20%

Excellent

60%

The training allows for learners to work together [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

14 of 20

4/23/14 7:20 PM

28

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

The program provides other learning tools and learning strategies [5

PEDAGOGY]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

The program gives user sufficient control (i.e. volume, pause, continue,
table of contents, etc.) [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

20%

Excellent

80%

Assessment questions are relevant to the learning objectives [5

15 of 20

4/23/14 7:20 PM

29

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

PEDAGOGY]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

Assessment questions encourage the learner to think and process the

information [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

0%

Excellent

80%

There is variety in the assessment questions (i.e. multiple choice, fill in

the blank, true/false, etc.) [5 PEDAGOGY]
16 of 20

4/23/14 7:20 PM

30

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Answering questions is clear and reasonable [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

20%

Acceptable

20%

Excellent

60%

Questions are not misleading or ambiguous [5 PEDAGOGY]

17 of 20

4/23/14 7:20 PM

31

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

0%

Good

20%

Acceptable

20%

Excellent

60%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Question feedback is constructive and appropriate [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

20%

Excellent

80%

The program teaches valuable information [5 PEDAGOGY]

18 of 20

4/23/14 7:20 PM

32

Band Orientation Evaluation Form - Google Forms

Poor

0%

Needs Work

0%

Good

0%

Acceptable

0%

Excellent

100%

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

Successful completion of the assessment illustrates mastery of the

information taught [5 PEDAGOGY]

Poor

0%

Needs Work

0%

Good

0%

Acceptable

20%

Excellent

80%

Pedagogy Comments
If I was a musician, hired to tour with the band, this module would give me a good

19 of 20

4/23/14 7:20 PM

33

Band Orientation Evaluation Form - Google Forms

https://docs.google.com/forms/d/1QoocXALhVbPTbhbz1me4...

foundation to function appropriately in the band. Good tips and clear goals. Excellent use of
photography without being distracting. Very professional layout.

I answered the quiz at the

end and it said that I failed showing that I didn't even get one answer correct - I am sure that
it was an error. Not sure why it would not let me start again. I did not hit "submit" after the
last question because I did not see it clearly. I suppose that this is why I FAILED. GREAT
presentation - makes me very proud to be a member of the Rocket Band!!!

Number of daily responses

20 of 20

4/23/14 7:20 PM

34