Runninghead:EFFECTIVENESSANDUSABILITYREPORT

EffectivenessandUsabilityReport:MoonLesson
TraciHogrefe,NikkiMarcel
CaliforniaStateUniversityMontereyBay

IST622AssessmentandEvaluation
Dr.BudeSu
June16,2015

EFFECTIVENESSANDUSABILITYREPORT
2

TABLEOFCONTENTS
Page
INTRODUCTION3
METHODOLOGY....4
PrototypeDescription4
Learners.5
TryoutConditions.6
Process..6
Pretest..6
LessonandObservations.7
Posttest7
ParticipantUsabilitySurvey..7
RESULTS..9
EntryConditions9
Instruction.9
Outcomes..10
Recommendations.14
SUMMARY..16
REFERENCES.17
APPENDICES.18
A.Pretest18
B.Lesson.22
C.Posttest..26
D.ParticipantUsabilitySurvey..30

EFFECTIVENESSANDUSABILITYREPORT
3

Introduction
TheMooncanbeseennearlyeverynight,andsometimesinthedaytimeaswell.Many
peoplemaynotbeawareofthevariousphasestheMoonhas,thecorrecttermsusedtodescribe
thosephases,andhowtheyactuallyoccur.
ThepurposeofthistryoutistomeasuretheeffectivenessandusabilityofaMoonlesson
designedanddevelopedbyBrianBirdsallandTraciHogrefe,respectively.Theadministratorsof
thistryoutsetouttoanswerthefollowingtwoquestions:1)Doesthelessoneffectivelyteachthe
statedlessonlearningobjectivesand2)Howintuitiveisthelesson,givenitisamultimedia
lessondesignedtobecompletedindependentlybytheuser?
Theadministratorsrecruitedelevenparticipantstocompleteapretest,theinteractive
lesson,aposttest,andausabilitysurvey,thenanalyzedthedatatodeterminetheeffectiveness
andusabilityofthelesson.Oftheelevenparticipants,theadministratorswereabletoobservesix
participantscompletethefulltryout,obtainingadditionalinsightintothelessonusability.
Belowarethelessonlearningobjectiveswhichwereassessedduringthistryout:
giventhecontentofthelesson,thelearnerwillbeabletostatethatthereareeightmain
phasesoftheMoon
giventhewords
waxing
and
waning
,thelearnerwillbeabletoindicatetheappropriate
definition
giventhecontentofthelesson,thelearnerwillbeabletodefine
geosynchronousorbit

giventhecontentofthelesson,thelearnerwillbeabletodescribetheonlytimetheEarth
castsashadowontheMoonand
givenimagesoftheMoonphases,thelearnerwillbeabletochoosetheappropriateterm.

EFFECTIVENESSANDUSABILITYREPORT
4

Methodology
PrototypeDescription
Usuallytaughtinlateelementaryschool,manyadultshavemisconceptionsofhowthe
perceivedshadowontheMoonssurfaceiscast.Thelessonincludesapreassessmentlinkingto
theprototype,whichthenlinkstothepostassessmentandusabilitytest.Theprototype,designed
byBrianBirdsallanddevelopedbyTraciHogrefeofthe20142015onlineonlyCSUMBMIST
cohort,isamultifacetedlessonwhichincludesinstruction,interaction,practice,andresults.
Followinginstructionalprinciples,suchastheMultimedia,Modality,andSignaling
principlesdevelopedbyRichardMayer(
Clark&Mayer,2011)
,theuserisguidedthroughoutthe
lessonwithnarration,signalingcues,andprompts.Thebeginningchallengestheuserscurrent
knowledgeofcertainvocabularythatislaterexplainedasthelessonunfolds.Thefirstinteraction
iswhentheuserisprovokedwithaquestionseeninthepretest:Howmanyphasesdoesthe
Moonhave?Asthelessonmovesalong,theuserispresentedwithamenudisplayingallofthe
Moonphasesandispromptedtoclickoneachphasetolearnmoreaboutit.Whentheuser
completesthislesson,theuserisconfrontedwithapracticeandanopportunitytoreviewthe
lessonifthepracticewasnotsuccessful.Finally,theusercomesacrossthegradedassessment,
completewitharesultspage.
AftertheuserfeelsconfidentintheirknowledgeoftheMoonphases,theusercanmove
ontothesecondlessononorbitsandeclipses.Followingthecompletionofthislessonand
assessmentincludingfillinblankquestions,theusercanreviewtheirresultsoranyofthe
lessonsaswellasviewthecreditsbyclickingontheappropriatebuttons.Alinktothe
posttestproceedsthecredits.

EFFECTIVENESSANDUSABILITYREPORT
5

Thelessonhasmanystrengths,suchasitbuildsuponpreviousknowledgeandnewly
learnedskills.Multimedia,interaction,andopportunitiestoviewresultsareelementslikely
appreciatedbytheusersandkeepstheirattentionthroughoutthelesson.Avarietyof
opportunitiesarepresentedtoallowtheusertointeractwiththelessonindiverseways.A
weaknessoftheprototypemaybealackofsignaling.Thelessoncanhavesomelevelof
improvementbyincludingsignalingduringtheindividualMoonphaseinstructiontoemphasize
keylearningaspectsandkeepthelearnersinterestasopposedtolisteningtonarrationwitha
stillimageforhalfaminute.
Learners
Thetargetedlearnersforthisevaluationwereanyadultseighteenyearsofageorolder
whoareproficientinEnglish.Additionally,learnersshouldhavebasiccomputeruse
experience,includingusingamouseortrackpad,typingskillstocompleteonetotwoword
answers,andinternetnavigationskills.Althoughnotarequirementtoparticipateinthis
evaluation,learnerswithadesiretolearnnewinformationandwhohaveapositiveattitude
towardselfpacedelearningweretargeted.
Duringthecourseoftheevaluation,elevenlearnersparticipated:sixfemalesandfive
males,withagesrangingfromeighteentosixtynine.Demographicinformationofeachofthe
learnerswasobtainedthroughaseriesofquestionsatthebeginningofthepretest(AppendixA).
Eachofthelearnershadabasicpriorknowledgeofthefollowingconcepts:Earth,Moon,orbit,
andmovement,whichwasdemonstratedthroughdiscussionsprevioustoparticipatinginthe
prototypeevaluation.

EFFECTIVENESSANDUSABILITYREPORT
6

TryoutConditions

Foroptimaltryoutconditions,theparticipantsneedacomputerwithinternetaccess,a

keyboard,speakersorheadphones,monitor,andamouseortrackpad.Ideally,theparticipantis
inalocationwithlittledistractionsandanynoisesshouldnotinterferewiththequalityofthe
soundprojectedbythelesson.
Theentirelesson,assessments,andusabilitysurveyareexpectedtobecompletedwithin
2530minutes.Ifauserchoosestoreviewthelessonseveraltimes,thelessoncantakelonger.
Thelessonaloneisexpectedtobecompletedinabout15minutesidealforthetypicaladults
attentionspanof20minutes(
Dukette&Cornish,2009).
Process
Theprocessofeffectivenessandusabilitytestingwasconductedinfourparts:1)pretest
tomeasurelearnerspriorknowledge,2)lessonandobservations,3)posttesttomeasure
participantslearning,and4)participantusabilitysurveytomeasurereactionandusabilitydata.
Ineachoftheparts,participantswereaskedtoidentifythemselvesonlywithachosenusername
inordertomatchalloftheirdataduringanalysis,buttokeeptheirinformationconfidential.
Pretest.
Thepretest(AppendixA)consistedofaseriesofdemographicquestionstoobtain
informationabouttheparticipants,aswellasquestionsaboutcontenttheparticipantswouldbe
exposedtoduringthelessoninordertoassessparticipantspriorknowledge.Thequestionson
lessoncontentincludedshortanswer,multiplechoice,andanumberofquestionsidentifyingthe
propertermforillustrationsofsomeoftheMoonphases.Thepretestwascreatedand

EFFECTIVENESSANDUSABILITYREPORT
7

administeredusingGoogleForms.Thefinalscreenofthepretestcontainedathankyou
messageandalinktotheinternetbasedlesson.
Lessonandobservations.
Theadministratorswereabletoobservesixoutoftheeleven
participantscompletetheentireprocess.Theparticipantswerepromptedtobeginthelessonand
movethroughitastheywouldiftheyweretakingacourseindependently,thatis,withno
administratorpresentforsupport.Theadministratorswereabletoobservehowtheparticipants
proceededthroughthelesson(AppendixB),recordtheamountoftimeusersspentwithinthe
lesson,andmakenoteofanyhardshipstheparticipantsmayhaveencounteredconcerningthe
intuitivenessofthelesson.Thefinalslideofthelessoncontainedalinktotheposttest.
Posttest.
Theposttest(AppendixC)consistedofninequestionssimilartobutnot
exactlythesameasthequestionsposedinthepretest.Theadministratorschosetochangethe
formatofmanyofthequestionssothatthelearningcouldbemeasuredbasedontheinformation
participantsgainedthroughthelesson,ratherthanpossiblybeinggainedfromthepretest
questions.Theknowledgebeingmeasured,however,remainedconsistentinboththepreand
posttests.TheposttestwascreatedandadministeredusingGoogleForms.Thefinalscreenof
theposttestthankedanddirectedtheuserwithalinktotheparticipantusabilitysurvey.
Participantusabilitysurvey.
Thefinalpartoftheprocesswastheparticipantusability
survey(AppendixD)usedtomeasuretheusabilityofthelessonandtheparticipantsreactionto
theformatofthelesson.ThisshortsurveywascreatedandadministeredusingGoogleForms.
Mostofthequestionsaskedtheparticipantstoratestatementsaboutvariousaspectsofthelesson
onaLikertstylescalerangingfrom
stronglydisagree(1)
to
stronglyagree(5)
.Thefinalfour
questionsaskedparticipantstoexplainanythingtheyliked/dislikedaboutthelessonandshare

EFFECTIVENESSANDUSABILITYREPORT
8

anyadditionalsuggestionstheyhad.Completionoftheparticipantusabilitysurveyendedthe
entiretryout.

EFFECTIVENESSANDUSABILITYREPORT
9

Results
EntryConditions
Participantswereexpectedtosetaside2530minutestocompletetheentireprocedure:
includingassessments,lesson,andsurvey.However,observedparticipantsonlyneeded2025
minutestocompletethefullpackage.
Asthetryouttestswereexpectedtobegin,CSUMBsserverwasdown,souserswerenot
abletoaccessthecontentviatheinternet.Fortunately,theadministratorshadthelesson
accessibleontheirpersonalcomputersandwereabletomanuallyconnecttheusersfromthe
pretesttothelocallessonandsoforth.Thisalsoallowedtheadministratorstobefirsthand
observersandtakenoteofanycomplicationsorreactionsusersmayhaveexperienced.
Observingtheparticipantsgaveadministratorsafirsthandlookatusersinteractionsand
involvementwiththelearningtool.WhentheCSUMBsserverwasonlineagain,thelessonwas
accessibletousersthatwerebothlocallyandvirtuallylocated.
Althoughnotintentionallydone,manyofthetryoutswereadministeredintheevening,
afterparticipantsreturnedfromwork.Thiscanbeacommonscenewithadulteducation,but
mayhaveadverseeffectsontiredadultsandtheirresults(
Barbera,2013)
.
Instruction
Thelessonisselfcontained,accessibleonlinethroughCSUMBsserver,ITCDLAND.
Thelessonitselfisintendedtobecompletedwithin1015minutes.Ofthesixobservedusers,the
lessonwascompletedbetween10to13minutes,withanaverageof11.5minutes.Ofthe
observedparticipants,thosethatdidwellinthepracticelessonsmovedalongquickerthanthose
thatchosetoreturntoasectionforreview.

EFFECTIVENESSANDUSABILITYREPORT
10

Observedparticipantsshowedexcitementwhentheyreceivedpositivedirectfeedback,as
wellasslightdisappointmentorfrustrationwhentheyreceivednegativefeedback.Feedbackwas
informativetohelptheuserunderstandtheircorrectorincorrectresponses.Theirpracticeresults
reflectedtheirlearningandoftencreatedajoltofexcitementintheobservedparticipants.
Outcomes
WhereH
0
isthenullhypothesis,

1
isthescoreinpercentageofthepretest,

2
isthe
scoreinpercentageoftheposttest,andH
A
isthealternativehypothesis:
H
0
:

1
=

2
H
A
:

1
<

2
Thefollowingtablesandfiguresanalyzethetestscoresandotherdatacollectedfromthe
userstoeitherrejectornotrejectthenullhypothesisofnodifferenceinparticipantslearning.
Assumingtheparticipantswillimprovetheiroverallscorefromthepretesttotheposttest,we
presentthealternativehypothesis:thepretestwillhavealoweraveragepercentagethanthe
posttestorH
A
:

1
<

2
.
Table1displaysthepreandposttestscoresforallelevenparticipants.Allusers
improvedtheirtestscoresafterparticipatinginthelessonwithameanof34.3%improvement.
Participants1and10showedtheleastimprovement(+11.2%)andparticipant3showedagreat
dealofimprovement(+67%).Figure1comparestheparticipantspretestscoreswiththeir
posttestscoresinpercentage.Thebargraphdisplaysthepreandposttestforeachindividual
user.Figure2isahistogramcomparing,asagroup,theparticipantspreandposttestscores,
showinganoverallimprovementinalloftheparticipants

EFFECTIVENESSANDUSABILITYREPORT
11

Table1
UsersPretestandPosttestScores

Figure1
.ParticipantPreandPosttestScores

EFFECTIVENESSANDUSABILITYREPORT
12

Figure2
.ComparisonofPreandPosttestScores
Consideringthatthenullhypothesis:pretestscore=posttestscoreandthealternative
hypothesis:pretestscore<posttestscore,attestwasusedtoanalyzetheabovedatatoreview
thesignificanceoftheresults.Morespecifically,apairedsamplettest(Table2)wasused
becausethetwosetsofdataaredependent:beforeandafterthelessonisadministered.Ifthet
StatisgreaterthanthetCritical,thenonecanrejectthenullhypothesis.ThetStatvalueis6.50
andthetCriticalvalueisat1.81,consideringaonetailtestduetothedifferencebeingequalor
greaterthanzero.Ignoringanynegativesymbols,thetStatisgreaterthanthetCriticalvalue
(6.50>1.82).Additionally,thepValueissmallerthan5%(0.05)at3.47E05or0.0035%.We
canrejectthenullhypothesiswitha95%statisticalsignificance.Inthiscase,wecanrejectthe
nullhypothesisofnodifferenceinlearningwitha99.99%statisticalsignificance.

EFFECTIVENESSANDUSABILITYREPORT
13

Table2
tTestPairedTwoSampleforMeans

Accordingtofeedbackfromtheparticipantsthroughtheusabilitysurvey,thelesson
overallwasdeemedanenjoyablelearningexperience.Manyparticipantsstatedtheyenjoyedthe
interactivityandfounditaddedtothelesson.Havingthelessonavailableasaselfpacedlearning
modulewasanotherpositiveformanyofthelearners,addingflexibilityandconvenience.
Whenconsideringparticipantsinterestlevel,itisinterestingtonotethatonly36%of
participantsstatedtheywereinterestedinspacesciencespecificallySun,Moon,andEarth
inthepretest,however90%ofparticipantsreportedtheyfoundthecontentwithinthelesson
interesting.EquallynoteworthywasparticipantsopinionsoftheiroverallMoonknowledge,
with100%ofparticipantsstatingtheyaremoreconfidentintheirMoonknowledgenow,than
priortocompletingthelesson.

EFFECTIVENESSANDUSABILITYREPORT
14

Recommendations
Thefollowingarerecommendationsforlessonimprovementbasedonparticipant
feedbackintheparticipantusabilitysurveyandadministratorobservations.
Highlightthecreditsbuttononthefinalresultsslidetosignalthelearnertoproceed.
Of
thesixobservedparticipants,onlytworealizeditwasnecessarytoclickthecreditsbuttononthe
finalresultspagetoproceedtotheendofthelesson,andbothofthoseparticipantsdidsoonly
afterwaitingashorttimebeforeunderstandinganadditionalactionwasnecessarytomove
forward.Althoughtheposttestwasspecificallycreatedforthistryouttotesttheeffectivenessof
thelesson,ifoneisincludedinfutureversions,itwouldbehelpfultotheusertohaveacue,such
ashighlightingthecreditsbuttonwhichleadstothepostassessmentlink.
Correctinformationingeosynchronousorbitcomponentofthelesson.

Itwasbroughtto
theattentionoftheadministratorsbytwoparticipantsthattheportionofthelessondiscussingthe
orbitoftheMoondescribedasgeosynchronousorbitinthelessonisinaccurate.Upon
furtherinvestigation,itwasfoundthatgeosynchronousorbitdoesnotaccuratelydescribethe
orbitoftheMoon,andthusitshouldbechangedwithinthelesson.
Changethedoubleclicktriggerwithinthefirstassessmenttosingleclick.
Itwasreported
throughanopenendedquestionintheusabilitysurveythatparticipantsfelttherequirementto
doubleclicktheMoonphasesintheassessmentinordertotriggerthenextquestionwas
unnecessaryandcouldpossiblybeconfusingtosomeusers.Thequestioninteractioniseffective
byhavingthelearnerclickthecorrectimage,soitisrecommendedthatthesettingischangedto
asingleclick.

EFFECTIVENESSANDUSABILITYREPORT
15

IncludeacuetoremindthelearnertoclickthenextMoonphaseinthefirstpartofthe
lesson.
Duringtwooftheobservations,theadministratorsnoticedthelearnershesitatedatthe
MoonmenuaftertheFullMooninstruction,lettingsometimepassbeforerememberingthey
weretoclickthenextMoonphasetoproceed.Anotherlearneralsomentionedintheparticipant
usabilitysurveythatitwouldbehelpfultohaveanadditionalprompttoremindthelearnerto
clickthenextMoonphasetocontinueoraddafeaturetoautomaticallyadvanceafterashort
periodoftimehaspassed.Consideringinteractivityisoneofthetargetedfeaturesofthislesson,
itisrecommendedthatanadditionalprompt,suchasflashingorasound,beinsertedafterthe
FullMooninstruction,remindingthelearnerofhowtoproceed.

EFFECTIVENESSANDUSABILITYREPORT
16

Summary
ThegoalofthistryoutwastodeterminetheeffectivenessandusabilityoftheMoon
lesson.Throughcarefulanalysisofthepreandposttestresults,itwasdeterminedthat
significantlearninggainswereobtainedbyallofthetryoutparticipants,showingthatthelesson
isindeedaneffectivewaytoteachthephasesoftheMoon.Throughobservationsandparticipant
feedbackintheparticipantusabilitysurvey,itwasfoundthatthelessonwasoverallintuitiveand
easyforthelearnerstonavigate,aswellasanenjoyablelearningexperience.Basedon
observationsandparticipantfeedback,someminorchangeswererecommendedwithinthis
reportandonerecommendationofachangeincontent.Overall,participantshaveclaimedto
enjoythelessonandwereimpressedwiththeamountofinteractivityandlearningthattook
place.Althoughnotcommunicatedorallyorthroughtheusabilitytest,observedusersshowed
excitementwhentheyviewedtheirresults,likelystimulatingtheirinteresttocontinue.

EFFECTIVENESSANDUSABILITYREPORT
17

References
Barbera,E.(Ed.).(2013).
Assessmentandevaluationoftimefactorsinonlineteachingand
learning
.IGIGlobal.p.96
Clark,R.C.,&Mayer,R.E.(2011).
Elearningandthescienceofinstruction:Proven
guidelinesforconsumersanddesignersofmultimedialearning
.JohnWiley&Sons.
Dukette,D.&CornishD.(2009).
TheEssential20:TwentyComponentsofanExcellentHealth
CareTeam
.RoseDogBooks.p.73

EFFECTIVENESSANDUSABILITYREPORT
18

AppendixA

EFFECTIVENESSANDUSABILITYREPORT
19

EFFECTIVENESSANDUSABILITYREPORT
20

EFFECTIVENESSANDUSABILITYREPORT
21

EFFECTIVENESSANDUSABILITYREPORT
22

AppendixB

EFFECTIVENESSANDUSABILITYREPORT
23

EFFECTIVENESSANDUSABILITYREPORT
24

EFFECTIVENESSANDUSABILITYREPORT
25

EFFECTIVENESSANDUSABILITYREPORT
26

AppendixC

EFFECTIVENESSANDUSABILITYREPORT
27

EFFECTIVENESSANDUSABILITYREPORT
28

EFFECTIVENESSANDUSABILITYREPORT
29

EFFECTIVENESSANDUSABILITYREPORT
30

AppendixD

EFFECTIVENESSANDUSABILITYREPORT
31

EFFECTIVENESSANDUSABILITYREPORT
32