PISA 2012 FIELD TRIAL PROBLEM SOLVING FRAMEWORK

DRAFT SUBJECT TO POSSIBLE REVISION AFTER THE FIELD TRIAL

Doc: ProbSolvFrmwrk_FT2012

30 September 2010

Consortium: Australian Council for Educational Research (ACER, Australia) cApStAn Linguistic Quality Control (Belgium) Deutsches Institut fr Interup to nationale Pdagogische Forschung (DIPF, Germany) Educational Testing Services (ETS, USA) Institutt for Lrerutdanning og Skoleutvikling (ILS, Norway) Leibniz - Institute for Science Education (IPN, Germany) National Institute for Educational Policy Research (NIER, Japan) The Tao Initiative: CRP - Henri Tudor and Universit de Luxembourg - EMACS (Luxembourg) Unit d'analyse des systmes et des pratiques d'enseignement (aSPe, Belgium) Westat (USA)

FrameworkforPISA2012ProblemSolving 30September2010 TABLEOFCONTENTS

PREAMBLE................................................................................................................................... 5 Chapter1:INTRODUCTION........................................................................................................... 7 ProblemSolvinginPIAAC................................................................................................................. 9 Chapter2:DEFININGTHEDOMAIN.............................................................................................11 ScopeoftheAssessment................................................................................................................ 5 1 Chapter3:ORGANISINGTHEDOMAIN........................................................................................16 a. ProblemContext................................................................................................................... 6 1 b. NatureofProblemSituation................................................................................................. 7 1 InteractiveProblemSituations..................................................................................................... 8 1 StaticProblemSituations............................................................................................................. 9 1 IlldefinedProblems..................................................................................................................... 9 1 c. ProblemSolvingProcesses.................................................................................................... 9 1 ReasoningSkills............................................................................................................................ 1 2 Chapter4:ASSESSINGPROBLEMSOLVINGCOMPETENCY...........................................................23 a. StructureoftheAssessment................................................................................................. 3 2 FunctionalityProvidedbyComputerDelivery.............................................................................. 3 2 b. TaskCharacteristicsandDifficulty........................................................................................ 4 2 ResponseFormatsandCoding..................................................................................................... 5 2 InteractiveProblems.................................................................................................................... 6 2 c. DistributionofItems............................................................................................................. 7 2 Chapter5:REPORTINGPROBLEMSOLVINGCOMPETENCY..........................................................29 Chapter6:SAMPLETASKS..........................................................................................................30 REFERENCES...............................................................................................................................31 ANNEXA: OVERVIEWOFPROBLEMSOLVINGRESEARCH.........................................................35 HistoricalandTheoreticalFoundations.......................................................................................... 5 3 EarlyConceptions......................................................................................................................... 5 3 Associationism............................................................................................................................. 5 3 GestaltPsychology....................................................................................................................... 6 3 GeorgePolya................................................................................................................................ 6 3 InformationProcessing................................................................................................................ 7 3 CurrentLinesofResearchonProblemSolving............................................................................... 7 3 DecisionMaking........................................................................................................................... 7 3 Reasoning .................................................................................................................................... 9 . 3 IntelligenceandCreativity........................................................................................................... 9 3 TeachingofThinkingSkills........................................................................................................... 0 4 ExpertProblemSolving................................................................................................................ 0 4 ThinkingbyAnalogy..................................................................................................................... 1 4 MathematicalandScientificProblemSolving............................................................................. 1 4 SituatedCognition ....................................................................................................................... 2 . 4 CognitiveNeuroscienceofProblemSolving................................................................................. 2 4 ComplexProblemSolving............................................................................................................. 3 4 Conclusions..................................................................................................................................... 4 4 References...................................................................................................................................... 6 4
3

ANNEXB: PROBLEMSOLVINGEXPERTGROUP........................................................................49

FrameworkforPISA2012ProblemSolving 30September2010
PREAMBLE 1. Thisdocumentpresentstherecommendedframeworkforassessmentofproblem solvinginthePISA2012fieldtrial.Theframeworkwillbefinetunedforthemain surveyfollowingconsiderationoftheoutcomesofthefieldtrial,andsampleitems fromthefieldtrialwillbeincludedasadditionalexamples.Thefollowingparagraphs describethedevelopmentoftheframeworktothisstage. 2. AfirstdraftoftheProblemSolvingFrameworkwasconsideredatthefirstProblem SolvingExpertGroup(PEG)meetingheldinMelbournefrom1012February2010.A seconddraftwaspreparedbyConsortiumstaffimmediatelyfollowingthePEG meetingforpresentationtotheNationalProjectManagersmeetingheldinHongKong from15March.FeedbackreceivedattheHongKongNPMmeeting,togetherwith extensivewrittenfeedbackfromPEGmembers,wasusedtoprepareathirddraft intendedforconsiderationatthePISAGoverningBoardmeetingwhichwastobeheld inCopenhagenfrom1921April2010. 3. WiththegroundingofEuropeanflightsduetovolcanicashcloud,theCopenhagen meetingwascancelledandreplacedwithawrittenconsultationonissuesthatwereto besettledatthemeeting.Asaconsequence,commentsfromPGBmembersonthe ProblemSolvingFrameworkwerenotavailablewhenthefourthdraftwasprepared forconsiderationatthePEGmeetingheldinBostonfrom2123June2010.However, thefourthdraftdidincorporatefeedbackonthethirddraftfromPEGmembersand testdevelopers.Onlyasmallnumberofchangesweremadetothepreviousdraftbut somecommentsrelatingtounresolvedmatters,andtonewissuesraisedbyPEG members,wereincludedfordiscussioninBoston. 4. FeedbackfromPGBmembersonthefourthdraftwasconsideredbythePEGatits Bostonmeeting,alongwithrecommendationsfromthePISAStrategicDevelopment Group(SDG)presentedbyEugeneOwen.FollowingthePEGmeeting,Consortiumstaff preparedafifthdraftreflectingdecisionsmadeinresponsetothePGBandSDG feedbackandthecommentscontainedinthefourthdraft.Thisfifthdraftwas circulatedtoPEGmembersforreviewandtheirfeedbackwasincorporatedinthe versionsubmittedtotheOECDatthestartofAugust.Thepresentversionincorporates changesagreedtoatthePEGmeetingheldinBudapestfrom2729September.
5

CHAPTER1:INTRODUCTION 1. Problemsolvingcompetencyisacentralobjectivewithintheeducational programmesofmanycountries.Theacquisitionofincreasedlevelsofproblem solvingcompetencyprovidesabasisforfuturelearning,foreffectiveparticipation insocietyandforconductingpersonalactivities.Studentsneedtobeabletoapply whattheyhavelearnedtonewsituations.Thestudyofindividualsproblem solvingstrengthsprovidesawindowontheircapabilitiestoemploybasic thinkingandothergeneralcognitiveapproachestoconfrontingchallengesinlife (Lesh&Zawojewski,2007). 2. ProblemSolvingwasanadditionalassessmentdomaininPISA2003.Somekey findingsofthesurveywereasfollows(OECD,2004): Insomecountries70%ofstudentscouldsolverelativelycomplexproblems, whileinotherslessthan5%coulddoso. Inmostcountries,morethan10%ofstudentswereunabletosolvebasic problems. OnaverageinOECDcountries,halfofthestudentswereunabletosolve problemsthataremoredifficultthanbasicproblems. Patternsof withincountryvariation instudents problemsolvingproficiency differedconsiderablyacrosscountries. Patternsofwithincountrydifferencesbetweenproblemsolvingproficiency anddomainrelatedproficiencies(mathematics,reading,science),differed considerablyacrosscountries. 3. Sincethe2003problemsolvingassessmentframework(OECD,2003a)was developed,considerableresearchhasbeencarriedoutintheareasofcomplex problemsolving,transfer,computerbasedassessmentofproblemsolving,and largescaleassessmentofproblemsolvingcompetency(e.g.Blech&Funke,2005; Funke&Frensch,2007;Greiff&Funke,2008;Klieme,2004;Klieme,Leutner,& Wirth,2005;Leutner,Klieme,Meyer,&Wirth,2004;Mayer,2002;Mayer& Wittrock,2006;ONeil,2002;Osman,2010;Reeff,Zabal&Blech,2006;Wirth& Klieme,2004).Thisresearchhasledtoadvancesinunderstandingandmeasuring individualsproblemsolvingcapabilities.

4. Inaddition,advancesinsoftwaredevelopmenttoolsandtheuseofnetworked computershavemadepossiblegreaterefficiencyandeffectivenessofassessment, includingthecapabilitytoadministerdynamicandinteractiveproblems,engage studentsinterestmorefullyandcapturemoreinformationaboutthecourseofthe problemsolvingprocess.Onthislastpoint,computerdeliveryofassessmenttasks makesitpossibletorecorddataaboutsuchthingsasthetype,frequency,length andsequenceofactionsperformedbystudentsinrespondingtoitems. 5. Itisappropriate,therefore,toonceagainmakeproblemsolvinganassessment domaininPISA,butindoingsotodeviseanewframeworkandimplement additionalassessmentmethodologiesthatallowfortherealtimecaptureof studentscapabilities.Inparticular,thePISA2012assessmentofproblemsolving willbecomputerbasedandinteractivityofthestudentwiththeproblemwillbea centralcomponentoftheinformationgathered. 6. PISA2012problemsolvingisanassessmentofindividualproblemsolving competency.Collaborativeproblemsolvingskillstheskillsrequiredtosolve problemsasamemberofagroupareessentialforsuccessfulemployment,where theindividualisoftenamemberofateamofdiversespecialistsworkingin separatelocations.However,significantmeasurementchallengesstillstandinthe wayofcollaborativetasksbecomingafeatureoflargescaleinternationalsurveys suchasPISA(Reeff,Zabal&Blech,2006).Foremostamongthesechallengesare howtoassigncredittoindividualgroupmembersifthisisrequired,howto accountfordifferencesacrossgroupsthatmaybiasindividualperformance,and howtoaccountforculturaldifferencesingroupdynamics. 7. Aconsistentresearchfindingisthatproblemsolvingisdependentondomain specificknowledgeandstrategies(e.g.Mayer,1992;Funke&Frensch,2007) 1 .The PISA2012assessmentwillavoidtheneedforexpertpriorknowledgeasmuchas possibleinordertofocusonmeasuringthecognitiveprocessesfundamentalto problemsolving.Thisalsodistinguishestheassessmentfromproblemsolving tasksinthecorePISAliteracydomainsofreading,mathematicsandscience,which callonexpertknowledgeintheseareas. 8. Anotherconclusionthatcanbedrawnfromtheresearchoverviewisthat authentic,relativelycomplexproblems,particularlythosethatrequiredirect interactionbythesolvertouncoveranddiscoverrelevantinformation,shouldbea
1AnoverviewofscientificresearchintoindividualproblemsolvingisincludedasAnnexA.This

reviewneednotnecessarilybeincludedinthefinalframeworkpublication. 8

centralfeatureofthePISA2012problemsolvingassessment.Examplesarethe problemscommonlyfacedwhenusingunfamiliareverydaydevicessuchasremote controls,personaldigitaldevices(e.g.mobilephones),homeappliancesand vendingmachines.Otherexamplesariseinsituationssuchasphysical conditioning,feedinganimals,growingplantsandsocialinteractions.Problem solvingskillsarenecessarytoachievemorethanabasiclevelofskillindealing withsuchsituationsandthereisevidencethatskillsadditionaltothoseinvolvedin traditionalreasoningbasedproblemsolvingarerequired(e.g.Klieme,2004).This willbethefirsttime,madepossiblebycomputerdeliveryoftheassessment,that suchinteractiveproblemshavebeenincludedinalargescaleinternational survey. 9. Problemsolvingcompetencycanbedevelopedbyhighqualityeducation. Progressiveteachingmethods,likeproblembasedlearning,inquirybased learning,andindividualandgroupprojectwork,canbeusedtofosterdeep understandingandpreparestudentstoapplytheirknowledgeinnovelsituations. Goodteachingpromotesselfregulatedlearningandmetacognitionanddevelops thecognitiveprocessesthatunderpinproblemsolving.Itpreparesstudentsto reasoneffectivelyinunfamiliarsituations,andtofillgapsintheirknowledgeby observation,explorationandinteractionwithunknownsystems.ThePISA2012 computerbasedassessmentofproblemsolvingaimstoexaminehowstudentsare preparedtomeetunknownfuturechallengesforwhichdirectteachingoftodays knowledgeisnotsufficient. ProblemSolvinginPIAAC 10. TheOECDsProgrammefortheInternationalAssessmentofAdultCompetencies (PIAAC),isanassessmentofreadingcomponentskills,literacy,numeracy,and problemsolvingintechnologyrichenvironments.Itisafacetofacesample householdsurveyofpeopleaged1665yearsandwillbeconductedforthefirst timein2012. 11. PIAACsassessmentofproblemsolvingintechnologyrichenvironmentsdiffers fromthePISA2012assessmentofproblemsolvingintwoimportantaspects 2 . Firstly,itisprimarilyconcernedwithinformationrichproblems.Examples includeneedingtolocateandevaluateinformationontheWeboronsocial
2PIAACdefinesproblemsolvingintechnologyrichenvironmentsasfollows:Problemsolvingin

technologyrichenvironmentsinvolvesusingdigitaltechnology,communicationtoolsandnetworks toacquireandevaluateinformation,communicatewithothersandperformpracticaltasks.(OECD, March2009,p.7) 9

networkingsites,navigatingthroughunfamiliarwebpagesandmakingdecisions aboutwhatinformationisrelevantandirrelevantforatask. 12. Asecondmajordifferenceisthatproblemsolutionsrequiretheuseofoneormore computersoftwareapplications(filemanagement,webbrowser,email,and spreadsheet).InPISA,ICTisintegraltotheassessmentofproblemsolvingbutitis notintegraltoitsdefinitionofproblemsolving.OnlyfoundationalICTskills(based onuseofakeyboardandmouse)arenecessarytoundertakethePISAassessment ofproblemsolving.Softwaretoolsarecommonandpowerfulaidsforinformation richproblemsolvingandahighlevelofICTliteracyisessentialinthisdigitalage. However,PISAsfocusisassessmentofthefundamentalcognitiveprocessesthat areessentialforsuccessfulproblemsolvingwithorwithoutICTassistance.

10

CHAPTER2:DEFININGTHEDOMAIN 13. TheaimofthePISA2012problemsolvingassessmentistoassessindividual problemsolvingcompetency.Beforedefiningwhatismeantbythetermproblem solvingcompetencyinthiscontext,itisimportanttoclarifywhatismeantbythe termsproblemandproblemsolvingbyresearchersinthefield. 14. Aproblemexistswhenapersonhasagoalbutdoesntknowhowtoachieveit (Duncker,1945).ThisdefinitionisenlargeduponinFigure1.Thegivenstate (givens)istheknowledgethepersonhasabouttheproblemattheoutsetandthe operatorsaretheadmissibleactionsthatcanbeperformedtoachievethedesired goalstate(outcomes)withtheassistanceoftheavailabletools.Barriersthatmust beovercome(e.g.lackofknowledgeorobviousstrategies)standinthewayof achievingthegoal.Overcomingthebarriersmayinvolvenotonlycognition,but motivationalandaffectivemeans(Funke,2010). Figure1.ProblemSituation(afterFunke&Frensch,1995)
GIVENS GOAL

Barriers

OPERATORS & TOOLS

15. Asanexample,considerthesimpleproblemoffindingtheroutebetweentwo townsthatislikelytobethequickest,givenaroadmapwithestimatedtravel timesmarkedandacalculator.Thegivenstateisthegiveninformationthemap withnoroutemarkedandthegoalstateisthedesiredanswerthequickest route.Theallowableactions(operators)areselectingapossibleroute,calculating itstotaltimeandcomparingitwiththetimesforotherroutes.Atool(calculator) isavailableforassistanceinaddingtimes. 16. Consistentwiththisunderstandingofwhatismeantbyaproblem,Mayer(1990) definesproblemsolvingascognitiveprocessingdirectedattransformingagiven situationintoagoalsituationwhennoobviousmethodofsolutionisavailable. Thisdefinitioniswidelyacceptedintheproblemsolvingcommunity(e.g.see Klieme,2004;Mayer&Wittrock,2006;Reeff,Zabal&Blech,2006).

11

17. ThePISA2012definitionofproblemsolvingcompetencyisgroundedinthese generallyacceptedmeaningsofproblemandproblemsolving.Itisasfollows: Problemsolvingcompetencyisanindividualscapacitytoengageincognitive processingtounderstandandresolveproblemsituationswhereamethodof solutionisnotimmediatelyobvious.Itincludesthewillingnesstoengagewith suchsituationsinordertoachieveonespotentialasaconstructiveandreflective citizen. 18. Notsurprisingly,thefirstsentenceofthedefinitionisalmostidenticaltothefirst partofthedefinitionusedforthePISA2003assessmentofproblemsolving 3 . However,whereasthe2003definitionhadonlyacognitivedimension,withits latterparthighlightingthecrosscurricularnatureoftheassessment,anaffective componentisintroducedinthe2012definitioninkeepingwiththedefinitionof competencyasrecognisedbytheOECD(OECD,2003a). 19. Whatdistinguishesthe2012assessmentofproblemsolvingfromthe2003 assessmentisnotsomuchthedefinitionofproblemsolvingcompetency,butthe modeofdelivery(computerbased)ofthe2012assessmentandtheinclusionof problemswhichcannotbesolvedwithoutthesolverinteractingwiththeproblem situation. 20. Inthefollowingparagraphs,eachpartofthePISA2012definitionofproblem solvingcompetencyisconsideredinturntohelpclarifyitsmeaninginrelationto theassessment. Problemsolvingcompetency 21. Acompetencyinvolvesfarmorethanthebasicreproductionofaccumulated knowledge.Itinvolvesamobilisationofcognitiveandpracticalskills,creative abilitiesandotherpsychosocialresourcessuchasattitudes,motivationandvalues (OECD,2003b).ThePISA2012assessmentofproblemsolvingcompetencywillnot testsimplereproductionofdomainbasedknowledge;ratheritwillfocusonthe cognitiveskillsrequiredtosolveunfamiliarproblemsencounteredinlife 4 and lyingoutsidetraditionalcurriculardomains.

3Problemsolvingisanindividualscapacitytousecognitiveprocessestoconfrontandresolve

real,crossdisciplinarysituationswherethesolutionpathisnotimmediatelyobviousandwhere theliteracydomainsorcurriculaareasthatmightbeapplicablearenotwithinasingledomainof mathematics,scienceorreading(OECD,2003a,p.156). 4Includingeducationalandprofessionalencounters. 12

22. Priorknowledgeisimportantinsolvingproblems.However,problemsolving competencyinvolvestheabilitytoacquireandusenewknowledge,ortouseold knowledgeinanewway,tosolvenovelproblems(i.e.problemsthatarenot routine). isanindividualscapacitytoengageincognitiveprocessing 23. Problemsolvingoccursinternallyinanindividualscognitivesystemandcanonly beinferredindirectlybythepersonsactionsandproducts.Itinvolves representingandmanipulatingvarioustypesofknowledgeintheproblemsolvers cognitivesystem(Mayer&Wittrock,2006).Studentsresponsestoassessment itemstheirexplorationstrategies,therepresentationstheyemployinmodelling theproblem,numericalandnonnumericalanswers,orextendedexplanationsof howaproblemwassolvedwillbeusedtomakeinferencesaboutthecognitive processestheyemployed. 24. Creative(divergent)thinkingandcriticalthinkingareimportantcomponentsof problemsolvingcompetency(Mayer,1992).Creativethinkingisacognitive activitythatresultsinfindingsolutionstoanovelproblem.Criticalthinking accompaniescreativethinkingandisemployedtoevaluatepossiblesolutions.The assessmentwilltargetbothcomponents. tounderstandandresolveproblemsituations 25. Towhatdegreecanindividualsmeetthechallengesofaproblemsituationand movetowardsresolvingit?Inadditiontoexplicitresponsestoitems,the assessmentaimstomeasureindividualsprogressinsolvingaproblem,including thestrategiestheyemploy.Whereappropriatethesestrategiescanbetrackedby meansofbehaviouraldatacapturedbythecomputerdeliverysystem:thetype, frequency,lengthandsequenceofinteractionsmadewiththesystemcanbe capturedandusedinscoringorinsubsequentanalysesofstudentperformance. 26. Problemsolvingbeginswithrecognisingthataproblemsituationexistsand establishinganunderstandingofthenatureofthesituation.Itrequiresthesolver toidentifythespecificproblem(s)tobesolvedandtoplanandcarryouta solution,alongwithmonitoringandevaluatingprogressthroughouttheactivity. 27. Ofteninrealworldproblemstheremaynotbeauniqueorexactsolution.In addition,theproblemsituationmaychangeduringthesolvingprocess,possibly duetointeractionwiththeproblemsolverorasaresultofitsowndynamic
13

nature.Thesecomplexitieswillbeaddressedwhensettingassessmenttasksanda balancestruckbetweenauthenticityofasituationandpracticalityofassessment. whereamethodofsolutionisnotimmediatelyobvious. 28. Themeansoffindingasolutionpathshouldnotbeimmediatelyobvioustothe problemsolver.Therewillbebarriersofvarioussortsintheway,ormissing information.Theassessmentisconcernedwithnonroutineproblems,notroutine ones(i.e.problemsforwhichapreviouslylearnedsolutionprocedureisclearly applicable):theproblemsolvermustactivelyexploreandunderstandtheproblem andeitherdeviseanewstrategyorapplyastrategylearntinadifferentcontextto worktowardsasolution. 29. Thestatusofaproblemwhetheritisroutineornotdependsonthesolvers familiaritywiththeproblem.Aproblemforonepersonmayhaveanobvious solutiontoanotherpersonwhoisexperiencedandpractisedwithsuchproblems. Accordingly,carewillbetakentosetproblemsthatshouldbenonroutineforthe greatmajorityof15yearolds. 30. Itisnotnecessarilythecasethatthecontextorgoalswillthemselvesbeunfamiliar tothesolver;whatisimportantisthattheparticularproblemsarenovelorthe waysofachievingthegoalsarenotimmediatelyobvious.Theproblemsolver mightneedtoexploreorinteractwiththeproblemsituationbeforeattemptingto solvetheproblem.Directinteractionismadefeasiblebytheuseofcomputer deliveredassessmentinPISA2012. Itincludesthewillingnesstoengagewithsuchsituations 31. Problemsolvingispersonalanddirected,thatis,theproblemsolversprocessingis guidedbytheirpersonalgoals(Mayer&Wittrock,2006).Theproblemsolvers individualknowledgeandskillshelpdeterminethedifficultyoreasewithwhich obstaclestosolutionscanbeovercome.However,theoperationofsuchknowledge andskillisaffectedbymotivationalandaffectivefactorssuchasbeliefs(e.g.self confidence)andfeelingsaboutonesinterestandabilitytosolvetheproblem (Mayer,1998). 32. Inaddition,thecontextofaproblem(whetheritisfamiliarandunderstood),the externalresourcesavailabletothesolver(suchasaccesstotools),andthe environmentinwhichthesolveroperates(e.g.anexaminationsetting)willaffect thewayapersonapproachesandengageswiththeproblem.
14

33. Motivationandaffectivefactorswillnotbemeasuredintheproblemsolving assessmentbutsomemaybeaddressedgenerally,orwithparticularreferenceto mathematics,inthestudentquestionnaire. inordertoachieveonespotentialasaconstructiveandreflectivecitizen. 34. Competencyisanimportantfactorinthewaysthatindividualshelptoshapethe world,notjusttocopewithit:keycompetenciescanbenefitbothindividuals andsocieties(Rychen&Salganik,2003).Theyshouldmanagetheirlivesin meaningfulandresponsiblewaysbyexercisingcontrolovertheirlivingand workingconditions(ibid).Individualsneedtobeproficientproblemsolversto achievetheirpotentialasconstructive,concernedandreflectivecitizens. ScopeoftheAssessment 35. ThePISA2012assessmentofproblemsolvingwillnotincludeproblemsthat requireexpertpriorknowledgefortheirsolution.Inparticular,problemsthat couldreasonablybeincludedinanassessmentofoneofthethreecorePISA domainswillnotbeincluded.Assessmenttaskswillcentreoneverydaysituations, withawiderangeofcontextsemployedasameansofcontrollingforprior knowledgeingeneral. 36. Mobilisationofpriorknowledgeisnotsufficienttosolvenovelproblemsinmany everydaysituations.Gapsinknowledgemustbefilledbyobservationand explorationoftheproblemsituation.Thisofteninvolvesinteractionwithanew systemtodiscoverrulesthatinturnmustbeappliedtosolvetheproblem.Instead ofastraightforwardapplicationofpreviouslymasteredknowledge,existing knowledgeneedstobereorganisedandcombinedwithnewknowledgeusinga rangeofreasoningskills.

15

CHAPTER3:ORGANISINGTHEDOMAIN 37. Howthedomainisrepresentedandorganiseddeterminestheassessmentdesign and,ultimately,theevidenceaboutstudentproficienciesthatcanbecollectedand reported.Manyelementsarepartoftheconstruct,notallofwhichcanbetaken intoaccountandvariedinanassessmentsuchasPISA.Itisnecessarytoselectthe mostimportantelementsthatcanbevariedtoensureconstructionofan assessmentthatcontainsitemswhichhaveanappropriaterangeofdifficultyand provideabroadcoverageofthedomain. 38. ThedomainelementsofkeyimportanceforthePISA2012assessmentofproblem solvingareasfollows: Theproblemcontext:whetheritinvolvesatechnologicaldeviceornot,and whetherthefocusoftheproblemispersonalorsocial. Thenatureoftheproblemsituation:whetheritisinteractiveorstatic. Theproblemsolvingprocesses:thecognitiveprocessesinvolvedinsolvinga problem. 39. Taskswillbeconstructedtomeasurehowwellstudentsperformwhenthevarious cognitiveprocessesinvolvedinproblemsolvingareexercisedwithinthetwo differenttypesofproblemsituationsacrossarangeofcontexts.Eachofthesekey domainelementsisdiscussedandillustratedinthefollowingsections.The characteristicsofthetasksthemselvesarediscussedinChapter4. a. ProblemContext 40. Anindividualsfamiliarityandunderstandingoftheproblemcontextwillaffect howdifficulttheproblemistosolveforthatperson.Twodimensionshavebeen identifiedtoensurethatassessmenttaskssampleacrossarangeofcontextsthat areauthenticandofinterestto15yearolds:thesetting(technologyornot)and thefocus(personalorsocial). 41. Problemssetinatechnologycontexthavethefunctionalityofatechnological deviceastheirbasis.Examplesincludemobilephones,remotecontrolsfor appliancesandticketvendingmachines.Knowledgeoftheinnerworkingsofthese deviceswillnotberequired:typically,problemsolverswillbeledtoexploreand understandthefunctionalityofadevice,aspreparationforcontrollingthedevice orfortroubleshootingitsmalfunctioning.Situationsthatgiverisetoothertypesof
16

problems,suchasrouteplanning,taskschedulinganddecisionmaking,havenon technologycontexts. 42. Personalcontextsincludethoserelatingprimarilytotheself,familyandpeer groups.Socialcontextsrelatetosituationsencounteredinthecommunityor societyingeneral.Asillustrations,thecontextofanitemaboutsettingthetimeon adigitalwatchwouldbeclassifiedasTechnologyandPersonal,whereasthe contextofanitemrequiringtheconstructionofabasketballteamrosterwouldbe classifiedasNontechnologyandSocial. b. NatureofProblemSituation 43. Howaproblemispresentedhasimportantconsequencesforhowitcanbesolved. Ofcrucialimportanceiswhethertheinformationabouttheproblemdisclosedto theproblemsolverattheoutsetiscomplete.Thisisthecaseforthequickestroute problemdiscussedearlier(seeparagraph15).Werefertosuchproblemsituations asbeingstatic. 44. Bycontrast,problemsituationsmaybeinteractive,meaningthatexplorationofthe situationtouncoveradditionalrelevantinformationispossible 5 .Realtime navigationusingaGPSsystem,wheretrafficcongestionisreportedautomatically orbyquery,presentssuchasituation. 45. Interactiveproblemsituationscanbesimulatedinatestsettingbymeansofa computer.IncludinginteractiveproblemsituationsinthecomputerbasedPISA 2012problemsolvingassessmentallowsawiderangeofmoreauthentic,reallife scenariostobepresentedthanwouldotherwisebepossibleusingpenciland papertests.Problemswherethestudentexploresandcontrolsasimulated environmentwillbeadistinctivefeatureoftheassessment. 46. Aselectionofstaticproblemsituationsalsowillbeincludedintheassessment.The assessmentofsuchproblemshastraditionallytakenplaceusingpencilandpaper tests.However,theircomputerbasedassessmenthasmanyadvantagesincluding thecapabilityofpresentingabroaderrangeofscenarios,involvingmultimedia elementssuchasanimation;theavailabilityofonlinetools;and,theuseofawide rangeofresponseformatsthatcanbeautomaticallycoded.

5Thetermintransparentissometimesusedtodescribeproblemswhencompleteinformation abouttheproblemsituationisnotavailableattheoutset(seeFunke&Frensch,1995).

17

47. Somestudiessuggestthatknowledgeacquisitioninexploringaprobleminan interactiveenvironment,andhowthatknowledgeisapplied,arecompetencies distinctfromthetypicalskillsusedinsolvingstaticproblems(seeKlieme,2004; Wirth&Klieme,2004;Leutner&Wirth,2005).Includingamixtureofinteractive andstaticproblemsinthePISA2012assessmentwillprovideabroadermeasure ofproblemsolvingcompetencythanhasbeenpossiblewithpencilandpaper instruments. InteractiveProblemSituations 48. Interactiveproblemsituationsoftenarisewhenencounteringrealworldartefacts suchasticketvendingmachines,airconditioningsystemsormobiletelephonesfor thefirsttime,especiallyiftheinstructionsforuseofsuchdevicesarenotclearor notavailable.Understandinghowtocontrolsuchdevicesisaproblemfaced universallyineverydaylife.Inthesesituationsitisoftenthecasethatsome relevantinformationisnotapparentattheoutset.Forexample,theeffectof applyinganoperation(say,pushingabuttononaremotecontrol)maynotbe knownandcannotbededuced,butrathermustbeinferredbyinteractingwiththe scenariothroughactuallyperformingtheoperation(pushingthebutton)and formingahypothesisaboutitsfunctionbasedontheoutcome.Ingeneral,some explorationorexperimentationmustbedonetoacquiretheknowledgenecessary tocontrolthedevice.Anothercommonscenarioiswhenapersonmust troubleshootafaultormalfunctioninadevice.Hereacertainamountof experimentationmusttakeplacetocollectdataonthecircumstancesunderwhich themachineryfails. 49. Aninteractiveproblemsituationcanbedynamic,meaningthatitsstatemight changeofitsownaccordduetoinfluencesbeyondtheproblemsolverscontrol(i.e. withoutanyinterventionbytheproblemsolver) 6 .Forexample,inthecaseofa ticketvendingmachine,ifduringatransactionnobuttonsarepressedfor20 secondsthemachinemightreset.Suchautonomousbehaviourofasystemmustbe observedandunderstoodsoitcanbetakenintoaccountinattainingthedesired goal(purchasingaticket).

6Thetermdynamicisusedbysomeresearcherstodescribeanysimulatedphysicalsystemthata

problemsolvercaninteractwithandreceivefeedback.Insuchcases,problemsituationsthat changeautonomouslyaresometimestermedeigendynamic(e.g.seeBlech&Funke,2005). 18

StaticProblemSituations 50. Staticproblemsituationscangiverisetowelldefinedorilldefinedproblems.Ina welldefinedproblem,suchasthequickestrouteproblem(seeparagraph15),the givenstate,goalstateandallowableoperatorsareclearlyspecified(Mayer& Wittrock,2006).Theproblemsituationisnotdynamic(i.e.doesnotchangeofits ownaccordduringthecourseofsolvingtheproblem),allrelevantinformationis disclosedattheoutsetandthereisasinglegoal. 51. Otherexamplesofwelldefinedproblemsaretraditionallogicpuzzlessuchasthe TowerofHanoiandthewaterjarsproblems(see,forexample,Robertson,2001); decisionmakingproblems,wherethesolverisrequiredtounderstandasituation involvinganumberofwelldefinedalternativesandconstraintssoastomakea decisionthatsatisfiestheconstraints(e.g.choosingtherightpainkiller,given sufficientdetailsaboutthepatient,thecomplaintandtheavailablepainkillers); and,schedulingproblemsforprojectssuchasbuildingahouseorproducing computersoftware,wherealistoftaskswithdurationsanddependenciesbetween tasksisgiven. IlldefinedProblems 52. MayerandWittrock(2006)pointoutthateducationalmaterialsoftenemphasise welldefinedproblems,althoughmostrealproblemsareilldefined[i.e.notwell defined].Theseproblemsofteninvolvemultiplegoalswhichareinconflictsothat progresstowardonemaydetractfromprogresstowardtheother(s).Elaboration andweighingofprioritiesisrequiredfortheproblemsolvertoachieveabalance betweenthegoals(Blech&Funke,2010).Anexampleisfindingthebestroute betweentwoplacesshouldthisbetheshortestroute?,thelikelyquickestroute?, themoststraightforwardroute?,theroutewithminimumvariationintime?,etc.A morecomplexexampleisdesigningacarwherehighefficiency,lowcost,high safetyandloweffectontheenvironmentaredesired. c. ProblemSolvingProcesses 53. Differentauthorsconceiveofthecognitiveprocessesinvolvedinsolvingaproblem indifferentways,butthereisagreatdealofcommonalityintheirviews.The processesidentifiedbelowarederivedfromtheworkonproblemsolvingand reasoningofcognitivepsychologists(e.g.,Baxter&Glaser,1997;Bransfordetal, 1999;Mayer&Wittrock,1996;Mayer&Wittrock,2006;Vosniadou&Ortony, 1989),aswellasbytheseminalworkofPolya(1945).Additionally,recentwork
19

oncomplexanddynamicproblemsolving(Blech&Funke,2005,2010;Funke& Frensch,2007;Greiff&Funke2008;Klieme,2004;Osman,2010;Reeff,Zabal& Blech,2006;Wirth&Klieme,2004)hasbeentakenintoaccount. 54. Noassumptionismadethattheprocessesinvolvedinsolvingaparticularproblem aresequentialorthatalloftheprocesseslistedarenecessaryinthesolutionofa particularproblem.Asindividualsconfront,structure,representandsolve authenticproblemsrepresentingemerginglifedemands,theymaymovetoa solutioninawaythattranscendstheboundariesofalinear,stepbystepmodel. Mostoftheinformationaboutthefunctioningofthehumancognitivesystemnow supportstheviewthatitiscapableofparallelinformationprocessing(Lesh& Zawojewski,2007). 55. ForthepurposesofthePISA2012problemsolvingassessment,theprocesses involvedinproblemsolvingaretakentobe: Exploringandunderstanding Representingandformulating Planningandexecuting Monitoringandreflecting

56. Exploringandunderstanding.Theobjectivehereistobuildmental representationsofeachofthepiecesofinformationpresentedintheproblem.This involves: exploringtheproblemsituation:observingit;interactingwithit;searchingfor information;findinglimitationsorobstacles;and understandinggiveninformationandinformationdiscoveredwhileinteracting withtheproblemsituation;demonstratingunderstandingofrelevant concepts. 57. Representingandformulating.Theobjectivehereistobuildacoherentmental representationoftheproblemsituation(i.e.asituationmodeloraproblem model).Todothis,relevantinformationmustbeselected,mentallyorganised,and integratedwithrelevantpriorknowledge.Thismayinvolve:

20

representingtheproblembyconstructingtabular,graphical,symbolicor verbalrepresentations,andshiftingbetweenrepresentationalformats;and

formulatinghypothesesbyidentifyingtherelevantfactorsintheproblemand theirinterrelationships;organisingandcriticallyevaluatinginformation.

58. Planningandexecuting.Thisincludes: planning,whichconsistsofgoalsetting,includingclarifyingtheoverallgoal, andsettingsubgoals,wherenecessary;anddevisingaplanorstrategyto reachthegoalstate,includingthestepstobeundertaken;and executing,whichconsistsofcarryingoutaplan.

59. Monitoringandreflectingincludes: monitoringprogresstowardsthegoalateachstage,includingchecking intermediateandfinalresults,detectingunexpectedevents,andtaking remedialactionwhenrequired;and reflectingonsolutionsfromdifferentperspectives;criticallyevaluating assumptionsandalternativesolutions;andlookingforadditionalinformation orclarification. ReasoningSkills 60. Eachoftheproblemsolvingprocessesdrawsupononeormorereasoningskills.In understandingaproblemsituation,theproblemsolvermayneedtodistinguish betweenfactsandopinion;informulatingasolution,theproblemsolvermayneed toidentifyrelationshipsbetweenvariables;inselectingastrategy,theproblem solvermayneedtoconsidercauseandeffect;and,incommunicatingtheresults, theproblemsolvermayneedtoorganiseinformationinalogicalmanner.The reasoningskillsassociatedwiththeseprocessesareembeddedwithinproblem solving.TheyareimportantinthePISAcontextsincetheycanbetaughtand modelledinclassroominstruction(e.g.Adeyetal,2007;Klauer&Phye,2008). 61. Examplesofreasoningskillsemployedinproblemsolvingincludedeductive, inductive,quantitative,correlational,analogical,combinatorial,and multidimensionalreasoning.Thesereasoningskillsarenotmutuallyexclusiveand ofteninpracticeproblemsolversmovefromonetoanotheringatheringevidence andtestingpotentialsolutionpathsbeforesettlingintothemajoruseofone
21

methodoverothersinfindingthesolutiontoagivenproblem.Reasoningskillswill bebroadlysampledwhendevisingassessmentitemsastheyarelikelytoinfluence thedifficultyofitems.

22

CHAPTER4:ASSESSINGPROBLEMSOLVINGCOMPETENCY a. StructureoftheAssessment 62. Thedurationofthemainsurveyassessmentwillbe40minutes.Therewillbea totalof80minutesofmaterialorganisedintofour20minuteclusters,witheach studentdoingtwoclustersaccordingtoabalancedrotationdesign.Abouttwice thisamountofmaterialwillbeincludedinthefieldtrial.Itisestimatedthat2025 itemswillbeincludedinthemainsurvey,withaboutonethirdofthembeing partialcredititems.Timinginformationautomaticallycapturedduringthefield trialwillbeusedtodeterminetheactualnumberofitemsthatcanbeincluded. 63. AsisnormalforPISAassessments,itemswillbegroupedinunitsbasedarounda commonstimulusthatwilldescribetheproblemsituation.Tominimisethelevelof readingliteracyrequired,stimulusmaterial(andtaskstatements)willbeasclear, simpleandbriefaspossible.Animations,photographsordiagramswillbeusedto avoidlengthypassagesoftext.Numeracydemandsalsowillbekepttoaminimum with,forexample,runningtotalsprovidedwhereappropriate. 64. Theassessmentwillincludeabroadsampleofitemscoveringarangeofdifficulty thatwillenablethestrengthsandweaknessesofpopulationsandkeysubgroupsto bedeterminedwithrespecttothecognitiveprocessesinvolvedinproblemsolving. FunctionalityProvidedbyComputerDelivery 65. Aprincipalbenefitofmeasuringproblemsolvingcompetencybycomputeristhe opportunitytocollectandanalysedatathatrelatetoprocessesandstrategies,in additiontocapturingandscoringintermediateandfinalresults.Thisislikelytobe amajorcontributionofthePISA2012assessmentofproblemsolving.With appropriateitemauthoring,datasuchasthetype,frequency,lengthandsequence ofactionsperformedbystudentscanbecapturedforthispurpose. 66. Anotherbenefitisthatthetimestudentsspendonanyparticularitemcanbe monitoredandrestricted.Thisfacilitymaybeusedwhereitisconsidered appropriate,suchasinlimitingthetimestudentsareallowedtoexploreacomplex problemsituation(or,toputitanotherway,forlimitingthetimestudentsmay wasteinthismanner). 67. OnlyfoundationalICTskillswillbeassumedintheassessment,suchaskeyboard use,manipulatingapointer(e.g.amouse),clickingbuttons,draganddrop, scrolling,anduseofpulldownmenusandhyperlinks.Carewillbetakentoensure
23

thatinterferencewiththemeasurementofproblemsolvingcompetencybyICT demandandpresentationiskepttoaminimum. 68. Bothunitsanditemswithinunitswillbedeliveredinafixedorder,orlockstep fashion.Thelockstepproceduremeansthatstudentsarenotabletoreturntoan itemorunitoncetheyhavemovedtothenextone.Eachtimestudentsclickthe Nextbuttonadialogboxwilldisplayawarningthattheyareabouttomoveonto thenextitemandthatitwillnotbepossibletoreturntothepreviousitem.Atthis point,studentscaneitherconfirmtheywanttomoveonorcanceltheactionand returntothecurrentitem. b. TaskCharacteristicsandDifficulty 69. Generally,eachitemwillfocusonasingleproblemsolvingprocessasfaras possible.Accordingly,forsomeitems,demonstratingarecognitionoftheproblem willbesufficient;inothers,describingamethodofsolutionwillbeenough;in many,theactualsolution(s)willberequiredwitheffectivenessandefficiencyof methodbeingimportantcharacteristics;inyetothers,thetaskwillbetoevaluate proposedsolutionsanddecideonthemostappropriatesolutionfortheproblem posed.Althoughexecutingisoftenemphasisedinclassroominstruction,themajor difficultiesformostproblemsolversinvolverepresenting,planningandself regulating(Mayer,2003). 70. Someproblemsareinherentlymorecomplexthanothers(Funke&Frensch,2007). Furthermore,increasedcomplexitygenerallymeansgreaterdifficulty.Table1 summarisestaskcharacteristicsthatwillbevariedintheassessmenttoensure thattheitemscoveranappropriaterangeofdifficulty.

24

Table1.TaskCharacteristics Characteristic Amountofinformation Representationofinformation Effectontaskdifficulty Themoreinformationthathastobeconsidered,the moredifficultthetaskislikelytobe. Unfamilarrepresentations, andmultiple representations(especiallyiftheinformationhasto berelated),tendtoincreasedifficulty. Themorerelevantinformationthatisnotdisclosed attheoutsetandthereforehastobediscovered(e.g. effectofoperations,autonomousbehaviour, unanticipatedobstacles),themoredifficultthetask islikelytobe. Internalcomplexityofaproblemsituationincreases asthenumberofvariablesandtheirdependencies increases.Taskswithahighlevelofinternal complexityarelikelytobeharderthanthosewith lowerlevelsofinternalcomplexity. Tasksinwhichtherearefewconstaintsonthe valuesofvariablearelikelytobeeasierthanthose inwhichmanyconstraintsmustbesatisfied. Thegreaterthenumberof stepsneededtosolvea problem,themoredifficultitislikelytobe. Problemsthatrequiretheapplicationofsometypes ofreasoningskills(e.g.combinatorialreasoning)are likleytobeharderthanthosethatdonot.

Disclosureofinformation

Internalcomplexity

Constraintstobesatisfied

Distancetogoal Reasoningskillsrequired

ResponseFormatsandCoding 71. Atleast75%ofitemswillhavearesponseformatthatcanbeautomaticallycoded initscapturedform.Thiswillincludesimpleandcomplexmultiplechoiceitems thatareansweredbyclickingoption(radio)buttons,itemsthatrequireshapesto beselectedanddraggedintoposition,itemsinwhichselectionshavetobemade frompulldownmenustofilltablecells,anditemsthatrequirepartsofdiagramsto behighlighted. 72. Openconstructedresponseitemswillbeusedwherenecessary,forexample whereitisconsideredimportanttoaskstudentstoexplaintheirmethodorjustify theirsolution.Fortheseitems,studentswillentertheirresponsesintextboxes. 73. Constructedresponseswillbecollectedautomaticallybythecomputerdelivery systemandanOnlineCodingSystemwillbedevelopedtofacilitatetheircodingby experts.Thiseliminatestheneedforseparatedataentry,minimisestheneedfor datacleaning,andallowscodingtotakeplaceoffsiteifdesired.

25

74. Thecodingschemeforanitemwillallowforpartialcreditifappropriate,suchas whenmultiplecorrectanswersarerequiredforfullcreditorwhenacorrect strategyisemployedbutisnotexecutedproperly.Designatedbehaviours(suchas explorationsstrategies)thatprovidereliableevidenceaboutproblemsolving competencyoverandabovethefulfilmentoftaskdemandswillbecapturedand contributetoscoring. InteractiveProblems 75. Interactiveproblemscanbebuiltonunderlyingmathematicalmodelswhose parameterscanbevariedsystematicallytoachievedifferingdegreesofdifficulty. Therearetwowellusedparadigms:lineardifferenceequationsandfinitestate machines. 76. Withproblemsituationsmodelledbylineardifferenceequations(alsoreferredto aslinearstructuralequations) 7 ,theproblemsolvermustmanipulateoneormore inputvariables(suchascontrolsforaclimatecontrolsystem)andconsiderthe effectthishasononeormoreoutputvariables(suchastemperatureand humidity);theoutputvariablesmayalsoinfluencethemselvessothatthesystemis dynamic.Examplecontextsincluderemotecontrols,thermostats,paintmixingand ecosystems. 77. Afinitestatemachineisasystemwithafinitenumberofstates,inputsignalsand outputsignals(Buchner&Funke1993) 8 .Thesystemsnextstate(andoutput signal)isuniquelydeterminedbyitscurrentstateandthespecificinputsignal. Withproblemsituationsmodelledbyfinitestatemachines,theproblemsolver mustsupplyinputsignals(usuallyintheformofasequenceofbuttonpresses)to determinetheeffectonthesystemsstatesinanefforttounderstandits underlyingstructureandmoveittowardsagoalstate.Manyeverydaydevicesand contextsaregovernedorconstrainedbytherulesofafinitestatemachine structure.Examplesincludedigitalwatches,mobilephones,microwaveovens, MP3players,ticketvendingmachinesandwashingmachines. 78. Thetypicaltaskdemandsforsuchinteractiveproblemsareasfollows(seeBlech& Funke,2005andGreiff&Funke,2008foradditionaldetail):

7SeeGreiff&Funke(2008)whousethetermMicroDYNtodescribethesesystems.Anearlier

implementationofsuchasystemisknownasDynamisseeBlech&Funke(2005).

8Finitestatemachinesforassessmentpurposeshavebeenimplementedunderthename

MicroFinseehttp://www.psychologie.uniheidelberg.de/ae/allg_en/forschun/probleml.html 26

Exploration.Acquireknowledgeofsystemstructureeitherbyactiveor directedexploration(interaction).[Explorationstrategiescanbetrackedand capturedbythecomputerdeliverysystem.]

Identification.Giveorcompletearepresentationofthementalmodelofthe systemthatisformedduringexploration.Thismaybeindrawingortextform. [Theaccuracyofthemodelhelpsinassessingacquiredcausalknowledge.]

Control.Practicalapplicationofacquiredknowledge:transformagivenstate intoagoalstateand(forappropriatesystems)maintainthegoalstateover time.Acorrectmodelofthesystemmaybeprovidedtominimisedependence onpreviousitems.[Transferofacquiredknowledgeisassessedinthisway.]

Explanation.Describestrategiesusedtoreachagoal;explainhowasystem works;orsuggestcausesofamalfunctionofadevice.

79. Studentsmayalreadyhavesomeideaoftherelationshipsbetweensystem variablesinproblemsituationsbecauseoftheirfamiliaritywithsimilar,actual devices.Suchpriorknowledgewillvarybetweenindividualsandsoavarietyof common,everydayproblemcontextswillbeusedtohelpovercomethiseffect acrosstheassessment.Inaddition,afewmoreunusualbutengaginggamelike contextswillbeincludedwheretherelationshipsmustbeinferredsolelyby manipulationandobservationofsystemvariables. 80. Thedifficultyofproblemsofthesetypesislargelydependentontheinternal complexityofthemathematicalmodelsunderlyingthesituations.Problemsof varyingdifficultycanbesetbysystematicallyvaryingthiscomplexitywhichis determinedbythenumberofvariablesinvolvedandhowtheyareconnected.For example,aprobleminvolvingonlyafewvariablescanbeveryeasyifitonly involvesdirecteffectsbetweeninputandoutputvariables,butcanbemade extremelydifficultbytheinclusionofmultipleeffectsandsideeffectsbetween outputvariables. c. DistributionofItems 81. Therecommendedpercentagedistributionofscorepointsaccordingtothe cognitiveprocessesinvolvedinproblemsolvingisgiveninTable2.Highest weightingisgiventoPlanningandexecutinginrecognitionoftheimportanceof beingabletocarrythroughasolutiontoasuccessfulconclusion.Lowerthan averageweightisgiventoMonitoringandreflectingbecauseitisanintegralpartof
27

theotherthreeprocessesandthereforealsoisassessed(indirectly)initemsthat targetthoseprocesses. Table2.DistributionofScorePointsbyProcess Exploring& Representing understanding &formulating 2025% 2025% 82. Table3indicatestherecommendeddistributionofitemsacrossthetwootherkey domainelements,problemcontextandnatureoftheproblemsituation 9 .Theclear emphasisoninteractiveproblemsoverstaticproblems(aratioofabout2:1) reflectstheintentionthattheassessmentshouldconcentrateonthisimportant classofproblemswhich,withtheadvantageofcomputerdelivery,itispossibleto includeinalargescaleinternationalsurveyforthefirsttime. Table3.DistributionofItemsbyProblemNatureandContext Static problemsituation Interactive problemsituation Technology context 510% 4045% 4555% Nontechnology context 2025% 2530% 4555% 2535% 6575% 100% Planning& executing 3545% Monitoring &reflecting 1020% Total 100%

9Notethatthesetwoclassificationswillbeassignedattheunitlevelandapplytoallitemsinthe

unit.

28

CHAPTER5:REPORTINGPROBLEMSOLVINGCOMPETENCY 10 83. Itisexpectedthatatleastfourlevelsofproficiencywillbeabletobeidentifiedand describedtoshowhowindividualsproblemsolvingcompetencygrowsand develops,andtoenablecomparisonsofstudentperformancebetweenandwithin participatingcountriesandeconomies.Therewillnotbeenoughitemsinthemain surveytoreportonsubscales. 84. Proficiencydescriptionscharacterisingtypicalstudentperformanceateachlevel willbedevelopedbyanalysingtheknowledgeandskillsrequiredtoanswerthe itemsatthatlevel.Itisexpectedthatthefollowingabilitieswillcharacterisehigh performingstudents: Abilitytoplanandexecutesolutionsthatinvolvearelativelyhighnumberof steps,andtoapplyawiderangeofreasoningskills. Abilitytodealwithsituationsinvolvingmanyvariableswherethereishigh dependencybetweenthevariablesoralargenumberofconstraintsontheir values. Abilitytounderstandandrelateinformationpresentedinavarietyoffamiliar orunfamilarrepresentations. Abilitytointeractwithproblemstodiscoverundisclosedinformationor handleunanticipatedobstacles. 85. Lowperformingstudentsareexpectedtohavethefollowingcharacteristics: Atbestbeabletoplanandexecutesolutionsthatinvolveonlyafewstepsand simplereasoning. Beunabletounderstandinformationpresentedinunfamiliarrepresentations ortorelateinformationbetweenfamiliarrepresentations. Beabletomakelittleornoprogressinsolvingaproblemunlessitinvolves onlyoneortwovariableswithverylimiteddependencyandnoconstraints. Onlybeabletodiscoverundisclosedinformationifinstructionsareprovided todirectexplorationactivity.

10Furtherdetailwillbeaddedwhenfieldtrialresultsareavailable.

29

CHAPTER6:SAMPLETASKS 11 86. DIGITALWATCH Asimulationofadigitalwatchispresented.Thewatchiscontrolledbyfourbuttons, thefunctionsofwhichareunknowntothestudentattheoutsetoftheproblem.The studentisrequiredto:(Q1)determinethroughguidedexplorationhowthebuttons workinTIMEmode;(Q2)completeadiagramshowinghowtocyclethroughthe variousmodes;and(Q3)usethisknowledgetocontrolthewatch(setthetime). ProblemContext:Technology,Personal NatureofProblem:Interactive ProblemSolvingProcesses:(Q1)Exploringandunderstanding;(Q2) Representingandformulating;(Q3)Planningandexecuting ReasoningSkills:Correlational,multidimensional

87. BASKETBALL Therulesforabasketballtournamentrelatingtothewayinwhichmatchtime shouldbedistributedbetweenplayersaregiven.Therearetwomoreplayersthan required(5)andeachplayermustbeoncourtforatleast25ofthe40minutes playingtime.Studentsarerequiredto:(Q1)createascheduleforteammembers thatsatisfiesthetournamentrules;and(Q2)reflectontherulesbycritiquingan existingschedule. ProblemContext:Social,Nontechnology NatureofProblem:Static ProblemSolvingProcess:(Q1)Planningandexecuting;(Q2) Monitoringand reflecting ReasoningSkills:Combinatorial,deductive

11Fieldtrialtasksnotneededforthemainsurveywillbeaddedoncethemainsurveyselectionis made,withsomeexamplesbeingdistributedthroughoutthetextandcrossreferencedtoillustrate thediscussion.

30

REFERENCES Adey,P.,Csapo,B.,Demetriou,A.,Hautamki,J.&Shayer,M.(2007).Canwebe intelligentaboutintelligence?Whyeducationneedstheconceptofplasticgeneral ability.EducationalResearchReview2,7597. Anderson,L.W.,Krathwohl,D.R.,Airasian,P.W.,Cruikshank,K.A.,Mayer,R.E., Pintrich,P.R.,Raths,J.&Wittrock,M.C.(2001).ATaxonomyforLearning,Teaching, andAssessing:ARevisionofBloomsTaxonomyofEducationalObjectives.NewYork: Longman. Baxter,G.P.&Glaser,R.(1997).Anapproachtoanalysingthecognitivecomplexityof scienceperformanceassessments(TechnicalReport452),NationalCenterfor ResearchonEvaluation,StandardsandStudentTesting(CRESST),LosAngeles,CA. Blech,C.&Funke,J.(2005).Dynamisreview:Anoverviewaboutapplicationsofthe Dynamisapproachincognitivepsychology.Bonn:DeutschesInstitutfr Erwachsenenbildung(availableathttp://www.diebonn.de/esprid/dokumente/doc 2005/blech05_01.pdf). Blech,C.&Funke,J.(2010).Youcannothaveyourcakeandeatit,too:Howinduced goalconflictsaffectcomplexproblemsolving.OpenPsychologyJournal3,4253. Bransford,J.D.,Brown,A.OL.&Cockling,R.R.(Eds.)(1999).HowPeopleLearn:Brain, Mind,Experience,andSchool.NationalAcademyPress:Washington,DC. Buchner, A. & Funke, J. (1993).Finitestate automata: Dynamic task environments in problemsolving research. The Quarterly Journal of Experimental Psychology 46A (1), 83118. Duncker,K.(1945).Onproblemsolving.PsychologicalMonographs58(3)(WholeNo. 270). Funke,J.(2001).Dynamicsystemsastoolsforanalysinghumanjudgement.Thinking andReasoning,2001,7(1),6989. Funke,J.(2010).Complexproblemsolving:Acaseforcomplexcognition?Cognitive Processing11,133142. Funke,J.&Frensch,P.A.(2007).Complexproblemsolving:TheEuropeanperspective 10yearsafter.InD.H.Jonassen(Ed.),LearningtoSolveComplexScientificProblems (pp.2547).NewYork:LawrenceErlbaum. Greiff,S.&Funke,J.(2008).IndikatorenderProblemlseleistung:SinnundUnsinn verschiedenerBerechnungsvorschriften.BerichtausdemMicroDYNProjekt[Measuring ComplexProblemSolving:TheMicroDYNapproach].Heidelberg:Psychologisches Institut. Klauer,K.&Phye,G.(2008).Inductivereasoning:atrainingapproach.Reviewof EducationalResearch,78(1),85123. Klieme,E.(2004).Assessmentofcrosscurricularproblemsolvingcompetencies.In J.H.MoskowitzandM.Stephens(Eds.).ComparingLearningOutcomes.International AssessmentsandEducationPolicy(pp.81107).London:RoutledgeFalmer.
31

Klieme,E.,Leutner,D.&Wirth,J.(Eds.).(2005).Problemlsekompetenzvon SchlerinnenundSchlern.DiagnostischeAnstze,theoretischeGrundlagenund empirischeBefundederdeutschenPISA2000Studie[Problemsolvingcompetencyof students.Assessmentapproaches,theoreticalbasics,andempiricalresultsofthe GermanPISA2000study].Wiesbaden,Germany:VSVerlagfrSozialwissenschaften. Lesh,R.&Zawojewski,J.S.(2007).Problemsolvingandmodeling.InF.Lester(Ed.), TheHandbookofResearchonMathematicsTeachingandLearning(2nded.)(pp.763 804).Reston,VA:NationalCouncilofTeachersofMathematics;Charlotte,NC: InformationAgePublishing(jointpublication). Leutner,D.,Klieme,E.,Meyer,K.&Wirth,J.(2004).Problemlsen[Problemsolving].In M.Prenzel,J.Baumert,W.Blum,R.Lehmann,D.Leutner,M.Neubrand,R.Pekrun,J. Rost&U.Schiefele(PISAKonsortiumDeutschland)(Eds.),PISA2003:Der BildungsstandderJugendlicheninDeutschlandErgebnissedeszweiteninternationalen Vergleichs(pp.147175).Mnster,Germany:Waxmann. Leutner,D.&Wirth,J.(2005).WhatwehavelearnedfromPISAsofar:aGerman educationalpsychologypointofview.KEDIJournalofEducationalPolicy2(2),3956. Mayer,R.E.(1990).Problemsolving.InM.W.Eysenck(Ed.),TheBlackwellDictionary ofCognitivePsychology(pp.284288).Oxford:BasilBlackwell. Mayer,R.E.(1992).Thinking,Problemsolving,Cognition(2ndEd.).NewYork,NY: Freeman. Mayer,R.E.(1998).Cognitive,metacognitive,andmotivationalaspectsofproblem solving.InstructionalScience26,4963. Mayer,R.E.(2002).Ataxonomyforcomputerbasedassessmentofproblemsolving. ComputersinHumanBehavior18,623632. Mayer,R.E.(2003).LearningandInstruction.UpperSaddleRiver,NJ:MerrillPrentice Hall. Mayer,R.E.&Wittrock,M.C.(1996).Problemsolvingtransfer.InR.Calfee&R. Berliner(Eds.),HandbookofEducationalPsychology(pp.4762).NewYork: Macmillan. Mayer,R.E.&Wittrock,M.C.(2006)ProblemSolving.InP.A.AlexanderandP. H.Winne(Eds.),HandbookofEducationalPsychology(2nded.)(ch.13).Mahwah,NJ: LawrenceErlbaumAssociates. OECD.(2003a).ThePISA2003AssessmentFramework.Mathematics,Reading,Science andProblemSolvingKnowledgeandSkills.Paris:OECD. OECD.(2003b).Thedefinitionandselectionofcompetencies(DeSeCo):Executive summaryofthefinalreport.Paris:OECD. http://www.oecd.org/dataoecd/47/61/35070367.pdf OECD.(2004).ProblemSolvingforTomorrowsWorld.FirstMeasuresofCross CurricularCompetenciesfromPISA2003.Paris:OECD. OECD.(March2009).PIAACproblemsolvingintechnologyrichenvironments: Conceptualframework.Paris:OECD.
32

 ONeil,H.F.(2002).Perspectivesoncomputerbasedassessmentofproblemsolving. ComputersinHumanBehavior18,605607. OsmanM.(2010).Controllinguncertainty:Areviewofhumanbehaviorincomplex dynamicenvironments.PsychologicalBulletin136,6586. Plya,G.(1945).HowtoSolveIt.Princeton,NJ:PrincetonUniversityPress. Reeff,J.P.,Zabal,A.&Blech.C.(2006).TheAssessmentofProblemSolving Competencies.ADraftVersionofaGeneralFramework.Bonn:DeutschesInstitutfr Erwachsenenbildung.(RetrievedMay8,2008,fromhttp://www.die bonn.de/esprid/dokumente/doc2006/reeff06_01.pdf). Robertson,S.I.(2001).ProblemSolving.EastSussex:PsychologyPress. RychenD.S.&Salganik,L.H.(Eds.).(2003).KeyCompetenciesforaSuccessfulLifeand aWellFunctioningSociety.Gttingen,Germany:HogrefeandHuber. Vosniadou,S.&Ortony,A.(1989).SimilarityandAnalogicalReasoning.NewYork: CambridgeUniversityPress. Wirth,J.&Klieme,E.(2004).Computerbasedassessmentofproblemsolving competence.AssessmentinEducation:Principles,PolicyandPractice10(3),329345.

33

ANNEXA:OVERVIEWOFPROBLEMSOLVINGRESEARCH 1 HistoricalandTheoreticalFoundations 1. Psychologicalresearchonproblemsolvingbeganintheearly1900s,asan outgrowthofmentalphilosophy(Humphrey,1963;Mandler&Mandler,1964). Throughoutthe20thcenturyfourtheoreticalapproachesdeveloped:early conceptions,associationism,Gestaltpsychology,andinformationprocessing (Mayer,inpress). EarlyConceptions 2. Thestartofpsychologyasasciencecanbesetat1879theyearWilhelmWundt openedthefirstworldspsychologylaboratoryinLeipzig,Germany,andsoughtto traintheworldsfirstcohortofexperimentalpsychologists.Wundtsmain contributiontothestudyofproblemsolving,however,wastocallforits banishmentonthegroundsthatcomplexcognitiveprocessingwastoo complicatedtobestudiedbyexperimentalmethods(Wundt,1911/1973).Inspite ofhisadmonishments,agroupofhisformerstudentswhocametobeknownas theWurzburggroupsoughttostudythinkingbyaskingstudentstodescribe theirthoughtprocessastheysolvedwordassociationproblems,suchasfindinga superordinateofnewspaper(e.g.,ananswerispublication).Althoughtheydid notproduceanewtheoreticalapproach,theWurzburggroupfoundempirical evidencethatchallengedsomeofthekeyassumptionsofmentalphilosophy, includingchallengestotheideathatallthinkinginvolvesmentalimagery. Associationism 3. Duringthe1920sthroughthe1950s,thefirstmajortheoreticalapproachtotake holdinthescientificstudyofproblemsolvingwasassociationismtheideathat cognitiverepresentationsinthemindconsistofideasandlinksbetweenthemand thatcognitiveprocessinginthemindinvolvesfollowingachainofassociations fromoneideatothenext(Mandler&Mandler,1964;Mayer,1992).Forexample, inaclassicstudy,Thorndike(1911)placedahungrycatinwhathecalledapuzzle boxawoodencrateinwhichpullingaloopofstringthathungfromoverhead wouldopenatrapdoortoallowthecattoescapetoabowloffoodoutsidethe crate.AccordingtoThorndike,overthecourseofmanytrials,thecatlearnedto
1Exceptforthematerialpertainingtocomplexproblemsolving,thisreviewwaspreparedbyRich MayerinFebruary2010.ThesectiononComplexProblemSolvingismainlyderivedfromFunke andFrensch(2007).ProfessorsMayer(UniversityofCalifornia,SantaBarbara)andFunke (UniversityofHeidelberg,Germany)aremembersofthePISA2012ProblemSolvingExpertGroup.

35

escapethroughaprocessthathecalledthelawofeffect:responsesthatleadto dissatisfactionbecomelessassociatedwiththesituationandresponsesthatlead tosatisfactionbecomemoreassociatedwiththesituation.Solvingaproblemis simplyamatteroftrialanderrorandaccidentalsuccess,accordingtoThorndikes associationistview.Amajorchallengetoassociationisttheoryconcernsthenature oftransferthatis,explaininghowaproblemsolverinventsacreativesolution thathasneverbeenperformedbefore.Associationistconceptionsofcognitioncan beseenincurrentresearch,includingneuralnetworks,connectionistmodels,and paralleldistributedprocessingmodels(Rogers&McClelland,2004). GestaltPsychology 4. TheGestaltapproachtoproblemsolvingdevelopedinthe1930sand1940sasa counterbalancetotheassociationistapproach.AccordingtotheGestaltapproach cognitiverepresentationsconsistofcoherentstructures(ratherthanindividual associations)andthecognitiveprocessofproblemsolvinginvolvesbuildinga coherentstructure(ratherthanstrengtheningandweakeningofassociations).A majorfocusisonthenatureofinsighthowproblemsolversmovefromnot knowinghowtosolveaproblemtoknowinghowtosolveit(Duncker,1945; Mayer,1995).Forexample,inaclassicstudy,Kohler(1925)observedhowa hungryapeinaplayyardfiguredouthowtostackcratestoallowhimtoreacha bananahangingoverhead.Kohlerreferredtotheunderlyingmechanismasinsight literallyseeingintothestructureofthesituation.AmajorchallengetoGestalt theoryisitslackofprecision.Gestaltconceptionscanbeseeninmodernresearch onmentalmodelsandschemas(Gentner&Stevens,1983). GeorgePolya 5. GeorgePolyawasborninHungaryin1887andemigratedtotheUnitedStatesin 1940.Adistinguishedmathematician,hequicklybecameknownforhisresearch andteachingsinproblemsolvingatStanfordUniversity.Inhisclassbooks,Howto SolveIt(1957),firstpublishedin1945,andMathematicalDiscovery(1965),Polya brokeproblemsolvingintofourphases:(1)understandingtheproblem,(2) devisingaplan,(3)carryingouttheplan,and(4)lookingback.Animportant contributionofPolyasworkishisobservationthatproblemsolvingisalearnable skill:youcanlearnitonlybyimitationandpractice(Polya,1965,p.ix).Drawing mainlyonexamplesinmathematicalproblemsolving,Polyasuggestednumerous problemsolvingheuristicsincludingthinkingofarelatedproblem,breakinga problemintoparts,andrestatingthegivensorgoal.
36

InformationProcessing 6. Theinformationprocessingapproachtoproblemsolvingdevelopedinthe1960s and1970sandwasbasedonthecomputermetaphortheideathathumansare processorsofinformation(Mayer,2009).Accordingtotheinformationprocessing approach,problemsolvinginvolvesaseriesofmentalcomputationseachof whichconsistsofapplyingaprocesstoamentalrepresentation(suchas comparingtwoelementstodetermineiftheydiffer).Intheirclassicbook,Human ProblemSolving,NewellandSimon(1972)proposedthatproblemsolvinginvolved aproblemspacearepresentationofalltheinterveningstatesbetweenthegiven andgoalstatesandsearchheuristicsstrategiesformovingfromthegiventothe goalstate.NewellandSimonusedcomputersimulationasaresearchmethodto testtheirconceptionofhumanproblemsolving,inwhichtheybuiltcomputer programsthatsolvedproblemsusinghumanlikestrategies.Animportant advantageoftheinformationprocessingapproachisthatproblemsolvingcanbe describedwithgreatclarityasacomputerprogram.Animportantlimitationof theinformationprocessingapproachisthatitismostusefulfordescribing problemsolvingforwelldefinedproblemsratherthanilldefinedproblems.The informationprocessingconceptionofcognitionlivesonasakeystoneoftodays cognitivescience(Mayer,2009). CurrentLinesofResearchonProblemSolving 7. Thestudyofproblemsolvinghasasomewhatfragmentedtraditioninwhich varioustopicshavebeenstudiedinrelativeisolationfromoneanother.Some majorlinesofproblemsolvingresearchincludedecisionmaking,reasoning, intelligenceandcreativity,teachingofthinkingskills,expertproblemsolving, thinkingbyanalogy,mathematicalandscientificproblemsolving,situated cognition,cognitiveneuroscienceandcomplexproblemsolving. 8. Aunifyingthemeofseveralofthesediverselinesofproblemsolvingresearchis theideathatproblemsolvingperformancedependsontheproblemsolvers knowledge,includingdomainspecificknowledge(Mayer&Wittrock,2006). Anotherunifyingthemeconcernstheroleofthehumaninformationprocessing systemincludingseverelimitsonworkingmemorycapacity. DecisionMaking 9. Decisionmakingreferstocognitiveprocessinginvolvedinchoosingbetweentwo ormorealternatives(Barron,2000;Kahneman&Tversky,2000;Markman&
37

Medin,2002).Forexample,adecisionmakingtaskistochoosebetween(a)an 85%chanceofwinning$1000(witha15%chanceofwinningnothing)and(b) receiving$800forsure. 10. Researchondecisionmakinghasproducedaprogressionoffourtheories prescriptivetheories,descriptivetheories,heuristictheories,andconstructive theories.Prescriptivetheoriessometimescalledeconomictheoriesspecify whatpeopleshoulddoiftheyarecompletelyrational.Prescriptivetheoriessuch asexpectedvaluetheorypredictthatpeopleshouldoptforthefirstalternative becauseithasahigherexpectedvalue(i.e.,$850)thanthesecondalternative(i.e., $800);however,psychologicalresearchshowsthatpeoplepreferthesecond alternative(Kahneman&Tversky,1984).Descriptivetheoriesdescribewhat peopleactuallydowhenconfrontedwithdecisiontasks.Descriptivetheories suchasprospecttheoryarebasedontheideathatpeopleprefertoavoidaloss morethantheyprefertomakeanequivalentgain,andthatpeopletendto overestimatesmallprobabilitiesandunderestimatelargeprobabilities.For examplemostpeoplepreferasurewinof$3000toan80%chanceofwinning $4000,butmostpeoplepreferan80%chanceoflosing$4000toasurelossof $3000(Kahneman&Tversky,1979).Heuristictheoriesbuildonthesefindingsby focusingonthedomainspecificstrategiesthatpeopleusetomakedecisions (Gigerenzeretal.,1999).Forexample,mostpeopledecidethattheletterris morelikelytooccurinthefirstpositionofanEnglishwordthaninthethird positionofaword(Tversky&Kahneman,1973).Anexplanationisthattheyare usingtheavailabilityheuristic,inwhichitiseasiertothinkofwordsthatstartwith rthantothinkofwordsthathaverasthethirdletter.Constructivetheories alsobuildonthesefindingsbyfocusingondecisionmakingasaprocessof constructingacoherentmentalmodelofthesituation.Forexample,peopleare likelytosaytheywouldbuyanew$10theatreticketiftheydiscoverthattheyhad lost$10butarenotlikelytobuyanew$10ticketiftheydiscovertheyhadlostthe tickettheyhadalreadypurchased(Kahneman&Tversky,1984).Anexplanation forthisfindingisthatpeopleengageinmentalaccountingbywhichtheticket moneywaslostinthesecondscenariobutnotthefirst. 11. Animportantthemeincurrentevidencebasedtheoriesofdecisionmaking concernsthecentralroleofdomainspecificknowledgeandcognitiveprocessing thatpeoplebringtothedecisionmakingtask.Anothercentralthemeisthatpeople useshortcutstrategiestoovercomethelimitsoninformationprocessingimposed

38

bythehumaninformationsystem,therebyrelyingonwhatSimon(1982)calls satisficingselectinganacceptablechoicethatmeetsonescriteria. Reasoning 12. Reasoningreferstodeterminingwhetheraconclusionfollowsfrompremises (Evans,2005;JohnsonLaird,2005).Forexample,conditionalreasoningmaybe examinedusingacardturningproblemsuchas,Ifacardhasavowelononeside, ithasanevennumberontheotherside.Selectthosecardsthatyoudefinitelyneed toturnovertofindoutwhetherornottheyviolatetherule:AD47(Wason, 1966).Researchshowsthatpeopleperformbetterwhenthecardturningproblem ispresentedinconcreteform,suchas,Ifapersonisdrinkingbeer,thenthe personmustbeover19yearsofage.Selectthecardorcardsyouwoulddefinitely needtoturnovertodeterminewhetherornotthepeopleareviolatingtherule: DRINKINGABEER,DRINKINGACOKE,16YEARSOFAGE,22YEARSOFAGE (Griggs&Cox,1982).Subsequentresearchhasshownthatpeopleperformbetter whenthethemeofaconcreteprobleminvolvesdetectingcheaters(Cosmides, 1989)orseekingpermission(Cheng&Holyoak,1985).Basedonaseriesof empiricalinvestigations,JohnsonLairds(2005)mentalmodeltheoryprovidesa similaraccountofdeductivereasoninginwhichpeoplebuildconcretemental modelsofthesituationdescribedinthepremises.Thesefindingspointtothe domainspecificityofreasoning,thatis,theideathatpeopleusetheconcrete contextoftheproblemtoguidetheirreasoningratherthanapplythesamegeneral logicalrulestoallproblems. IntelligenceandCreativity 13. Researchonintelligenceandcreativityexplorestheroleofindividualdifferencesin humancognitiveabilityinproblemsolving(Guilford,1967;Sternberg,1990,1999; Sternberg&Grigorenko,2003).Psychometricresearchexaminesthecorrelations amongcognitivetests(Carroll,1993)andevidenceforgeneralintelligence(or whathasbeencalledg)wouldbereflectedinhighcorrelationsamongallcognitive tests.However,factoranalyses(andsimilarstatisticaltools)revealmanysmaller specialisedfactors,suchasformsofverbalability,mathematicalability,andspatial ability(Carroll,1993;Sternberg,1999).Moderncognitivescienceresearchfocuses onidentifyingcomponentprocessesthatsupportperformanceonintelligence tests,includingcognitivetaskanalysesofintelligencetestitems;andpinpointing individualdifferencesintheinformationprocessingsystemrelatedtocognitive testperformance.Thisworkhighlightstheroleofdomainspecificknowledgeand
39

processingincognitiveperformance,aswellasthewayspeoplesthinkingis dependentonthearchitectureofthehumaninformationprocessingsystem. TeachingofThinkingSkills 14. Researchonteachingofthinkingskillsfocusesontrainingthathelpspeople becomebetterproblemsolvers(Bloom&Broder,1950;Covington,Crutchfield, Davies,&Olton,1974;Nickerson,1999;Ritchhart&Perkins,2005).Forexample, BloomandBroder(1950)taughtcollegestudentstosolveeconomicsproblemsby havingthemwatchamodelsolveproblemswhilethinkingaloud.Similarly, Covington,CrutchfieldDavies,andOlton(1974)taughtelementaryschoolchildren tosolvemysteryproblemsbyaskingthemtoreadcomicbooksthatmodelledhow togenerateandtesthypothesesforvariousmysteryproblems.Apersistentfinding isthatproblemsolvingtrainingtendstobeeffectivewhenitfocusesonspecific skillsneededforthetask(suchasgeneratingandtestinghypotheses),whena successfulproblemsolvermodelstheskills,andwhenthetestinvolvesproblems thataresimilartothoseusedduringtraining.Thereisnotstrongevidencethat peoplecanbetaughtgeneralproblemsolvingstrategiesthatimproveperformance acrossadiversesetofproblemsituations(Nickerson,1999;Ritchhart&Perkins, 2005).Thus,animportantthemeofresearchonthinkingskillstrainingconcerns theroleofdomainspecificknowledgeinsupportingproblemsolving. ExpertProblemSolving 15. Researchonexpertproblemsolvinginvolvescomparinghowexpertsandnovices solveproblems(Chase&Simon,1973;deGroot,1965;Ericsson,Feltovich,& Hoffman,2006).Forexample,expertchessplayersarebetterablethannovicesto rememberthepositionofchesspiecesonachessboardfromarealgamebutthe differencebetweenexpertsandnovicesdisappearswhenthechesspiecesare randomlyplacedontheboard(Chase&Simon,1973).Similarly,whengivena chessboardfromarealgameandaskedtomakethenextmove,expertsdonot considermoremovesthannovicesbutexpertsconsiderbettermovesthannovices (deGroot,1965).Researchinthedevelopmentofexpertiseshowsthatexperts mustengageinapproximatelytenyearsofpracticetogaintheknowledgebase neededforexpertproblemsolving,butthattheknowledgeneededforsuccessin onedomaindoesnoteasilytransfertoanother(Ericsson,Feltovich,&Hoffman, 2006). 16. Thesefindingareconsistentwiththeideathatexpertsdonotpossessbetter generalcognitiveskillsthannovicessuchasbettermemorycapacitybutrather
40

expertshavebetterdomainknowledgebasedontheirexperience(suchas schemasforhowseveralpiecesformasingleconfiguration).Similarly,these resultsalsodemonstratethelimitsonthehumaninformationprocessingsystem, whichnecessitatesstrategiessuchasmentallyarrangingthepiecesinto configurations. ThinkingbyAnalogy 17. Thinkingbyanalogyreferstosolvingaproblembyusingonesknowledgeabouta similarproblem(Holyoak,2005).Forexample,GickandHolyoak(1980,1983) askedstudentstosolveDunckers(1945)tumourproblem,inwhichtheymust figureouthowtodestroyaninoperabletumourbyusingraysthatatsufficient intensitycandestroytissueintheirpath.Studentsgenerallywerenotsuccessfulin producingthedesiredsolutiontohavemanyweakraysallconvergeonthe tumoureveniftheywerepreviouslygivenapassageaboutageneralattackinga fortressthatwasbasedonthesamegeneralsolutionofhavingmanysmallforces allconvergeonthefortress.Amajorconclusionisthatpeoplehavetrouble engaginginanalogicaltransferapplyingthesolutionmethodfromoneproblem thattheyknowtoanewproblemthatcanbesolvedbythesamemethod particularlywhenthecoverstoriesdiffer.Thisfindingalsosuggeststhathuman cognitiontendstobedomainspecificsothatsolutionstrategiesusedinone contextarenotautomaticallytransferredtoanewcontext. MathematicalandScientificProblemSolving 18. Researchonmathematicalandscientificproblemsolvingfocusesonhowstudents solveproblemsinthespecificsubjectdomainofmathematicsorscience(Mayer, 2008).Researchshowsthatproficiencyinsolvingarithmeticwordproblems,asan example,dependsonspecifickindsofknowledgeintheproblemsolver,including facts,concepts,procedures,strategies,andbeliefs(Andersonetal.,2001; Kilpatrick,Swafford,&Findell,2001;Mayer,2008).Forexample,studentsareable tosortarithmeticwordproblemsintocategories,suggestingthattheypossess schemasforspecificproblemtypes(Hinsley,Hays,&Simon,1977).Riley,Greeno, &Heller(1982)foundthatyoungstudentsareabletosolvewordproblems presentedinachangesituation(e.g.,Johnhas3marbles.Hegets2moremarbles. Howmanydoeshehavenow?)moreeasilythaninacomparesituation(e.g.,John has3marbles.Petehas2moremarblesthanJohn.HowmanymarblesdoesPete have?).Olderstudentsperformedwellonbothproblemtypes.Thisfinding

41

suggeststhatstudentsmaybeginwithonesimpleschemaanddevelopadditional oneswithmoreexperience. 19. Inthefieldofscientificproblemsolving,apersistentfindingisthatstudentsenter thescienceclassroomwithasetofpreexistingconceptionsofhowthingswork, whichcanbecalledpreconceptionsormisconceptions.Forexample,manyphysics studentsbelievethatifanobjectisinmotionsomeforcemustactonittokeepitin motionamisconceptionsometimesreferredtoasimpetustheory(McCloskey, 1983).Suchmisconceptionsaffecthowstudentsmakepredictionsandobserve resultsinascientificproblemsolvingtask,oftenresultinginconfirmationbias attemptstoproveoneshypothesisandignorediscrepantdata(Chinn&Malhotra, 2002;Dunbar,1993).Overall,researchonmathematicalandscientificproblem solvingpointstotheroleofthelearnersspecificknowledgeindeterminingthe processofproblemsolving. SituatedCognition 20. Situatedcognitionreferstosolvingproblemswithinspecificphysical,socialand culturalcontexts(Nunes,Schlieman,&Carraher,1993;Robbins&Aydede,2009). Forexample,inaclassicstudy,Nunes,Schlieman,andCarraher(1993)foundthat Brazilianstudentsusedcompletelydifferentcomputationalstrategiesforsolving arithmeticproblemsthattheyreceivedonpaperwithinaschoolversusproblems theyencounteredintheirjobasstreetvendors.Inschool,whengivenaproblem suchas35x10=___theytriedtoapplytheschooltaughtproceduresfor multiplication,whereasinthestreettheyinventedastrategybasedonrepeated addition(105+105+105+35=350).Inastudyofcomputationalstrategiesused bygrocerystoreshopperstodeterminewhichoftwoitemswasbettertobuy,Lave (1988)foundthatpeopleneverusedtheschooltaughtstrategyofcomputingunit costandinsteaduseddifferentkindsofstrategiesfordifferentsituations.For example,tochoosebetweena10ouncecanofpeanutsfor90centsanda4ounce canfor45cents,theyusedaratiostrategyinwhichthelargercanisabetterbuy becauseitcoststwiceasmuchbuthasmorethantwiceasmanyounces.Research onsituatedcognitionorwhatcanbecalledeverydaythinkingshowsthatthe contextinwhichsomeoneencountersaprobleminfluenceshowtheygoabout solvingtheproblem. CognitiveNeuroscienceofProblemSolving 21. Researchonthecognitiveneuroscienceofproblemsolvingfocusesonthebrain activitythatoccursduringproblemsolving(Goel,2005).Forexample,Goel(2005)
42

founddifferentpartsofthebrainarerecruitedforsolvingreasoningproblemsthat arepresentedinabstractform(e.g.,AllPareB.AllCareP.Therefore,allCareB.) versusinconcreteform(e.g.,Alldogsarepets.Allpoodlesaredogs.Therefore,all poodlesarepets.).Suchfindingsagainsuggestthatreasoningissomewhat domainspecificratherthanbeingbasedonapplyingauniversalsetofreasoning rules. ComplexProblemSolving 22. About30yearsago,Drner(1975)adoptedthetermcomplexproblemsolvingto describeaparticulartypeofproblemtobestudied,whichdifferedfromsimple problemsolvingintermsofcomplexityandotherattributes,aswellasanew methodofinvestigation,theuseofcomputersimulatedmicroworlds.Atthistime, researcherswerebecomingincreasinglyconvincedthatempiricalfindingsand theoreticalconceptsderivedfromsimplenoveltaskscouldnoteasilybe generalisedtomorecomplex,reallifeproblems,andthatdomainspecific knowledgeandstrategiesplayanimportantpartinproblemsolving. 23. InEurope,twomainapproachestocomplexproblemsolvingresearchemerged havingincommonanemphasisonrelativelycommon,semanticallyrich, computerisedlaboratorytasksconstructedtobesimilartoreallifeproblems (Frensch&Funke,1995;Funke&Frensch,2007).Oneapproach,initiatedby Broadbent(1977;seeBerry&Broadbent,1995),focusesonthedistinction betweenexplicitproblemsolving(problemsolvingcontrolledbythesolvers intentions)andimplicitproblemsolving(problemsolvingthatisautomaticor nonconscious),andtypicallyemploysmathematicallywelldefinedcomputer models.Theotherapproach,initiatedbyDrner(e.g.,Drner,1980;Drneretal, 1983),isinterestedintheinterplayofthecognitive,motivational,andsocial componentsofproblemsolving(Funke&Frensch,2007),andutilisesvery complexcomputersimulationsinvolvingupto2000interconnectedvariables. 24. FrenschandFunke(1995b,p.18)proposethefollowingdistinctivecharacteristics ofcomplexproblemsolvinginkeepingwiththeresearchtraditionsestablishedby BroadbentandDrner 2 :Thegivenstate,goalstate,andbarriersbetweengiven stateandgoalstatearecomplex,changedynamicallyduringproblemsolving,and

2ThisdefinitionisrootedintheEuropeantraditionofBroadbentandDrner.Amuchbroaderview

ofwhatconstitutesacomplexproblem,incontrasttoasimpleproblem,istakenbymanyauthors (e.g., Sternberg & Frensch, 1991). Some use it to denote authentic tasks, irrespective of their properties;othersuseitinasimilarmannertoFrenschandFunke,butallowcomplexproblemsto bedynamicorintransparent. 43

areintransparent.Theexactpropertiesofthegivenstate,goalstate,andbarriersare unknowntothesolverattheoutset.Thesefeaturescanbeillustratedwithan example(duetoVollmeyer,Burns&Holyoak,1996).Acomputersimulatedfish tankispresentedtosubjects.Inthesimulationtherearefourspeciesofseaanimal (crabs,prawns,lobstersandseabass);thesespeciesareaffectedbyfourquantities (temperature,salt,oxygenandcurrent).Thefourquantitiescanbemanipulatedby thesubjects(usersofthesystem),buttherelationshipsbetweenthequantitiesand theseaanimalsarenotgiventosubjects.Changesmadebysubjectsto temperature,salt,oxygenandcurrentwillcausechangesintheseaanimal populations.Inaddition,intheabsenceofanysuchsubjectinterventionthelobster populationwilldeclineofitsownaccord.Subjectsareaskedtotrytoworkoutthe relationshipsbetweenthequantitiesandtheseaanimalsbyexperimentingwith differentinputvalues,andarethenaskedtocontrolthesysteminordertoreach specifictargetvaluesofeachseaanimal.Itshouldbenotedthattheequations governingtheunderlyingsystemarequitearbitrary,implyingthatknowledgeof realworldmarinebiologywillbeatbestuselessandatworstcounterproductive. 25. Research(Funke&Frensch,2007)doesnotsupportastronglinkbetweenglobal intelligenceandcomplexproblemsolvingcompetencywhengoalspecificityand transparencyarelowandwhenthesemanticcontentisrich,butsomeintelligence components(inparticular,processingcapacityreasoningabilityandlearning potential)appeartobecorrelatedwithcomplexproblemsolvingcompetencyeven whenthetaskisintransparent.However,asisthecasewithsimpleproblems, complexproblemsolvingcompetencyishighlydependentondomainspecific knowledgeandstrategies.Finally,whethercomplexproblemsolvingcompetency isanindependentconstructtosimpleproblemsolvingisanopenquestion thoughtheresultsofthefindingsoftheGermannationalextensionsofPISA2000 (Wirth&Klieme,2004)andPISA2003(Leutneretal,2004;Leutner&Wirth, 2005)givesomesupporttotheexistenceofanindependentcomplexproblem solvingconstruct. Conclusions 26. AmajorimplicationofproblemsolvingresearchforthePISA2012Problem Solvingassessmentisthatitshouldtakeintoaccounttheroleofthestudents domainknowledgeinproblemsolvingratherthantryingtoassessproblemsolving ingeneral.Inaddition,theproblemsolvingliteraturesuggeststhatthePISA2012 ProblemSolvingassessmentshouldbebasedonproblemsolvinginauthentic concretecontextsratherthanwithabstracttasks.Theproblemsolvingliterature
44

alsosuggeststhattheproblemsolvingtasksusedinthePISA2012Problem Solvingassessmentshouldbesufficientlyrichtoallowforstudentstousetheir informationprocessingsystemstomanageacollectionofdifferentkindsoftheir knowledgeratherthanbebasedonshortpuzzlelikeitems. 27. Increasingly,theproblemsfacingcitizensofthe21stcenturyarecomplexproblems andmethodsforsolvingsimpleproblemscannotbeeasilygeneralisedtosolve suchproblems.Furthermore,thereisnoempiricalevidenceforarelationbetween complexproblemsolvingcompetencyandglobalintelligence,butthereissome evidence(thoughnotconclusive)thatcomplexproblemsolvingcompetencyisa separateconstructandnotjusttheapplicationofnormalproblemsolving processestocomplexsituations.Theclearimplicationisthatcomplexproblem solvingshouldbeacentralfeatureofthePISA2012ProblemSolvingassessment.

45

References Anderson,L.W.,Krathwohl,D.R.,Airasian,P.W.,Cruikshank,K.A.,Mayer,R.E., Pintrich,P.R.,Raths,J.,&Wittrock,M.C.(2001).Ataxonomyforlearning,teaching, andassessing:ArevisionofBloomstaxonomyofeducationalobjectives.NewYork: Longman. Baron,J.(2000).Thinkinganddeciding(3rded).NewYork:CambridgeUniversity Press. Bloom,B.S.,&Broder,B.J.(1950).Problemsolvingprocessesofcollegestudents:An exploratoryinvestigation.Chicago:UniversityofChicagoPress. Berry,D.C.,&Broadbent,D.E.(1995).Implicitlearninginthecontrolofcomplex systems.InP.A.Frensch&J.Funke(Eds.),ComplexProblemSolving:TheEuropean Perspective(pp.325).Hillsdale,NJ:LawrenceErlbaumAssociates. Broadbent,D.E.(1977).Levels,hierarchies,andthelocusofcontrol.QuarterlyJournal ofExperimentalPsychology29,181200. Carroll,J.B.(1993).Humancognitiveability.NewYork:CambridgeUniversityPress. Chase,W.G.,&Simon,H.A.(1973).Perceptioninchess.CognitivePsychology,4,5581. Cheng,P.W.,&Holyoak,K.(1985).Pragmaticreasoningschemas.Cognitive Psychology,17,391416. Chinn,C.A.,&Malhotra,B.A.(2002).Childrensresponsestoanomalousscientific data:Howisconceptualchangeimpeded?JournalofEducationalPsychology,94, 327343. Cosmides,L.(1989).Thelogicofsocialexchange:Hasnaturalselectionshapedhow humansreason?Cognition,31,187276. Covington,M.V.,Crutchfield,R.S.,Davies,L.B.,&Olton,R.M.(1974).Theproductive thinkingprogram.Columbus,OH:Merrill. deGroot,A.D.(1965).Thoughtandchoiceinchess.TheHague,Netherlands:Mouton. Drner,D.,Kreuzig,H.W.,Reither,F.,&Studel,T.(1983).Lohhausen.VomUnigangmit Unbestimmtheit und Komplexitt [Lohhausen. On dealing with uncertainty and complexity].Bern:Huber. Dorner,D.(1980).Onthedifficultypeoplehaveindealingwithcomplexity.Simulation &Games11,87106. Dienes, Z., & Berry, D. (1997). Implicit learning: Below the subjective threshold. PsychonomicBulletin&Review4,323. Dunbar,K.(1993).Conceptdiscoveryinascientificdomain.CognitiveScience,17,397 434. Duncker,K.(1945).Onproblemsolving.PsychologicalMonographs58:3(WholeNo. 270). Ericsson,K.A.,Feltovich,P.J.,&Hoffman,R.R.(Eds.)(2006).TheCambridgehandbook ofexpertiseandexpertperformance.NewYork:CambridgeUniversityPress. Evans,J.S.B.T.(2005).Deductivereasoning.InK.J.Holyoak,&R.G.Morrison(Eds.), TheCambridgehandbookofthinkingandreasoning(pp.169184).NewYork: CambridgeUniversityPress. Frensch,P.A.&Funke,J.(Eds.).(1995).ComplexProblemSolving:TheEuropean Perspective.Hillsdale,NJ:LawrenceErlbaumAssociates. Frensch,P.A.&Rnger,D.(2003)Implicitlearning.CurrentDirectionsinPsychological Science12,1318. Funke,J.(2001).Dynamicsystemsastoolsforanalysinghumanjudgement.Thinking andReasoning7(1),6989. Funke,J.andFrensch,P.A.(2007).Complexproblemsolving:TheEuropean perspective10yearsafter.InD.H.Jonassen(Ed.),LearningtoSolveComplex ScientificProblems(pp.2547).NewYork:LawrenceErlbaum. Gentner,D.,&Stevens,A.L.(Eds.).(1983).Mentalmodels.Hillsdale,NJ:Erlbaum. Gick,M.I.,&Holyoak,K.J.(1980).Analogicalproblemsolving.CognitivePsychology, 12,306355.
46

Gick,M.I.,&Holyoak,K.J.(1983).Schemainductionandanalogicaltransfer.Cognitive Psychology,15,138. Gigerenzer,G.,Todd,P.M.,&theABCResearchGroup(Eds.).(1999).Simpleheuristics thatmakeussmart.Oxford,UK:OxfordUniversityPress. Goel,V.(2005).Cognitiveneuroscienceofdeductivereasoning.InK.J.Holyoak,&R.G. Morrison(Eds.),TheCambridgehandbookofthinkingandreasoning(pp.475492). NewYork:CambridgeUniversityPress. Griggs,R.A.,&Cox,J.R.(1982).TheelusivethematicmaterialseffectinWasons selectiontask.BritishJournalofPsychology,73,407420. Guilford,J.P.(1967).Thenatureofhumanintelligence.NewYork:McGrawHill. Hinsley,D.,Hayes,J.R.,&Simon,H.A.(1977).Fromwordstoequations.InP. Carpenter&M.Just(Eds.),Cognitiveprocessesincomprehension.Hillsdale,NJ: Erlbaum. Holyoak,K.J.(2005).Analogy.InK.J.Holyoak,&R.G.Morrison(Eds.),TheCambridge handbookofthinkingandreasoning(pp.117142).NewYork:Cambridge UniversityPress. Humphrey,G.(1963).Thinking:Anintroductiontoexperimentalpsychology.NewYork: Wiley. JohnsonLaird,P.N.(2005).Mentalmodelsandthought.InK.J.Holyoak,&R.G. Morrison(Eds.),TheCambridgehandbookofthinkingandreasoning(pp.185208). NewYork:CambridgeUniversityPress. Kahneman,D.,&Tversky,A.(1979).Prospecttheory:Ananalysisofdecisionunder risk.Econometrica,47,263291. Kahneman,D.,&Tversky,A.(1984).Choice,values,andframes.AmericanPsychologist, 39,341350. Kahneman,D.,&Tversky,A.(Eds.).(2000).Choices,values,andframes.NewYork: CambridgeUniversityPress. Kilpatrick,J.,Swafford,J.,&Findell,B.(Eds.).(2001).Addingitup:Helpingchildren learnmathematics.Washington,DC:NationalAcademyPress. Klieme,E.(2004).Assessmentofcrosscurricularproblemsolvingcompetencies.In J.H.MoskowitzandM.Stephens(Eds.).ComparingLearningOutcomes. InternationalAssessmentsandEducationPolicy(pp.81107).London:Routledge Falmer. Kohler,W.(1925).Thementalityofapes.NewYork:Liveright. Lave,J.(1988).Cognitioninpractice.Cambridge,England:CambridgeUniversityPress. Leutner,D.,Klieme,E.,Meyer,K.&Wirth,J.(2004).Problemlsen[Problemsolving]. InM.Prenzel,J.Baumert,W.Blum,R.Lehmann,D.Leutner,M.Neubrand,R. Pekrun,J.Rost&U.Schiefele(PISAKonsortiumDeutschland)(Eds.),PISA2003: DerBildungsstandderJugendlicheninDeutschlandErgebnissedeszweiten internationalenVergleichs(pp.147175).Mnster,Germany:Waxmann. Mandler,J.M.,&Mandler,G.(1964).ThinkingfromassociationismtoGestalt.New York:Wiley. Markman,A.B.,&Medin,D.L.(2002).Decisionmaking.InD.Medin(Ed.),Stevens handbookofexperimentalpsychology,Volume2:Memoryandcognitiveprocesses (2nded;pp.413466).NewYork:Wiley. Mayer,R.E.(1992).Thinking,problemsolving,cognition(2nded).NewYork:Freeman. Mayer,R.E.(1995).Thesearchforinsight:GrapplingwithGestaltpsychologys unansweredquestions.InR.J.Sternberg&J.E.Davidson(Eds.),Thenatureof insight(pp.332).Cambridge,MA:MITPress. Mayer,R.E.(2008).Learningandinstruction.UpperSaddleRiver,NJ:MerrillPrentice Hall. Mayer,R.E.(2009).Informationprocessing.InT.L.Good(Ed.),21stcenturyeducation: Areferencehandbook(pp.168174).ThousandOaks,CA:SAGE. Mayer,R.E.(inpress).Problemsolving.InD.Reisberg(Ed.),Oxfordhandbookof cognitivepsychology.NewYork:OxfordUniversityPress.
47

Mayer,R.E.,&Wittrock,M.C.(2006).Problemsolving.InP.A.Alexander&P.H. Winne(Eds.),Handbookofeducationalpsychology(2nded;pp.287304).Mahwah, NJ:Erlbaum. McCloskey,M.(1983).Intuitivephysics.ScientificAmerican,248(4),122130. Newell,A.,&Simon,H.A.(1972).Humanproblemsolving.EnglewoodCliffs,NJ: PrenticeHall. Nickerson,R.S.(1999).Enhancingcreativity.InR.J.Sternberg(Ed.),Handbookof creativity(pp.392430).NewYork:CambridgeUniversityPress. Nunes,T.,Schliemann,A.D.,&Carraher,D.W,(1993).Streetmathematicsandschool mathematics.Cambridge,UK:CambridgeUniversityPress. Polya,G.(1957).Howtosolveit.GardenCity,NY:Doubleday.[Originallypublishedin 1945byPrincetonUniversityPress.] Polya,G.(1965).Mathematicaldiscovery(vol.2).NewYork:Wiley. OsmanM.(2010).Controllinguncertainty:Areviewofhumanbehaviorincomplex dynamicenvironments.PsychologicalBulletin136,6586. Riley,M.,Greeno,J.G.,&Heller,J.(1982).Thedevelopmentofchildrensproblem solvingabilityinarithmetic.InH.Ginsberg(Ed.),Thedevelopmentofmathematical thinking(pp.153199).NewYork:AcademicPress. Ritchhart,R.,&Perkins,D.N.(2005).Learningtothink:Thechallengeofteaching thinking.InK.J.Holyoak,&R.G.Morrison(Eds.),TheCambridgehandbookof thinkingandreasoning(pp.775802).NewYork:CambridgeUniversityPress. Robbins,P.,&Aydede,M.(Eds.).(2009).TheCambridgehandbookofsituated cognition.NewYork:CambridgeUniversityPress. Rogers,T.T.,&McClelland,J.L.(2004).Semanticcognition:Aparalleldistributed processingapproach.Cambridge,MA:MITPress. Simon,H.A.(1982).Modelsofboundedrationality.Cambridge,MA:MITPress. Sternberg,R.J.(1990).Metaphorsofmind:Conceptionsofthenatureofintelligence. NewYork:CambridgeUniversityPress. Sternberg,R.J.(1999).Handbookofcreativity.NewYork:CambridgeUniversityPress. Sternberg, R. J., & Frensch, P. A. (Eds.). (1991). Complex problem solving: Principles and mechanisms. Hillsdale, NJ: Lawrence Erlbaum Associates Sternberg,R.J.,&Grigorenko,E.L.(Eds.).(2003).Thepsychologyofabilities, competencies,andexpertise.NewYork:CambridgeUniversityPress. Thorndike,E.L.(1911).Animalintelligence.NewYork:Hafner. Tversky,A.,&Kahneman,D.(1973).Availability:Aheuristicforjudgingfrequencyand probability.CognitivePsychology,5,207232. Vollmeyer,R.,Burns,B.D.,&Holyoak,K.J.(1996).Theimpactofgoalspecificityand systematicityofstrategiesontheacquisitionofproblemstructure.Cognitive Science20,75100. Wason,P.C.(1966).Reasoning.InB.M.Foss(Ed.),Newhorizonsinpsychology. Harmondsworth,England:Penguin. Wirth,J.andKlieme,E.(2004).Computerbasedassessmentofproblemsolving competence.AssessmentinEducation:Principles,PolicyandPractice10(3),329 345. Wundt,W.(1973).Anintroductiontoexperimentalpsychology.NewYork:ArnoPress. [OriginallypublishedinGermanin1911.]

48

ANNEXB:PROBLEMSOLVINGEXPERTGROUP ThePEGmembershipforPISA2012isasfollows:

PEGMember
JoachimFunke(Chair) BenCsap JohnDossey ArtGraesser DetlevLeutner RichardMayer TanMingMing RomainMartin

Affiliation
HeidelbergUniversity,Germany UniversityofSzeged,Hungary (ExofficioPGBrepresentative) IllinoisStateUniversity,USA UniversityofMemphis,USA DuisburgEssenUniversity,Germany UniversityofCalifornia,USA MinistryofEducation,Singapore UniversityofLuxembourg,Luxembourg

49