Vous êtes sur la page 1sur 500

10th International Conference of the Learning Sciences

The Future of Learning

Proceedings
Volume 1 - Full papers
July 2-6, 2012 July 2 nd -Australia 6 th, 2012 Sydney,

Sydney, Australia

www.isls.org/icls2012
SPONSORED BY: HOSTED BY:

Centre for Research on Computer Supported Learning & Cognition (CoCo)

ICLS 2012 Conference Proceedings Volume 1 – Full Papers
10th International Conference of the Learning Sciences The University of Sydney, Sydney, NSW, AUSTRALIA July 2nd – 6th, 2012, Sydney

THE FUTURE OF LEARNING

Editors: Jan van Aalst, Kate Thompson, Michael J. Jacobson, and Peter Reimann

Title:

Cite as:

Editors:

ISBN: 978-0-578-10641-0

© 2012, INTERNATIONAL SOCIETY OF THE LEARNING SCIENCES [ISLS] Copyright 2012 International Society of the Learning Sciences, Inc. Rights reserved

van Aalst, J., Thompson, K., Jacobson, M. J., & Reimann, P. (Eds.) (2012). The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) – Volume 1, Full papers. International Society of the Learning Sciences: Sydney, NSW, AUSTRALIA

The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (ICLS 2012) – Volume 1, Full Papers Jan van Aalst, Kate Thompson, Michael J. Jacobson, and Peter Reimann

Published by: International Society of the Learning Sciences (ISLS) Proceedings printed by: LuLu http://www.lulu.com Proceedings distributed by: LuLu and Amazon Book Cover Design: Dorian Peters Book Cover Photo: Christine Bree

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the International Society of the Learning Sciences. The International Society of the Learning Sciences is not responsible for the use which might be made of the information contained in this book. http://www.isls.org

OUR SPONSORS

ICLS2012 would like to thank our sponsors:
• • •

NSW Trade & Investment The Centre for Research on Computer Supported Learning and Cognition (CoCo) Smart Services Collaborative Research Centre

The Early Career workshop and Doctoral Consortium would particularly like to thank the following for their support:
• • • National Science Foundation The International Society of the Learning Sciences Asia-Pacific Society of Computers in Education

We would also like to thank the following for their generosity in donating their time, skills and information to the conference.
• Inquirium

ICLS 2012

Volume 1: Full Papers

ORGANISERS AND COMMITTEES CONFERENCE CHAIRS
• Michael J Jacobson | The University of • Wouter Van Joolingen | University of

Twente, the Netherlands
• Anindito Aditomo | The University of

Sydney, Australia • Peter Reimann | The University of Sydney, Australia CONFERENCE ADVISORY BOARD
• Shaaron • • • • • • • • • • •

Sydney, Australia EARLY CAREER WORKSHOP
• Tom Moher | University of Illinois, United

States
• Anne Newstead | The University of Sydney,

• •

Ainsworth | University of Nottingham, UK Michael Baker | Telecom ParisTech, Paris, France Katerine Bielaczyc | Nanyang Technological University, Singapore Paul Chandler | Wollongong University, Australia Susan Goldman | University of Illinois, United States Kai Hakkarainnen | University of Helsinki, Finland Yasmin Kafai | University of Pennsylvania, United States Paul Kirschner | Open University of the Netherlands, the Netherlands Marcia Linn | University of California, United States Ric Lowe | Curtin University, Australia Naomi Miyake | University of Tokyo, Japan Stella Vosniadou | University of Athens (Greece) and University of Adelaide (Australia) Uri Wilensky | Northwestern University, United States James Pellegrino | University of Illinois, United States

Australia
• Tak-Wai Chan | National Central Taiwan

University, Taiwan PRE-CONFERENCE WORKSHOPS AND SPECIAL SESSIONS
• Lina Markauskaite | The University of

Sydney, Australia
• Dan Suthers | University of Hawaii, United

States
• Carol Chan | The University of Hong Kong,

Hong Kong
• Nikol Rummel | Ruhr-Universitat Bochum,

Germany PAPER REVIEW COORDINATION
• Peter Reimann | The University of Sydney,

Australia
• Peter Freebody | The University of Sydney,

Australia Kyza | Cyprus University of Technology, Cyprus • Ton de Jong | University of Twente, the Netherlands • Nick Kelly | The University of Sydney, Australia
• Eleni

PROCEEDINGS
• Jan van Aalst | The University of Hong

Kong, Hong Kong
• Brian J. Reiser | Northwestern University,

United States
• Cindy Hmelo-Silver | Rutgers University,

United States
• Kate Thompson | The University of Sydney,

LOGISTICS AND ADMINISTRATION SUPPORT • Sadhbh Warren | The University of Sydney, Australia • Sadhana Puntambekar | The International Society of the Learning Sciences • Dana Gnesdilow | The International Society of the Learning Sciences

Australia
• Stella Wen Tian | The University of Hong

Kong, Hong Kong DOCTORAL CONSORTIUM
• Susan Yoon | University of Pennsylvania,

United States

© ISLS

Page vi

ICLS 2012

Volume 1: Full Papers

PAPER REVIEWERS Abrahamson, Dor, University of California, United States Aditomo, Anindito,, The University of Sydney, Australia Ahn, June, University of Maryland, United States Ainsworth, Shaaron, University of Nottingham, United Kingdom Alac, Morana, University of California, United States Alonzo, Alicia, Michigan State University, United States Alterman, Rick, Brandeis University, United States Andriessen, Jerry, Utrecht University, The Netherlands Ares, Nancy, University of Rochester, United States Arnseth, Hans Christian, University of Oslo, Norway Arthur-Kelly, Michael, University of Newcastle, Australia Asterhan, Christa, Hebrew University of Jerusalem, Israel Avramides, Katerina, London Knowledge Lab, United Kingdom Bagley, Elizabeth, University of Illinois, United States Bairral, Marcelo, Federal Rural University of Rio de Janeiro, Brazil Baker, Michael, CNRS - Telecom ParisTech, France Baker, Ryan, Worcester Polytechnic Institute, United Kingdom Barnes, Jacqueline, Indiana University, United States Barth-Cohen, Lauren, University of California, United States Belland, Brian, Utah State University, United States Berland, Leema, University of Texas, United States Bers, Marina, Tufts University, United States Bielaczyc, Katerine, Singapore National Institute of Education, Singapore Booker, Angela, University of California, United States Brett, Clare, University of Toronto, Canada Brian, Reiser, Northwestern University, United States Buckingham, Brandy, Northwestern University, United States Burke, Quinn, University of Pennsylvania, United States Calabrese Barton, Angela, Michigan State University, United States Carmela, Aprea, Swiss Federal Institute for Vocational Education and Training, Switzerland Carpendale, Jeremy, Simon Fraser University, Canada Castro-Alonso, Cris, University of New South Wales, Australia Cesar, Collazos, Universidad del Cauca, Colombia Chan, Carol, The University of Hong Kong, Hong Kong Chang, Yi-Hsing, Southern Taiwan University, Taiwan Charoenying, Timothy, University of California, United States Chase, Catherine, Carnegie Mellon University, United States Chen, Ying-Chih, University of Minnesota, United States Childs, Joshua, University of Pittsburgh, United States Ching, Cynthia Carter, University of California, United States Chinn, Clark, Rutgers University, United States Christophe, Reffay, ENS Cachan, France Chye, Stefanie, Singapore National Institute of Education, Singapore Clegg, Tamara, University of Maryland, United States Cooper, Benny, University of California, United States Correia, Ana-Paula, Iowa State University, United States Crain, Rhiannon, Cornell University, United States Cress, Ulrike, Knowledge Media Research Center, Germany D'Angelo, Cynthia, University of Wisconsin, United States Damsa, Crina, Utrecht University, The Netherlands Davis, Pryce, Northwestern University, United States De Jong, Ton, University of Twente, The Netherlands de Vries, Erica, University of Grenoble, France Dillenbourg, Pierre, Swiss Federal Institute of Technology, Lausanne, Switzerland

© ISLS

Page vii

ICLS 2012

Volume 1: Full Papers

Drago, Kathryn, University of Michigan, United States Dugan, Therese, University of Washington, United States Duncan, Ravit, Rutgers University, United States Durga, Shree, University of Wisconsin, United States Dyke, Gregory, Carnegie Mellon University, United States Earnest, Darrell, University of California, United States Ehret, Christian, Vanderbilt University, United States Eisenberg, Michael, University of Colorado, United States Elen, Jan, K.U.Leuven, Belgium Engelmann, Tanja, Knowledge Media Research Center, Germany Engle, Randi A., University of California, United States Enyedy, Noel, University of California, United States Erkens, Gijsbert, Utrecht University, The Netherlands Erlandson, Benjamin, Arizona State University, United States Ertl, Bernhard, Universität der Bundeswehr, Germany Fatos, Xhafa, Universitat Politecnica de Catalunya, Spain Femke, Kirschner, Utrecht University, The Netherlands Fields, Deborah, University of California, United States Fischer, Frank, University of Munich, Germany Fishman, Barry, University of Michigan, United States Forest, Dominique, Universitat de Brest, France Freebody, Peter, The University of Sydney, Australia Frode, Guriybe, University of Bergen, Germany Furtak, Erin Marie, University of Colorado, United States Gegenfurtner, Andreas, Technical University of Munich, Germany Gerofsky, Susan, University of British Columbia, Canada Gillen, Julia, Lancaster University, United Kingdom Ginns, Paul, The University of Sydney, Australia Goldman, Susan R, University of Illinois, United States

Goldman, Ricki, New York University, United States Gomez, Kimberley, University of Pittsburgh, United States Gomez, Kimberley, University of California, United States Gonzalez, Carlos, Pontificia Universidad Catolica, Chile Gresalfi, Melissa, Indiana University, United States Groza, Gabriela, University of Illinois, United States Gruson, Brigitte, IUFM de Bretagne, France Gutierrez, Jose, University of California, United States Hajime, Shirouzu, Chukyo University, Japan Hakkarainen, Kai, University of Turku, Finland Halverson, Erica, University Of Wisconsin, United States Halverson, Richard, University of Wisconsin, United States Hawi, Nazir, Notre Dame University, Lebanon Herman, Phillip, University of Pittsburgh, United States Herold, David, Hong Kong Polytechnic University, Hong Kong Herrmann, Thomas, University of Bochum, Germany Hershkovitz, Arnon, Worcester Polytechnic Institute, United Kingdom Hesse, Friedrich, Knowledge Media Research Center, Germany Hickey, Daniel, Indiana University, United States Hmelo-Silver, Cindy, Rutgers University, United States Hoadley, Christopher, Penn State University, United States Holbert, Nathan, Northwestern University, United States Hong, Huang-Yao, National Chengchi University, Taiwan Hora, Matthew, University of Wisconsin, United States Horn, Ilana, Vanderbilt University, United States Howard, Sarah, University of Wollongong, Australia Hulshof, Casper, University of Utrecht, The Netherlands Inge, Molenaar, University of Amsterdam, The Netherlands Ingerman, Åke, University of Gothenburg, Sweden Jackson, Kara, McGill University, Canada
Page viii

© ISLS

ICLS 2012

Volume 1: Full Papers

Jacobson, Michael, The University of Sydney, Australia Jamaludin, Azilawati, National Institute of Education, Singapore Jay, Tim, University of Bristol, United Kingdom Jochen, Rick, Saarland University, Germany Joiner, Richard, University of Bath, United Kingdom Jurow, A. Susan, University of Colorado, United States Kafai, Yasmin, University of Pennsylvania, United States Kanselaar, Gellof, Utrecht University, The Netherlands Kapur, Manu, National Institute of Education, Singapore Keifert, Danielle, Northwestern University, United States Kimmerle, Joachim, University of Tuebingen, Germany King Chen, Jennifer, University of California, United States Kirschner, Paul, Open University, The Netherlands Kollöffel, Bas, University of Twente, The Netherlands Kollar, Ingo, University of Munich, Germany Koschmann, Timothy, Southern Illinois University, United States Kraemer, Nicole, University Duisburg-Essen, Germany Krauskopf, Karsten, Knowledge Media Research Center, Germany Kwon, Samuel, Learning Point Associates, United States Kyza, Eleni, Cyprus University of Technology, Cyprus Lai, Jiang, Katholieke University of Leuven, Belgium Lakkala, Minna, University of Helsinki, Finland Langer-Osuna, Jennifer, University of Miami, United States Law, Nancy, University of Hong Kong, Hong Kong Lazonder, Ard, University of Twente, The Netherlands Leary, Heather, University of Colorado, United States Lee, Kyungmee, University of Toronto, Canada Lee, Chien Sing, National Central University, China Lee, Tiffany, University of Washington, United States Lee, Victor, Utah State University, United States

Lehrer, Rich, Vanderbilt University, United States Lei, Jing, Syracuse University, United States Liao, Chang-Yen, National Central University, China Liesbeth, Kester, Open University of the Netherlands, The Netherlands Limon, Margarita, University Autonoma of Madrid, Spain Lin, Hsien-Ta, National Chengchi University, Taiwan Lindgren, Robb, University of Central Florida, United States Lindwall, Oskar, University of Gotheburg, Sweden Liu, Shiyu, University of Minnesota, United States Liu, Ru-De, Beijing Normal University, China Lone, Dirckinck-Holmfeld, Aalborg University, Denmark Lonn, Steven, University of Michigan, United States Looi, Chee Kit, National Institute of Education, Singapore Louca, Loucas T., European University, Cyprus Lowe, Ric, Curtin University, Australia Lozano, Maritza, University of California, United States Luckin, Rose, London Knowledge Lab, United Kingdom Ludvigsen, Sten, University of Oslo, Norway Lund, Kristine, CNRS, France Magnifico, Alecia Marie, University of Illinois United States Magno, Carlo, De La Salle University, Philippines Manches, Andrew, University of Nottingham, United Kingdom Martell, Sandra, National Science Foundation, United States Martin, Crystle, University of Wisconsin, United States McElhaney, Kevin, University of California, United States McGee, Steven, The Learning Partnership, Canada McKenney, Susan, Open University of the Netherlands, The Netherlands Medina, Richard, University of Hawaii, United States Mercier, Emma, Durham University, United Kingdom Miyake, Naomi, the University of Tokyo, Japan Moher, Tom, University of Illinois, United States
Page ix

© ISLS

ICLS 2012

Volume 1: Full Papers

Moore, Joyce, University of Iowa, United States Mor, Yishay, Open University, United Kingdom Morch, Anders, University of Oslo, Norway Moskaliuk, Johannes, Knowledge Media Research Center, Germany Munoz-Baell, Irma M, University of Alicante, Spain Myint Swe, Khine, Bahrain Teachers College, Bahrain Nückles, Matthias, University of Freiburg, Germany Naidoo, Jayaluxmi, University of KwaZuluNatal, South Africa Nicolau, Juan L., University of Alicante, Spain Noroozi, Omid, Wageningen University, The Netherlands O'Malley, Claire, University of Nottingham, United Kingdom O'Neill, Kevin, Simon Fraser University, Canada Oeberst, Aileen, Knowledge Media Research Center, Germany Olimpo, Jeffrey, University of Maryland, United States Opfermann, Maria, University of DuisburgEssen, Germany Orrill, Chandra, University of Massachusetts, United States Oshima, Jun, Shizuoka University, Japan Oztok, Murat, University of Toronto, Canada Paavola, Sami, University of Helsinki, Finland Papademetri-Kachrimani, Chrystalla, European Univesrity Cyprus, Cyprus Parisio, Martin, The University of Sydney, Australia Pea, Roy, Stanford University, United States Pedaste, Margus, University of Tartu, Estonia Penuel, William, University of Colorado, United States Pluta, William, Rutgers, The State University of New Jersey, United States Polman, Joe, University of Missouri, United States Potgieter, Marietjie, University of Pretoria, South Africa Prins, Frans, Utrecht University, The Netherlands Radinsky, Josh, University of Illinois, United States Raes, Annelies, Ghent University, Belgium Ranney, Michael, University of California, United States Reber, Rolf, University of Bergen, Norway Reimann, Peter, The University of Sydney, Australia

Renninger, K. Ann, Swarthmore College, United States Rivet, Ann, Columbia University, United States Rogotneva, Elena, Tomsk Polytechnic University, Russia Rose, Carolyn, Carnegie Mellon University, United States Rummel, Nikol, University of Freiburg, Germany Rusman, Ellen, Open University of the Netherlands, The Netherlands Ryan, Stephanie, University of Illinois, United States Saleh, Asmalina, Indiana University, United States Sandoval, William, University of California, United States Sayre, Eleanor, Kansas State University, United States Schnotz, Wolfgang, University of KoblenzLandau, Germany Schoerning, Emily, University of Iowa, United States Schwarz, Baruch, Hebrew University, Israel Schwendimann, Beat, The University of Sydney, Australia Scott Curwood, Jen, University of Wisconsin, United States Seifert, Colleen, Univ. of Michigan, United States Sensevy, Gérard, University of Western Brittany, France Shapiro, Amy, University of Massachusetts, United States Shelton, Brett, Utah State University, United States Singer, Susan, Carleton College, United States Slof, Bert, Utrecht University, The Netherlands Smith, Debbie, Clemson University, United States Solomou, Maria, Indiana University, United States Spada, Hans, University of Freiburg, Germany Specht, Marcus, Open University, The Netherlands Stager, Sarah, Pennsylvania State University, United States Stevens, Reed, Northwestern University, United States Stieff, Mike, University of Illinois, United States Strijbos, Jan-Willem, Ludwig-MaximiliansUniversity München, Germany Sudol-DeLyser, Leigh Ann, Carnegie Mellon University, United States

© ISLS

Page x

ICLS 2012

Volume 1: Full Papers

Suthers, Daniel, University of Hawaii, United States Tan, Seng-Chee, National Institute of Education, Singapore Taylor, Martin, University of Texas, United States Tchounikine, Pierre, University of Grenoble, France Tee, Meng Yew, University of Malaya, Malaysia Thayer, Alexander, University of Washington, United States Thompson, Kate, The University of Sydney, Australia Tok, ükran, Pamukkale University, Turkey Trausan-Matu, Stefan, University "Politehnica" of Bucharest, Romania Trninic, Dragan, University of California, United States Tung, I-Pei (Vicky), McGill University, Canada Turns, Jennifer, University of Washington, United States Tzialli, Dora, European University Cyprus, Cyprus van Aalst, Jan, The University of Hong Kong, Hong Kong van Amelsvoort, Marije, Tilburg University, The Netherlands van Es, Beth, University of California, United States van Oers, Bert, University Amsterdam, The Netherlands van Oostendorp, Herre, Utrecht University, The Netherlands Vandewaetere, Mieke, K.U. Leuven, Belgium Vanover, Charles, University of South Florida, United States Vatrapu, Ravi, Copenhagen Business School, Denmark Venkataswamy, Arjun, Community Links High School, United States Virnes, Marjo, University of Eastern Finland, Finland Volker, Wulf, University of Siegen, Germany Walker, Richard, The University of Sydney, Australia Walkington, Candace, University of Wisconsin, United States

Wang, Tsungjuang, National Taipei University of Technology, Taiwan Wardrip, Peter, Northwestern University, United States Warren, Scott, Indiana University, United States Wecker, Christof, University of Munich, Germany Wee, Juan Dee, National Institute of Education, Singapore Weinberger, Armin, Saarland University, Germany Weinstock, Michael, Ben-Gurion University of the Negev, Israel Wessel, Daniel, Knowledge Media Research Center, Germany Wessner, Martin, Fraunhofer IESE, Germany White, Tobin, University of California, United States Wichmann, Astrid, Ruhr University Bochum, Germany Williams, Robert, Lawrence University, United States Wodzicki, Katrin, Knowledge Media Research Center, Germany Wong, Lung-Hsiang, National Institute of Education, Singapore Wouters, Pieter, Utrecht University, The Netherlands Wu, Hsin-Kai, National Taiwan Normal University, Taiwan Yael, Kali, University of Haifa, Israel Yeo, Ai Choo Jennifer, National Institute of Education, Singapore Yilmaz, Kaya, Marmara University, Turkey Young, Michael, University of Connecticut, United States Zagal, Jose, DePaul University, United States Zahn, Carmen, Knowledge Media Research Center, Germany Zannetou, Marilyn, EUC European University of Cyprus, Cyprus Zhang, Jianwei, University at Albany, United States Zingaro, Daniel, University of Toronto, Canada Zywica, Jolene, University of Pittsburgh, United States

© ISLS

Page xi

ICLS 2012

Volume 1: Full Papers

ABOUT ISLS

ISLS is a professional society dedicated to the interdisciplinary empirical investigation of learning as it exists in real-world settings and how learning may be facilitated both with and without technology. ISLS sponsors two professional conferences, held in alternate years. The International Conference of the Learning Sciences (ICLS), first held in 1991 and held biannually since 1996, covers the entire field of the learning sciences. http://www.isls.org/

ABOUT "COCO" AT THE UNIVERSITY OF SYDNEY

ICLS 2012 is hosted by the Centre for Research on Computer Supported Learning and Cognition (CoCo) at the University of Sydney. CoCo's mission is to contribute to theory and research in the field of the learning sciences in order to discover how innovative learning technologies and pedagogical approaches can enhance formal and informal learning. CoCo is a University of Sydney Research Centre operating within the Faculty of Education and Social Work. http://sydney.edu.au/edsw/coco

© ISLS

Page xii

ICLS 2012

Volume 1: Full Papers

PREFACE

Michael J. Jacobson, Peter Reimann, and Jan van Aalst The international and interdisciplinary field of the learning sciences brings together researchers from the fields of cognitive science, educational research, psychology, computer science, artificial intelligence, anthropology, neuroscience, and other fields to study learning in a wide variety of formal and informal contexts. This field emerged in the late 1980s and early 1990s, with the first International Conference of the Learning Sciences (ICLS) being held in 1991 at Northwestern University in Evanston, Illinois, USA. Subsequent meetings of ICLS were held again in Evanston, USA (1996), and then in Atlanta, USA (1998), Ann Arbor, USA (2000), Seattle, USA (2004), Santa Monica, USA (2004), and Bloomington, USA (2006). The first ICLS to be held outside of North America was in Utrecht, the Netherlands (2008), and then back to the USA in Chicago (2010). The ICLS 2012 in Sydney is the first hosting of the conference in the Asia-Pacific region. Papers for this conference were submitted in November 2011, and then went through a process of peer review. Full papers and symposia submissions received three anonymous reviews with a member of the Program Committee summarizing the reviews and making a recommendation. Short paper poster submissions went through the same process, except with two anonymous reviews. See Table 1 for a summary of the conference paper statistics. Table 1. Paper submissions, acceptance, and rejection rates.
Full papers 264 Accepted as Full Papers: 65 (25%) Accepted as Short Paper: 54 (20%) Accepted as Poster: 42 (16%) Rejected: 103 (39%) Short Papers 79 Accepted: 15 (19%) Accepted as poster: 11 (14%) Rejected: 53 (67%) Posters 76 Accepted: 37 (49%) Rejected: 39 (51%) Symposia 27 Accepted: 18 (67%) Rejected: 9 (33%) Total Submissions: 446

The final papers included in the ICLS 2012 Proceedings are for 18 symposia, 60 full papers, 61 short papers, and 62 poster papers, as well as abstracts for the four keynote talks and three special sessions, including the invited Presidential Session on The Future of Learning, and nine workshops. The themes reflected in the papers and presentations at ICLS 2012 cover a wide range of issues and research areas. Some papers deal with long standing theoretical issues, such as conceptual change and knowledge transfer, whereas other papers report on new learning research in conventional subject areas such as science, mathematics, and literacy. Research is also reported on newer knowledge areas such as complex systems, as well as more recent
© ISLS Page xiii

ICLS 2012

Volume 1: Full Papers

perspectives on pedagogy and learning such as knowledge building, inquiry, and productive failure. A number of learning sciences research projects internationally are also exploring ways in which innovative environments for learning may be designed with new technologies such virtual and game environments, modeling and visualization systems, robotics, collaboration technologies, mobile and hand held devices, and educational data mining and learning analytics. Emerging areas in learning sciences research that perhaps may lead to newer perspectives and directions in our understanding of learning include creatively, identity, and embodied cognition. The field continues to engage the broad issues of contributing to and impacting policy and practice more generally. Overall, the research presented in the proceedings of ICSL 2012 contributes numerous perspectives on the conference theme of the future of learning, both as trajectories of research that have been maturing over a number of years and bold new perspectives that promise to shape new trajectories for the future. Making this conference possible, we thank the hard work and the countless hours put in by our international Conference Advisory Board and the various conference subcommittees. We would like to thank all those who assisted with the review process. The Conference is also fortunate to have a number of financial sponsors whose support contributes to conference events: NSW Trade & Investment, Smart Services Collaborative Research Centre, United States National Science Foundation, The International Society of the Learning Sciences, Asia-Pacific Society of Computers in Education, Inquirium, LLC, and The Centre for Research on Computer Supported Learning and Cognition. Any conference is but a snapshot in the dynamic process of articulating and vetting scientifically principled ideas and approaches. That the International Conference of the Learning Sciences has now had its 10th meeting and has entered its third decade are exciting milestones that this proceedings helps to document. We close by reflecting that just as a great movie is about the journey of discovery and development that the characters experience, so might our research field be a journey of discovery and development to more deeply understand how people learn now and as the future unfolds, knowing learning itself as a core essence of life and our humanity.

© ISLS

Page xiv

ICLS2012

Volume 1: Full Papers

CONTENTS - VOLUME 1 Using Controversies for Knowledge Construction: Thinking and Writing about Alternative Medicine
Joachim Kimmerle, Johannes Moskaliuk, Martina Bientzle, Ansgar Thiel, Ulrike Cress

1

Understanding How Learners Grapple with Wicked Problems in Environmental Science
Brian Slattery, Chandan Dasgupta, Tia Shelley, Leilah Lyons, Emily Minor, Moira Zellner

9 17

Semiosis Process in Joint Didactic Action
Gérard Sensevy , Dominique Forest

Agent-Based Computer Models for Learning About Climate Change and Process Analysis Techniques
Nick Kelly, Michael Jacobson, Lina Markauskaite, Vilaythong Southavilay

25

Learning How to Create: Toward A Learning Sciences of Art and Design
Keith Sawyer

33

A Microgenetic Analysis of How Expansive Framing Led to Transfer with One Struggling Student
Diane Lam, Xenia Meyer, Randi A. Engle, Lloyd Goldwasser, Kathleen Zheng, Erica Naves, Danny Tan, Richard Hsu, Hernan Rosas, Sarah Perez

40

How Metacognitive Awareness Caused A Domino Effect in Learning
Kristin Schmidt, Andreas Lachner, Björn Stucke, Sabine Rey, Cornelius Frömmel, Matthias Nückles

48

Initial Validation of Listening Behavior Typologies for Online Discussions Using Microanalytic Case Studies
Alyssa Friend Wise, Ying-Ting Hsiao, Farshid Marbouti, Jennifer Speer, Nishan Perera

56

How to Schedule Multiple Graphical Representations? A Classroom Experiment With an Intelligent Tutoring System for Fractions
Martina Rau, Nikol Rummel, Vincent Aleven, Laura Pacilio, Zelha Tunc-Pekkan

64

Investigating the Relative Difficulty of Various Complex Systems Ideas in Biology
Sao-Ee Goh, Susan Yoon, Joyce Wang, Zhitong Yang, Eric Klopfer

72

Using Adaptive Learning Technologies to Personalize Instruction: The Impact of InterestBased Scenarios on Performance in Algebra
Candace Walkington, Milan Sherman

80

The Impact of Students’ Exploration Strategies in Discovery-Based Instructional Software
Barney Dalgarno, Gregor Kennedy, Sue Bennett

88 96 103

Design Research in Early Literacy within the Zone of Proximal Implementation
Susan McKenney, Paul Kirschner, Joke Voogt

Kitchen Chemistry: Supporting Learners’ Decisions in Science
Jason Yip, Tamara Clegg, Elizabeth Bonsignore, Helene Gelderblom, Becky Lewittes, Mona Leigh Guha, Allison Druin

Students’ Intuitive Understanding of Promisingness and Promisingness Judgments to Facilitate Knowledge Advancement
Bodong Chen, Marlene Scardamalia, Monica Resendes, Maria Chuy, Carl Bereiter

111

Designing Video Games that Encourage Players to Integrate Formal Representations with Informal Play
Nathan Holbert, Uri Wilensky

119

Dilemmas of Promoting Expansive Educational Transformation through Information and Communication Technologies (ICTs) in Botswana

127

© ISLS

Page xv

ICLS2012

Volume 1: Full Papers

Tshepo Batane, Ritva Engeström, Kai Hakkarainen, Denise Newnham, Paul Nleya, Jaakko Virkkunen

Visible Spending: Information Visualization of Spending Behavior and the Thought-Act Gap
Emily S. Lin, Holly A. Taylor

135

Beyond the Screen: Game-Based Learning As A Nexus of Identification
Ben DeVane

143

Working With Teenagers to Design Technology That Supports Learning About Energy in Informal Contexts
Katerina Avramides, Brock Craft, Rosemary Luckin, Janet Read

151

Unpacking the Use of Talk and Writing in Argument-based Inquiry: Instruction and Cognition
Ying-Chih Chen, Soonhye Park, Brian Hand

159

A Data-driven Path Model of Student Attributes, Affect, and Engagement in a Computerbased Science Inquiry Microworld
Arnon Hershkovitz, Ryan Baker, Janice Gobert, Adam Nakama

167

Exploring Connectedness: Applying ENA to Teacher Knowledge
Chandra Hawley Orrill, David Williamson Shaffer

175 180

Investigating the Effects of Varying Labels as Scaffolds for Visitor Learning
Joyce Wang, Susan Yoon, Karen Elinich, Jackie Van Schooneveld

Making Technology Visible: Connecting the Learning of Crafts, Circuitry and Coding in E188 textiles by Youth Designers
yasmin kafai, Deborah Fields, Kristin Searle

Functional Aesthetics for Learning: Creative Tensions in Youth e-Textile Designs
Deborah A. Fields, Yasmin B. Kafai, Kristin Searle

196 204 212

Learning Progressions, Learning Trajectories, and Equity
Cesar Delgado, Karisma Morton

A Teacher's Journey in Knowledge Building Pedagogy
Nancy Law, Johnny Yuen, Hidy Tse

Is Computer Support More Significant than Collaboration in Promoting Self-Efficacy and Transfer?
Andreas Gegenfurtner, Marja Vauras, Koen Veermans

220

Teacher Paradigm Shifts for 21st Practice Skills: The Role of Scaffolded Reflection Within 227 A Peer Community
Cheryl-Ann Madeira, Jim Slotta

The Impacts of Flexible Grouping in a Mobile-Assisted Game-based Chinese Character Learning Approach
Lung-Hsiang Wong, Ching-Kun Hsu, Jizhen Sun, Ivica Boticki

235

Exploratory Study on the Physical Tool-based Conceptions of Learning of Young Students 243 in a Technology-Rich Primary School
Lung-Hsiang Wong, Mingfong Jan, Yancy Toh, Ching-Sing Chai

Alternate Reality Games: Platforms for Collaborative Learning
Elizabeth Bonsignore, Derek Hansen, Kari Kraus, June Ahn, Amanda Visconti, Ann Fraistat, Allison Druin

251

Learning Innovation Diffusion as Complex Adaptive Systems through Model Building, Simulation, Game Play and Reflections
Junsong Huang, Manu Kapur

259

© ISLS

Page xvi

ICLS2012

Volume 1: Full Papers

Peer Collaboration and Mediation in Elementary Students’ Lamp Designing Process
Kaiju Kangas, Pirita Seitamaa-Hakkarainen

267 275 283

Dynabook: Supporting Teacher Learning about Mathematical Thinking
Jeremy Roschelle, Charles Patton, Elizabeth Murray

Embodied Artifacts and Conceptual Performances
Dragan Trninic, Dor Abrahamson

The Impact of Structural Characteristics of Concept Maps on Automatic Quality Measurement
H. Ulrich Hoppe, Jan Engler, Stefan Weinbrenner

291

Toward a Cognitive Framework of Interdisciplinary Understanding
Shannon Sung, Ji Shen, Dongmei Zhang

299 307

Using Innovation with Contrasting Cases to Scaffold Collaborative Learning and Transfer
Seungyon Ha, David A. Sears

Young Children's Everyday Inquiry: A Field Study of a Young Girl's Play Across Contexts 315
Danielle Keifert

Using the Idea Manager to Promote Coherent Understanding of Inquiry Investigations
Kevin McElhaney, Camillia Matuk, David Miller, Marcia Linn

323

Using Heuristic Worked Examples and Collaboration Scripts to Help Learners Acquire Mathematical Argumentation Skills
Ingo Kollar, Stefan Ufer, Elisabeth Lorenz, Freydis Vogel, Kristina Reiss, Frank Fischer

331

Technology Supports in CSCL
Heisawn Jeong, Cindy Hmelo-Silver

339 347 355

Situating epistemological development
William Sandoval

Improving Revision in Wiki-based Writing: Coordination Pays off
Astrid Wichmann, Marina Becker, Nikol Rummel

Challenging Assumptions: using sliding window visualizations to reveal time-based irregularities in CSCL processes
Gregory Dyke, Rohit Kumar, Hua Ai, Carolyn Rose

363

Distributing Practice: Challenges and Opportunities for Inquiry Learning
Vanessa Svihla, Marcia Linn

371 379

Technology for Learning: Moving from the Cognitive to the Anthropological Stance
Michael Eisenberg

Re-presenting Complex Scientific Phenomena Using Agent-Based Modeling in Engineering 387 Education
Paulo Blikstein

Metadiscourse to Foster Student Collective Responsibility for Deepening Inquiry
Jianwei Zhang, Jiyeon Lee, Jane Wilde

395

Consequential Feedback as a Means of Supporting Student Engagement and Understanding 403
Melissa Gresalfi, Jacqueline Barnes

Multiple Trajectories for Understanding Ecosystems
Catherine Eberbach, Cindy Hmelo-Silver, Rebecca Jordan, Suparna Sinha, Ashok Goel

411

Supporting Learners to Conceptualize the Vast Range of Imperceptible Smallness with Temporal-aural-visual Representations
Minyoung Song, Chris Quintana

419

© ISLS

Page xvii

ICLS2012

Volume 1: Full Papers

Inter-Identity Technologies for Learning
Robb Lindgren, Roy Pea

427 435 443 451 459

Predicting Idea Co-Construction in Speech Data using Insights from Sociolinguistics
Gahgene Gweon, Mahaveer Jain, John McDonough, Bhiksha Raj, Carolyn Rose

Automatically extract interpretable topics from online discussion
Yonghe Zhang, Nancy Law, Yanyan Li, Ronghuai Huang

Finding Voices and Emerging Agency in Classroom Learning
Tuck Leong Lee, Beaumie Kim, Mi Song Kim, Jason Wen Yau Lee

Processes of decision-making with adaptive combinations of wiki and chat tools
Kate Thompson, Nick Kelly

Examining the Adequacy of Students' Priors and Teacher's Role in Attention to Critical Features in Designing for Productive Failure
Pee Li Leslie Toh, Manu Kapur

467

Author Index

475

© ISLS

Page xviii

ICLS2012

Volume 1: Full Papers

Using Controversies for Knowledge Construction: Thinking and Writing about Alternative Medicine
Joachim Kimmerle, University of Tuebingen, Schleichstr. 4, 72076 Tuebingen, j.kimmerle@iwm-kmrc.de Johannes Moskaliuk, University of Tuebingen, Schleichstr. 4, 72076 Tuebingen, j.moskaliuk@iwm-kmrc.de Martina Bientzle, KMRC, Schleichstr. 6, 72076 Tuebingen, m.bientzle@iwm-kmrc.de Ansgar Thiel, University of Tuebingen, Wilhelmstraße 124, 72072 Tuebingen, ansgar.thiel@uni-tuebingen.de Ulrike Cress, KMRC, Schleichstr. 6, 72076 Tuebingen, u.cress@iwm-kmrc.de Abstract: This paper presents a case study in which two participants with opposing points of view used a shared document for purposes of dealing with a controversy on the effectiveness of homeopathic medicine. The participants were working by turns on a common presentation that should outline their opinions on the topic. During this alternating process they had to think aloud to make individual processes observable. In addition, we analyzed the development of the shared text. We combined a content analysis and the think-aloud data to measure internal and external processes of knowledge construction, and we describe how these processes influenced each other. This procedure led to insights into the development of perception shared by the participants and into the resulting knowledge construction with respect to quantitative and qualitative processes. In particular, we analyzed the interplay between the participants’ thinking and writing processes as well as the co-evolution of their thinking and writing as cognitive and social processes. In our conclusion we discuss the implications of our findings for understanding processes of knowledge construction.

Introduction
Collaboratively developed shared digital artifacts may facilitate collaborative learning and the interplay between individual and collective knowledge processes (Kimmerle, Cress, & Held, 2010; Nussbaum, Winsor, Aqui, & Poliquin, 2007). In previous publications we have outlined a co-evolution model of cognitive and social systems, which is a theoretical approach that takes into account both the social processes facilitated by collaboratively developed digital artifacts and the cognitive processes of the individuals involved (Cress & Kimmerle, 2007, 2008). The aim of the current paper is to demonstrate in a detailed analysis how these social and cognitive processes may mutually influence each other in a process of collaboration and construction of knowledge. In an empirical study, we took into consideration both interdependent processes simultaneously: One person’s trains of thought during collaboration as well as concurrent contributions this person made to a shared digital artifact. We related these processes to the corresponding reactions of another person with an opposing point of view, revealing this second person’s trains of thought and contributions to the artifact, and so on. The basic idea is that it is a conflictual situation that initiates such an interdependency and interplay of social and cognitive processes. Piaget (1977) pointed out the significance of socio-cognitive conflicts for the development of knowledge. The utilization of cognitive conflicts for facilitating learning processes has been debated in the Learning Sciences for decades (Johnson & Johnson, 1987; Wadsworth, 1978; cf. also O’Donnell & O’Kelly, 1994). We are convinced that conflicts also play an important role when it comes to collaboration and knowledge construction with the use of shared digital artifacts (for the role of confronting cognitions in computer-supported collaborative learning settings cf. Andriessen, Baker, & Suthers, 2003). The controversy between two or more individuals is supposed to trigger the construction of emergent knowledge, both in the cognitive system of a user and in the social systems represented by the shared digital artifact. Tools that particularly suggest themselves for this purpose are wikis, as they are established as teaching tools and learning environments (Forte & Bruckman, 2010; Parker & Chao, 2007; cf. also Glassman & Kang, 2011; Pifarré & Kleine Staarman, 2011) and greatly allow dealing with conflicts creatively and constructively (Moskaliuk & Kimmerle, 2009): Wikis provide a space in which ideas can be formulated, revised, or rejected. The people involved can contribute to the advancement of knowledge in the social system. Here, advancing knowledge is supposed to take place in the form of an improvement of ideas, not as the development of a true or perfect solution (cf. Bereiter & Scardamalia, 1996; Scardamalia, 2002). In the following section we will briefly describe the underlying theoretical considerations with regard to the relationship of cognitive and social systems, as they apply to the interplay between individual and collective knowledge. Subsequently, we will describe the method that was applied in the empirical study reported here and present its main results regarding the processes of knowledge construction. Concluding, we will discuss the implications of our analysis for further research on knowledge construction.

© ISLS

1

ICLS2012

Volume 1: Full Papers

The Interplay between Cognitive and Social Systems
The collaborative construction of new knowledge may be described as an interplay between cognitive systems of individuals and a social system (Kimmerle, Moskaliuk, & Cress, 2009; Moskaliuk, Kimmerle, & Cress, 2008). We argue that research should focus equally on both systems by considering individual cognitive processes (such as perception, attention, learning, thinking, decision-making) as well as the collective development of knowledge. In this context we consider shared artifacts not just as a means to an end, but as the space where the collective development of knowledge takes place or, in other words, the space where collective knowledge and social structures manifest themselves. Our main assumption is that knowledge construction can only be understood if we take a very detailed look at the interplay between individuals and the collective. From a systems-theoretical point of view (Luhmann, 1995) the cognitive systems of individuals need to be distinguished from a social system—which is here represented by the interaction that takes place with the help of a shared digital artifact (cf. Kimmerle, Moskaliuk, & Cress, 2011). Cognitive systems and social systems, however, can affect each other in their development by providing ‘irritating’ information to the other system (Moskaliuk, Kimmerle, & Cress, 2012). This is the process we want to take into account in our analysis. We regard irritations, along with Piaget (1977), as cognitive conflicts. We assume that cognitive systems will develop when individuals solve cognitive conflicts. Cognitive conflicts occur when individuals realize that their prior knowledge and the information from the environment which they have to deal with are contradictory to some extent. People may solve these conflicts by means of two sorts of equilibration processes: They may either assimilate information (basically attach new information to their prior knowledge, i.e., a quantitative process) or accommodate prior knowledge to novel information, which comes along with qualitative improvements. Both processes contribute to the development of a cognitive system, that is, they represent processes of individual learning. We argue that those processes that occur in cognitive systems through internalization may happen in social systems in an analogous way through externalization of an individual’s own knowledge into a shared artifact. So, social systems can be shaped by integrating relevant information or by separating inappropriate information—in either an assimilative or accommodating way: They can develop by merely adding new content or by removing contradictory information (‘external assimilation’). Or, social systems may emerge by modifying their own structure (‘external accommodation’). Such external accommodation processes are supposed to lead to a more pronounced complexity of the shared digital artifact and, potentially, as a consequence, to new cognitive conflicts in other people. We emphasize that internalization and externalization, that is, individual learning and collaborative knowledge construction, are strongly interdependent. Thus, we need to examine internalization and externalization as interdependent processes that take place at the same time in order to understand this continuous exchange process between cognitive and social systems. This continuous exchange process is exactly what we want to trace and illustrate here. In the past years we have conducted a variety of empirical investigations into the interplay between cognitive and social systems, applying a range of methods, such as laboratory experiments (Moskaliuk, Kimmerle, & Cress, 2009) or Social Network Analysis (Kimmerle, Moskaliuk, Harrer, & Cress, 2010). Based on our experiences from previous studies, we have conducted an empirical examination in which we considered processes of individual thinking and related processes of social exchange at the same time. We will explicate the processes of internalization and externalization by examining how two people collaboratively developed a shared text on a controversial issue in a sequence of individual sessions. In doing so, we simultaneously collected thinking-aloud data and contributions to a text.

Method
The ‘wiki-approach’ can be applied well to situations in which people tend to have controversial opinions on a certain subject. They can introduce their own positions on equal terms (Moskaliuk & Kimmerle, 2009): “Participants may express opposing opinions, addressing each other, and they incorporate their own perspective into a coherent text [and] Individuals may acquire new knowledge when they internalize information from the wiki” (Kimmerle, Cress, Held, & Moskaliuk, 2010, pp. 13-14). Therefore, we applied the same wiki principle here and in this case used a collaborative writing tool in order to support participants in writing a shared text on a controversial topic. We provided them with a document that allowed them to revise all parts of a text, add, change, or delete anything (like with a wiki). We were selectively looking for participants who were interested in naturopathy and homeopathy and had an unambiguous opinion either for or against homeopathy. We chose this domain, as there was recently a controversial debate on the role of homeopathy in the public health sector in Germany, and much information on this topic was discussed in the media coverage. This allowed us to recruit participants with extensive prior knowledge on this topic who were motivated to take part in this discussion. We advertised the participation in this study and offered 60 EUR for a three-hour session. The advertisement pointed out that the participants’ task would be to collaboratively write a text on the issue “Should homeopathy still be allowed in Germany”. The

© ISLS

2

ICLS2012

Volume 1: Full Papers

advertisement explained that participants were supposed to advance their own point of view, but be willing to engage in a discussion with others. The advertisement outlined as the goal of the collaborative writing process the preparation of an article that expressed the opinions of the people involved and summarized the basic arguments. Candidates had to fill in a short questionnaire to apply for participation in this study. We asked them about their prior knowledge on the topic and for their personal opinion. In addition, they had to indicate their agreement or disagreement with two statements (pro and contra homeopathy) using a 7-point Likert Scale. We used the data of this questionnaire to select two university students as participants in our study: One woman (age: 25; German education and English education student) and one man (age: 32; chemistry student). The two participants had different opinions about the given issue. Their applications showed that they had detailed knowledge about homeopathy, were able to outline their opinions in an adequate way (for the role of argumentation in computer-supported collaboration cf. Stegmann, Weinberger, Fischer, 2007; Weinberger, Stegmann, Fischer, 2010), and were highly motivated to participate in this study. We used eduPad (http://edupad.ch/) as a collaborative writing tool. eduPad provides a simple text editor and allows participants to change the text online. The text written by the two participants was marked with two different colors during the writing process to identify the author of specific parts of the text. This supported the quick identification of ongoing changes in the text. All changes of the text made by the two participants were stored as different versions of the text. The two participants had to write in turns on the shared text: There were eight phases, with four writing phases for each of the two participants. Their goal was to develop a text that outlined their combined reasoning on the given question (“Should homeopathy still be allowed in Germany?”). They were instructed that the resulting text should represent both opinions and integrate the different lines of argument. Each of the two participants had ten minutes available per round to write on the text (however, if participants asked for more time in order to finish their current entry, we permitted up to four additional minutes). One participant had to wait in a separate room while the other one was working on the text. Participant 1 (P1), who advanced a pro homeopathy view, started with an almost empty file that contained no content except for a short headline: “Homeopathy in crisis”. After the first phase, participant 2 (P2), who advanced a contra homeopathy view, proceeded, followed again by P1 and so on. We instructed the participants to think aloud (cf. Ericsson & Simon, 1980) during their work with eduPad and recorded their comments digitally (for a discussion on the suitability of the think-aloud method cf. Ericsson & Fox, 2011; Schooler, 2011). In addition, a screen-capturing tool was used to log the writing process in detail. The verbal comments were transcribed (think-aloud protocols) and matched with the data from the screen-capturing tool and the stored text versions. This led to a rich data base that combined internal processes (think-aloud protocols) and external processes (text development).

Analysis
In this section we will present the core findings of our analysis. As described above, both individual learning and collaborative knowledge construction may either take place in the form of assimilation or accommodation. The former is chiefly a quantitative process while the latter is supposed to be a process that leads to qualitative improvement. This section is organized accordingly. First, we will briefly describe the development of the wiki text over time in quantitative terms, that is, the development of the absolute length of the text and the increments of text in the eight particular episodes. In doing so, we will illustrate the alternations between phases of quantitative and qualitative development. Subsequently, we will provide a qualitative analysis in which we will examine critical incidents in the knowledge construction process. In this process, we will analyze situations in which the participants decided to deal with controversies by either writing new text or deleting existing text. We will describe the interplay of thinking and writing and examine situations in which the participants decided not to introduce their opinions or knowledge into the text. In addition, we will analyze how participants compromised in the collaboration process and discuss our findings regarding this process as a co-evolution of thinking and writing. Finally, we will describe how norms of collaboration developed during the writing process.

Alternations between Quantitative and Qualitative Development
As was mentioned above P1 started with an almost empty document that contained just a short headline. In the beginning of the writing process there was, naturally, mainly a quantitative increase. After the first two rounds (one for each participant) the document contained already 40 % of the final text (311 of 780 words). In the last two versions (rounds 7 and 8) the addition of text was only about 5 % (42 of 780 words). In these final phases of the writing process, however, qualitative development instead of quantitative expansion of the text became more relevant. This increase of the text length can be seen in Figure 1. Such alternations between quantitative and qualitative writing activities did not merely occur between the first and the last versions, but over the entire writing process. Alternations between quantitative and qualitative phases of

© ISLS

3

ICLS2012

Volume 1: Full Papers

text-production are also known from previous research on individual writing processes (cf. Hayes & Flower, 1986).

Figure 1. Development of text length (cumulative) across the eight versions of the text. In the beginning of the collaborative writing process P1 merely added initial information (version 1), because there was no text that this participant could revise in any way. Afterward, there was an alternation from mainly quantity-orientated and to rather quality-orientated periods. We found a first shift from quantitative to qualitative writing activities in text version 3: In version 1, P1 had started with a short definition of homeopathy and tried to outline the controversy of the topic concerning the efficacy issue; in version 2, P2 complemented the efficacy discussion with an additional aspect, the placebo effect. In her second turn (version 3) P1 wrote a kind of conclusion trying to integrate the different points of view: P1, wrote text: “It’s a fact that homeopathy and other alternative therapeutic methods are controversial topics. There is no clear answer as to whether they are a true alternative, because their advocates just focus on the effects, whereas the opponents focus only on the facts which are not measurable. Everybody should decide for himself whether he is for or against it.” The think aloud protocols (TA) supported the assumption that P1 tried to integrate the conflicting approaches here: P1, TA: “I don’t want to contrast both perspectives, but bring them together somehow.” In version 4, in turn, we found a shift from qualitative to quantitative writing activities. In this version P2 contributed two new perspectives on the homeopathy controversy, a financial and an ethical aspect. In version 5 P1 complemented in turn the financial aspect. Then, in version 6 the next shift from quantitative to qualitative activities took place: Here, P2 tried to merge the different aspects of the text after having recognized: P2, TA: “It’s really noticeable that this whole thing doesn’t fit together.” Altogether, we may conclude that taking up a new perspective of the topic (by one of the writing partners) triggered quantitative processes in a first step which was, however, closely followed by qualitative processes, such as structuring the text or merging different points of view. So a quantitative focus alternates with a qualitative focus across the whole collaborative writing process. This is in line with our theoretical differentiation of external assimilation and external accommodation. The co-evolution model describes these processes as two interrelated aspects of knowledge construction in a social system.

The Interplay of Thinking and Writing
When people want to contribute to a shared document, they have to externalize particular aspects of their own knowledge. For this purpose they have to transform their thoughts into written language. In doing so, they have to reflect on pre-existing information in the text to ensure that they integrate their cognitive concepts adequately. In this section we focus on the process of externalization, considering, on the one hand, the situations and social

© ISLS

4

ICLS2012

Volume 1: Full Papers

conditions that seem to foster externalization and, on the other hand, situations that seem to inhibit externalization processes. In our assessment we take into account not only writing but also deleting activities. As we have captured thinking-aloud data simultaneously while participants edited the text, we have the opportunity to match internal and external processes in this analysis. So we can see that in many cases the process of externalization was clearly associated with the respective mindset of the particular participant, for example: P1, TA: “Homeopathy was—no, um—oh, how should I start… Homeopathy… Yes, right! I will start with a definition, namely with a translation of the term...” Accordingly, this is what P1 was writing directly thereafter: P1, wrote text: “Homeopathy means literally: To treat the same with the same.” A consistency of thinking and acting may also find expression in deleting text as can be seen from the following example: P2, TA [after reading the text]: “I don‘t think that’s so great, for example. And it colors everything, as well.” P2, deleted text: “One should first give homeopathy a try, and if it doesn’t have any effect, then fall back on allopathic medicine.” This strong association of internal and external processes is in line with the idea of the co-evolution model that externalization and internalization depend on each other. If the participant had a deviating opinion about the information introduced by the other participant or wanted to introduce new knowledge, s/he added, edited, or deleted the shared text. There is, so to speak, a direct interrelation between the internal processes recorded using the think-aloud protocols and the resulting external processes, that is, writing text. At the same time, we observed episodes in which externalization seemed to be influenced by additional processes. We argue that individual perception of the shared text influenced externalization. The previous interactions between the two participants, the current status of the collaboration, and the occurring socio-cognitive conflict became manifest in the shared document. This led to a mutual development of internal and external processes. In the following, however, we will report episodes in which internal processes differed from resulting external processes.

Co-Evolution of Thinking and Writing
A peculiar gap between the writing process and the present thinking of a participant could be observed several times during the knowledge construction process. In some cases, the participants decided to keep their opinions or their current thoughts out of the collaborative situation: P1, TA: “You can’t pigeon-hole people who use homeopathy. Like saying they are some kind of strange people. But I am not going to add any more about that now.” “Anyway, I’ll leave it like that, otherwise he will change it again.” In other situations we observed great differences or a missing relationship respectively between thinking and writing: P1, TA: “No, no. I do not agree with one thing he wrote there. With anything. Anything at all. I could delete all of his changes now and, and... Oh, never mind. Ok, what should I do with that?” But, instead, P1 addressed a completely different topic in the next moment: P1, wrote text: “Still, one should consider also that allopathic drugs are much more expensive, because…” In this example P1 decided not to add her own opinion, as she assumed that the other participant would delete her additions anyway. P1 seemed to look for an agreement, as the goal of the collaboration was to come to a common statement on the given topic. Instead of resuming the confrontation she tried to integrate the different perspectives. The observed gap between thinking and writing resulted from a kind of filter that decided which part of one’s own knowledge should be introduced to the shared text. Even if one’s own opinion or knowledge was different from that of the other participant, the written text was adjusted to accommodate the common goal—that is, coming to an agreement. In the first example, P1 perceived the socio-cognitive conflict between her own opinion and that of her counterpart but decided to ignore the difference. In the second example, she decided to elaborate more on the topic and introduced an additional counter-argument. Based on the co-evolution model, we might conclude that the social system (the shared text, the current collaboration, and previous experience during the interaction) influenced how thinking was related to writing. Individual perception of the social system filtered if and how the knowledge from the cognitive system was externalized into the shared document. We argue that the observed gap between thinking and writing is in line with the co-evolution model, as there was a mutual influence of internal and external processes. The externalization process depended on the current status of the social system. The collaborative writing processes were strongly influenced by the social conditions implemented here. Thus, we also observed situations in which internal processes and external

© ISLS

5

ICLS2012

Volume 1: Full Papers

process strongly influenced each other. In the following we present two additional episodes where this coevolution of thinking and writing became evident. P2 made clear his skeptical attitude towards homeopathy in the text: P2, wrote text: “[…] effect does not measurably differ from placebos […]” “[…] the most radical advocates of homeopathy do not accept this and often use outrageous pseudo-scientific ‘facts’ to sell their often expansive preparations […]” “[…] so-called alternative medicine.” When P1 returned to the computer and read the revised text for the first time (in phase 3), the conflict of opinion became evident (‘<…>’ means that a participant is reading the text aloud): P1, TA: “‘<What is indisputable is the placebo effect… most radical advocates…>’… What!? What?... so ‘<his self-healing powers…>’ yes, okay, well. If he thinks so. I don’t think so, but if he thinks so, mmh…” “… that is so … well, but, so, what strikes me is… he is obviously against homeopathy… yes, that is obvious. Well! I’m gonna write something into it.” P1, wrote text: “[…] effect does also occur with allopathic drugs”. P1, deleted: “so-called” in “so-called alternative medicine” and deleted: “outrageous” in “outrageous pseudo-scientific ‘facts’”. Even if P1 advanced a pro homeopathy view, she did only delete “outrageous” and did not edit the attribution “pseudo-scientific”. Again this seemed to be an attempt to find a compromise between the two contrary positions. P1 subordinated her own opinion to the goal of writing a common statement. We observed similar processes for P2. Even if he had a contra homeopathy view, he did try to find formulations with which he could anticipate the agreement of P1. In another episode P2 highlighted the empowerment of patients and the obligation of physicians and politics to extensively inform patients about homeopathy: P2, TA: “Enlightenment is an important point” P2, wrote text: “[…] a way that enlightens patients […] nonsensical and harmful […]” P2, TA: “Alas! It’s no use.” P2, deleted: “that enlightens patients […] nonsensical and harmful […]” P2, wrote instead: “In any case the patient should be the priority.” Probably, as result of the anticipation that P1 would not accept the description of homoeopathy as “nonsensical” or “harmful”, P2 deleted this part of the text and searched for more moderate wording. This, again, points out that internal processes are influenced by external, that is, (anticipated) social processes. P2 started to write down his opinion about homeopathy as a nonsensical and harmful method, but then deleted the written text and attenuated his point. Filtering his own knowledge before externalizing it to the shared document alleviated the perceived socio-cognitive conflict between his own knowledge and the information in the wiki. Again, the thinking processes seemed to be influenced by the perception of the social system. In this way, the previous collaboration, the externalized knowledge of both participants, and their common goals led to a more moderate formulation. In the presented data we observed that the participants tried to come to an agreement and made compromises. As the goal of their collaboration was to write a common text, the participants subordinated their individual goals. They solved occurring conflicts by filtering their own knowledge, or even by going so far as to not externalize specific arguments. Thus, internal processes and the resulting process of externalization were strongly interdependent. Both processes took place simultaneously as a continuous exchange between cognitive and social systems.

Conclusion
In this paper, we have elaborated the interdependence between internal and external processes of knowledge construction. We examined the internal process by asking the participants to think aloud and the external processes by recording the writing process closely. This allowed us to observe internalization and externalization as interdependent processes that take place at the same time. Our data highlight the continuous exchange processes between cognitive and social systems and the resulting process of knowledge construction. We interpret the observed processes as a co-evolution of cognitive and social systems. We created a specific setting in which two people with extensive knowledge on a given topic, but with contrary opinions, had to collaborate and construct a common statement. This setting provided ideal conditions to study the postulated process of knowledge construction. The socio-cognitive conflict was a key incitement factor of the collaborative construction of knowledge and had to be resolved by the two participants. Thus, the participants made compromises and integrated their different arguments. This led to the reported gap between thinking and writing as an indicator for the influence of the social system on individual processes. In the described collaboration of the two participants we observed a development of collaboration norms during the writing process. Even if there was no direct interaction via chat or face to face, the collaboration using the shared text was strongly influenced by the perceived or anticipated reactions of the other person and by the knowledge which was presented. The current status of the

© ISLS

6

ICLS2012

Volume 1: Full Papers

shared document as a social system strongly influenced how an individual chose to introduce his/her own knowledge into a shared artifact. We also observed a behavior that might be called a strategic component of knowledge construction: A controversial idea with an anticipated low chance of survival in the knowledge construction process was not even externalized. To sum up, new knowledge may develop when people interact with each other via a shared digital artifact such as a wiki text. The integration of different arguments is triggered by the perceived cognitive conflict. We could describe this process of knowledge construction as co-evolution of internal and external processes. The method of combining a think-aloud protocol with a close observation of the writing process emphasizes that research on knowledge construction should consider internal and external process at the same time. Future studies should aim at extending and modifying our research procedure in two respects. First, further analyses might consider examining a collaborative situation in which the participants do not have such a strong commitment to their respective positions. In many situations in research on computer-supported collaborative learning this is rather unusual, for example in scripted scenarios where one person is given different information than another. Second, in our research setting knowledge construction was more constricted than in typical collaborative interactions with wikis. Usually people have more freedom to organize their collaboration than with our predefined time periods. Future studies might apply our approach to examine collaboration scenarios with higher degrees of ecological validity—for example with wikis in educational contexts or on the Internet.

References
Andriessen, J., Baker, M., & Suthers, D. (2003). Arguing to learn: Confronting cognitions in computersupported collaborative learning environments. Dordrecht, Netherlands: Kluwer Academic Publishers. Bereiter, C., & Scardamalia, M. (1996). Rethinking learning. In D. Olson, & N. Torrance (Eds.), Handbook of education and human development: New models of learning, teaching and schooling (pp. 485–513). Cambridge, MA: Basil Blackwell. Cress, U., & Kimmerle, J. (2007). A theoretical framework of collaborative knowledge building with wikis—a systemic and cognitive perspective. In C. A. Chinn, G. Erkens, & S. Puntambekar. (Eds.), Proceedings of the 7th Computer Supported Collaborative Learning Conference (pp. 153–161). New Brunswick, NJ: International Society of the Learning Sciences. Cress, U., & Kimmerle, J. (2008). A systemic and cognitive view on collaborative knowledge building with wikis. International Journal of Computer-Supported Collaborative Learning, 3, 105–122. Ericsson, K. A., & Fox, M. C. (2011). Thinking aloud is not a form of introspection but a qualitatively different methodology: Reply to Schooler (2011). Psychological Bulletin, 137, 351–354. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215–251. Forte, A., & Bruckman, A. (2010). Citing, writing and participatory media: Wikis as learning environments in the high school classroom. International Journal of Learning and Media, 1(4), 23–44. Glassman, M., & Kang, M. J. (2011). The logic of wikis: The possibilities of the Web 2.0 classroom. International Journal of Computer-Supported Collaborative Learning, 6, 93–112. Hayes, J. R., & Flower, L. S. (1986). Writing research and the writer. American Psychologist, 41, 1106–1113. Johnson, D. W., & Johnson, R. T. (1987). Creative conflict. Edina, MN: Interaction Book Co. Kimmerle, J., Cress, U., & Held, C. (2010). The interplay between individual and collective knowledge: Technologies for organisational learning and knowledge building. Knowledge Management Research and Practice, 8, 33–44. Kimmerle, J., Cress, U., Held, C., & Moskaliuk, J. (2010). Social software and knowledge building: Supporting co-evolution of individual and collective knowledge. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (Vol. 1, pp. 9–16). Chicago, IL: International Society of the Learning Sciences. Kimmerle, J., Moskaliuk, J., & Cress, U. (2009). Learning and knowledge building with social software. In C. O'Malley, D. Suthers, P. Reimann, & A. Dimitracopoulou (Eds.), Computer Supported Collaborative Learning Practices: CSCL2009 Conference Proceedings (Vol. 1, pp. 459–468). Rhodes, Greece: International Society of the Learning Sciences. Kimmerle, J., Moskaliuk, J., & Cress, U. (2011). Using wikis for learning and knowledge building: Results of an experimental study. Educational Technology & Society, 14(4), 138–148. Kimmerle, J., Moskaliuk, J., Harrer, A., & Cress, U. (2010). Visualizing co-evolution of individual and collective knowledge. Information, Communication and Society, 13, 1099–1121. Luhmann, N. (1995). Social systems. Stanford: Stanford University Press. Moskaliuk, J., & Kimmerle, J. (2009). Using wikis for organizational learning: Functional and psycho-social principles. Development and Learning in Organizations, 23(4), 21–24.

© ISLS

7

ICLS2012

Volume 1: Full Papers

Moskaliuk, J., Kimmerle, J., & Cress, U. (2008). Learning and knowledge building with wikis: The impact of incongruity between people’s knowledge and a wiki’s information. Proceedings of the International Conference for the Learning Sciences 2008, Vol. 2 (pp. 99–106). Utrecht, The Netherlands: International Society of the Learning Sciences. Moskaliuk, J., Kimmerle, J., & Cress (2009). Wiki-supported learning and knowledge building: Effects of incongruity between knowledge and information. Journal of Computer Assisted Learning, 25, 549– 561. Moskaliuk, J., Kimmerle, J., & Cress, U. (2012). Collaborative knowledge building with wikis: The impact of redundancy and polarity. Computers & Education, 58, 1049–1057. Nussbaum, E. M., Winsor, D. L., Aqui, Y. M., & Poliquin, A. M. (2007). Putting the pieces together: Online argumentation vee diagrams enhance thinking during discussions. International Journal of ComputerSupported Collaborative Learning, 2, 479–500. O’Donnell, A. M. & O’Kelly, J. (1994). Learning from peers: Beyond the rhetoric of positive results. Educational Psychology Review, 6, 321–349. Parker, K. R., & Chao, J. T. (2007). Wiki as a teaching tool. Interdisciplinary Journal of Knowledge and Learning Objects, 3, 57–72. Piaget, J. (1977). The development of thought: Equilibration of cognitive structures. New York: Viking Press. Pifarré, M., & Kleine Staarman, J. (2011). Wiki-supported collaborative learning in primary education: How a dialogic space is created for thinking together. International Journal of Computer-Supported Collaborative Learning, 6, 187–205. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 67–98). Chicago: Open Court. Schooler, J. W. (2011). Introspecting in the spirit of William James: Comment on Fox, Ericsson, and Best (2011). Psychological Bulletin, 137, 345–350. Stegmann, K., Weinberger, A., & Fischer, F. (2007). Facilitating argumentative knowledge construction with computer-supported collaboration scripts International Journal of Computer-Supported Collaborative Learning, 2, 421–447. Wadsworth, B. (1978). Piaget for the classroom teacher. New York: Longman. Weinberger, A., Stegmann, K., & Fischer, F. (2010). Learning to argue online: Scripted groups surpass individuals (unscripted groups do not). Computers in Human Behavior, 26, 506–515.

© ISLS

8

ICLS2012

Volume 1: Full Papers

Understanding How Learners Grapple with Wicked Problems in Environmental Science
Brian Slattery, Chandan Dasgupta, Tia Shelley, Leilah Lyons, Emily Minor, Moira Zellner University of Illinois at Chicago, 851 S. Morgan, Chicago, IL 60607 bslatt2@uic.edu, cdasgu2@uic.edu, tshell2@uic.edu, llyons@uic.edu, eminor@uic.edu, mzellner@uic.edu Abstract: Environmental science standards are calling for a perspective that highlights how social and natural systems interact. In order to properly deal with the “wicked problems” arising from this interaction, learners must recognize that there is “no right answer”, since solutions require compromise. They must also use spatial concepts instrumentally to reason about these systems. We propose to address these challenges by adapting authentic complex human-natural systems models into collaborative learning experiences. To do so, we need to better understand the challenges learners face as they use simulations to link spatial reasoning with dynamic processes. This paper presents two cases where we examine learners’ spatial and problem-solving strategies as they interact with a modified stormwater management model. We show that learners require support for core spatial reasoning skills and for problem solving around wicked problems. We then recommend forms of scaffolding and further development.

Introduction
Over the past decade there has been increased acknowledgement that “mile wide, inch deep” coverage of largely factual content is insufficient to prepare future scientists and scientifically literate citizens (NAE, 2008), causing the College Board to embark on the redesign of four AP (Advanced Placement) science courses (Biology, Chemistry, Environmental Science, and Physics) in June 2006. The changes made to the AP Environmental Science standards have placed increased recognition on the importance of taking a complex systems perspective on scientific phenomena such as ecosystems (College Board, 2009). Agent-Based Models (ABMs), one approach to complex systems modeling, can facilitate multidisciplinary learning about the science and policy issues arising from human-environment interactions, but research is needed on how to make these models intellectually and pragmatically accessible to learners. This project addresses that need by developing and testing new tools and pedagogical strategies using an iterative design-based research approach. Our first exploration of these issues engages learners in Green Infrastructure (GI) planning, an authentic problem-based context incorporating Urban Planning (UP) and Environmental Science (ES) disciplines. GI is defined as "an interconnected network of green spaces that conserves natural ecosystem values and functions and provides associated benefits to human populations" (Schilling & Logan, 2008). GI often takes the form of vegetated swales or green roofs that aim to minimize urban stormwater runoff and associated pollution. Since communities do not have infinite resources, they must make strategic decisions about where GI elements will make the most impact. In our collaboration with disciplinary experts from both UP and ES, we have identified certain key reasoning skills required to grapple with challenges at the intersection of human and natural systems. While others have investigated challenges of learning about complex systems (Hmelo-Silver, Marathe, & Liu, 2007), two things set our learning challenges apart. First, the complex systems present in UP and ES are fundamentally defined by their spatial properties and relations. Second, in order to understand how human (UP) and natural (ES) systems interrelate, one must study how their spatial properties and relations intersect. Similarly, the integration of policy or economics transforms these complex problem spaces into “wicked problems”, a class of ill-structured problems originally defined in UP literature (described further below). This exploratory study focused on a subset of spatial reasoning skills, and on one critical skill for grappling with wicked problems: learning to compromise across multiple competing demands. The ABM we used in this study was adapted from a GI planning ABM, L-GrID, developed for the Illinois Environmental Protection Agency by the project’s disciplinary expert co-PIs (Zellner et al., in progress). We recruited pairs of undergraduates (to act as proxies for high school seniors who might be in AP Environmental Science courses), and asked them to engage in an authentic GI problem-solving task with the ABM. Because this was a lab-based study, to encourage them to take the problem solving task seriously, we borrowed from methods used in experimental economics. The ABM was structured to contain two competing reward functions, and participants received a bonus cash payout proportional to their optimization of these two functions. We wanted to observe how the participants strategized as they tried to optimize, to better understand how we can support the reasoning skills we identified as being key to the domain. This paper describes how we derived those key skills, then illustrates via case studies how participants used these skills to address a wicked problem, and concludes with recommendations for further development of both ABM tools and the supporting curriculum for multidisciplinary UP and ES learning.
© ISLS 9

ICLS2012

Volume 1: Full Papers

Background
Content Domain: The Intersection of Environmental Science and Urban Planning
To introduce students to the fields of ES and UP, we chose to work with a complex systems model that represents rare stormwater events. This model incorporates elevation, land usage, roads, and sewers, and demonstrates how the placement of green infrastructure (in the form of rain gardens) affects stormwater movement. Our study allowed students to place these gardens on a map of 20 city blocks with various land uses (residential, business, and industrial) that can support differing numbers of rain gardens. The simulation shows the effects of a “hundred year storm” on the area, illustrating how the arrangement of gardens leads to more or less effective stormwater management. Effectiveness is defined by the two competing reward functions: one stresses a social priority (cost), and the other stresses a natural priority (time for water to drain into the soil). While this problem is important to practicing urban planners and ecologists, it also provides a valuable context for learners new to those fields. Key reasoning practices found in these disciplines are described below.

Spatial Reasoning in Environmental Science and Urban Planning
Planners argue for the importance of spatiotemporal context and place-based analysis in their research. These tasks draw on planners’ ability to think spatially. Spatial thinking in the social sciences generally demands a simultaneous integration of multiple spatial concepts. In Table 1 we build on five general spatial skills identified by Janelle & Goodchild (2011), who situated these skills in the context of practice. Column 1 lists the general skills, while Column 2 provides examples of areas where those skills would be used in UP and ES. A host of other researchers (Golledge, 1995; Janelle & Goodchild, 2011; Jo & Bednarz, 2009; Kaufman, 2004), however, have also produced categorizations and ontologies of spatial skills, which we use to identify component skills in Column 3. Column 4 then illustrates where those component skills would arise in UP and ES. Table 1: Illustration of how general spatial skills can be broken into component skills
General spatial skill (Janelle & Goodchild, 2011) Example skill application areas, for UP and ES

Generalized component skills

Examples of component skills in use in UP and ES

Differentiating: 1. Detect changes in ES: habitats the uses of, and UP: land use, regional e.g., differentiation of, agricultural, spaces residential 2. Measure the physical arrangement and clustering of phenomena to identify spatial patterns

Satellite imagery and maps are often used to determine boundaries, but one must: The ability to define a set of salient ES: select habitat factors and cutoff points variables, based on the phenomena (e.g. % tree coverage that counts as habitat) of interest, with which to relevant to species of interest differentiate space into different UP: define properties (e.g., types of areas residences) and proportions that distinguish urban from exurban land use The ability to differentiate areas (see above) and to perceive their relative position; adapt concepts of distance and connectivity to context; qualitative language for patterns A distance that permits two areas to be “connected” can be influenced by: ES: the foraging range of species of interest UP: the type of a road (highway, surface street)

Identifying: ES: nest patterns UP: settlement patterns

The ability to recognize and distinguish among different spatial Documenting: patterns (see prior general skill), to 3. Document spatial ES: invasive associate different patterns with patterns over time to species spread aspects of dynamic process, and to infer processes UP: urban design consistent measurement sprawl schemes to track patterns of interest; integrate analysis of both relative and absolute location 4. Study flows between specific locations as indicators of spatiotemporal interactions 5. Measure spatial Spatial dependence: the ability to integrate understanding of a Studying: dynamic process with an ES: genetic understanding of how spatial drift locations attenuate that process; the UP: traffic flow ability to map key steps in process to key locations Investigating: A sense of time scale and spatial

Both UP and ES have been hampered by inconsistent schemes for measuring patterns. For example, over the last 30 years: ES: National Land Cover Data set (NLCD) has changed land cover definitions UP: Land use definitions (especially “urban”) have changed ES: Studying migration of genes across landscape as an indication of how connectivity of habitat may have changed UP: Studying road congestions as an indication of the interaction between signal light timing and vehicle acceleration ES: Test if connectivity affects rate of genetic

© ISLS

10

ICLS2012 associations (and space-time associations) to test hypotheses ES: effect of habitat features on organisms UP: effect of zoning on land use scale magnitudes, and quantitative metrics for measuring spatial properties and patterns

Volume 1: Full Papers drift by measuring genetic diversity over time as habitat pattern changes UP: Test if zoning affects land use by measuring density of development in zoned and unzoned areas over span of 10 years

Engaging in the spatial reasoning processes outlined above entails integrating a number of component spatial skills, many of which involve making nuanced judgements about spatial concepts. For example, although the concept of a “habitat patch” may be simple, operationalizing it in a spatial sense requires judgements about spatial “cut off” points, among other distinctions. While these judgements are certainly guided by the topic of interest (e.g., a specific species of animal, or certain impacts of land use), even experts recognize that these decisions are not objectively certain. It is just this sort of practice with making critical, qualitative judgements (“reasoned guesses”) that has been lacking in traditional STEM curricula, and whose lack often renders recent STEM graduates paralyzed when confronted with real-world ambiguity. Learning how to approach a problem space that requires such judgements, learning to not be paralyzed by uncertainty, but also recognizing that other judgements could have resulted in alternate results and remaining open to other problem formulations are part of authentic practice in these domains. While this is impossible to fully address in our study without a structured curriculum, extended time, and other supports, we do wish to discover how learners differentiate spatial regions based on the salience of observed characteristics (general skill 1), perceive and describe patterns (general skill 2), and test hypotheses by measuring spatiotemporal associations (general skill 3).

Wicked Problems
An important component of the planning and policy practices described above is the necessity to make consequential decisions when there is inadequate, conflicting, or changing information. This particularly complex quandary is known as a wicked problem (Rittel & Webber, 1973). Wicked problems—in contrast to "tame" or "benign" problems that may be difficult, but are definitely solvable—cannot be clearly defined or solved, resist objective judgment, and are unique and nested in larger issues, among other qualities. In this way, wicked problems share some similarities to ill-structured problems. Ill-structured problems don’t have a positive definition, however, but are instead defined in relation to well-structured problems, which have single solutions, optimal solution paths, and structured goals (Sinott, 1989). By contrast, ill-structured problems have unclear (and possibly multiple) solutions, although the mechanisms used to solve both classes of problems are thought to be the same (Simon, 1973). Wicked problems similarly have multiple possible solutions, no clear optimal solution path, and ill-defined goals. Wicked problems differ from ill-structured problems in three main ways, however: (1) wicked problems emphasize the importance of context in shaping the goals, solutions, and solution paths, meaning that solution strategies developed for one context may not be applicable to another context; (2) wicked problems stress the consequentiality of solutions on the real world, meaning that problem solvers are asked to take stances on values and morality in order to define goals; (3) wicked problems are collective in that the values and morality incorporated into goal definitions are informed by a range of diverse stakeholders (Munneke et al., 2007). These differences suggest that what is known about ill-structured problem solving strategies may not be sufficient to understand how to support learners as they attempt to solve wicked problems. When planners attempt to solve wicked problems, they must have the ability to frame and define problem spaces, identify possible goals and solutions in those spaces, enact procedures to reach those goals, and do all of these in a way that fits real-world constraints. With the short, 2-year professional training planners receive, they cannot practice exercising these skills enough to adopt them as part of their standard practice (Zellner & Campbell, 2011). Clearly, planners need earlier and more frequent exposure to these problem spaces, but others would benefit as well. Incorporating wicked problems into public education would better prepare future stakeholders for participating in the planning process, because stakeholders must ultimately buy into the tradeoffs inherent in plans for those plans to succeed. Thus, the wicked problems selected for incorporation into curricula should have relevance for both future planners and future stakeholders. For our study, we chose the domain of stormwater management, which is a planning problem that urban and suburban residents find relatable. This problem domain also has relevance to environmental science. Stormwater management (especially in urban areas) requires spatial specificity incorporating both human and natural systems, and has direct consequences on city residents and other stakeholders. Working with this rich domain, we decided to focus on how it could be used to highlight a subset of the reasoning required for dealing with wicked problems. The accessibility of the stormwater management domain allows us to gradually introduce novice learners to reasoning about wicked problems, with the goal of starting learners on the progression towards developing the skill of compromising across competing definitions and demands. This is similar to Songer’s (2006) BioKids curriculum, which similarly repackages professional-level concepts into a learning progression more suitable to novice learners.
© ISLS 11

ICLS2012

Volume 1: Full Papers

Complex Systems Models as Tools for Exploring Wicked Problems
Complex systems simulations are an existing approach for presenting domain-specific and real-world phenomena to learners (Goldstone, 2006). These simulations are dynamic, centrally rely on iteration and feedback, and are made up of many parts organized across multiple levels of scale. As such, these tools provide a new opportunity for confronting aspects of wicked problems otherwise considered intractible. For example, when Rittel & Webber (1973) conceptualized wicked problems, they assumed that any exploration of the problem space would take place in the real world. With complex systems simulations, practices that Rittel & Webber advocated against (e.g. experimenting with different strategies, examining multi-level interactions) are instead sources of explanatory power (Zellner & Campbell, 2011). Learners are being provided with new tools that represent the spatial and dynamic interactions that underlie wicked problems in urban planning and environmental science, allowing learners to reason about these problems in a different way than they could without modeling and simulation software. This contrasts with common school practices, which prioritize correct answers over exploratory processes, often fail to link classroom activities to real-world practices, and mainly deal with well-defined problems and procedures. This exacerbates students’ inability to deal with wicked problems by giving them practice in exactly the wrong skills. One value of complex systems simulations lies in their ability to help users recognize how spatial and dynamic interactions lead to the emergence of wicked problems, which allows them to explore a richer range of solutions. The tools also allow learners to actively explore the different tradeoffs involved in solving wicked problems. This exploration forces learners to voice and test their hidden assumptions, leading to the collective definition of values that is necessary when grappling with wicked problems.

Prior Work
Learning Technologies Applicable to Wicked Problems in Urban Planning and Environmental Science
Technological capabilities for scaffolding spatial thinking have been developing faster than our understanding of the acquisition of skills in fundamental spatial thinking (Janelle & Goodchild, 2011). Geographic information systems (GIS) are heralded as a potentially effective tool for teaching basic spatial concepts in K–12 classrooms (NRC, 2006). However, when implemented in their current state in classrooms the risk may become one of teaching ‘‘buttonology,’’ or point-and-click procedures, to obtain a specified outcome (Marsh, Golledge & Battersby, 2007). With the buttonology approach, learning how to use the software program often supercedes conceptual and procedural understanding of the spatial analysis the software program is performing. Some packages, like MyWorld (Brown & Edelson, 1998) intentionally simplify the interface to re-align focus on the critical concepts and content (e.g., global earthquake data). We are taking a similar approach by adapting complex system models designed for use by environmental scientists and urban planners. Researchers have studied introducing learners to complex systems perspectives, which typically addresses how complex science content should be structured (e.g., Liu, Marathe, & Hmelo-Silver, 2005) or how to regulate students’ learning about complex systems (e.g., Azevedo et al., 2004). Such ideas will influence us as we structure our curriculum in the future, but perhaps more relevant to this study is work exploring how ABMs can be used to teach complex systems principles (e.g., Wilensky & Resnick, 1999; Goldstone, 2006). What distinguishes our work from theirs is that we are not trying to induce students to understand basic principles like “emergence” – rather, we are using these tools to help students confront compromise.

Activity and Study Design
The original stormwater management model (L-GrID) was designed to allow researchers to investigate the impact of different types of ground cover on groundwater infiltration. To adapt it for use with novices in fifteenminute sessions we had to scale back the detail made apparent to the user. We reduced the types of ground cover to three: impermeable (road surfaces), highly-permeable (swales) and semi-permeable (non-road patches that were not yet converted to swales). Additional simulation elements included sewers (which drain water from surrounding patches), elevation (the map sloped from a high at the top right corner to a low at the bottom left), an output sink (at the lowest corner of the map), and rain (set to always reproduce a devastating “hundred year” storm). We presented different contexts by varying sewer placement and ground cover across three maps encountered by learners. We reduced the outputs of the simulation to two: infiltration speed (time taken to clear all of the storm’s water from the map) and cost (each swale cost “$10,000”). Users could move and place swales on patches of the map, hit “go”, and witness how their configuration affected both the score-based outputs (cost and infiltration) as well as view a visualization of the depth of the water as it flowed across the map. We recruited fourteen pairs of undergraduates (as proxies for high school seniors). Because this was a lab-based pilot study, to encourage them to take the problem solving task seriously we borrowed from methods used in experimental economics. The ABM was structured to use the two simulation outputs (cost and infiltration) as two competing reward functions, and the cash payout participants would receive could be
© ISLS 12

ICLS2012

Volume 1: Full Papers

improved upon by optimizing across these two reward functions. To avoid discouraging them from exploring the problem space, the final reward was based on the best score they were able to attain within the allotted time of 15 minutes. We recorded video, transcribed their conversations, and collected data on the scores they attained with each trial. This stormwater management task gives learners the opportunity to engage with the aspects of wicked problems that are not always addressed by ill-structured problem solving tasks (context, consequentiality, collectivity). The two monetary reward functions are simplistic placeholders for the values of different stakeholders. Learners must determine how to reconcile the scores produced by these functions, which drives further exploration of the problem space. This reconciliation process is an entry point for novices to confront the demands of consequentiality and collectivity involved in solving wicked problems. Furthermore, by exposing learners to multiple maps which—despite surface similarities—demand different approaches to swale placement, we highlight that the context shapes the selection of heuristics for solving wicked problems.

Figure 1. Screenshot of the GI ABM used in this study, adapted from a more extensive NetLogo-based model built by the co-PIs for the Illinois EPA-funded Green Infrastructure Plan for Illinois project.

Coding Schemes
The literature on problem solving has codified several common strategies, labeled “weak problem solving strategies” (Simon, 1978), that learners use when confronting ill-structured problems (e.g. analogy, hill climbing, generate-and-test, means-ends analysis, problem decomposition, problem conversion). However, these strategies were deduced from studies with individual learners, and thus may not encompass the full range of strategies we expect to see during wicked problem solution. In particular, the properties that distinguish wicked problems (context, consequentiality, collectivity) place special emphasis on evolving a collectively acceptable goal definition that fits a particular set of circumstances. This change in emphasis necessitated an emergent coding approach (Strauss & Corbin, 2008) to identify the kinds of strategies used by our participants. The strategy kinds we identified each describe a particular form of shared, sustained discussion or work on the model. These included budgeting the total amount of gardens used (formal budgeting), adding or removing a nonspecified number of gardens (informal budgeting), focusing on particular large (map-wide) or small (block- and road-level) scales (large- and small-scale spatial), reacting to the spatial dynamics of the last run of the simulation (reactive spatial), and exploring garden-placement maxima and minima (determining limits). Three strategies we identified correlated with higher scores, or were used more often (see Table 2). A formal budgeting strategy was defined as the pair of learners setting a shared cap on the number of gardens used. This strategy was unsurprisingly associated with better performance on our cost score metric, although the budgets we observed were seemingly arbitrary. For instance, they tended not to stray from a restricted band of values (a cap of around 20 gardens). Despite a surface similarity to a means-end strategy (Simon, 1978), we did not observe learners justifying their budget. This indicates that learners need to be scaffolded to explore a wider range of budgets which would have a more principled effect on cost score. Table 2: Average scores per trial and frequency of use for observed strategies.
Reactive Spatial 0.97 0.50 1.47 26.31% Formal Budgeting 1.03 0.34 1.37 6.88% Informal Budgeting 0.94 0.53 1.46 14.55% Determining Limits 0.88 0.56 1.43 3.33% Large-Scale Spatial 1.01 0.42 1.43 30.86% Small-Scale Spatial 1.01 0.42 1.43 13.78%

Avg. Cost Score Avg. Infiltration Score Avg. Overall Score % Freq. of strategy use

With regards to the infiltration score, learners were most effective using a reactive spatial strategy, so named because it involved incremental changes based on the visualization. Users would point out spots of the map where water flow slowed or stopped, and would accordingly shore those up with gardens or attempt to restructure their arrangement to guide water in different ways. An exemplar of hill-climbing (Simon, 1978),
© ISLS 13

ICLS2012

Volume 1: Full Papers

using a reactive spatial strategy was effective in incrementally improving the infiltration score, but not as effective as the determining limits strategy. This suggests that learners need to better identify which parts of the map are critical, which is tied to detection of potential changes in the use of space (Janelle & Goodchild, 2011). Unlike the prior strategies we identified, the small- and large-scale spatial strategies were not so much related to problem solving heuristics as they were descriptive of the spatial reasoning of learners. It is also significant that the majority of learners began their interaction with the model by using a large-scale spatial strategy (see Table 2), which involves learners selecting and working on large regions of the map (usually described by learners in cardinal or relative directions, e.g., north/south, up/down, left/right). This stands in contrast to the relative paucity of small-scale spatial strategies, which would involve attending to small-scale features (e.g., adjacency and relative distance of map elements). Adjacency and distance can in fact strongly affect flooding, so learners require support to notice partitions of space (general skill 1) and their relations to one another (general skill 2), and to transition between different granularities of scale (general skill 5) (see Table 1). This could be accomplished through instructional support that identifies domain-meaningful distinctions amongst regions that users could choose to adopt.

Case Studies
Case 1: Learning to compromise takes more than heuristics
In this section, we highlight group conversations that show engagement with compromises between competing reward systems – i.e, showing how they responded to the characteristic lack of true or false answers to wicked problems. The two reward systems highlighted in our experimental trial were cost reduction (the players were penalized for the cost of installing each green infrastructure garden), and rate of rainwater infiltration (the players were rewarded for reducing the time taken for rainwater to drain). These two reward systems combine to form an overall score – although it should be emphasized that these reward functions are often competing: more gardens improve infiltration, but also increase the overall cost, and vice-versa. Moreover, the specific spatial arrangements of the gardens can also affect their infiltration efficacy. The following conversations have been selected from two groups of students to illustrate the types of discussion about compromise that occurred. In one group, S17 and S18 have just completed a garden arrangement and are watching the simulated outcome. They discuss their next strategy, balancing an observation that a larger number of gardens was helping them do well with infiltration against the observation that the cost was prohibitive: S17: We're doing well. Is the answer more gardens? S18: Oooh, arright. Where's the drainage seem to be caught, though? S17: It’s going a lot faster, but it’s the garden cost, is, uh, kinda screwin’ us over S18: Well, you can remove—I wanna remove this one. Maybe we could put two down here, maybe S17’s observation of the decrease in cost score shapes the subsequent strategy. They attempt to balance these competing rewards and achieve an alternative solution in the problem space as they have constructed it. Their approach is incremental (“this one”, “we could put two”), suggesting that it did not occur to them that they could use a means-end strategy, working backwards from the desired cost score increase to determine the number of gardens to target for removal. Another group, however, showed a much more sophisticated approach. Initially, students S19 and S20 are trying to figure out their first garden-placement move, and seem to only consider the rate of infiltration (which they refer to as the “time score”): S20: In theory you should probably want to have more, uh, more sinks for the water up at the top, because that's where the source of them S19: Right S20: But, if you have them towards the bottom, then it would drain out the entire area probably more effectively S19: But in order to keep the time score low, maybe we should have some at the top and some at the bottom? S20: Um, let's see S19: To prevent all of the water from rushing down S20: Let’s see, so the top drains out first Their initial strategy is to consider the infiltration pattern of the given map, by “study[ing] flows between specific locations,” (general spatial skill 4, see Table 1). They discuss whether it would be more effective to place gardens near the high elevation region (they mistake it as the “source” of the water) or near the main outflow at the bottom of the map. Later, after watching the simulation run their arrangement, they incorporate the reward structure of garden cost: S20: I think we're, uh, seeing cost score coming—going—go down pretty quickly, so… S19: /Yeah S20: …that may mean we're—we're, uh—we're adding few S19: We're getting a good return on, uh, on the gardens. Um, but we're still not anywhere near 2.25 S20: Yeah, let’s see
© ISLS 14

ICLS2012

Volume 1: Full Papers

S19: If we have—if we're getting a good return on the gardens we have, but we're ideally looking for a score that's… S20: /Probably around 2? S19: Yeah, I mean 2.25 is the ideal S20: Yeah S19: I think we could probably have—20 gardens—on there, and still get a fairly good overall score. And a good return on gardens S19 and S20 not only identify the competing reward structure and factor it into their decision making process, but they actually generate an entirely new construct: “good return” on the gardens. The language they use here indicates that they are embedding cost and infiltration not as independent factors, but as a unified ratio – a tradeoff. Unsurprisingly, it is after this discussion that S19 and S20 achieve their highest combined score. In their reasoning they also use the “ideal” score (what one could earn with maximum infiltration and no gardens, an impossibility) as an anchor point to work backwards from, in order to make judgements about how many gardens they can add without a significant hit in cost score. From these cases it is evident that while students are thinking about the competing reward structures without any explicit external prompts, this alone is not enough to guarantee success. The pattern exemplified by the first group (S17 and S18) shows a tendency to vacillate between first trying to satisfy one constraint, then the other. While this generate-and-test strategy can result in improved scores, these improvements are gradual, and may not help the learners to understand the essential relationship between the reward structures. The second group, on the other hand, demonstrates what is possible when means-end reasoning can be brought to bear on the problem, and when intermediate score evaluation representations (such as ratios) can be constructed.

Case 2: Lack of consideration of distance and adjacency
The pattern of sewers on the city map is an important small-scale spatial feature of the problem space. If learners place the gardens relative to sewer locations, they have the opportunity to maximize infiltration with a reduced number of gardens. However, we seldom saw this occur. While sewers were spoken of in 16 out of 28 (57%) conditions, mentions of sewers were present in only 1.19% of all discussion turns, a very low rate of occurrence around the second spatial component skill of distance and adjacency (see Table 1). In groups that do talk about distance and adjacency of gardens to sewers, the idea is very shortlived. For example: S20: What do you think—edges—or towards the, um, towards the middle? I think, um, keep ‘em towards the streets, then—that's also where the sewers are S19: Yeah, I was thinking that perhaps we should orient the gardens near the sewers, um S20: Well then you have two sinks at the—at the same location—versus S19: Spreading them out S20: Yeah S19: Yeah. Ok. Well what if we added four more gardens, umm, in this general area but not near the sewers This conversation lasts for less than a minute, and sewers are only briefly mentioned once more towards the end of their activity. This is typical of the conversations happening in other groups, where sewers are not recurrent strategic elements. This is an example of where scaffolding to help learners attend to smallscale spatial features could assist learners to bolster their spatial strategizing.

Discussion and Conclusion
Overall, learners required more support to engage in the spatial component tasks we identified as valuable for spatial reasoning and dealing with wicked problems. In particular, we found a need to support learners as they decompose space into regions, notice patterns across regions, and apply strategies that operate across multiple spatial granularities. To afford better reasoning about wicked problems, we need to encourage learners to engage in boundary testing, as this allows them to bring other problem-solving strategies (like means-ends analysis) to bear on the problem space. Since this study reports on a pilot of the simulation, we are in the process of developing a larger curriculum to complement this model’s impact on the development of spatial reasoning and systems thinking skills. It is very difficult for learners to incorporate all elements of the model into their judgments and plans, so a curricular scaffold could allow them to better accomplish this task. By balancing the diverse inputs, outputs, and processes that make up the simulation, learners can make more intentional and justified choices to affect the model, instead of taking actions that are abitrary or needlessly restricted.

References
Azevedo, R., Winters, F.I., & Moos, D.C., (2004). Can students collaboratively use hypermedia to learn about science? The dynamics of self- and other-regulatory processes in an ecology classroom. Journal of Educational Computing Research, 31(3), 215–245.
© ISLS 15

ICLS2012

Volume 1: Full Papers

Brown, M., & Edelson, D. C. (1998). Software in context: Designing for students, teachers, and classroom enactment. In A. S. Bruckman, M. Guzdial, J. L. Kolodner, & A. Ram (Eds.), Proceedings of ICLS 98: International Conference on the Learning Sciences, Atlanta, GA (pp. 63-69). AACE. College Board (2009). College Board Standards for College Success, from http://professionals.collegeboard.com/profdownload/cbscs-science-standards-2009.pdf Goldstone, R. L., (2006). The Complex Systems See-Change in Education. Journal of the Learning Sciences, 15(1), 35–43. Golledge, R. G. (1995). Primitives of Spatial Knowledge. In Nyerges T.L., D.M. Mark, R. Laurini, & M.J. Egenhofer (Eds.), Cognitive Aspects of Human-Computer Interaction for Geographic Information Systems (pp. 29-44), Dordrecht: Kluwer Academic Publishers. Hmelo-Silver, C. E., Marathe, S., & Liu, L. (2007). Fish Swim, Rocks Sit, and Lungs Breathe: Expert-Novice Understanding of Complex Systems. Journal of the Learning Sciences, 16(3), 307 - 331. Janelle, D. G. and M. F. Goodchild (2011). Concepts, Principles, Tools, and Challenges in Spatially Integrated Social Science. In Nyerges, T.L., H. Couclelis, and R. McMaster (Eds.) The Sage Handbook of GIS & Society. Sage Publications. 27-45. Jo, I. & Bednarz, S. W. (2009). Evaluating Geography Textbook Questions from a Spatial Perspective: Using Concepts of Space, Tools of Representation, and Cognitive Processes to Evaluate Spatiality. Journal of Geography, 108(1), 4-13 Kaufman, M. M. (2004). Using Spatial-Temporal Primitives to Improve Geographic Skills for Preservice Teachers. Journal of Geography, 103(4), 171-181 Liu, L., Marathe, S., & Hmelo-Silver, C. E. (2005). Function before form: An alternative approach to learning about complex systems. Paper presented at AERA 2005: The 2005 Annual Meeting of the American Educational Research Association. Marsh, M., Golledge, R., & Battersby, S. E. (2007). Geospatial Concept Understanding and Recognition in G6– College Students: A Preliminary Argument for Minimal GIS. Annals of the Association of American Geographers, 97(4), 696-712. Munneke, L., Andriessen, J., Kanselaar, G., & Kirschner, P. (2007). Supporting interactive argumentation: Influence of representational tools on discussing a wicked problem. Computers in Human Behavior 23, 1072-1088. National Academy of Education (2008). Science and Mathematics Education: Education Policy White Paper. Washington, D.C.: National Academies Press. National Research Council (NRC) (2006). Learning to Think Spatially: GIS as a Support System in the K-12 Curriculum. Washington, DC: The National Academies Press. Rittel, H. & Webber, M. (1973). Dilemmas in a General Theory of Planning. Policy Sciences, 4, 155-169. Schilling, J., Logan, J. (2008). Greening the rust belt: A green infrastructure model for right sizing America's shrinking cities. Journal of the American Planning Association, 74, 451-466. Simon, H. A. (1973). The structure of ill-structured problems. Artificial Intelligence, 4(3), 181-201. Simon, H. A. (1978). Information processing theory of human problem solving. In W.K. Estes (Ed.), Handbook of Learning and Cognitive Processes (Vol 5, pp. 271-295). Hillsdale: Erlbaum. Sinnott, J.D. (1989). A model for solution of ill-structured problems: Implications for everyday and abstract problem solving. In J.D. Sinnott (Ed.), Everyday problem solving: Theory and applications (pp. 72-99). New York: Praeger. Songer, N. (2006). BioKIDS: An animated conversation on the development of curricular activity structures in inquiry science. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 355–371). New York: Cambridge University Press. Strauss, A. & Corbin. (2008). Basics of qualitative research: techniques and procedures for developing grounded theory. Los Angeles: Sage. Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8, 3-19. Zellner, M. & Campbell, S. D. (2011). New Tools for Deep-Rooted Problems: Using Complex Systems to Decode Wicked Problems. Paper presented at ACSP: The Association of Collegiate Schools of Planning Conference, Salt Lake City, UT. Zellner, M., Massey, D., Cotner, L., Minor, E., Gonzalez-Meler, M. (in progress). How Effective Is Green Infrastructure for Stormwater Management? Landscape and Urban Planning.

Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. 1020065. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

© ISLS

16

ICLS2012

Volume 1: Full Papers

Semiosis Process in Instructional Practice
Gé rard Sensevy, Dominique Forest, Brittany Institute of Education, University of Western Brittany, Rennes, France Email: gerard.sensevy@bretagne.iufm.fr, dominique.forest@bretagne.iufm.fr Abstract: In this paper, we aim to show how the Joint Action Theory in Didactics (JATD) may contribute to construct a science of instructional practice. From this perspective, we propose the two concepts of student’s twofold semiosis and teacher’s equilibration work, which allow us to understand the instructional practice and the learning process in the same conceptualization. We illustrate these concepts and our video research methodology within two case studies. The first case study concerns tactile reading with a visually impaired student at primary school. The second case study refers to a learning situation proposed by Brousseau (2004), the Treasure Game, which one can see as a fundamental situation for the representational process. By contrasting the two cases, we attempt to identify two different kinds of equilibration works. We show how these two different equilibration works involve different ways of articulating the twofold semiosis in the didactic transactions.

Introduction
In this paper, we address two major issues. First, we concentrate our analysis on practice, and particularly on the instructional practice. Sharing Koschmann’s question “How… do we begin to construct a real science of instructional practice?” (Koschmann, 2011, p. 6), we try to provide some elements to contribute to this construction. From that perspective, we sketch a theory, the Joint Action Theory in Didactics (Sensevy & Mercier, 2007; Sensevy, 2011a; 2011b; 2011c), stemmed from French Didactics (Brousseau, 1997; Chevallard, 2007; Sensevy et al., 2005; Laborde et al., 2005 ; Sensevy et al., 2008), a theory which aims at linking teaching and learning by postulating that one cannot understand learning practices without understanding the related teaching practices. In this theoretical frame, “didactic” has the general meaning of “relating to the teaching/learning process concerning a specific knowledge domain”. The second issue of our paper concerns the way the teaching/learning relationship can be seen as a semiosis process - that we define, in Peirce’s sense (Peirce, 1998), as the way of producing and deciphering signs - in which the teacher’s action plays a prominent role. Within the JATD theoretical frame, we argue that learning needs teaching, and we postulate that the teaching learning process is above all a communicative process. In order to learn, a student needs to understand the learning situation in a right way, i.e. in the “same” way as the teacher does. Under certain conditions, the teacher and the student have “to play the same game”. How can the teacher and the students acquire a common understanding of a given situation? We argue that students have to become able to see the signs in/of the situation in the same way the teacher does. They have to build an accurate seeing-as (Wittgenstein, 1997) in a specific thought style (Fleck, 1979).

The theoretical approach: the semiosis process in the joint action theory
The JATD is based on the notions of didactic game and learning game. The didactic game (Sensevy et Mercier, 2007; Sensevy, 2011a) describes the fundamental structure of the teaching/learning process. We consider the didactic game as a game in which one player (the teacher) wins if and only if the other player (the student) wins. The teacher cannot make the “winning moves” (i.e. cannot learn) in place of the student, even though she has to enact a specific guidance (Kirschner et al., 2006) to orient the students. All that he or she can do is “play” in an indirect way in order to enable the student to learn. In this regard, the student has to act in a corresponding way, and enact a first-hand relationship to the piece of knowledge involved. We have described (Sensevy, 2011b) this fundamental constraint by arguing that the teacher has to be reticent – the teacher must not tell the student all that he or she knows – in order to enable the student to learn on her own – a necessity we term the proprio motu clause. This fundamental property of the didactic game implies some other necessities that we have no space here to mention, but the main point is that the teacher’s reticence and the proprio motu clause strongly shape the kind of semiosis inherent to the didactic game. A learning game, which refers to the teacher’s game on the student’s game, can be described by relying on two elements. The first element is the didactic contract, an expectations system between the student and the teacher, which gathers the current habits (rules, norms, capacities) relating to the knowledge at stake, and which constitutes the current students’ strategic system. The didactic contract can be seen as a system of social and socio-epistemic norms (Yackel & Cobb, 1996; McClain & Cobb, 2001) relating to the didactic institution in which the didactic process unfolds. The second related element is the didactic milieu, which refers to the virtual

© ISLS

17

ICLS2012

Volume 1: Full Papers

knowledge to be acquired, as it is displayed in the learning game. This knowledge is embedded in the material and symbolic objects of the milieu, to the extent that this milieu enacts a problem. A main feature of the didactic transactions lies in the process of reaching a kind of equilibrium between contract and milieu in the learning games. We called this process didactic equilibration. The didactic equilibration can be seen as a semiosis process, in which, by relying on the didactic contract, the students become able to solve the problem, i.e. to interpret the milieu in a relevant way. In the semiosis process, the student has to produce a first semiosis, the deciphering of the signs of the milieu. These signs are non-intentional signs. By “non-intentional”, we mean that the signs are not directly produced within in intentional process. But this first kind of semiosis is not produced independently of the social structure in which the didactic process unfolds. It is enacted against the background of the previously taught knowledge as it has been structured in the preceding didactic joint action, the didactic contract background. This didactic contract may be seen as an epistemic system, which enables the students to cope with the milieu of the problem at stake, and a transactional system, in that the teacher may orient the student’s action in the milieu by relying on the content of the epistemic system provided by the contract. This transactional process refers to what we call the second semiosis. In the first semiosis, the student has to decipher the non-intentional signs of the milieu. In the second semiosis, she has to decipher the intentional signs that the teacher provides in order to orient her in the milieu. In the following, we document this twofold semiosis and the related teacher’s equilibration work in two empirical fields. The first case study refers to a situation of tactile reading, involving visually impaired students. The second case study elaborates on a didactic situation at Kindergarten proposed by Brousseau (2004).

The methodological approach: some elements
Our theoretical frame rests upon the analysis of the joint action between the teacher and the students. As we have seen, this theoretical viewpoint gives a major importance to the production and the deciphering of signs, in the semiosis process, which is partly a semiosis of others. The teacher has to identify the signs provided by the students, and, in what we have called the second semiosis, students have to recognize the teacher’s intentional signs. Moreover, the knowledge as stake can be conceived of a symbolic system in itself, that the students have to acknowledge and practice. For these reasons, we ground our inquiries in the video recording of teaching/learning practices, and we include our approach (Forest & Mercier, 2011) in the methodological paradigm of the Video Research (Goldman et al., 2007). As we have argued elsewhere (Sensevy, 2011a; Tiberghien & Sensevy, in press), video recordings enable to keep the analogical character of the situations with its specificity and its density of information (Dretske, 1981). Above all, as Goldman & McDermott (2007) state, it makes communication visible, and allows accounting for the embodied instructional communication. Our methodological endeavor consists of building what we call hybrid texts-images systems, in which is fostered a reciprocal annotation process between texts and images.

Case study 1: Visually impaired students
General description of the situation
The first situation we present involves tactile reading. Visually impaired students (5 years old), are trained by a special teacher to read a tactile book, which name is “Little point draws a man”, and which is structured as presented below.

Figure 1: 3/9 pages showing the structure of the tactile book “little point draws a man”. The teacher decided to give the book to the visually impaired student without reading it to her. The student had to recognize the shapes without any information. She first recognized “the head” of the man and the studied episode take place when the student has to identify what she first called «the sweater »of the man.

Solving the problem of representation: an oval for the belly
The student has just turned the page and she is exploring the tactile representation, i.e. a new piece of fabric below the head that she has previously identified.

© ISLS

18

ICLS2012

Volume 1: Full Papers

General location phase: the student Specific exploration phase (1): the uses the left index to locate the student enacts a vertical exploration “ear” element, while the right hand of the new element. recognizes the whole element “head”. In the same time the thumb starts to explore the new element (the “belly”). Table 1: first tactile exploration, in a silent way

Specific exploration phase (2): hemispheric dissociative exploration. In particular, she focuses on the hinge between the «head » element and the new element.

Specific exploration phase (3): the Specific exploration phase (4): the student focuses on the new element. student verifies the specific location She feels the texture of the new of the “belly” element. T: What can it represent, according to you? S: The belly T: what is the shape of the belly? + can you tell me the shape? Table 2: exploration with joint verbal action S (Student): The sweater+ T (Teacher): This is the sweater? S: Ups! no

Specific exploration phase (5): the students explores the new element, in search of new differential properties (other than texture) S: round, mm T: It’s a round? I do not totally agree S: an oval

In this episode, from the student’s point of view, the twofold semiosis means a milieu semiosis (S M), the first semiosis, and a teacher semiosis (ST ), the second one. The learning game at stake is a “form recognition” game, i.e. a game where the student must identify the forms she touches on the book. The epistemic elements of the didactic contract on which the student relies are the prototypic body image, her textures knowledge, a ”common geometric” knowledge, and the “currently taught” geometric knowledge. The student explores the didactic milieu constituted by the different elements of “the man”. The fact that the student is visually impaired put forward the necessity she meets to explore the didactic milieu, as the annotated photograms show. The relationship to the milieu is a necessary condition of her learning. Nevertheless, as we will see, the didactic contract plays a prominent role in this episode. We can describe the didactic transactions in this excerpt according to four phases. Phase 1: (SM1/T1). The semiosis is oriented to the milieu (First Semiosis), following the teacher’s general indication. It’s the meaning of the symbolic notation S M1/T1: a semiosis of the milieu (first letter M), following the teacher’s orientation (second letter T). The student explores the specific location of the new element in the body image and relies on its texture. She finds a first answer (The sweater), by focusing on one may call a non-intentional sign (the fabric on the tactile book) Phase 2: Then the teacher asks a question (T: This is the sweater? …What does it can represent, according to you?) (ST1). Here the student’s semiosis is of the second kind. The student deciphers the teacher’s signs, against the background of the didactic contract. One can make the hypothesis that she has understood that the game is about the parts of the body, she identifies an intentional sign, and she answers by guessing the likely teacher’s expectations (The Belly) Phase 3: Then the teacher asks a new question (T: What is the shape of the belly?)(ST2). Again, the student’s semiosis is of the second kind. The teacher’s word “shape” probably allows her to identify the geometric didactic contract, which rule this transaction. She understands she has to focus on the geometric property of the belly, and she answers with this respect (Round).

Tactile reading: the twofold semiosis process and the teacher’s equilibration work

© ISLS

19

ICLS2012

Volume 1: Full Papers

Phase 4: Now the teacher tries to obtain a better “geometric” response from the student (T: It’s a round? I do not totally agree)(S T2/M2). The student’s semiosis continues to unfold in the deciphering of the teacher’s signs (her utterances). She explores the form of the “belly” and produces a new answer (Oval). This case study enables us to make a clear distinction between the two forms of semiosis. At the beginning of the episode, the student really faces the milieu, and she has to decipher the tactile signs by relying on her knowledge of the body image and the texture she recognizes. But after this first kind of semiosis, she has to take into account the teacher’s questions and assertions. In doing that, she orients her deciphering towards the expectations of the teacher, that she interprets against the background of the didactic contract. From the teacher’s action viewpoint, one can recognize an equilibration work in which the milieu is an auxiliary to the contract. Everything occurs as if the main intention of the teacher was to elicit some words (belly, oval) that the student already knows. The milieu, and the exploratory work of the student in this milieu, has the main function of producing the “right” predefined seeing-as: “seeing” the “sweater” as a body part; “seeing” the “round” as an oval.

Case study 2: The Treasures Game
General description of the situation
Brousseau and his team designed the treasures game for kindergarten students at the beginning of the nineteen eighty. This learning sequence takes place over several months and it aims to have students build a system of graphical representations. Brousseau (2004) has presented a strong theorization of this research design, which he considered as a fundamental situation for the notion of a representation. The situation was recently reimplemented in some classes in Switzerland (Leutenegger & Ligozat, 2009) and in France. It consists of producing a list of objects to be remembered and communicated. The game is organized in four stages, the rules changing as the game progresses. In the first stage, the teacher presents two or three small new objects each day. These objects belong to the world of children and are passed from hand to hand. The teacher asks the students to name them, and she puts them in a box (the treasure chest). Then she asks: what's in my box? A student then calls out the name of an object, the teacher pulls the object out of the box and places it in full view. Then she asks “Is my box empty?” and if not, the game continues, and so on. Every two or three days, two or three new objects are presented by the teacher and are added to the previous ones. At the end of one month, the whole class can empty a box of 40 objects (thus being able to collectively memorize 40 objects). These objects can seem disparate but they were carefully chosen. One can look at some examples of objects on the figure 1, including “the lens” and “the pan” which will support the problem of representation in the episode we will study in this section.

Figure 2: some examples from the reference collection of 40 objects. This stage is played out with the entire group of students and focuses on the creation of a verbal system of reference for the objects inside the treasure chest. The stage 2 proceeds in two memory games. Firstly the students have to play an individual memory game, as each pupil must individually remember 3 objects that are daily hidden in the treasure chest. But when all the pupils have understood this memory game with 3 objects, the teacher introduces an important change: she asks each pupil to remember ten (10) hidden objects. This informational leap makes it impossible to win only with an “internal” memory. The jump from 3 to 10 objects is intended to oblige pupils to produce individual graphical lists, in order to remember the specific objects and win the game. The stage 3 aimed at communicating with lists. This new part of the game takes place in small groups of 5 pupils. 4 object representations are written by a pupil, who is “the designer”, and these objects are hidden in the box. The other 4 pupils have to “read” the graphic representations of the designer, and name each object to get it out of the box. This third stage gives pupils opportunities to debate, firstly in their small group, and secondly in the whole classroom during the 4th stage of the game. In stage 4, the pupils are building a common code for the whole class. In the studied carrying on of this engineering, stage 3 and 4 are intertwined. Some changes in stage 3 prepare the debates in stage 4.

Solving the problem of pan's representation
The small episode we study takes place in stage 3, when pupils have to read the representation of the designer. On this episode, the teacher gave the designer four quite different objects; one of them is “the pan” (cf. fig.2, above). When students have to “read” the designer's

© ISLS

20

ICLS2012

Volume 1: Full Papers

production (cf. fig.3, shown against), they fail to recognize “the pan” (the designed object, fig. 2, above), confusing it with “the lens”, another object of the collection of reference, which have a similar form (cf. fig.2, above). In the short moment of joint didactic action we present, the teacher has first Figure 3: the representation shared with the students the failure of the designing and recognizing the pan. of the pan by the “designer” She then asks students to find some ideas to draw and recognize the pan better. student Like in the previous example, our description and analysis use a hybrid system with photograms, utterances and commentaries on twofold semiosis. The English discourse is a translation from French. The commentaries are informed by proxemic analysis (Hall, 1966; Forest, 2009) of non-verbal behaviors. Table 3: giving the problem Iman: we have to make a small line, we have to do this... The teacher shows the object in her left hand, and its representation (cf. fig.3 above) by pointing it with her finger, in her right hand, and asks the students for a comparison. A student, Iman, points a part of the object, which is characteristic of it (a hole on the handle). In that way, the student links a sensory feature to a specific characteristic of the object, which is potentially relevant to solve the problem. One can notice the direction of the student's eyes as an indication of a joint semiosis between the teacher and the student. At this moment, the local learning game consists in representing in a better way the pan, in order to distinguish it from the lens. By showing the object and its representation, the teacher gives the student the specific problem of the distinction lens-pan in the graphic representation. According to the teacher’s expectations, the student (Iman) acts in the milieu, and suggests by pointing on the object that it would be accurate to represent the hole (we have to do this). We can argue that the student plays a relevant move in the representation game by deciphering the signs of the milieu (semiosis 1, she indicates a possible characteristic to design). Even though she responds to the teacher’s expectation (semiosis 2), she positions herself as a potential provider of relevant design features. The semiosis is of the form (SM/T ). Table 4: another strategy is proposed Iman: a big ring, and for the lens a small ring The teacher looks herself at the representation drawn by the designer (cf. fig.3 above). She points it by her left hand finger. Iman “writes” with her finger on the table by evoking a possible representation of the difference between the pan and the lens (big ring-small ring). By showing the representation to the students, and looking herself at it, the teacher drives student's attention onto the design. Iman tries to draw on the table, and a generic sign emerges (a ring). But the representation “big-small” is not a distinctive sign (which may allow to recognize the object in itself) and it will not really work if only one object (the lens or the pan) is to be recognized among others: in this case only the pan is represented, and the comparisons of representations (between the lens and the pan) cannot work. Interpreting in a general way the teacher’s expectations about the drawings, the student produces a kind of strategy which can be useful, but which is not a winning one in this specific learning game. Even if it is relatively ineffective, the semiosis is always of the form (SM/T ), in that it enables an explorative and imaginative inquiry into the milieu, not oriented to pre-conceived meanings. Table 5: the less relevant strategy starts spreading Another student: ...a small ring, and a stick As in table 3, the teacher shows the object and its representation. Again, she asks the students for a comparison. Another student writes with his finger on the table, as Iman did. He repeats Iman’s proposition (a ring), adding the possibility of drawing a stick (for the handle). It’s worth noticing that the attention of the students focuses on Iman’s action. The teacher’ attention is oriented to the last student who had made a proposal.

© ISLS

21

ICLS2012

Volume 1: Full Papers

The teacher goes on showing the object and its representation in order to have the students “comparing” them, but Iman's proposal is taken up and meliorated by others. The new proposal (a small ring and a stick) possibly solves the problem of a generic representation for the pan and the lens. From this point of view, it is relevant. But it doesn't work as a distinctive representation between the two objects. So it is less relevant in this representation game. The enacted semiosis is of the same kind as the previously analyzed (SM/T ). The students are oriented by the strong teacher’s guidance, but in an open way, which allows them to propose different representations. Table 6: the teacher do precise the problem Showing the pan and its Staying in her posture in front of Moving on her right side, near the representation, the teacher reminds the students, the teacher moves the designer, she formulates the the students the current difficulty object and its representation on her problem for the pan. (we had confused with the lens). left side. When talking, the teacher directs One can notice the central posture In the same time, she reformulates her eyes toward different children, of the teacher, and the convergence the problem, relating to the lens (so finishing toward the designer. of the children's eyes. that one guesses every time that it's the lens…)

The teacher: It's true that we had We should try to find a drawing for ...and a design for the pan, so that confused with the lens, because it the lens, so that one guesses every one guesses every time that it's the was like the drawing of the pan. time that it's the lens… pan… One can see, at this moment, how the teacher works in the semiosis 2 (the student’s deciphering of teacher’s signs) to take on the issues of the learning game (we had confused... we should try to find a design... so that one guesses every time). However, the teacher remains reticent. To support her reticence, she uses the resources of the proxemics, joining gesture to her talk: first, she shows the object and its representation like in the photogram 1, showing them to all the students; second, she evokes the lens (which is absent from the package) putting the pan and its representation on the left side; third, she moves on the other side to finally show the representation which failed, to the designer. The learning game is strongly restated at this time (a drawing for the lens/the pan, so that one guesses every time that it's the lens/the pan…): the students have to produce by themselves a strategy to design the pan, while taking into account its similarity with the lens. One can analyze this episode by relying on the contract-milieu dialectics within the learning game. In order to explore the milieu (to inquire into the different ways of representing the objects) in a relevant way, the students have to play the right game. The students have to become able to differentiate some very similar objects (from the students’ representational viewpoint), as the pan and the lens. In this perspective, the teacher’s monitoring consists of reminding these contract rules, which embed the end-in-view of their game. Table 7: Joint action provides a new relevant move Tilio: Ah yes! so one can make a small hole here. The teacher continues showing the object and its representation. A student (Tilio) points a part of the pan, the same that Iman had pointed on table 3. We can perceive the convergence of student’s gaze on Tilio’s gesture. According to the teacher’ expectation, Tilio acts directly into the milieu, by pointing the hole on the object and by naming it (a small hole for the handle). In a more theoretical language, we can say that Tilio deciphered the teacher’s expectation (semiosis 2) about the playing of the “right game”. He then plays a relevant move in the representation game, by deciphering a sign of the milieu (semiosis 1), and categorizing it in an efficient way. In effect, if the representation of the pan includes “a small hole” on the handle, it will allow to
© ISLS 22

ICLS2012

Volume 1: Full Papers

differentiate the pan from the lens, even though if the general form of the two representations is similar (a small ring and a stick). Tilio’s move will be followed by other students and his proposal will be institutionalized. Once again, the semiosis process is of the form (SM/T ). The students produce imaginative and accurate ways of seeing the milieu, but one has to keep in mind that such performance is made possible by the enduring teacher’s work.

The Treasure Game: the twofold semiosis process and the teacher’s equilibration work
According to us, the above analysis is in line with a transactional perspective (Clancey, 1997, 2001; Koschmann, 2001, 2011), which looks at the teaching-learning process from a Deweyan viewpoint. In that way, we agree with Koschmann’s contention according to which “Deweyan inquiry, when successfully carried out not only effects a change in the problem solver (what psychologists treat as “learning”) but also leads to a reconstruction of the problematic situation that led to the inquiry in the first place” (Koschmann, 2011, p. 12). In the Treasure Game situation we have presented in this paper, one may appreciate an actual form of inquiry endorsed by young students (5 years old). The theoretical categories of the Joint Action Theory in Didactics enable us to understand what could be the sources of such an inquiry. Grounded in a representation game, this situation entitles the students to propose signs as means of representing objects. But this game does obey to strict rules, as we have seen above. If a sign does not provide the collective with a successful way to recognize an object, it has to be rejected, not by the teacher, not even by the students, but in the unfolding of the actual game (at is was the case at the beginning of the studied episode). It’s worth noting that such a situation does not achieve its goal without a specific teacher’s equilibration work. One can imagine two opposed counterfactuals in order to figure out this point. In the first counterfactual, one can think about a teacher who would abandon her reticence, and try to directly influence the students, for example by immediately rejecting the ineffective signs, or by proposing one particular “good” sign to the students. In the second counterfactual, one can envisage a teacher who would let the students explore the milieu, for example without reminding them the overarching goal of the learning game. In contrast with these both counterfactuals, in the actual situation, the teacher enacts a third way of doing. She provides the students with a system of meanings, embedded in the signs she uses, while keeping a strong reticence about the representational knowledge in question. In her equilibration work, the contract can be seen as an auxiliary to the milieu, which means that in the twofold semiosis that the students enact, the deciphering of the teacher’s signs gains its sense firstly as a way of carrying out the inquiry (Dewey, 1938/2008).

Discussion: an outline of a comparative analysis
The comparison of practices in the two case studies could provide an exemplar, in Kuhn’s sense (Kuhn, 1979), of two possible relationships between the student’s deciphering of signs and the structure of the instructional process. In effect, in the first case study (Tactile Reading by a visually impaired student), the twofold semiosis is mainly of the form (ST/M), even (ST ). It is to say that the student, when she explores the milieu, does it by enabling herself to recognize the didactic contract meanings that stand as a background against which she acts. Her agency is constrained by the predefined meanings that the teacher has in mind (i.e. “belly” rather than “sweater”, or “oval” rather than “round”). In this perspective, the teacher’s equilibration work is contractdriven, in that her main intention consists of making the student more familiar with a specific seeing-as (see the round as an oval) in a particular though style (the geometric though style), that has been previously studied. By contrast, in the second case study (The Treasure Game), the twofold semiosis is mainly of the form (SM/T ). The deciphering of the teacher’s signs, in the second semiosis, is never an end in itself. The main function of this second semiosis consists of making the student able to carry on the inquiry process. The didactic contract stands as a background for the student’s activity, in that it orients in a decisive way the students’ action (the weakness of the contract would be very detrimental to the joint action), but it is not sufficient to reach the learning games goal. The more important feature of the students’ activity lies in their inquiry process into the milieu (i.e. becoming able to “invent” effective representations), and rests on the way they endorse the first semiosis. In that way, the structure of the learning game fosters students’ agency. In this perspective, the teacher’s equilibration work is milieu-driven, in that her main intention consists of making the students able to enter the representational process by directly understanding the nature and properties of signs through the consequences of their decisions.

References
Brousseau, G. (1997). The theory of didactic situations in mathematics. Dordrecht: Kluwer. Brousseau, G. (2004). Les repré sentations : é tude en thé orie des situations didactiques. Revue des sciences de l’éducation, 2(30), 241-277. Chevallard, Y. (2007). Readjusting didactics to a changing epistemology. European Educational Research Journal, 6(2), 131–134.

© ISLS

23

ICLS2012

Volume 1: Full Papers

Clancey, W. J. (1997). Situated Cognition: On Human Knowledge and Computer Representations. Cambridge: Cambridge University Press. Clancey, W. J. (2011). A Transactional Perspective on the Practice-Based Science of Teaching and Learning. In T. D. Koschmann (Ed.), Theories of Learning and Studies of Instructional Practice (pp. 247-278). New York: Springer. Dewey, J. (2008). John Dewey The Later Works, 1925-1953: 1938: Logic: The Theory of Inquiry. Chicago: Southern Illinois University Press. Fleck, L. (1981). Genesis and development of a scientific fact. Chicago: University of Chicago press. Forest, D. (2009). Agencements didactiques, pour une analyse fonctionnelle du comportement non-verbal du professeur. Revue franç aise de pé dagogie, 165, 77-89. Forest, D., & Mercier, A. (2011). Classroom Video Data and Ressources for Teaching. In G. Gueudet, B. Pepin & L. Trouche (Eds.), From Text to "Lived" Ressources : Mathematics Curriculum Materials and Teacher’s Development. Berlin : Springer. Goldman, R., Pea, R., Barron, B., & Derry, S. J. (2007). Video Research in the Learning Sciences. Mahwah, NJ: Lawrence Erlbaum Associates Inc. Goldman, S., & McDermott, R. (2007). Staying the course with video analysis. In R. Goldman & R. Pea & B. Barron & S. Derry (Eds.), Video research in the learning sciences (pp. 101-113). Mahwah New Jersey: Lawrence Erlbaum Associates. Hall, E. (1966). The Hidden Dimension. New York: Anchor Books Doubleday Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and InquiryBased Teaching. Educational Psychologist, 41, 75-86. doi:10.1207/s15326985ep4102_1 Koschmann, T. D. (2001). A third metaphor for learning: Toward a Deweyan form of transactional inquiry. In S. M. Carver & D. Klahr (Eds.), Cognition and Instruction: 25 Years of Progress (pp. 402-415). Mahwah, NJ: Lawrence Erlbaum Associates Inc. Koschmann, T. D. (2011). Theories of Learning and Studies of Instructional Practice. New York: Springer. Kuhn, T. (1979). The Essential Tension. Chicago: University of Chicago Press. Laborde, C., Perrin-Glorian, M.-J., & Sierpinska, A. (2005). Beyond the Apparent Banality of the Mathematics Classroom. New-York: Springer. Leutenegger, F & Ligozat, F. (2009). The Treasure Game : grasping the premises of the subject matter norms in pre-school classes. Communication, Network 27, European Congress in Educational Research (ECER) "Theory and evidence in European educational research?", [28-30 september] University of Vienna, Austria. McClain, K., & Cobb, P. (2001). An Analysis of Development of Sociomathematical Norms in One First-Grade Classroom. Journal for Research in Mathematics Education, 32(3), 236-266. Mauss, M. (2006). Techniques, Technology And Civilization. London: Berghahn Books. Peirce, C. S. (1998). The essential Peirce: (1893-1913). Bloomington: Indiana University Press. Sensevy, G., Mercier, A., Schubauer-Leoni, M-L., Ligozat, F. Perrot, G (2005). An attempt to model the teacher’s action in mathematics. Educational Studies in mathematics, 59(1), 153-181. Sensevy, G. & Mercier, A. (2007). Agir ensemble. Rennes : Presses Universitaires de Rennes. Sensevy, G., Tiberghien, A., Santini, J., Laubé S. & Griggs, P. (2008). Modelling, an epistemological approach : , cases studies and implications for science teaching. Science Education, 92(3), 424-446. Sensevy, G. (2011a). Le sens du savoir. Éléments pour une théorie de l’action conjointe en didactique. Bruxelles : De Boeck. Sensevy, G. (2011b). Overcoming fragmentation : Towards a joint action theory in didactics. In B. Hudson & M. Meyer (Eds.), Beyond Fragmentation : Didactics, Learning and Teaching in Europe (pp. 60-76). Opladen and Farmington Hills : Barbara Budrich. Sensevy, G. (2011c). Patterns of Didactic Intentions. Thought Collective and Documentation Work. In G. Gueudet, B. Pepin, & L. Trouche (Eds.), From Text to “Lived” Resources : Mathematics Curriculum Materials and Teacher Development (pp. 43-57). New-York : Springer. Tiberghien, A., & Sensevy, G. (in press). Video studies : Time and duration in the teaching-learning processes In J. Dillon & D. Jorde (Eds.), Handbook "The World of Science Education” volume 4. Rotterdam/Boston/Taipei : Sense Publishers. Yackel, E., & Cobb, P. (1996). Sociomathematical Norms, Argumentation, and Autonomy in Mathematics. Journal for Research in Mathematics Education, 27(4), 458-477. Wittgenstein, L. (1997). Philosophical Investigations. Oxford: Blackwell.

© ISLS

24

ICLS2012

Volume 1: Full Papers

Agent-based computer models for learning about climate change and process analysis techniques
Nick Kelly, Michael J Jacobson, Lina Markauskaite, Vilaythong Southavilay The University of Sydney, Australia Email: nick.kelly.mail@gmail.edu.au, michael.jacobson@sydney.edu.au, lina.markauskaite@sydney.edu.au, vstoto@it.usyd.edu.au Abstract: This paper describes a design-based research project that investigates the learning of scientific knowledge about climate change through computational models. The design experiment used two NetLogo models and problem-based learning materials developed in partnership with a high school science teacher. In the study, three classes of year nine science students were divided into two groups based upon different levels of structure that was provided during learning activities with the models. The results indicate that there was significant learning of concepts about greenhouse gases and the carbon cycle through engagement with the models. We also describe the process analysis techniques being developed to analyze the log files of the interactions the students had with the computer models. This paper presents results from the first iteration of a design-based research project that is exploring how students may learn scientific knowledge about climate systems using computer models as part of a four year federally funded project in Australia. There are three main research areas of this project. The first area is investigating if different patterns for the sequencing and structure of pedagogical guidance during learning activities can enhance learning with computer models. Whereas the notion of strict discovery-learning has been largely discredited (Mayer, 2004) there is recent evidence that a sequence of low-structured activities followed by high structured activities can result in enhanced learning outcomes when compared to purely high-structured activities only (Kapur, 2008, 2010; Schwartz & Bransford, 1998; VanLehn, Siler, & Murray, 2003). Second, there are increasing calls for students to learn important scientific knowledge about systems, including topics related to complex climate systems and global warming (ACARA, 2011; National Research Council, 2012). An important recent approach for helping students to successfully learn more standard science education topics involves model-based learning (MBL) (Clement, 2000; Zhang, Liu, & Krajcik, 2005). In general, MBL approaches engage the learners with computer simulation and visualization systems and learning the targeted science concepts interactively via problem solving. There has also been research involving the use of agent-based models (ABM), which computationally represent phenomena as a number of agents or elements that each has particular rules they follow, and for which the apparent complexity of the system being modeled emerges from the interactions of the agents in the systems (Railsback & Grimm, 2011). A number of recent studies have documented significant learning with ABMs, especially for learning about various types of complex physical and biological systems (Goldstone & Wilensky, 2008; Jacobson, Kapur, So, & Lee, 2011; Wilensky & Reisman, 2006). In this study, we investigate how ABMs can help students learn important concepts about climate systems. Finally, when students work with computer models, a large stream of fine-grained process data can be collected. Recently, researchers have begun to explore various process analysis techniques for analyzing data of human-computer interactions, such as data mining, process mining, and learning analytics (Reimann, 2009; Romero, Ventura, Pechenizkiy, Ryan, & Baker, 2010; Trčka, Pechenizkiy, & van der Aalst, 2011; van der Aalst, 2011). These techniques have demonstrated potential for discovering student learning patterns and predicting unproductive behaviors where early intervention may be possible. However, research is needed to investigate how these techniques might be used to assist teachers and students during MBL. Therefore, the third area of our research is to provide a proof of concept for specific data mining techniques that might be useful for analysis of MBL. This study involved MBL of scientific knowledge about climate change to explore these three areas. In particular, the research design involved different sequences of learning activities as students worked with agent-based computer models of the carbon cycle and the greenhouse effect. In addition to the pre-post test assessments of conceptual understanding of climate change knowledge, our analysis of students’ learning makes use of process discovery techniques involving log file data of student dyads using the computer models.

Method
A classroom experiment was devised in which students from three year nine science classes were randomly assigned to two treatment groups: Challenge and Guided Learning (CGL) (i.e., low-to-high sequence of pedagogical structure) and Guided Learning (GL) (i.e., high-to-high sequence of pedagogical structure). Both

© ISLS

25

ICLS2012

Volume 1: Full Papers

groups experienced two classroom periods of learning with two computational models relating to climate science (described below). For each of the two models, three problem-based activities were prepared. The two experimental groups (CGL and GL) differed only in the structure provided in the first of two activities with each model, so that the second activity and assessment were identical across both groups. For the CGL group, this involved the posing of an initial challenging question with no instructional support, e.g., “What is the difference between the behavior of visible light (yellow rays) and infrared light (red rays)?” The GL group had the same question but was provided additional instructional support in the form of small guiding tasks that structured their exploration of the models in a way intended to help them answer the question successfully. This experimental design was used in previous studies in which significant learning outcomes were found (Kapur, 2006; Pathak, Kim, Jacobson, & Zhang, 2009). Students individually completed a pre-test and a post-test in which their knowledge about climate science was assessed, and then all learning activities with the computer models were carried out in dyads.

Figure 1: A NetLogo model of the greenhouse effect The agent-based models of the carbon cycle and the greenhouse effect were developed with NetLogo (Wilensky, 1999). The models were co-created by the first author and a collaborating schoolteacher with a PhD in physics, and were then verified by an earth sciences graduate. The carbon cycle model represents a fixed number of carbon molecules within a closed system. Carbon molecules exchange at the boundaries between atmosphere, land, and ocean. Students can alter the rates of transfer at these boundaries as well as control the rate of release of deeply stored carbon into the atmosphere. Carbon moves in different regions as well as in different life forms, represented as trees (plants) and sheep (animals). The greenhouse effect model allows students to explore the interaction between the sun’s energy (i.e., electromagnetic radiation) and the carbon cycle. This model builds on the simpler carbon cycle model that the students use first. Students can release carbon dioxide into the atmosphere and observe the effects over time. The output shows atmospheric gases and surface temperatures. The model includes parameters for the quantity of the greenhouse gasses of CO2 and H2O and includes a parameter for the Earth’s albedo or reflectiveness. Rules are specified for the interaction between electromagnetic radiation from the sun and the Earth’s surface (reflection, absorption) as well as for heat released from the Earth’s surface (infrared radiation [IR]) and greenhouse gases (CO2 and H2O). The resulting complex system has the emergent property of “global heat” that changes based upon the proportions of solar radiation/greenhouse gases/reflection in the system. The simulation generates output that was calibrated to be an approximation of real data for carbon and temperature (Intergovernmental Panel on Climate Change, 2007). Students interact with the models by changing the parameters and observing the effects. For example, in the

© ISLS

26

ICLS2012

Volume 1: Full Papers

model of the greenhouse effect shown in Figure 1, releasing CO2 into the atmosphere (by changing a slider, left side) leads to real-time visual feedback (in the visualization window, right side) as well as more CO2 molecules being present and an increased incidence of re-radiation of IR light leaving Earth’s surface. This is displayed in the graphs of temperature and carbon (lower left) that show an increase in the former and a shift in the equilibrium in the latter over time. The study was conducted in 2011 at an all girl’s high school in Australia, with three year nine classes (third year of secondary school) and a total of 90 students (30 students each). Students completed individual pre-tests and post-tests of their knowledge of climate science and complex systems. The pre- and post-tests consisted of 20 questions—six multiple choice and 14 open-ended short answer questions—that were intended to assess both declarative and conceptual knowledge of climate science. Students were given 20 minutes to complete each test. During the experiment, students carried out three activities with each of the two models in pairs (45 dyads in total). Each pair was randomly assigned to one of two conditions: (a) CGL where dyads had no structure for the first activity of each model, and (b) GL where dyads had a structured task for the first activity with each model. Students were allocated 80 minutes for working with each of the two models and completing the given activities. Computer screen recordings were made and log files of NetLogo interactions were maintained for all dyads. Additional webcam audio and video recordings were made for six dyad groups (three in each of the treatment groups).

Results
This section presents the main results related to our three research areas from the first year of the project. First we discuss the pre- and post-test assessment results. Second, we illustrate the potential of process analysis techniques to detect student behavior patterns from the NetLogo log files in our analysis of one high achieving and one low achieving dyad from CGL group. Of the initial 90 students, one of the classes was used as a pilot to tune data collection processes, and therefore was not included in the final analysis. Of the remaining 60 students, only 33 completed all tasks due to absences and technical difficulties (e.g., computer failures resulting in missing data). Ten short answer questions related to scientific knowledge of climate change phenomena were coded using an adapted five-point knowledge integration rubric (Gerard, Spitulnik, & Linn, 2010). This coding scheme elucidates students’ capabilities to make connections between different elements of the phenomenon, thus was appropriate for investigating the depth of students understanding. One question was removed as we determined it had unclear language, leaving nine questions. Initially, two individuals each coded 60% of the results with a 20% overlap for reliability. Cohen’s Kappa on this initial 20% was 0.64. The coders then met to discuss discrepancies and coded the remaining results so that each had rated all of the responses. Following discussion of discrepancies, 100% agreement was reached. These scores were scaled to a maximum of 100 and Table 1 summarizes these results. Table 1: Pre- and post- test results N Challenge & Guided Learning Guided Learning All students 15 18 33 Mean 31.73 34.67 33.33 Pre-Test SD (15.81) (12.27) (13.84) Mean 41.07 38.89 39.88 Post-Test SD (14.38) (13.22) (13.58)

Overall, there was a significant increase in the mean post-test scores of all the students, F(1,31)=9.367, p=.005, η2 =.232. Repeated measures ANOVA on the summed short answer responses with a between-subjects factor of the CGL/GL group found no significant differences, F(1,31) = 1.532, p = .225, η2 =.047, although the CGL group did have a higher score on the post-test than the GL group. The problem-based activity and assessment questions the dyads worked on were similarly coded and scaled (Table 2). Repeated measures ANOVA on the activity and assessment responses with a between-subjects factor of the CGL/GL group found no significant group differences, F(1,12) = 0.300, p = .594, η2 =.024, although again the CGL group had a higher mean scores than the GL group on both of the assessment tasks for each models.

© ISLS

27

ICLS2012

Volume 1: Full Papers

Table 2: Student scores for activity and assessment questions N (Dyads) Challenge & Guided Learning Guided Learning All dyads 6 8 14 Model 1 (Carbon Cycle) Mean SD 61.11 (16.39) 55.21 (9.90) 57.74 (12.85) Model 2 (Greenhouse Effect) Mean SD 38.89 (18.26) 37.50 (18.72) 38.10 (17.82)

Process Analysis
In this section we outline an approach that we are developing for analyzing the stream of data that is available from students engaged in learning activities with NetLogo agent-based models. The main goal of this early phase of our work is to explore possible ways that data mining techniques may provide insights into the students’ interactions with the NetLogo models. Educational data mining employs various algorithmic techniques on large amounts of institutional data records or student learning log files that are associated with online learning activities in order to discover student learning patterns (Romero & Ventura, 2006; Romero et al., 2010). This information then could be used to improve the design of software and learning materials or provide individualized feedback and support for learners during MBL activities. As a part of our initial analysis we are exploring several possible techniques to analyze students’ interaction patterns: Hidden Markov Model (HMM), process mining, and various statistical visualization techniques. In our initial analysis, we used the HMM constructing algorithm adapted from Jeong, Biswas, Johnson, & Howard, (2010). On the basis of sequences of student interactions with NetLogo models recorded in log files, the algorithm extracted HMMs consisting of hidden states that depict how students interact with the models. Student activities or actions are equivalent to the use of controls on the model interface: Setup button, Go button (that initiates Start and Stop of the simulation), Speed control, Fossil Fuel Use control, and Tracking CO2 Molecule control. The HMM shows the composition and the percentage of activities in each state as well as transitioning probabilities between these states. Students’ activities within individual states and likelihoods with which they move from one state to another together depict students’ overall behavior patterns. Each HMM is made up of a set of states and transitions: the behavior probabilities associated with each state and the transition probabilities between the states. For example, the HMM in Figure 2 shows that a high achieving dyad in State D (Fossil Fuel and Speed) explored the model by controlling the simulation in two distinct ways: by changing the amount of fossil fuel use 61% of the times (14 actions in total) and by changing modeling speed 39% of the time (9 actions in total). The link from one state to another and the percentage associated with the link indicate how likely it is that a certain state will be followed by another. For example, the HMM in Figure 2 shows the probabilities that the dyad would transition from State A (in which the they set up the simulation) to states B, C, and D. Specifically, the HMM for the high achieving dyad was composed from five states (Figure 2). Three simple states were associated with model control: Setup (State A), Stop (State C), and Stop Tracing of a CO2 molecule action (State E). Two composite states were associated with the model exploration: (a) start simulation and initiate tracing of a CO2 molecule actions (State B), and (b) control the fossil-fuel use and speed control (State D). The HMM shows that the high achieving pair often moved from the initial model setup (State A) to the control of the main model parameters and speed (State D), with the transition probability as high as 70%. The transition probability between the model manipulation (State D) and Start (State B) was also relatively high (48%). In contrast, the likelihoods of transition between Setup (State A) and Start (State B) and Setup (State A) and Stop (State C) were as low as 20% and 10% respectively. This suggests that after resetting the model (State A), the dyad often initially configured the main model parameters and only then started exploration of the model by switching on CO2 tracking or pressing Start (State B). The HMM also indicated that these students often explored the model by running and pausing it, with the transition probabilities between Start (State B) and Stop (State C), and vice versa, ranging between 47% and 50%. The HMM for the low achieving pair indicates that these students interacted with the model differently than did the high achieving dyad. The model was composed of four states only (Figure 3). Two simple states and one composite state were similar to the states in the HMM for high achieving students: Setup (Sate A); Stop (State C), and model manipulation (State D). One further simple state was formed from Start action (State B) only. The transitions indicated that the low achieving dyad’s behavior patterns could be characterized by two dominant moves between the three basic control buttons: Setup, Start, and Stop. The transition probabilities between Setup (State A) and Start (State B), and between Start (State B) and Stop (State C) were as high as 65% and 70%, respectively. All other students’ transitions among a range of states had medium or low probabilities, suggesting that the exploration was less systematic. In contrast to the high achievers’ HMM, the transition probability from the initial Setup (State A) to the model manipulation (State D) was only 10%. This result suggests that these students rarely setup the main parameters after resetting the model, but rushed to start the simulation (State B). Nevertheless, this dyad sometimes also manipulated some model parameters (State D). The students often stopped the simulation (State C), changed the parameters (State D), and then resumed (State
© ISLS 28

ICLS2012

Volume 1: Full Papers

B). It is also interesting to note that log file analysis revealed this dyad never attempted to track an individual CO2 molecule, unlike the high achieving dyad.

Figure 2: Hidden Markov Model of the high achieving dyad These results indicated that the model exploration strategies adopted by two dyads were different. The HMM for the high achieving dyad indicates that their explorations can be characterized by the interaction with the model, systematic control of the parameters, and model tracking behaviors. It also suggests that these students combined qualitative and quantitative explorations depicted by State D and State B, respectively. Their dominant interaction behavior can be broadly called a “Setup-Configure-Start/Follow” pattern. The HMM for the less successful dyad depicts more passive student interactions with the model and more surface exploration behaviors. Their dominant behavior pattern can be broadly called “Setup-Start-Stop,” which was then followed by the manipulation of model parameters. Overall, the HMM for the low achieving dyad indicates that these students did not engage in systematic explorations of the agents depicted in the carbon cycle model.

Figure 3: Hidden Markov Model of the low achieving dyad In order to triangulate the HMM results, we used other core process visualization and statistical techniques to explore student interactions by constructing interaction graphs (Figure 4 and Figure 5). These graphs depict dyads’ interactions with the carbon cycle models during the experimental period. The vertical axis represents the time line, and the horizontal axis shows the total number of various model control actions taken from the beginning of the session. The interaction graph for the high achieving dyad confirms that the students actively explored the model during the entire experimental sequence by using three main simulation controls and parameters: fossil fuel use, speed, and tracking CO2 molecule (Figure 4). It also depicts that during the first half of the session

© ISLS

29

ICLS2012

Volume 1: Full Papers

students changed the fossil fuel rates and speed, which suggests simultaneous quantitative and qualitative explorations for tracking of a CO2 molecule. However, it also reveals that there were noticeable changes in the student behaviors over time. For example, these students changed the CO2 tracking setting during the first half of the session only. This was followed by a shorter, but a noticeable period, when students actively controlled the fossil fuel use. At the end of the session, they interacted with the model simply by pausing and resuming the simulation without changing its settings.

Figure 4: Interaction graph of the high achieving dyad Similarly the interaction graph for the low achieving dyad generally reaffirmed the pattern depicted by the HMM (Figure 5). It showed that these students interacted with the model much longer, but their interaction was often limited to Setup, Start, and Stop actions. The interaction graph further revealed noticeable changes in the student behavior during the session. For example, in more than half of the session, these two students changed only the fossil fuel use values. This was followed by a short interval when the students started to change and manipulate only speed, and they then proceeded to changing the two parameters simultaneously.

Figure 5: Interaction of the low achieving dyad

Discussion and Conclusion
In terms of the implications of these findings for the research interest related to sequences of pedagogical structure, we note that participants in both the Challenge and Guided Learning (CGL) and Guided Learning (GL) sequences made significant learning gains. Although there were not significant learning differences between the two treatment groups, the main findings were actually in the hypothesized directions, with the CGL group scoring higher on the model problem solutions (especially Model 1) and on the posttest than did the GL group. We note that this study involved a relatively small number of students, and we are planning future studies that will involve larger numbers by collecting data in additional schools, which should provide greater statistical power to further investigate aspects of the sequencing of pedagogical guidance for agent-based
© ISLS 30

ICLS2012

Volume 1: Full Papers

models and learning. Still, given there have been few reported empirical studies of learning in this area, an important finding in this study is that on average all participants significantly enhanced their understanding of important scientific knowledge about greenhouse gases and the carbon cycle. The process analysis is an ongoing research theme in this project. We aim to find key patterns of students’ interactions with the models, and find ways to identify “on-the-fly” effective and inefficient behaviors for learning with agent-based computer models. A current limitation is that the algorithm that we used to construct the HMMs has a local maxima problem and the log data files are relatively small. In our study, we executed the algorithm one hundred times with random initializations by sampling the initial parameter values from uniform distributions (Jeong et al., 2010; Southavilay, Yacef, & Callvo., 2010). The HMMs were constructed when these executions converged to the same configuration. A better solution is needed to execute this algorithm if we are to use the HMM for providing feedback to students and teachers in real time. Further, the HMM algorithm applied to the student model generated log files identifies patterns that are directly rooted in students’ technical fine-grained actions and constructs only one model for the entire interaction sequence. Two key challenges need further exploration and solution. First, as we illustrated above, students’ behavior patterns may change during their interaction with the model. For example, we expect that one of possible reasons for the change in student interactions depicted in the timelines (Figure 4 and 5) was the change between the three activities associated with each model. Other research (Levy & Wilensky, 2010) has that found students tend to apply similar strategies when they interact with different models. In addition, their interaction patterns are not fixed and students, particularly successful learners, may flexibly adapt their interaction strategies as their goals change (i.e., when students complete one activity or answer a question and move to the other). The analytical techniques should be flexible enough to identify such macro level changes (i.e., shifts between stages or goals) in student exploration strategies and then construct interaction patterns for each stage separately. Second, when modeling is based on the raw log data, it directly links students’ activities to the students’ technical fine-grained interactions with the model via the interface (i.e., “button click” is equal to student activity). These “technical actions” are likely not identical to the “meaning actions” that are associated with the students’ intentions and conscious behavior. Drawing on evidence from cognitive research, we can expect that “meaning actions” are likely to be more abstract “chunks” of common sequences of student behavior (i.e., clusters of common strings of “technical actions”). Mining techniques are needed for identifying “meaning actions” in student logs initially, which then could be used for constructing further student interaction models. To address the first challenge, we are currently working on two goals. First, we will put a time/date stamp into the NetLogo log files to mark the beginning and ending of the specific problem solving activities the students are working on. This time/date stamp will allow us to identify possible changes in students’ strategies as they are engaged in the different model-based problem activities; in particular, different model use behaviors in sequences of low and high structured tasks associated with Challenge and Guided Learning and Guided Learning. Second, in order to gain deeper insights into how students performed their activities, we are also exploring possibilities to incorporate process mining techniques, particularly in the initial exploratory stage of the analysis (Trčka et al., 2011; van der Aalst, 2011). Addressing the second challenge, we plan to utilize a Hidden Markov Model clustering algorithm (Shih, Koedinger, & Scheines, 2010) and explore other similar techniques. By clustering and learning HMMs rather than just one for each student group, we can discover different strategies that students employed and to use those strategies to understand and, perhaps in the future, to predict learning outcomes. This combination of techniques should provide a good fit between automatically identified behavior patterns to students’ conscious behavior. In terms of more general research issues highlighted at the start of this paper, these results are encouraging for the continuation of this design-based research, and we are hopeful that the process mining approaches we are developing may be useful in providing further insights into the dynamics of learning scientific knowledge with computer model based learning activities.

References
ACARA. (2011). The Australian Curriculum: Science (Version 1.2). Sydney: Australian Curriculum Assessment and Reporting Authority. Clement, J. (2000). Model based learning as a key research area for science education. International Journal of Science Education, 22(9), 1041-1053. Gerard, L. F., Spitulnik, M., & Linn, M. C. (2010). Teacher use of evidence to customize inquiry science instruction. Journal of Research in Science Teaching, 47(9), 1037-1063. Goldstone, R. L., & Wilensky, U. (2008). Promoting transfer through complex systems principles. Journal of the Learning Sciences, 17(4), 465-516. Intergovernmental Panel on Climate Change. (2007). Fourth Assessment Report of the Intergovernmental Panel on Climate Change, .

© ISLS

31

ICLS2012

Volume 1: Full Papers

Jacobson, M. J., Kapur, M., So, H.-J., & Lee, J. (2011). The ontologies of complexity and learning about complex systems. Instructional Science, 39, 763-783. doi: 10.1007/s11251-010-9147-0 Jeong, H., Biswas, G., Johnson, J., & Howard, L. (2010). Analysis of productive learning behaviors in a structured inquiry cycle using hidden markov models. Paper presented at the 3rd International Conference on Educational Data Mining, Pittsburgh, PA, USA. Kapur, M. (2006). Productive failure. In S. Barab, K. Hay & D. Hickey (Eds.), Proceedings of the International Conference of the Learning Sciences (pp. 307-313). Mahwah, NJ: Erlbaum. Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379 - 424. Kapur, M. (2010). A further study of productive failure in mathematical problem solving: Unpacking the design components. Instructional Science, DOI: 10.1007/s11251-11010-19144-11253. Levy, S. T., & Wilensky, U. (2010). Mining students’ inquiry actions for understanding of complex systems. Computers and Education, 56(3), 556-573. doi: 10.1016/j.compedu.2010.09.015 Mayer, R. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14-19. National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a Conceptual Framework for New K-12 Science Education Standards. Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press. Pathak, S., Kim, B., Jacobson, M. J., & Zhang, B. (2009). Failures and successes in collaborative inquiry: Learning the physics of electricity with agent-based models. In C. O’Malley, D. Suthers, P. Reimann & A. Dimitracopoulou (Eds.), Computer-supported collaborative learning practices: CSCL2009 conference proceedings (pp. 199-203). Rhodes, Greece: International Society of the Learning Sciences. Railsback, S. F., & Grimm, V. (2011). Agent-based and individual-based modeling: A Practical Introduction. New Jersey: Princeton University Press. Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239-257. Romero, C., & Ventura, S. (Eds.). (2006). Data mining in e-learning. Southampton: WITpress. Romero, C., Ventura, S., Pechenizkiy, M., Ryan, S., & Baker, R. (Eds.). (2010). Handbook of Educational Data Mining. Boca Raton: CRC Press. Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–522. Shih, B., Koedinger, K. R., & Scheines, R. (2010). Discovery of learning tactics using Hidden Markov Model clustering. Paper presented at the 3rd International Conference on Educational Data Mining, 2010, Pittsburg, PA, USA. Southavilay, V., Yacef, K., & Callvo., R. A. (2010, June 11-13). Process Mining to Support Students' Collaborative Writing. Paper presented at the International Conference on Educational Data Mining, Pittsburgh, PA, USA. Trčka, N., Pechenizkiy, M., & van der Aalst, W. (2011). Process Mining from Educational Data. In C. Romero, S. Ventura, M. Pechenizkiy & R. S. J. d. Baker (Eds.), Handbook of Educational Data Mining (pp. 123-142). Boca Raton: CRC Press. van der Aalst, W. M. P. (2011). Process Mining: Discovery, Conformance and Enhancement of Business Processes: Springer. VanLehn, K., Siler, S., & Murray, C. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 2(3), 209-249. Wilensky, U. (1999). NetLogo. Evanston, IL: Center for Connected Learning and Computer-Based Modeling. Northwestern University (http://ccl.northwestern.edu/netlogo). Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep or a firefly: Learning biology through constructing and testing computational theories -- an embodied modeling approach. Cognition & Instruction, 24(2), 171-209. Zhang, B., Liu, X., & Krajcik, J. S. (2005). Expert models and modeling processes associated with a computermodeling tool. Science Education, 90, 579-604.

Acknowledgments
The research reported in this paper was funded by an Australia Research Council Linkage Grant No. LP100100594 to the second and third authors. We thank Dr. Paul Stokes, Kashmira Dave, and the participating teachers and students for their invaluable assistance in this study.

© ISLS

32

ICLS2012

Volume 1: Full Papers

Learning how to create: Toward a learning sciences of art and design
R. Keith Sawyer, Washington University in St. Louis Abstract: This paper is the first report on an extensive ethnographic study of two professional schools of art and design in the United States. The overall purpose of the study is to identify general principles for how to design learning environments that prepare learners to be creative. First, I document the cultural model of teaching and learning held by the faculty and students, and analyze the pedagogical practices used. This studio model is of interest because it emerged naturally in a community of educational practice. I argue that it is distinct from the two cultural models most familiar to learning scientists: instructionism and apprenticeship. Second, I argue that the studio model is more closely aligned with learning sciences principles than either instructionism or apprenticeship. Third, I draw lessons from this studio model for designing learning environments in all school subjects, in particular STEM subjects.

Introduction
How can we teach students to be creative? How can we teach them disciplinary knowledge in a way that prepares them to build on it and to go beyond? These questions are increasingly important in education due to a widely acknowledged global transformation to a knowledge-based society and a creative economy (Florida, 2002; Sawyer, 2008). In the United States, many learning scientists have joined with policy makers to advocate that schools should teach “21st century skills” including creativity (Trilling & Fadel, 2011). In the United Kingdom, national policy since the late 1990s has emphasized the creative industries as an economic strategy, and creativity is a part of the national curriculum (NACCCE, 1999). Internationally, the OECD has generated a series of reports about the need for schools to become creative learning environments (e.g., OECD, 2006, 2008). The future of learning must centrally involve creativity. This paper is the first report on an extensive ethnographic study of two professional schools of art and design in the United States. The goal of the study was to document the cultural model of teaching and learning held by the faculty and students, and to document and analyze the pedagogical practices used. Two institutions were chosen: the Savannah College of Art & Design (SCAD), a free-standing art and design college dedicated to art and design with a career focus; and Washington University (WU), where the art and design school is embedded within a comprehensive research university, with a more theoretical and research focus. These very different sites were chosen to ensure that the identified cultural model would be more likely to be general to all art and design schools rather than specific to a certain type of institution. Data include videotapes of studio classes and critiques, audiotaped interviews with professors, audiotaped interviews with students, photos of student work, and copies of curricular materials provided by professors. As of Fall 2011, the data-gathering phase is complete and transcription has begun. Although the transcription and coding is expected to continue through 2012, it is already possible to identify a core set of features of the cultural model of teaching and learning, and this paper presents the first preliminary findings from the study. This study is of importance to learning scientists for three reasons. First, it documents a naturally occurring cultural model, one that emerged over time in a professional community of practice. Thus it occupies an analogous position to the various ethnographies of apprenticeship that so strongly influenced learning sciences in the 1980s and 1990s (e.g., Collins, Brown, & Newman, 1989; Lave & Wenger, 1991; Rogoff, 1990). Second, I will argue that this cultural model is strongly consistent with learning sciences findings, and yet that it presents an intriguingly different vision of teaching and learning. Third, I will argue that this cultural model can help us to resolve a fundamental paradox facing learning sciences theory and research: How can students learn the basic facts, concepts, and skills of a discipline, and yet learn them in such a way as to be prepared to be creative and to go beyond them? To date, learning scientists have rarely addressed creativity explicitly. However, many of the reforms advocated by learning scientists are designed to result in learning outcomes that researchers believe result in higher creative potential (Sawyer, 2008). For example, when one learns deeper conceptual understanding rather than superficial knowledge, one is better prepared to transfer that understanding to a new situation, resulting in adaptive expertise (Hatano & Inagaki, 1986; Schwartz, Bransford, & Sears, 2005) or adaptive competence (de Corte, 2010). Several learning scientists have been inspired by studio practices in design education and engineering education, and have adopted studio practices in K-12 math and science classes (e.g., Cossentino & Shaffer, 1999; Cox, Harrison, & Hoadley, 2009; Greeno, 1997; Jacobson & Lehrer, 2000; Hmelo, Holton, & Kolodner, 2000), but without empirically documenting the range of practices found in art and design education, and without fully theorizing and articulating how such environments might foster greater creativity. This project intends to build on and extend this previous work.

© ISLS

33

ICLS2012

Volume 1: Full Papers

Learning scientists have argued that the traditional cultural model of teaching and learning, referred to as instructionism (Papert, 1980), transmission and acquisition (Rogoff, 1990), or the standard model (OECD, 2008) is not grounded in the science of learning. Beginning in the 1980s, education researchers searched “in the wild” for other cultural models of teaching and learning, and were drawn to apprenticeship (Collins, Brown, & Newman, 1989; Lave & Wenger, 1991; Rogoff, 1990). These two cultural models have often presented as two poles of an axis from formal learning to informal learning. I argue that the studio model represents a third cultural model of teaching and learning, distinct from both instructionism and apprenticeship. Documenting this cultural model is the first goal of this paper. The second goal is to demonstrate that this cultural model is closely aligned with learning sciences findings, and thus to demonstrate that those findings are not restricted to STEM disciplines. Learning scientists have argued that their findings reflect general properties of human cognition that underlie all learning. And yet, learning sciences research has occurred almost exclusively in STEM disciplines. This study serves as an important validation test of the generality of learning sciences findings. The third goal is to argue that the studio model has implications beyond art and design education; that it provides lessons for how to resolve longstanding tensions faced by learning scientists and educators in STEM disciplines. Most importantly, I argue that studio learning environments resolve a fundamental paradox of constructivist teaching: how to ensure that learners master core facts and skills, while at the same time learning how to use that knowledge creatively? Sawyer (2011) argued that such teaching is best thought of as an improvisational activity, and that the best teaching is disciplined improvisation: teaching that provides space for the flexibility required for constructivist learning, but guided within structures and frameworks—in a similar fashion to professionally performed improvisations found in jazz and improv theater. In this paper, I show that the teaching practices of professional schools of art and design represent a form of disciplined improvisation, and thus show us how to teach for creativity.

Related studies
Hetland et al. (2007) studied the teaching practices of five teachers at two secondary schools, and identified three practices: demonstration/lecture, student work, and critique. Their analysis was focused on studio art, and did not draw connections to the learning sciences. As in most other academic disciplines, university instructors often engage in reflection and writing about their own teaching practice. Design educators have written about design education (e.g., Design Studies 1999 special issue); and architecture educators have written about architecture education (e.g., Dutton, 1991; Nicol & Pilling, 2000). Although these writings are helpful, they tend to be focused on practitioner issues and rarely explore underlying models of learning. Two exceptions are (Schön, 1985) and (Shaffer, 2003) which report on ethnographies of educational practice in architecture studios. There have been several studies of design expertise (Cross, 2011; Lawson & Dorst, 2009) and design practice (Hall & Stevens, 1996). Many of these studies identify various stages of the design process and the iterative cycling through them; a typical model includes stages such as (1) explore the problem, (2) generate a range of concepts, (3) select the most promising concept, (4) develop the concept into a detailed design, (5) communicate the final proposed design. This study builds on previous work in several ways: (1) although prior studies have described specific practices, none document the complete cultural model of teaching and learning shared in this community of practice; (2) for the most part, each study tends to focus on one discipline (art, architecture, or engineering) whereas this study attempts to identify what is common across all art and design disciplines; (3) previous studies tend to focus on one class, teacher, or project, and thus are not able to identify the way that learning trajectories are designed for the learner, nor the place of any one practice in it; (4) finally, this study aims to identify general principles, grounded in the learning sciences, that can be applied to all academic disciplines, particularly STEM disciplines. The most closely related prior studies are (Shaffer, 2003) and (Hetland et al., 2007); this study is inspired by those studies, and attempts to be both broader and deeper.

Methodology
The author conducted an ethnographic study of two professional schools of art and design. Two very different types of institutions were chosen to ensure that the cultural model identified would be more likely to be general to all art and design schools rather than specific to a certain type of institution. The Savannah College of Art & Design (SCAD) is a freestanding institution dedicated to art and design. It is proud of its focus on “creative careers” and its ability to place students in creative industry jobs after graduation. It is the largest art and design school in the United States, with over 9,000 students and more than 40 different majors offered, ranging from painting and sculpture to architecture, industrial design, furniture design, advertising, videogame design, and movie production.

© ISLS

34

ICLS2012

Volume 1: Full Papers

The Sam Fox School of Design & Visual Arts is embedded within Washington University (WU), a comprehensive research university. Students are required to take one course each semester outside of the Sam Fox School. Most students take classes in the College of Arts and Sciences, but some take classes in the School of Business or even the School of Engineering. Compared to SCAD, there is more of a focus on theoretical foundations, conceptual underpinnings, and a notion of arts practice as research. WU is substantially smaller, with under 1,000 students and just over ten undergraduate and graduate degrees. At SCAD the author served as a Visiting Professor for two academic quarters (a total of six months) and conducted three other shorter site visits. At WU the author has been a professor since 1996, in a different academic unit but with an office about ½ mile distant. Observations and interviews occurred during two academic semesters. Both sites combined resulted in a total of approximately 75 hours of video observations and audiotape interviews, as follows: Table 1: Data gathered. Classes videotapes 6 (2.5 hours each) 6 Instructors interviewed 18 16 Students interviewed 6 4

SCAD WU

This conference paper is a preliminary report; not all materials have been transcribed and coded. However, based on initial analyses of these materials, several preliminary findings have emerged. By the time of the conference in Summer 2012, specific quotations and video episodes will be available for presentation.

Findings
Based on a preliminary review of the ethnographic materials, the studio model can be characterized by the following features.

Complex, authentic, real-world projects
In all of the classes observed, instructors carefully design projects to initiate and guide student work. A project poses a problem that is open ended, so that there is no single obvious way to resolve the problem, and so that each student must identify their own path to a solution. In design terms, these are ill defined or wicked problems. In the first two years of the curriculum, the projects typically take two to three weeks to complete, and over the course of a quarter or semester term, four to five projects will be completed. In the latter two years of the curriculum, projects become progressively more open ended and involve longer periods of work, fewer constraints, and greater student discretion. In these later classes, it is common for one project to absorb the entire term. The projects are designed so that students will not be able to complete them unless they master specific knowledge and skills that have been identified by the instructor as the desired learning outcomes. Instructors believe that students will learn the required knowledge and skills more effectively if they learn them while engaged in a process of authentic problem solving. Instructors justify this method of instruction by pointing out that in the art and design professions, problems are complex wholes, and cannot be reduced to individual component skills. “Real world projects are complex and all the parts are interrelated. It’s not linear” (Ella, SCAD interior design).1 They believe that teaching individual skills apart from authentic projects would not properly prepare students for professional disciplinary practice. In many classes in design disciplines, instructors work with external industry partners as clients. This is particularly common in disciplines like industrial design and graphic design. Executives from the client firm will attend a class early in the term to present the problem to be solved, and will visit again at the end of the term, when student teams will present their final work to the client. A common theme in learning sciences based curricula is that students are encouraged to engage in activities that are similar to those practiced by professionals in the discipline (e.g., Edelson, 2006; Krajcik & Blumenfeld, 2006). Because this study analyzes professional schools with the goal of preparing students for professional practice, it is not surprising that authentic practice is heavily emphasized.

Guided problem solving
Although projects are open ended so that each student must find their own way to a solution, projects also typically specify a set of constraints or parameters that restrict how the student can proceed. Here are two representative examples of project assignments. In a Communication Design class: The attached population data compares populations in 2000 and 2010 for all countries and regions with more than 5000 people. Using at least the first two columns, design two different

© ISLS

35

ICLS2012

Volume 1: Full Papers

approaches to visualize this data. One of these approaches must make some use of some aspect of a world map (shape of countries, relative position in the world, etc.) Each of your two solutions must be able to fit within a 20” x 16” format, but can go smaller. You may use a maximum of two colors, black and a second color of your choice. (Heather Corcoran, WU, Spring 2011) In a first year 2D design class: Create a project that conveys (1) something that just happened; (2) something happening; or (3) something about to happen. You can only use a quarter or less of a figure, you must use these fifteen objects provided, and it must be in an architecturally significant interior. (Kathy, SCAD, Winter 2010) When I asked professors why they provided such specific constraints, they provided a variety of intriguing answers that invoke many learning sciences principles and warrant further exploration. First, the goal of project parameters is to limit students’ creative options, so that their creative effort is focused on the specific learning outcomes desirable at that point in their learning trajectory. John Hendrix (WU, illustration) said “Project constraints set you free. It focuses the learning because it removes the number of variables they have to think about.” Second, instructors often said that without such constraints, students would fall into familiar patterns. Julie Varland (SCAD, architecture) said “We are peeling away their assumptions and forcing them into discomfort… without structure, they’d do what they know already”. Arne Nadler (WU, core) said “You find these things through doing, not while you are thinking about it.” Other instructors said “Students need the structure. It gives them something to push against” (SCAD reading group). This practice is consistent with learning sciences studies of misconceptions (diSessa, 2006): The constraints are thought to be necessary to help the students to deeply challenge their own misconceptions, and then to guide them to a more advanced conception. Third, instructors said that the constraints were valuable because they would lead students to fail early, thus more deeply realizing the need to move beyond their existing misconceptions. John Hendrix said, “Students should come in with an expectation and then have it proven wrong. Then, we redirect them down the right path.” Scott Thorpe (SCAD, sculpture) said, “People don’t change their model of thinking unless there’s a consequence. I wanted them to fail.” Rhonda Arntsen (SCAD, graphic design) said “I encourage failure.” This practice is consistent with several cognitive studies showing that learner failure results in more effective learning (impasse driven learning of Van Lehn, 1988; desirable difficulties of Bjork & Linn, 2006; productive failure of Kapur, 2008). Fourth, instructors often said that constraints were necessary to force students to slow down. Professors report that beginning art and design students attempt to move too quickly toward a solution. “I want students to stop and think. Don’t start working right away” (Rhonda Arntsen). An important outcome of art and design education is that students master a deliberate process that will result in consistently effective work. Prior research (see “Related studies”) has documented that expert designers engage in an early exploratory phase, which students tend to rush through. One painting instructor at SCAD, Susan, assigned a project that required students to use the lost medieval technique of glazes—not only because glazes were a valuable technique for a painter, but also because glazes take a long time to dry, and cannot be applied until the under painting has dried, thus forcing students to slow down. The constraints of a project assignment are the subject of substantial thought and explicit discussion among instructors. Project assignments are continually revised, a process familiar to learning scientists as design based research (Barab, 2006). Instructors often talked about projects that didn’t work and required revision. The most common problem was that not enough constraints were specified. In this situation, what typically happens is that all students generate a too-obvious solution; their misconceptions are not successfully challenged. A less common problem is that too many constraints are specified. In this situation, all students tend to generate very similar solutions. The studio model is similar to project-based learning, a learning-sciences based science curriculum (Krajcik & Blumenfeld, 2006). According to Krajcik and Blumenfeld, learning environments that are project based have five key features: 1. 2. They start with a driving question, a problem to be solved. Learners explore the driving question by participating in authentic, situated inquiry—processes of problem solving that are central to expert performance in the discipline. As students explore the driving question, they learn and apply important ideas in the discipline. Students, teachers, and community members engage in collaborative activities to find solutions to the driving question.
36

3.

© ISLS

ICLS2012

Volume 1: Full Papers

4. 5.

While engaged in the inquiry process, students are scaffolded with learning technologies that help them participate in activities normally beyond student ability. Students create a set of tangible products that address the driving question. These are shared artifacts, publicly accessible external representations of the class’s learning.

Only feature (4), technology scaffolding, is not found in art and design education. Feature (3), collaboration, was often observed in art and design classes, but collaboration is beyond the scope of this paper. Feature (5) is at the core of art and design education and will be discussed further below.

Curriculum design
Instructors design open-ended projects so that as students develop their own creative solutions, they attain the desired learning outcomes. Douglas Dowd (WU, illustration) said, “The learning is embedded in the act.” Instructors believe that simply warning students about common problems, or describing how to use standard techniques, will not work. Students have to learn through problem solving. Each term, a class begins with high scaffolding and progressively fades to less scaffolding. Over the four year undergraduate curriculum, fading occurs on a broader scale. Project assignments are carefully sequenced so that desired techniques and domain knowledge are acquired in the process of engaging in authentic problem solving activity, and so that each project’s knowledge prepares the learner to begin the next project. Amy Thompson (WU, printmaking) said “Every project builds on the project before it”. In many design disciplines, there is a standard sequential practice in the discipline (see “Related studies” above) that students must learn. In the course of a two-week project, a studio class will typically meet for four times—twice each week for 2.5 hours—and for each of these four classes, the instructor will specify an assignment that moves the student to the next stage of the design process. This practice scaffolds and externalizes the design process used by professionals in the discipline. For example, in a 3D graphics studio in SCAD’s advertising department (Joe diGioia), a term-length project required students to work through all of the stages of a real project: identify a target market, create a logo, develop prototypes, provide preliminary sketches, and at the end, present a final model. As students engaged in this project, at each stage they learned specific techniques and skills; yet these were contextualized within an ongoing authentic activity. Instructors’ comments frequently remind students of what they were working on in previous weeks, how their work has evolved, and mention potential next steps for the work. In a painting class, Susan Russell asked students to estimate what percentage of the work they had completed so far, and then asked “What comes next in your painting?” One student was asked, “What will you do in your next hour of work on this painting?” Instructors constantly reinforce the embedded, authentic nature of creative work. Learning sciences research has demonstrated that constructivist learning processes must be carefully guided to result in maximum effectiveness. One group of studies has shown that discovery learning, a relatively unguided version of constructivism, is less effective than instructionism (Mayer, 2004). The consensus emerging from learning sciences research is that the most effective learning environments are fairly highly constrained, while nonetheless providing students with the opportunity to engage in authentic, situated inquiry practices. Art and design students are substantially scaffolded by the parameters of the project assignments, and by the sequencing of projects over the course of a school term.

Externalization and reflection
One of the most fundamental characteristics of art and design learning is that a learner’s developing understandings are externally represented in a visible artifact. “Ideas come from making things… in design, making is thinking” (Ken Botnick, WU, printmaking). Catalina Freixas (WU, architecture) had an explicit rule in her class, “If it’s not on your drawing then we can’t discuss it,” because of her frequent experience of students wanting to talk about what they thought they were doing (but had not actually done). Studio conversation is always focused on the external artifact generated by the student. “Students have to explain their choices and back them up” (Amy Thompson). In one architecture critique I observed, a visiting architect’s first comment to a student was “I don’t understand what we are looking at” and later, “You’re not saying what you are doing and giving it a name” (Freixas studio 11 May 2011). Such comments refer to one of the most common problems with student work: that the student’s intention for the work is not realized in the actual work. When this happens, instructors ask questions like: “Can anyone help him figure out what this is about?” (Susan); “Why do you want to pursue a solution like this?” (Ella). In the studio, reflection is constantly required. Why this choice? Why this action? What are you trying to do? Why are you using this technique? Learning sciences has found that environments that foster reflection about learning are more effective (e.g., Bransford et al., 2000). These project-based learning environments are designed to guide students toward an increasing ability to link intention with making.

© ISLS

37

ICLS2012

Volume 1: Full Papers

Balancing domain skill learning and creative abilities
Expert performance in most domains requires a great deal of repetitive practice, and most knowledge intensive professions require that a large body of facts and skills be mastered (Ericsson, 2002). In professional schools of art and design—where creativity is more valued than in any other discipline—basic skills are viewed as essential and are constantly emphasized. Yet their learning is embedded in the authentic professional practices of the discipline. Skills are often taught opportunistically, as a question or problem arises in the course of a student’s project work. For example, in Susan’s painting class (22 Feb 2010), a review of one student’s painted portrait gave the instructor an opportunity to tell the entire class about classic techniques for shadow and background. Responding to these “teachable moments” requires a high degree of pedagogical content knowledge; because the syllabus is not structured around specific facts and skills, instructors must keep in mind their own set of desired learning outcomes and wait for good opportunities to introduce them. The studio model demonstrates that teaching creativity is not opposed to repetition, craft, and skill— that in fact, creative performance depends on their mastery. Thus the studio model can help STEM educators to resolve an ever-present tension: how to design scaffolded constructivist learning environments, such as projectbased and inquiry-based approaches, that also ensure that students master basic facts and skills.

Conclusion
This study contributes to the learning sciences in the three ways identified in the introduction. First, the learning sciences have almost exclusively studied learning in science, technology, engineering, and math (STEM). Learning scientists believe that their findings apply to learning in all domains, and yet this belief has not been empirically validated because there has been so little learning sciences research in other disciplines, particularly in the arts. This study confirms that the principles of effective learning that have been identified in the learning sciences, primarily through the study of STEM disciplines, apply equally to art and design. Second, I have argued that the studio model is closely aligned with learning sciences findings—more so than either instructionism or apprenticeship. This is particularly intriguing because the studio model emerged over historical time in a community of practice. Instructors all reported that they learned how to teach on their own, drawing on memories of their own favorite instructors, or on conversation with colleagues. Each instructor develops their own project assignments (they do not use textbooks or online repositories of good project ideas). Third, the learning environments that are found in schools of art and design are well articulated and carefully designed, to an extent that has been difficult to attain in STEM curricular reforms based on learning sciences findings. Projects are linked together sequentially through each term, and the classes are linked together across the four year undergraduate curriculum. A sustained study of the nature of this sequencing could provide valuable insights into how to redesign STEM curricula. Most importantly, these learning environments are designed with the goal of resolving a fundamental paradox of constructivist teaching: how to ensure that learners master core facts and skills, while at the same time learning how to use that knowledge creatively? In the future of learning, in the 21st century’s creative age, these are the same goals that we have for the reform of STEM education.

Endnotes
(1) Many participants requested that I use their actual name, wanting to be credited for their pedagogical practices, so for those participants I provide both first and last name. Other participants requested a pseudonym, and for those participants I use only a first name pseudonym.

References
Barab, S. (2006). Design-based research: A methodological toolkit for the learning scientist. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 153-169). New York: Cambridge University Press. Bjork, R. A., & Linn, M. C. (2006). The science of learning and the learning of science: Introducing desirable difficulties. APS Observer, 19(3), 29,39. Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453-494). Mahwah, NJ: Erlbaum. Cossentino, J., & Shaffer, D. W. (1999). The math studio: Harnessing the power of the arts to teach across disciplines. Journal of Aesthetic Education, 33(2), 99-109. Cox, C., Harrison, S., & Hoadley, C. (2009). Applying the "studio model" to learning technology design. In C. DiGiano, S. Goldman & M. Chorost (Eds.), Educating learning technology designers (pp. 145-164). New York: Routledge. Cross, N. (2011). Design thinking: Understanding how designers think and work. Oxford: Berg.
© ISLS 38

ICLS2012

Volume 1: Full Papers

de Corte, E. (2010). Historical understandings in the understanding of learning. In H. Dumont, F. Benavides & D. Istance (Eds.), The nature of learning: Using research to inspire practice (pp. 35-67). Paris: OECD. diSessa, A. A. (2006). A history of conceptual change research. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 265-281). New York: Cambridge University Press. Dutton, T. A. (Ed.). (1991). Voices in architectural education: Cultural politics and pedagogy. New York: Bergin & Garvey. Edelson, D. C., & Reiser, B. J. (2006). Making authentic practices accessible to learners: Design challenges and strategies. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 335-354). New York: Cambridge University Press. Ericsson, K. A. (2002). Attaining excellence through deliberate practice: Insights from the study of expert performance. In M. Ferrari (Ed.), The pursuit of excellence in education (pp. 21-55). Hillsdale, NJ: Erlbaum. Florida, R. (2002). The rise of the creative class and how it's transforming work, life, community and everyday life. New York: Basic Books. Greeno, J. G., & The Middle School Mathematics Through Applications Project. (1997). Theories and practices of thinking and learning to think. American Journal of Education, 106, 85-126. Hall, R., & Stevens, R. (1996). Teaching/learning events in the workplace: A comparative analysis of their organizational and interactional structure. In G. W. Cottrell (Ed.), Proceedings of the Eighteenth Annual Conference of the Cognitive Science Society (pp. 160-165). Hillsdale, NJ: Erlbaum. Hatano, G., & Inagaki, K. (1986). Two courses of expertise. Child Development and Education in Japan, 262272. Hmelo, C. E., Holton, D. L., & Kolodner, J. L. (2000). Designing to learn about complex systems. Journal of the Learning Sciences, 9(3), 247-298. Jacobson, C., & Lehrer, R. (2000). Teacher appropriation and student learning of geometry through design. Journal for Research in Mathematics Education, 31(1), 71-88. Kapur, M. (2008). Productive failure. Curriculum and Instruction, 26(3), 379-424. Kolodner, J. L. (2006). Case-based reasoning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 225-242). New York: Cambridge University Press. Krajcik, J. S., & Blumenfeld, P. (2006). Project based learning. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 317-333). New York: Cambridge University Press. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press. Lawson, B., & Dorst, K. (2009). Design expertise. Burlington, MA: Architectural Press. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of instruction. American Psychologist, 59(1), 14-19. NACCCE (National Advisory Committee on Creative and Cultural Education). (1999). All our futures: Creativity, culture and education. London, UK: DFEE. Nicol, D., & Pilling, S. (Eds.). (2000). Changing architectural education: Towards a new professionalism. New York: Spon Press. OECD. (2006). 21st century learning environments. Paris: OECD. OECD. (2008). Innovating to learn, learning to innovate. Paris, France: OECD. Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York: Basic Books. Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social context. New York: Oxford University Press. Sawyer, R. K. (2008). Optimizing learning: Implications of learning sciences research. In Innovating to learn, learning to innovate (pp. 45-65). Paris: OECD. Sawyer, R. K. (Ed.). (2011). Structure and improvisation in creative teaching. Cambridge, UK: Cambridge University Press. Schön, D. (1985). The design studio: An exploration of its traditions and potentials. London, UK: RIBA Publications Limited. Schwartz, D. L., Bransford, J. D., & Sears, D. A. (2006). Efficiency and innovation in transfer. In J. P. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 1-51). Greenwich, CT: Information Age Publishing. Shaffer, D. W. (2003). Portrait of the Oxford design studio: An ethnography of design pedagogy (Working paper No. 2003-11). Madison, WI: Wisconsin Center for Education Research. Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times. San Francisco, CA: JosseyBass. VanLehn, K. A. (1988). Towards a theory of impasse-driven learning. In H. Mandl & A. Lesgold (Eds.), Learning Issues for Intelligent Tutoring Systems (pp. 19-41). New York, NY: Springer.

© ISLS

39

ICLS2012

Volume 1: Full Papers

A Microgenetic Analysis of How Expansive Framing Led to Transfer with One Struggling Student
Diane P. Lam, Xenia S. Meyer, Randi A. Engle, Lloyd Goldwasser, Sarah Perez, Kathleen Zheng, Jim Clark, Erica Naves, Hernan Rosas, Danny Tan Graduate School of Education, UC Berkeley, 4646 Tolman Hall Berkeley, CA. 94720 dianelam@berkeley.edu, xenia.meyer@berkeley.edu, raengle@berkeley.edu Abstract: Research has shown that it is uncommon for students to transfer what they learn about science to their other classes, let alone their everyday lives. However, a set of recent studies provides empirical evidence that the expansive framing of learning contexts can foster transfer. In this paper, we further our investigations of the idea that transfer is promoted by the instructional practice of framing learning contexts expansively. We do so by offering a microgenetic analysis of a relatively surprising case of transfer by a student in a high school biology class. In particular, we use her case to evaluate five recently proposed explanations for exactly how expansive framing may foster transfer, and close with additional hypotheses for how transfer may have been supported with this particular student.

Introduction
If students are to be successful and if schooling is to have a significant impact on their lives, it is essential that students regularly transfer what they learn (Renkl, Mandl & Gruber, 1996; Schwartz, Bransford & Sears, 2005). Specifically, this research is based on the premise that it is not just the nature of the particular science content that students learn that matters for transfer, but also how learning and transfer contexts are framed. In this paper, we examine the idea that transfer can be promoted through an expansive framing of learning contexts by conducting a microgenetic analysis of a case of one struggling student in a high school biology class (Maxwell, 2004; Siegler, 2006). First, we explain and describe what we mean by expansive framing, and then briefly present recent studies that provide evidence that framing affects transfer. In the core of this paper, we evaluate how expansive framing may have promoted transfer for this particular student by evaluating five recently proposed explanations for how it may do so (Engle, Lam, Meyer & Nix, in press). We present relatively strong evidence for four out of the five explanations with data that we have collected in this target classroom. Finally, we close by proposing additional explanations for this student‟s unexpectedly high rate of transfer that may be less related to the expansive framing of her instruction.

What is Expansive Framing?
Framing is the meta-communicative act of characterizing what is happening and how different people are participating in a given context (Bateson, 1972; Goffman, 1974; Tannen, 1993). For example, a teacher can frame a lesson as a one-time event of learning something that students are unlikely to ever use again, or as beginning to discuss an issue that students are likely to be actively engaging with throughout their lives. Our claim is that the first kind of framing, which we refer to as “bounded,” will tend to discourage students from later transferring what they learn, while the second, which we refer to as “expansive,” will tend to encourage students to transfer what they have learned. This claim is consistent with several prior proposals by situative and sociocultural theorists (Greeno, Smith & Moore, 1993; Laboratory for Comparative Human Cognition, 1983; Pea, 1987) who argue that it is not just the physical aspects of contexts that matter for transfer (Catrambone & Holyoak, 1989; Spencer & Weisberg, 1986), but also how social interactions frame contexts as particular kinds of social realities (Gee & Green, 1998; Searle, 1995). There are several aspects of learning contexts that can be framed in ways that can affect transfer (see Figure 1). First, a learning setting may be framed in an expansive manner by extending it to include past and future times, different places, and additional participants. In contrast, a bounded framing of a learning setting may constrain it to the present time, place, and just a few participants. Furthermore, in an expansive framing of roles, learners may be positioned as active contributors and authors of their own ideas. In contrast, in a bounded framing of roles, learners may be positioned on the periphery of a learning context, where they are expected to report on the ideas of others. In the biology class we studied, the teacher exhibited expansive framing of all of these aspects with great frequency (Meyer, Mendelson, Engle & Clark, 2011). With regards to settings, the teacher regularly referenced other times like the beginning of the school year and later in life; other places like college and afterschool environments; and other participants like friends, family members, and scientists. For example, during a lesson on cells, the teacher explained the following:

© ISLS

40

ICLS2012

Volume 1: Full Papers

Now all this, all of this cell stuff, is kind of described on its own as the cell theory. And every biology student in the world, okay, maybe not the whole world but [inaudible] learns the cell theory. So whether you're going to be in this class, or you're moving to Texas, or you're going to be going to Berkeley, eventually someday you're gonna walk in, and they're gonna say, "Hey, tell me about the cell theory," so here you go. Here, the teacher explicitly mentioned times other than the present by saying “eventually someday,” he also referred to places outside of the classroom such as “the world,” “Texas,” and “Berkeley,” and involved other participants by including “biology students” and “they.” This type of expansive framing of settings was frequent in his instruction. Furthermore, with respect to roles, the teacher routinely expected students to author their own ideas, credited students for ideas that they shared with the class, and held them accountable for their contributions. A common activity he had students engage in was a class discussion of double multiple-choice questions that he projected on the board. During these activities, students were asked to individually choose an answer before the class was polled for their choices by a raise of hands. If discrepancies arose from the poll, students were expected to respectfully argue with each other for their answer choices until all students understood why one of the answers was the best choice. While these discussions took place, the teacher would stand quietly to the side unless he believed his input was needed to guide the students‟ conversation. Typically, the teacher‟s only contributions to the discussion were to ensure that the students directed their conversations at each other and not at him, credit a student for a statement he/she made and ask others to respond to it, and encourage individual students to argue for the answers they chose.

Evidence that Expansive Framing Promotes Transfer
A growing series of studies has empirically investigated connections between the framing of contexts and transfer. Several classroom case studies have provided evidence that transfer may be affected by framing in realworld learning environments (Engle, 2006; Engle Meyer, Clark, White & Mendelson, 2010; Hammer et al., 2005; Mendelson, 2010). Additionally, two experiments have systematically tested one or more aspects of this hypothesis, with results consistent with a causal connection between expansive framing and transfer (Engle, Nguyen & Mendelson, 2011; Hart and Albarracin, 2009).

Figure 1. How different aspects of learning environments can be framed expansively.

Five Proposed Hypotheses for How Expansive Framing May Foster Transfer
Although prior research has shown that framing contexts differently can affect transfer, it has not explained how this occurs. In Engle et al. (in press), we proposed five explanations--not mutually exclusive--for how expansive framing may promote transfer: 1) Connecting settings with each other encourages transfer during learning when students expect they will later need to transfer what they are learning and in response may be more likely to prepare for this possibility (e.g., Bereiter, 1995; Brown, 1989). 2) Connecting settings also encourages transfer during potential transfer contexts, when students view prior content as continuing to be relevant and so are reminded of it (e.g., Leander, 2001; Pea, 1987; Ross, 1984). 3) By connecting to prior settings and by positioning students as authors, students are likely to view their own prior knowledge as relevant to current learning, encouraging them to “transfer-in” more of their prior knowledge during learning, as they construct new understandings, which can enhance the quality of learning and therefore transfer (e.g., Bransford, Brown & Cocking, 1999; Schwartz, Bransford & Sears, 2005).

© ISLS

41

ICLS2012

Volume 1: Full Papers

4) Framing students as authors of particular content may foster student accountability to that content, making students more likely to use this content in transfer contexts (e.g., Greeno, 2006; Jacoby & Gonzales, 1991). 5) If authorship becomes a general practice that students regularly participate in, it may promote the practices of generating new knowledge and engaging in adaptive problem solving (e.g., Boaler, 2002; Greeno, 2006; Hatano & Oura, 2003; Schwartz et al., 2005). In this paper, we investigate which of these five potential explanations are helpful in understanding why and how one particular student had an unexpectedly high rate of transfer.

A Somewhat Surprising Case of Transfer
Elaine was a Hispanic 9th grader who qualified for the free and reduced lunch program, and whose parents had a high school level education. Her 8th grade standardized test score in science was not particularly strong (“basic,” as opposed to “proficient” or “advanced”). For the first month of school she often did not attend class consistently nor do her biology homework, earning a D the first semester. She was also a student who had a long history of getting into trouble in school. She was identified by the high school as a student at risk and also reported “everybody assumes that I have straight F‟s”. Thus, Elaine did not fit the typical profile of a student that would be expected to demonstrate strong learning, let alone transfer. We assessed Elaine and her classmates‟ transfer of the concept of concentration gradient, as it was central to the course. In particular, we had students explain scenarios about osmosis and diffusion (Odom & Barrow, 1995) that were designed to be more or less analogous to what students had learned in class (Gick & Holyoak, 1983). Scenarios that were analogous were included in a knowledge assessment given after the class unit on osmosis and diffusion a month into the second semester. Scenarios that were less analogous were included in the transfer assessment administered 3 months later. Elaine scored surprisingly high on the transfer test in comparison to the rest of the class. She transferred 78% (z=+1.22) of the knowledge elements she had known on the knowledge assessment while the class as a whole transferred a mean of 57% of what they knew. In addition, her score on the transfer test was 67% higher than her score on the knowledge assessment, suggesting she used more knowledge on the more challenging transfer questions than on the more straightforward questions in the knowledge assessment.

Data Sources for Explaining Elaine’s Transfer
1. Student framing survey: Conducted midway through the school year, it used open-ended and Likert-style questions to measure the degree to which students were aware of their teacher‟s expansive framing of their biology class and the degree to which they were responsive to his expansive framing in their own actions. 2. Video and audio recordings: One video camera captured a view of the students and the other, a view of the instructor. The audio recorders captured conversations among pairs and groups of students during class activities. 3. Instructional materials and graded student work: These were collected to supplement the video and audio recordings, as well as to track individual students‟ performances. 4. Student interviews: Conducted at the end of the school year, Elaine and other students were asked to update and elaborate on their responses to the framing survey and share their perspectives on two recorded excerpts from class in which the teacher explained aspects of his expansive framing of settings and roles. 5. Teacher interviews: We draw on excerpts in which he discussed Elaine.

Explaining How Expansive Framing May Have Led to Elaine’s Transfer
We first show that Elaine‟s higher-than-expected transfer was paralleled by high awareness and responsiveness to the teacher‟s expansive framing. We then use her case to investigate evidence relevant to each of the five proposed hypotheses for how framing can foster transfer.

Elaine was especially aware of and responsive to the teacher’s expansive framing
Survey results indicated that Elaine perceived and responded to the teacher‟s expansive framing to a greater extent than her classmates (Figure 2). On average, students in this biology class scored 3.66 when asked questions aimed at the degree to which they were aware of their teacher‟s expansive framing, and 3.33 when asked about how they responded to this framing (on a five point scale from 1 [“never”] to 5 [“all the time”]). However, Elaine was aware of the teacher‟s expansive framing with an average score of 4.42 and responded to the framing with an average score of 3.99, both about one standard deviation unit above the class means. Thus, Elaine appears to have been more aware of and responsive to the teacher‟s expansive framing than the average student in her biology class. Specifically, many of Elaine‟s responses to survey questions about her awareness of particular aspects of framing were more than a full standard deviation above the average score of her peers: the teacher‟s framing of place (4.17, z=+1.07), participants (4.40, z=+1.56), and roles (5.00, z=+1.32). She also indicated particularly higher responsiveness to expansive framing related to time (4.88, z=+1.50) and place

© ISLS

42

ICLS2012

Volume 1: Full Papers

(4.33, z=+1.12). According to the prior studies cited above, these results predict that Elaine would have a higher rate transfer of than her classmates.

Figure 2. Student awareness of and responsiveness to the teacher's expansive framing: the class vs. Elaine

Explaining Elaine’s case by examining evidence relevant to each of the five hypotheses 1. The student expects future transfer, so prepares for it by studying more or better
There is only partial evidence that this explanation helps explain Elaine‟s enhanced transfer as we have strong evidence that Elaine expected she would need to transfer what she had learned in the future, but weak evidence that she prepared for transfer by studying more or better. When asked in the survey about the frequency with which the teacher told the students what they are learning may be useful to them at various times in the future (the next day, the next few weeks, the next year, and beyond the next year), Elaine frequently reported that this happened “often” or “all the time” (4.50, z = +0.97). Furthermore, she reported thinking “all the time” about how to apply what she learned at all of these times in the future (5.00, z = +1.17). On the other hand, evidence that Elaine strongly prepared for future transfer is weak. When asked about her studying habits she stated “I‟m lucky if I look over my notes like, once, like I never study for anything…„cause I‟m lazy or I‟m doing something else.” However, evidence from the teacher and grade reports shows that Elaine completed more biology homework as the year progressed, gradually improving her grade. When asked specifically about turning in assignments, Elaine said, “I kinda caught up now, „cause um how I said [the teacher] kinda pushed me to do better…I‟m doing better in his class, in specific, so everything‟s kinda working out.” Moreover, she also appeared to recognize the value in studying and doing homework, even though she did not regularly do these things herself. For example, when asked if she thought that completing the pre-lab assignments were helpful she responded “like me, if you kinda wait „til the last minute or try to copy it… it doesn‟t help you at all,” suggesting that she was aware of the fact that completing the assignments would have helped her had she prepared better. In summary, while Elaine may not have exhibited strong preparation for future transfer to the extent that we would have expected her to be, considering her successful rate of transfer, her class performance improved over the course of the school year and she appeared to be aware of the value of preparation for class.

2. Viewing what was learned before as having continued relevance in transfer situations
There is strong evidence that this explanation is significant to Elaine‟s case as we found she viewed ideas from the past as continuing to be relevant. Based on survey and interview data, we found that Elaine noticed the teacher regularly referred to what the students had learned at various prior times and recognized that topics in class built upon each other: It‟s not always something brand new… we always elaborate on what we know. So … like the only time it‟s really brand new is in the beginning of the year … [Otherwise,] we always refer back to certain stuff….Like um, with mitosis and meiosis, we had to know that for evolution …we always just kinda adds -- add something new, but it builds on what we already know. In this example, Elaine showed that she was aware that the content in class built upon prior lessons and that what she previously learned had continued relevance. Elaine also reported that she frequently drew on ideas from previous times in class. For example, Elaine once asked: “I have a question....It just relates to sorta what we were doing last week, but you know how...we said that...an animal will blend in with like its environment? So, um, does that kinda happen with

© ISLS

43

ICLS2012

Volume 1: Full Papers

plants and stuff too?” Elaine drew on what she had learned before about animals to pose a new transfer question to the class about whether the same idea extends to plants, and then began engaging the class in answering her question.

3. Prior knowledge as author is relevant, so transfer-in more prior knowledge, including examples or generalizations that can enhance learning and therefore later transfer.
We also have evidence that this explanation is relevant to explaining Elaine‟s transfer. Elaine reported that she appreciated participating in class, which in this particular case meant drawing on her prior knowledge to author her own ideas. As evidence, we saw Elaine bring in knowledge from outside of school to help her learn biology, which is necessary for future transfer. For example, when the teacher was introducing different cell parts and displayed a picture of a centriole (a long, tube-shaped structure), Elaine exclaimed “it looks like a churro!,” a similarly long and tube-shaped Hispanic pastry. By comparing the shape of centrioles to churros, Elaine brought in her cultural knowledge in ways that likely enhanced her memory of the structure of centrioles, allowing her to draw on it later. In one lesson, Elaine was also able to transfer-in her cultural knowledge to interpret the species name of a cougar, concolor: “In Spanish, it‟s „with color,‟ except that there is a space.” Potentially, Elaine may generalize the idea that her cultural knowledge is applicable to learning contexts to help her remember concepts and interpret Latin words. We also have evidence of Elaine transferring-in prior knowledge that is not based on culture. During a lesson that began with the topic of animal traits, students became engaged in a discussion about plant characteristics that may deter predators from injuring them. Elaine contributed to the discussion by saying, “so a cactus has thorns to stop animals from eating them,” bringing in a specific example of cacti characteristics to help understand the general idea that plants have traits that help prevent them from being eaten by predators. This act of transferring-in additional examples and comparing them to each other can also directly enhance transfer (e.g., Gentner, Loewenstein & Thompson, 2003; Gick & Holyoak, 1983).

4. Students become publicly recognized as the author of particular content that they then become accountable for sharing
Elaine‟s ability to transfer was likely affected by being regularly positioned as an author of ideas and questions during discussions. As a result, she was regularly held accountable for the biology content that she was authoring in class. For example, during a particular class discussion about chromosomes, Elaine shared with the class that she did not think human eggs and sperm have 46 chromosomes. The teacher followed her contribution to the discussion by asking the class “what do you think about what Elaine said?” In doing so, he publicly credited Elaine for her ideas and made them a part of the instruction. Later in the discussion, the teacher even asks Elaine to repeat what she had shared, re-emphasizing her ideas. Our survey results indicated that Elaine noticed that the teacher regularly credited her “for coming up with important ideas in class” and that it came with an expectation that she would continue to “comment intelligently on what we are learning in class...all the time.” Our data also reveal evidence that Elaine was recognized as an author and accountable for sharing her knowledge outside of class. During her interview, when she was asked for details about her survey response where she indicated that she talked about biology to those around her at home, Elaine elaborated: Just last night, my step-dad said that he was sick or something, and that -- something about amino acids and then I asked him… „Well, you know what that is?‟ Like, „how do you even know what that is?‟ And um, he started laughing and he -- he told me, um, well how do you not know what that is... And he says, um, do YOU know what it is? And I was like yeah, we just learned that this year. And I kinda explained it him, like what I -- MY understanding of what they were and all that, and he just kinda stayed quiet and was like, „mm hmm‟ and shook his head like, „yeah.‟ In this example, Elaine was authoring her own ideas about amino acids, emphasizing “MY understanding of what they were” while her step-father held her accountable for sharing her understanding. This accountability appears and is likely to increase opportunities and the social expectation for students to continue contributing to conversations about the ideas that they have and the ideas that they have previously authored.

5. Authorship becomes a practice so students regularly adapt what has been learned to earnestly attempt or propose novel problems.
There is strong evidence that Elaine‟s practice of being an author may have contributed to her high transfer results. First, it is clear that over time Elaine strongly adopted authorship as a practice. By the end of the year, she said “I participate as much as I can.” With respect to being generative, Elaine always made a genuine attempt to answer all of the questions on the transfer assessment while other classmates occasionally left questions blank or responded with “I don‟t know.”

© ISLS

44

ICLS2012

Volume 1: Full Papers

Elaine‟s role as an author is best summarized by an excerpt from the teacher during a class discussion several months into the school year when Elaine proposed an incorrect answer to a question: Teacher: I think without question I can say the reason Elaine is doing so much better in here is because she is willing to say „No, I think it‟s … A because it‟s what a gamete is‟ and that takes some guts because she‟s not sure but it‟s what she thinks … and that‟s really good. Is that the right thing to say though? I mean, is that a correct statement? Students [in unison]: No Teacher: No, it‟s not a correct statement, ok, but she‟s not over there crying because she was wrong, but what it did then was give ... everybody else who thought it and get a chance to correct her which reinforces the right answers for them, allows Elaine to rethink through her response and change her mind. She‟s more likely to remember that now and everybody, now everybody benefits … because Elaine had, you know, was able to say something. From this transcript, it is clear that the teacher recognized that Elaine earnestly attempted to answer questions and shared with the class even when she was uncertain of her answers. In our survey, Elaine also stated that she shared what she thought in class, even though she was not sure if she was right, “all of the time” (5, z= 1.5). She further reported being “able to say what I feel” and always shared her ideas in class even if she did not know if her answer was correct. These survey data suggest that Elaine attempted problems presented in class and was always willing to share her ideas. In addition to earnestly attempting novel problems in class, video data captured many instances of Elaine proposing her own novel problems by adapting her prior knowledge to the current learning context. For example, during a lesson when students discussed a visualization of equilibration, Elaine posed the question, “Does it stay at equilibrium after it reaches equilibrium?” In presenting this question to the class, Elaine extended the discussion beyond initial attempts to understand the definition of equilibrium, to the process of how equilibrium is reached and what occurs after that point. The teacher subsequently supported Elaine‟s proposal and encouraged students to propose similar questions by responding to Elaine‟s question saying “That, ladies and gentlemen, is a ten thousand dollar question. Smart kids don‟t give the best answers, they ask the best questions. So in groups, after equilibrium is reached, does it stay that way?” The teacher credited Elaine as the author of a “smart” question, and used her question as a point for further discussion and instruction. Occasions such as this, where Elaine proposed novel problems built from her understanding of the content, occurred often throughout the school year. We believe that this practice of being an author, and attempting or proposing novel problems in class, ultimately enhanced her ability to transfer.

Other hypotheses for how Elaine’s transfer may have been supported
In addition to these 5 hypotheses, we also considered how data from Elaine‟s case might suggest other hypotheses for how her transfer was supported that may be less related to the effects of the teacher‟s expansive framing. The first hypothesis suggested by Elaine‟s data was that she learned more and more deeply in the class because of the quality of the teaching she received, which then allowed her to transfer more of what she learned (Bransford, Brown & Cocking, 1999). Elaine specifically referred to the quality of the teaching in her biology class in an open-ended survey question that asked her to describe her biology class and teacher to a new student: “[He] always finds fun ways for us to learn all the material….. Everything he teaches us is a pretty hard subject and yet this is the class I‟ve learned the most in…. It‟s complicated, but [the teacher] does a great job of explaining and breaking things down until we understand.” Consistent with Elaine‟s comments, we can report that classes taught by this teacher, including this section, generally received higher scores both on standardized tests and in-class conceptual tests as compared to the other biology teachers in this same school. However, while it is clear that the teacher‟s success at helping Elaine learn the material was a necessary prerequisite for her transfer, this does not help explain why we observed more transfer with her than other students, as other students also presumably benefited from the high quality of in-class instruction. So by itself, this hypothesis is not sufficient for explaining Elaine‟s higher than usual rate of transfer. We then investigated whether there were any aspects of support that Elaine received that were different other students and found that her engagement in class was supported more strongly, and eventually led to greater engagement when it was originally much weaker than most students. In particular, Elaine had been identified as a potential at-risk student, leading to her biology teacher‟s decision to visit her home and better understand her out-of-school constraints and resources. This led to him regularly taking her aside in school to encourage and push her, which Elaine said helped her do better in class as we illustrated in discussing Elaine‟s study practices around Hypothesis 1. When we asked exactly how her biology teacher helped her improve, Elaine explained:

© ISLS

45

ICLS2012

Volume 1: Full Papers

“[By] him always reminding me. Having somebody tell me that I have to do this, that I have to do it for myself. Reminding me of the consequences, um, just telling me, like, that I‟m capable of doing better. [So I] get home and don‟t forget…that I actually have homework to do.” So the fact that Elaine began doing more schoolwork and therefore learning more that she could then transfer, we attribute to the teacher‟s extra efforts to support her engagement with school. However, while this hypothesis would explain why Elaine had learned more material to transfer, it does not fully explain why she transferred a higher proportion of material than other students. Our final hypothesis centers on the often tricky and subtle issue of student motivation. From Elaine‟s interview we found out that she was particularly interested in taking this class as she‟d heard from older students about its culminating forensics unit, a career she was seriously considering after a school project in the prior year. In addition, she expressed a particular interest in animals and noted that this also made her more interested in biology. Finally, she reported particularly enjoying this biology class because of the teacher‟s sense of humor: “I really enjoy being in [the teacher‟s] class….[He] always finds fun ways for us to learn all the material… He‟s very funny too. I come in this class knowing that by the end of the one-hour period, I‟ll have a huge grin on my face.” Both Elaine‟s prior individual and current situational interests (Hidi & Renninger, 2006) as well as the future utility value she perceived with the content for her future (Hulleman & Harackiewicz, 2009) could be expected to support Elaine‟s engagement with the class as well as the kinds of re-engagement with the content at later times (Renninger, 2000) that become transfer. However, at first glance this hypothesis of Elaine‟s strong motivation helping to foster transfer seems inconsistent with the fact that Elaine needed to be reminded to do homework and reported never studying for tests. In the end, though, we do not think this is the case as we learned through the teacher‟s home visit that (a) doing schoolwork at home often conflicted with Elaine‟s family responsibilities, and (b) completing out-of-class assignments takes more time and effort than the kinds of transfer we observed Elaine evidencing that involved using more of her knowledge on required in-class transfer assessments, making comments in class, or bringing up what she had learned about biology with her friends or family when relevant to ongoing activity. So we believe that it may have been Elaine‟s individual and situationally supported interests that interacted positively with the teacher‟s expansive framing and other support that fostered her transfer.

Conclusion
Based on our analyses of Elaine‟s case of transfer, we found relatively strong support for four of the five proposed explanations for how expansive framing fostered transfer for a student that was not expected to demonstrate strong transfer. We have also developed additional hypotheses that may further contribute to her high rate of transfer such as the quality of instruction she received, teacher support, and personal motivation. In our presentation, we will further describe how the processes suggested by each explanation unfolded throughout the school year, drawing on data to describe how Elaine‟s student activities changed over time. Moreover, we believe that the explanations are not mutually exclusive. Many of the examples we have provided under each of the five explanations are closely related to and even demonstrate one or more of the other four explanations. Therefore, we aim to further develop our theories about how these five explanations are related.

References
Bateson, G. (1972). A theory of play and fantasy. In Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology (pp. 177–193). New York: Ballantine. Bereiter, C. (1995). A dispositional view of transfer. In A. McKeough, J. Lupart, & A. Marini (Eds.), Teaching for transfer: Fostering generalization in learning (pp. 21–34). Mahwah, NJ: Erlbaum. Boaler, J. (2002). Experiencing school mathematics: Traditional and reform approaches to teaching and their effect on student learning. Mahwah, NJ: Erlbaum. Bransford, J., Brown, A. L., & Cocking, R. (Eds.). (1999). Learning and transfer. In How people learn: Brain, mind, experience, and school (pp. 39–66). Washington, DC: National Academy Press. Brown, A. L. (1989). Analogical learning and transfer: What develops? In S. Vosniadou & A. Ortony (Eds.), Similarity and analogical reasoning (pp. 369-412). Cambridge, UK: Cambridge U. Press. Engle, R. A. (2006). Framing interactions to foster generative learning: A situative account of transfer in a community of learners classroom. J. of the Learning Sciences, 15(4), 451-498. Engle, R. A., Lam, D. P., Meyer, X. S., & Nix, S. (in press). How does expansive framing promote transfer? Several proposed explanations. Educational Psychologist. Engle, R. A., Meyer, X., Clark, J., White, J. & Mendelson, A. (2010). Expansive framing and transfer in a high school biology class: Hybridizing settings and promoting connections within a larger learning community. In R. A. Engle (organizer), Applying new mechanisms and conceptualizations of the

© ISLS

46

ICLS2012

Volume 1: Full Papers

„transfer-of-learning‟ to science classrooms: The dynamic role of contexts and interactions. Symposium presented at the National Association for Research in Science Teaching Annual International Conference, Philadelphia, PA. Engle, R. A., Nguyen, P. D. & Mendelson, A. (2011). The influence of framing on transfer: Initial evidence from a tutoring experiment. Instructional Science, 39(5), 603-628. Gentner, D., Loewenstein, J. & Thompson, L. (2003). Learning and transfer: A general role for analogical encoding. J. of Educational Psychology, 95(2), 393-408. Gick, M. L. & Holyoak, K. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15, 1–38. Goffman, E. (1974). Frame analysis: An essay on the organization of experience. New York: Harper & Row. Greeno, J. G. (2006). Authoritative, accountable positioning and connected, general knowing: Progressive themes in understanding transfer. J. of the Learning Sciences, 15(4), 537-547. Greeno, J. G., Smith, D. R., & Moore, J. L. (1993). Transfer of situated learning. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 99–127). Norwood, NJ: Ablex. Hammer, D., Elby, A., Scherr, R. E., & Redish, E. F. (2005). Resources, framing, and transfer. In J. Mestre (Ed.), Transfer of learning: Research and perspectives. Greenwich, CT: Information Age Publishing. Hart, W. & Albarracin, D. (2009). What was doing vs. what I did: Verb aspect influences memory and future actions. Psychological Science, 20(2), 238-244. Hatano, G. & Oura, Y. (2003). Commentary: Reconceptualizing school learning using insight from expertise research. Educational Researcher, 32(8), 26-29. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Ed. Psychologist, 41, 111– 127. Hulleman, C. S. & Harackiewicz, J. M. (2009). Promoting interest and performance in high school science classes. Science, 326, 1410-1412. Jacoby, S. & Gonzales, P. (1991). The construction of expert–novice in scientific discourse. Issues in Applied Linguistics, 2(2), 149–181. Laboratory of Comparative Human Cognition. (1983). Culture and cognitive development. In P. H. Mussen (Ed.), Handbook of child psychology: Vol. 1. History, theory and methods (pp. 295–356). New York: Wiley. Leander, K.M. (2001). “This is our freedom bus going home right now”: Producing and hybridizing space-time contexts in pedagogical discourse. J. of Literacy Research, 33(4), 637-679. Maxwell, J. A. (2004). Using qualitative methods for causal explanation. Field Methods, 16(3), 243-264. Mendelson, A. (2010). Using online forums to scaffold oral participation in foreign language instruction. L2 Journal, 2(1), 23-44. Meyer, X., Mendelson, A., Engle, R.A. & Clark, J (2011). Expansive framing and transfer in a high school biology class: Promoting connections within and beyond the classroom. Manuscript submitted for publication. Pea, R. D. (1987). Socializing the knowledge transfer problem. Int‟l. J. of Educational Research, 11, 639–663. Renninger, K. A. (2000). Individual interest and its implications for understanding intrinsic motivation. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 373-404). San Diego, CA: Academic Press. Ross, B. H. (1984). Reminding and their effects in learning a cognitive skill. Cognitive Psychology, 16(3), 371– 416. Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer. In J. Mestre (Ed.), Transfer of learning: Research and perspectives. Greenwich, CT: Information Age Publishing. Siegler, R. S. (2006). Microgenetic analyses of learning. In W. Damon & R.M. Lerner (Series Eds.) & D. Kuhn & R.S. Siegler (Vol. Eds.), Handbook of child psychology: Volume 2: Cognition, perception, and language (6th ed., pp. 464-510). Hoboken, NJ: Wiley. Tannen, D. (1993). Introduction. In D. Tannen (Ed.), Framing in discourse (pp.3–13). New York: Oxford U. Press.

© ISLS

47

ICLS2012

Volume 1: Full Papers

How metacognitive awareness caused a domino effect in learning
Kristin Schmidt, University of Freiburg, kristin.schmidt@ezw.uni-freiburg.de Andreas Lachner, University of Göttingen, andreas.lachner@med.uni-goettingen.de Björn Stucke, University of Göttingen, bjoern.stucke@med.uni-goettingen.de Sabine Rey, University of Göttingen, srey@med.uni-goettingen.de Cornelius Frömmel, University of Göttingen, cfroemmel@med.uni-goettingen.de Matthias Nückles, University of Freiburg, matthias.nueckles@ezw.uni-freiburg.de Abstract: The scope of this study was to investigate effects of a metacognitive awareness tool on self-regulated learning. In a series of web-based self-monitoring protocols, participants (N=49) recorded their class preparation. In the experimental condition, they received individual scores of dilatory behavior mirrored in a line chart. We were interested in whether potential advantages of the line chart can be traced back to non-specific signaling of dilatory behavior or to specific individual feedback. Therefore, we compared the experimental group with two control groups: without visualization or with random visualization. The random visualization was expected to have a signaling but no feedback effect. Results supported the effects of signaling and individual feedback. Participants with random or veridical visualizations were more likely to engage in self-reflection, but only those who received individual feedback were more likely to reduce their dilatory behavior and improved selfregulated learning.

Introduction
Although there is an increasing demand for self-regulation strategies in learning, many students have problems in regulating themselves (Zimmerman, 2002). Consider the case of Tony, a university student of medicine, who is in the last year of his clinical studies. An important exam on neurology is three weeks away, and Tony has not started studying yet. To calm himself, he tells himself that enough time is left and studying too long before the exam would cause him to forget important facts. In this way, he keeps on postponing his tasks, although he experiences anxiety due to the growing pressure. Two days before the exam, Tony starts repeating the facts without understanding anything because there is no time for deeper understanding. Anyway, he argues, he learns best under pressure. However, he does not set any learning goals except of “studying”, and does not reflect his preparation. In some cases, students like Tony truly receive satisfying grades for their exam performance, but in the majority of cases, these students suffer from dilatory behavior and often fail to achieve their goals. Taken together, Tony shows deficient self-regulation. One possibility to improve self-regulation is to guide students’ metacognitive awareness to processes that need to be improved (Schraw, 1998), for example by using time management techniques. Metacognitive awareness is expected to facilitate self-reactivity, which is an improvement of behavior without further treatments (Zimmerman, 2002). In the present article, we want to discuss the role of metacognitive awareness in self-regulated learning and present an empirical study on the effectiveness of a metacognitive awareness tool.

Dilatory Behavior – Failure of Self-Regulation
A ubiquitous problem of self-regulation is students’ irrational postponing of important tasks, which is known as dilatory behavior (Lay & Silverman, 1996; Steel, 2007). It often results from ineffective metacognitive strategy use when high standards of self-regulation are required (Wolters, 2003). Many learners delay the beginning of unpleasant but important learning tasks to an unspecified later date. It seems to be difficult for them to motivate themselves and shield motivation against interruptions. Thus, they seem to lack volitional control (Boekaerts, 1996). Several studies showed that dilatory behavior is negatively related to self-regulated learning. It interferes with goal-setting (Lay & Schouwenburg, 1993), self-efficacy (Wolters, 2003; Ferrari, Parker, & Ware, 1992), strategy use (Wolters, 2003), and learning outcomes (Lay & Schouwenburg, 1993; Klassen, Krawchuk, & Rajani, 2008). Dilatory behavior is influenced by a personal tendency to delay tasks (procrastination as a personal trait), task characteristics and a learner’s repertoire of self-regulation strategies (Steel, 2007). Thus, the tendency to procrastinate needs not necessarily result in dilatory behavior, because strategies for self-regulation are expected to mediate personal and contextual characteristics and behavior (Pintrich, 2004). The correlation between trait procrastination and dilatory behavior (Stainton, Lay, & Flett, 2000: r = .51) indicates that trait procrastination and dilatory behavior are related but not the same. Thus, trait procrastination could be compensated by strategies for self-regulation.

Metacognitive Strategies

© ISLS

48

ICLS2012

Volume 1: Full Papers

To self-regulate learning processes successfully and to avoid problems in motivation and understanding, sophisticated metacognitive strategies are needed (Schraw, 1998; Zimmerman, 2000). These strategies encompass planning, monitoring and self-reflection of the learning process (Schraw, 1998; Zimmerman, 2002). To explain the relations between metacognitive, motivational, and behavioral aspects of self-regulated learning in a more systematic manner, different researchers draw on prescriptive models of self-regulated learning (Pintrich, 2004; Winne & Hadwin, 1998; Zimmerman, 2000). According to Zimmerman (2002), self-regulated learning involves the selective and adaptive use of specific metacognitive strategies to monitor and regulate cognitive processes. These cognitive processes ensure understanding and recall of knowledge as well as high degrees of motivation. Some of these metacognitive strategies are goal setting, self-motivation, strategy selection and adaptation, comprehension monitoring, and self-reflection of products and efficiency (Schraw, 1998; Zimmerman, 2002). Each of these prescriptive models of self-regulated learning includes a planning phase, an implementation phase, and a reflection phase, each depending on each other. Planning refers to the selection of personal standards that should be achieved during the learning episode and the selection of appropriate cognitive strategies in relation to the standards. During the implementation of cognitive strategies, self-monitoring of performance and understanding is essential (Zimmerman, 2000). Therefore, learners’ deliberate observation of the learning process is necessary in order to recognize discrepancies between the observed learning process and the personal ideal of the learning process. Afterwards, the learners reflect upon these discrepancies (Winne & Hadwin, 1998). Self-reflection includes an attribution of success and failure to internal and external responsibilities (Zimmerman, 2000). In doing so, learners can deduce consequences for later learning episodes to avoid making the same mistakes. Thus, via metacognitive regulation, learners have the possibility to improve their learning self-reactively (Zimmerman, 2002).

Volitional Control
Strongly related to metacognitive regulation is the regulation of motivation. This regulation is also called volitional control (Boekaerts, 1996; Corno, 1986). Volitional control is expected to prevent postponements and to shield studying against interruptions. The relation between metacognitive strategies and volitional control during the learning process can be illustrated for goal-setting. Goal setting is a metacognitive strategy that is built around a reference to decide whether the learning behavior is appropriate (Zimmerman, Bandura, & Martinez-Pons, 1992). There is considerable evidence for increased academic success related to specific learning goals (Locke & Latham, 2002; Schunk, 2001). Specific, proximal goals facilitate the selection of appropriate cognitive strategies, self-monitoring of performance and strategy regulation, and the assessment of goal achievement. Additionally, goal setting is important for motivation. Personally relevant learning goals contribute to high motivation to achieve the task. Specific, challenging but obtainable learning goals support self-efficacy, which is a central source for motivation (Locke & Latham, 2002; Schunk, 2001). Vice versa, a higher motivation to deal with a particular subject could lead to more challenging learning goals, and in this way to more learning effort, deep-learning strategies (Sankaran & Bui, 2001) and better learning outcomes (Zimmerman et al., 1992; Boekaerts, 1996).

Reducing Dilatory Behavior by Supporting Self-Monitoring
Self-monitoring provides learners with data about their learning process (e.g., “What goals are appropriate?”), and the learner themselves (e.g., “What motivates me?”) (Zimmerman, 2002). Hence, self-monitoring establishes the base for increased metacognitive knowledge that may help to improve future studying (Zohar & Peled, 2008). Via self-monitoring, students become aware of ineffective or inappropriate behavior and try to better fit their personal standards by changing their behavior. Several studies impressively showed that improved self-monitoring can improve various learning tasks, like reading (Joseph & Eveleigh, 2011), writing (Cho, Cho, & Hacker, 2010), and academic performance (Chang, 2007). Additionally, self-monitoring can improve motivational beliefs (Chang, 2007). Thus, accurate, deliberate and critical self-monitoring is essential for self-reactive improvements of learning behavior (Boekaerts, 1996). Self-monitoring of dilatory behavior could be improved by guiding students’ awareness to dilatory behavior and by supporting the accuracy of self-monitoring. This self-monitoring can be improved by asking learners to record their learning behavior in self-monitoring protocols (Lan, 1996; Schmitz & Wiese, 2006). The self-monitoring protocols require students to monitor their learning behavior in order to answer questions about it. These questions can guide a learner’s attention to aspects that should be improved. Monitoring dilatory behavior provides students with data that conflict with the normative ideal of studying that students are expected to have. The discrepancies between observed learning behavior and personal standards could lead to a selfreactive improvement of studying. However, self-regulated learning is a very complex activity that involves numerous processes. Therefore, students have the possibility to choose data concordant with their self-image and monitor and reflect primarily on positive aspects of their studying (Lewicki, 1983). This allows for a protection of their self-image, but inhibits improvements in learning behavior. To avoid biased self-monitoring,

© ISLS

49

ICLS2012

Volume 1: Full Papers

a tool would be helpful that guides students’ awareness to rather problematic aspects of learning, for instance to dilatory behavior. To support students’ deliberate self- monitoring and metacognitive awareness of their dilatory behavior, we investigated the effectiveness of a graphical visualization that provided learners with data about their own dilatory behavior. Awareness tools are often used to facilitate computer-supported collaborative learning (CSCL). They sensitize students to the collaborative setting. Furthermore, specific information about the collaborative partner’s activities helps students to calibrate their own learning behavior in accordance with their collaboration partner’s behavior (Dehler, Bodemer, Buder, & Hesse, 2011).

Research questions of the current study
However, we were interested in whether web-based metacognitive awareness tools can also be used to improve one’s own learning processes by decreasing dilatory behavior (1) and increasing self-reflection (2). Additionally, we were interested in whether such improvements are related to motivation and cognitive strategy use (3) and whether this affects learning outcomes (4). More precisely, we investigated the effects of systematic feedback by an individual visualization of dilatory behavior within self-monitoring protocols. Every week, students recorded their class preparations in a web-based self-monitoring protocol. To improve the effects of self-monitoring protocols, we included a graphical visualization of dilatory behavior. We expected the visualization to boost deliberate self-monitoring for two reasons. The signaling effect (or eye catching effect) of the traffic light colors could sensitize participants for dilatory behavior and thus cause increased self-reflection. Additionally, individual feedback in a veridical line chart could help students to estimate their behavior more accurately and thus facilitate selfreactivity, which helps to decrease the degree of dilatory behavior. To investigate whether the effect of the veridical visualization could be traced back to general awareness or to individual feedback, we compared the experimental condition with a no-visualization and a random condition. This random group received the same type of visualization with traffic light colors but with a random line chart. In this way, we could separate out the effects of the signaling effect of the traffic light colors, irrespective of the veridical values in the line chart from individual feedback.

Method
Participants, Design and Procedure
49 medical students (freshmen) with an average age of 20.9 years (SD = 2.17) participated in the study. The proportion of females was 67%. During an anatomy course (12 weeks, grades were not relevant for the preliminary medical examination), participants were asked to record their class preparation and homework in a web-based self-monitoring protocol once a week. The participants were randomly assigned to one of three experimental conditions: a veridical condition, a random condition, or a no-visualization condition. Students in the veridical condition received their average score of dilatory behavior ratings mirrored back in an individual line chart. The random group received the same visualization but with random values. The no-visualization group received no chart. During the self-monitoring period, students wrote 12 protocols at most. Before and after the self-monitoring period, students took part in a knowledge test that included declarative and conceptual knowledge tasks. In the study, dependent variables encompassed learning goals, motivation, dilatory behavior, strategy use, and learning outcomes.

Self-monitoring protocols
The self-monitoring protocols were realized in a web-based learning management system. Thus, to make their entries, students only needed Internet access. The self-monitoring protocols consisted of different sections related to models of self-regulated learning. Except for the learning goals, participants had to estimate their agreement with the items on a 5-point Likert scale ranging from low to high agreement. First, participants assessed whether they achieved their learning goals in the last week. Furthermore, they estimated the degree of their dilatory behavior based on four items (e.g. “I often delayed important tasks to a later date.”), which were adapted and translated from a dilatory behavior questionnaire (Lay & Silverman, 1996). Second, participants estimated their cognitive and metacognitive strategy use. The items were slightly adapted and translated from the MSLQ (Pintrich, Smith, Garcia, & McKeachie, 1993). As cognitive strategies, we collected self-reported use of elaboration strategies (e.g., “I thought about examples that illustrate abstract learning content.”), organization strategies (e.g., “I tried to structure the learning material.”), and rehearsal strategies (e.g., “I repeated the learning materials more than once.”). The items of the metacognitive strategy scale encompassed planning (e.g., “Before studying, I thought about how to organize the materials.”), monitoring (e.g., “While studying, I monitored whether my learning strategy use is efficient to achieve my goals.”), and self-reflection (e.g., “I thought about possibilities to optimize my studying.”).

© ISLS

50

ICLS2012

Volume 1: Full Papers

Third, participants defined up to five learning goals for the following week and estimated their motivation to achieve these goals (e.g., “The topic of learning content is relevant for me.”). Learning goals were categorized in terms of their specificity level either as non-specific or specific. Inter-rater reliability was good (κ =.78.). Non-specific goals neither included concrete topics nor strategic planning (e.g., “studying anatomy”). Accordingly, students might have problems in deciding whether they had reached these goals. It was presumably easier for students to assess whether they had actually reached specific goals. Specific goals included a concrete topic and strategic planning (e.g., “repeating the organs of the abdomen”). Fourth, to compare knowledge gains, a knowledge test was administered as part of the first and the last self-monitoring protocols. It contained several declarative knowledge and conceptual knowledge tasks. In declarative knowledge tasks, students were asked to mention facts about a particular topic (e.g., “Name the muscles of the human face.”). In conceptual knowledge tasks, students were asked to explain complex medical processes (e.g., “Explain the follow-ups of a pancreas excision for patients.”).

The graphical visualization
Participants in the veridical condition received their averaged responses of dilatory behavior items mirrored back in an individual line chart that expanded on a weekly basis. Thus, when students filled out the protocol they were confronted with their scores of dilatory behavior since the beginning of the study. The visualizations included the course of dilatory behavior since the beginning of the study, and in accordance with traffic lights, a classification into low (green), medium (yellow), and critical (red) values of dilatory behavior (see Figure 1). Participants in the random condition received computer-generated random values that could take values between 1 and 5. The participants in that condition were debriefed after the end of the study.
You tend to postpone your tasks.

Your time management is fine.

Figure 1. Example of a visualization of dilatory behavior

Results
To obtain mean scores, we averaged for each student across her/his self-monitoring protocols of the first and second week, the third and fourth week, the fifth and sixth week, and so on. By using average scores, we improved the reliability of a participant’s individual scores and we avoided an unnecessary loss of data. As we analyzed the protocol data with repeated measures analyses of variance, students who had one or more missing journal entries otherwise would have been completely excluded as cases.

Reduction of Dilatory Behavior
To investigate the course of dilatory behavior, we conducted an analysis of variance with repeated measurement. The individual mean scores of dilatory behavior items were treated as within subject factor, and experimental condition (veridical, random, no-visualization) as between subject factors. The main effect of measurement was significant, F(1, 30) = 5.12, p < .05, partial η² = .15. Hence, dilatory behavior changed during the surveyed time span. The main effect of experimental condition failed to reach statistical significance F(2, 30) = 0.40, n.s.. Nevertheless, the interaction effect between measurements and experimental condition was significant, F(2, 30) = 3.62, p < .05, partial η² = .19, indicating that the courses of dilatory behavior differed between the experimental conditions. The courses of dilatory behavior indicated that students in the veridical condition learned more consistently (see Figure 2, left). To investigate this assumption, we computed the intra-individual variance of each participant’s dilatory behavior that were subjected as dependent variable to a one-factorial analysis of variance with experimental condition (veridical, random, no-visualization) as independent variable. The results indicated that the variance of dilatory behavior could be reduced in the conditions with graphical support, especially in the veridical condition, F(2, 43) = 3.93, p < .05, partial η² = .16. A Tukey post hoc test revealed that the difference between the no-visualization and the veridical condition was significant. Thus, mirroring students their self-reported dilatory behavior resulted in more consistent studying.

Improvement of Self-Reflection
To investigate whether individual feedback provided by a graphical visualization of dilatory behavior improved self-reflection, we conducted an analysis of variance with repeated measurement. We included the mean degrees of strategy use as within subject factor and the experimental condition (veridical, random, no-visualization) as between subject factor. The main effect of measurement time failed statistical significance, F(1, 30) = 3.99, p =
© ISLS 51

ICLS2012

Volume 1: Full Papers

.06. The main effect of experimental condition, however, was significant F(2, 30) = 3.84, p < .05, partial η² = .20, as well as the interaction effect of experimental condition and measurement time, F(2, 30) = 4.24, p < .05, partial η² = .22. A Tukey post hoc test revealed that participants in the no-visualization condition reported to use less self-reflection strategies than participants in the veridical or random condition. Thus, participants with a visualization tool (veridical and random) recorded a more constant and higher level of self-reflection than the no-visualization group (see Figure 2, right).
2,7
4,5

Degree of dilatory behavior

2,6 2,5 2,4 2,3 2,2 2,1 2 1,9 1,8 1,7

control condition

Degree of self-reflection

random condition 4,0 feedback condition 3,5

random condition

3,0

control condition

feedback condition

2,5

week 1/2

week 3/4

week 5/6

week 7/8

week 9/10

week 11/12

week 1/2

week 3/4

week 5/6

week 7/8

week 9/10

week 11/12

Figure 2. Course of dilatory behavior (left) and self-reflection (right) during the semester in the experimental conditions (rated on a 5-point scale from 1 to 5)

Relations between components of self-regulated learning
First, we conducted an exploratory analysis of correlations in which we investigated the relations between components of self-regulated learning. For this purpose, we averaged the weekly collected individual scores of the variables of interest and computed Pearson correlations. Table 1 provides an overview of the intercorrelations. Non-specific goals were negatively related to both the average motivation (r = -.48, p < .01) and learning outcomes (declarative knowledge: r = -.39, p < .05; conceptual knowledge: r = -.43, p < .05). In contrast, specific goals were related to better learning outcomes (declarative knowledge: r = .47, p < .01; conceptual knowledge: r = .59, p < .01), higher motivation (r = .29, p < .05), and less intra-individual variance in dilatory behavior (r = -.37, p < .05), and thus contributed to a consistent learning progress. Dilatory behavior was related to low motivation (r = -.42, p< .01). Related to self-reported strategy use, we found few correlations. However, the results of the correlation analyses indicated different relations between motivational, cognitive, metacognitive and behavioral aspects in self-regulated learning. Thus, the results offered the opportunity to improve the whole process by supporting one particular aspect of self-regulated learning, for example dilatory behavior. Table 1. Summary of inter-correlations among variables of self-regulated learning and learning outcomes Measure N 1 2 3 4 5 6 7 8 9 10 1. Declarative knowledge 35 -.55** .47** -.39* -.33 .12 -05 .00 -.09 .11 2. Conceptual knowledge 35 -.59** -.43* -.18 .10 .08 07 -.22 .25 3. Specific goals 46 --.49** -.03 .29* -.37* -.07 -.05 .13 4. Non-specific goals 46 -.25 -.48** -.06 -.21 -.21 -.22 5. Dilatory behavior DB 49 --.42** -.02 -14 .05 -.23 6. Motivation 49 --.16 .24 .40** .01 7. Variance of DB 49 -.08 .04 .00 8. Metacognitive Strat. 49 -.37* .03 9. Elaboration 49 --.16 10. Organization 49 -11. Rehearsal strategies 49 Note. * p < .05; ** p < .01

11 -.03 .13 -.08 .05 -.23 .04 -.10 .27 .00 .37** --

Improvement of learning outcomes
In the next step, we investigated improvements of learning outcomes due to the experimental conditions. Table 2 provides an overview of means and standard deviations in the knowledge tests separately for the experimental conditions. To investigate whether prior knowledge was comparable, we conducted a multivariate analysis of variance with declarative knowledge and conceptual knowledge as dependent variables and experimental condition (veridical, random, no-visualization) as independent variable. The results showed that the experimental conditions were comparable with regard to this important prerequisite, Pillay’s trace = .07, F(4, 90) = 0.84, p = .50. Second, we conducted a multivariate analysis of variance with posttest results (declarative and conceptual knowledge) as dependent variables and experimental condition (veridical, random, no-

© ISLS

52

ICLS2012

Volume 1: Full Papers

visualization) as independent variable. The difference between the conditions failed statistical significance, Pillay’s trace = .15, F(4, 64) = 1.30, p = .28. Thus, no statistically significant, direct effect of experimental conditions on learning outcomes could be found. Table 2. Means and standard deviations of the knowledge test results Experimental condition No-visualization Random visualization Knowledge tests N M (SD) N M (SD) Conceptual Pre 14 7.29(4.68) 18 6.83(3.43) knowledge Post 14 6.81(2.75) 18 7.13(3.18) Declarative Pre 11 4.00(5.14) 15 6.11(6.10) knowledge Post 11 16.80(7.97) 15 17.09(5.91)

Veridical visualization N M (SD) 16 6.44(3.16) 16 6.67(1.87) 9 5.94(5.34) 9 12.33(4.74)

As we expected that metacognitive strategies and volitional control influenced learning outcomes, we first conducted two separate multiple regression analyses (backward) with indicators of metacognition and volitional control (goal categories, metacognitive strategies, motivation, intra-individual variance of dilatory behavior) as potential predictors of conceptual and declarative knowledge – which were the dependent variables. Additionally we included the pretest as predictor. Only specific goals were included in the final regression model for conceptual knowledge, t = 4.09, p < .01, β = .59, R² = .34. For declarative knowledge, specific goals, t = 3.43, p < .01, β = .47, and prior knowledge t = 2.73, p < .05, β = .38, R² = .50, were included in the final regression model. The more often students set up specific goals, the higher were learning outcomes in conceptual and declarative knowledge tasks as measured by the knowledge test. To investigate aspects that are positively related to specific goal setting, we conducted another multiple regression analysis. We included motivation, variance of dilatory behavior and prior knowledge as potential predictors and specific goals as dependent variable. We identified variance of dilatory behavior as having a negative impact on specific goals, t = -3.05, p< .01, β = .37, and motivation having a positive impact, t = 2.66, p< .05, β = .37, R² = .26. Thus, students who reported to learn more consistently and assessed the topic as personally relevant set a higher number of specific learning goals. As reported above, the graphical visualization with veridical feedback improved consistent studying and reduced the variance of dilatory behavior. Thus, one can speculate that the tool had an indirect influence on goal setting and on conceptual and declarative knowledge tasks, although the direct effect of the visualization on learning outcomes did not become significant, probably because of the low number of participants. However, supporting self-monitoring of dilatory behavior with a graphical visualization reduced ups and downs in postponing that supported specific goal setting that improved learning outcomes. Thus, providing a graphical visualization initiated a domino effect in selfregulated learning. To investigate this assumption, we conducted a path analysis with a linear contrast coding for experimental condition (feedback: 1; random: 0; no-visualization: -1) to account for both the signaling effect and the individual feedback. The results are illustrated in Figure 3. The model fit was satisfactory, CFI =.97; CMIN/df = 1.20, indicating that the relationships in these data could be well represented by this model.

Figure 3. Path model illustrating the domino effect initiated by a metacognitive awareness tool (all regression weights are significant at least at the .05-α-level)

Discussion
The aim of the current study was to evaluate a tool for self-regulated learning that should help students to reduce their dilatory behavior and improve their self-regulated learning. Deliberately monitoring of one’s own problematic behavior was expected to induce a reactivity effect that can promote changes in behavior without further instructional support (Zimmerman, 2002). A graphical visualization of dilatory behavior should help students to become aware of their dilatory behavior and improve it (Schraw, 1998). In web-based selfmonitoring protocols, students recorded their self-regulated learning behavior once a week. To support such monitoring and increase its effects, we used a line chart and reflected back to students their self-reported degree of dilatory behavior and categorized their scores of dilatory behavior in low, medium and critical levels using traffic light colors. To evaluate this metacognitive awareness tool, we conducted an experimental field study with three different groups: The veridical feedback group received their dilatory behavior scores in an individual

© ISLS

53

ICLS2012

Volume 1: Full Papers

line-chart. The random feedback group received random scores in the same chart. The no-visualization group did not receive any visualization. The results of the experimental field-study can be summarized as follows. First, directing students’ attention to dilatory behavior using traffic light colors (signaling effect) caused a greater degree of self-reported self-reflection in both groups with visualizations (random and veridical). Thus, students in these conditions were more likely to think about their comprehension and effectiveness of learning. This is a precondition for developing self-regulated learning and motivational beliefs (Pintrich, 2004; Zimmerman, 2002). Visualizations seem to have signaling effects that helped students to become aware of their learning processes that were reflected more intensively. Second, by directing students’ attention to individual dilatory behavior, and thus supporting a realistic self-observation, the veridical feedback tool reduced ups and downs in learning. Students in that condition avoided postponing to a substantial extent. In this way, the veridical visualization contributed to more continuous studying. Third, a low variance of dilatory behavior was related to a greater number of specific goals. Specific goals included both a concrete topic and strategic planning. Such goals are expected to be related to better learning outcomes (Schunk, 2001). A path analysis showed that providing a metacognitive awareness tool that mirrors students’ dilatory behavior and thus supports self-monitoring of one’s behavior could have caused a domino effect in the learning process. In the current study, a visualization of dilatory behavior increased regular studying and improved goal setting, which improved learning outcomes. This domino effect is an indicator for the strong connectivity of behavioral (postponing), motivational (goal setting), and cognitive processes (learning outcomes) in self-regulated learning. As a consequence, students who engaged in learning activities regularly might have experienced a lower stress level before their exams (Tice & Baumeister, 1997). Furthermore, studying regularly could have eliminated the pressure to learn and understand high amounts of learning contents in a very short time, because it enhanced time-on-task (Tabak, Nguyen, Basuray & Darrow, 2009). As mentioned above, a lower stress level and the will to study regularly might enable students to take responsibility for their studying, and thus plan and reflect on it. These potential effects need to be proved in future studies. It seems to be important for future research to examine self-regulated learning as a complex system of different interacting aspects. Despite the promising results, there are some limitations in the current study. First, the low number of participants diminishes the statistical power of complex analyses and should therefore be interpreted carefully. The low number of participants can be traced back to the field-experimental design in demanding courses of study. Nevertheless, a field study raises the ecological validity of the data. Second, we partially referred to selfreported strategy use. As noted in prior research and supported by the missing relations between strategy use and learning outcomes in the present study, self-reported strategy use is not always reliable and thus of limited interpretability (Boekaerts & Corno, 2005).Therefore, we added alternative methods to investigate self-regulated learning, like the categorization of students’ learning goals. Third, our baseline - students in the no-visualization group - also experienced an intervention. As they recorded their class preparation and homework, selfmonitoring was supported concerning important aspects of learning that were included in the self-monitoring protocols. This might lead to an underestimation of the effect of metacognitive awareness tools. Nevertheless, though self-monitoring protocols are indeed an intervention tool, they are at the same time a sophisticated tool to diagnose self-regulated learning efficiently (Schmitz & Wiese, 2006). In summary, the metacognitive awareness tool used in the present study had beneficial effects on selfregulated learning. The study showed that metacognitive awareness tools, used for individual learning, had two main effects: raising students’ metacognitive awareness and supporting the accuracy of self-monitoring. More precisely, signaling dilatory behavior caused stronger self-reflection, and supporting realistic self-observation by veridical feedback induced behavior calibration in regard to dilatory behavior. Therefore, graphical visualizations of dilatory behavior, which can be easily integrated in online management systems, seem to be promising to improve the effects of self-monitoring protocols. They guide students’ awareness to problematic behavior that is ubiquitous among university students.

References
Boekaerts, M. (1996). Self-regulated learning at the junction of cognition and motivation. European Psychologist, 1(2), 100-112. Boekaerts, M., & Corno, L. (2005). Self‐regulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54(2), 199-231. Chang, M.-M. (2007). Enhancing web-based language learning through self-monitoring. Journal of Computer Assisted Learning, 23(3), 187-196. Cho, K., Cho, M.-H., & Hacker, D. J. (2010). Self-monitoring support for learning to write. Interactive Learning Environments, 18(2), 101-113. Corno, L. (1986). The metacognitive control components of self-regulated learning. Contemporary Educational Psychology, 11(4), 333-346. Dehler, J., Bodemer, D., Buder, J., & Hesse, F. W. (2011). Guiding knowledge communication in CSCL via group knowledge awareness. Computers in Human Behavior, 27(3), 1068-1078.

© ISLS

54

ICLS2012

Volume 1: Full Papers

Ferrari, J., Parker, J., & Ware, C. (1992). Academic procrastination: Personality correlates with Myers-Briggs types, self-efficacy, and academic locus of control. Journal of Social Behavior and Personality, 7, 495502. Joseph, L. M., & Eveleigh, E. L. (2011). A review of the effects of self-monitoring on reading performance of students with disabilities. The Journal of Special Education, 45(1), 43-53. Klassen, R. M., Krawchuk, L. L., & Rajani, S. (2008). Academic procrastination of undergraduates: Low selfefficacy to self-regulate predicts higher levels of procrastination. Contemporary Educational Psychology, 33(4), 915-931. Lan, W. Y. (1996). The effects of self-monitoring on students’ course performance, use of learning strategies, attitude, self-judgment ability, and knowledge representation. The Journal of Experimental Education, 64, 101-115. Lay, C., & Schouwenburg, H. (1993). Trait procrastination, time management, and academic behavior. Journal of Social Behavior and Personality, 8, 647-662. Lay, C., & Silverman, S. (1996). Trait procrastination, anxiety, and dilatory behavior. Personality and Individual Differences, 21(1), 61-67. Lewicki, P. (1983). Self-image bias in person perception. Journal of Personality and Social Psychology, 45(2), 384-393. Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation: A 35-year odyssey. American Psychologist, 57(9), 705-717. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16, 385-407. Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801-813. Sankaran, S. R., &Bui, T. (2001). Impact of learning strategies and motivation on performance: A study in webbased instruction. Journal of Instructional Psychology, 28(3), 191-198. Schmitz, B., & Wiese, B. S. (2006). New perspectives for the evaluation of training sessions in self-regulated learning: Time-series analyses of diary data. Contemporary Educational Psychology, 31(1), 64-96. Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113-125. Schunk, D.H. (2001). Social cognitive theory and self-regulated learning. In B.J. Zimmerman & D.H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed., pp. 125152). Mahwah, NJ: Erlbaum. Stainton, M., Lay, C. H., & Flett, G. L. (2000). Trait procrastinators and behavior/ trait-specific cognitions. Journal of Social Behavior & Personality, 15(5), 297-312. Steel, P. (2007). The nature of procrastination: A meta-analytic and theoretical review of quintessential selfregulatory failure. Psychological Bulletin, 133(1), 65-94. Tabak, F., Nguyen, N., Basuray, T., & Darrow, W. (2009). Exploring the impact of personality on performance: How time-on-task moderates the mediation by self-efficacy. Personality and Individual Differences, 47(8), 823-828. Tice, D. M., & Baumeister, R. F. (1997). Longitudinal study of procrastination, performance, stress, and health: The costs and benefits of dawdling. Psychological Science, 8, 454-458. Winne, P. H., & Hadwin, A. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277-304). Hillsdale, NJ: Erlbaum. Wolters, C. A. (2003). Understanding procrastination from a self-regulated learning perspective. Journal of Educational Psychology, 95(1), 179-187. Zimmerman, B.J. (2000). Attainment of self-regulation: A social cognitive perspective. In M. Boekaerts, P.R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-39). San Diego, CA: Academic Press. Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64-70. Zimmerman, B. J., Bandura, A., &Martinez-Pons, M. (1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29(3), 663676. Zohar, A., & Peled, B. (2008). The effects of explicit teaching of metastrategic knowledge on low- and highachieving students. Learning and Instruction, 18(4), 337-353.

Acknowledgments
The research reported in this paper was supported by the German Federal Ministry of Education and Research (BMBF) under contract 01PH08031A.

© ISLS

55

ICLS2012

Volume 1: Full Papers

Initial Validation of “Listening” Behavior Typologies for Online Discussions Using Microanalytic Case Studies
Alyssa Friend Wise, Ying-Ting Hsiao, Farshid Marbouti, Jennifer Speer, Nishan Perera Simon Fraser University, 250-13450 102 Avenue, Surrey BC, Canada, V3T 0A3 Email: afw3@sfu.ca, yha73@sfu.ca, fmarbout@sfu.ca, jspeer@sfu.ca, ncp3@sfu.ca Abstract: Collaborative participation in online discussions requires interacting with the ideas posted by others, yet many students do so ineffectually. This study conceptualizes these actions as online “listening” behaviors and proposes six prototypical behavioral patterns. To test these prototypes, microanalytic case studies were conducted. Five cases with distinct listening profiles are described: “Gigi”, showed a content coverage approach, viewing all her classmates’ posts but not drawing on them in her own; “Stephanie”, showed an interactive approach, focusing on a smaller number of posts, but building on them in her contributions. “Ron” showed a targeted approach, focusing on particular kinds of posts in the discussion and disregarding the rest; “Darren” showed a social coverage approach to listening, briefly scanning many posts and acknowledging his classmate’s contributions in a conversational tone; and “Julie”, showed a disregard for listening, ignoring the majority of the posts in the discussion. Theoretical and practical implications of the findings are discussed.

Introduction
In previous work we have argued for the importance of students’ “listening” in online discussions and the need for research to understand different listening approaches (Wise, Marbouti et al., 2011). For the context of online discussions, we define listening behaviors as the ways learners engage with others’ posts (e.g. which posts they attend to, at what point(s) in time, and in what ways). It is through these behaviors that learners become aware of others’ ideas, a pre-requisite for taking-up these ideas and creating comments that respond to them (Suthers, 2006). As attending to others’ ideas is necessary as a first step towards interactivity, a thorough notion of discussion participation must include not only the activity of making posts (i.e. “speaking”), but also that of viewing posts of others (i.e. “listening”). Prior work has suggested that in general learners in online discussions attend to each other’s ideas superficially (e.g. Hewitt, 2003; Palmer et al., 2008; Thomas, 2002) and that there is a relationship between the amount of engagement with other’s posts and course performance (Hamann et al., 2009; Morris et al., 2005). However, this work is based on examinations of student behaviors in aggregate. As we have argued previously, there is no reason to assume that students all listen in similar ways; in fact, initial evidence suggests the contrary (Wise, Perera et al., 2012). Even for individual students, participation is unlikely to be uniform over time; such temporal variations are artificially smoothed (and thus analytically eliminated) in the creation of aggregate measures. Thus, patterns of student listening in online discussions and their relationships to learning may be much more complex than previously suggested. To take full advantage of the analytical opportunities afforded by rich log-file data we need methods that unpack both the individual (Peters & Hewitt, 2010) and temporal (Dringus & Ellis, 2010) aspects of discussion participation. In past work, we found that different students interacted with prior messages in distinct ways (Wise, Perera et al., 2012) and documented three particular approaches to online listening (Wise, Marbouti et al., 2011). While our initial work on listening behaviors focused on empirically-derived patterns across multiple students, the current study moves this line of research forward by generating a set of theoretical listening typologies and comparing them with in-depth case studies of a small number of students’ individual behaviors. In this way we work towards a taxonomy of online listening behaviors that can be used to analyze and support online discussions.

Conceptualizing Online Listening Behaviors as Part of Discussion Participation
To conceptualize listening as an aspect of participation in online discussions, we draw on Knowlton’s (2005) taxonomy of five online discussion participation types characterized in terms of their perspectives on collaborative knowledge construction and its enactment in online discussions. Knowlton argues that based on different conceptualization of the meaning and purpose of online discussions, learners can participate differently and advance to more sophisticated participation types over time. Expanding on this general theorization of participation types, we propose specific listening behaviors that correspond with each category in the taxonomy (see Table 1). Knowlton’s (2005) model begins with the category of passive participation: those students who do not actively contribute posts. He suggests two explanations for this behavior. First, students’ inactivity may be due to a lack of interest in learning, a lack of knowledge in engaging in a collaborative learning environment, or a

© ISLS

56

ICLS2012

Volume 1: Full Papers

poor ability to manage their time. We would expect these students to view few posts and spend little time on them. Second, these students may be engaging in initial “legitimate peripheral participation” (Lave & Wenger, 1991) as they familiarize themselves with the discussion. This second class of students would be expected to actively listen to the discussion as they prepare to eventually contribute, thus we consider them as a separate listening type. The second level of participation is developmental, in which students regard discussions as a social space for building community (rather than a place for learning the course material). Consequently, they are responsive in acknowledging their peers and their efforts in a conversational tone but the content of their comments lacks depth of thinking. In terms of listening, we would expect them to orient socially based on relationships, reading posts made by their friends and those that reply directly to them. The third level of participation is generative. Students at this level consider discussions as an individual channel to demonstrate their understanding and thus are more likely to contribute lecture-like posts in an academic writing style. Concentrating on articulating their own ideas to the instructor, we would expect them to listen primarily to the posts of the instructor rather than those of their peers. The two highest levels of participation are dialogical and metacognitive. At these levels learners participate with a collaborative mindset. While focused on their peers, they craft their comments with the goal of more than just community-building, recognizing a need to connect with others in a collective process of building ideas. For example, learners exhibiting dialogical participation are expected to be interactive in their listening, attending in depth to the contents’ of others posts. This enables them to build on other’s ideas in their own posts, and even synthesize disparate viewpoints to pull the group’s conversation together. Learners at the metacognitive level exhibit dialogical participation plus have the additional characteristic of reflecting on their own thinking. Their goal of participation is not only to understand the topic discussed, but also recognize the influence of others’ ideas in changes in their own thinking. As metacognition often occurs internally, it can be difficult to detect (Knowlton, 2005); in terms of listening behaviors it might be reflected in substantial time spent re-reading posts made by others and oneself. Table 1: Knowlton’s (2005) participation types and proposed listening behaviors (Knowlton, 2005) Description Sees the discussion as a channel to receive information; does not contribute posts Sees the discussion as a social space; posts to acknowledge classmates Posts to construct their own ideas and report them to the instructor Posts to clarify their understanding through interacting with others Dialogical participation plus reflects on the process of knowledge development Proposed Listening Behaviors Disregardful. Minimal attention to others’ ideas. OR Preparatory. Listens to others’ ideas in preparation to contribute Social. Attends to peers’ posts in a social sense, superficially based on reciprocity Targeted. Attends primarily to the instructor’s posts, not those of their peers Interactive. Actively attends to content of peers’ posts (draws on them to contribute) Reflective. Dialogical plus reviews own and others’ posts throughout the discussion

Participation Type Passive

Developmental Generative Dialogical Metacognitive

Purpose of the Research
As a first step to empirically ground the proposed typologies of listening behaviors, we constructed a series of microanalytic case studies of student interactions in online discussions as part of a university class. Extreme case sampling (Patton, 2002) was used to identify patterns of interaction that were clearly distinct and would thus allow for well-defined characterization and potential alignment with the theoretical-generated typologies. Specifically, we attempted to answer two research questions: 1. What distinct listening patterns do individual students enact over time as they participate in asynchronous discussions as part of an online course? 2. How do these patterns align with theoretically-predicted typologies of listening behaviors?

Methods
Learning Environment and Discussion Task
Students in a fully-online undergraduate course on Educational Psychology participated in six week-long discussions, in groups of 10-13 students, worth 20% of their grade. Discussions were conducted in Phorum, a basic asynchronous threaded discussion forum. Students were required to discuss contrasting perspectives on authentic, contested questions in educational psychology. Discussions ran from Monday to Friday and required
© ISLS 57

ICLS2012

Volume 1: Full Papers

students to contribute three posts on different days. Each week two students were assigned roles: Synthesizer (summarize the group’s early ideas) and Wrapper (pull the conversation together at the end).

Data Extraction and Processing
Thirty-three consenting students’ log-file data was collected based on their activity in the discussion forum. The system automatically logged every action students performed in the discussions, noting the identity of the post read or created and applying a time-date stamp. Three types of actions were processed: view, post, and review (for details see Wise, Marbouti et al., 2011). Views were subdivided into reads or scans based on a threshold reading speed of 6.5 words per second (see Hewitt et al., 2007). Action duration was calculated by subtracting the time between actions for each student. One limitation of this calculation is that it does not account for the fact that students may engage in off-task activities during some portion of this time. Adjustments were made for unlogged system exits and actions were divided into sessions (a series of consecutive actions in the discussions) following the procedure described in Wise, Perera et al. (2012). Sixteen interaction variables (e.g. number and duration of sessions, total time spent reading, percentage of total posts viewed) were calculated for all students based on the log-file data.

Case Selection and Framing
Potential cases were selected for analysis by ranking students based on each interaction variable. Extreme sampling was enacted by identifying students who appeared most frequently in the top or bottom 10% for each variable (indicating extremely high or low participation on that indicator). Eight cases were selected for further examination of listening patterns. Preliminary investigation compared these cases with respect to the selection criteria of being extreme and unique; three cases were eliminated and five were retained (three high-volume and two low-volume participants). The fourth discussion week (on the topic of critical thinking) was chosen as the focus week for case study; none of the students studied had assigned discussion roles this week. The analytic frame included all actions performed by a student during this week; to capture related early/late activity, any actions in the critical thinking discussion up to two days before or after were also included.

Microanalytic Case Analysis Process
To conduct the case studies, temporal microanalysis of log-file data (Wise, Perera et al., 2012) was employed. This is an analytic approach that provides a methodology for constructing a meaningful account of students’ participation in an online discussion by providing a detailed on-the-ground account of student behaviors. The first analytic step was to get a broad sense of each student’s behaviors in the discussions globally by looking at their overall data patterns (see Table 2). The number and duration of each kind of action were also represented with pie charts to examine their relative fraction of total activity (see Figure 1). The second analytic step narrowed in on the focal discussion. A session-by-session table of activity was built for each student (see Tables 3-7) and this data was used to make a preliminary characterization of the student’s participation and listening behaviors. This interpretation would be tested and further developed in the third stage of the analysis. The third analytic step was to reconstruct learners’ experiences participating in the discussion during the focus week, with a focus on their listening behaviors. To do this, we created a dynamic discussion map: a spreadsheet of all posts organized by date, time and threading structure that could be filtered to reproduce the state of the forum at any point in the discussion. The log-file data for each case was examined using this map to provide context; this allowed us to make sense of the actions students performed within the historical setting of the discussion as it appeared at that time. (For example, we could distinguish when a student read posts scattered across a discussion versus those in sequence, even if by the end of the discussion the posts were no longer adjacent due to interspersed replies.) Analyzing students’ behaviors action-by-action in this way, we produced a narrative for each student reconstructing their participation in the discussion. Reference to the content of the student’s posts was used to contextualize actions in the discussion as needed.

Results
The five students studied were Gigi, Stephanie, Ron, Darren, and Julie (pseudonyms). The first three were high participation students; all spent relatively long time in the discussions, though activity patterns were distinct (see Table 2 and Figure 1). The latter two were low participation students; specific activity patterns also differed.

Gigi: A content coverage approach to listening
Gigi visited the discussions often, logging-in for 80 sessions over 23 hours (see Table 2). A large percentage of her sessions involved only reading actions (no posts made). Gigi viewed all posts in each discussion; however, many of these views were superficial scans. She made 24 posts in total, six more than required, and reviewed them multiple times during the discussions. In the Critical Thinking discussion, Gigi visited the forum every day, for a total of 13 sessions (see Table 3). She began early Monday morning, quickly viewing several posts from the preceding discussion. She then read the current week’s discussion prompt before returning to the previous forum to review her final post

© ISLS

58

ICLS2012

Volume 1: Full Papers

and the last post in the discussion. Later Monday, she viewed 11 classmates’ posts in a linear sequence of scans and quick reads. Prior to making a reply, Gigi reviewed her final post from the previous discussion (about ‘overlearning’ in math) one more time. She then contributed her first post, building on this concept and its relationship to transfer, while integrating academic language from the textbook. Finally, she quickly viewed three posts and logged-off. Table 2: Aggregate measures of listening and speaking behaviors in the discussions Behavior Time in the system (hrs) Number of sessions Percent of sessions with posts made Percent of time spent reading Number of posts viewed Percent of views that were reads (not scans) Av. time reading a post (min) Percent of posts viewed at least once Percent of posts read at least once Number posts made Av. length of post (words) Av. time making a post (min) Av. number of reads before posting Av. number of reads after posting Number of reviews Final course grade Gigi 22.8 80 28% 64% 413 56% 3.8 100% 80% 24 258 18.0 2.6 1 30 92 Stephanie 30.7 62 29% 97% 330 55% 9.8 100% 62% 18 247 0.8 4.8 0.4 5 87 Ron 24.1 28 68% 71% 217 65% 7.2 38% 31% 19 174 18.2 5.7 0.5 9 72 Darren 5.5 33 61% 48% 304 33% 1.6 89% 36% 26 134 4.9 2.2 0.6 2 67 Julie 6.6 17 82% 47% 158 25% 4.7 39% 14% 14 288 12.4 1.6 0 3 87

The next day, Gigi worked through the forum methodically and efficiently: she quickly read all four new posts in order, scanning the discussion prompt a couple of times between reads. She also reviewed her post from the previous evening. Later that night, she logged-in again and replied to the final post in the discussion, picking up on the notion of transfer she mentioned earlier. She then read one post and left. On Wednesday, Gigi had two quick sessions checking the discussion for new posts. In two subsequent sessions, she viewed a small number of new posts in a distinctive pattern, toggling between them and the one she had made. Very late that night (1am Thursday morning), Gigi read one new post, created her third post for the week (emphasizing the importance of overlearning), and left. Although she had met her posting requirement for the week, Gigi returned to the discussion daily, engaging in short sessions to briefly view all new posts linearly. Table 3: Gigi’s activities in the critical thinking discussion by session 1† Session 2 3 4 5 6 7 8 9 10 11 12 13 Day Mon Mon Tue Tue Wed Wed Wed Wed Thur Thur Fri Sat Sun Time 2:32 21:59 19:55 22:31 1:30 17:13 18:27 21:42 1:41 21:19 1:02 2:35 4:30 Started Length 17 42 10 22 2 2 6 27 28 3 3 6 1 (min) Scans 3 7 2 1 0 0 2 5 1 2 1 4 1 (# of) Reads 4 4 5 1 1 1 3 5 0 1 4 7 1 (# of) Posts 0 1 0 1 0 0 0 0 1 0 0 0 0 (# of) Reviews 2 1 1 0 0 0 2 2 0 0 0 0 0 (# of) † Several actions in this session took place in the forum for the previous discussion, see text for details. Total Time

167 min 5 min 71 min 83 min 8 min

Stephanie: An interactive approach to listening
Stephanie spent the most time in the discussions overall, logging in 62 times for a total of 31 hours (see Table 2). Her primary activity in the forum was reading (see Figure 1) and she spent more time per post than each of the other four students. However, she only spent 15 minutes to make her 18 required posts, thus it is likely that she composed her posts in an outside tool during some of her long reading times.

© ISLS

59

ICLS2012

Volume 1: Full Papers

Figure 1. Proportion of total number of events and amount of time reading, scanning, reviewing and posting for (a) Gigi (b) Stephanie (c) Ron (d) Darren and (e) Julie. Pie size is proportional to total amount of activity. In the Critical Thinking discussion, Stephanie visited the discussion seven times on three different days, with most of her time spent on reading (see Table 4). She viewed the entire discussion, though she focused her attention on a small number of posts. On Monday afternoon, she began by scanning the discussion prompt and several posts from the prior discussion. She then spent a few minutes reading the Critical Thinking prompt and one of the existing replies. A few hours later, she logged in again and read another reply. That evening she spent a long time on the discussion prompt and the two posts she viewed earlier; she then viewed the rest of the posts in the forum. After, she replied to the prompt stating her position that critical thinking should be taught as part of a coordinated approach with other subjects. She then scanned the remaining two posts and logged-out. Stephanie returned to the discussion on Thursday evening for another session. During this time, she read two posts for almost an hour and then viewed other posts in a linear fashion. At last she made her second post trying to present a group position and respond to a question posed by a classmate. On Friday, Stephanie briefly viewed the discussion prompt again, but did not read any new posts. Later that evening, she spent almost two hours viewing all of the posts, three of which she spent substantial time on. She then made her final post, which built on the wrapper’s conclusion post, drawing on her own relevant personal experiences. Table 4: Stephanie’s activities in the critical thinking discussion by session Session Day Time Started Length (min) 1† Mon 14:07 6 2 Mon 16:15 3 3 Mon 21:05 129 4 Thur 20:38 125 5 Fri 15:52 4 6 Fri 19:53 164 7 Mon 20:38 <1 Total Time

430 min

Scans (# of) 2 1 6 2 0 3 1 4 min Reads (# of) 2 1 6 11 1 11 1 425 min Posts (# of) 0 0 1 1 0 1 0 1 min Reviews (# of) 0 0 0 0 0 0 0 0 min † Several actions in this session took place in the forum for the previous discussion, see text for details.

© ISLS

60

ICLS2012

Volume 1: Full Papers

Ron: A targeted approach to listening
Ron spent a similarly long amount of time in the discussions as Gigi (see Table 2), with 24 hours across 28 sessions. While he only viewed 38% of his classmates’ posts, when he did read, he took his time. Most sessions included posting actions, and he contributed one post more than the assignment required. In the Critical Thinking discussion, Ron logged-in on three days, always in the evening. The first day he had three separate sessions; the other two days he only had one session each (see Table 5). Early Monday Ron visited the prompts for the current and previous discussions. Later Monday, he spent substantial time reading all five existing posts, cycling between them and the prompt. He was then inactive for two hours before creating his own reply to the prompt. Because of the extended time between actions, this was considered a new session. Ron then left the forum. On Thursday Ron quickly viewed three new posts and then read a post by the Synthesizer, viewed its first reply, and reread the synthesis again. He then scanned his previous post, and replied to the synthesis, explicitly responding to a question raised. Finally, he spent 30 minutes revisiting the synthesis post, and 5 minutes on its first reply before exiting the discussion leaving 14 posts unviewed. On Friday Ron scanned a post midway through the discussion, before reading the final four new posts in reverse chronological order. He returned to the synthesis post, quickly viewed two of the replies to this post and posted, again addressing other questions raised by the Synthesizer. In total, Ron viewed only 18 of the 39 posts available this week. Table 5: Ron’s activities in the critical thinking discussion by session Session Day Time Started Length (min) Scans (# of) Reads (# of) Posts (# of) Reviews (# of) * Estimated value 1 Mon 2:03 4 0 2 0 0 2 Mon 18:44 96 2 5 0 0 3 Mon 20:21 13* 0 0 1 0 4 Thur 23:14 106 1 7 1 1 5 Fri 19:23 25 3 6 1 0 Total Time

244 min 3 min 190 min 51 min <1 min

Darren: A social coverage approach to listening
Darren spent little time in the discussions (33 sessions, 5.5 hours, see Table 2). He viewed most posts at least once, but the majority of his views were scans. He was active in making many short posts (26 in total). In the Critical Thinking discussion Darren had several brief sessions and spent most of his time on a small number of posts (see Table 6). On Monday evening he “finished up” the prior discussion, viewing its remaining new posts. In the current week’s discussion he read the discussion prompt, scanned four of the seven posts and read one post extensively. Later he viewed the prompt, the two skipped posts, and a previously scanned post; he then replied to the post he read earlier. On Wednesday, Darren first spent a few minutes reading a new post in the prior discussion. In the Critical Thinking discussion, he read the first reply to his post, scanned 11 new posts, and read the last one (contributed by a classmate, Mike). Later, Darren revisited nine of these posts, and posted a reply to Mike explicitly responding to his question. He also acknowledged another classmate (Sundeep) for sharing an example. Thursday, Darren continued the conversation by viewing the replies to his post (one by Mike) and contributing a reply to Mike’s reply. He then re-viewed several posts in the same thread and made another post replying to the last one; this time he acknowledged a third classmate (Melissa). Darren returned to the discussion twice more, both times viewing all new posts in under a minute. Table 6: Darren’s activities in the critical thinking discussion by session Session Day Time Started Length (min) Scans (# of) Reads (# of) Posts (# of) Reviews (# of)
© ISLS

1† Mon 19:27 12 7 4 0 0

2 Mon 21:09 14 2 3 1 0

3† Wed 18:19 4 13 3 0 0

4 Wed 22:13 7 8 2 1 0

5 Thur 21:17 8 4 3 2 1

6 Fri 16:36 1 4 1 0 0

7 Mon 11:59 1 4 0 0 0

Total Time

47 min 8 min 27 min 12 min <1 min
61

ICLS2012

Volume 1: Full Papers

Several actions in this session took place in the forum for the previous discussion, see text for details.

Julie: A disregard for listening
Julie had the fewest sessions and least amount of time in the discussions (Figure 1). She viewed only 39% of others’ posts and read less than half of those. She was also inactive in speaking, contributing only 14 posts total. During the Critical Thinking Discussion, Julie logged-in three times, creating a single post each time (see Table 7). On Monday, she quickly viewed the discussion prompt and the four existing posts in order. While she spent 54 minutes on the last post she read before making a post, it is likely that some of this time indicates off-task behavior because the post only consisted of 190 words and she did not refer to it in her own. Instead, her post (made in the final action of the session), replied directly to the discussion prompt and shared her own ideas on the subject. On Wednesday, Julie had a short session in which she viewed the synthesizer post, skipped its several replies, and replied directly to it. This post also did not refer to any other posts from the discussion. In her final session on Friday, Julie viewed only five posts. These posts were distributed across the forum and half of them were only scanned. Julie created a post as a reply to one of these, but its content repeated the same ideas as her previous posts. At the end of the discussion Julie had left two-thirds of the posts (22 of 33) unread. Table 7: Julie’s activities in the critical thinking discussion by session Session Day Time Started Length (min) Scans (# of) Reads (# of) Posts (# of) Reviews (# of) 1 Mon 13:48 64 3 3 1 0 2 Wed 22:32 6 0 1 1 0 3 Fri 12:45 18 3 3 1 0 Total Time

89 min 1 min 60 min 28 min 0 min

Discussion
The case studies displayed unique interaction styles, showing approaches to listening that emphasized different aspects of the conversation. Gigi, Stephanie and Darren viewed a large percentage of their classmate’s posts, but in different ways for different apparent purposes. Gigi’s listening activity was broad yet superficial; she seemed to “cover” the content by glancing over all the posts without spending much time on any one post and briefly revisiting the forum to check for new posts after fulfilling her posting requirement. She also seemed to compare her classmate’s posts to her own but never drew on their ideas. Her posts often used academic language. Darren also “covered” the discussion by opening a high percentage of posts for short periods of time; however his use of casual language and focus on acknowledging classmates by name suggests a more social purpose to his behavior. The inverse levels of interactivity and formality of language used by Darren and Gigi has also been seen in previous research (Thomas, 2002), suggesting that the goals of having a high level of academic discourse and building trust and relationships between students may at times be at cross purposes. While Stephanie also viewed a large percentage of the discussion, she is distinguished from Gigi and Darren in that she not only attended to all of her classmates’ posts, but also built substantively on their ideas. In addition, Stephanie tended to concentrate her reading activity on only a portion of the posts made, spending a substantial amount of time reading (and re-reading) them. Her posts drew on these posts, at times synthesizing them together, and built her ideas on top of them to move the conversation forward. In contrast to these three, Ron and Julie did not seem to worry about “covering” the conversation. Instead, Ron focused on strategically reading certain posts (particularly by the Synthesizer) to respond to the prompt while Julie made no effort to read deeply or thoroughly. While Julie’s behaviors echo common findings of minimal student effort (Palmer et al., 2008; Webb et al., 2004), the other students’ behaviors extend the knowledge base by presenting evidence of several new, distinct, and seemingly purposeful strategies for listening. Comparing the cases with the proposed listening typologies (see Table 1), Julie’s actions align with the predicted behaviors for disregardful listening, indicating that she did not value, or understand how to engage in collaborative processes in online discussions. No student seemed to exhibit a preparatory listening style; this is unsurprising in a formal discussion context requiring students to post. Social listening appeared present in Darren’s behaviors of reading posts by particular classmates and acknowledging people without engaging deeply with their ideas. Targeted listening was evidenced in Ron’s focus on the “authoritative” voices of the prompt and synthesizer, and Interactive listening was clearly enacted by Stephanie as she spent extensive time reading and responding to others’ ideas. Stephanie’s focus on a subset of the posts was not predicted as part of Interactive listening and thus is an interesting phenomenon for further investigation. It may be that this selective attention make the tasks of interactivity and synthesis more manageable, or simply that these were the posts she

© ISLS

62

ICLS2012

Volume 1: Full Papers

found most valuable. One student’s behavior did not clearly align with a single typology. Gigi paid superficial attention to many of her peer’s comments, but did not respond to them socially. The posts she made herself focused on her own ideas, but her listening was not targeted towards the instructor or authoritative others. Finally, despite her lack of interactivity in posting, she spent considerable time comparing her posts with those of others. Further work is needed to examine if Gigi’s behaviors can be explained as a hybrid of existing categories, if an additional listening typology is needed, or if these patterns simply reflect one person’s idiosyncratic behaviors.

Conclusion
Past research has indicated that student interactions with prior messages in online discussions are often suboptimal; however, little empirical work has examined the different approaches students take to “listen” to others’ ideas. The general fit between the theoretically proposed typologies and the empirical cases studied here presents some initial validation for a taxonomy of listening in online discussion, showing that students’ listening is more varied and nuanced than previously known. The hybrid of behaviors seen for Gigi also indicates that categories may not be mutually exclusive. Further work is needed to probe the motivation for and usefulness of Gigi’s eclectic behaviors, as well as Stephanie’s concentration on subsets of the discussion. This work lays the foundation for building a more detailed understanding of different learner participation approaches that can be used to analyze and support online discussions. Specifically, this can provide useful information for educators in designing early detection and support (e.g. feedback, listening strategies and interaction guidelines) tailored for different participation types. Additional work is ongoing to test, validate, and expand these listening prototypes for different discussion contexts using additional measures and larger data sets. Future research will extend this work by examining motivations behind different behaviors as well as their impact on the quality of individual and group learning using additional data and measures such as interviews and content analysis.

References
Dringus, L. P., & Ellis, T. (2010). Temporal transitions in participation flow in an asynchronous discussion forum. Computers in Education, 54(2), 340–349. Hamann, K., Pollock, P. H., & Wilson, B. M. (2009). Learning from “listening” to peers in online political science classes. Journal of Political Science Education, 5(1), 1–11. Hewitt, J. (2003). How habitual online practices affect the development of asynchronous discussion threads. Journal Educational Computing Research, 28(1), 31-45. Hewitt, J., Brett, C., & Peters, V. (2007). Scan Rate: A new metric for the analysis of reading behaviors in asynchronous computer conferencing environments. AJDE, 21(4), 1-17. Knowlton, D.S. (2005). A taxonomy of learning through asynchronous discussion. Journal of Interactive Learning Research, 16(2), 155-177. Lave, J., & Wenger, E. (1991). Situated learning: legitimate peripheral participation. New York: Cambridge. Morris, L. V., Finnegan, C., & Wu, S. (2005). Tracking student behavior, persistence, and achievement in online courses. The Internet and Higher Education, 8(3), 221–231. Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion help? The impact of a formally assessed online discussion on final student results. British Journal of Educational Technology, 39(5), 847-858. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage. Peters, V., & Hewitt, J. (2010). An investigation of student practices in asynchronous computer conferencing courses. Computers in Education, 54(4), 951–961. Suthers, D. (2006). Technology affordances for intersubjective meaning making: A research agenda for CSCL. International Journal of Computer-Supported Collaborative Learning, 1(3), 315-337. Thomas, M. J. W. (2002). Learning within incoherent structures: the space of online discussion forums. Journal of Computer Assisted Learning, 18, 351-366. Webb, E., Jones, A., Barker, P., & van Schaik, P. (2004). Using e-learning dialogues in higher education. Innovations in Education and Teaching International, 41(1), 93-103. Wise, A. F., Marbouti, F. Speer, J., & Hsiao, Y. T., (2011). Towards an understanding of “listening” in online discussions: A cluster analysis of learners’ interaction patterns. In H. Spada, G. Stahl, N. Miyake & N. Law (Eds.) Proceedings of the International Conference on Computer Supported Learning 2011, Volume I (pp. 88-95), Hong Kong, China: International Society of the Learning Sciences. Wise, A. F., Perera, N., Hsiao, Y. Speer, J., & Marbouti, F. (2012). Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course. Internet and Higher Education, 15(2), 108–117.

Acknowledgments
This work was supported by the Social Sciences and Humanities Council of Canada.

© ISLS

63

ICLS2012

Volume 1: Full Papers

How to schedule multiple graphical representations? A classroom experiment with an intelligent tutoring system for fractions
Martina A. Rau, Carnegie Mellon University, Human-Computer Interaction Institute, 5000 Forbes Ave, Pittsburgh, PA 15213, USA, marau@cs.cmu.edu, Nikol Rummel, Ruhr-Universität Bochum, Institute of Educational Research, Universitätsstraße 150, 44801 Bochum, Germany, nikol.rummel@rub.de Vincent Aleven, Laura Pacilio, Zelha Tunc-Pekkan, Carnegie Mellon University, Human-Computer Interaction Institute, 5000 Forbes Ave, Pittsburgh, PA 15213, USA, aleven@cs.cmu.edu, lpacilio@andrew.cmu.edu, zelha@cs.cmu.edu Abstract: Providing learners with multiple representations of the learning content has been shown to enhance learning outcomes. When designing problem sequences with multiple representations, designers of intelligent tutoring systems must decide how to schedule the representations. Prior research on contextual interference has demonstrated that interleaving different types of learning tasks can foster a deep understanding of the underlying concepts. Do the same advantages apply to interleaving representations? In a classroom experiment, we compared four conditions that varied the practice schedules of multiple graphical representations between interleaving and blocking. The multiple-representation conditions were compared to three single-representation control conditions. During their regular classroom instruction, 290 4th and 5th-grade students worked for five hours with versions of an intelligent tutoring system for fractions. On several dependent measures, interleaving multiple graphical representations led to better learning results than blocking multiple graphical representations. Findings from a think-aloud study give insight into the underlying cognitive processes.

Introduction
Graphical representations of learning contents are often used for instruction (Ainsworth, 2006). When used in learning technology, graphical representations can be especially useful since they allow for interactions that are physically impossible or very difficult to realize, for instance by dragging and dropping symbolic statements into a chart that automatically updates to display the information (Moyer, Bolyard, & Spikell, 2002). However, learning with multiple graphical representations is challenging. An important prerequisite for benefiting from multiple representations is the acquisition of representational fluency: students need to conceptually understand each of the representations, and they need to be able to use them to solve problems (Ainsworth, 2006). Furthermore, students need to develop representational flexibility: they need to understand the differences and similarities between the representations, they need to learn to relate the different representations to one another, and to use the different representations interchangeably to solve problems (Ainsworth, 2006; de Jong et al., 1998). Fractions are one of many areas in mathematics where multiple graphical representations are used extensively (National Mathematics Advisory Panel, 2008). There are different conceptual interpretations of fractions, such as the measurement concept and the part-whole interpretation (Charalambous & Pitta-Pantazi, 2007). Each conceptual interpretation can be illustrated using a different graphical representation, such as number lines for measurement, or area models (e.g., circles or rectangles) to support part-whole interpretations. Multiple graphical representations may help students understand different conceptual aspects of fractions and thus gain a robust understanding of fractions. In a prior study, we found experimental evidence that students working with multiple graphical representation of fractions (e.g., circles, rectangles, and number lines) outperform students who work with a single graphical representation (a number line), although only when prompted to explain how the graphical representations (e.g., half a circle) relate to the symbolic representation (e.g., 1/2) (Rau, Aleven, & Rummel, 2009). These results demonstrate that understanding individual graphical representations (i.e., by relating graphically displayed information to the concepts of numerator and denominator) is essential in order for learners to benefit from multiple graphical representations. When designing instruction that uses multiple graphical representations, curriculum designers must decide how to temporally sequence the different graphical representations. How frequently should the curriculum alternate between graphical representations? Practice schedules are likely to impact how students understand each graphical representation and, consequently, how well they learn the underlying mathematical concepts. In particular, it may matter whether the different representations are practiced in a “blocked” manner (e.g., A – A – B – B) or are interleaved with practice of other representations (e.g., A – B – A – B). Research on contextual interference has investigated scheduling effects of different task types. Results show that interleaving task types leads to better learning results than blocked practice (Battig, 1972; de Croock, van Merrienboer, &

© ISLS

64

ICLS2012

Volume 1: Full Papers

Paas, 1998). A common interpretation of this finding is that interleaved practice encourages deep processing (de Croock et al., 1998). Since students cannot hold all relevant knowledge in working memory, they must reactivate task-specific knowledge as it comes up again in the task sequence. Another interpretation is that when frequently switching between task types, students will be more likely to abstract a common principle from the tasks than when switching infrequently between them (de Croock et al., 1998). They can do so, for instance, by comparing different task types to one another. Against the background of research on blocking versus interleaving different task types, one could hypothesize that interleaving practice with different graphical representations (i.e. switching frequently between them) may lead to deeper processing of conceptual fractions knowledge and may encourage students to abstract a robust conceptual understanding from the multiplicity of graphical representations. These processes may help students acquire representational flexibility. On the other hand, our own previous work (Rau et al., 2009) leads to the hypothesis that developing representational fluency (i.e., coming to understand each single representation) may be a prerequisite for developing representational flexibility. Blocked practice with the representations may be successful in promoting representational fluency because it provides students with the opportunity to become fluent with one graphical representation before starting to work with a second graphical representation. This could be achieved by switching infrequently or with moderate frequency between the representations. If representation flexibility builds on representational fluency, as just argued, then students may benefit most from a condition that gradually moves from blocking to an increasingly interleaved schedule of multiple graphical representations. To test the hypotheses mentioned above, we compared four conditions that varied the practice schedules of multiple graphical representations in a classroom experiment. Students in all conditions worked on the same problems, but multiple graphical representations were presented either in a blocked, moderately interleaved, fully interleaved, or increasingly interleaved manner. We also employed three single graphical representation control conditions in which students worked either with only a circle, a rectangle, or a number line. These control conditions allow us to replicate the results from our first study (which, as mentioned showed advantages for learning with multiple graphical representations over learning with a number line; see Rau, et al., 2009) and to extend this finding to working with only area models. In order to gain insights into the cognitive processes underlying the most successful practice schedule, we additionally conducted a small think-aloud study. We report the results of the think-aloud study after discussing the results from the classroom experiment. We investigated the effects of interleaving multiple graphical representations in the context of a proven intelligent tutoring system technology, namely Cognitive Tutors (Koedinger & Corbett, 2006). Cognitive Tutors have a proven track record in improving students’ mathematics achievement (Koedinger & Corbett, 2006). We developed several versions of an example-tracing tutor for fractions learning, using Cognitive Tutor Authoring Tools (Aleven et al., 2009). This type of tutor behaves like a Cognitive Tutor but relies on examples of correct and incorrect solution paths rather than on a cognitive model. The design of the fractions tutor was informed by results from our previous studies (Rau, Aleven, & Rummel, 2009; 2010), and by small-scale user studies.

Experimental Study
We investigated the effects of blocking versus interleaving multiple graphical representations in a classroom experiment. We expected that interleaving multiple graphical representations of fractions would enhance the acquisition of representational flexibility whereas blocking and moderately interleaving representations would enhance the acquisition of representational fluency. We expected that increasingly interleaving representations would be most successful if representational flexibility builds on representational fluency.

Methods
The tutors used in the study included three interactive graphical representations of fractions: circles, rectangles, and number lines (Figure 1). Our fractions tutor curriculum covered six task types: identifying fractions from multiple graphical representations, making multiple graphical representations of symbolic fractions, reconstructing the unit from unit fractions, reconstructing the unit from proper fractions, identifying improper fractions from multiple graphical representations, and making multiple graphical representations of improper fractions. The tutoring system takes a conceptually-focused approach in introducing fractions. A common theme throughout the fractions tutor was the unit of the fraction (i.e., what the fraction is taken of). The concept of the unit is being introduced in the first task types, and revisited during the later task types as students learn about improper fractions. Figure 2 shows an example of a problem in which students make circle representations for two given symbolic fractions and are then prompted to reflect on the relative size of the two fractions. Students solved each problem by interacting both with symbols and with interactive graphical representations. Students manipulated the graphical representations in various ways: by clicking on fraction pieces to highlight them, by dragging and dropping fraction pieces, and through buttons to change the partitioning of the graphical representations. The tutor interfaces updated interactively after each step to show

© ISLS

65

ICLS2012

Volume 1: Full Papers

the next step to work on, and to emphasize parts of the graphical representations that were conceptually relevant for the subtask at hand through color-highlighting. Students received error feedback and hints on all steps. Error feedback messages were designed to make students reconsider their answer using the multiple graphical representations, or by reminding them of a previously introduced principle. Hint messages provided conceptually oriented help in relation to the graphical representation. Each problem included conceptually oriented prompts to help students relate the multiple graphical representations to the symbolic notation of fractions. We found these prompts to be effective in an earlier experimental study (Rau et al., 2009).

Figure 1. Interactive representations used in fractions tutor: circle, rectangle, and number line.

Figure 2. Making a circle given a symbolic fraction, combined with prompts to compare the two fractions. Reflection prompts are implemented with drop-down menus shown in the bottom half of each problem. A total of 587 4th- and 5th-grade students from six different schools (31 classes) participated in the study during their regular mathematics instruction. We excluded students who missed at least one test day, and who completed less than 67% of all tutor problems (to ensure that students in the multiple graphical representations conditions encountered all three graphical representations). This results in a total of N = 290 (n = 63 in blocked, n = 53 in moderate, n = 52 in fully interleaved, n = 62 in increased, n = 21 in single-circle, n = 20 in single-rectangle, n = 19 in single-number-line). Prior to working on the fractions tutor, students completed a pretest. The pretest took about 30 minutes. On the following day, all students started working with the fractions tutor. Students accessed the tutoring system from the computer lab at their schools and worked on the tutor for fractions for about five hours as part of their regular math instruction for five to six consecutive school days (depending on the length of the respective school’s class periods). All students worked on the fractions tutor at their own pace, but the time students spent with the system was held constant across classrooms and across experimental conditions. On the day following the tutoring sessions, students took the immediate posttest which took about 30 minutes to complete. Seven days after the posttest, students completed an equivalent delayed posttest. Figure 3 illustrates the practice schedules of task types and graphical representations for the four multiple graphical representations conditions. In all conditions, students worked through the same sequence of task types and fraction problems, and switched task types after every 9 of a total of 108 problems. Each task type was revisited three times. This procedure corresponds to the most successful level of interleaving task types in our prior experiment (Rau et al., 2010). We randomly assigned students to one of seven conditions. In the blocked condition, students switched graphical representations after 36 problems. In the moderate condition,
© ISLS 66

ICLS2012

Volume 1: Full Papers

students switched representations after every six problems. In the fully interleaved condition, students switched representations after each problem. In the increased condition, the length of the blocks was gradually reduced from twelve problems at the beginning to a single problem at the end. To account for possible effects of the order of graphical representations, we randomized the order in which students encountered the graphical representations. Finally, students in the three single graphical representation conditions worked on all tutor problems with only the circle, the rectangle, or the number line, respectively.

Figure 3: Practice schedule for multiple graphical representations conditions for all six task types. Each task type was revisited three times. Task types are indicated by numbers 1-6, representations are indicated by the different shapes. We assessed students’ knowledge of fractions at three test times. Three equivalent test forms were created, and we randomized the order in which they were administered. The tests included four knowledge types: fluency with area models (i.e., circles and rectangles), fluency with the number line, conceptual transfer and procedural transfer. The fluency items included identifying fractions given a graphical representation, making a graphical representation given a symbolic fraction, and recreating the unit given a graphical representation of both unit fractions and proper fractions. Conceptual transfer items included proportional reasoning questions with and without graphical representations. Procedural transfer items included comparison questions with and without graphical representations. The theoretical structure of the test (i.e., the four knowledge types just mentioned) resulted from a factor analysis performed on the pretest data. Test items including the number line seemed to be more challenging for students than area models.

Analysis
As mentioned, we analyzed the data of N = 290 students. There was no significant difference between conditions with respect to the number of students excluded (χ² < 1). There were no significant differences between conditions at pretest for any dependent measure, ps > .10. There was no significant effect for order of multiple graphical representations for any dependent measure, F(5, 285) = 1.56, ps > .10. We used a hierarchical linear model (HLM, see Raudenbush & Bryk, 2002) with four nested levels to analyze the data. We modeled performance on the tests for each student (level 1), differences between students

© ISLS

67

ICLS2012

Volume 1: Full Papers

nested within classes (level 2), differences between classes nested within schools (level 3), and differences between schools (level 4). More specifically, the following HLM model was fitted to the data: scoreij = testj + conditioni + testj*conditioni + preScorei + preScorei*conditioni + numProblemsi + 1) student(class)i+ class(school)i + schooli, with the dependent variable scoreij being studenti’s score on the dependent measures at testj (i.e., immediate or delayed posttest). In order to analyze whether students with different levels of prior knowledge benefit differently from our conditions, we included students’ pretest scores as a covariate (preScorei), and modeled the interaction of pretest score with condition (preScorei*conditioni). Since the HLM described in (1) uses students’ pretest scores as a covariate, it does not allow us to analyze whether students in the various conditions improved from pretest to immediate and delayed posttest. To analyze learning gains, we included pretest score in the dependent variable, yielding: scoreij = testj + conditioni + testj*conditioni + numProblemsi + student(class)i+ class(school)i + 2) schooli, with the dependent variable scoreij being studenti’s score on the dependent measures at testj (i.e., pretest, immediate posttest, or delayed posttest). We used planned contrasts and post-hoc comparisons to clarify results from the HLM analysis. All reported p-values were adjusted using the Bonferroni correction for multiple comparisons. Table 1: Improvement of test scores at immediate posttest (post) over pretest (pre) and delayed posttest (delayed) over pretest by knowledge types and conditions. Condition
blocked moderately interleaved fully interleaved increasingly interleaved singlecircle singlerectangle singlenumber-line

Effect
post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre post > pre delayed > pre

Fluency with area models n.s. p < .05, d = .52 n.s. n.s. p < .05, d = .45 p < .05, d = .38 p < .05, d = .38 p < .05, d = .55 n.s. n.s. n.s. n.s. n.s. n.s.

Fluency with the number line n.s. p < .01, d = .39 n.s. p < .01, d = .50 p < .01, d = .51 p < .01, d = .75 p < .01, d = .43 p < .01, d = .46 n.s. n.s. n.s. n.s. n.s. n.s.

Conceptual transfer p < .05, d = .42 p < .05, d = .39 p < .05, d = .29 p < .05, d = .30 p < .01, d = .34 p < .01, d = .60 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.

Procedural transfer n.s. n.s. n.s. p < .05, d = .45 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.

Table 2: Differences between conditions at immediate posttest (post) and delayed posttest (delayed) by type of knowledge. “n.s.” indicates non-significant results. “-“ indicates that no post-hoc comparisons were computed.
Effect fully interleaved > blocked, moderately interleaved, increasingly interleaved increasingly interleaved > blocked, moderately interleaved, fully interleaved moderately interleaved > blocked, fully interleaved, increasingly interleaved Test post delayed post delayed post delayed Fluency with area models p < .10, d = .30 p < .10, d = .30 Fluency with number line n.s. n.s. Conceptual transfer n.s. p < .05, d = .33 Procedural transfer n.s. n.s.

Results
To investigate whether students learned from the fractions tutor, we analyzed learning gains using the simpler HLM described in formula (2). The main effect of test time was significant for fluency with the number line, F(2, 867) = 20.09, p < .01, partial η² = .03, for fluency with area models, F(2, 867) = 17.54, p < .01, η² = .02, conceptual transfer, F(2, 867) = 38.78, p < .01, partial η² = .03, and marginally significant for procedural transfer, F(2, 867) = 2.84, p < .10, partial η² = .01. The interaction between test time and condition was significant for fluency with area models F(12, 862) = 2.06, p < .05, partial η² = .01. These results show that students (regardless of condition) improved on fluency with the number line, area models, procedural and conceptual transfer. On fluency with area models, students’ learning gains also depended on the condition. To further clarify these results, we computed post-hoc comparisons that compared students’ scores at the immediate posttest and the delayed posttest compared to the pretest, respectively. Table 1 provides a

© ISLS

68

ICLS2012

Volume 1: Full Papers

summary of these post-hoc comparisons. Generally, we found significant learning gains at the delayed posttest for most of the multiple graphical representations conditions on fluency with area models, fluency with the number line, and conceptual transfer. On procedural transfer, only the moderate condition showed significant learning gains at the delayed posttest. Finally, we found no significant learning gains for the single graphical representation conditions except for the single-circle condition at the delayed posttest on conceptual transfer. To analyze the effect of practice schedules of multiple graphical representations, we computed the HLM presented in formula (1) for only the multiple graphical representations conditions. There was no significant main effect of condition on any posttest scale, indicating that there was no global effect of practice schedules of multiple graphical representations across immediate and delayed posttests. An interaction between test time and condition was marginally significant for fluency with area models, F(3, 867) = 2.57, p < .10, η² = .01, indicating that the effect of practice schedules depends on test time. The interaction between pretest score and condition was marginally significant for conceptual transfer, F(3, 219) = 2.52, p < .10, η² = .02, demonstrating that students with different pretest scores benefit from different practice schedules. To clarify the interaction between test time and condition, we used post-hoc comparisons separately for the immediate and the delayed posttest. To limit the number of comparisons, we only compared the most successful multiple graphical representations condition against the other three multiple graphical representations conditions, as summarized in Table 2. We found some support for a benefit of interleaving multiple graphical representations: the fully interleaved condition significantly outperformed the blocked, the moderately interleaved, and the increasingly interleaved conditions on conceptual transfer at the delayed posttest. Furthermore, we found a marginally significant advantage for the increasingly interleaved condition over the blocked, moderately interleaved, and fully interleaved conditions on fluency with the number line at the immediate and the delayed posttests. To clarify the interaction between pretest score and condition on proportional reasoning items, we computed post-hoc comparisons for students with extremely low or high pretest scores. For students with a pretest score of 15%, 20%, and 25%, we found a significant advantage for the interleaved over the blocked condition (ps < .05). We found no differences for high prior knowledge students. To analyze the difference between multiple graphical representations conditions and the single graphical representation conditions, we computed the HLM described in formula (1). We used planned contrasts to compare the multiple graphical representations conditions to the single graphical representation conditions. We found a significant advantage for the multiple graphical representations conditions over the single graphical representation conditions for number line test items at delayed posttest (p < .05, d =.29).

Think-aloud study
In order to gain further insight into the cognitive processes underlying the benefits of the interleaved practice schedules, we conducted a small-scale think-aloud study with six students who worked on the fully interleaved version of the tutoring system. As argued above, one hypothesized mechanism is that interleaved practice leads students to abstract across multiple graphical representations, for instance by comparing them to one another. Alternatively, it has been hypothesized that interleaved practice leads to the reactivation of representationspecific knowledge. Specifically, we were interested in what kinds of spontaneous comparisons students are making between the graphical representations, at what points in the curriculum they make comparisons, and whether students who fail to make spontaneous comparisons can be prompted to do so.

Methods
Six 5th-grade students participated in the think-aloud study. The think-aloud study was conducted in our laboratory and included three sessions. During the first session, students took the same pretest that was used in our experimental study. The pretest took about 30 minutes to complete. During the second session, students worked for one hour on a subset of problems taken from the interleaved version of the tutoring system while being prompted to think aloud, following the procedure described in Ericsson and Simon (1984). In the third session, students worked with similar tutor problems for one hour while being prompted to relate the different graphical representations to one another. We varied the type of prompts based on a within-subjects design: the prompt questions were either implicit (i.e., without directly prompting comparisons between the representations; e.g. “How is this problem the same as the last two you did?” or “How is this problem different from the last one you did?”), or explicit (i.e., directly referring to aspects that the different representations share; e.g., “What is the unit in the circle / rectangle / number line?” or “How are the rectangle and the circle and the number line the same / different?”). All students received two implicit prompts and four explicit prompts, in a fixed sequence. Students’ utterances were recorded and transcribed. We combined top-down and bottom-up approaches in developing a coding scheme: the experimenters identified types of comparisons that students might make prior to the think-aloud study, and then refined the coding scheme after viewing the transcripts from the thinkaloud study. Comparisons between graphical representations were coded as surface comparisons if they either referred to the color of the representation, the shape of the representation, or the action performed on the

© ISLS

69

ICLS2012

Volume 1: Full Papers

representation (e.g., dragging and dropping). Comparisons were coded as conceptual if they referred to the corresponding features of the representations (i.e., numerator, denominator, unit), or the magnitude represented.

Results and Discussion
The results from the pretest indicate that all students had a good understanding of fractions. During the spontaneous comparison phase of the think-aloud study, we found only five instances of comparisons. These five comparisons were uttered by five of the six students. All five comparisons were surface comparisons. When prompted to compare the different representations, students generated 138 instances of comparisons overall. Table 3 summarizes the average number of comparisons coded as surface and conceptual comparisons per implicit and explicit prompt. Given the small number of students, a statistical test on the types of comparisons in response to implicit and explicit prompts is not warranted. Table 3 shows that students generated substantially more surface than conceptual comparisons per prompt. We can also see that the implicit prompts yielded most of the surface comparisons, but almost none of the conceptual comparisons. In contrast, explicit prompts seem to have yielded more of the conceptual comparisons and fewer of the surface comparisons, compared to the implicit prompts. Table 3: Average number of surface and conceptual comparisons per implicit and explicit prompts. Implicit prompts 4.17 0.58 2.38 Explicit prompts 2.33 1.63 1.98

Surface Conceptual

2.94 1.28

Conclusions
Taken together, the results from the experiment demonstrate significant learning gains for students who worked with a tutoring system that supports learning with multiple graphical representations of fractions, but not for those students who worked with only a single graphical representation. The gains persist until one week after the study when we administered the delayed posttest. Learning gains were found for students in all multiple graphical representations conditions on all posttest scales except procedural transfer. The fact that students’ performance on procedural transfer does not improve may be due to the fact that comparing fractions was not the focus of the tutor, and that the comparison tasks in the procedural transfer items were too difficult for students. We did not find evidence for learning in the single representation groups. The lack of learning gains in the single representation control conditions demonstrates that the learning gains in the other conditions are not due to practice effects with the test format. Furthermore, the finding that the multiple graphical representations conditions outperform the single graphical representation conditions on number line items at the delayed posttest suggests an advantage of learning with multiple graphical representations over learning with a single graphical representation. Taken together with the lack of learning gains in the single representation conditions, these results replicate our earlier finding that multiple graphical representations lead to better learning of fractions than a single graphical representation (Rau et al., 2009), and extend this finding from working with a number line only to working with circles or rectangles only. We argued that interleaving multiple graphical representations may help students acquire representational flexibility, whereas blocking graphical representations will help students acquire fluency with each representation. Our results confirm that the practice schedule for learning with multiple graphical representations matters: interleaving multiple graphical representations enhances the acquisition of representational flexibility as assessed by the conceptual transfer scale of the test – in particular for students with low prior knowledge. On the other hand, we did not find differences between experimental conditions for representational fluency (i.e., on area model and number line items) – with the exception of a marginally significant advantage of the increasingly interleaved condition on fluency with area models. Our results therefore suggest that students in all multiple representations conditions were able to acquire representational fluency from the tutoring system; and that – contrary to our hypothesis – blocking graphical representations does not contribute to the development of representational fluency. We hypothesized that interleaving graphical representations will promote the acquisition of representational flexibility. The significant advantage of the fully interleaved condition on conceptual transfer supports this hypothesis. Reasoning that representational flexibility may build on representational fluency, we had hypothesized that increasingly interleaving representations is the best option. Our findings do not support this hypothesis. Rather, the finding that especially low prior knowledge students benefit from fully interleaved representations may suggest that developing representational flexibility is particularly important for novice learners and that, perhaps, representational flexibility is a prerequisite for the development of representational fluency. Although more research is needed to investigate whether our findings generalize to other domains, based on our findings we can carefully conclude that designers of intelligent

© ISLS

70

ICLS2012

Volume 1: Full Papers

tutoring systems should employ an interleaved practice schedule of multiple graphical representations in order to enhance robust conceptual understanding of fractions. What might be the mechanisms leading to the advantage of interleaving multiple graphical representations over blocking multiple graphical representations? Are students actively abstracting across graphical representations by explicitly comparing them to one another? Or are they, as argued above, reactivating knowledge that is specific to the graphical representations? Findings from our think-aloud study with the fully interleaved version of the tutoring system suggest that students did not spontaneously relate the different graphical representations to one another. The benefit from interleaving multiple graphical representations does not seem to stem from conscious abstraction across the different representations, but from more subliminal processes, such as repeatedly reactivating knowledge about the specific representations. On the other hand, the finding that students generate a good number of conceptual comparisons between the graphical representations when explicitly prompted to do so, suggests that students might benefit from explicit support in relating the different representations to one another. Although these considerations are based on the findings from only a small-scale think-aloud study, we believe that they provide interesting insights into the processes that may underlie the benefits of interleaving multiple graphical representations. Future work should investigate whether indeed the advantage of interleaving representations results from repeated reactivation of representational knowledge. Furthermore, future work should explore the benefit of explicitly supporting students in conceptually relating multiple graphical representations and in making sense of their differences and similarities in how they depict fractions.

References
Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183-198. Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009). Example-tracing tutors: A new paradigm for intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 19(2), 105154. Battig, W. F. (1972). Intratask interference as a source of facilitation in transfer and retention. In R. F. Thompson & J. F. Vos (Eds.), Topics in learning and performance (pp. 131-159). New York: Academic Press. Charalambous, C. Y., & Pitta-Pantazi, D. (2007). Drawing on a Theoretical Model to Study Students' Understandings of Fractions. Educational Studies in Mathematics, 64(3), 293-316. de Croock, M. B. M., Van Merrienboer, J. J. G., & Paas, F. (1998). High versus low contextual interference in simulation-based training of troubleshooting skills: Effects on transfer performance and invested mental effort. Computers in Human Behavior, 14(2), 249-267. de Jong, T., Ainsworth, S. E., Dobson, M., Van der Meij, J., Levonen, J., Reimann, P., et al. (1998). Acquiring knowledge in science and mathematics: The use of multiple representations in technology-based learning environments. In M. W. Van Someren, W. Reimers, H. P. A. Boshuizen & T. de Jong (Eds.), Learning with Multiple Representations. Oxford. Ericsson, & Simon. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press. Koedinger, K., & Corbett, A. (2006). Cognitive Tutors: Technology bringing learning science to the classroom. In K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 61-78): Cambridge University Press. Moyer, P., Bolyard, J., & Spikell, M. A. (2002). What are virtual manipulatives? Teaching children mathematics, 8, 372-377. National Mathematics Advisory Panel (2008). Foundations for Success: Report of the National Mathematics Advisory Board Panel: U.S. Government Printing Office. Rau, M. A., Aleven, V., & Rummel, N. (2009). Intelligent tutoring systems with multiple representations and self-explanation prompts support learning of fractions. In Dimibrova, V., Mizoguchi, R., du Boulay, B. (Eds.). 14th International Conference on Artificial Intelligence in Education, pp. 441-448. IOS Press, Amsterdam. Rau, M. A., Aleven, V., & Rummel, N. (2010). Blocked versus Interleaved Practice with Multiple Representations in an Intelligent Tutoring System for Fractions In Aleven, V., Kay, J., Mostow, J. (Eds.) 10th International Conference of Intelligent Tutoring Systems, pp. 412-422, Springer, Heidelberg. Raudenbush, S.W., Bryk, A.S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods. Sage Publications, Newbury Park.

Acknowledgments
This work was supported by the National Science Foundation, REESE-21851-1-1121307. We thank Ken Koedinger, Mitchell Nathan, Kathy Cramer, Peg Smith, Jay Raspat, Michael Ringenberg, the Datashop and CTAT teams, Brian Junker, Howard Seltman, Cassandra Studer, the students, teachers, and school principals.
© ISLS 71

ICLS2012

Volume 1: Full Papers

Investigating the Relative Difficulty of Complex Systems Ideas in Biology
Sao-Ee Goha, Susan A. Yoona, Joyce Wanga, Zhitong Yanga, Eric Klopferb Graduate School of Education, University of Pennsylvania, 3700 Walnut Street, Philadelphia, PA 19104 goh@dolphin.upenn.edu, yoonsa@gse.upenn.edu, joyce.s.wang@gmail.com, zhitongy@hotmail.com b Scheller MIT Teacher Education Program, 77 Mass. Ave., MIT Building 10-337, Cambridge, MA 02139 klopfer@mit.edu
a

Abstract: A number of students’ biology misconceptions can be attributed to their lack of understanding in complex systems. Comprehending complex systems can be counterintuitive and difficult. A literature review reveals that we have yet to systematically determine a learning approach to address these learning challenges. In this paper, we propose that learning progressions research offers a methodical approach to organize the learning pathways students take to improve conceptual competence in complex systems. As a first step, we articulate a sequence of complex systems ideas - from the least to most difficult - by analyzing students’ written responses. Using an Item Response Theory model, we found that the easiest ideas to grasp are those that relate to the interconnected nature of these systems whereas the most difficult ideas are those concerning the decentralized organization of the system, and the predictability of the system effects.

Introduction
Studies in science education have revealed robust misconceptions about concepts and processes in school biology that directly impact students’ abilities to understand scientific topics. Misconceptions have been found across scales, from atoms (Taber & Garcia-Franco, 2010), to cells and genes (Garvin-Doxas & Klymkowsky, 2008), to organisms and ecology (Gotwals & Songer, 2010). Difficulties in understanding the relationships between these various scales have also been documented (Sewell, 2002). Researchers have speculated that these problems may exist due to a lack of understanding of the complex systems realms in which these scientific phenomena reside and interact (Charles & d’Apollonia, 2004; Chi, 2005). Thus, educational agencies in the US have urged science curriculum and instruction to emphasize systems content (AAAS, 2009; NRC, 2011). A complex system can be defined as an organization of interconnected components that as a whole, exhibits patterns and properties not obvious from those of the individual components (Mitchell, 2009). The way in which herds of ungulates roam across the savannah provides one of the most vivid examples of how individual actions and interactions lead to large scale patterns. To understand complex systems is to be able to reason that the individual components interact with one another in multiple and nonlinear ways, and recognize that the patterns observed at the system level emerge from these interactions at the component level (Sweeney & Sterman, 2007; Yoon, 2008). Research has documented that grasping complex systems ideas can be counterintuitive and difficult (Chi, 2005; Jacobson & Wilensky, 2006). While there have been several studies that promote the learning of particular complex systems or specific complexity ideas (e.g., Klopfer et al., 2009; Levy & Wilensky, 2009), the literature also suggests that we have yet to systematically determine a learning approach to address the challenges associated with a comprehensive understanding of the various ideas (HmeloSilver & Azevedo, 2006). We believe that the learning progressions methodology offers one such systematic approach in structuring the learning of various complex systems ideas. Learning progressions are defined as sequences of ordered descriptions that illustrate the learning pathways students can take to improve conceptual competence in science (Alonso & Steedle, 2006; CPRE, 2009). For example, Mohan and her colleagues (2009) identified levels of increasing sophistication in students’ perception of carbon-transforming events in complex socioecological systems. These ordered descriptions represent a research-informed framework for structuring the learning of core scientific ideas (NRC, 2007). Curriculum and instructional activities can in turn be mapped onto the learning progressions so as to better facilitate the learning of the core ideas (Gotwals & Songer, 2010; Songer et al., 2009). Our study extends the research on learning progressions by focusing on complex systems ideas grounded in high school biology content. Our broad goal is to eventually design valid curricular and instructional activities based on a learning progression of complex systems ideas. As a first step, in this paper we report on an exploratory work that investigates a conceptual sequence of complex systems ideas, from the least to the most difficult, by analyzing a diverse group of students’ written responses on the effects of geese on a park ecosystem.

Theoretical Considerations
© ISLS 72

ICLS2012

Volume 1: Full Papers

Our research lies at the intersection of two fields – understanding complex systems and learning progressions. We first discuss known challenges in understanding complex systems, before illustrating how learning progressions as a methodology can help address these challenges.

Understanding complex systems
As previously mentioned students typically possess misconceptions about complex systems (Ben-Zvi Assaraf & Orion, 2005; Hmelo, Holton, & Kolodner, 2000; Jacobson & Wilensky, 2006). For example, students tend to adopt a linear approach when thinking about the relationships among system components (Gotwals & Songer, 2010; Hogan, 2000; Riess & Mischo, 2010; Sweeney & Sterman, 2007). That is, they perceive single cause-andeffect relationships, where small actions lead to small effects. However, a complex system with its multiple connections among the components can often result in an action that gives rise to widespread effects throughout the system. Students also generally conceive of systems being controlled by certain components, and systemic patterns as designed with specific functions in mind (Penner, 2000; Resnick, 1996; Taber & Garcia-Franco, 2010). In other words, students are unable to interpret the decentralized nature and emergent patterns of complex systems. Wilensky and Resnick (1999) argue that this attribution of centralized control and intentionality to system processes is due to students’ confusion about the various levels of representation (e.g., microscopic and macroscopic) of complex systems. Despite the valuable information these studies provide, they examine single aspects of complex systems understanding. Given the current overall systems thrust in science education (AAAS, 2009; NRC, 2011) it is prudent to turn our attention to developing a more comprehensive view of complex systems content. Some studies have already articulated analytical frameworks that encompass multiple complex systems ideas and beliefs (Hmelo-Silver et al., 2004; 2007; Jacobson, 2001; 2011). For example Jacobson and his colleagues (2001) developed a framework that delineates distinct categories of complex systems ideas. Using this framework in an expert-novice study, he discovered that pre-college students tended to believe that systems operate in reductive, centralized and predictable way, whereas, science experts described phenomena as nonreductive, decentralized, and non-linear. We adapt this framework and evolutions of it (Jacobson, 2011; Yoon, 2008; 2011) to analyze levels of understanding for various ideas of complex systems. In doing so, we unpack and reveal the relative levels of difficulties that can eventually inform the development of a learning progression.

Learning Progressions
Learning progressions provide a methodical approach to organize curriculum and instructional activities to learn and teach about complex systems ideas, as they illustrate the cognitive pathways and skills students are likely to follow over a period of time in mastering the scientific concepts (Alonso & Steedle, 2006; NRC, 2007). Learning progressions structure the sequences based on what cognitive researchers know about science learning from empirical research findings (CPRE, 2009). To date, there have been three published works on learning progressions on biology topics related to complex systems concepts (Gotwals & Songer, 2010; Mohan et al., 2009; Songer et al., 2009). These studies have collectively ascertained that students’ learning of biological systems, or the development in the way they think about systems, follows some trajectory. However, these progressions address particular science content such as ecosystems and biodiversity rather than a more generalized complexity perspective. Although they have not characterized their work in terms of learning progressions, Ben-Zvi Assaraf and Orion (2005; 2010a; 2010b) examine students’ development of systems thinking abilities, and construct a hierarchical model of the stages by which students’ reason about complex systems. They explain that at the most rudimentary level students are able to identify the components of a system and processes within the system. At the intermediate level, students were able to identify relationships between system components, and organize the systems’ components, processes, and their interactions within a framework of relationships. At the most sophisticated level students can perceive hidden system components, explain the history of current patterns, and make predictions on the evolution of the system. However, their model comprises only a subset of essential complex systems components as identified in Jacobson (2001). A sequence or markers for learning about other equally salient ideas such as decentralization and emergence are also necessary to ensure that students acquire an adequate overall understanding of complex systems. As an initial development of a learning progression on complex systems ideas, a comprehensive set of ideas is first differentiated in order of difficulty levels. Such delineation can help organize the sequence of complexity ideas to be learned. There are generally two broad approaches in constructing the initial learning sequences (CPRE, 2009). The first approach begins by conjecturing a possible sequence from existing literature and then validating it. For example, Songer and her team (2009) first hypothesized their learning progression on biodiversity from literature reviews, and then tested the validity of the progression by comparing student learning outcomes in a control-treatment study. The second approach starts by analyzing a cross-sectional sample of students’ responses to provide indicators of their understanding and then derives the levels of
© ISLS 73

ICLS2012

Volume 1: Full Papers

sophistication from the analysis. For instance, Mohan and her colleagues (2009) analyzed almost 300 elementary through high school students’ written accounts of the biogeochemical carbon cycling processes in socio-ecological systems for distinct levels of sophistication, and constructed their progression based on these levels. The latter approach presents an advantage when existing literature does not inform much about which ideas should be learned before others. In our exploratory study, we adopt this more ‘grounds-up’ approach in drawing up the initial sequence of our learning progression.

Methods
Context and Participants
This study is part of a larger National Science Foundation-funded research project in which we design and develop curriculum and instructional activities using computational modeling tools in four high school biology units – Chemistry of Life, Population Ecology, Community Ecology, and Evolution that promote student learning of complex systems. While the activities are being constructed and tested, we focus concurrently on building the learning progression that informs them. To begin, we developed eight open-ended short-answer questions, two for each biology unit. These were selected or derived from the OECD Programme for International Student Assessment (PISA, 2006) international science test and Ph.D. level biology content experts. We then administered this test to Grades 8 to 12 students in order to determine the range of conceptual difficulties. To recruit students to participate in our study, we enlisted the support of science teachers with whom we had previously worked in another study. We targeted different kinds of schools (including 3 magnet and 4 public) to account for differences in science learning experience. All teachers and students were recruited from a large urban school district in the northeast part of the US. Two days prior to the implementation, a researcher went to the teachers’ classes and gave a short presentation regarding the goals of the study and the roles of the students (should they participate). Those who participated were self-selected and were offered a $10 gift card in exchange for their participation. Their involvement lasted about one hour after school in which they answered questions on the open-ended biology test. The students were given as much time as they required to answer the questions. In total 44 students of various ethnicities from 7 urban schools participated. There were 20 males and 24 females in grade 8 (5), grade 9 (13), grade 10 (14), and grade 12 (12). The recruitment and data collection occurred over 3 months.

Data Sources, Data Coding and Analyses
For this paper, only the responses to one question were analyzed: “Imagine a flock of geese arriving in a park in [your city], where geese haven’t lived before. Describe how the addition of these geese to the park affects the ecosystem over time. Consider both the living and non-living parts of the ecosystem.” This question, written by an expert in biology, sought to solicit students’ understanding of biology and complex systems in an ecological context. A content analysis of the students’ responses was performed using six categories of complex systems understanding - Agent Effects, Action Effects, Networked Interactions, Multiple Causes, Order, and Processes derived from Jacobson’s (2001; 2011) and Yoon’s (2008; 2011) studies. Table 1 provides an abridged description of each category. To account for variation in students’ understanding of the complexity ideas within each category, each response was coded six times - once for each category - for four levels of increasing sophistication: Completely Clockwork (Level 1), Somewhat Clockwork (Level 2), Somewhat Complex (Level 3), and Completely Complex (Level 4). Clockwork responses encompassed those that showed linear, deterministic, isolated, centralized, single-cause, and predictable system interactions or states, whereas complex responses included those that demonstrated non-linear, non-deterministic, networked, decentralized, multiplecauses, and random system interactions or states. After the coding manual was constructed and vetted, its reliability was assessed with two independent doctoral student raters coding 20% of the written responses. An inter-rater agreement of 0.8 was achieved collectively across categories using the Cronbach alpha reliability test. The remainder of the responses was subsequently coded by two of the authors using the coding scheme, with any discrepancies discussed. Below we provide an example of a response that achieved mostly Levels 3 or 4 for all categories of complexity ideas to show how the coding was done: Well if geese arrived it would probably help the ecosystem. The bird droppings might make the soil fertile. It would start to look a lot greener. The problem with that is erosion. The increase of plants and root size might cause paths or walkways to be damaged or destroyed. Statues might start to fall apart from the constant weight of birds. Plus the increase of plants in amount and size make O2 levels higher. Which could cause a warm and wet ecosystem, much similar to a swamp. Over a long period of time of course. (Student response, March 2011)

© ISLS

74

ICLS2012

Volume 1: Full Papers

Table 1: Complex Systems Category Code Descriptions
Category Agent Effects The emphasis is the predictability of the effects caused by the part in question. Completely Clockwork (Level 1) Response shows that the way in which a part operates or affects other parts is completely predictable. No alternative is offered in the response. Somewhat Clockwork (Level 2) Response shows that the way in which a part operates or affects other parts is somewhat predictable. There are 1-2 possibilities suggested in the response. Response contains one complex component (out of three) of action effects. (See Level 4.) Somewhat Complex (Level 3) Response shows that the way in which a part operates or affects other parts is somewhat unpredictable. There are 3-4 possibilities suggested in the response. Response contains two complex components (out of three) of action effects. (See Level 4.) Completely Complex (Level 4) Response shows that the way in which a part operates or affects other parts is completely unpredictable. There are many possibilities suggested in the response. Response indicates (i) small actions can lead to large effects; (ii) the action can produce both localized changes and cascading effects; and (iii) the changes can take place both immediately and over a long period of time. Response attributes the outcome(s) of an event to four or more causes/factors.

Action Effects Three components are considered: (i) the relative scale of outcomes caused by action; (ii) the cascading effects of the action; and (iii) the time scale at which changes happen. Multiple Causes The focus is on the number of causes that may/will attribute to the outcome(s) of an event. Networked Interactions Three components are assessed: (i) interdependency among parts in the system; (ii) nonlinearity in reasoning; and (iii) emergent patterns over scale. Order The focus is the organization of the system or phenomenon – centralized or decentralized.

Response indicates (i) small actions only lead to small effects; (ii) there is a sense that the action causes localized changes only; and (iii) the changes are immediate and do not sustain for a long time.

Response attributes the outcome(s) of an event to one cause/factor.

Response attributes the outcome(s) of an event to two causes/factors.

Response attributes the outcome(s) of an event to three causes/factors.

Response indicates that (i) the parts of a system are isolated with no interdependency among them; (ii) the interactions between parts are linear with no feedback; and (iii) the patterns at the system level are the same from those at the component level. Response indicates that the system is controlled by one central agent, that is, all action is dictated by a leader. Order in the system is established ‘top-down’ or determined with a specific purpose in mind. Response indicates that the system is composed of static events. While perturbations in the system cause change to occur, the change terminates once an outcome is achieved (i.e., a definite end).

Response contains one complex component (out of three) of networked interactions. (See Level 4.)

Response contains two complex components (out of three) of networked interactions. (See Level 4.)

Response indicates that (i) the parts are interdependent; (ii) the interactions between parts are non-linear with feedback; and (iii) the patterns at the system level are emergent.

Processes Processes refer to the dynamism of the mechanisms that underlie the phenomena; in other words, how the system works or is thought to work.

Response indicates that the system is largely controlled by 2-3 central agents, i.e., there are other parts that may dictate how the system behaves. Order in the system is established ‘top-down.’ Response indicates that the system is somewhat composed of static events with suggestions that these events take time to reach the outcome(s).

Response indicates that the system is largely decentralized and the control lies with 4-5 components. However, there is little evidence to show that the order in the system is selforganized. Response indicates that the system is somewhat of an ongoing process. Perturbations take a long time to reach the final outcomes, which are at larger scale than the initial event(s).

Response indicates that the system is decentralized and control lies with a myriad (more than 5) of parts. Order in the system is self-organized or ‘bottom-up’, and emerges spontaneously. Response indicates that the system is an ongoing, dynamic process. System continues to be in a state of flux. The parts adapt or evolve, and continue to do so accordingly.

In this exemplar response, the student repeatedly used non-deterministic words, such as “might” and “could,” and suggested three possibilities of the effects due to the geese’s arrival, which indicates his uncertainty in the effects of the geese’s arrival (Level 3 Agent Effects). There are suggestions that the geese’s
© ISLS 75

ICLS2012

Volume 1: Full Papers

arrival can cause cascading effects to the soil, the plant population, the oxygen levels, may lead to large-scale and long-term effects such as causing a “warm and wet ecosystem” (Level 4 Action Effects). More than four factors (e.g., droppings, soil, erosion, plants, oxygen) have been identified as attributing to possible outcomes (Level 4 Multiple Causes). The response also hints of an understanding of the interdependence of the various components in the park ecosystem, the feedback mechanisms present in the system (i.e., paths or walkways damaged by more plants), and the possible emergence of a systemic pattern (i.e., warm and wet environment) (Level 4 Networked Interactions). In addition, the idea of decentralization is clearly demonstrated as four actors (i.e., geese, plants, soil, and oxygen) are said to be involved (Level 3 Order). Furthermore, the response implies that the perturbations in the park ecosystem are somewhat on-going, and may take a long time to arrive at a final state (Level 3 Processes). As we are interested in determining the differences in conceptual difficulties among the six categories of complex systems ideas, and given that we have employed successively ordered rating scales (i.e., 4 levels of understanding), the Generalized Partial Credit Model (GPCM: Muraki, 1992), which is an item response theory model, was deemed appropriate for the analysis of the responses. The items, or categories of ideas in our case, are conceptualized as a series of hierarchical levels of performances where respondent receives partial credit for successfully performing at a particular level. Since our coding included four levels of understanding for six categories, the GPCM was an appropriate means to provide the information that we wanted. Students’ raw scores were first standardized on a continuum between -3 to +3 to reveal the difficulty level of each category. On this continuum of logit scale, 0 is set as the mean of the item difficulty parameter. On the positive direction toward +3, each increase indicates that the item is becoming more difficult; conversely, on the negative direction toward -3, each decrease signals that the item is turning less difficult. In addition, the model is able to show us how well each item can distinguish students with different abilities by the discrimination parameter. The discrimination parameter typically ranges between 0.5 and 2.5 in value. The larger this parameter, the more effective the item can distinguish students with varying levels of understanding.

Results
We rated the written responses for their understanding in each category of complex systems ideas. A breakdown of how they scored is given in Table 2. A simple frequency count shows most of the responses were at a level 2 (somewhat clockwork) or level 3 (somewhat complex) understanding of various complex systems ideas, with very few showing a level 4 (completely complex) understanding. We expected this because as described earlier, the literature indicated that most students face difficulty in grasping complex systems ideas. Table 2: Breakdown of scoring for each category of complex systems ideas
Category Agent Effects Action Effects Multiple Causes Networked Interactions Processes Order Total: Level 1 16 8 11 6 11 15 67 Level 2 14 11 19 12 23 16 95 Level 3 12 14 9 22 7 12 76 Level 4 2 11 5 4 3 1 26

We then ran the GPCM on the 44 sets of ratings, using PARSCLE 4.1. The model converged at critical value = 0.005, and had no items fit statistical significance, indicating a good model fit. The mean difficulty of the six items or categories is 0.49 (SD = 0.45), and mean discrimination parameter was 1.10 (SD =0.39). The categories were found reliable to measure students’ understanding of complex system (composite reliability = 0.87). The total test information peaked between 0 and 1, indicating the categories could measure students with slightly higher than average ability with most precision. Table 3 presents the six categories, their difficulty parameters, and their discrimination parameters. As indicated by the difficulty parameters, Agent Effects, Order, and Processes categories of ideas were found to be the most difficult among these students; Multiple Causes was found to be at the intermediate level; and Networked Interaction and Action Effects categories were the easiest. All six categories can also distinguish students based on their abilities, as their discrimination parameters ranged from 0.69 to 1.64. That is, the question is suitable for measuring the categories theorized. In addition, an examination of the test characteristic curves for this model indicated that the six items collectively provide adequate information for students with ability levels both below and above average. Table 3: Generalized Partial Credit Model
Category Difficulty (SE) Slope (SE)

© ISLS

76

ICLS2012 Action Effects Networked Interaction Multiple Causes Processes Order Agent Effects N = 44. -0.17 (.018) 0.04 (0.22) 0.54 (0.21) 0.77 (0.26) 0.86 (0.22) 0.87 (0.29) 1.42 (0.50) 1.64 (0.58) 1.24 (0.51) 0.69 (0.39) 0.72 (0.98) 0.91 (0.34)

Volume 1: Full Papers

Discussion and Implications
In the introduction, we highlighted that a number of students’ misconceptions in biology can be attributed to their lack of understanding in complex systems. Comprehending complex systems ideas can be counterintuitive and difficult, and a review of the literature has revealed that we have yet to systematically determine a learning approach to address these learning challenges. We have proposed that learning progressions research can help to systematize the learning pathways students take to improve conceptual competence in complex systems. These learning pathways can in turn guide the curriculum and instructional activities in promoting complex systems learning, which is our broad goal. This paper articulates the initial development of a learning progression of complex systems ideas. A sequence of a comprehensive set of these ideas, from the least to the most difficult to understand, has been determined. As there is no prior literature to guide us in hypothesizing such a sequence, we have adopted a ‘grounds-up’ approach in constructing the progression similar to Mohan et al.’s study (2009). Using the GPCM, we analyzed 44 students’ written responses to a biology question on the effects of geese arrival on a park ecosystem for their understanding of various complexity ideas. We acknowledge the small sample size in this exploratory study and the implication this may have in our analyses, but we believe we have taken sufficient steps to ensure that different grade levels and types of schools are represented so as to make our findings as generalizable as possible. We found that two of the easiest categories of ideas for students to comprehend are those that relate to the effects of actions in complex systems (Action Effects), and the interconnected and emergent nature of these systems (Networked Interactions). A “completely complex” understanding of these ideas includes the ability to reason that: small actions can lead to large effects; these actions can produce both localized and cascading changes; the changes can occur immediately and over a long period of time; the parts in the system are interdependent; the relationships among the parts in the system are nonlinear with feedback mechanisms; and the parts interact to produce emergent patterns that are not obvious at the component level. Students seem to experience more difficulty in interpreting that there are multiple causes that may attribute to the outcome(s) of a change (Multiple Causes), and perceiving that the system continues to undergo adaptation or evolution (Processes). The categories of complex systems ideas that apparently are the most complicated to understand are Order and Agent Effects; these complexity ideas concern the decentralized organization of the system, and the predictability of the effects caused by the parts of the system. We believe this ordered sequence of complexity ideas is valid – at least in the context of our test question. In a question that frames the complexity within a familiar park ecosystem, the students might have found it easier to observe the manifestation of complexity ideas associated with the interconnectedness and interdependency of the various plant and animal species in the park, the cascading, emergent, and non-linear effects the arrival of the geese can have on the rest of the ecosystem (i.e., the Action Effects and Networked Interactions categories). In contrast, complexity ideas in the Order and Agent Effects categories are less ‘visible;’ the ideas that there is a myriad of other components (e.g., plants, predators and prey to geese, climate) that may contribute to how the system is organized, and that it is not possible to predict with precision the effects of geese’s arrival at the park could well fall outside of students’ perceptual abilities. Chi (2005) also proposes that some scientific concepts and ideas may be easier to comprehend because of the ontological categories they belong to. Chi explains that “ontological categories refer to the basic categories of realities or the kinds of existence in the world, such as concrete objects, events, and abstractions” (p. 163). The ‘more visible’ ideas of complex systems may have represented ontologically-easier categories for students to understand. In sum, our findings align well with current literature on complex systems understanding, and extend it by establishing an initial progression of the various ideas. Resnick (1996) and other researchers (e.g., Wilensky & Resnick, 1999; Wilensky & Reisman, 2006; Jacobson et al., 2011) have repeatedly iterated the deterministic and centralized mindset students typically have. Likewise, Charles and d’Apollonia (2004) and others (Ben-Zvi Assaraf & Orion, 2010a; Hogan, 2000; Lin & Hu, 2003; Perkins & Grozter, 2005) have observed that students tend to think linearly about relationships among the components or parts of complex systems. Collectively, these researchers have shown that when students are exposed to appropriate instruction, the learning challenges can be overcome. However, as we have argued, most of the interventions are targeted at improving student understanding of particular systems or specific complexity concepts rather than promoting a more holistic understanding of complex systems. A learning sequence of complexity ideas that specifies levels of difficulty

© ISLS

77

ICLS2012

Volume 1: Full Papers

which we have found seeks to address this gap. This initial conception of difficulty levels is only the first, but important, step to developing a systematic approach in designing curriculum and instructional activities for the learning of complex systems ideas. In subsequent studies we aim to further validate the sequence with a larger sample size and use questions that involve other types of complex systems in order to determine how robust it is as a general complex systems heuristic.

Reference
Alonzo, A. & Steedle, J. (2008). Developing and assessing a force and motion learning progression. Science Education, 93, 389-421. doi: 10.1002/sce.20303. American Association for the Advancement of Science, AAAS (2009). Benchmarks for scientific literacy. Retrieved from http://www.project2061.org/publications/bsl/online/index.php on Mar 1, 2010. Ben-Zvi Assaraf, O., & Orion, N. (2005). Development of system thinking skills in the context of earth system education. Journal of Research in Science Teaching, 42(5), 518-560. doi: 10.1002/tea.20061. Ben-Zvi Assaraf, O., & Orion, N. (2010a). System thinking skills at the elementary school level. Journal of Research in Science Teaching, 47(5), 540-563. doi: 10.1002/tea.20351. Ben-Zvi Assaraf, O., & Orion, N. (2010b). Four case studies, six years later: Developing system thinking skills in junior high school and sustaining them over time. Journal of Research in Science Teaching, 47(10), 1253-1280. doi: 10.1002/tea.20383. Charles, E. S., & d’Apollonia, S. (2004). Developing a conceptual framework to explain emergent causality: Overcoming ontological beliefs to achieve conceptual change. In K. Forbus, D. Gentner, & T. Reiger (Eds.), Proceedings of the 26th annual cognitive science society. Mahwah, NJ: Lawrence Erlbaum Associates. Chi, M.T.H. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 14(2), 161-199. doi: 10.1207/s15327809jls1402_1. Consortium for Policy Research in Education, CPRE (2009). Learning Progressions in Science: An EvidenceBased Approach to Reform. CPRE Research Report # RR-63. New York: Teachers College Columbia University. Retrieved from http://www.cpre.org/images/stories/cpre_pdfs/lp_science_rr63.pdf on August 20th, 2010. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum. Garvin-Doxas, K.,& Klymkowsky, M.W. (2008). Understanding randomness and its impact on student learning: Lessons learned from building the biology concept inventory (BCI). Life Science Education, 7, 227233. doi: 10.1187/cbe.07-08-0063. Gotwals, A.W., & Songer, N.B. (2010). Reasoning and down a food chain: Using an assessment framework to investigate students’ middle knowledge. Science Education, 94, 259-281. doi: 10.1002/sce.20368. Hmelo-Silver, C.E., & Azevedo, R. (2006). Understanding complex systems: some core challenges. The Journal of the Learning Sciences, 15(1), 53-61. doi: 10.1207/s15327809jls1501_7. Hmelo, C. E., Holton, D., & Kolodner, J. L. (2000). Designing to learn about complex systems. The Journal of the Learning Sciences, 9, 247–298. doi: 10.1207/S15327809JLS0903_2. Hmelo-Silver, C.E., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-novice understanding of complex systems. The Journal of the Learning Sciences, 16(3), 307-331. doi: 10.1080/10508400901413401. Hmelo-Silver, C.E., & Pfeffer, M.G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28, 127-138. doi: 10.1207/s15516709cog2801_7. Hogan, K. (2000). Assessing students’ systems reasoning in ecology. Journal of Biological Education, 35(1), 22-28. Jacobson, M. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41-49. doi: 10.1002/cplx.1027. Jacobson, M. J., Kapur, M., So, H.J., & Lee, J. (2011). The ontologies of complexity and learning about complex systems. Instructional Science, 39, 763-783. doi: 10.1007/s11251-010-9147-0. Jacobson, M.J., & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. The Journal of the Learning Sciences, 15(1), 11-34. doi: 10.1207/s15327809jls1501_4. Johnson, S. (2002). Emergence. New York: Touchstone. Klopfer, E., Scheintaub, H., Huang, W., Wendel, D., & Roque, R. (2009). The simulation cycle: combining games, simulations, engineering and science using StarLogo TNG. E-Learning, 6(1), 71-96. Liu, L., & Hmelo-Silver, C.E. (2009). Promoting complex systems learning through the use of conceptual representations in hypermedia. Journal of Research in Science Teaching, 46(9), 1023-1040. doi: 10.1002/tea.20297.

© ISLS

78

ICLS2012

Volume 1: Full Papers

Levy, S.T., & Wilensky, U. (2009). Students’ learning with the connected chemistry (CCI) curriculum: Navigating the complexities of the particulate world. Journal of Science Education and Technology, 18, 243-254. doi: 10.1007/s10956-009-9145-7. Levy, S.T., & Wilensky, U. (2008). Inventing a “mid level” to make ends meet: Reasoning between the levels of complexity. Cognition and Instruction, 26(1), 1-47. doi: 10.1080/07370000701798479. Levy, S.T., & Wilensky, U. (2011). Mining students’ inquiry actions for understanding of complex systems. Computers & Education, 56, 556-573. doi: 10.1016/j.compedu.2010.09.015. Mitchell, M. (2009). Complexity: A Guided Tour. New York: Oxford University Press. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176. doi: 10.1177/014662169201600206. National Research Council, NRC (2007). Generating and evaluating scientific evidence and explanation (Chapter 5). In, R. Duschl, H. Schweingruber, and A. Shouse (Eds.), Taking Science to School: Learning and Teaching Science in Grades K-8 (pp. 129-167). Washington D.C.: National Academies Press. National Science Foundation, NSF. (2011). Empowering the Nation through Discovery and Innovation – NSF Strategic Plan for Fiscal Years (FY) 2011-2016. Arlington, VA: National Science Foundation. Mohan, L., Chen, J., & Anderson, C.W. (2009). Developing a multi-year learning progression for carbon cycling in socio-ecological systems. Journal of Research in Science Teaching, 46(6), 675-698. doi: 10.1002/tea.20314 Program for International Student Assessment, PISA. (2006). PISA Released Items – Science. Paris: OECD. Retrieved from http://www.oecd.org/dataoecd/13/33/38709385.pdf on Oct 14, 2011. Penner, D.E. (2000). Explaining systems: Investigating middle school students’ understanding of emergent phenomena. Journal of Research in Science Teaching, 37(8), 784-806. doi: 10.1002/10982736(200010)37:8<784::AID-TEA3>3.0.CO;2-E. Resnick, M. (1996). Beyond the centralized mindset. Journal of the Learning Sciences, 5, 1–22. doi: 10.1207/s15327809jls0501_1. Riess, W., & Mischo, C. (2010). Promoting systems thinking through biology lessons. International Journal of Science Education, 32(6), 705-725. doi: 10.1080/09500690902769946. Sewell, A. (2002). Constructivism and student misconceptions. Why every teacher needs to know about them. Australian Science Teachers Journal, 48(4), 24-28. Songer, N.B., Kelcey, B., & Gotwals, A.W. (2009). How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity. Journal of Research in Science Teaching, 46(6), 610-631. doi: 10.1002/tea.20313. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories-an embodied modeling approach. Cognition and Instruction, 24(2), 171-209. doi: 10.1207/s1532690xci2402_1. Wilensky, U., & Resnick, M. (1999). Thinking in levels: A dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8(1), 3-19. doi: 10.1023/A:1009421303064. Yoon, S.A. (2008). An evolutionary approach to harnessing complex systems thinking in the science and technology classroom. International Journal of Science Education, 30(1), 1-32. doi: 10.1080/09500690601101672. Yoon, S. A. (2011). Using social network graphs as visualization tools to influence peer selection decisionmaking strategies to access information about complex socioscientific issues. Journal of the Learning Sciences, 20(4), 549-588. doi: 10.1080/10508406.2011.563655.

Acknowledgement
This material is based upon work supported by the National Science Foundation under Grant No. 1019228. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

© ISLS

79

ICLS2012

Volume 1: Full Papers

Using Adaptive Learning Technologies to Personalize Instruction: The Impact of Interest-Based Scenarios on Performance in Algebra
Candace Walkington, University of Wisconsin-Madison, 1025 W. Johnson St., Madison, WI, cwalkington@wisc.edu Milan Sherman, Portland State University, P.O. Box 751, Portland, OR, milan3@pdx.edu Abstract: Context personalization refers to the idea of adapting learning activities based on students‟ interests and experiences. While new learning technologies make such innovations feasible, little research supports whether and how context personalization may mediate important learner outcomes, especially in mathematics. Here, we present results of a experimental study of Algebra I students (N = 145) receiving context personalization within an intelligent tutoring system, Cognitive Tutor Algebra. Results suggest that personalization allowed students to form more accurate and meaningful situation models of story problems, facilitating algebraic symbolization. Interest-based interventions may allow learners to make critical connections between personally relevant scenarios and abstract representational systems. Adaptive learning environments that personalize instruction by leveraging interests are an important future direction for advanced learning technologies.

Introduction
Collins and Halverson (2009) in the book Rethinking Education in the Age of Technology describe a compelling vision for how technology could change the nature of schooling. Among the key strengths that technologybased interactive learning environments could bring to educational practice is customization of learning. New technology innovations allow learning to be personalized to students‟ interests, abilities, and preferences, in order to provide assistance when needed, and present instruction that is understandable, engaging, and situated in the context of what is important to leaners. Such technologies are timely, given the pressing problems with motivation that face schools today (Hidi & Harackiewicz, 2000), especially in secondary mathematics (Mitchell, 1993; Loveless, Fennel, Williams, Ball, & Banfield, 2008). One powerful way to adapt instruction using technology that is relatively widespread in mathematics education is intelligent tutoring systems (ITS; Koedinger & Corbett, 2006). These systems build a cognitive model of a learner‟s current knowledge, and use these models to adapt problem selection, track mastery, and provide just-in-time feedback and hints. However, beyond adapting instruction to students‟ knowledge, recently these systems have begun to consider a new and potentially important form of personalized learning – presenting instruction in the context of students‟ interests (Hidi & Renninger, 2006; Renninger, Ewen, & Lasher, 2002). Carnegie Learning‟s new ITS MATHia personalizes middle school mathematics problems to different topics of student interest, such as sports, art, and music (Carnegie Learning, 2011). Here, we refer to this type of adaption based on topic interest as “context personalization,” or, for simplicity, “personalization.” While many stakeholders in education, including teachers (Fives & Manning, 2005), believe such interventions support learning, there is little evidence of their effectiveness, especially in secondary mathematics. Further, little research exists that confirms or provides an explanation as to why personalization could support learning in more abstract domains such as high school algebra, which has been framed as a gatekeeper to higher-level mathematics (Kaput, 2000) with significant implications for equity and access (Moses & Cobb, 2001). Here, we present the results of a context personalization intervention within the Cognitive Tutor software that was carried out in Algebra I classes. We explore at how such technology-based innovations have the potential to mediate learning outcomes by matching instruction to student interests.

Literature Review
Interest
Personalization, as conceptualized here, has the potential to enable productive pathways to learning by activating topic interest (Ainley, Hidi, & Berndorff, 2002). Interest is defined as “the psychological state of engaging and the predisposition to re-engage with particular classes of objects, events, or ideas over time” (Hidi & Renninger, 2006, p. 112). Topic interest has components of both situational interest, activated by surprising, salient, or evocative features of the environment, and individual interest, a learner‟s enduring preferences towards particular topics, activities, or objects (Mitchell, 1993; Hidi & Renninger, 2006). Individual interest is assumed to have both knowledge-related and value-related components, meaning that it is associated with both learners‟ feelings of value for and knowledge of the topic (Renninger et al., 2002). In some cases, when learners are presented with a topic they are interested in, their familiarity with and connection to the particular scenario may allow for the knowledge and value components of individual interest to be activated. In other cases, even

© ISLS

80

ICLS2012

Volume 1: Full Papers

though a problem is personalized to a general topic that interests the student, their connection with the particular story might not be very strong. In this case, an affective reaction from the situational interest spurred by topic familiarity may be more likely (Ainley et al., 2002). Thus topic interest is influenced by both individual and situational factors, and may be an important support for learning, as research has demonstrated that interest mediates attention, persistence, and engagement (Ainley, Hidi, & Berndorff, 2002; Durik & Harackiewicz, 2007; Hidi, 2001; McDaniel et al., 2000; Renninger & Wozinak, 1985). Prior studies on personalization in mathematics have shown mixed results, suggesting that there is a need to better understand how personalization can impact learner outcomes. Several studies have found learning gains for personalization (Anand and Ross, 1987; Cordova & Lepper, 1996; Chen & Liu, 2007), while others have not (Bates & Weist, 2004; Cakir & Simsek, 2010; Ku & Sullivan, 2000). In previous work (Walkington & Maull, 2011) we hypothesized that these mixed results are due to the fact that in order for personalization to be effective, the difficulty of the problems being personalized must be at the edge of learners‟ abilities. This makes the ITS system used in the present study an especially interesting platform to study personalization, as it intelligently selects problems based on a cognitive model of performance. In addition to the mixed results in the literature, all of the previous studies have been conducted with elementary school students, and few look at personalization during the actual course of instruction where issues of interest and motivation would be most salient. Here, we present an ITS-based study of personalization during the course of algebra instruction.

Algebraic Reasoning
The present study focuses on algebra story problems (Figure 1), specifically the skill of writing of symbolic expressions from story problems. Koedinger and Nathan (2004) found that algebra story problems were easier to solve than matched symbolic equations, and attributed this to difficulties students have learning the language of algebra. Indeed, a number of studies have shown that algebraic expression-writing is a challenging skill (Walkington, Sherman, & Petrosino, 2012; Bardini, Pierce, & Stacey, 2004; Koedginer & McLaughlin, 2010; Clement, 1982; Stacey & MacGregor, 1999). Students may have a tendency to conceptualize a symbolic equation as a string of calculations rather than a statement about equality, and may assign variables multiple or shifting values (Clement, 1982; Stacey & MacGregor, 1999). Students may also have difficulty combining different parts of symbolic expressions (Koedinger & McLaughlin, 2010; Heffernan & Koedinger, 1997). Nathan, Kintsch, and Young (1992) proposed a model of algebra story problem solving where learners negotiate three levels of representation: (1) a propositional textbase containing the information given in the story in propositional form, (2) a situation model representing a qualitative understanding of the actions and relationships in the story, and (3) a problem model which includes formal mathematical operands, variables, and equations. They found that when students were given support in forming situation models through computer animations, they were better able to write equations from stories. Similarly, in a pilot study of 24 students solving normal and personalized algebra story problems, we found that personalization supported students in forming more detailed and accurate situation models of problem scenarios. Students were more likely to attempt personalized problems, reported that personalized problems easier to solve and more related to their lives, and were more likely to use informal strategies on that closely mirrored the problem‟s action (Walkington et al., 2012). Here, we investigate this idea of supporting learner‟s situation models on a larger scale, during the course of actual classroom instruction with an intelligent tutoring system.

Research Question
In previous work (Walkington, 2012) we found that personalization of algebra story problems promotes performance and learning gains for algebraic symbolization. However, it is not clear from our previous studies precisely why this result occurred. Here, we engage in a deeper investigation of the positive performance effect for personalization, in order to try to uncover how personalization supports learning. Thus our research question is: How and under what circumstances does personalization of algebra story problems support learning pathways? To investigate this question, we will examine the interaction between the context of a story problem and the impact of personalization, the effect of the readability level of the problem on the impact of personalization, and the impact of personalization on student mistake patterns.

Methods
This study took place during instruction as 145 9th grade Algebra I students used the Cognitive Tutor Algebra software. Cognitive Tutor is an intelligent tutoring system for Algebra I that individualizes instruction through adaptive problem selection and feedback (Koedinger & Corbett, 2006). Participants used Cognitive Tutor two days per week, most students spent several sessions working through the unit examined in this study, Unit 6. Participants were from a suburban school in the Northeastern United States that was majority Caucasian (96%), with 18% of students eligible for free/reduced lunch and 71% proficient in math on the 11th grade assessment. Participants were randomly assigned to two conditions when they entered Unit 6, which contained algebra story problems on linear functions. The control group received the standard problems for the unit, while

© ISLS

81

ICLS2012

Volume 1: Full Papers

the experimental group received problems matched to their topic interests through a computer survey administered to both conditions. The survey asked students to rate their level of interest in (i.e., “liking of”) 9 topic areas: sports, music, movies, games, TV, computers, food, art, and shopping. Personalized problems corresponding to each topic were written based on prior surveys (N = 60) and interviews (N = 29). Although the topic of each problem was matched to students‟ interests at the individual level, there is no measure in this study of whether the specific story in each problem was relevant to individual students‟ experiences with that topic. However, our prior survey, interview, and observational work revealed a great deal of overlap in the individual experiences and preferences high school students typically had in relation to each of the 9 topics. There were 27 problems in the original unit, and 4 variations on each problem were written to correspond to the interest topics (Table 1). For each problem, students filled in different cells in a spreadsheet as they solved result and start unknowns. In result unknowns (questions 1-2, Figure 1), students solved for the y variable in a linear function given a specific x value, which involves directly applying the described operations. For start unknowns (questions 3-4, Figure 1), students solved for the x variable in a linear function given a y value, which may require working backwards. Students also wrote algebraic expressions from story scenarios. Table 1: Example of four personalized variations on original algebra story problem from Unit 6 Original Problem Sports Music Art Games One method for estimating the cost of new home construction is based on the proposed square footage of the home. Locally, the average cost per square foot is estimated to be $46.50. You are working at the ticket office for a college football team. Each ticket to the first home football game costs $46.50. You are helping to organize a concert where some local R&B artists will be performing. Each ticket to the concert costs $46.50. You have been working for the school yearbook, taking pictures and designing pages, and now it‟s time for the school to sell the yearbooks for $46.50 each. You work for a Best Buy store that is selling the newest Rock Band game for $46.50.

Figure 1. Example of normal story problem scenario - upper text shows result and start unknown questions posed, while lower table displays correct responses to each problem part The data were analyzed using hierarchical logistic regression (Snijders & Bosker, 1999), where problem parts were nested within students who were nested within teachers. Random intercept terms were included for which story problem the student was working on, and what linear function (i.e., y=-1550 – 7x) or “item” was being described in the story problem. The dependent variable was whether the student got the problem part correct on the first attempt. Fixed effects included which condition the student was in (experimental group - personalized problems or control group - normal problems), the difficulty of the knowledge component being assessed in the problem part, and the interaction between condition and knowledge component difficulty. A knowledge component (KC; Koedinger & Aleven, 2007) is a skill that is tracked for mastery by Cognitive Tutor, and KCs were classified as easy, medium, or hard based on student performance data. Hard KCs included writing algebraic expressions with slope and intercept terms, while medium KCs included solving result and start unknowns, working with different types of numbers, and writing expressions with only a slope. Easy KCs included identifying independent and dependent units and entering given values.
© ISLS 82

ICLS2012

Volume 1: Full Papers

Results and Discussion
The output from the regression model of performance in Unit 6 is shown in Table 2. Personalization increased performance when students were solving problem parts with hard KCs (odds = 1.5228, p < .001) and easy KCs (odds = 1.5228*.9368 = 1.4266, p < .001). The effect of personalization on medium KCs did not quite reach significance (odds = 1.5228*.7689 = 1.1709, p = 0.0601). Personalization supported students‟ algebraic expression-writing for linear functions that had both a slope and intercept term (hard KCs), and also facilitated the solving of less mathematically-relevant portions of the problem, like identifying units (easy KCs). Table 2: Output for hierarchical logistic regression model of performance within Unit 6 Raw Coeff Std. Error Exp(Coeff) (Intercept) -0.27378 0.22707 0.7605 Condition-Control Ref. Condition-Experimental 0.42053 0.11134 1.5228 KC-Hard Ref. KC-Medium 1.32951 0.05986 3.7792 KC-Easy 2.17917 0.06105 8.8390 Condition-Exp:KC-Medium -0.26276 0.08255 0.7689 Condition-Exp:KC-Easy -0.06528 0.08484 0.9368 Note. Significance codes: „***‟ 0.001, „**‟ 0.01, „*‟ 0.05 z value -1.21 3.78 22.21 35.69 -3.18 -0.77 Sig

*** *** *** **

Although it is not included here for brevity, analyses of students‟ work in a future unit of Cognitive Tutor showed that personalization not only increased performance on hard KCs within Unit 6, it significantly increased performance on hard KCs in the subsequent unit, after the personalization intervention had been removed (Walkington, 2012). This suggests that the performance boost associated with personalization in Unit 6 facilitated students‟ learning of the concept of algebraic expression-writing. Thus we determined that it is critical to engage in a deeper investigation of what promoted or caused the performance differences seen in Unit 6, in order to better understand why personalization ultimately impacted students‟ long-term learning.

Problem Cover Stories
We begin by examining which of the story problems in Unit 6 benefitted most from being personalized. This may indicate classes of story problems for which personalization is particularity helpful. To do this, we added a condition by item random slope to the model shown in Table 3. This term essentially shows how the condition (personalized versus non-personalized) interacted with each base story problem to impact student performance. We concentrated on two groups of problems for this analysis. First, we looked at problems with random slopes indicating that personalization reduced the odds of getting the problem correct, corresponding to odds values of 0.7 to 0.9. Second, we looked at problems with random slopes indicating that personalization increased the odds of getting the problem correct, corresponding to odds values of 1.1 to 1.4. The results are shown in Table 3. Table 3: The impact of personalization on performance for selected story problems in Unit 6 Topic of Original Problem Cell phone service Getting raise at work Pay from work Selling paintings Mail-order shopping Sending cards to friends Paying cell phone bill Making wages at work Nitrogen in an asteroid Home square footage Taking pictures with film Installing carpet Trading wampum shells Taking vitamins Breaking crayons Linear Function y = 60-5x y = .04x y = .72x y = 64-6x y = x -.025x+10 y = 55-.5x y = -.23x+7.87 y = 10.50x y = 90-2x y = 46.50x y = 36-6x y = 12.95x y = 80-6x y = 50-2x y = 24-2x Personalization’s Impact on Odds 0.73 0.74 0.75 0.82 0.85 0.86 0.86 0.87 1.13 1.13 1.15 1.19 1.22 1.22 1.39

Personalization… Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Reduced performance Increased performance Increased performance Increased performance Increased performance Increased performance Increased performance Increased performance

© ISLS

83

ICLS2012

Volume 1: Full Papers

Several trends are apparent from Table 3 that provide insight into how personalization may support learning. First, personalization was not particularly helpful when the problem‟s topic was already relevant to students‟ lives and interests – like paying for their cell phone or getting paid at work. In these cases, the original problem could actually be slightly easier for students to solve than the personalized versions. Most of the problems where personalization was not beneficial dealt with money. High school students already have strong familiarity with doing calculations with money, especially in the context of work and shopping (Walkington et al, 2012), thus it makes sense that personalizing such a problem is unlikely to further improve performance. The instances in Table 3 where personalization increased performance are also revealing – high school students are not likely to have direct experience with calculations relating to nitrogen in an asteroid or installing carpet, thus personalization of these problems was particularly beneficial. Taken together, these results suggest that personalization supports performance when irrelevant story contexts become more situated within students‟ experiences and interests, and the ways in which they encounter numbers in their everyday life. Thus personalization may be effective in supporting learners in forming detailed and comprehensible situation models (Nathan et al., 1992) of problem scenarios, allowing them to leverage their prior knowledge of familiar contexts.

Readability Level
One reason why personalization may increase performance is that personalized problems are easier for students to read and interpret. This is supported by the data shown in the previous section, which suggested that personalization had the largest impact when it was used on problems with contexts that were not familiar or relevant to students‟ experiences. To examine this hypothesis, we added the readability level (Flesch, 1948) of the story scenario to the hierarchical model as a predictor. Readability level is an index used to measure comprehension difficulty, and uses indicators such as number of words, number of sentences, and syllables. Each story problem was classified as “below grade level” if the Flesch-Kincaid grade level test predicted its readability was appropriate for grades 4-7, “at grade level” if it was appropriate for grades 8-10, and “above grade level” if it was appropriate for grades 11-15. This addition also allowed us to check whether differences in performance for personalized problems were being driven solely by systematic differences in readability level. Results show that personalization still had a significant positive effect on performance on easy (odds = 1.397, p < .001) and hard (odds = 1.489, p < .001) KCs when controlling for readability level. The model also showed a significant interaction between readability level and condition (χ2(2) = 6.364, p < .05), and readability level and knowledge component difficulty (χ2 (4) = 39.532, p < .001). The results are summarized in Figure 2. For both easy and hard KCs, when a problem was easy to read (below grade level) the personalization condition had higher performance. However, the advantage of the personalization condition became even greater as the problems became more difficult to read. So, for example, if a student was presented with normal problem that was difficult to read, and a personalized problem that was also difficult to read, their performance would be considerably higher on the personalized problem. Whereas different readability levels did not seem to impact performance on personalized problems, increasing grade level readability had a considerable negative effect on performance on normal problems. The model showed that personalization had significantly greater positive impact on performance when the text of the problem was above grade level, for both hard and easy KCs (odds = 1.424, p < .05), compared to performance on these KCs on below grade level texts.

Figure 2. The impact of readability level on problem-solving performance for control (normal problems) and experimental (personalized problems) groups, separated by Easy KCs (left) and Hard KCs (right) These results suggest that personalization allows students to better interpret difficult verbal scenarios of complex mathematical situations, and that this effect is especially large when students are asked to write algebraic expressions from story scenarios. The students‟ familiarity with personalized story contexts, due to the

© ISLS

84

ICLS2012

Volume 1: Full Papers

connection with their interests, may allow students to better understand the relationships within a story scenario with a complicated verbal structure, facilitating the formation of detailed and accurate situation models that support expression-writing. Thus personalization may allow learners to better move from their propositional textbase to an elaborated understanding of the story‟s actions and relationships.

Analysis of Mistakes
Personalization had an important positive impact on performance and learning for hard KCs, when students were writing symbolic equations from story scenarios. The analyses above suggest that successfully writing a symbolic equation from a story is tied to the students‟ verbal comprehension of the text and their ability to draw on their prior knowledge to form a situation model of the actions and relationships being described. We conducted an analysis of students‟ interactions with the tutor when solving hard KCs in order to see if the situation model support provided by personalization allowed students to avoid certain kinds of mistakes. As can be seen from Table 4, when students were presented with personalized problems, they were more likely to write an algebraic expression successfully on their first attempt, were less likely to initially ask for a hint, and were less likely to make a mistake and write the expression incorrectly. Results of the HLM analysis presented at the beginning of the section show that the difference in correct answers for the control group (42.6%) versus the experimental group (51.6%) is a statistically significant (p < .001) trend. Table 4: Student performance on Hard KCs Outcome First Attempt Correct First Attempt Hint First Attempt Incorrect Control 42.6% 8.2% 49.2% Experimental 51.6% 4.9% 43.4%

As can be seen from Table 5, there are also differences in mistake patterns that may account for the greater success of the experimental group in writing algebraic expressions. The experimental group was equally likely to make simple sign reversal errors on either the slope or the intercept term. However, the experimental group was less likely to mix up the slope and intercept parameters or use incorrect signs for both the slope and intercept. The experimental group was also slightly less likely to not include a slope coefficient (i.e., use a slope of 1), not include an intercept term, or leave out both the slope term and the independent variable. Generally, this seems to imply that personalization assists students in remembering to include the appropriate parameters and variables in their expression, while also helping them make sense of how these parameters should be combined. This suggests that personalization reduces the likelihood of large conceptual misspecifications of the functional relationship by supporting students‟ situation models. In this way, personalization may allow students to meaningfully create a mathematical model for a situation, allowing them to coordinate their understanding of the story with a formal symbolic representation. This is contrasted with a purely procedural or syntactic construction of an algebraic expression, where situational understanding is not well connected to symbolic reasoning. Such approaches may be more prone to major conceptual errors, as shown by research on direct translation strategies when solving arithmetic word problems (Hegarty, Mayer, & Monk, 1995). Table 5: Mistakes made by students when writing symbolic expressions from story scenarios Write Equation Mistake Entered slope of 1 Mixed up slope and intercept No intercept Only intercept Intercept wrong sign Slope wrong sign Wrong sign, slope and intercept Control 111 57 146 16 166 116 104 Experimental 103 39 136 9 165 114 118

Summary and Conclusion
In previous work, we found that personalization to students‟ interests mediates performance and learning in potentially powerful ways, supporting students in navigating the abstract representational systems in algebra. Here, we took a closer look at the data from our studies in order to investigate why such an effect occurs. We conclude that personalization may support students‟ formation of more detailed, accurate, and meaningful situation models of problem scenarios. This is evidenced by findings that the effect of personalization is largest when it makes an irrelevant problem relevant to students‟ experiences and when it makes the actions and relationships in a complex verbal scenario more easily read and interpreted. Personalization also seemed to
© ISLS 85

ICLS2012

Volume 1: Full Papers

allow students to make fewer large conceptual mistakes when specifying the relationships in a story, suggesting that personalization supports coordination of situation and problem models. Rather than creating a general motivational effect that was seen throughout learners‟ problem-solving, the impact of topic interest on performance was targeted. Interest facilitated performance when students were working directly with the scenario, naming quantities and units and formulating a symbolic equation of the relationships. When students wrote equations, interest seemed to support understanding of the deeper, quantitative structure of the story. This suggests that for algebra story problems, the effectiveness of interestbased interventions lies in their ability to allow productive connections between situational contexts and formal models of situations. As the context becomes more comprehensible and connected to prior experience, students are better able to coordinate their situation and problem models in the process of problem-solving. We hypothesize that personalization could be considered a form of grounding (Goldstone & Son, 2005; Koedinger, Alibali, & Nathan, 2008) that allows for abstract formalisms to be coordinated with concrete everyday experiences. Here, we presented evidence that the grounding provided by personalization becomes more effective as the story scenarios become more related to the interests, language, and activities the students are familiar with. Further, we described how in related work (Walkington, 2012) providing such grounding not only facilitated performance on personalized problems, but also allowed for robust learning of underlying algebraic ideas. This suggests that grounding interventions based on topic interest are not simply a crutch for learners to lean on when they struggle or become disengaged – they are a scaffold that has the potential to allow learners to come to understand powerful systems of representation. With the rise of advanced learning technologies, the potential to connect instruction to both students‟ interests and to their current level of understanding has become a reality. It is critical that we understand how this technology impacts students‟ cognition and problem-solving, such that it can be leveraged in schools to enable productive pathways to learning. Here, we show how interest-based interventions can mediate students‟ situational understanding of mathematical relationships in story problems, fostering critical connections between personalized scenarios and abstract representational systems. It is both surprising and promising that such a simple alteration to instructional materials to make them more adaptive had an important impact on both performance and long-term learning of difficult concepts in secondary mathematics. As learning technologies advance, we thus propose that an important future direction is designing environments that can adapt to student background and preferences in a deeper and more meaningful manner, by leveraging the interests that drive and motivate students in authentic and powerful ways.

References
Ainley, M., Hidi, S., & Berndorff, D. (2002). Interest, learning, and the psychological processes that mediate their relationship. Journal of Educational Psychology, 94(3), 545-561. Anand, P., & Ross, S. (1987). Using computer-assisted instruction to personalize arithmetic materials for elementary school children. Journal of Educational Psychology, 79(1), 72-78. Bardini, C., Pierce, R., & Stacey, K. (2004) Teaching linear functions in context with graphics calculators: students' responses and the impact of the approach on their use of algebraic symbols. International Journal of Science and Mathematics Education, 2, 353-376. Bates, E., & Wiest, L. (2004). The impact of personalization of mathematical word problems on student performance. The Mathematics Educator, 14(2), 17-26. Caker, O., & Simsek, N. (2010). A comparative analysis of computer and paper-based personalization on student achievement. Computers & Education, 55, 1524-1531. Carnegie Learning (2011). Carnegie Learning Math Series: Carnegie Learning MATHia Software. Retrieved May 24, 2011 from http://mathseries.carnegielearning.com/product-info/software Chen, C., & Liu, P. (2007). Personalized computer-assisted mathematics problem-solving program and its impact on Taiwanese students. Journal of Computers in Mathematics and Science Teaching, 26(2), 105-121. Clement, J. (1982). Algebra word problem solutions: Thought processes underlying a common misconception. Journal for Research in Mathematics Education, 13(1), 16-30. Collins, A., & Halverson, R. (2009). Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America. New York: Teachers College Press. Cordova, D., & Lepper, M. (1996). Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. Journal of Educational Psychology, 88(4), 715-730. Durik, A., & Harackiewicz, J. (2007). Different strokes for different folks: How individual interest moderates effects of situational factors on task interest. Journal of Educational Psychology, 99(3), 597-610. Fives, H., & Manning, D. (2005). Teachers' strategies for student engagement: Comparing research to demonstrated knowledge. Paper presented at 2005 Annual Meeting of American Psychological Association, Washington DC. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221-233.

© ISLS

86

ICLS2012

Volume 1: Full Papers

Goldstone, R., & Son, J. (2005). The transfer of scientific principles using concrete and idealized simulations. Journal of the Learning Sciences, 14(1), 69-110. Heffernan, N. T., & Koedinger, K. R. (1997).The composition effect in symbolizing: The role of symbol production vs. text comprehension. In Proceedings of the nineteenth annual meeting of the Cognitive Science Society (pp. 307-312). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Hegarty, M., Mayer, R., & Monk, C. (1995). Comprehension of arithmetic word problems: A comparison of successful and unsuccessful problem solvers. Journal of Educational Psychology, 87(1), 18-32. Hidi, S. (2001). Interest, reading and learning: Theoretical and practical considerations. Educational Psychology Review, 13, 191–210. Hidi, S., & Harackiewicz, J. (2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70, 151–179. Kaput, J. J. (2000). Teaching and learning a new algebra with understanding. U.S.; Massachusetts: National Center for Improving Student Learning and Achievement. Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19, 239-264. Koedinger, K., Alibali, M., & Nathan, M. (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science, 32, 366-397. Koedinger, K. R., & Corbett, A. (2006). Cognitive Tutors - Technology Bringing Learning Sciences to the Classroom. In R. K. Sawyer (ed.) The Cambridge Handbook of the Learning Sciences. St. Louis, Cambridge University Press: 61-77. Koedinger, K., & McLaughlin, E. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society. (pp. 471-476.) Austin, TX: Cognitive Science Society. Koedinger, K., & Nathan, M. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129-164. Ku, H., & Sullivan, H. (2000). Personalization of mathematics word problems in Taiwan. Educational Technology Research and Development, 48(3), 49-59. Loveless, T, Fennel, F., Williams, V., Ball, D., & Banfield, M. (2008). Chapter 9: Report of the Subcommittee on the National Survey of Algebra I Teachers. In Foundations for Success: Report of the National Mathematics Advisory Panel. Retrieved 14 October 2010 from http://www2.ed.gov/about/bdscomm/list/mathpanel/report/nsat.pdf. McDaniel, M., Waddill, P., Finstad, K., & Bourg, T. (2000). The effects of text-based interest on attention and recall. Journal of Educational Psychology, 92(3), 492-502. Mitchell, M. (1993). Situational interest: Its multifaceted structure in the secondary school mathematics classroom. Journal of Educational Psychology, 85, 424–436. Moses, R., & Cobb, C. (2001). Radical Equations: Math Literacy and Civil Rights. Boston: Beacon Press. Nathan, M., Kintsch, W., & Young, E. (1992). A theory of algebra-word-problem comprehension and its implications for the design of learning environments. Cognition and Instruction, 9(4), 329-389. Renninger, K., Ewen, L., & Lasher, A. (2002). Individual interest as context in expository text and mathematical word problems. Learning and Instruction, 12(4), 467-490. Renninger, K., & Wozinak, R. (1985). Effect of interest on attentional shift, recognition, and recall in young children. Developmental Psychology, 21(4), 624-632. Snijders, T. & Bosker, R. (1999). Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Sage Publications. Stacey, K., & MacGregor, M. (1999). Learning the algebraic method of solving problems. Journal of Mathematical Behavior, 18(2), 149-167. Walkington, C. (April, 2012). Context personalization in algebra: Supporting connections between relevant stories and symbolic representations. Paper presented at the 2012 Annual Meeting of the American Educational Research Association. Vancouver, Canada. Walkington, C., & Maull, K. (2011). Exploring the assistance dilemma: The case of context personalization. In L. Carlson, C. Hö lscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 90-95). Boston, MA: Cognitive Science Society. Walkington, C., Sherman, M., & Petrosino, A. (2012). „Playing the game‟ of story problems: Coordinating situation-based reasoning with algebraic representation. Journal of Mathematical Behavior, 31(2), 174195.

Acknowledgments
This work was conducted in partnership with Carnegie Learning, and was supported by the Pittsburgh Science of Learning Center which is funded by the National Science Foundation award # SBE-0354420.

© ISLS

87

ICLS2012

Volume 1: Full Papers

The Impact of Students’ Exploration Strategies in Discovery-Based Instructional Software
Associate Professor Barney Dalgarno, Charles Sturt University, bdalgarno@csu.edu.au Associate Professor Gregor Kennedy, University of Melbourne, gek@unimelb.edu.au Associate Professor Sue Bennett, University of Wollongong, sbennett@uow.edu.au Abstract: Active learning is a key element of constructivist learning theory and has been used as an argument for employing discovery-based designs with instructional software. On the other hand, researchers have highlighted empirical evidence showing that ‘pure’ discoverybased learning is of limited value. This suggests that how learners interact is important in predicting whether learning occurs. This paper reports on a study of 158 university students who each used two instructional simulations – one with a discovery-based design and the other with a tutorial-based design. Students’ learning outcomes were assessed via pre-tests and post-tests of conceptual understanding. Students’ interactions using the discovery-based program were recorded and coded as either systematic or unsystematic. The results showed that when compared with the tutorial-based learning program systematic exploration resulted in learning benefits, while unsystematic exploration did not. These results have implications for the design of instructional software if such resources are to be used effectively.

Introduction
A key element of constructivist theories of learning is the idea that each person forms their own representation of knowledge, building on their individual experiences – an idea attributed primarily to Piaget (1973). This knowledge representation is constantly reviewed and revised as inconsistencies between the current knowledge representation and experience are encountered through active exploration (Bruner, 1962; Piaget, 1973). Piaget (1973) explains the learning process in terms of equilibration. Equilibration begins with the construction by the individual of their own internal knowledge representation, or in Piaget’s terms they accommodate their knowledge representation to fit with their experience. Subsequent experiences that are consistent with this knowledge representation are then assimilated into their knowledge representation. New experiences that do not fit with their current knowledge representation result in a further accommodation of their knowledge representation to fit with this new experience. This idea that learning involves active knowledge construction has been used in support of various learning design frameworks, including inquiry-based learning (particularly in the sciences), and discovery-based learning using educational multimedia and computer-based simulations. On the other hand, researchers such as Mayer (2004) and Kirschner, Sweller and Clarke (2006) have highlighted empirical evidence showing that ‘pure’ discovery-based learning is of limited value. This suggests that how learners interact is an important avenue for further research. Specifically, the relationship between a learner’s interaction with an instructional software program and his or her cognition has been identified as an important factor in predicting whether learning occurs (see Kennedy, 2004). This paper reports on research investigating how learners’ different exploration strategies impacted on their cognitive processes and consequently on their learning outcomes.

Background
Interest in the potential learning benefits of interactive multimedia over the past 15 to 20 years was driven initially by the advent of the personal computer and its evolving multimedia capabilities, and subsequently by the advent of the Internet, making resources more readily located and acquired. Early multimedia took the form of programmed instruction or tutorials in which the learner would work through a linear sequence of instructional screens or drill-and-practice activities which provided an opportunity for learners to perfect their responses with immediate feedback. These designs were typically based on behaviourist theories of learning and so limited choice was available to learners. More recently the range of instructional software available has expanded but this type is still widely available (eg. Mathletics). Alternative types of multimedia software began to emerge in the 1990s, with resources like Investigating Lake Iluka (Harper, Hedberg & Brown, 1995) and the Jasper Woodbury series (Cognition and Technology Group At Vanderbilt, 1992), which allowed non-sequential inquiry-based exploration of a computer-based learning environment scaffolded by the provision of holistic problem scenarios. The design of these types of interactive learning environment has been underpinned by the principles of active, inquiry-based learning that occurs in context and promotes knowledge construction and articulation (see Duffy & Cunningham, 1996). Early research questions included: how multiple media should be combined to present content; how multimedia information might be structured; how learners might control the pace or the route by which they
© ISLS 88

ICLS2012

Volume 1: Full Papers

moved through the data; and how the interface might be designed so learners would know where to click. This research agenda led to numerous experimental studies drawing on cognitive theories through which principles for designers and developers were derived (eg. Chandler and Sweller, 1991; Mayer; 2001; Sweller, 2005). Another body of research in the field emerged in the form of small-scale naturalistic studies of the use of multimedia resources by learners in particular contexts resulting in the derivation of design guidelines incorporating active exploratory learning underpinned by constructivist approaches to teaching (eg. Harper & Hedberg, 1997; Herrington & Oliver, 2000). These two research agendas have developed in parallel and at times at odds with one another. Recently, Mayer’s (2004) conclusion that unscaffolded discovery learning is of limited or no value has re-ignited debate over how much direction is optimal in the design and use of instructional software (see also Kirschner, Sweller & Clarke, 2006; Kapur, 2008; Hmelo-Silver, Duncan & Chinn, 2007) In summary, despite changes in technology over the past decade, interactive multimedia remains relevant, while researchers’ understanding of the role interactivity plays in learning is limited. It is well accepted from a theoretical standpoint that there is great potential in exploratory resources that allow learners to actively construct their own knowledge representation through interaction with the environment, however the empirical results are mixed. Consequently there is a need for further research that explores the relationship between the design of the resource, the interactive tasks undertaken by the learner, and the cognition and learning that occurs as a result. The study described in this paper was intended to contribute to knowledge about the relationship between interactive learning and cognition through an exploration of the learning performance of students using a multimedia resource with a discovery-based design compared to students using a resource with a tutorialbased design. As well as allowing for a direct comparison of the two groups the study also collected data on student prior knowledge, engagement and cognitive load as well as logging student interactions in order to characterise their exploration strategies. The focus of this paper is on the way in which exploration strategy affects learning performance. Future articles will report on the relationship between prior knowledge, engagement, cognitive load and learning performance.

Methodology
The study compared learning performance using a discovery-based design with performance using a tutorialbased design in each of two content domains, as shown in Table 1. The study was conducted with 158 University of Wollongong teacher education students. Each student completed a discovery-based learning condition using a multimedia resource focussing on one content domain and a tutorial-based learning condition using a multimedia resource focussing on the other content domain. The order was varied so that approximately half of the students undertook their tutorial learning condition first while the other half undertook their discovery learning condition first. Prior to each learning condition the students completed a knowledge pre-test. After each learning condition the students completed a knowledge post-test and a questionnaire with engagement and cognitive load items. Student actions within the learning resources were also logged to allow later analysis of their exploration strategies. For analysis purposes the two content domains were treated as distinct experiments. In each case the independent variable was learning condition, with two levels, Tutorial and Discovery, and the dependent variable was learning performance. The decision to essentially undertake two concurrent experiments using two distinct content domains and two distinct sets of multimedia resources was made because, being mindful of the potential for content or resource specific factors to impact on the results, we were keen to have two distinct data sets to draw upon in making conclusions from the data. The content domains, blood alcohol concentration and global warming, were chosen because of their widespread interest within the likely participant community. Table 1: Experimental Design Content Domain Blood Alcohol Concentration N=73 N=85

Condition Tutorial Discovery

Global Warming N=85 N=73

To address issues encountered in earlier studies (see, for example, Dalgarno, Bennett & Harper, 2010), resources and learning tasks were designed to ensure alignment with the intended learning outcomes. The discovery resources were designed around a ‘predict-observe-explain’ learning design (White & Gunstone, 1992), with students encouraged to mentally predict the effect of their simulation parameter changes, observe the results of the change and mentally try to explain the observed results. Figures 1 and 2 provide excerpts from the discovery-based resources showing the screens on which participants could change variables on the lefthand side of the screen before running the simulation to see the results. In the case of global warming, the four graphs on the right-hand side of the screen, showing values for ozone layer thickness, CO 2 concentration,
© ISLS 89

ICLS2012

Volume 1: Full Papers

greenhouse insulation and surface temperature, changed if the variables were altered. In the blood alcohol concentration simulation the graph on the right-hand side showing values for blood alcohol concentration over time, changed if the input variables were altered. As far as possible the user interface was consistent across the two topics to maximise comparability between the resources.

Figure 1. Excerpt from the Global Warming Discovery Resource

Figure 2. Excerpt from the Blood Alcohol Concentration Discovery Resource The tutorial-based resources for each topic area were created from the discovery resource, but instead of allowing students to change the variables a series of pre-defined values and the relevant graphical outputs were shown. Students were provided with a single option, a continue button, which resulted in the next set of predefined values being loaded and the corresponding output displayed. This is illustrated in Figures 3 and 4.

© ISLS

90

ICLS2012

Volume 1: Full Papers

Figure 3. Excerpt from the Global Warming Tutorial Resource

Figure 4. Excerpt from the Blood Alcohol Concentration Tutorial Resource The knowledge pre-tests and post-tests contained a series of identical items designed to measure understanding of key concepts within the content area. For example, within the Blood Alcohol Concentration content area, the following item was included within the pre-test and post-test to explore the participants’ understanding of the relationship between of a person’s weight and their blood alcohol concentration: A person with greater body weight: a) Will have a higher Blood Alcohol Concentration (BAC) than a lighter person. b) Will have a lower BAC than a lighter person. c) Will have the same BAC as a lighter person. d) Will have their BAC increase at a greater rate than a lighter person

© ISLS

91

ICLS2012

Volume 1: Full Papers

The following item was included within the Global Warming pre-test and post-test to test the participants on their understanding of the way various environmental factors affect the global temperature: Which of the following environmental factors have a direct or an indirect effect on the Global Average Surface Temperature (GAST)? a) The amount of Carbon Dioxide (CO2) absorbed by plants b) The thickness of the ozone layer c) The percentage of CO2 in the atmosphere d) The Greenhouse insulation effect Some questions were specifically designed to test whether participants held certain common misconceptions within each content area. Some of the questions, such as the Global Warming example question above, had multiple correct stems while some, such as the Blood Alcohol Concentration example question above, had a single correct stem. Students scored one point for each question in which only the correct stems were chosen. There were seven questions in the Global Warming test and nine questions in the Blood Alcohol Concentration test.

Results
The first analysis considered whether participants who had completed the program with a discovery-based or tutorial-based learning design showed improved understanding. As shown in Table 2, an initial analysis of the learning which occurred in each condition was undertaken using paired T tests which compared the Pre-Test and Post-Test scores. The results of this analysis indicated that for the Global Warming content area, which participants found quite challenging, there was a slight decrease in test performance from the Pre-Test to the Post-Test by Tutorial participants and no difference in performance for Discovery participants. For the Blood Alcohol Concentration content area there was no difference in test performance for the Tutorial participants and an increase for Discovery participants. For each content domain (Global Warming and Blood Alcohol Concentration) an analysis of covariance (ANCOVA) test was completed which included post-test score as the dependent variable, learning design condition as the independent variable (discovery, tutorial), and pre-test score as a covariate. For the Global Warming content domain there was no effect for condition (F (1,155) = 2.40; p = .124) but there was a significant effect for participants’ pre-test scores on their post-test understanding (F (1,155) = 27.50; p < .001). For the Blood Alcohol Concentration, there was an effect for both condition (F (1, 155) = 5.52; p = .02) and the participants pre-test scores on their post test understanding (F (1,155) = 16.40; p < .001). It can be seen from the mean scores and standard deviations presented in Table 2, that despite these significant result, particularly in the content area of Blood Alcohol Concentration, in general participants showed very little improvement in performance from pre- to post-test, regardless of content domain or condition. Given the maximum scores that could be obtained by students, performance scores were well below the mid point of the scale for both content domains. Table 2: Mean pre- and post-test scores for both conditions across the two content domains Content Domain Global Warming+ Blood Alcohol^
+max = 7; ^max = 9

Condition Tutorial (n=85) Discovery (n=73) Tutorial (n=73) Discovery (n=85)

Pre-Test M (SD) 1.82 (1.51) 1.68 (1.42) 3.55 (1.25) 3.60 (1.24)

Post Test M (SD) 1.42 (1.29) 1.72 (1.85) 3.42 (1.31) 3.93 (1.40)

T 2.26 (p=0.027) 0.20 (p=0.841) 0.60 (p=0.552) 2.33 (p=0.022)

© ISLS

92

ICLS2012

Volume 1: Full Papers

We also noticed that the variance in post-test scores for participants’ discovery-based condition, particularly for the global warming content domain, was quite high; and we wondered whether there may be variation within the discovery-based group that reflected differences in how the simulation was used and explored by participants. A preliminary analysis showed that some participants seemed to have explored the simulation systematically and others were more unsystematic in their approach. Given this, we characterised participants’ strategies while interacting with the simulation-based learning activity as Systematic Discovery if they investigated the content area by ‘running’ the simulation by changing only one variable from the provided example (e.g. ‘Bill’s values’ or ‘2006 values’) or from previous simulation outputs on four or more occasions. All other discovery participants’ strategies were classified as Non Systematic Discovery. Table 3 shows the implicit experimental design once the discovery participants were separated into two groups using this characterisation. The new design again has learning condition as the independent variable, but with three levels, Tutorial, Non-Systematic Discovery and Systematic Discovery. The criteria and thresholds for distinguishing between non-systematic and systematic discovery participants could have been arrived at in a number of ways. We experimented with cluster analysis using a number of the indicators of exploration approach (time spent, iterations undertaken, variables changed etc) but ultimately decided to use the number of variables changed as a simple, intuitively meaningful criteria, and chose thresholds that ensured sufficiently sized and meaningful sub-groups. In order to investigate whether this characterisation of participants’ exploration strategies in the simulator was better able to account for variance in post-test performance scores, an ANCOVA for each content area was again calculated but this time with a three-level independent variable (tutorial, non-systematic discovery, and systematic discovery), as shown in Table 4. For both content domains there were again significant covariate effects for the pre-test scores on the post-test scores (Global Warming (F (1,154) = 23.72; p < .001); Blood Alcohol (F (1,154) = 23.72; p < .001)). Main effects for condition were also recorded for both content domains. In both content domains the pattern of results was the same. After accounting for pre-test knowledge, post-test comparisons indicated that participants who were in the systematic discovery group recorded significantly higher post-test scores than participants in both the non-systematic discovery and the tutorial groups. Interestingly, there were no differences between these latter two groups in terms of their understanding (post-test) for either content domain. Table 3: Experimental Design after Characterising Discovery Strategies Content Domain Blood Alcohol Concentration N=73 N=51 N=34

Condition Tutorial Non-Systematic Discovery Systematic Discovery

Global Warming N=85 N=48 N=25

Table 4: Significant main effects for condition (three levels) across the two content domains Content Domain Post-Test Tutorial M (SD) 1.42 (1.29) a 3.42 (1.31) a Post-Test Non Systematic Discovery M (SD) 1.33 (1.52) a 3.51 (1.30) a Post-Test Systematic Discovery M (SD) 2.48 (2.20) b 4.56 (1.33) b F p

Global Warming+ Blood Alcohol^
ab

4.17 8.69

.017 <.001

different superscript across rows indicate between group differences (p < .001).

Discussion
Take Home Messages and Implications
The comparison of the post-test performance of students who undertook their learning using a learning resource with a tutorial-based design with those who used a resource with a discovery-based design found a small but significant difference in test performance in one content domain and no significant difference in the other. This absence of a consistent benefit for pure discovery learning is consistent with Mayer’s (2004) argument against the use of such learning designs. However, when the exploration strategies of participants using the discoverybased resources were analysed it became clear that some students explored in a systematic fashion and some did not, and that there was a clear difference in test performance of systematic and unsystematic explorers within

© ISLS

93

ICLS2012

Volume 1: Full Papers

each content area. Additionally, systematic explorers in each content area performed significantly better than tutorial participants. This suggests that, contrary to Mayer’s (2004) arguments, for some students, an active discovery-based design may be ideal. However, for others, the benefits of active exploration are countered by the confusion caused by unsystematic exploration strategies and consequently they do no better than tutorial participants. An important question, then, is what led some students to explore the simulation systematically and others nonsystematically. Kirschner, Sweller and Clarke (2006) suggest that the value of instructional guidance only begins to diminish when learners have sufficient prior knowledge to allow for internal guidance. On the other hand, it may be that some learners have a natural aptitude for systematic exploration or through prior experience of more structured inquiry-based learning designs, such as problem-based learning (see Hmelo-Silver, Duncan & Chinn, 2007), they have developed more systematic exploratory approaches. The concept of scaffolding (Wood, Bruner, & Ross, 1976) is helpful in thinking about learning designs which capitalise on Piaget’s ideas about active learning while providing sufficient guidance to help prevent the learner becoming confused. Scaffolding could be provided by a teacher, with the simulation explored by a student on an interactive whiteboard while the teacher directs students’ attention to the emergent concepts. Alternatively, the tutorial resource could be enhanced by the inclusion of explanatory text or audio, similarly directing the students’ attention to the key concepts (see also Lee & Dalgarno, 2011, for a more detailed discussion of scaffolding in discovery-based learning environments).

Follow up Research
We are currently analysing the results of the engagement and cognitive load questionnaires to determine whether there is a relationship between resource type and these factors, or between these factors and learning outcomes. The relationship between prior knowledge and engagement and between prior knowledge and exploration strategy is another avenue being explored, which will help to inform on Kirschner, Sweller and Clarke’s (2006) proposition that learners with higher prior knowledge have less need for instructional guidance. We have also undertaken a pilot study using the same experimental design but with participants in an MRI scanner to determine whether brain activation differences are evident between the two conditions (see Dalgarno, Kennedy & Bennett, 2010) and we are also currently analysing these results.

References
Bruner, J.S. (1962). On knowing: Essays for the left hand. Cambridge, MA: Harvard University Press. Chandler, P. & Sweller, J. (1991). Cognitive Load Theory and the Format of Instruction. Cognition and Instruction, 8(4), 293-332. Cognition and Technology Group at Vanderbilt University (1992). The Jasper experiment: An exploration of issues in learning and instructional design. Educational Technology Research and Development, 40(1), 66-80. Dalgarno, B., Kennedy, G. & Bennett, S. (2010). Can functional brain imaging be used to explore interactivity and cognition in multimedia learning environments? Journal of Interactive Learning Research, 21(3), 317-342. Dalgarno, B., Bennett, S. & Harper, B. (2010). The Importance of Active Exploration, Optical Flow and Task Alignment for Spatial Learning in Desktop 3D Environments. Human Computer Interaction, 25(1), 2566. Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 170-198). New York: Macmillan Library Reference. Harper, B., Hedberg, J. & Brown, C. (1995). Investigating Lake Iluka [computer software]. University of Wollongong NSW: Interactive Multimedia Unit. Harper, B., & Hedberg, J. (1997). Creating motivating interactive learning environments: a constructivist view. In R. Kevill, R. Oliver, & R. Phillips, (Eds.), What works and why, proceedings of the 14th annual conference of the Australasian Society for Computers in Learning in Tertiary Education Perth: Curtin University of Technology. Hawkins, D. (1994). Constructivism: Some history. In P.J. Fensham, R.F. Gunstone & R.T. White (Eds), The content of science: A constructivist approach to its teaching and learning (pp. 9-13). London: Falmer. Hmelo-Silver, C.E., Duncan, R.G., & Chinn, C.A. (2007). Scaffolding and Achievement in Problem-Based and Inquiry Learning: A Response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99-107. Kapur, M. (2008). Productive Failure. Cognition and Instruction, 26(3), 379-424. Kennedy, G. E. (2004). Promoting cognition in multimedia interactivity research. Journal of Interactive Learning Research, 15(1), 43-61.

© ISLS

94

ICLS2012

Volume 1: Full Papers

Kirschner, P.A., Sweller, J. & Clarke, R.E. (2006). Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and InquiryBased Teaching. Educational Psychologist, 41(2), 75-86. Lee, M.J.W. & Dalgarno, B. (2011). Scaffolding discovery learning in 3D virtual environments: Challenges and considerations for instructional design. In S. Hai-Jew (Ed) Virtual Immersive and 3D Learning Spaces: Emerging Technologies and Trends, IGI-Global. Mayer, R. E. (2001). Multimedia Learning. New York: Cambridge University Press. Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14-19. Piaget, J. (1973). To understand is to invent: The future of education. New York: Grossman. Sweller, J. (2005). Implications of Cognitive Load Theory for Multimedia Learning, In R.E. Mayer (Ed), The Cambridge Handbook of Multimedia Learning, von Glasserfeld, E. (1984). An introduction to radical constructivism. In P.W. Watzlawick (Ed.), The invented reality: How do we know what we believe we know? (pp. 17-40). New York: Norton. Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. White, R. T., & Gunstone, R. F. (1992). Probing Understanding. Great Britain: Falmer Press

Acknowledgements
Support for this project has been provided through a Charles Sturt University Small Grant and a University of Wollongong Centre for Research in Interactive Learning Environments Seed Grant. The authors would also like to acknowledge the contributions of Dr Terry Judd and Dr Michael Lew of the University of Melbourne to the development of the multimedia resources.

© ISLS

95

ICLS2012

Volume 1: Full Papers

Design Research in Early Literacy within the Zone of Proximal Implementation
Susan McKenney, Open University of the Netherlands & Twente University, PO Box 2960, 6401DL Heerlen, susan.mckenney@ou.nl Paul Kirschner, Open University of the Netherlands, PO Box 2960, 6401DL Heerlen, paul.kirschner@ou.nl Joke Voogt, Twente University, PO Box 217, 7500 AE, Enschede, j.m.voogt@utwente.nl

Abstract: Despite intentions to the contrary, insights on pedagogically appropriate innovations with representative teachers in everyday school settings are severely limited. In part, this is because (design) research is often conducted at the bleeding edge of what is possible, exploring innovative uses of new technologies and/or emerging theories, while insufficient research and development work focuses on what is practical, today. This leaves a problematic gap between what could be useful research in theory, and what can be useful research in practice. This paper calls for (design) researchers to attend to factors that determine if and how innovations are understood, adopted and used by teachers and schools, and gives one example of how this was tackled in the domain of early literacy. Across ten studies, researchers collected data that helped shape an intervention that can be implemented by representative teachers, for diverse learners, in varied school settings.

Purpose
Educational (design) research strives to develop theoretical understanding that will ultimately improve the quality of teaching and learning. But even when research contributes to programs that are robustly designed and promising in specific settings, most prove difficult to scale up (cf. Penuel, Fishman, Cheng, & Sabelli, 2011). In part, this is because research is conducted at the bleeding edge of what is possible (i.e., exploring new technologies and/or emerging theories). There is no disputing that such work is greatly needed to seek out new ways to potentially enhance the quality of teaching and learning. However, in the excitement of exploring what is possible tomorrow, there is insufficient research and development work focusing on what is practical today. This leaves a problematic gap between what could be useful research in theory, and what can be useful research in practice. With the aim of generating „usable knowledge‟ (cf. Lagemann, 2002) and creating innovations that truly serve teaching and learning in practice, this contribution calls for researchers to devote attention to not only fine-grained issues of pupil learning and instruction, but also to broader factors that determine if and how innovations are understood, adopted and used by teachers and schools. Allowing these issues to steer design is necessary to yield innovations that can feasibly be implemented outside of (often highly enabling) research and development trajectories. We argue for more research conducted from the perspective of actual implementation and provide an example of how this was accomplished across nearly a decade of design research related to early literacy.

Toward Relevant and Useful (Design) Research
Revisiting a Familiar Challenge
A hundred years ago, psychologists Hugo Musterburg and John Dewey called for a linking science that would connect theoretical and practical work. In the last 20 years, researchers in the field of curriculum have increasingly called for the development of theoretical understanding that can guide the design of educational innovation (van den Akker, 1999; Walker, 1992). At the same time, researchers in the learning sciences have emphasized the need for research to be situated in authentic contexts to understand learning as it naturally occurs and specifically, what that means for how we shape education (Bransford, Brown, & Cocking, 2000; Brown, 1992). In addition, researchers have stressed the need for innovations to fit into the complex and diverse systems in which they are implemented (Hall & Hord, 2010). Doing so requires developing theories and models to underpin such innovations (Penuel, et al., 2011). Such work must be fed by research and development investigations that are tightly connected to practical realities such as dominant curricula; school cultures; high stakes assessments; interest and expertise of teachers (McKenney, Nieveen, & van den Akker, 2006). They must also be sufficiently problematized to focus efforts not on quasi-problems (e.g., “our teachers need ideas for how to use the iPads® we gave them”), but on problems that are urgent enough to warrant scientific investigation to yield solutions and/or theoretical understanding that can help solve them (McKenney & Reeves, 2012).

The Zone of Proximal Implementation (ZPI)
For research to be relevant and useful, it must reflect an understanding of practical realities and concerns of the domain studied. Studying the status quo of teaching, learning and settings, and designing innovations such that

© ISLS

96

ICLS2012

Volume 1: Full Papers

they gradually bridge from the current situation to the desired situation, is essential to developing both the knowledge and the tools required to address real needs in today‟s classrooms. This perspective has been referred to as the Zone of Proximal Implementation (ZPI; McKenney, 2011). Vygotsky‟s concept of the zone of proximal development – the distance between what learners can accomplish independently and what they can accomplish through guidance or collaboration – has previously been applied to large scale reform (Rogan, 2007; Rogan & Grayson, 2003); school leadership (McGivney & Moynihan, 1972); and the mediation of educational partnerships (Oakes, Welner, Yonezawa, & Allen, 1998). Similarly, others have referred to the need to pursue certain innovation goals in stepwise fashion, gradually moving from the current situation toward what is desired (Sullivan, 2004). A similar basic concept is applied to the design of educational innovations; but rather than focusing on what can be achieved by learners, it focuses on what can be implemented by teachers and/or schools. The ZPI refers to the distance between what teachers and schools can implement independently and what they can implement through guidance or collaboration (McKenney, 2011). Designing for the ZPI means explicitly tailoring products and processes to fit the capabilities, opportunities and limitations present among teachers and in schools. It also means planning for implementation scaffolding (e.g., honoraria or researcher co-teaching) to fade away in a timely fashion, while simultaneously developing the ownership and expertise among practitioners that will engender the desire and ability to sustain innovation. This is done, in part, through responsive (and sometimes participatory) design, fed by insights concerning learners, practitioners and context.

Implementation-prone Designs
In their book on conducting educational design research, McKenney and Reeves (2012) identify four characteristics of innovations that are prone to successful implementation, namely innovations that are valueadded, clear, compatible and tolerant. During the inception, creation and testing of educational innovations at the ZPI, these characteristics may be considered criteria to be met. Value-added innovations offer something better than what is already in place. Similar to Rogers‟ (Rogers, 2003) notion of the relative advantage, the potential benefits of value-added innovations visibly outweigh the investments required to yield them. Clear innovations enable participants to easily envision their involvement. Innovations may be clear through high levels of explicitness via a priori specifications of procedures and/or interactive mechanisms whereby developers and users co-define the innovation or elements thereof. Compatible innovations are congruent with existing values, cultures, practices and beliefs. They are innovative, but the innovations and/or their underlying assumptions do not violate or reject fundamental concerns and principles of those involved. Compatible innovations are also aligned with non-changeable aspects of the educational system, such as assessment frameworks or policies. Finally, tolerant innovations are those that “degrade gracefully” (cf. Walker, 2006) as opposed to yielding “lethal mutations” (cf. Brown & Campione, 1996) during the natural variation in enactment that inevitably comes along with differing contexts, resources, expertise, acceptance levels and so on. Tolerance refers to how precisely core components must be enacted for the innovation to be true to its goals, and how well an innovation withstands local adaptations. The remainder of this contribution describes how a multi-year (design) research series of studies was conducted within the zone of proximal implementation. Attention is given to how data were collected pertaining to the characteristics described above. The innovation was PictoPal: a technology-supported intervention designed to foster the development of emergent reading and writing skills in four and five year old children. Two main strands of inquiry have been attached to the PictoPal endeavor: technology for early literacy; and the influential role of teachers designing the PictoPal materials.

The PictoPal Studies
Technology for Early Literacy
Clay, (1966) emphasized that literacy begins long before school entry, calling this phenomenon emergent literacy. Underpinning this notion, which involves synergistic development of listening, speaking, reading, writing and viewing from birth, are several assertions which have been stressed by other experts as well. First, well-known theorists have long claimed that children play active roles in their own development (Bruner, 1983; Piaget, 1952; Piaget & Inhelder, 1969; Vygotsky, 1962). Clay‟s position that children are active learners about print long before they can read or write is consistent with this view. Second, Macnamara (1972) argued that language learning is driven by and dependent on the capacity to understand and participate in social situations. This is well-aligned with Clay‟s view that social interaction is the basis of emergent literacy. The computer‟s potential to promulgate discourse and thereby knowledge creation has been examined across various age ranges (McLoughlin & Oliver, 1998; Scardmalia, Bereiter, & Lamon, 1994). In terms of early childhood literacy, studies have shown that properly shaped collaborative use of the computer can contribute to pro-social behaviours, including: lively interactions, shared vocabularies, mutual enjoyment and spontaneous, active off-computer play (Brooker & Siraj-Blatchford, 2002; Van Scoter, 2008). In such ways, technology can serve as a catalyst for social interaction and contribute positively to fostering early literacy (Van Scoter, 2008).

© ISLS

97

ICLS2012

Volume 1: Full Papers

Yet teachers struggle to integrate technology with their classroom cultures (Labbo et al., 2003; Olson, 2000). This situation is exacerbated by a lack of high-quality emergent literacy materials (de Jong & Bus, 2003; Segers & Verhoeven, 2002). Appropriate software for fostering literacy skills in young children should be created in such a way that the learner‟s previous knowledge is taken into account, involve learners actively and encourage the use of language and the explorative nature and curiosity of young children (Brooker & Siraj-Blatchford; Plowman & Stephen, 2005). In addition, computer activities of young learners should be integrated with related classroom activities (Van Scoter, 2008) and embedded in appropriate pedagogical models for technology applications for young children (Plowman & Stephen, 2005). The PictoPal studies examine how the computer can contribute to an understanding of the nature and function of written language. Doing so requires that attention be given to the relationship between spoken and written language and the purposes for which reading and writing are used. As in oral language learning, children‟s concept formation regarding written language is driven by its use, in an environment rich with meaningful messages and functional print (Warash, Strong, & Donoho, 1999; Van Scoter, 2008). The main question shaping this strand of inquiry is “How can the technology-supported learning environment, PictoPal, contribute to helping kindergarteners understand the nature and function of written language?”

Teachers as Designers
Teacher involvement in the development of classroom curricula often fosters a sense of ownership, which increases the chances of actual curriculum use (Fullan, 2003). When teachers are supported during the design of innovative curricula, they can learn more about the innovation (Crow & Pounder, 2000), which also increases the chances of the implementation being successful. This is partly because teachers are then better informed, and able to visualize how curriculum enactment could look. Being able to „see‟ a curriculum in action is an important factor considered by teachers as they weigh off the amount of effort they invest and the potential benefits of the innovative curriculum (cf. Doyle & Ponder, 1978). Designing requires teacher time and effort, but also has the potential to improve implementation of an innovative curriculum. There are various ways to involve teachers in design and support that involvement. This strand of inquiry is concerned with answering the research question, “What forms and levels of involvement are feasible, and still yield the benefits of ownership, and understanding a new curriculum?”

Research approach The Designed Intervention
PictoPal is an ICT-rich learning environment with two main components: (a) on-computer activities through which pre-readers use words, sound and images to construct written texts; and (b) off-computer activities that prompt children to „use‟ their printed documents for authentic purposes. For example, children create grocery lists using the computer and then „shop‟ for the items on the printed list in the ´store´ corner of the kindergarten classroom. Alternatively, they prepare a weather forecast with the aid of the computer, and then „deliver‟ the forecast to their class from the television corner (from inside a „television‟ fashioned by the children from a large cardboard box). Figure 1 shows children, composing a recipe for vegetable soup (left) and then following the cooking instructions on their printed recipe (right). For more information about the PictoPal learning environment, please refer to McKenney and Voogt (2009).

Figure 1. Children Creating the Recipe on the Computer (left) and then Following it in the Classroom (right).

The Methods Used
The PictoPal studies have been underway for nearly a decade. Evolving through five different prototypes, ten different studies have been completed relating to the two strands of inquiry described. Comprehensive portrayal
© ISLS 98

ICLS2012

Volume 1: Full Papers

of the studies is therefore not possible here. Table 1 provides an overview of the studies including the prototype concerned, focus of the data collection in each study, methods used and the report reference for additional details. Table 1: Research overview of focus and methods in the 10 studies to date Study 1 2 3 4 5 Teacher/school habits, beliefs, values, resources Children‟s habits, attitudes Pupil learning gains Pupil engagement Integration Teacher experiences Teacher and child questionnaires Site visits, field notes Document analysis Pre/post-test On-pc observation Off-pc observation Teacher interviews (S. McKenney & Voogt, 2005) (S. McKenney & Voogt, 2010) (S. McKenney & Voogt, 2009) (S. McKenney & Voogt, in press) (Cviko, McKenney, & Voogt, 2012) Cviko, McKenney Voogt, 2010) Cviko, McKenney, Voogt, 2011) A B C D E F G H I J

Prototype

Focus

Methods

Reporting

Selected Results Practical Innovation: How the Research Informed Intervention Design
Throughout the series of studies, data were collected that informed the design of both PictoPal and its accompanying supportive mechanisms (teacher guides, professional development workshops). In hindsight, it is clear that the data contributed to developing an intervention that was, as characterized earlier, value-added, clear, compatible and tolerant. Table 2 provides an overview of how the research helped to develop PictoPal within the zone of proximal implementation. Table 2. Research methods used to feed design within the zone of proximal implementation Pre-design (needs/context analysis) Value-added (better than status quo) Clear (participants can envision their involvement) Site visits to see learning practices and ask about problems, in the baseline situation Teacher interviews to explore mindsets, habits and conventions within the classroom/school in the baseline situation Teacher interviews and child During development (prototyping and formative evaluation) Pre/post-tests of pupil learning during use With stable design (used as means to study teacher design practices) Pre/post-tests of pupil learning with implementation scaffolds removed Teacher interviews to explore mindsets, habits and conventions within the classroom/school that are sustained or changed after the innovation Teacher interviews to

Teacher interviews to explore mindsets, habits and conventions within the classroom/school during use Field notes concerning

Compatible

© ISLS

99

ICLS2012

Volume 1: Full Papers

(compatible with values, beliefs, surrounding educational context/system) Tolerant (withstands the natural variation of actual use)

questionnaire to explore values, cultures, beliefs, priorities, and contextual /system factors in the baseline situation Site visits and field notes of actual behaviors of teachers and learners and reasons for them in the baseline situation

values, cultures, beliefs, priorities, and contextual /system factors that help or hinder implementation

Observation of teachers and learners (and conjectured reasons for their conduct) during use

explore values, cultures, beliefs, priorities, and contextual /system factors that are sustained or changed after the innovation Observation of teachers and learners, and document analysis of both teacher products and learner products resulting from use with implementation scaffolds removed

Relevant Theoretical Understanding: Insights on Developing Technology for Early Literacy
This decade of design studies has yielded multiple reports and different kinds of insights. The papers cited in Table 1 offer detailed descriptions of findings and the data supporting them. In this section, examples of theoretical understanding emerging from the PictoPal studies are given: - Related to shaping technology for early literacy, several important understandings include: o Combining on- and off-computer activities can significantly influence children‟s understanding of functional literacy. o Children engage in dialogue about their work on and off the computer with peers or adults, though the quality of the conversation differs. o Once the software has been mastered (about two turns), new elements can be introduced into the learning environment (e.g., contrary to initial hypotheses, there is no need to teach the image-based vocabulary). o Children can use PictoPal semi-independently; the limited guidance they need is best given by an adult, but can be handled by an older child (e.g. Grade 6). - Related to understanding the influence of teachers as designers, several important understandings include: o Curricular ownership is positively related to the level of technology integration. o Kindergarten teachers tend to limit new initiatives in the classroom - even if they support them until a safe, trusting, routine and predictable classroom climate has been firmly established. o Well-structured, even modest design involvement fosters curricular ownership, facilitating implementation. o Providing tailored support can build teacher understanding and endorsement of core ideas, while freedom and creativity should be encouraged to develop different manifestations of those core ideas.

Reflections
The PictoPal studies were carried out over nearly a decade. After an initial needs and context analysis, the evaluation of early prototypes could be characterized as proof-of-concept work, focusing especially on the soundness and feasibility of the envisioned intervention. Thereafter, additional prototypes were developed and iteratively tested. During this work, implementation concerns received particular attention. Once the design stabilized, a more extensive look was taken at issues that could potentially increase effectiveness. Earlier work thus fed the research on shaping technology for early literacy. After one study (Study D in Table 1) showed substantial learning gains using materials that had been created by teachers themselves, research turned more explicitly toward exploring teachers as designers, how ownership is created through the design process, and its resulting influences in technology integration. Pupil learning was still examined, but mostly in light of these processes. The PictoPal intervention was designed to be usable in a wide range of schools, by different kinds of kindergarten teachers, with diverse groups of learners. It has also been tested under varying conditions. The fact that teachers in several schools have continued using PictoPal, even outside of the research activities and with no external support suggests that it did fall within their ZPI. This is likely due, in part, to being value-added, clear, compatible and tolerant. The research findings indicate that PictoPal grew to become value-added because, in addition to regularly resulting in significant pupil learning gains, teachers appreciated how it addressed a known and disconcerting gap between the existing language curriculum used in schools and the national interim targets for early literacy. Use of PictoPal became clear to teachers through their involvement in designing content, and through examples, passed along from previous teachers (e.g. through video), of how PictoPal came to life in the classroom. While PictoPal was extremely innovative, being joined by very few other technologies for early literacy that support the development of „discursive prowess‟ (Lankshear and Knoebel, 2003), it was also

© ISLS

100

ICLS2012

Volume 1: Full Papers

designed to be compatible with existing values, cultures, beliefs and priorities that are shared by most kindergarten teachers, e.g. that children learn through play (dramatic play features frequently in the off-computer activities); or that children shape understanding about the world through first-hand experiences (using printed texts for authentic purposes). The studies also explored different scenarios that would help teachers orchestrate the implementation of PictoPal, especially providing guidance during on-computer activities (e.g. adding an agent to the software, peer tutors, adult guides) in ways that are compatible with the options available in representative settings. Finally, PictoPal grew tolerant over time. As the intervention matured, it became clear which features had to be adhered to more strictly to achieve pupil learning (e.g. duration of intervention; structure and layout of words in the software grid; and a gradual increase in difficulty level); and which ones could be tailored by teachers to their own preferences, themes or inspiration. The need for researchers to seek understanding about where teachers and schools are, how they perceive the problem(s) to be addressed, and to and frame innovations within a reachable distance, has been described in literature (e.g. Bielaczyc, 2006; Blumenfeld, 2000; McKenney & Voogt, in press). This paper further emphasizes the importance of explicitly collecting data that can attune both interventions and theoretical insights accordingly. It offers some examples of how that was accomplished in one case relating to technology for early literacy. Toward increasing the relevance and usefulness of educational (design) research, less examples are needed of what might be potentially possible, and more examples are needed of how to understand and design for what is realistically feasible: in the Zone of Proximal Implementation.

References
Bielaczyc, K. (2006). Designing social infrastructure: Critical issues in creating learning environments with technology. Journal of the Learning Sciences, 15(3), 301-329. Blumenfeld, P., Fishman, B. J., Krajcik, J., Marx, R. W., & Soloway, E. (2000). Creating Usable Innovations in Systemic Reform: Scaling Up Technology-Embedded Project-Based Science in Urban Schools. Educational Psychologist, 35(3), 149-164. Bransford, J., Brown, A., & Cocking, R. (2000). How people learn: Brain, mind, experience and school. Washington, DC.: National Academy Press. Brooker, L., & Siraj-Blatchford, J. (2002). „Click on Miaow!‟: how children of three and four years experience the nursery computer. Contemporary Issues in Early Childhood, 3(2), 251-273. Brown, A. L. (1992). Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. Journal of the Learning Sciences, 2(2), 141-178. Brown, A. L., & Campione, J. (1996). Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. In L. Schauble & R. Glaser (Eds.), Innovations in learning (pp. 289-325). Mahwah, NJ: Earlbaum. Clay, M. (1966). Emergent Reading Behaviour. Unpublished doctoral dissertation, University of Auckland, New Zealand. Crow, G.M., & Pounder, D. G. (2000). Interdisciplinary Teacher Teams: Context, Design, and Process. Educational Administration Quarterly, 36, 216-254. Cviko, A., McKenney, S. & Voogt, J. (2010). Docenten als ontwerpers en uitvoerders van een ICT-rijk curriculum voor beginnende geletterdheid. Round table presentation at the ORD annual meeting, June 23-25: Enschede. Cviko, A., McKenney, S. & Voogt, J. (2011). Teachers as (re-)designers of an ICT-rich learning environment for early literacy. Paper presentation at the ECER annual meeting, September13-16: Berlin. Cviko, A., McKenney, S., & Voogt, J. (2012). Teachers enacting a technology-rich curriculum for emergent literacy Educational Technology Research and Development, 60(1), 31-54. doi: I 10.1007/s11423-0119208-3. de Jong, M. T., & Bus, A. G. (2003). How Well Suited are Electronic Books to Supporting Literacy? Journal of Early Childhood Literacy, 3(2), 147-164. doi: 10.1177/14687984030032002 Doyle, W., & Ponder, G. (1978). The practicality ethic in teacher decision-making. Interchange, 8(3), 1-12. Fullan, M. (2003). Change forces with a vengeance. London: RoutledgeFalmer. Hall, G., & Hord, S. (2010). Implementing change: Patterns, principles and potholes. Upper Saddle River, NJ: Prentice Hall. Labbo, L., Leu, D., Kinzer, C., Teale, W., Cammack, D., Kara-Soteriou, J., & Sanny, R. (2003). Teacher wisdom stories: Cautions and recommentaions for using computer-related technologies for literacy instruction. The Reading Teacher, 57(3), 300-304. Lagemann, E. (2002). An elusive science: The troubling history of education research. Chicago: University of Chicago Press. Lankshear, C., & Knoebel, M. (2003). New technologies in early childhood literacy research: A review of research. Journal of Early Childhood Literacy, 3(1), 59-82. Macnamara, J. (1972). Cognitive basis of lanaugage learning in infants. Psychological Review, 79, 1-12.

© ISLS

101

ICLS2012

Volume 1: Full Papers

McGivney, J., & Moynihan, W. (1972). School and community. Teachers College Record, 74, 209-224. McKenney, S. (2011). Designing and researching technology enhanced learning fort he zone of proximal implementation. Invited paper presentation at the Art and Science of Learning Design workshop. September 13-14: London Knowledge Lab, London. McKenney, S., & Reeves, T. (2012). Conducting Educational Design Research. London: Routledge. McKenney, S., & Voogt, J. (2005). Using ICT to foster (pre) reading and writing skills in young children. Paper presented at the American Educational Research Association, Montreal. McKenney, S., & Voogt, J. (2009). Designing technology for emergent literacy: the PictoPal initiative. Computers & Education, 52, 719-729. McKenney, S., & Voogt, J. (2010). Technology and young children: How 4-7 year olds perceive their own use of computers. Computers in Human Behavior, 26, 656-664. McKenney, S., & Voogt, J. (in press). Teacher design of technology for emergent literacy: An explorative feasibility study. Australasian Journal of Early Childhood. McKenney, S., Nieveen, N., & van den Akker, J. (2006). Design research from the curriculum perspective. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 67-90). London: Routledge. McLoughlin, C., & Oliver, R. (1998). Maximising the language and learning link in computer learning environments. British Journal of Educational Technology, 29(2), 125-136. doi: doi:10.1111/14678535.00054 Oakes, J., Welner, K., Yonezawa, S., & Allen, R. (1998). Norms and policies of enquiry-minded change: Reaching the 'zone of mediation'. In A. Hargreaves, A. Liebermann, M. Fullan & D. Hopkins (Eds.), International handbook of educational change (pp. 952-975). Dordrecht: Kluwer. Olson, J. (2000). Trojan horse or teacher‟s pet? Computer and the culture of the school. Journal of Curriculum Studies, 32, 1-8. Penuel, W., Fishman, B., Cheng, B., & Sabelli, N. (2011). Organizing research and development ad the intersection of learning, implementation and design. Educational Researcher, 40(7), 331-337. Piaget, J. (1952). The Origins of Intelligence in Children. New York: W. W. Norton. Piaget, J., & Inhelder, B. (1969). The psychology of the child. New York: Basic Books. Plowman, L., & Stephen, C. (2005). Children, play and computers in pre-school education. British Journal of Educational Technology, 36(2), 145-157. Reeves, T. (2000). Enhancing the worth of instructional technology research through “design experiments” and other development research strategies. Paper presented at the Annual AERA Meeting, New Orleans. Rogan, J. (2007). How much curriculum change is appropriate? Defining a zone of feasible innovation. Science Education, 91, 439-460. Rogan, J., & Grayson, D. (2003). Towards a theory of curriculum implementation with particular referenc to science education in developing countries. International Journal of Science Education, 25, 1171-1204. Rogers, E. (2003). Diffusion of innovations (5th ed.). New York: Free Press. Scardmalia, M., Bereiter, C., & Lamon, M. (1994). The CSILE Project: Trying to bring the classroom into World 3. ). : . In K. McGilly (Ed.), Classroom lessons: Integrating cognitive theory and classroom practice (pp. 201-228). Cambridge, MA: MIT Press/Bradford Books. Segers, E., & Verhoeven, L. (2002). Multimedia support of early literacy learning. Computers in Education, 39(3), 207-221. Sullivan, M. (2004). The reconceptualisation of learner-centred approaches: a Namibian case study. International Journal of Educational Development, 24, 585-602. Tebbutt, M. (2000). ICT in science: problems, possibilities and principles. School Science Revie, 81(297), 57-64. van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. Branch, K. Gustafson, N. Nieveen & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1-14). Dordrecht: Kluwer Academic Publishers. Van Scoter, J. (2008). The Potential of IT to Foster Literacy Development in Kindergarten. In J. V. Knezek (Ed.), International Handbook of Information Technology in Education (pp. 149-161). London: Springer. Vygotsky, L. (1962). Thought and language. Cambridge, MA: MIT Press. Walker, D. (1992). Methodological Issues in Curriculum Research. In P. W. Jackson (Ed.), Handbook of Research on Curriculum (pp. 98-118). New York: Macmillan. Walker, D. (2006). Toward productive design studies. In J. Van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. ZZ-XX). London: Routledge. Warash, B., Strong, M., & Donoho, R. (1999). Approaches to environmental print with young children. . In O. G. Nelson & W. M. Linek (Eds.), Practical classroom applications of language experience: Looking back, looking forward (pp. 53-58). Boston, MA: Allyn & Bacon.

© ISLS

102

ICLS2012

Volume 1: Full Papers

Kitchen Chemistry: Supporting Learners’ Decisions in Science
Jason C. Yip, Tamara Clegg, Elizabeth Bonsignore, Becky Lewittes, Mona Leigh Guha, and Allison Druin, Univeristy of Maryland – College Park, Human-Computer Interaction Lab, 2117 Hornbake Bldg, South Wing, College Park, MD 20742 Helene Gelderblom, University of South Africa, School of Computing, PO Box 392, Pretoria, South Africa, 0003 jasonyip@umd.edu, tclegg@umd.edu, ebonsign@umd.edu, geldejh@unisa.ac.za, charley@cs.umd.edu, mona@cs.umd.edu, allisond@umiacs.umd.edu Abstract: Students often find science to be disconnected from their everyday lives. One reason for this disengagement is that learners are often not given the chance to choose how to pursue their personal goals using science reasoning. Therefore, we are creating science programs that emphasize life-relevant learning - the ability to engage science learners in the context of achieving their own goals. We developed Kitchen Chemistry to engage and support children in the design of their own personal investigations. In this paper, we use a case study analysis to examine three groups of learners in Kitchen Chemistry. We analyze the decisions that learners make, how learners make these decisions, and the supports needed to make informed choices. We examine how the use of semi-structured activities, whole group discussions, adult facilitation, and mobile technologies interact and support learners in their decision-making practices.

Introduction
Researchers have documented that some youth find aspects of traditional school science to be disconnected from learners’ daily lives and interests (e.g., Atwater, 1996). Indeed, within the didactic instructional model, particularly within US schools, abstract knowledge is often presented without making direct connections to learners’ lives outside the classroom. When science learning does not make connections to learners’ interests and experiences, students may have pervasive negative views of science (e.g., Basu & Barton, 2007). We believe that one reason for this disengagement is that learners are often not given the chance to make choices and develop agency in how to pursue their personal goals using science reasoning. A challenge now facing educators is the development of learning activities that connect learners’ personal goals to science knowledge and processes. This is important because all citizens must be scientifically literate and be able to reason well about complex evidence to make educated decisions about important issues, such as health and environmental policies (Chinn & Malhotra, 2002). We aim to address this issue of science disengagement through the development a life-relevant learning (LRL) environment called Kitchen Chemistry (KC). LRL environments are learning environments that engage learners in science through the pursuit of personally meaningful and relevant goals, in either formal (e.g., Bouillion & Gomez, 2001) or informal learning contexts (e.g., Fusco, 2001). One everyday context we believe can support science learning is the kitchen. Therefore, we developed KC as a guided-inquiry LRL program that allows participants to learn science and engage in scientific practices within the context of cooking (e.g., Clegg, Gardner, & Kolodner, 2010). Through semi-structured activities and discussions, learners use their science knowledge to later develop their own personal investigations into cooking. We assert that learners are capable of playing an active role in designing their own avenues of scientific inquiry. However, having learners design scientific investigations based on their own personal goals is no easy task. Learners often need support as inquiry tasks become more complex (e.g., Reiser, 2004). We argue that giving learners agency in how they make decisions in inquiry practices is an important means of getting learners engaged in science. We believe there is much importance in supporting learners’ choices in science, but that we need to be mindful of what supports are needed to allow for appropriate science choices. Therefore, in this paper, we ask what supports are needed for learners to make informed personal choices about the designs of their investigations within a guided-inquiry based science-learning environment. We seek to find out 1) what decisions learners make in the development of their own investigations; 2) how learners come to make such decisions; and 3) what aspects of the learning environment support these choices.

Theoretical Framework
To help engage learners in science, we must bridge the gap between science knowledge and learners’ lifeworlds (Aikenhead, 1996). We have to provide a means for learners to engage in an authentic scientific experience in the context of their own personal goals (Clegg et al., 2010) and give learners a chance to make decisions in what they want to pursue. We recognize that while we need learners to make personal choices to accomplish their goals, they need help developing the skill set to make decisions to achieve those goals. Indeed, learners come

© ISLS

103

ICLS2012

Volume 1: Full Papers

with diverse funds of knowledge that are rooted in their everyday-life experience and participation in their community (e.g., Barton & Tan, 2009). However, much of the science experience in schools does not acknowledge learners’ own personal goals and resources. Often, reforms in school science push learners into the unfamiliar world of scientists, as opposed to engaging learners in their own culture (Aikenhead, 1996). In particular, the lack of relevance of science knowledge to learners’ lives leads to these higher levels of disinterest in later years of schooling (e.g., Aikenhead, 1996). Much of school science focuses on simple experiments and tasks that bear little resemblance to professional and authentic science. Often these simple science tasks do not connect to learners’ own funds of knowledge and can promote an unscientific epistemology (Chinn & Malhotra, 2002). More authentic science learning gives learners a chance to solve problems through designing investigations and complex experiments, collaborating with others, considering multiple variables for testing, and establishing rigor through checks and balances. Because authentic science learning requires learners to socially construct knowledge, we wanted to promote a participatory culture in KC. Jenkins, Clinton, Purushotma, Robinson, and Weigel (2006) define a participatory culture as a culture “with relatively low barriers to artistic expression and civic engagement, strong support for creating and sharing one’s creations, and some type of informal mentorship whereby what is known by the most experienced is passed along to novices” (p. 3). In a participatory culture, learners believe that their contributions matter and they develop social connections between each other. To help support this participatory culture, we developed cultural practices based on our experiences with two frameworks for design: Cooperative Inquiry (e.g., Druin, 2005) and Learning By Design (Kolodner et al., 2003). In Cooperative Inquiry, children are partners with adults in the design process. From our previous work with Cooperative Inquiry, we established customs that attempted to minimize existing power structures. Our team insisted on three basic practices: sitting together in a circle with children in discussion, calling each other by our first names, and speaking in ordered turn without raising our hands. From Learning by Design, we adhered to two principles. First, we adhered to the culture of iteration: members of the community develop the responsibility for helping each other learn through critique of one another and iteration on their designs. Second, we supported scientific reasoning: members of the community must adhere to principles of science and causality in explanations and refer to evidence to back up claims.

Design of Kitchen Chemistry
KC is an after-school or summer camp program where learners engage in scientific practices within the context of cooking. Holding the program outside of school enables learners to choose the directions of their scientific inquiry without being bound to a particular curriculum. To provide an environment in which learners can participate in scientific practices to design investigations that are personally meaningful to them, we developed two activity sequences. First, learners engage in semi-structured activities that help familiarize them with cooking and science practices. In the cooking practices, learners are given the tasks such as observing what eggs do in brownies and what leaveners (e.g., baking powder, baking soda) do in cookies. In these cooking experiments, learners vary the amounts of eggs and types of leaveners in the recipes and examine the results. The semi-structured activities also include non-cooking experiments in which the ingredient variations in their cooking experiments are highlighted specifically to help learners think about the underlying scientific phenomena. For example, learners mix different amounts of eggs into fixed amounts of oil and water to observe how eggs act as emulsifiers in the mixture. They also compare the heights and amount of foam that are generated from shaking the egg, water, and oil mixture. For the experiment on leaveners, participants mix different amounts of baking soda, baking powder, and cream of tartar into water and lemon juice and observe the amount of bubbles generated from the mixture. These experiments help learners to see how eggs act as emulsifiers to bind together water and oils in the brownies and how different leaveners incorporate gas bubbles into dough. Learners participate in these semi-structured activities to prepare for the second activity sequence flexible exploratory activities that we called Choice Days. Learners are given the opportunity to use what they have learned to prepare an investigation into a recipe of their choice. Here, learners make decisions about their investigations, such as which recipes to explore, what modifications to make to ingredient amounts, and what observations they will make. Embedded in both the semi-structured activities and Choice Days are whole group conversations in which learners further discuss and reason about the observed phenomena. To develop a community that invites participation, both adults and learners sit together in a circle. We discuss what we observed, our thoughts on the results, and what we think about the outcomes. We also discuss authentic scientific practices (Chinn & Malhotra, 2002), such as making qualitative and quantitative observations, tying evidence to claims and reasoning, thinking about the underlying mechanism that causes the observed phenomenon, and considering how independent variables may change multiple dependent variables. All participants work together to think about the underlying phenomenon and build on each other’s scientific arguments. Learners also use mobile technologies on the iPad™ to conduct their investigations and reflect on their observations. In particular, learners use StoryKit (e.g., Quinn, Bederson, Bonsignore, & Druin, 2009) and Zydeco (e.g., Cahill, Kuhn,

© ISLS

104

ICLS2012

Volume 1: Full Papers

Schmoll, Pompe, & Quintana, 2010). StoryKit is an iPhone™ application for creating and sharing audio-visual stories through text, photos, drawings, and audio recordings. Zydeco is also an iPhone™ application that can be used to photograph, tag, and annotate information within different contexts. From this collective information, users can develop claims, evidence and reasoning. We use both StoryKit and Zydeco together to support learners’ scientific practice in the context of choice-based activities and to help them reflect during and after the investigations on the scientific aspects of the activities (Clegg, Gardner, & Kolodner, 2011).

Methods
For this study, we used the standards of a comparative case study (Yin, 2009) of a one-week implementation of KC. We employed a multiple case study design, that is, within the single implementation there are multiple units of analysis. In this exploratory case study, we are comparing three different groups of learners within the same context. We examined the decisions that learners made, how learners made these decisions, and what factors supported their decision-making practices.

Context and Data Collection
KC was implemented as an all day, one-week summer program at a local private school. Six to nine learners between the ages of 9 to 13 participated in the program each day. The participants were from public and private schools in the area. Each day, we collected video recordings of all semi-structured activities and whole group discussions. Facilitators also recorded post-observation field notes that described what we thought were the most significant aspects of the day. We collected learners’ entries in the Zydeco and Storykit software systems. We conducted interviews with four learners who agreed to do interviews to understand their perspectives on their experiences with the program, activities, and technology.

Criteria for Case Selection and Data Analysis
Since we are interested in the decision-making processes of the learners, we selected three of the four Choice Day groups as cases. These are three cases for which we had sufficient data recordings (we were unable to fully record the fourth case). All learner names are pseudonyms. We began the data analysis through an initial examination of the interviews, videos, software artifacts, and facilitators’ field notes to find evidence that links to our questions on decision-making. During this time, we wrote content logs of the videos and transcribed salient quotes. As particular segments emerged as significant, we expanded the content logs into transcriptions. Since the unit of analysis was the small group, we coded instances when the groups or interviewees talked about group-based choices, how the groups made a decision, and the outcomes of their decisions. We triangulated the data to make sure that all pieces of evidence supported each other (Merriam, 2009). From our answers, we developed a formal assembly of the evidence into a case study database in which field notes, transcripts, and software artifacts were put online for each author to independently examine (Yin, 2009). Using this database, we maintained a chain of evidence for each embedded case that builds connections between the variety of evidence, the questions we had on decision-making and our initial conclusions. To establish validity, we presented each case to the corresponding facilitator to ensure that the cases were representative of the learners and their decision-making practices. Once each case was thoroughly developed, we conducted a cross case analysis between the three embedded cases to see if similarities and patterns were present in the data.

Key Findings
We begin each case with a description of the activity that the learners and facilitators designed for Choice Day. We then analyze each case to understand what decisions learners made and how they came to make them.

Case 1: Vegan Brownies
For this Choice Day activity, Ari and Clyde, two 10-year-old boys, wanted to make vegan brownies. Ari was motivated to try this recipe because he had tasted a vegan brownie from a prior activity and found the taste to be displeasing. Ari decided to take up the challenge to make a vegan brownie that would taste better and be gooey and fudgy. Clyde decided to join Ari in this endeavor. Becky (facilitator) was given the task of supporting this group. What sets Becky apart from the other facilitators is that this session was her first time at KC. Therefore, Becky was learning about the prior activities and discussions as she was helping to facilitate. The lead facilitators provided the group with a vegan brownie recipe to try. However, the boys did not want to blindly follow a recipe whose outcome might not meet their “gooey and fudgy” goals. Instead, Ari and Clyde wanted to figure out if they should use the vegan brownie recipe or modify a non-vegan recipe. To help them decide, Ari and Clyde immediately chose to run an egg experiment using an egg substitute powder. To test the emulsifying properties, the learners made a mixture of oil and water and added the egg substitute powder. They added different amounts of egg substitute powder as their new independent variable. Their dependent variable was the amount of foam generated in the new mixture. They added the egg substitute in one-tablespoon

© ISLS

105

ICLS2012

Volume 1: Full Papers

increments, shook the bottle, and observed the height of the foam. Using two tablespoons of the egg substitute powder, Ari and Clyde found that the foam did not rise as much as when they used the real eggs. Based on these observations, the boys concluded that the egg substitute could be used as an emulsifier. Meanwhile, Becky observed them, asking questions about why they were engaged in this process. Ari and Clyde also referred to the prior StoryKit photographs to make comparisons between the new experimental foam and the prior egg experiments. The learners explained to Becky, who was not present for the original egg experiment, that the lower amount of bubbles from two tablespoons of egg substitute would create a more gooey brownie that was less leavened. Because they wanted that outcome, Ari and Clyde decided to go ahead and make modifications to the recipe to include two tablespoons of egg substitute.

Case 1: Analysis
We observed that the learners immediately chose to use the egg experiment with the new variable. Becky stated, "They knew to set up the experiment. And then they added the oil, the water, and the fake eggs." Here, the learners took aspects of the semi-structured activities and were able to tailor them to help them make their decisions. Second, although Becky was a first time visitor, she was able facilitate their decisions. Being new to the activity, Becky did not know all the answers, which she reported may have helped the whole group make decisions together. Although Becky spent a lot of time trying to keep Clyde on task, she also made sure to spend time in discussion with both learners to help them make their investigation decisions. For example, the vegan brownie group spent the most time of any group discussing their decisions. While Becky was not involved in the prior activities, she knew what questions they were asking and what decision points they were heading toward. Becky recalls “prompting them on making the decision between the two [original and new] recipes” and “thinking that they needed me to prompt a bit in terms of what those two things [leaveners and emulsifiers] were and like, making the decision.” While Becky was able to prompt Ari and Clyde on the decision points, the two learners also used the mobile technologies to help guide their choice. Ari and Clyde chose to use StoryKit as a means to reflect on the process of making their decisions. During the setup, Ari and Clyde ran their egg-oil-water experiment using the egg substitute and observed the foam generated from shaking the mixture. Once a slight bit of foam was created, Becky noticed that Ari and Clyde were flipping through prior photos and recordings from StoryKit of the foam from the prior egg-oil-water experiment. They referred back to these photos of the foam results to compare with the current foam results they had with the egg substitute to decide if the egg substitute would be a good emulsifier. From these comparisons between the prior data and their new data, the learners made the decision to use their modified investigation.

Case 2: “Togo” Cakes
For the entire week, Lisa, a 10-year-old girl, consistently reminded us that she had a brilliant idea to make “Togo” Cakes (a name she coined herself, derived from “to go”). Instead of normal pancakes with the syrup drenched all over, Lisa wanted to develop a pancake that could enclose the syrup in the middle. This way, she could toast a frozen Togo Cake that would have a pocket of syrup in the middle so that she could eat it on the go. Lisa easily decided to choose Togo Cakes as the focus of her investigation. The more difficult decision was how to bring this project to fruition. Working with Tammy (facilitator), Lisa first had to determine what she wanted from a pancake recipe. She writes in StoryKit, “We want to find out: how much baking powder we need for it not to be as fluffy and still taste right. We don't want it to be fluffy because we don't want the syrup to absorb into the pancakes.” Lisa wanted to design a pancake with two features. First, the pancake could not be too fluffy or the syrup would absorb into the pancake. Second, the pancake had to have a crispy top so the syrup would not leak out. For the crispy top, Lisa chose to use three eggs because in their brownie experiment, she observed that the three-egg variation had a harder crust. Next, Lisa and Tammy ran a series of experiments to determine the amount of baking powder to use in their pancakes. They heated varying amounts of baking powder in water, observing how much foam the “cooked” solution generated before deciding that 1¾ teaspoon produced the desired amount of foam and fluff. Lisa also came across a new problem in her investigation. As part of her engineering task, she needed to figure out a way to put the syrup into the middle of the pancake. Tammy and Lisa tried to freeze the syrup on top of a small plate. They found, however, that maple syrup does not freeze very well. Lisa decided to try another viscous and sweet liquid that would be similar to maple syrup. She designed another set of experiments to test out whether honey, corn syrup or maple syrup would freeze enough so that she could lift it off the plate and place into the middle of the pancake. Placing the three liquids side by side in the freezer, Lisa determined that honey had the consistency and solidity that worked for her Togo Cakes. Lisa and Tammy concluded their Storykit story, “Success at last, we got honey in the pancakes without absorbing.” In the end, Lisa found the Togo Cakes were successful; the final product had the right crispiness and fluffiness and was able to hold the honey in the middle.

© ISLS

106

ICLS2012

Volume 1: Full Papers

Case 2: Analysis
Based on the tasks, Lisa had to make several key decisions in her investigation. She did not make haphazard guesses, but instead, had a very methodical approach to her design. First, Lisa depended on the semi-structured activities to develop her series of experiments. In particular, she used the leavener experiment in which learners had to make observations about different combinations of leaveners (e.g., baking soda, baking powder, cream of tartar) and whether they reacted with lemon juice, vinegar or water to produce bubbles. Learners also made observations of how warmer temperatures affected these reactions. Without hesitation, Lisa gravitated towards conducting an experiment on baking powder alone, to determine how much she would use in her recipe. The choice of using baking powder, as opposed to baking soda, may have also come from her experience from baking cookies using baking soda. Her use of baking soda in her cookies resulted in a very flat and unleavened product. Lisa stated in one of the discussions, “the baking soda needs an acid to react because it was a base” and that “the baking soda didn’t react until we added vinegar to the cup experiment.” Her choice in using baking powder for this design is reflected in her thoughts that baking powder added, “a little more acid” to the bicarbonate to produce the bubbles. Many of these statements of Lisa’s understanding of the ingredients emerged during our discussion time. Lisa also used StoryKit as a way to organize her investigation. For example, in Lisa’s story, there are multiple pictures and descriptions of the three sets of experiments she performed. Each photo showed the foam coming out. Lisa wrote about her decision, “This one [referring to 2½ teaspoons] boiled over (not quite as high as the 3-1/2 tsp) too! So we’re going to use 1-3/4 tsp. of baking powder.” We saw that Lisa’s decision to use 1¾ teaspoon of baking powder was done carefully and methodically. Although Lisa was very organized, we observed that Tammy, the facilitator, still helped her sort through her decisions. Tammy recalls that she prompted Lisa to think about what goals she had, “It was sort of like thinking about what goals she had for it. …She knew she wanted it (pancake) to be thicker, thicker than normal, and that was why she thought about it more, the leaveners." Lisa told Tammy the issues and the goals that she had. Tammy felt that she wanted to help her “find solutions” to meet that goal. This meant that as a facilitator, Tammy helped to plan out ways to adapt the semi-structured activities to help meet Lisa’s goals.

Case 3: Red Velvet Cake
Denise, a 10-year-old girl, choose to make red velvet cakes, because her aunt made very good red velvet cakes. Lily and Meg (10- and 13-year-old respectively) decided to join Denise in her endeavor. The girls and their facilitators, Helene and Beth, discussed how they were going to make this into a science investigation. Denise initially did not want to think about the science behind baking this cake; she just wanted to make a cake as delicious as her aunt’s. Compared with Cases 1 and 2, Denise talked more about liking cake and icing, as opposed to wanting to conduct an experiment to determine ways to make her cake better. Based on this motivation, Beth and Helene prompted the group with questions about the choices they could make in the cake. They asked the learners questions about how the recipe could be modified to achieve their desired results in terms of texture and taste. The adults recalled structuring the development of the red velvet investigation using a goals chart that prompted learners to think about what choices they want to make and what outcomes they predicted would occur. Learners recorded information onto the goals chart, such as taste, texture, mouthfeel, handfeel, smell, and look, and what leavener proportion they thought would achieve these outcomes. Using StoryKit, Meg took a photo of the goal sheet, and the group worked together using the iPad™ to complete the goals chart. Based on these prompts and the structure of the goals chart, the learners chose to compare a recipe they found online and a modified recipe by adding eggs and substituting baking powder for vinegar and baking soda. The group realized that they only had enough resources to make two smaller comparison cakes, so they decided to cut the recipe in half. To make these modifications, the learners relied on their knowledge from the prior experiments. For example, Denise decided that since the ingredients in the brownies recipe were similar to the red velvet, she chose two eggs (doubling the original recipe) for the modification because she predicted this would yield a more moist cake (one of her recipe goals). The learners and facilitators made an additional modification to their experimental recipe by replacing the original vinegar and baking soda ingredients with baking powder. Since baking powder is both dry acid and base combined, the group wanted to see if this ingredient would produce the same leavening effects as vinegar and baking soda. As the group made progress through their Choice Day efforts, each learner raised more scientific questions. For example, Denise began to inquire why different icing mixes exhibit different consistencies (e.g., “the cream cheese icing is thicker and not as melty as the butter cream”). Lily also made more objective observations about the outcome of the cake. Instead of making subjective statements about whether the cake was yummy or tasty, Lily noticed that, “one (the experimental) has more bubbles than the other, well not more bubbles, but they’re bigger than the other one (original outcome).” In their StoryKit story, the group concluded that the experimental cake with “more eggs make cake-ier, heavier, more packed cake.” The group recorded comparative height measurements of each baked cake, noting that the original recipe rose to 3 cm, while the experimental one rose to 4.9 cm.

© ISLS

107

ICLS2012

Volume 1: Full Papers

Case 3: Analysis
We observed that the learners’ prediction of the outcome of these modifications was tied to their participation in the semi-structured activities. The learners used past knowledge from these activities to make their choices. Even though the group did not conduct an experiment, they determined to use two eggs in the recipe because of the connection to the prior brownie activity. For example, in a prior StoryKit story, the learners wrote that the addition of more eggs produced brownies that were, “Spongy and sticky. A lot more like cake than like brownie.” Because they wanted a cake that is spongier, the learners made the modification to double the amount of eggs that was originally called for. However, for the replacement of baking soda and vinegar with baking powder, the adults worked carefully with the learners using the goals chart as a “catalyst” that helped them to structure their initial discussion so that they could ask the learners what they wanted in the cake. We gave each group the goals sheet to help them work through their decision-making process. The goal charts helped the adults to structure the discussion. For example, the adults prompted the learners with questions, such as, “what do we want it (the cake) to taste like?”, “do cakey brownies have more eggs or less eggs?”, and “did it (the cookies) have more baking powder or less?”. For each of the outcomes on the goals chart, the learners wanted a product that was “cakey, moist, and soft.” From this, they made as many modifications as they could to make sure they would have a moist outcome.

Discussion
One of the primary goals of KC was to support learners’ agency and personal investment into inquiry based scientific practices. We wanted learners to walk away not just with content knowledge and better understanding of the social practices of science, but with the notion that learners can make informed choices in science. Four common themes from the cross-case analysis emerged that identified aspects of the learning environment that support learners’ personal decisions: 1) Semi-structured activities helped learners to think about how they might structure their own investigations. 2) Facilitation from adults allowed learners to think scientifically about what decisions needed to be made. 3) Participation in whole group discussions helped learners to reflect and build knowledge for their choices. 4) The practice of using mobile technology allowed learners to build personalized narratives for reflection and discussion to help make informed choices for their personal investigations. Although we have parsed out these four practices, we recognize that each of them interacts with one another to help learners make informed decisions to support their personal goals. Each of the themes is an extension of our vision of Cooperative Inquiry (e.g., Druin, 2005), that is, learners can be design partners within the development of a learning environment. Our four cultural practices helped to support learners’ choice in design because we attempted to take seriously the challenge that children can make viable decisions in their science learning. This culture of learner agency and partnership permeated throughout the week. Lisa describes KC as a place where “the kids can be adults, and the adults can be kids” and that “I was really excited, I was excited because I don’t get to do a lot of this in school. You don’t get to pick anything you want. I mean, they [teachers at school] give you a choice between one and two and half of the time, and like one and two and sometimes that’s not what you want at all.” Each of these four practices manifested and intertwined in three aspects of KC. First, the sequence of the activities gives learners a chance to develop ways to make decisions. Informed decision-making practices are not the sole responsibility of the learners, the facilitators, or the technology. Instead, our four cultural practices worked together to support these learning goals. We sequenced KC so that during the semi-structured activities and whole group conversations, learners would familiarize themselves with scientific and cooking practices in preparation for the decisions they would make in Choice Day. However, this sequence of activities would not have prepared the learners for decision-making if learners were not already engaged in this practice. For instance, using the mobile technologies in the activities, learners chose when to take photos, when to interview each other, and what belonged in their digital artifacts. During discussions, facilitators routinely worked with learners to help them make informed decisions about their arguments and claims. In the cases we have presented, the four practices work in tandem to help guide learners and their choices. Second, guidance and organization helps learners recognize what decisions they can make. In each of the three cases, while the sequence of the activities helped learners engage in the practices of inquiry to build independent investigations, we found that learners still needed help to use scientific reasoning to justify all choices. They often worked together with the adults and used the mobile technologies to guide them. During the activities, facilitators helped to break down their larger goals into smaller decision points. Facilitators prompted and reminded the learners of their goals and asked them what they wanted. Learners still had to make decisions, but justify those choices with evidence and reasoning. We observed from Cases 1 and 2 that mobile technologies helped to organize learners’ decisions to structure their investigations. From Case 1, Ari and Clyde
© ISLS 108

ICLS2012

Volume 1: Full Papers

used the digital artifacts from the prior activities to guide their decisions. In Case 2, Lisa used the technology to record and make comparisons between her leavening experiments to select which amount of baking powder to use. Lisa states, “I like StoryKit because it lets you use sequence. And I’m a big fan of sequence because it helps you get in order.” We recognize, however, that there is a tension between guidance and learners’ choices. Our description of Case 3 illustrates this point. Initially, Denise’s group was just interested in baking the cake, as opposed to engaging in scientific practices and understanding the phenomenon. We argue that choice in science can be an entry point for engagement and participation in science learning. However, Furman and Barton (2006) caution that giving learners choice means that facilitators need to work with learners in how to make choices in an informed way. The role of the facilitator is not just to prompt for decision points, but also to challenge learners to provide evidence or reasons in a way that makes their choices stronger. In our case, Beth and Helene attempted to work with the girls to bake their cake and help the learners make well-informed modifications and comparisons to the recipe. Lastly, opportunities for reflection are an important means for learners to make informed decisions. We argue reflection supports learners’ decision practices in two ways. First, learners needed to work together and listen to each other’s arguments. Part of the practices we put forth in KC are intended to help learners to focus more attention on group deliberation to ensure all members listen and learn from each other’s ideas and arguments. Learners had to feel secure enough to ask questions, make claims, and provide evidence for their reasoning. During whole group discussion time, adults, and learners worked together to come up with viable explanations for the observed phenomena. Both children and adults comfortably built on each other’s arguments. For example, we noticed that during discussions, Lisa jumped into discussions with ease and elaborated on an adult’s argument: Tammy: So you think less eggs, makes it more chewy? Lisa: Yeah, less eggs makes it more chewy because there’s not enough egg particles to hold everything together. Jason: What do you mean by hold everything together? Lisa: Because you don’t, I hear Tammy say that it [eggs] like water and oil, so the only reason that the oil and the water mix is that the egg particles forces to mix because it hung on to both. In the prior discussion, Tammy referred to eggs as emulsifiers that help oil and water mix together. Here, we see Lisa further adding to Tammy’s definition, giving a possible causal reason for why the oil and water mix. Our argument is that during these discussions, Lisa developed a comfort level with the adults, which later allowed her to develop in her decision-making practices. Second, in-the-moment captures of learners’ experiences are useful for helping them to stop and reflect on the activities. In the semi-structured activities, learners took photos and recorded their thoughts in situ on the iPads™. We believe that in so doing, the learners slowed down from the activities and began to reflect. In discussions, learners would refer back to their stories and present them to the group as evidence for their claims. For instance, the participants engaged in a whole group discussion on bubbles and their role in the outcome of their semi-structured experiments on eggs and leaveners. During this time, learners shared their stories with the group. For example, Ari shared this portion of his StoryKit story with the group, “The bubbles probably make the brownies rise.” Within the whole group setting both learners and adults attempted to determine what bubbles are composed of, how bubbles might be formed (both from a macroscopic and submicroscopic perspective), and what role did bubbles have in producing the outcomes they observed in the brownies and cookies. There is a chance that Ari and Clyde’s group utilized the ideas from the whole group discussion and their story presentation to connect the foam to the future outcome of the vegan brownies.

Conclusions
This work is not the prescriptive, singular manual for how to help learners make choices in science learning. Instead, we suggest that supporting learners’ decisions in science learning is a dynamic and complex challenge that requires practitioners and researchers to consider many issues. We argue that the cultural practices are important for supporting decision-making practices in science, not because they are formal scaffolds, but because they interact to support a participatory culture in which informed choices are a form of empowerment for learners. As mentioned before, learners’ lack of interest in science may be in part due to their perception of disempowerment, including the lack of opportunities to use science in meaningful real life contexts. Based on the work presented here, we suggest that empowerment can come from helping learners make personal decisions in science with contexts and goals that support their interests.

References

© ISLS

109

ICLS2012

Volume 1: Full Papers

Aikenhead, G. S. (1996). Science education: Border crossing into the subculture of science. Studies in Science Education, 27, 1-52. Atwater, M. M. (1996). Social constructivism: Infusion into the multicultural science education research agenda. Journal of Research in Science Teaching, 33(8), 821–837. Barton, A. C., & Tan, E. (2009). Funds of knowledge and discourses and hybrid space. Journal of Research in Science Teaching, 46(1), 50–73. Basu, S. J., & Barton, A. C. (2007). Developing a sustained interest in science among urban minority youth. Journal of Research in Science Teaching, 44(3), 466–489. Bouillion, L. M., & Gomez, L. M. (2001). Connecting school and community with science learning: Real world problems and school–community partnerships as contextual scaffolds. Journal of Research in Science Teaching, 38(8), 878–898. Cahill, C., Kuhn, A., Schmoll, S., Pompe, A., & Quintana, C. (2010). Zydeco: Using mobile and web technologies to support seamless inquiry between museum and school contexts. Proceedings of the Ninth International Conference on Interaction Design and Children (pp. 174–177). Chinn, C. A., & Malhotra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86(2), 175–218. Clegg, T. L., Gardner, C., & Kolodner, J. (2011). Technology for supporting learners in out-of-school learning environments. In Proceedings of the Ninth International CSCL Conference. Hong Kong, China. Clegg, T. L., Gardner, C., & Kolodner, J. L. (2010). Playing with food: Moving from interests and goals into scientifically meaningful experiences. In Proceedings of the Ninth International Conference of the Learning Sciences (ICLS) (Vol. 1, pp. 1135-1142). Druin, A. (2005). What children can teach us: Developing digital libraries for children with children. The Library Quarterly, 75(1), 20–41. Furman, M., & Barton, A. C. (2006). Capturing urban student voices in the creation of a science minidocumentary. Journal of Research in Science Teaching, 43(7), 667–694. Fusco, D. (2001). Creating relevant science through urban planning and gardening. Journal of Research in Science Teaching, 38(8), 860–877. Jenkins, H., Clinton, K., Purushotma, R., Robinson, A., & Weigel, M. (2006). Confronting the challenges of participatory culture: Media education for the 21st century. Chicago: MacArthur Foundation. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., et al. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting Learning by DesignTM into practice. Journal of the Learning Sciences, 12(4), 495–547. Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. San Francisco, CA: John Wiley and Sons. Quinn, A., Bederson, B., Bonsignore, E., & Druin, A. (2009). StoryKit: Designing a mobile application for story creation by children and older adults (No. HCIL-2009-22) (pp. 1-10). College Park, MD: Human Computer Interaction Lab, University of Maryland. Reiser, B. J. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13, 273–304. Yin, R. K. (2009). Case study research: Design and methods (4th ed., Vol. 5). Thousand Oaks, CA: Sage Publications, Inc.

Acknowledgements
We want to thank the CI Fellows program for funding this work. We thank David Cavallo for his advice, Alex Quinn and Ben Bederson for the use of StoryKit, and Alex Kuhn and Chris Quintana for the use of Zydeco. Finally, we acknowledge our participants and the local school community that partnered with us for this work.

© ISLS

110

ICLS2012

Volume 1: Full Papers

Students’ Intuitive Understanding of Promisingness and Promisingness Judgments to Facilitate Knowledge Advancement
Bodong Chen, Marlene Scardamalia, Monica Resendes, Maria Chuy, Carl Bereiter, University of Toronto, 252 Bloor Street West, IKIT 9th floor, Toronto, ON, M5S 1V6, Canada Email: bodong.chen@utoronto.ca, marlene.scardamalia@utoronto.ca, monica.resendes@utoronto.ca, maria.chuy@gmail.com, carl.bereiter@utoronto.ca Abstract: The ability to identify promising ideas is an important but obscure and undeveloped aspect of knowledge building. The goal of this research was to examine the extent to which young students can make promisingness judgments and, as a result, engage in more effective knowledge building. Toward this end we embedded a design experiment in a Grade 3 classroom. In this experiment students were engaged in discussion and reflection of the concept of promisingness and used a Promising Ideas tool to identify promising ideas in their written online discourse. They used the tool for two refinements of idea selections to focus ongoing community dialogue. Results suggest that students as young as 8 years of age can make promisingness judgments that facilitate knowledge advancement in their work. These results inform future work in classroom interventions and tool development to promote promisingness judgments in collaborative knowledge building. Like scientists in research laboratories (Dunbar, 1995), students engaged in knowledge building participate in constructive and progressive knowledge-building discourse, in which they contribute to group dialogue in distinctive ways, including proposing theories, synthesizing ideas, and making analogies (Chuy, Zhang, Resendes, Scardamalia, & Bereiter, 2011). A knowledge building principle that frames such discourse is “collective cognitive responsibility” (Scardamalia, 2002), according to which students share responsibility for advancing the knowledge of their community. To advance community knowledge, knowledge building calls for risk taking in pursing novel solutions to problems, with an ongoing commitment to continual idea improvement. Knowledge Forum (Scardamalia, 2004) is an online, community space specially designed to support Knowledge Building pedagogy. This pedagogy and technology has been adopted in international contexts and across the school curriculum (Scardamalia & Egnatoff, eds., 2010). The ability to identify promising ideas—ideas that with development might grow to something of consequence—is essential for creative work with ideas, but remains unexplored (Bereiter & Scardamalia, 1993). According to Bereiter (2002), knowledge of promisingness is crucial in every kind of creative work at all levels, and accordingly should be recognized as an important component in the knowledge-building process. Knowledge of promisingness is acquired over time as people engage in creative practices, by taking risks and learning from the successes and failures that are integral to the creative process. This type of knowledge increases with creative expertise, helping people improve their ability to take successful risks. Thus, to prepare young generations to be future creative achievers in various fields, we should encourage them to make promisingness judgments during their knowledge building work. This study serves as a first step in a program of research that seeks to explore the extent to which young students can make promisingness judgments and how educators can support them. The study addresses two central research questions: Can young students assess promisingness, or the knowledge building potential of their own ideas? Do their selections of promising ideas, and further community discourse on these selections, facilitate knowledge advancement in the community?

Promisingness and Knowledge Building Discourse
Substantial work is normally needed to develop ideas to address knowledge creation goals. Single ideas seldom constitute problem solutions; neither do simple combinations of ideas. When the goal is knowledge creation, the problem space tends to be complex and the extent to which an idea will prove valuable cannot be known at the beginning. As Johnson (2010) points out, “an idea is not a single thing, but more like a swarm (p. 43);” to better grow, an idea should live in a “liquid network” where it can more easily connect with its “adjacent possible” (Kauffman, 2002, p.47). However, an abundance of possible connections to an idea does not guarantee the fulfillment of a knowledge creation goal. A significant challenge in all creative work, in both the fine and broad grain, is to identify promising directions and to avoid wasting time or becoming entrapped by unpromising ones (Bereiter & Scardamalia, 1993). In the knowledge-creating context, promising ideas provide indication of future success or good results in a specific problem domain. Evaluation of risk and promisingness is a natural component of the knowledge creation process. For example, in his study of scientific reasoning in real-world laboratories, Dunbar (1995) found that scientists tend to categorize their projects into different levels of risk: a high risk project has a low probability of working out but may lead to breakthroughs, whereas a low risk project has a high probability of

© ISLS

111

ICLS2012

Volume 1: Full Papers

success but does not promise important discoveries. Assessments of promisingness help scientists determine how and when to invest their resources and time. Promisingness guides action of the moment as well as overall approach. To become a creative achiever in any field, one has to take risks at many levels and learn from the results (Bereiter & Scardamalia, 1993). The current study focuses on knowledge-building discourse by young students, exploring the extent to which promisingness judgments might facilitate their knowledge advancement.

Method
Participants
Participants were students from a grade 3 class at the Dr. Eric Jackman Institute of Child Study in downtown Toronto. There were 11 boys and 9 girls taught by a novice Knowledge Building teacher in 2011. They were studying earth sciences, with focus on soil and worms, for eight weeks. In this elementary school, Knowledge Building pedagogy and technology are used from grade 1 on, so students each year had roughly equivalent experience in the pedagogy and technology.

Promising Ideas Tool
A Promising Ideas tool was added to Knowledge Forum to facilitate this research. It includes two components: (a) a highlight feature to allow students to tag an idea within a note using a customizable categorization scheme (see Figure 1, left side) and (b) a component aggregates all selected ideas within a same view, merges overlapping ideas, and visualizes them in a list (see Figure 1, right side). An early version of this tool was implemented and tested in a small-scale pilot study (Chen, Chuy, Resendes, & Scardamalia, 2010). The goal of that study was not to enhance students’ ability to make judgments on idea promisingness, but to identify whether they are able to make these judgments at all. Results indicated that without any explanation of the concept of promisingness, grade 5/6 students tended to identify important-sounding facts and questions as promising. Thus, we had reason to doubt that students were making promisingness judgments based on the potential for deepening their understanding or identifying new directions in their research and development (Chen, Chuy, Resendes, Scardamalia, & Bereiter, 2011). Based on results from the first pilot study, we reengineered this tool to emphasize the concept of promisingness as the essence of this tool. The new version maintained functionalities from the previous version, including highlighting, tagging, and filtering. To make the endeavor of identifying promising ideas more meaningful, we implemented a third component that allows students to export a subset of identified promising ideas from the aggregate list to a new workspace for further inquiry. By reviewing the promising idea list together, a class could collectively determine which ideas or questions they wish to pursue for next steps. These ideas can be selected by clicking on checkboxes corresponding to each chosen idea on the list (see Figure 1, right side). Then, notes which contain those promising ideas can be exported to a new Knowledge Forum view by clicking on the “Export Notes” button. After that, students can work to develop those exported promising ideas to a deeper level. This function could engage students in identifying promising ideas with a commitment to define new directions for their work.

Figure 1. The Promising Ideas tool currently has three components: (a) tag an idea—on the left, a student can identify an idea with a customizable scheme; (b) list tagged ideas—in the background window on the right side, all identified promising ideas from a view are listed; and (c) export selected ideas to a new view—users can then select a subset of ideas from this list and export them to a new view (the foreground window on the right).

© ISLS

112

ICLS2012

Volume 1: Full Papers

Procedure
Beside the grade 3 class which participated in this study, another grade 3 class from the previous year in the same school was included to provide baseline data. Both grade 3 classes studied “Soil in the Environment” for around 8 weeks, a unit in the Ontario Science and Technology Curriculum. In studying this unit, both classes were following a knowledge-building approach guided by twelve knowledge building principles (Scardamalia, 2002; Zhang, Hong, Scardamalia, Teo, & Morley, 2011). Knowledge building practice in this school usually contains four components: Knowledge Building talk (“KB Talk”), online discourse in Knowledge Forum, study on authoritative sources, and experiments or observations. There is no routine that defines how these four components should proceed. Rather, these components are organized in a classroom in a flexible and organic manner that facilitates student’s collective knowledge building. Both classes started with a KB talk, with students sitting in a circle sharing their initial authentic questions and ideas about soil. After the first KB talk, “what is soil made of?” and “how to make soil?” emerged as the focus of students’ study of soil in both classes. To keep students’ ideas alive and to extend student discussion to a broader scale, students then moved to Knowledge Forum to record their ideas and start online dialogue. They contributed to their discourse in various ways, including proposing their own theories, building on each other’s theories, providing observation results to defend their claims, designing and conducting experiments to test hypotheses, and reading and introducing authoritative sources such as books or online materials. Although two classes were led by two different teachers, dynamics in both classrooms followed a similar knowledge building process. In the experimental class we implemented a promisingness intervention into regular knowledgebuilding discourse. This intervention mainly comprised two components. The first component was a 30 minute pedagogical intervention targeting students’ understanding of promisingness. As the first step, we asked grade 3 students to brainstorm with their partners the meaning of “a promising idea” and “a promising question”, and recorded their group thoughts on paper. Student notes were collected for later analysis. Ten minutes later, students brought their thoughts back to a class discussion, advancing different but plausible definitions of the concept and supporting them with meaningful examples. In this discussion, the teacher specifically invited students to ponder the difference between promising ideas and verified facts, to help them distinguish promisingness from “truthfulness.” By the end of the 30-minute discussion, the class arrived at a shared understanding that promising ideas are ideas that “they wish to spend time on,” “may change in further inquiry,” and would “deepen their shared understanding.” The second component of this intervention engaged students in progressive use of the Promising Ideas tool. The whole process involved two iterations of note writing and promisingness intervention. During the first two weeks, students worked on a view named “Grade 3 Soil 2010/11” in Knowledge Forum. After that, students were directed to conduct promisingness judgments on this view, picking ideas that they thought were promising using the Promising Ideas tool. Then the whole class had a discussion on identified ideas, and decided to export several promising ideas to a new view called “Where does soil come from?” The second iteration of note writing and promisingness evaluation then followed. During the next three weeks students kept working on the exported view. After that, students did a second round of promisingness evaluation on that view, and exported their promising ideas to another new view named “Worms and Soil,” on which students worked for another two weeks. In the whole process, promisingness evaluation was integrated into regular knowledge-building discourse as a component to determine future steps of student inquiry. Student notes and selections of ideas were collected for data analysis.

Data Analysis
Students’ notes of their intuitive understanding of promisingness were collected from the brainstorm session in the experimental class. Overall, 8 groups of students produced 35 distinctive notes. We coded data based on a “grounded theory” approach (Glaser & Strauss, 1967), with a goal to identify popular conceptions, especially misleading ones, of promisingness among this group of students. In work on this soil unit, students from the experimental group created 125 notes in three views (respectively 38, 47, and 40 notes). During two sessions of promisingness judgments, students identified 57 and 94 “promising ideas” from the first two views. Each single highlighting action was counted as an idea. Since several students could independently select a same idea, there were a number of repetitions of ideas. In the baseline group, students produced 130 notes in five views. They were not making promisingness judgments in any form, so no idea was identified. Two independent raters used the following criteria to code the notes and ideas. They agreed on 82.5% of the coding. Discrepancies were discussed to reach a final agreement. Contribution types of identified promising ideas. Because promisingness judgments in knowledge building focus on potential for growth, we expect ideas identified by students as promising to be of a “theoretical nature” rather than pure facts (Chen et al., 2011). In order to understand the nature of identified ideas and assess whether the pedagogical intervention was effective in transforming students’ understanding of

© ISLS

113

ICLS2012

Volume 1: Full Papers

promisingness, we applied an extended list of ways of contributing scheme previously developed by Chuy and colleagues (2011). This scheme includes six major categories, including: formulating thought-provoking questions (e.g., explanatory, factual, and design questions), theorizing (e.g., proposing, supporting, and improving a theory), obtaining evidence (e.g., seeking evidence, reporting experiment results, introducing facts from sources or experience), working with evidence (e.g., using evidence to support or discard a theory), synthesizing and comparing (e.g., synthesizing discussion, creating rise-aboves, and making analogies), and supporting discussion (e.g., facilitate discussion, give an opinion). Measurements of individual and community knowledge advancement. Knowledge advancement by students in both the experimental and baseline groups was assessed through content analysis of student notes in Knowledge Forum. First of all, scientificness of student ideas was assessed using a four-point scale developed by Zhang and colleagues (2007): 1–pre-scientific (containing a misconception while applying a naive conceptual framework), 2–hybrid (containing misconceptions that have incorporated scientific information), 3–basically scientific (containing ideas based on a scientific framework, but not precise), or 4–scientific (containing explanations that are consistent with scientific knowledge). To evaluate whether the experimental class has made any progress in terms of scientificness of their ideas, we analyzed all student notes that fell into the theorizing category in the ways of contributing analysis, and compared across three views produced by three phases of student work. The same analysis was performed for the baseline group, and results were compared between two groups. Second, latent semantic similarities between student notes and authoritative sources were calculated through Latent Semantic Analysis (LSA) to approximately represent students’ level of collective understanding of the subject matter. LSA is a theory and model for extracting and inferring contextualized, latent meaning of words from statistical analysis of a corpus of text (Landauer, Foltz, & Laham, 1998). It has been widely applied to compare semantic meaning of different documents, and its adequacy in reflecting human knowledge has been established in extensive educational settings (Foltz, Kintsch, & Landauer, 1998; Forster & Dunbar, 2009; Landauer & Dumais, 1997; Rehder et al., 1998; Teplovs & Fujita, 2009). In LSA, each document is treated as a “bag of words”; comparison between documents is based on latent connections between their words in a predefined semantic space trained through complex mathematical computations. To train a semantic space about soil, we first selected four pieces of authoritative sources that represent varied levels of complexity, including Soil in the Environment unit in the Ontario curriculum standards (2007), Wikepedia article about Soil (2011), an entry level article about soil from US Environmental Protection Agency (2011), and a more sophisticated education article from US Department of Agriculture (2011). Then we exported student discourse in three views to three individual documents, and compared them with the list of authoritative sources for similarity. Similarity scores between student discourse and authoritative sources represented the advancement of community knowledge at different knowledge building phases.

Results and Discussion
Student Understanding of Promisingness Students’ Intuitive Understanding of Promisingness
As shown in previous studies, when asked to select “promising” ideas, young students often select “important facts” of the sort they address in their school work (Chen et al., 2011). In this investigation we considered students’ intuitive understanding of promisingness. Analysis indicated that some students thought a promising idea was “a true idea,” “an idea that promises you something” or “actually happened;” for a promising question, some students thought this could be explained as “a question you promise to answer,” “a question that has multiple answers,” “a question that has not many answers,” or “a question that is difficult and should be solved.” Overall, students revealed different understandings of promisingness in different contexts. However, three themes emerged from students’ notes about the meaning of promisingness. First of all, consistent with the results from previous studies (Chen et al., 2011), the feature of “being true” or “truthfulness” was a popular notion present in notes of all groups. For example, some students thought a promising idea was: Group 1: “A true idea” or “an idea that is not incorrect.” Group 2: “True.” Group 3: “You know that an idea is true”, or “you promise that it is right.” Group 5: “An idea that you are pretty sure is right.” These thoughts could explain why students in our previous study were identifying facts such as “The universe is 13,000,000,000 years old!” and “the grand canyon could have 912,456 layers of rock” (Chen et al., 2011). The second cluster of thoughts focused on the notion of “likelihood.” For example, a number of groups mentioned that a promising idea was: Group 4: “An idea that is very good and probably be right.”

© ISLS

114

ICLS2012

Volume 1: Full Papers

Group 5: “An idea that might work.” Group 7: “This is probably a right idea.” Group 8: “Idea that is most likely to be right,” or “90% sure it’s right.” This notion of “likelihood” went further than emphasizing “truthfulness,” in that there is a recognition that we cannot tell whether a promising idea is true for sure, but a promising idea has high odds of being true. According to dictionaries such as Merriam-Webster, promising means “showing signs of future success or excellence,” “likely to turn out well,” or “likely to succeed or yield good results.” Bereiter (2002, p.337) explained promisingness by the metaphor of “hill climbing,” highlighting that a promisingness evaluation is like to find the way up the hill although you cannot see a path to the summit. In this sense, the notion of “likelihood” students captured is an essential component of promisingness. However, it is important to note that students were thinking of likelihood in terms of an idea’s current status of being true. Apparently, they were not thinking of ideas with a trajectory for growth. The third notion that emerged from students’ notes was about “future actions.” When thinking about what is “a promising question,” a few groups wrote: Group 1: “A question you can spend time on.” Group 2: “A question you need to know.” Group 5: “A question that can help you do something.” The notion that a promising idea or question is something that you can spend time on or can help you better know or do something fit the notion that a promising idea could be flawed but worth laboring to improve it (Bereiter & Scardamalia, 1993; Bereiter, 2002). This notion provided a good basis to work from to prepare students with a favorable understanding for promisingness judgments. To summarize, without intervention the concept of promisingness proved to have different meanings under different conditions for grade 3 students.

Was promisingness talk effective?
Previous research shows young students tend to select important facts of the sort they address in their school work as being promising (Chen et al., 2011). In the current study we introduced a pedagogical intervention module to engage grade 3 students in thinking and discussing about the meaning of promisingness as a group. Given this was our first attempt to engage them actively in the concept of promisingness, it is important to evaluate whether this intervention was effective in preparing students to consider promisingness as extending beyond factual content to ideas that might advance work in the field. To answer this question, we applied the ways of contributing scheme described above. Results showed that grade 3 students identified a large portion of theorizing (68.9%) and much less obtaining evidence contributions (6.7%) as being promising. Students also identified other contribution types, including questioning (9.8%), working with evidence (4.9%), synthesizing and comparing (1.8%), and supporting discussion (8.0%). Overall, the proportion of fact/evidence has decreased significantly, comparing to selections by grade 5/6 students reported in previous research (Chen et al., 2011). This finding indicated that the concept of promisingness could be easily extended, so that rather than a focus on “true ideas” students have more of a sense of ideas with a promising growth trajectory. To further understand whether evidence picked by grade 3 students was introduced with a goal to move discourse forward, we analyzed the dialogue context of the contributions related to obtaining evidence. Results indicated most these contributions fell into the introducing new information sub-category, mostly containing new facts or new pieces of evidence; this situation was similar to that in two grade 5/6 classes (Chen et al., 2011). However, further analysis on the grade 3 discourse revealed that obtaining evidence contributions identified by students as promising usually co-occurred with the contribution type of using evidence or reference to support a theory (which falls within the working with evidence contribution category). Apparently, students from this grade 3 class were identifying those obtaining evidence contributions which were made to drive discourse forward. In contrast, as reported in previous study, fact/evidence contributions were identified by grade 5/6 students, who did not spend time brainstorming or discussing the concept of promisingness, were more likely to be standalone. The shift in understanding can be seen in the following contrast. In an idea identified by one grade 3 student, the new information “when it’s night it’s cooler … in the day the sun is shining and it’s warmer” was introduced to support her theory that “worms sense light by temperature.” However, grade 5/6 students were more likely to identify standalone facts, such as “the solar system formed 4570 million years ago.” Notably, grade 3 students who went through the promisingness discussion were more likely to identify facts or evidence that contributed to their knowledge-building discourse, while grade 5/6 students tended to identify pure facts that had some distinctive “attractiveness” (Chen et al., 2011).

Promisingness Judgments to Facilitate Knowledge Advancement
Promisingness judgments are to boost community knowledge by identification of promising directions that are refocusing in ongoing discourse. Evaluating the promisingness of their own ideas engages students in deep reflection on the state of their knowledge work and helps them recognize ideas that they deem worth extended work. In this way, students can denote their limited time and energy to more promising ideas. As a result, their

© ISLS

115

ICLS2012

Volume 1: Full Papers

individual and collective understanding may get more opportunities to grow. In this study, our hypothesis was that students in the experimental class, who performed promisingness judgments, would achieve greater knowledge advancement than students in the baseline class. To test this hypothesis, in this section we examine idea improvement and knowledge advancement in two classes by two means: rating students’ theorizing contributions for scientificness, and comparing semantic space of student discourse with expert content.

Scientificness of notes and conceptual change
First of all, we rated students’ theorizing contributions in two classes on the continuum from pre-scientific to scientific understanding. In the experimental class, we identified and analyzed 91 theorizing notes, 26 notes from “Grade 3 Soil 2010/11” (view 1), 42 notes from “Where does soil come from” (view 2), and the rest 23 from “Worms and Soil” (view 3). In the baseline class, we identified a total number of 68 theorizing notes. Because the baseline group did not integrate promisingness judgments into their discourse, there was no natural distinction of different phases; so we sequenced student notes according to the time of creation and manually divided them into three phases with equivalent number of notes. Mean scores and standard deviations for three discourse phases in both classes are presented in Table 1. A 2 (Group) × 3 (Discourse Phase) factorial ANOVA was performed to assess whether scientificness scores of student ideas could be predicted from student group (experimental vs. baseline), phases of discourse (phase 1, 2 and 3), and the interaction between these two factors. This analysis of variance revealed a significant main effect for discourse phases, F (2, 30) = 13.51, p < .001, η2 = .15, indicating the mean scientificness scores were different among three phases. The main effect for group difference was not significant, F (1, 30) = .47, p = .49, η2 = .00. The interaction between discourse phases and group difference was significant, F (2, 30) = 3.81, p < .05, η2 = .05. Table 1: Mean and Standard Deviations for Three Discourse Phases in Experimental and Baseline Groups. Phase 1 Experimental Group M SD n Baseline Group M SD n 1.29 .38 26 1.50 .58 23 Phase 2 1.45 .55 42 1.50 .62 23 Phase 3 2.24 .89 23 1.77 .67 22

Figure 2. Mean scores of scientificness of theories across three discourse phases. The graph of mean scores (in Figure 2) indicated that scientificness scores in the experimental class have been substantially improved across three phases, while scientificness scores in the baseline group were
© ISLS 116

ICLS2012

Volume 1: Full Papers

only slightly improved. Planned contrasts were done with L-Matrix subcommand to assess whether these differences (between phase 1, 2, and 3) were significant within the experimental and baseline groups. For the experimental group, the improvement from phase 1 to 3 was significant, F (1, 153) = 28.70, p < .001, η2 = .16; so was the improvement from phase 2 and 3, F (1, 153) = 23.94, p < .05, η2 = .14. In contrast, none of the differences in the baseline group was significant. Notably, the experimental group started with less scientific ideas than the baseline group, and finished with more scientific understanding about soil and worms. In conclusion, students in the experimental class, who performed promisingness judgments, achieved greater knowledge advancement than students in the baseline class.

Growth of subject domain knowledge
To further explore the changes of community knowledge in the experimental group, we applied Latent Semantic Analysis (LSA) as another distinctive analysis technique. LSA is able to accurately identify semantic similarity between two documents within a semantic space. In this analysis, student notes in three different views were exported to three individual documents, and then respectively compared with authoritative sources about soil. LSA semantic similarity scores between student written dialogues and authoritative sources were used to infer students’ levels of understanding in different phases. Because view 3 (“Worms and Soil”) in the experimental group focused on worms rather than soil, we only included view 1 (“Grade 3 Soil 2010/11”) and view 2 (“Where does soil come from?”) into this analysis. We also included a view (“Grade Three Soil 2009/10”) produced by the baseline group for comparison. Figure 3 presents LSA cosine similarity scores of three views when they are respectively compared to four pieces of authoritative sources, i.e. Ontario curriculum standards, Wikipedia article, US Environmental Protection Agency (US EPA) article, and US Department of Agriculture (US DA) article.

Figure 3. LSA cosine similarity scores between views and different authoritative sources. The first two views, i.e. “Grade 3 Soil 2010/11” and “Where does soil come from?”, were produced by the experimental class, and the third view, “Grade Three Soil 2009/10”, was from the baseline group. From this analysis a notable pattern emerged that the similarity scores of view 2 (M = .24) were substantially higher than view 1 (M = .08) in comparing with those different pieces of authoritative sources. The similarity scores of view 2 were even higher than the baseline group (M = .18) taught by a more experienced Knowledge Building teacher. These results from LSA indicated that students in the experimental group had improved their understanding on soil across the first two dialogue phases, to an equivalent or higher level of the baseline group.

Discussion and Conclusion
The goal of this study was to examine the extent to which young students can make promisingness judgments on their own ideas, and accordingly achieve more advancement in their knowledge-building discourse. Toward this end, a promisingness intervention was employed in a grade 3 class, which engaged students in discussing the concept of promisingness and then using the Promising Ideas tool to identify promising ideas during knowledge-building discourse in a progressive manner. Analysis of results indicated that with discussion and analysis of the concept of promisingness, students as young as 8 to 9 years old could make promisingness judgments which, compared to those of grade 5/6 students (Chen et al., 2011), advanced beyond the idea that factual content defines promisingness. Scientificness and knowledge coverage of student notes were further analyzed across three discourse phases divided by two sessions of promisingness judgments. Results showed a significant growth of scientificness as well as domain knowledge across three phases in the experimental group,

© ISLS

117

ICLS2012

Volume 1: Full Papers

while the baseline group, who did not engage in making promisingness judgments on their work, did not reveal such growth. Promisingness evaluation is an important, but obscure component in knowledge-building discourse (Bereiter & Scardamalia, 1993). Substantial further research is needed to examine the role promisingness evaluations play in different contexts. New studies will tackle fundamental research issues such as how people make promisingness judgments, a developmental trajectory for making promisingness judgments, and means to help students gain deeper understanding through making more promisingness judgments.

References
Bereiter, C. (2002). Education and Mind in the Knowledge Age (p. 544). Lawrence Erlbaum. Bereiter, C., & Scardamalia, M. (1993). Surpassing Ourselves: An Inquiry Into the Nature and Implications of Expertise (p. 296). Open Court Publishing Company. Chen, B., Chuy, M., Resendes, M., & Scardamalia, M. (2010, August 4). “Big Ideas Tool” as a New Feature of Knowledge Forum. Poser presented at 2010 Knowledge Building Summer Institute. Toronto, Ontario, Canada. Chen, B., Chuy, M., Resendes, M., Scardamalia, M., & Bereiter, C. (2011). Evaluation by Grade 5 and 6 Students of the Promisingness of Ideas in Knowledge-Building Discourse. In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings. Volume II - Short Papers and Posters (pp. 571-575). International Society of the Learning Sciences. Chuy, M., Zhang, J., Resendes, M., Scardamalia, M., & Bereiter, C. (2011). Does Contributing to a Knowledge Building Dialogue lead to Individual Advancement of Knowledge? In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings. Volume I - Long Papers (pp. 57-63). International Society of the Learning Sciences. Dunbar, K. N. (1995). How scientists really reason: Scientific reasoning in real-world laboratories. In R. J. Sternberg & J. Davidson (Eds.), The Nature of Insight (pp. 365-395). MIT Press. Foltz, P., Kintsch, W., & Landauer, T. (1998). The measurement of textual coherence with latent semantic analysis. Discourse Processes, 25(2), 285-307. doi:10.1080/01638539809545029 Forster, E. A., & Dunbar, K. N. (2009). Creativity Evaluation through Latent Semantic Analysis. Proceedings of the Annual Conference of the Cognitive Science Society. Glaser, B. G., & Strauss, A. L. (1967). The Discovery of Grounded Theory. Discovery (p. 271). Aldine. Johnson, S. (2010). Where Good Ideas Come From: The Natural History of Innovation. Riverhead Hardcover. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211-240. doi:10.1037//0033-295X.104.2.211 Landauer, T., Foltz, P., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2), 259-284. doi:10.1080/01638539809545028 Rehder, B., Schreiner, M. E., Wolfe, M. B. W., Laham, D., Landauer, T. K., & Kintsch, W. (1998). Using latent semantic analysis to assess knowledge: Some technical considerations. Discourse Processes, 25(2), 337 354 ST - Using latent semantic analysis to. doi:10.1080/01638539809545031 Scardamalia, M. (2004). CSILE/Knowledge Forum®. In A. Kovalchick & K. Dawson (Eds.), Education and technology: An encyclopedia (pp. 183-192). Santa Barbara, CA: ABC-CLIO. Scardamalia, M., & Egnatoff, W. (Eds.). (2010). Canadian Journal of Learning and Technology, Special Issue on Knowledge Building, 36(1). Teplovs, C., & Fujita, N. (2009). Determining curricular coverage of student contributions to an online discourse environment: Using latent semantic analysis to construct differential term clouds. Proceedings of the 9th International Conference of Computer Supported Collaborative Learning 2009 - Volume 2 (pp. 165-167). Rhodes, Greece: International Society of the Learning Sciences. Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics of knowledge building in the work of 9- and 10-year-olds. Educational Technology Research and Development, 55(2), 117-145. doi:10.1007/s11423-006-9019-0

Acknowledgments
This research was funded by a Research Grant from the Social Sciences and Humanities Research Council of Canada titled “Ways of contributing to dialogue in elementary school science and history.” We would like to express our gratitude to the students, teachers and principals of the Dr. Eric Jackman Institute of Child Study, University of Toronto for the insights and the opportunities enabled by their involvement.

© ISLS

118

ICLS2012

Volume 1: Full Papers

Designing Video Games that Encourage Players to Integrate Formal Representations with Informal Play
Nathan R. Holbert, Uri Wilensky, Northwestern University, Evanston, IL Email: nholbert@u.northwestern.edu, uri@northwestern.edu Abstract: In this paper we present a case study of a player of the game FormulaT Racing—a prototype racing video game we created to explore design principles that could transform popular games into powerful learning environments—shifting between and ultimately integrating video game and school-based epistemological framings. We show a player shifting from describing acceleration and velocity in a purely game-based narrative during a pre-game interview to descriptions that seamlessly integrated both game and formal representations in a post-game interview. We contend that this change is due to novel interactions with key representations introduced throughout the game and discuss design implications for future games.

Introduction
As technology continues to improve, video games are becoming a preferred way many American children spend their free time. The PEW Internet and American Life Project claims that as many as 97% of all American teens (regardless of gender, age, or socioeconomic status) play video games in some way and 50% play games daily for an hour or more (Lenhart et al., 2008). A more recent report by the Kaiser Family Foundation suggests young “tweens” and Black and Hispanic children (Rideout, Foehr, & Roberts, 2010) spend even more time playing video games daily. With such a time investment by all of our children it is important to not only try to understand the cognitive implications of this common experience, but also to consider ways in which we might enhance this activity that is so central to youth culture. A number of video games have recently been designed to teach science content in the classroom (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005; Clark, Nelson, D'Angelo, Slack, & Menekse, 2010; Squire, Barnett, Grant, & Higginbotham, 2004). Like our colleagues, we believe video games are excellent vehicles for experimenting with science phenomena, but rather than design science video games for the classroom, we have chosen to explore design principles that could transform popular commercial video games, games that are typically played outside of school settings, into powerful educational experiences. With this in mind, we have designed a prototype video game, FormulaT Racing (Holbert & Wilensky, 2010a) to be an archetype of the racing genre while providing a platform for experimentation with new representations and interaction mechanisms. Instead of trying to explicitly “teach” players embedded physics concepts, FormulaT Racing (FTR) is designed to tap into children’s intuitive notions of motion and to connect these intuitions to formal representations. Consequently, while players may not become experts in kinematics by playing FTR, they should be left with a sense that their experiences in the game are relevant to non-game motion experiences and likewise, that formal definitions and representations of kinematic content are relevant to game experiences. Many video games designed with educational outcomes in mind have shown moderate success on related assessment measures, however, gains in related but not directly equivalent concepts are often less pronounced (Clark et al., 2010). In this paper we propose this “failure to transfer” may be due to incongruent epistemological framing by players and explore how the design of FTR helps one player overcome this dilemma. In particular, we show Brian, a fourteen year old gamer, shifting from describing acceleration and velocity in a purely game-based narrative during a pre-game interview to descriptions that seamlessly integrated both game and formal representations in a post-game interview. We contend that this epistemological integration is due to interactions with key representational changes introduced throughout the game and discuss design implications for future games.

FormulaT Racing's Design
FormulaT Racing allows players to explore kinematics by putting the player in the shoes of a new driver as they train to join the FormulaT Racing Circuit. The design principles of FTR specifically target relevant phenomenological causal schema (diSessa, 1993) bringing typically intuitive interpretations of in-game motion to the foreground through motion-sensitive and “body-syntonic” controls (Papert 1980) so that they can be explored and manipulated. FTR is designed to look and feel like a traditional commercial racing game though it includes a number of key design changes. One important design feature is the inclusion of intuitive and bodysyntonic controls. Using a Nintendo Wiimote players manipulate their vehicle’s acceleration by rolling the controller forward or backwards for positive and negative acceleration respectively. Track features and level challenges force players to frequently adjust the roll applied to the controller encouraging them to reflect on acceleration as a changing quantity that directly impacts velocity.
© ISLS 119

ICLS2012

Volume 1: Full Papers

Traditionally racing video games have provided kinematic feedback to the player through visual cues such as a passing background, haptic cues such as a controller, and numerical cues such as a speedometer. FormulaT Racing retains the traditional “passing background” visual cue, but adds a new “color-trails” representation. In some of the game’s levels, velocity is represented by a color-trail left by the player vehicle that changes with the player car’s velocity. The color-trail representation encourages the players to make connections between the motion sensitive controls, changes in acceleration and velocity, and the varying structure and features of the track. Interviews with children playing popular commercial racing games suggested players rarely attended to provided speedometers (Holbert, 2009). Rather than continue the tradition of providing discrete quantitative indications of the player car’s speed, we chose to utilize a velocity versus time graph. This representation serves two important goals. First, as a widely used formal representation, the velocity versus time graph acts as an anchor to formal knowledge within the informal game-space. As we will argue later, we believe this mixing of formal representations and gameplay encourages the player to see her game-knowledge as relevant across contexts. Since the player often identifies as an expert gamer, the application of this game-knowledge to nongame spaces is extremely powerful. Second, rather than a speedometer, which highlights instantaneous speed, a velocity versus time graph foregrounds changing velocity, or acceleration. Though acceleration has been shown to be a challenging concept for children to understand (Tasar, 2010; Trowbridge & McDermott, 1981), we believe that it is a powerful idea that provides an entry point to many other difficult and useful concepts (Papert, 1980). FTR also includes important construction tools that fundamentally change the way the player causes motion. These designs, inspired by constructionist methods (Papert, 1980; Papert & Harel, 1991) as well as work examining expert and novice embodied interactions with formal graphic representations (diSessa, Hammer, Sherin, & Kolpakowski, 1991; Nemirovsky, Tierney, & Wright, 1998; Roschelle, Kaput, & Stroup, 2000; Sherin, 2000; Wilensky & Reisman, 2006) are intimately connected to previously discussed controls and visual cues but are not explicitly introduced until the third phase of the game. In this phase, rather than drive the car directly, the player constructs personal notions of motion by interacting with the representations of motion rather than the car itself. Specifically this occurs in one of two ways, either by painting the track using the palette presented by the color-trails, or by constructing a velocity versus time graph. In the “painting” mode of the construction phase the player uses colors corresponding to the previously seen color-trails to paint various parts of the track. Once the player is done painting the entire track, the car begins the race, changing velocity based on the color being driven over. The player can paint the track in any way she prefers, however, because each color corresponds to a particular velocity, and the car’s ability to turn is impacted by its current velocity, the choices made in painting the track determine whether or not the car will successfully complete the race. The player may also interact with the car by constructing a velocity versus position graph. In the “graphing” mode, the player constructs a velocity versus position graph using the motion sensitive controller. To construct the graph, the player “accelerates” points added to the graph by rolling the controller forward and backward. Once the graph is constructed, the car “downloads” the data and drives around the track according to the player-constructed graph. In this way the player directly connects the intuitive feeling of acceleration to formal graphic representations and can also explore how varying graphic features, such as sharp drops or plateaus in velocity correspond to particular track features.

Theoretical Framework
While the design of FTR draws upon many important cognitive theories including embodied cognition (Lakoff & Nùñez, 2000; Wilson, 2002) and knowledge-in-pieces (diSessa, 1993; Hammer, 1996; Sherin, 2006), in this paper we draw on two key lines of research to understand how FTR's design impacts player knowledge: restructurations (Wilensky & Papert, 2010) and epistemological framing (Hammer, Eleby, Scherr, & Redish, 2005). Representations in the world are often created with the intent to store, embody, or alter knowledge and processes. In this way, external representations “become in a very real sense part of our thinking, remembering, and communicating” (diSessa, 2000 p. 6). The term restructuration was proposed by Wilensky and Papert (2006, 2010) to describe the dramatic cognitive shift that can occur due to changes in representational infrastructure. In the same way the processes of multiplication and division—skills only practiced by the priestly few—were democratized after the shift from Roman to Hindu-Arabic numerals, so too do we believe advanced topics such as electricity (Sengupta & Wilensky, 2009), differential equations (Wilkerson-Jerde & Wilensky, 2010), and kinematics (Holbert & Wilensky, 2011a, 2011b) can be taught to younger children. While we have designed FTR to include important representations for kinematic restructuration, how a player interprets and interacts with these representations in a video game is not always straightforward. In particular, getting players to connect experiences in video games to formal concepts and representations has proved challenging. We believe player epistemological framing might explain much of this difficulty (Hammer et al., 2005). In education research the term “epistemology” is often used to describe an individual’s beliefs

© ISLS

120

ICLS2012

Volume 1: Full Papers

about the form and type of knowledge in a domain or context. Drawing on the theory of knowledge-in-pieces (diSessa, 1993), which suggests that cognition is emergent from the activation of a large number of fine-grained, highly distributed, context-sensitive primitive elements, Hammer and colleagues (2005) suggests that individual epistemologies change in-the-moment as various cognitive resources are activated. They go on to propose the idea of epistemological framing and define it as, “Phenomenologically, a set of expectations an individual has about the situation in which she finds herself that affect what she notices and how she thinks to act” (Hammer et al., 2005 p. 98). In this model, the ability to recall or apply knowledge in a given context then is affected by both the epistemological framing of the learning situation as well as the framing of the recall situation. Designers interested in creating educationally relevant video games must wrestle with the tradeoff of making the learning goals explicit, versus foregrounding the game narrative and fantasy (Collins, 1995). An overemphasis on learning goals often results in an experience that feels more like interactive flashcards than a game – school work dressed up with animation and sound (Ito, 2007). Here the content remains clearly schoollike and as such, likely activates a “knowledge as propagated stuff” epistemology (Hammer & Eleby, 2002). In contrast, by focusing solely on a game narrative, the embedded content that the researcher hopes to share with the player, may become clouded. Here the player sees the game as just that, a game, and not something that is relevant in a traditional school setting (Ito, 2007). The challenge then is trying to find a way to keep players firmly embedded in the game narrative, while making the embedded content flexible enough to be utilized in non-game contexts. To achieve this delicate balance, we have created FTR to be what Clark et al. (2011) call a “conceptually integrated game” (Habgood and Ainsworth, 2010, discuss a similar idea they refer to as “intrinsic integration”). Such a design ensures the learning outcomes match the game mechanics and are tightly integrated into the game narrative. Furthermore, FTR is “representationally integrated,” in that domain relevant representations become highly tied to the player interface and game-mechanics. As we will show in the discussion section, we believe this encourages players to connect formal representations to informal epistemological framings (Hammer et al., 2005) leading to a kinematic restructuration.

Method
To explore the impact of specific design features incorporated into FormulaT Racing we have conducted a number of studies of children playing the prototype game. These studies are conducted exclusively in informal settings to remain consistent with the typical video game playing contexts and employ multiple methodologies. We have written elsewhere about how increasingly complex player strategies in the painting phase match common programming strategies (Holbert & Wilensky, 2011a) and how FTR players learn to construct velocity versus time graphs (2011b, 2011c). For this paper we discuss semi-clinical interviews conducted with participants before and after they play FTR to identify changes in player knowledge about ideas and representations of motion. In these interviews, participants were asked a series of questions focused on real-world examples of acceleration and velocity using toy cars, adjustable speedometers, and graphing paper. Using methods described by Russ, Lee, & Sherin (in press) we analyzed video data and transcripts of the interviews looking for moments where participant behavior and language indicates a shift in how he frames the interview activity. While we currently have data on 12 participants, in this paper we focus on one participant, Brian, who serves as an exemplar of epistemological integration.

Results
In this paper, the focus of our analysis is on instances of pre- and post-game epistemological shifts. In particular we present a case study of one participant, Brian, shifting from an epistemological stance where game knowledge is separate from formal mathematic and physics knowledge, to a stance that seamlessly integrates the two. While we noticed other interesting shifts, such as players that shifted from a game to a “real world” focused epistemology, we believe the integrated epistemology discussed here highlights the importance of specific design features and offers important lessons for future iterations of FormulaT Racing and other such games. Brian is a very confident 14 year old boy from a low income neighborhood in a large Midwestern city who prides himself in his ability to not only beat nearly every game he plays, but also to create the “hardest” games. Most of Brian’s game building experience has involved personal constructions he’s made in Scratch at a computer clubhouse he attends near his home. During interviews with the first author, Brian asked many offthe-script questions about the programming environment used to create FTR as well as how he could “hack” the Nintendo Wiimote to work with his own Scratch games. There was no doubt that Brian considers himself an expert gamer.

Pre-game interview: Non-fluid epistemological shifting
This sense of himself as an expert gamer impacted how Brian perceived the activity and relevant knowledge for the pre-game interview. Brian approached nearly every question in the pre-game interview from
© ISLS 121

ICLS2012

Volume 1: Full Papers

an epistemological frame that privileged video game knowledge – that is, Brian answers all questions about motion as if they are about how motion occurs in the context of a video game – despite the fact that the representations and questions asked in the interview are agnostic about context. When asked to describe phases of acceleration, or situations involving constant velocity, Brian assumed the appropriate epistemological activity was one of story telling. In particular, Brian answered as if he was describing action occurring in a racing video game. In the exchange below Brian was asked, “Let's imagine this car is at a red light. I want you to sort of show me, by moving the car and also describing with words, what happens when the light turns green.” Before speaking, Brian pushes the toy car used in the interview quickly so that it speeds across the table into the interviewer's arm. When asked to describe the motion he states: Well I tried, I just kept moving as fast as I could to make sure no one else is coming after me. Maybe I may have used a turbo boost, like a mushroom. And then I bumped into the obstacle, which was your arm. And then I got hurt, and then the other racers passed me. Brian’s description of motion here directly invokes experiences with the commercial racing video games Mario Kart (indicated by his reference to a “mushroom” turbo boost). It’s important to note again that the interview questions do not reference video games, and the toy cars used by the participants to answer questions more closely resemble “real” cars than the cartoon-like ones of games like Mario Kart. Despite this, Brian sees the activity of answering the questions of the interviewer to be about describing events that might occur in the video game world – what we might call a “game story” framing. In this framing, relevant knowledge comes from video game experiences and is about recalling or creating action-filled stories of game play. It is not surprising that participants might associate the interviewer with video games. While participants have not yet played our racing game at this point in the study, they have been told that we have designed a game that we want them to test. Because we are interested in studying how kids interact with games in informal environments, associating the interviewer with video games is preferred to an association with the classroom. In addition, such a focus on video games in the pre-game interview, like we saw with Brian, increases our confidence in the external validity of the experience. If we want to explore the possibility of designs in real video games impacting the cognition of players, we want to be sure the games are being tested in real gaming settings by individuals that consider themselves gamers. Furthermore, as we will show later in this section, the video game knowledge that Brian draws from throughout the interview seems to be incompatible with formal mathematics knowledge. Moments where Brian is forced to shift into a different epistemological frame are particularly enlightening for theorizing about game design. The most apparent epistemological shift occurs when the researcher introduces the graphing task. In this task participants are given a sheet of graphing paper with the x-axis labeled as “time” and the y-axis labeled as “velocity.” The interviewer uses a paper speedometer with a movable needle to indicate different actions of a hypothetical car, and the participant is asked to “graph what you think is happening.” Brian, who has talked freely and playfully during the interviewer while enacting a “game story” framing, becomes uncomfortable and stumbles a bit. Brian pauses, and for the next ten seconds mumbles, “Uh, wait. I’m trying to think which way to go. Lets see, the faster...the faster it moves, which means time…alright, ok, I’m ready.” In this moment Brian’s physical demeanor changes, as does his playful attitude. Both his verbal “wait” and visible behavior shift indicates Brian is likely experiencing a frame mismatch between his previous “game story” frame and a “school knowledge” frame he interprets the question to be about. Brian sees this “school knowledge” frame as being about providing a formal representation that involves certain rules and vocabulary. This mismatch means abandoning the previous “game-story” frame in favor of the alternate “school knowledge” frame. As he discusses his graph with the researcher, Brian doesn’t invoke a fictional car as it might act in a video game as he has done throughout the interview when confronted with other non-game representations (such as the speedometer). Instead, Brian describes the action of “plotted points.” He states: Brian: Let's see...so, since the y is the speed velocity, like miles per hour, uh...cars, the car increased as the plot, as the plotted points are moving more up. As long...the steeper the slope is the speed of the car. Interviewer: The, the steeper the slope is the speed? Brian: Yep. Interviewer: Okay. Brian: The speed rate. At this point of the interview Brian struggles to think of the correct word and the result is statements that seem initiated to use science vocabulary like “slope” as well as mixed terms such as “speed velocity” and “speed rate.” This remarkable shift continues for a few moments until he once again relaxes, and then returns to describing his graph as how it might be formed by a car in a video game — sharp phases of acceleration are described as “getting a mushroom” or “hitting a turbo boost” and sharp phases of negative acceleration are described as “bumping into something.” Of interest in this analysis is how his earlier “game story”

© ISLS

122

ICLS2012

Volume 1: Full Papers

epistemological framing of the activity shifts to a very different “school knowledge” framing. This “school knowledge” framing assumes knowledge comes from expert sources (books, teachers, etc), requires specialized representations and vocabulary, and is about recitation, rather than knowledge creation. While in this frame Brian does not see his expert gaming knowledge as relevant.

Post-game interview: An integrated epistemological framing
While there is still some residue of his earlier framing in the post-game interview, Brian seems to have engaged a highly flexible epistemological frame that privileges both game and formal knowledge. While segments of the post-game interview show Brian continues to answer some motion questions using video game stories, such as describing phases of acceleration as “boosts” and occasionally giving the interviewer an “instant replay” of his movements with the toy car, when describing the motion of two different toy cars being pushed by the interviewer across the table (one is being pushed with constant velocity, the other constant acceleration), he casually states “Well the other one was just this constant rate, but what you did just now, be where...where the slope would increase.” Brian’s use of the term “slope,” invoking a graph representation, is particularly interesting as no graphs have been used yet at this point in the post-game interview. When pushed by the interviewer to explain what he means by slope, Brian states, “well that’s what I learned in algebra.” Later, when describing his post-game graph, Brian frequently uses terms like “constant rate,” intermingling this formal vocabulary with more informal game-like terms like “boost.” Of note is that Brian’s use of this formal vocabulary is not marked by visible shifts in behavior or language patterns, as was the case in the pre-game interview. In the post-game interview, Brian freely uses terms like “slope,” “constant rate,” and “turbo boost” apparently without shifting between epistemological frames. In the pre-game interview, Brian almost exclusively utilized a “game story” frame when answering motion questions using the toy cars. It was only when given the graphing task that Brian attempted to use formal school-like knowledge after a visible shift from his previous epistemological framing. While further investigation shows Brian’s explanation of slope is somewhat faulty, the fact that Brian spontaneously and smoothly integrates this formal knowledge from his algebra class into his discussion of the motion of toy cars indicates he is engaged in a very different epistemological framing, one that draws on both game and formal knowledge and is more about knowledge creation, rather than knowledge regurgitation.

Discussion
In this section, we propose a possible mechanism provided by the design of FTR that might lead to the integrated epistemology enacted by Brian in the post-game interview. We also discuss the implication of this case study for future iterations of FTR and other games intending to take advantage of informal game play.

Figure 1. Some representations are better at foregrounding kinematic concepts than others. Before playing the game, video game and formal representations cue different epistemological framings. In the pre-game interview, Brian’s answers to questions and the language used to describe various representations suggest he is framing the epistemological activity as being about drawing on video game experiences. This is particularly true when interview questions utilize a speedometer or toy car and track representations. While it is possible that these representations prompt Brian to engage this gaming epistemological framing, we think it is more likely that the association of the interviewer with games prompted this framing, and since these representations are common in typical commercial racing games, they did not prompt an epistemological shift. Utilizing the toy car and track and speedometer representations, Brian was able

© ISLS

123

ICLS2012

Volume 1: Full Papers

to reason about velocity and acceleration with little trouble, though answers to questions typically focused on one concept at a time, with little discussion of how these concepts intertwine (Figure 1). In contrast, the formal representation of a velocity versus time graph is particularly suited to the integration of velocity and acceleration. While Brian is familiar with this representation, in the interview, the simple introduction of graphing papers causes Brian to shift into a new epistemological framing. Now Brian sees the task as being about formal school-like knowledge. While in this framing Brian ceases to draw on video game knowledge, something he has done throughout the interview, and instead focuses on answering questions using sanctioned science vocabulary. While his explanation is not entirely correct, we see evidence of Brian attempting to use the graphing representation to reason about both speed and acceleration as he not only discusses the height of the “plotted points” but also the steepness of the slope. During the post-game interview Brian engages a new epistemological framing that allows him to freely use both formal and gaming knowledge. Where did this epistemological framing come from? We propose that specific designs of FormulaT Racing – particularly the intuitive and embodied control interface and the programming phases of the game – encourage players to tie representations that highlight individual concepts of velocity and acceleration to powerful formal representations that integrate the two (Figure 2). Once these formal and informal representations are effectively linked, the player no longer sees the interview task as privileging either game or formal knowledge and instead sees the task as being about constructing new knowledge by drawing on game, “real-world,” and formal experiences.

Figure 2. Programming by painting and graphing in the construction phase, and the use of the motion sensitive controller in the racing and construction phases effectively ties the various representations in FTR together. This results in a new epistemological framing for Brian that privileges knowledge construction. As discussed in the game design section of this paper, FTR includes a number of important representations meant to foreground various kinematic concepts. Of the new representations, most important to this discussion is the color-trail representation. This representation is particularly good at highlighting the player vehicle’s current velocity. Moreover, this representation allows players to visually see how the vehicle velocity relates to various track features. The track itself is a representation that is good at highlighting acceleration. The animation of traditional racing video games, the controlling scheme used, as well as real-world experiences shared by American youth provide the player with enough knowledge to know he should slow down around turns and speed up when the track is straight. Finally, a formal graph, which is included instead of a more traditional speedometer, is very effective at foregrounding both acceleration and velocity. However, like the speedometer in traditional racing video games, players rarely consider this representation during a race. While the new representations in FTR are important, the construction phase of the game effectively ties these representations together in a new way, encouraging players to integrate game and formal representations. This happens first by pairing the painting and graphing constructions within the same game “level.” While the interactions in the painting and graphing constructions are somewhat different, because they share a common narrative and goal in the game (that of directing other cars as the pit boss) the representations become highly intertwined. Finally, the two phases share the activity of programming in some form. Whether painting or graphing we see evidence of players systematically enacting computational strategies (Holbert & Wilensky, 2010b). Video evidence suggests players rarely reference the velocity versus time graph during the racing phases, however, the graphing representation utilized in the construction phase becomes tied to specific track features through the accelerometer-based interface. When players drive the car in the racing phases, they do this using a motion-sensitive controller. This interface foregrounds acceleration as it allows players to feel moments of speeding up and slowing down when they approach straightaways and turns. We’ve adopted this same
© ISLS 124

ICLS2012

Volume 1: Full Papers

interface for the graph construction phase. Here, as we’ve suggested before, players construct a graph that the car will use to drive around the track. Players construct these graphs by “accelerating” points up and down the y-axis in the same way they accelerated their car around the track—though here, instead of dealing with acceleration in the moment, players consider acceleration over the course of an entire race. By tying these two representations, the track features and velocity versus time graph, though embodied controller use, we believe players more easily see the graphs as being highly relevant to specific race action. As the graph, track, and color-trail representations begin to be tied together by programming in the construction phases and the embodied controller use throughout the game, Brian more easily integrates previously incongruent epistemological frames. While we did see epistemological shifts with many participants, Brian’s merger of game and formal epistemologies was unique. We believe this might be due to Brian’s very salient identity as an expert gamer. Furthermore, as our oldest participant (14), Brian likely has more formal experience with the mathematics embedded in FTR. His ability to integrate previously learned knowledge from the classroom with his FTR video game experience gives us hope that younger players might utilize FTR experiences in the classroom. This case study highlights the importance of both 1) designing new representations that provide infrastructure for complex cognitive tasks and 2) providing mechanisms for connecting representations to alternate contexts and epistemological frames. The very best representation will only be effective if it is seen as relevant in various contexts. As we move forward, we should continue to strive to make connections to more casual gamers as well as those that consider themselves experts. This may mean making certain relevant game features less nuanced. For example, the “forced braking” is only noticed by expert gamers and as such, may be missed entirely by novice gamers. In addition, some features that are large breaks from gaming tradition, such as the painting and graphing phases, may remain unconnected to other commercial racing video games (or even the more traditional phases in FTR) for novice and expert gamers. In this case more care should be taken to gradually introduce these important phases so they can continue to be seen as connected to the rest of the racing genre and typical gaming culture.

Conclusion
Video games permeate the lives of young people in America and continue to be highly relevant in youth culture. Video games are a rich space in which children can experiment with scientific phenomena. Whether we design games for the classroom or beanbag chair, it will be important for designers and researchers to consider player’s personal epistemologies both in and out of the gaming context. If neglected, we run the risk of designing isolated experiences that only matter in the moment, rather than providing a rich space for integrating the virtual with the real and the informal with the formal.

References
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86-107. Clark, D. B., Nelson, B., Chang, H., Martinez-Garza, M., Slack, K., & D'Angelo, C. M. (2011). Exploring Newtonian Mechanics in a conceptually-integrated digital game: Comparisons of learning and affective outcomes for students in Taiwan and the United States. Computers and Education, 57, 2178-2195. Clark, D. B., Nelson, B., D'Angelo, C. M., Slack, K., & Menekse, M. (2010). SURGE: Assessing students' intuitive and formalized understandings about kinematics and Newtonian mechanics through immersive game play. Paper presented at the Annual American Educational Research Association 2010 meeting, Denver, CO. Collins, A. (1995). Design issues for learning environments. In S. Vosniadou, E. d. Corte & H. Mandle (Eds.), International perspectives on the psychological foundations of technology-based learning environments (pp. 347-361). Hillsdale, NJ: Lawrence Erlebaum. diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and instruction, 10(2 & 3), 105-225. diSessa, A. A. (2000). Changing Minds: Computers, Learning, and Literacy: The MIT Press. diSessa, A. A., Hammer, D., Sherin, B., & Kolpakowski, T. (1991). Inventing graphing; Meta-representational expertise in children. Journal of Mathematical Behavior, 10(2), 117-160. Habgood, J. M. P., & Ainsworth, S. E. (2011). Motivating children to learn effectively: Exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences, 20(2), 169-206. Hammer, D. (1996). Misconceptions or p-prims: How may alternative perspectives of cognitive structure influence instructional perceptions and intentions? Journal of the Learning Sciences, 5(2), 97-127. Hammer, D., & Eleby, A. (2002). On the form of personal epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs and knowledge and knowing (pp. 169-190). Mahwah, NJ: Erlbaum.

© ISLS

125

ICLS2012

Volume 1: Full Papers

Hammer, D., Eleby, A., Scherr, R. E., & Redish, E. F. (2005). Resources, framing, and transfer. In J. Mestre (Ed.), Transfer of learning from a modern multidisciplinary perspective. Greenwich, CT: Information Age Publishing. Holbert, N. (2009). Learning Newton while crashing cars. Games, Learning, and Society 2009: Madison, WI. Holbert, N., & Wilensky, U. (2010a). FormulaT Racing. Evanston, IL: Center for Connected Learning and Computer-based Modeling. Holbert, N., & Wilensky, U. (2010b). FormulaT Racing: Brining kinematics to the bean bag chair. Paper presented at the Games, Learning, and Society 6.0, Madison, WI. Holbert, N., & Wilensky, U. (2011a). FormulaT Racing: Designing a game for kinematic exploration and computational thinking. Proceedings of the 7th Annual Games, Learning, and Society Conference, Madison, WI. Holbert, N., & Wilensky, U. (2011b). Putting the turtle on the racetrack: Investigating a constructionist racing game for exploring kinematics. Paper presented at the NARST, Orlando, FL. Holbert, N., & Wilensky, U. (2011c). Racing games for exploring kinematics: A computational thinking approach. Paper presented at the American Educational Research Association 2011 meeting, New Orleans, Louisiana. Ito, M. (2007). Education vs. Entertainment: A Cultural History of Children's Software. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning, 89-116. Lakoff, G., & Nùñez, R. E. (2000). Where mathematics comes from: How the embodied mind brings mathematics into being. New York: Basic Books. Lenhart, A., Kahne, J., Middaugh, E., Macgill, A. R., Evans, C., & Vitak, J. (2008). Teens, Video Games, and Civics: PEW Internet & American Life Project. Nemirovsky, R., Tierney, C., & Wright, T. (1998). Body motion and graphing. Cognition and Instruction, 16(2), 199-172. Papert, S. (1980). Mindstorms. New York: Basic Books. Papert, S., & Harel, I. (1991). Situating constructionism. In S. Papert & I. Harel (Eds.), Constructionism. New York: Ablex Publishing. Rideout, V. J., Foehr, U. G., & Roberts, D. F. (2010). Generation M2: Media in the lives of 8- to 18-year olds: The Henry J. Kaiser Family Foundation. Roschelle, J., Kaput, J. J., & Stroup, W. (2000). SimCalc: Accelerating students' engagement with the mathematics of change. In M. J. Jacobson & R. B. Kozma (Eds.), Innovations in science and mathematics education: Advanced designs for technologies of learning (pp. 47-76). Hillsdale, NJ: Earlbaum. Sengupta, P., & Wilensky, U. (2009). Learning electricity with NIELS: Thinking with electrons and thinking in levels. International Journal of Computers for Mathematical Learning, 14(1), 21-50. Sherin, B. (2000). How students invent representations of motion: A genetic account. Journal of Mathematical Behavior, 19(4), 399-441. Sherin, B. (2006). Common sense clarified: The role of intuitive knowledge in physics problem solving. Journal of Research in Science Teaching, 43(6), 535-555. Squire, K., Barnett, M., Grant, J. M., & Higginbotham, T. (2004). Electromagnetism supercharged!: Learning physics with digital simulation games. Proceedings of the 6th International conference on Learning Sciences, Santa Monica, California. Tasar, M. H. (2010). What part of the concept of acceleration is difficult to understand: the mathematics, the physics, or both? ZDM Mathematics Education, 42, 469-482. Trowbridge, D. E., & McDermott, L. C. (1981). Investigation of student understanding of the concept of acceleration in one dimension. American Journal of Physics, 49(3), 242-253. Wilensky, U. (2001). Embodied learning: Students enacting complex dynamic phenomena with the HubNet architecture. Paper presented at the annual meeting of the American Educational Research Association, Seattle, Washington. Wilensky, U., & Papert, S. (2006). Restructurations: Reformulations of knowledge disciplines through a change in representational forms. [Unpublished working paper]. Center for Connected Learning and Computer-Based Modeling. Northwestern University. Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories--and embodied modeling approach. Cognition and Instruction, 24(2), 171-209. Wilensky, U., & Papert, S. (2010). Restructurations: Reformulating knowledge disciplines through new representational forms. Proceedings of Constructionism 2010, Paris, France. Wilkerson-Jerde, M., & Wilensky, U. (2010). Restructuring change, interpreting changes: The deltatick modeling and analysis toolkit. Proceedings of Constructionism 2010, Paris, France. Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625.

© ISLS

126

ICLS2012

Volume 1: Full Papers

Dilemmas of Promoting Expansive Educational Transformation through Information and Communication Technologies (ICTs) in Botswana
Batane, Tsepo, University of Botswana, Private Bag 0022 Gaborone, Botswana, email: batane@mopipi.ub.bw Engeströ Ritva, Institute of Behavioural Sciences, P.O Box 9, 00014 University of Helsinki, Finland, m, email: ritva.engestrom@helsinki.fi Hakkarainen, Kai, Department of Education, Assistentinkatu 5, 20014 University of Turku, Finland, email: kai.hakkarainen@helsinki.fi Newnham, Denise Shelley, Route de Chailly 14 1815, Clarens, Switzerland, email: deniseshelleynewnham1@gmail.com N‟leya, Paul, University of Bostwana, Private Bag 0022 Gaborone, Botswana, email: NLEYAPT@mopipi.ub.bw. Virkkunen, Jaakko, Institute of Behavioural Sciences, P.O Box 9, 00014 University of Helsinki, Finland, email: jaakko.virkkunen@helsinki.fi Abstract: The purpose of the present paper is to address challenges and obstacles that educators and researchers face while promoting expansive educational transformation through technology-mediated learning in developing countries. The paper reports challenges and possibilities encountered in Botswana Expansive School transformation (BeST) project by describing preliminary results of creating cross-functional networks, organizing ICT workshops and Change-Laboratory interventions in collaboration between European and African researchers.

Introduction
The Revised National Policy on Education in Botswana is designed to prepare the country for the transition from an agro-based economy to an industrial economy driven by information technology that the country aspires to. In light of the rapid technological changes that are taking place both within the country and in the rest of the world, the policy has called for the introduction of computer use in schools. The purpose of the present investigation is to use methods of the Cultural-Historical Activity Theory (CHAT), such as Change-Laboratory (CL) interventions, to promote educational transformations in the context of ICT-mediated learning in Botswana. Rather than seeing computers as mere technological tools, CHAT guides one to examine technology-mediated learning as an integral part of the contradiction-laden historical development of educational activity calling for profound transformation of social practices prevailing at school and well as renegotiation of relations between schools and the surrounding community. Accordingly, there appears to be an complex interplay between technological and social innovations (Perez, 2002). When a revolutionary combination of technological innovations emerges, it is usually followed by intensive capital investment in technology implementation and creation of the required novel infrastructure. This phase is characterized by a contradictory combination of overoptimistic technology „hype‟ and a propensity to apply new technologies within the existing institutional practices. After the economic bubble based on such illusory hopes explodes, a qualitatively different phase emerges that is characterized by broad application of the new technology across society as well as transformation of prevailing institutional practices according to the possibilities of the new technological infrastructure. Venkatraman (1991) argued that the gains achieved when ICTs are used to support local needs for internal integration of activities in business organizations are relatively meager compared to their use for business-processes engineering, for transforming enterprise networks, and for redefining the scope of the business activity. A corresponding potential of facilitating revolutionary development of educational activities has emerged. Initially, ICT were used to elicit traditional learning tasks. Gradually integration of compartmentalized subject domains has become possible in project-based learning activities of students and teachers. A more revolutionary development would involve re-conceptualizing learning processes in terms of technology-mediated investigative learning and knowledge-building projects, transforming and creating a new collaborative network, and redefining the scope and object of the school‟s educational activity. The first wave of implementing and using this technology in education in developed countries in the 80s and 90s was characterized by the computerization of traditional forms of teaching and school activities. Movement toward the later, transformative phase of educational use of ICTs has been gradually taking place in developed countries since the turn of the millennium. However, an on-going phase involving the re-definition and recreation of educational functions of the school with the help of ICTs is predominantly lacking in developing countries. The introduction of ICTs has somewhat remained at the level of eliciting traditional learning tasks and teaching styles. Such a historical point of stagnation can only be overcome through the introduction of new ways of perceiving the school activity. Rather than taking the school and its functioning as a given object of
© ISLS 127

ICLS2012

Volume 1: Full Papers

development, the challenge is to radically re-conceptualize what school and learning are all about, redefine the boundaries of school and community, and explore expansive possibilities in practice. Prevailing social practices in schools channel teachers‟ and students‟ activities in a way that is very hard to change without extended deliberate efforts and support from outside intervenors. In order to elicit expansive educational transformation, it appears necessary to involve a heterogeneous network of actors with expanded mandates and perspectives of development. Both vertical dialogue between different levels of authority and horizontal dialogue between various actors involved in transformation are needed. Rather than directly transferring models of using ICTs, it might be advantageous for developing countries to utilize the historical experiences of developed countries as resources for reflecting critically on their own current practices and trajectories of development of school. In order to avoid typical pitfalls of interventions carried out between developed and developing countries sophisticated interventional methodology is needed that allows the participants to make visible, reflect on and deliberately master various aspects of their collective school transformation efforts (Long, 2001), Change Laboratory (CL), based on and the methodology of Developmental Work Research (DRW), Engeströ 1987), is an already well-defined intervention method to support m, organizational transformations (Engeströ et al., 1996). In CL interventions, workplace communities reflect m upon their prevailing practices on the basis of data; they analyze the historical development of their current activity system and its inner contradictions, as well as envision and implement, through practical experimentation with a new model of the activity. These collaborative reflections are triggered and guided by the researchers, who collect and prepare ethnographic “mirror” data for assisting participants to reflect on their work. CL comprises a series of workgroup meetings to reflect on current practices and envision future activities. Essential aspects of CLs are; a) using videotaped practices as a “mirror” for assessing current activity; b) generating ideas and tools (e.g., charts) that help to assess past, present and future activity; and c) modeling present practices by using an activity-system analysis. Sharing participants‟ memories of the past assist the workplace community in analyzing the historical development of its activities. Within a CL process, a facilitator in the initial organizes, provokes and guides participants‟ learning activity through discourse, and expansive learning actions aimed at questioning, analyzing, modeling and transforming the activity‟s system. Learning activity is a process, in which the participants ascend from the perspective of their individual work to the joint analysis and development of the system of their actual productive activity in order to radically transform the former into a new and prolific future form which has surpassed its initial contradictions. As a result a new form of productive activity evolves in which the participants increasingly cohere toward a shared object of transformation which is less determined by pre-planned tasks set by the facilitator but rather increasingly by the evolving object. For example, a dialogue between people who examine the school and children‟s everyday life from varying perspectives is complemented with a dialogue focused on analyzing the historical development of prevailing practices of schooling, examining educational practices in detail, and deliberately experimenting with various new solutions. The participants are themselves engaged in analyzing disturbances that hinder pupils‟ learning and the development of the school community in its extended social and cultural context. They are provided with conceptual tools for analyzing the data and ways of modeling the dynamic development of the activity system are suggested.

Figure 1. Elements of the setting of the Change Laboratory (Engeströ et al., 1996). m The introduction of ICTs into school activities opens up new possibilities for teachers‟ educational work and teaching as well as children‟s learning. In Figure 2, four ideal typical conceptions of school have been depicted by crossing two dimensions of expansion of the traditional school discussed in the literature (Engeströ et al., 2002). One of these depicts the type of problems school children are working on and m identifies the developmental trend from learning by solving given tasks for which the right answer is known to complex real-life problems that can have varying creative solutions. The other dimension describes a school‟s relationship to the surrounding community and goes from an isolated school to an open school, where teaching
© ISLS 128

ICLS2012

Volume 1: Full Papers

and education is realized in collaboration with the surrounding community. Taking the school texts as objects that have to be remembered in their own right is strengthened in a relatively isolated school that has little contact with the surrounding community. Opening the school up to the community and taking the vital problems of the community, and more broadly of mankind as objects to be solved with the help of culturally developed intellectual tools, can strengthen the pupils‟ agency in mastering their lives as members of a community. This model introduces the concept of ICTs as a tool for expanding the school ethos. For historical reasons, the western model of school brought to developing countries carries many of the characteristics of the old-time isolated school (Sector 1) in which teachers‟ work has been defined by subject-by-subject curriculum packages, stage-by-stage teaching models, standardized tests, and corresponding ICTs. The present trend of introducing test-based control systems in schools have strengthened the tendency to highlight memorizing isolated pieces of knowledge and skill on the cost of understanding and creative use knowledge and education (Ravitch, 2010). The increasing emphasis on test scores has broadened the gulf between school knowledge and students‟ creative interaction with and learning from real life problems and utilizing the new possibilities of digital networks of open innovation (Tapscott & Williams, 2006) and collaborative building of knowledge (Bereiter, 2002). Sector 2 represents various reform-pedagogical ideas of a school that follows ideas of technology-enhanced projectand problem-based learning. Sector 3 represents approaches that highlight the importance of creating students‟ and teachers‟ classroom learning communities focused on collaborative building of knowledge with the help of the new ICTs (Scardamalia & Bereiter, 2006); there is a tendency to examine the school community as relatively isolated from the surrounding society. Sector 4 combines the two dimensions of expansion for examining the collective zone of proximal development of school. In such a school, learning is to a great extent carried out by solving real-world problems in contexts of various hybrid forms of activities and networks of learning; these are processes, also, in which representatives of local productive work are involved in and in which the rules are different from those in the classroom (Hakkarainen et al. 2004; Yamazumi, 2006; Roth & Lee, 2007). Various ICTs from word-processing and spreadsheets to collaborative web publishing provide support for such activities. There is, however currently an aggravating contradiction between these new possibilities and the tendency to strengthen the traditional school through increasing testing.

Figure 2. Two dimensions of expansive school transformation.

Research Aims
The purpose of this paper is to describe experiences from the Botswana Expansive School Transformation (BeST) project by addressing the following questions: 1) How is expansive school transformation facilitated through Change-Laboratory (CL) interventions; 2) What kinds of actors and agencies are needed for facilitating ICT-mediated educational interventions in Botswana; and 3) What kinds of challenges and constraints emerge?

Methods, Data Collection, Data Analysis
In order to elicit sustained education transformation of developing countries, based on their needs and developmental potentials, one has to employ sophisticated intervention methodology beyond standard academic research methods. Toward that end, the present developmental intervention relies on multiple methods and associated data sources. Two five-day workshops of 10 pilot schools (2005; 2008) and several smaller meetings have been documented by audio and video recording. Beyond within-school activities, ethnographic data were collected, by interviewing parents and community members and observing local activities, such as Gotla (municipal) gatherings, animal husbandry and vegetable gardening, child care as well as health and social services. These observations gave an insight into how the school activity and the existing way of life of the community related one to the other. Such understanding is paramount, acting as a springboard to unearthing the contradictions existing within the activity. Multiple voices have been recorded by interviewing principals, teachers, students, parents, and other relevant peripheral or distant actors. Change-Laboratory, that allows local communities to deliberately make visible, reflect on, and transform their practices with the help of researchers, has had a central role in the present developmental intervention. Relevant parts of the ethnographic material were used as mirror material in CL interventions. The material collected provides a rich body of data concerning the prevailing practices of learning and instruction at the intervention schools and school-community relations. Qualitative content analysis has been used to assess ethnographic data, such as interviews and videotaped participant observations as well as discourse taking place in Change-Laboratory sessions.
© ISLS 129

ICLS2012

Volume 1: Full Papers

Preliminary Results of the Project
In the following, preliminary results of the present project will be summarized with a focus on the creation of new developmental collaboration. In order to achieve this end, the structure of the project and the first two CL interventions and data illustrating discussions between collective actors shall be presented and discussed.

Actors and Structure of the Project
The present project relies on collaboration between the universities of Botswana and Helsinki for supporting the educational use of ICTs in schools in Botswana and the educational administrators from Botswana. In the initial stages of the project, The University of Mauritus had been part of the project at its initial stage and some staff members attended the first workshops held but they later dropped out. As a result of negotiations and workshops between organizations involved a new unofficial boundary-crossing network of actors was created. The University of Helsinki held a one month Activity Theory Workshop for AS-TIG during 2008. Ten schools, representing five different areas of the country and different types of schools were selected as pilots in the educational use of ICTs and in each school a group of interested teachers, that represented the activity vertically and horizontally, were selected to be responsible for the measures of developing ICT use in the school. The workshops arranged for the ICT-groups of the ten schools provided opportunities for hands on learning of ICTs as well as planning and modeling transformation with the help of the activity system model. In order to have a bigger impact on education, it was essential to involve all ten schools rather than only three CL schools as well as a network of mandated change stakeholders: “There are drivers, the people who are mandated with providing infrastructure or guidelines so I suppose if they understand our needs and they can re-align our needs towards the vision …, they will ensure that whatever we are thinking of, they will provide a platform for us to be going in that line" (a ministry official). Teachers from the ten pilot schools (two teachers from each) and varying groups of local school officials, their supervisors, representatives of the universities of Botswana and Helsinki, Ministry of Education as well as other organizations took part in various workshops. In these workshops, the basics of activity theory was explained, its application in school development as well as the Change Laboratory method. Later on, an unofficial Activity Theory Special Interest Group (AT-SIG) that involved a heterogeneous group of researchers and educational administrators to discuss how ICT can be used to develop schools, was formed. The idea was that the group would function as a cross-functional and cross-organizational forum for developing a pedagogically grounded concept for ICT use in schools (Figure 3).

Figure 3. Cross-functional network for developing methods of using ICTs at schools The research team is currently analyzing results concerning ICT development workshops organized during the project; the workshops were video recorded and there are tens of hours of video data. The local participants‟ contributions to joint discussion as well as their narrative descriptions of situations at school have been transcribed. Researcher Lauri Hietajä is analyzing those contributions in which the participants addressed rvi challenges of implementing ICTs in the school activity. Preliminary results of the study indicate that at the discourse level, there was some improvement in the integration of ICTs in the pilot schools from 2005 to 2008. Discussions that focused on arising awareness of ICTs transformed toward discourse addressing possibilities of transforming educational practices.

Intervention at School A
The first CL-intervention took place at a senior secondary-level school A (with a population of 1800 students and about 150 teachers) in fall 2008. The school is situated in a village close to a major city. Ethnographic data were collected in school A across ten days by interviewing teachers, principals, community institutions, students, ICT‟s community centers, and parents in order to gain insight into the activity and to prepare models and mirror data for the CL. The audio and video recorded data were transcribed and field notes analyzed. The school had a very limited ICT infrastructure –about 20 computers in a computer class – and there was only a minimal ICTrelated transformation in educational practices in terms of teaching ICTs as a new school subject. While

© ISLS

130

ICLS2012

Volume 1: Full Papers

government policy required teaching ICT literacy to all students, it was hard to implement such expansion of ICT education due to lack of computers, frequent malfunctioning, constantly occurring viruses, slow or nonexistent computer network, and limited teachers‟ ICT competence. Ethnographic data indicated a strong tradition of categorizing students into different curricular groups on the basis of tests in sciences. Accordingly, computer classes were reserved for high achieving “double” or “triple science” students whereas automatically promoted low achievers (“single” science students) were stigmatized and their consequent tarnished feeling of self-worth aggravated by a lack of access to ICTs and consequently a developing world: R: “What about access to the computer lab T: It is a bit of a problem because the computer lab there is only one and there are something like 20 computers and there are 80 students doing ComputerStudies in the mornings they are attending their lessons and in the afternoons they are doing their projects so it is a bit of a problem to have access to a computer R: How many students do you have in the school? T: 1800 R: What is the nature of computer literacy across the board, do all of the students have computer awareness and what access do they have? T: Well like I said, there are only about 80 students, 40 form 4 and 40 form 5, in our committee, one of the responsibilities is that we would like to start computer awareness that is just the basic of ... or introduction to computers. And if we can start computer awareness, then we can get somewhere. R: So other students who are not doing computer studies are not having...don‟t have access to computers; T: Ya they do not have...” An interview with a Computer Science student revealed that the problem was not only due to a limited amount of technology but also instruction as well: “It is not like I am blaming any one but we have a very big problem. They just give those many textbooks and then they say go and read them but some of us are not conversant with the computer terms which makes it very difficult to understand and we do not get enough end of month tests. I am not going after anybody, I am just trying to remedy the situation”. Eight (8) to thirteen (13) teachers, representing multiple subject domains and levels of authority, formed the school‟s ICT group and, consequently, participated in the CL process. Two members of the local media and teacher training and development centre participated in all the sessions together with the moderating intervenors. The process consisted of eight (8) two-hour sessions at a rate of two, two hourly session per week for 4 weeks. All sessions were video and audio recorded and transcribed. The sessions followed the cycle of expansive learning in terms of analyzing and questioning the prevailing practices, examining the historical development of the activity, planning some concrete changes, implementing the changes, and reflecting on the process. The moderator set tasks in order to stimulate questioning of their activity such as: causes of joys and sorrows for teachers, students and parents; examining the historical development of the school activity in order to track the central contradiction, and envisioned changes that would make educational activity more meaningful for the participants. In order to stimulate the construction of new ideas, each task was supported by a second stimulus such as mirror data or theoretical models of schooling and learning. In one of the first CL sessions, the researchers projected a video clip, onto the mirror board, of a student testifying how being categorized as a single science student engendered stigmatization by teachers and fellow pupils and that some teachers refused to answer questions and provide added support. This piece of mirror data elicited a lively discussion with tones of self-critique among the participants, the consequence of which was a request for further data on the problem. Later on, the same theme came up in another form when the teachers reflected on methods of Science education that involve asking the students to plan and carry out experimental projects; it was pointed out that the single-science students would not have capability to do it and should, therefore, not be given such tasks. A historical analysis revealed that a former established government policy, requiring automatic promotion of all students to junior and senior secondary schools regardless of their educational achievements, has made the student population more heterogeneous. In order to cope with the prevailing heterogeneity of student learning abilities, categorization was installed. As a consequence, there appeared to be a tension-laden contradiction between the object of teacher‟s instructional activity for all and prevailing instructional methods and practices that did not tailor and personal instructional efforts according to students‟ competencies. The tendency to categorize and segregate students according to different levels of accomplishments created some visible contradiction in the school system. Participants envisioned two solutions for overcoming the double binds together with the intervenors: 1) In order to facilitate students‟ motivation, the participants designed a new tool (personal study plan) for addressing a student‟s studying conditions, problems, and study objectives. The design aimed at bringing student weaknesses and strengths to the fore and subsequently to create a peer support network across the categories. 2) the teachers initiated, moreover, experiments with co-teaching so as to provide more time for individual student support. Neither of these solutions was based on the immediate use of ICTs, which is

© ISLS

131

ICLS2012

Volume 1: Full Papers

understandable in view of the limited ICT resources and lack of ICT competencies of the teachers. Yet, the new ideas of a more personalized way of teaching could later be sustained with the use of ICTs as a tool to lead to a more enabling learning environment. Furthermore, the teachers organized collaboration with the local Audiovisual Center for support and teaching in creating ICT based teaching aids. Related to the model in Figure 1, these initiatives would lead to changes on levels 2 and 3. An analysis of transcribed CL data indicates that while ICT-related discourse surfaced frequently, presumably because of limited experience, the participants were not able to sustain concrete ICT-related discussions, their focus tended to gravitate toward other themes. Nevertheless, making the computer into an instrument in teaching other subjects would, however, mean a major qualitative step in the development that could only have been taken with an expanded ICT infrastructure. The results of the first CL intervention in the AT-SIG surfaced tensions between ICT implementation and school development. Firstly, the teachers, due to new measures, had difficulties in finding time for the further development and experimentation of the new solutions. Secondly, after leaving the country, the foreign intervenors could not adequately continue supporting the educational transformations needed after the 8 CL sessions and the internal change agents were displaced to new geographical areas. Thirdly, as a consequence of the centralized management of Botswana teaching service , which enables teachers to move from one school to another across their careers, (in pursuance for career progression), some key CL teachers participating in developing the new solutions were transferred to other schools before they could elaborate the new tools and practices. Accordingly, the teachers did not have a sufficient support network to expand on the new practices despite adequate analyses of the developmental challenges having occurred through the CL. The envisaged virtual support platform that was planned to be provided by the University of Mauritius as a collaborating partner did not come forth and thus much was lost in this domain.

The CL Intervention at School B.
A recurrent problem in the development of ICT-based learning environments has been the lack of student motivation related to the lack of authenticity of the problems dealt with. The second CL intervention specifically addressed this problem at school B. A school located in a remote and rather deprived rural area in the Kalahari Desert in which the livelihood was mainly based on animal husbandry. School B had 800 students 80% of which are boarders and 42 relatively inexperienced teachers inexperienced in what?sent to the school from elsewhere for a 5-year-period and who maintained a strict separation from the village population of about 1590. The ethnography based on interviews and discussions with teachers, principals, community institutions, students, ICT centers and parents revealed a cleavage between school, parents and the village officials each blaming the other of problems in child rearing in the village. Moreover, teachers are far away from their families which as the following testimony indicates, impacts on their teaching: “Every time I come back here I have my body here but my heart is with my wife and girls…they need my affection and support…what is a dad that is not at home…we cannot be good teachers with our hearts broken like this” (field-notes, 2007). Teachers generally blamed parents for a lack of interest in the children‟s education which they said was manifested by a lack of school function attendance and alcohol abuse. Because parents felt that they had little control over their children and little to do with their school going activity, they could not find productive solutions. The intervenors hypothesized that the problems lay in the school-community relationships rather than within the school. Instead of inviting only the teachers to develop the educational practices within the school, village officials, teachers from the primary and secondary school and parents were invited to take part in a boundary crossing CL process. Participants of CL were composed of eight (8) members of the Junior secondary school, five (5) members of the primary school, two (2) members of the vocational school, two (2) members of the non-formal sector, several parents (later this number was around 8), and odd members of the different institutions, a total of twenty-five (25) people. The sessions were organized as two hourly- twice per week sessions for 5 weeks; most of them involving both school and community. Introductory sessions were held separately with each school. All session were video-taped and voice recorded and transcribed. The CL experimental space was held at the local „Kgotla‟ (court house), a thatched roofed open- to- the-environment building situated half way between the three educational institutions. The boundary crossing CL began by showing part of the ethnographic data including the interview with a parent welder who testified to the estrangement between the school culture and arts and crafts culture prevalent in this area. The participants then discussed the historical changes that had taken place in child rearing practices between home and school in the village. They came to a collective decision that the children performed better when school lessons were given under the tree and child rearing had been a shared responsibility. They began to question what had led to this transformation: Head teacher: “It is something like teacher 2 said, mind your own business. It is like parents are not very much supportive to the program for the school, so the parents live their own life there, teachers live their own life, other people in other departments live their own life and people do not come together and support one another it is like there are some barriers in between if the school ask for assistance from the community maybe to say come for a board collection or come to address the issues related to peoples performance and so forth they

© ISLS

132

ICLS2012

Volume 1: Full Papers

come with a different mind to be against what the school is trying to do or to try to be on the side of the kids even when they have done something wrong. So I think that it is a very serious limiting factor because when they come they come to say no even when their kids are wrong they say that they are not wrong. As if we are not good enough to the kids we always say they have done ABCD which is not good and so I think that there is a problem there”. Children‟s voices were as well part of the mirror data and used to maintain focus on the object. A metaphor of a “school under the tree” arose and grew until it became the pivoting point where parents and teachers began to reconstruct models for a learning environment that was both for the wider world and the everyday. In the extract above the school head suggested a new way to understanding the problem of teacher parent collaboration. This insight was strengthened through the introduction of further ethnographic data in which parents discussed a “bad child” and led to discussion in which it was agreed that “parents, teachers and community are all ultimately responsible for outcomes of children under their care; all should work to promote cases of the „good child‟ and to minimize cases of the „the bad child.‟ The computer room of the Junior Secondary school contains twenty-five (25) a computers and all children have access at different periods of the week. All teachers are encouraged to plan their lessons and write reports as well as produce exam and test results on the computers. During the first field study period, this laboratory was relatively under exploited but two years later, it was always busy and the level of computer literacy amongst the teaching staff appeared much higher. However, teachers were not using ICTs as a tool to stimulate the learning and development of their students. It appeared clear that creating novel ways of using ICTs as instructional tools and overcoming contradictions embedded within educational the institution and its relation to surrounding society were needed for bringing pedagogical benefits about. Their new designs included the cultivation of vegetables gardens, an arts and crafts center and a bakery projects. These work groups consisted of teachers of the Junior secondary and primary schools as well as the vocational schools, principals of the schools, members of all of the village institutions and parents. These projects planned to make use of all the subjects offered within the curriculum such as Mathematics, Geography, Science and the integration of computers. The concept is that of a school-community type of learning where children are actively involved in the creation of their own knowledge. ICTs are to be used in order to measure, monitor, and design implementation and progress of the projects. Such a leap requires careful planning and nurturing as it is far easier once the contradictions between the old activity and the new design surface to return to the former way of working. Spirits were high in the community after the conclusion of the CL. This was expressed in the words of a member of the Village extension team: “I never thought that I would see the day when parents and teachers came together to work on their children‟s future”. Table 1. A Summary of focuses on the two Change-Laboratory interventions School A School B School Senior secondary school Junior secondary school and Primary school Location Relatively urban Rural Ethnography Interviews of students, teachers, principals, Interviews of students, teachers, parents, parents, educational facilitators principal and community members Teachers, an intervenor, media centre, AS-TIG Community stakeholders, teachers, AT-SIG Participants members members and an intervenor Process of Nine sessions.(3 with the two schools and 6 Seven sessions. intervention with the schools and community) Developmental Collaborative study planning to enhance student Joint pursuit of an agricultural project and motivation; learning to create audiovisual an arts and crafts project experiments teaching aids to enhance teacher motivation; coteaching, system of controlling the progress in the implementation of new policies.

Discussion
The present investigation focused on examining how CL-intervention is used to elicit expansive school transformation in the context of technology-mediated learning. We considered future possibilities of such usage. The analysis revealed how teachers and educational institutions supported by interventionists explored ways to facilitate educational transformations in difficult conditions. Although there was not an adequate infrastructure, a high shortage of computers, and limited human resources, many critical issues were addressed regarding student motivation, collaborative teaching, or transformed school-community relations. Research on the use of ICTs in schools has shown that the greatest gains from new technology are achieved by redefining the content of the educational activity and using the new artifacts for realizing something that was previously not possible.

© ISLS

133

ICLS2012

Volume 1: Full Papers

Regarding teaching, this means, that the teachers have to rethink the pedagogical principles of their teaching and to use ICTs to realize a new kind of pedagogy. The present experiences of using CL for promoting such educational transformation and experiences are encouraging, in spite of the challenges involved in working through various linguistic, social, and cultural boundaries. It appears, however, that without personal and collaborative experiences of using ICTs, discussion in CL sessions tended to drift to issues other than ICTs (such as how to deal with diversity, overcome prevailing strong categorization practices and cope with common place corporate punishment). As far as CL sessions involve teachers only talking about technology – without appropriating ICTs as a part of their everyday activity – the interventions may not have sustained effect. Thus far, interventions have only hinted as to expansive possibilities provided by ICTs, and there is more talk about technology-mediated learning and instruction than its actual concrete exploration. The slow progress reported in the African context does not, however, appear to be an exception but a rule of technology-mediated learning. Our experiences from ICT in education projects from Finland and Europe (Lehtinen et al., 2001; Sinko & Lehtinen, 1999) indicate that educational transformations take years to consolidate. The present investigation aimed also at analyzing challenges and constraints of ICT-mediated educational intervention. Much of the research on the use of ICTs in schools has focused on analyzing the use and reasons of lack of use of ICTs, assuming uncritically that these technologies help the schools to carry out their societal task. Fewer studies have been carried out on what inner developmental contradictions in the current school education could be overcome with the introduction of these technologies. The expansive use of ICTs cannot, however, derive sufficient motivation from the new technology as such but must be based on well analyzed developmental challenges of the school that the teachers recognize and are ready to work on. According to the previously quoted theory of Perez (2002) the broad societal utilization of new technologies only started after the implementation of the technology has been carried out by actors representing the new technology. A turn from technology push to need pull takes place as the agency in the development moves incrementally from the providers to the users. In this research we have used the Change Laboratory method to help the practitioners to analyze the central developmental challenges and possibilities of their educational activity. One can argue that this need-based approach came to the pilot schools too early in view of the implementation of the new technologies. On the other hand, the new technologies can never be the only and the whole solution to a major historical challenge of development in an activity. Rather the needs and possibilities of the activities are more aptly recognized by a group of actors involved in an ongoing process of developing their educational activity, rather than by individuals carrying out their traditional work routines.

References
Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Erlbaum. Engeströ Y. (1987). Learning by Expanding. Helsinki: Orienta-Konsultit Oy m, Engeströ Y, Engeströ R., & Suntio, A. (2002). Can a school community learn to master its own future? In m, m, G. Wells & G. Claxton (Eds.), Learning for life in the 21st century (pp.211-224). Cambridge, MA: Blackwell. Engeströ Y., Virkkunen, J., Helle, M., Pihlaja, J., & Poikela, R. (1996). The Change Laboratory as a tool for m, transforming work. Lifelong Learning in Europe, 2/1996. Pp. 10-17. Hakkarainen, K. Palonen, T. Paavola, S. & Lehtinen, E. (2004). Communities of networked expertise: Educational and professional perspectives. Amsterdan: Elsevier Lehtinen, E., Sinko, M. & Hakkarainen, K. (2001). ICT in Finnish education: How to scale up best practices. International Journal of Educational Policy, Research and Practice, 2 (1), 77-89. Long, N. (2001). Development sociology: Actor perspectives. London: Routledge. Perez, C. (2002) Technological revolution and financial capital. Cheltenham, UK: Edward Elgar. Ravitch, D.(2010). The death and life of the great American school system: How testing and choice are undermining education. New York: Basic Books. Roth, W.-M. & Lee, Y. J. (2007) “Vygotsky‟s neglected legacy”. Review of Educational Research, 77, 2, 186232. Scardamalia, M. & Bereiter, C. (2006). Knowledge building. In K. Sawyer (Ed.) The Cambridge handbook of the learning sciences (pp. 97-115). Cambridge, MA: Cambridge University Press. Sinko, M. & Lehtinen, E. (1999). The challenges of ICT in Finnish education. Juva: Arena. Venkatraman, N. (1994). It-enable business transformation. Sloan Management Review, 35, 73-87. Yamazumi, K. (2006) Learning for critical and creative agency. In K. Yamazumi (Ed.) Building activity theory in practice (pp. 73-97). Osaka: Center for Human Activity Theory. Kansai University.

Acknowledgments
The present project was supported by grant 1127019 from the Academy of Finland. The study relies on collaboration between Universities of Helsinki and Botswana.

© ISLS

134

ICLS2012

Volume 1: Full Papers

Visible Spending: Information Visualization of Spending Behavior and the Thought-Act Gap
Emily S. Lin, Tufts University, 105 College Ave, Medford, MA 02155, es.lin@tufts.edu Holly A. Taylor, Tufts University, 490 Boston Ave, Medford, MA 02155, holly.taylor@tufts.edu Abstract: Most people know they ought not to spend beyond their means. Yet, many do so anyway, often without being aware of it. To explore how to narrow this thought-act gap, we conducted a three-month mixed-methods pilot study (N=25) of the effects of graphical vs. textual feedback on the relationship between estimated and actual spending. Results of a 2 x 2 x 3 repeated measures, doubly multivariate ANOVA indicated that high-debt participants experienced improvement in their estimations more than low-debt participants, and that the estimations of participants who received graphical visualizations became more accurate. Excerpts from interview data shed light on the learning processes underlying such changes.

Introduction
Individuals make private financial decisions daily; sometimes these decisions result in taking unwise financial risks or simply spending beyond one’s means. The consequences of these decisions can be damaging on the individual and family levels, leading to such consequences as emotional stress, an inability to access needed resources, or bankruptcy (Drentea, 2000). Individual financial choices can also play a role in larger-scale crises. While multiple factors contributed to the recent global economic recession, a massive wave of consumer-level defaults on credit card debt and mortgages counts among them. Even at the bull economy’s peak, economists warned that individual consumer behavior – in the form of rising credit card and housing debt – could negatively impact the nation's economic future (Greenspan, 2004). Given that most people would agree that they ought not to spend beyond their means, we posit that some people simply may not be aware of the extent to which their day-to-day actions lead them to spend more than they take in. An individual’s financial profile thus calls on three different types of representations – a mental measure of intended spending, a mental measure of estimated spending, and an external measure of actual spending. For brevity's sake, we call these representations “oughts,” “thoughts,” and “acts,” respectively. Clearly, there are practical implications to understanding how we might “nudge” individuals toward aligning their oughts, thoughts, and acts. The present study builds on research on visual perspective, cognition, and behavior to explore how thoughts might play a role in affecting that alignment. In particular, we studied whether participants who received interactive graphical reports of their spending behavior modified their thoughts to produce more “desirable” financial outcomes than people who only received text-based feedback.

Thoughts and Failures of Self-Knowledge
Thoughts are usually discussed using an analytic frame. When I “think” that I am in control of my spending, I am able to give evidence – e.g., I was tempted to go out to a pub with friends but instead opted to stay in and cook, which was much cheaper – to frame my analysis of the situation. Because good thinking is understood to be tied to externally available evidence as well as a subjective analysis of that evidence, it is reasonable that thoughts would mediate the alignment between oughts and acts. However, when thoughts falter in their analytic rigor, oughts and acts – while appearing to be aligned – may in fact be at odds. Revisiting the earlier example, I might think I am in control of my spending because I opted to cook dinner at home rather than going out with friends, but I may be overlooking the sandwich and chips I bought every day for lunch, which add up to more than I would have spent going out to the pub that one night. People are predictably more likely to make these kinds of analytic slips of thought – and subsequently construct inaccurate mental models of their own behavior – when viewing situations from particular perspectives or through particular cognitive frames. For example, research has consistently shown that using a first-person rather than a third-person perspective yields less accurate perceptions of phenomena ranging from navigational orientation (Brunyé, Mahoney, Gardony, & Taylor, 2010) to judgment of one's causal understanding (Keil, 2008). Another common heuristic in these cases is confirmation bias (e.g., Nickerson, Perkins, & Smith, 1985), a framing that predisposes us to seek information and adopt interpretations consistent with pre-existing beliefs. To return to the prior example, if I believe I am “good” at resisting temptation, I am more likely to remember the confirming incident when I resisted going to the pub, and also to neglect the disconfirming incidents of buying convenience food. In addition to selective memory, confirmation bias can manifest in the face of disconfirming representations of self by making more salient contextual information that “explains” or “excuses” one's behavior in a way consistent with self-concept. Jones and Nisbett (1973) present a classic example of this “attribution error,” where others' behavior is seen as having dispositional causes but our own behavior is explained by situational causes.

© ISLS

135

ICLS2012

Volume 1: Full Papers

The question that naturally arises is how to enable perspectives and frames of mind that promote the development of more productive mental models of one's own behavior. Integrating the lines of research mentioned above, we suggest that the differentially accurate mental models produced by taking first- and thirdperson perspectives may be a result not of simply changing perspective, but of what changing perspectives gives you – a chance to see yourself through a different, and disconfirming, mental frame. Indeed, in their study of voting behavior, Libby, Shaeffer, Eibach, and Slemmer (2007) showed that the perspective changes that affected behavioral changes were mediated by changes in dispositional understandings of self. With this insight into the mechanism by which perspective shifts influence behavior change, we can propose the role of thoughts in connecting oughts and acts. Assume I have an ought α (e.g., “I should not spend too much money eating out,”) and want my act a to be in line with that α. My acts appear to be motivated by my thoughts about my disposition. That is, given thought A about my own disposition (e.g., “I am thrifty,”), I am motivated to commit act a (e.g., not going to a pub), which in concordance with that disposition. Additionally, though, my acts are also interpreted through my thoughts. After committing a, if I hold dispositional thought A about myself, I may overlook (via confirmation bias) or discount (via situational attribution) an act b (e.g., buying sandwiches for lunch) that does not appear to be in concordance with A. In this case, if I have ought α and dispositional thought A, but have actually committed act b, then I have an ought-act mismatch and am either not aware of it, or at least not likely to act differently should act b present itself again in the future because I did not incorporate b into a revision of A. Therefore, while A motivates a, it also pushes us to neglect instances of b.

Getting to “Good Spending”
From this analysis we ask, what next? Is our goal to change act b into a? To change ought α into a new ought β? Change thought A into a thought B that more accurately describes my disposition? We have arrived at a basic question – what kind of ought-thought-act relationship constitutes “good spending?” The typical “accounting” conceptualization is to hold oughts constant and get acts to match them – that is, good spending is spending within my budget. If my spending does not match my budget, then I will change my spending to match my budget, or change my budget to match my spending. Any dispositional thinking is fairly trivial in this view. However, there is another way to operationalize good spending, and that is to say that good spending is spending of which I'm aware. In this view, my judgment of myself as a “good” or “bad” spender is not contingent on achieving some pre-determined goal, but on whether I know what I'm doing. A focus on thoughtact gaps, rather than ought-act gaps, encourages us to think of spenders not as accountants, but as scientists, whose investigations are driven not on quick judgment but on testing and generating hypotheses, a process in which paying attention to unexpected findings (e.g., a thought-act gap) is a critical strategy (Dunbar, 2000). While the scientist’s framing of good spending may seem counterintuitive by appearing to value, for example, overspending of which I am aware more than under-spending of which I am not aware, it also takes a longer, more global view of spending than the accountant’s view. Indeed, the accountant may be more likely to stay within a month-to-month budget, but the scientist may learn how to shift resources adaptively, overspending when it is strategically worthwhile (e.g., taking financial risks to start a small business, or buying a higher-quality car that will last, rather than a more affordable one that will yield years of costly repairs). Motivated by the idea that good spending of the scientific sort can be learned with the appropriate cognitive scaffolds, the present work comprised a three-month mixed-methods pilot study to examine whether participants who received graphical representations of spending behavior achieved greater alignment between estimated and actual spending as compared to participants who received text-only representations. We propose that graphical representation of data (also known as information visualization) can help overcome confirmation biases in a way similar to the mechanism by which dispositional thinking circumvents those biases. Evidence for differential analytic results based on types of visual representations is found in decisionmaking. People select different options when the same information is presented differently or when decisions are made through different means (e.g., Kahneman & Tversky, 1984, 2000; Redelmeier & Tversky, 1992). In addition, the inability to make rational decisions is exacerbated by too much information (Lurie, 2004). Lurie and Mason (2007) have suggested that visualizations can address these analytic slips, and the growing field of information visualization (Card, MacKinlay, & Shneiderman, 1999) is producing relevant tools. We suggest that presenting information on personal spending through visualizations can take advantage of humans' consummate pattern seeking ability (Wolford, Newman, Miller, & Wig, 2004), therefore generating, in a single view, alternative but plausible dispositional interpretations of their spending pattern. Thus, in the present study, we predicted that participants who received graphical feedback would show a greater narrowing of the gap between estimated and actual spending – that is, between their spending thoughts and their spending acts.

Method Participants and Procedures

© ISLS

136

ICLS2012

Volume 1: Full Papers

The present analysis was conducted using data from a pilot study examining the spending behavior of 29 adults. Using e-mail, electronic advertising, and posters, we recruited people with either high credit card debt (>$2000 carried over from month-to-month) or low credit card debt (<$200 carried over from month-to-month) to participate in a study that would track their spending over three months. Of these participants, 25 contributed valid data for this analysis; 4 were removed from the final sample for failing to follow directions to accurately report their data. Participants who completed any part of the study were financially compensated. The participants’ mean age was 29 years (SD=7.59), with 23 participants self-identifying as White, 1 as Asian, 1 as Hispanic, and 1 as both Asian and Hispanic. The data were collected over three months and included visits to a campus laboratory as well as electronic communications. Participants were required to have on-line access to checking and credit card accounts that comprised at least 80% of their daily financial transactions and to link them to a commercially available financial tracking website. High-debt and low-debt participants were differentiated, and then within each group were separated into pairs, with the first person in the pair being randomly assigned to either a “visualization” (VIS) group or a “no-visualization” (NOVIS) group, and the second person assigned to the other group. The group assignments can be seen in Table 1. VIS participants had online access to real-time graphical representations (bar and pie charts) of their spending by category, and NOVIS participants received weekly spreadsheets with text-only lists of their transactions, as well as text-only reports of spending by category. Table 1: Participant Group Assignments. Low-Debt 5 8 High-Debt 7 5

NOVIS VIS

All participants set their own initial budgets by category, then estimated and monitored their actual spending through weekly on-line surveys and monthly in-person questionnaires. Weekly surveys included monitoring and knowledge questions (e.g., “According to your spending summary, how much have you spent in each of the following categories so far this month?”) to ensure that participants regularly processed their spending reports. To check that participants had roughly the same exposure to their spending information, weekly surveys asked participants to report how many times they had looked at any kind of spending report (i.e., transaction lists not provided by the experimenters, credit card statements, bank statements, and, for the VIS group, their real-time online reports). Because personal spending is frequently budgeted and evaluated on a monthly timeframe, more extensive monthly questionnaires were completed in the laboratory, requiring participants to estimate their monthly spending in every budgeted category, then to compare their estimates with their actual spending and respond. All values were checked against automated reports and participants' selfreports of spending, and were corrected for notable errors (e.g., the mortgage payment for both the current and the following month being deducted in one month). Some participants were interviewed at monthly visits and were videotaped using the on-line financial tracking tool to elicit information about the psychological processes that might explain spending oughts, thoughts, and acts, as well as the relationship between them.

Measures
Both the signed and unsigned gap between estimated and actual spending were calculated as outcome measures for each participant at three time points – after one month, two months, and three months of tracking. The signed estimation gap was designed to give a standardized index of how “well” participants were able to estimate their actual spending. This measure captures accuracy of estimation but also gives information about whether participants are underestimating or overestimating their spending, a valuable differentiation in the “accounting” perspective, despite the lack of budgetary anchors. The signed gap was calculated by subtracting the total amount of actual spending for each participant each month from the total amount of estimated spending for that month, then it was normalized by dividing that difference by the total amount of estimated spending. Thus, negative values indicate underestimation (i.e., spending more than one thinks), positive values indicate overestimation (i.e., spending less than one thinks), and the distance of the values from zero indicates the accuracy of the estimation. The absolute value of the signed estimation measure each month was used as a standardized index of unsigned estimation accuracy, independent of whether the individual over- or underestimated his or her actual spending. This measure is useful for the “scientific” perspective on good spending and allows for the calculation of meaningful change scores to compare accuracy from month-to-month. For example, if Participant 1 overestimated her spending in Month 1 by 20%, then underestimated in Month 2 by 20%, and Participant 2 overestimated her spending in Month 1 by 5%, then underestimated in Month 2 by 35%, a signed measure would give them both the same change score of 40 percentage points. However, the unsigned accuracy score shows that Participant 2's accuracy became worse over time, while Participant 1 remained roughly on target.

© ISLS

137

ICLS2012

Volume 1: Full Papers

Results
A 2 x 2 x 3 repeated measures, doubly multivariate analysis of variance (ANOVA), with debt status and visualization condition as the between-subjects factors and three points of measurement as the within-subjects factor was conducted with both the unsigned accuracy gap and the signed estimation gap as outcomes.

Unsigned accuracy gap
A marginally significant main effect of visualization condition on unsigned accuracy was found, with the VIS group (M=16.78, SD=12.53) being more accurate, on average, than the NOVIS group (M=27.85, SD=22.25), F(1, 21)=4.34, p=.05. No significant main effect for time was found, but tests of within-subjects contrasts reveal significant effects on unsigned accuracy between months 1 and 2 for the two-way interaction of time and visualization condition (F(1, 21)=5.32, p<.05). Calculation of the accuracy “change scores” from Month 1 to Month 2 showed that, on average, the VIS group became more accurate in their estimates (M=-4.03), whereas the NOVIS group became less accurate (M=11.35). While both the VIS and NOVIS groups had mean accuracy gaps of about 20% at Month 1, by Month 2 alone the VIS group had, on average, improved to a 15% gap, whereas the NOVIS group had actually gotten worse, with an accuracy gap of about 30%, as shown in Figure 1.

Figure 1. Mean unsigned accuracy gap by visualization condition over three months (N=25). In addition, the NOVIS group had greater variability (SD=28.85) in their individual change scores between Month 1 and Month 2 than did the VIS group (SD=13.40), F(11, 12)=4.64, p<.05, as seen in Figure 2.

Figure 2. Individual unsigned accuracy gap change scores between Month 1 and Month 2, grouped by visualization condition (N=25).

Signed estimation gap
No significant main effect of visualization condition or time on the signed estimation gap was found. However, as shown in Figure 3, there was a significant interaction between time and debt status, F(2, 42)=5.14, p=.01. Within-subjects contrasts indicate significant interactions between time and debt status between Months 1 and 2 (F(1, 21)=4.51, p<.05) and between Months 2 and 3 (F(1, 21)=10.17, p<.01).

© ISLS

138

ICLS2012

Volume 1: Full Papers

Figure 3. Mean signed estimation gap by debt group over three months (N=25). High-debt participants began the study underestimating their actual spending by about 20% (mean gap at month 1 was -21.65, SD=26.38) but by the end of 3 months of participation, their estimates were higher than their actual spending by about 6% (M=5.68, SD=33.01). On the other hand, low-debt participants began the study underestimating their spending by about 4% (M=-3.93, SD=13.9) and ended doing slightly worse, underestimating their spending by about 9% (M=-8.79, SD=10.29). A marginally significant three-way interaction between time, debt status, and visualization condition was also found on the signed estimation gap between Months 1 and 2, F(1, 21)=4.06, p=.057. A visual representation of the complete 2 x 2 x 3 analysis for the signed gap is shown in Figure 4.

Figure 4. Mean signed estimation gap by debt and visualization group over three months (N=25).

Explaining Process: Preliminary Qualitative Results
Given the significant interaction between time and debt status, as well as the marginally significant interactions with visualization condition, we examined interview data from two high-debt VIS participants to shed light on processes that might underlie their improved spending awareness. While a deep qualitative analysis is beyond the scope of the current paper, the following excerpts are nevertheless illuminating and point toward potentially fruitful paths for further investigating these processes. Participant A: Beginning: “I don't make lots of frivolous purchases; I don't eat out much and I don't buy clothes or things so when I think, where can I skimp, because I'm making less than I spend, often, so I think, what's my biggest extravagance, or the thing I could live without?” Month 1: “When I click onto each individual pie [piece of the chart], that's where I really noticed that I was eating out too much for lunch, or buying too many little treats ... 'Cause in my head I'm like, all I spend every month are the big ticket things, like, oh, I spend my car payment, I

© ISLS

139

ICLS2012

Volume 1: Full Papers

spend on this, so that's how this is helpful for me, looking at these little tiny ways I could have saved, almost a hundred dollars this month.” Month 2: “The fees are just stupid. Whereas before I was like, 'Well, I'll get to that bill...' now, I've been calling my creditors, my student loan people... So it's sort of that, just seeing it again, that big chunk, altogether, seeing it sort of in its sum total, because I can write off 5 dollars here and there, but I can't write off 200 dollars. It's made me feel like I had more, sort of, agency in what's going on with it … I don't even know that I could have told you that I spent more than a hundred dollars, or even a hundred dollars on fees and charges, back in May. So that wasn't really part of how I was thinking about my spending.” Month 3: “I got really excited about the different parts of the graph I knew I wanted to see whittled down … and be able to click through and see them get smaller and smaller. I went from hundreds and hundreds [in fees] the first month, down to $87 this month, which is good. [It’s] definitely something that I saw it, and I made some changes ... when bills are getting paid and ... moving things to different credit cards with lower fees and stuff like that ... it's been nice to know that I have a little bit of control over that. Looking at it every time, I was like, 'Oh gosh, what invisible thing is going to pop up … ' Luckily that all went down over time because I made those changes, but ... I guess awareness equals stress, in this situation. I think I need to be more honest about the way I spend with myself.” Participant A describes an arc that begins with a dispositional thought about himself – “I don’t make lots of frivolous purchases.” By the end of the study, he acknowledges that something about his experience has allowed him to overcome confirmation bias of his initial dispositional assessment of his spending. The “monitoring” effect is evident in repeated references to becoming more aware of “tiny ways” to cut spending that he had previously overlooked in favor of larger, more salient expenditures. However, this effect appears to be notably influenced by the fact that feedback was being provided graphically. The participant describes his interaction with his data using visual metaphors, naming the act of clicking on pieces of a pie graph as a key strategy for understanding his spending better, as well as expressing excitement about seeing undesirable pieces of that graph “get smaller and smaller.” Though these descriptions could be applied to numerical representations of spending, this participant appears to have used the information visualizations not only to improve his mental awareness of his spending (a “scientific” success) but to actually alter his real-life spending, with the effect of significantly reducing the amount of money he was paying toward fees (an “accounting” success). Participant B: Month 3: “My narrative about money, all along, has been that there's never enough, I'm always overspending … so everything we do financially is by definition wrong … so there's no point paying attention or trying to figure out how to do it right. Being in the study has been really interesting because it's really drawn my attention to the ways that's not true anymore … I spend a lot of time and emotional energy fussing over things … and it's like, really? Thirty dollars is not where I should be spending hours fussing. So it really made me realize that, you know, I do have enough money, and I have a lot of personal tools to manage it wisely, and I don't always do it right... but it's never this, like, crisis that it would have been in the past. Can I just relax about this and like … put my energy back into other things in my life? It's like right now I'm spending an evening a week fussing over that stuff and that's time I could spend hopefully doing really creative work, you know? Maybe I'll write a book.” Participant B also experienced improved “accounting” success, but through circumstances external to the study (a new, more lucrative job). Even in this case, though, she experienced improved “scientific” success, reducing her unsigned accuracy gap from about 25% in Month 1 to about 2% in Month 3. She repeatedly cited the graphs and charts as more useful than transaction lists to her improved understanding, which not only helped her circumvent pre-existing notions of herself as an irresponsible spender, but that helped her alleviate anxiety and learn how to act more strategically.

Discussion
This study examined whether participants who received graphical representations of their spending behavior achieved greater alignment over three months between their estimated and actual spending behavior, as

© ISLS

140

ICLS2012

Volume 1: Full Papers

compared to participants who received text-only representations. Additionally, we investigated whether any effects were moderated by participants' pre-existing consumer debt status. We found one significant effect, the interaction of time and debt status on the signed estimation gap. On average, high-debt participants moved from underestimating to overestimating their spending and improved their estimation accuracy, whereas low-debt participants underestimated their spending with relatively consistent accuracy over the entire study duration. This finding might be indicative of a “monitoring” effect, whereby the act of tracking one's finances, regardless of feedback format, itself results in a change in awareness or behavior. A possible explanation is that the intense tracking required for participation in the study introduced a significant change from the prior habits of the high-debt participants, whereas the low-debt participants entered the study already doing a reasonable level of monitoring of their spending and so experienced a less dramatic change. Though the effects involving visualization condition were not found to be statistically significant, we discuss here the marginally significant and contrast effects as indicative of interesting trends to be investigated more rigorously. The main effect of visualization condition on the unsigned accuracy gap aligns with the hypothesis that graphical representations of spending support more salient and persistent mental models of actual spending than do textual representations, such that VIS participants were able to narrow their initial thought-act gaps by the second month of participation. NOVIS participants, on the other hand, increased their thought-act gaps, on average, regardless of direction (i.e., whether over- or underestimating). In addition, the significantly larger variance in the change scores of the NOVIS group can be interpreted to mean that while graphical feedback presented a perspective on spending to VIS participants that was salient enough to overcome other, non-systematic biases around estimation of spending, textual feedback presented a relatively “weak” mental model whose effect was “washed out” by whatever other factors normally influence the estimated models of NOVIS participants, independent of feedback format. We did not have a measure of dispositional factors in this change, but at the least we can propose that if graphical feedback can produce a predictable effect on estimation accuracy, dispositional biases toward overlooking overspending violations can be circumvented through perspective manipulation, if not directly changed. This proposal helps frame the finding of a significant contrast between Months 1 and 2 on the signed estimation gap for the three-way interaction of time, debt, and visualization condition. Figure 4 shows that highdebt participants had quite different experiences between Months 1 and 2, depending on their visualization condition. High-debt VIS participants became slightly less underestimating of their spending, whereas high-debt NOVIS participants went from overestimating to underestimating their spending. For low-debt participants, VIS participants remained fairly accurate in their estimations (though slightly underestimating), but NOVIS participants moved from being fairly accurate to highly overestimating their spending. The paths of these four groups lend support to the proposal that while textual feedback on spending appears to have no systematic effect on mental models of spending, graphical feedback produces a systematic and desirable effect on improving one's awareness of one's spending, and that this effect holds across “types” of spenders. In fact, the group that showed the most “improved” awareness (i.e., both an increase in accuracy and a move from underestimating to overestimating spending) was the high-debt VIS group, a finding that has implications for policies and interventions to help consumers remediate overspending and suggests that providing high-debt consumers with salient, visual feedback on their spending habits can circumvent the mental and behavioral habits that have put them into debt in the first place. While more systematic analyses of the interview data will be necessary to make interpretive claims about the processes that may lead to improved spending from both the accounting and scientific perspectives, the interview excerpts we have included above suggest the importance of future research on feedback format as it relates to the role of developing agency in changing the alignment between oughts, thoughts, and acts, and to surfacing and challenging pre-existing notions of one’s self as a spender. Such processes may be favorable to both improving one’s estimations of spending and becoming more strategic about how one uses valuable resources like time, mental energy, and, of course, money. The reflectiveness shown by the participants also underscores the importance of observing processes that relate oughts, thoughts, and acts in context. Follow-up work could, for example, observe participants in the moment of exploring their spending data and more deeply probe the processes by which these awareness shifts happen and actual changes in day-to-day spending occur. As noted above, all results, interpretations, and recommendations in this pilot study are made with caution, due to the small sample size and therefore low statistical power, as well as methodological limitations of this pilot study. For example, while we checked the validity of actual spending measures against both selfand automated reports, we did not test the reliability of our measures of estimated spending, nor did we rigorously control for the “feedback dosage” participants received, as measured by the number of times they reported looking at their data. A baseline measure of the estimated-actual spending gap before study participation begins would also be useful in future, more rigorous studies, in order to capture any changes in the gap from pre-intervention to post-intervention, as would a more theoretically predicated cutoff between highand low-debt participants.

© ISLS

141

ICLS2012

Volume 1: Full Papers

In addition, our framing theory for the study proposes taking a “scientific” perspective on spending in which thought-act misalignments can produce investigations that result in “better” spending – that is, spending in which thoughts and acts become more aligned. However, while we can look for evidence of these investigative processes in interview data, we have no reliable measures of their existence, nor of whether changes in situational or dispositional attributions occurred. Our evidence is therefore consistent with but not confirming of our theory that the mental framing of spending presented by graphic representations can help overcome confirmation biases or situational attributions. More systematic qualitative analyses could trace the relevance of these processes and explore the associations between aspects of those processes and increased thought-act alignment. Finally, a stronger connection between this work and the traditional “accounting” view of good spending, where acts and oughts align, should be made. We did include oughts in our study by having participants set a budget at the beginning of their participation, but we did not have a reliable way of assessing whether that initial budget accurately represented participants' intended spending goals, especially as those goals evolved in response to participants' changing awareness of their actual spending patterns. Though these limitations present major constraints on the generalizability of these results and any related recommendations on mitigating the negative consequences of overspending, the pilot study has served its purpose in producing data and results that point toward the value of future research on the role of information visualization in changing the relationships between oughts, thoughts, and acts. The present study may therefore be seen as a preliminary empirical step toward bringing the body of work on visual perspective, mental representation, and behavior to bear on the question of how to nudge people toward bringing their intentions, estimations, and actions into better, more productive alignment.

References
Brunyé, T.T., Mahoney, C.R., Gardony, A.L., & Taylor, H.A. (2010). North is up(hill): Route planning heuristics in real-world environments. Memory & Cognition, 38(6), 700-712. Card, S., MacKinlay, J., & Shneiderman, B. (1999). Readings in information visualization: Using vision to think. San Francisco, CA: Morgan Kaufmann. Drentea, P. (2000). Age, debt, and anxiety. Journal of Health and Social Behavior, 41(4), 437-350. Dunbar, K. (2000). How scientists think in the real world: Implications for science education. Journal of Applied Developmental Psychology, 21(1), 49-58. Frank, M.G., & Gilovich, T. (1989). Effect of memory perspective on retrospective causal attributions. Journal of Personality and Social Psychology, 57, 399–403. Greenspan, A. (2004). “The mortgage market and consumer debt.” America’s Community Bankers Annual Convention, Washington, D.C. 19 October 2004. http://www.federalreserve.gov/boarddocs/speeches/2004/20041019/default.htm Jones, E.E., & Nisbett, R.E. (1971). The actor and the observer: Divergent perceptions of the causes of behavior. In E.E. Jones, D.E. Kanouse, H.H. Kelley, R.E. Nisbett, S. Valins, & B. Weiner (Eds.), Attribution: Perceiving the causes of behavior (pp. 79–94). New York: General Learning Press. Kahneman, D., & Tversky, A. (1984). Choices, values, and frames. American Psychologist, 39, 341-350. Kahneman, D., & Tversky, A. (2000). Choices, values, and frames. New York: Cambridge University Press. Keil, F.C. (2008). Getting to the truth: Grounding incomplete knowledge. Brooklyn Law Review, 73 (3), 10351052. Libby, L.K., Shaeffer, E.M., Eibach, R.P., & Slemmer, J.A. (2007). Picture yourself at the polls: Visual perspective in mental imagery affects self-perception and behavior. Psychological Science, 18(3), 199203. Lurie, N.H. (2004). Decision making in information-rich environments: The role of information structure. Journal of Consumer Research, 30, 473-486. Lurie, N.H., & Mason, C.H. (2007). Visual representation: Implications for decision making. Journal of Marketing, 71, 160-177. Nickerson, R., Perkins, D., & Smith, E. (1985). The teaching of thinking. Hillsdale, NJ: Lawrence Erlbaum. Redelmeier, D.A., & Tversky, A. (1992). On the framing of multiple prospects. Psychological Science, 3(3), 191-193. Wolford, G., Newman, S.E., Miller, M.B., & Wig, G.S. (2004). Searching for patterns in random sequences. Canadian Journal of Experimental Psychology, 58(4), 221-228.

Acknowledgments
This study was supported in part by a grant from the Tufts University Faculty Research Awards Committee. The authors thank Thomas C. Mann, Jason Tsai, and Vivien Lim for their assistance in conducting this research, and Remco Chang, Sara Su, Fernanda Viégas, and Martin Wattenberg for their early advice and introduction to the field of information visualization.

© ISLS

142

ICLS2012

Volume 1: Full Papers

Beyond the screen: game-based learning as a nexus of identification
Ben DeVane, University of Florida, P.O.Box 115810, 101 Norman Gym, Gainesville, Florida 32611-5810, ben@digitalworlds.ufl.edu Abstract: Drawing on data from a four-year ethnographic research study of CivWorld, a game-based learning community, this paper argues that moments of identity enactment (e.g. a conversational performance of a student’s knowledge) result in part from the nexus of longterm projects of identity formation (e.g. the project of forming an identity as a student, a teen, a brother). It uses positioning analysis to examine the relationship between moments of identity enactment in social learning communities and temporally longer trajectories of identification. It does so by examining the trajectory of identification (and by extension the arc of participation) of an eight year-old CivWorld participant named Salim as he moves from peripheral towards more full participation in the community, moving from a novice in history and game-play toward becoming an expert over a six month period.

Introduction
It is hard to overstate the extent to which research in the learning sciences, specifically in the field of games and learning, have increased its focus on identity as an important mediator of learning and cognitive processes. Identity has become an increasingly central construct in learning sciences that ties learning in context to longterm knowledge and practices (Gee, 2000; Nasir, 2002; Wortham, 2006; Enyedy et al., 2006). ). Within the learning sciences, scholarship on ‘games and learning’ has embraced this new analytic direction, emphasizing the ability of games and virtual worlds to recruit players into new identities and knowledgeable practices (Gee, 2003; Squire & Barab, 2004; Steinkuehler, 2004; Barab et al., 2005; Shaffer, 2005). Yet these still exists no clear consensus regarding which descriptive model of identity is widely accepted, or even what the major frameworks for understanding identity. In particular, there exists no clear understanding of how to connect moments of identity enactment to long-term trajectories of identity formation as learner in a given knowledge domain. In the context of a four-year ethnographic research study of CivWorld, a game-based learning community, this paper argues that moments of identity enactment (e.g. a conversational performance of a student’s knowledge) result in part from the nexus of long-term projects of identity formation (e.g. the project of forming an identity as a student, a teen, a brother). Furthermore, it contends that these moments of identity enactment in learning communities must be understood as being in dialogue with temporally longer trajectories of identification. It does so by examining the trajectory of identification (and by extension the arc of participation) of a young CivWorld participant named Salim (a pseudonym) as he moves from peripheral towards more full participation in the community, moving from a novice in history and game-play toward becoming an expert over a six month period.

Literature Review
A major family of frameworks understands identity to be driven by social interaction – the socially-contingent roles, contexts and macro-structures in which a person exists in their day to day life (see Figure 1). Influenced by linguistic anthropology (Sapir, 1921; Hymes, 1964), pragmatism (Mead, 1934) and interactionist sociology (Goffman, 1956, Blumer, 1962), sociolinguistic research in the post-war period increasingly showed that people’s social, ethnic and gender identities to greatly influence how they speak, act and think in the world (Labov, 1972, Gumperz & Hymes, 1972; Scollon & Scollon, 1981; Gee, 1989). While previous research sociological and anthropological research had previously taken “gender, ethnicity and class as given pramaters and boundaries within which we [humans] create our own social identities”, new research found “language as interactional discourse demonstrates that these parameters are not constants that can be taken for granted but are communicatively produced” (Gumperz & Cook-Gumperz, 1982, pg. 1). Language and identity, in this understanding, are emergent phenomena that mediate each other to some extent, while maintaining some continuity across contexts (Royce, 1982; Le Page & Tabouret-Keller, 1985).

© ISLS

143

ICLS2012

Volume 1: Full Papers

Figure 1 - Major frameworks of identity While applied, design-oriented games and learning research has for the most part eschewed a focus on identity, some research has emphasized the importance of identity in learning with games. This research has adopted two general ways of understanding identity relative to the design of a game or game-based learning community. First, it has sought to understand how the design of a learning environment shapes, but does not determine, the way enact different identities within game-related spaces (Gee, 2003; Steinkuehler, 2006). Gee’s (2003) theory of projective identity has been influential in this regard, but it explicitly restricts its focus to the identity work that occurs between a single player and the semiotic-activity space of a single-player role-playing game. Second, this research has examined the long-term, overarching identity transformations that long-term participants in game-based learning communities experience over the course of moving from periphery to full participation in the community (Squire et al., 2005; 2008; see also Barab et al., 2007). These two strands of research have cumulatively generated a tremendous amount of insight into a) how identity works in different genres of commercial video games, and b) how game-based learning environments transform players’ identities (and attendant knowledge and skills) into those of experts. However, there still exist significant problems yet to be resolved on the topic. For an analytic construct that remains at the heart of analyses of games and learning, however, the theory of identity in games and learning scholarship has not been explicated well in much of the research literature.. If a major affordance of games and other new media learning environments is the ability to perform significant identity work, then the successful design of game-based learning communities should theoretically depend on explicit understandings of how to ‘engineer’ both day-to-day engagement via identity resources and overarching transformations of how a person sees themself in terms of the relevant knowledge or activities around which the community is centered. For a game-based learning community to continually spark a young person’s interest and engage them in a long-term process of identity transformation, it must have a certain compelling local alignment of social identity resources at the proximate level of activity.

Research Background
This paper is part of a larger study of a four-year long research project focused on creating, sustaining and studying a game-based learning community, CivWorld, that centered on a curriculum created using researchermodified versions of the Civilization historical strategy game series. The program located at an after-school center in a medium-sized Midwestern city, which had both weekly after-school and daily summer programs, of which CivWorld was one, for young people who were mostly aged nine to fourteen. Attendees almost entirely came from working-class or working-poor families and a eighty-percent of were African-American. Participants both learned about history and geography through custom designed game modules, and learned about information technology and game design by redesigning the game themselves This paper focuses on the twin trajectories of participation and identification of young man named Salim, who was Salim was an eight year-old African American male when he started in the CivWorld program in the fall of 2005. With one older brother and one older sister, he is the youngest child in his family, and also

© ISLS

144

ICLS2012

Volume 1: Full Papers

the only child in his immigrant family that was born in the United States. Salim was introduced to the CivWorld program through his brother Malik, who at age 12 was his oldest sibling. It draws on some 70 hours of audiovideo data over the course of 6 months of game play to examine changes in Salim’s patterns of participation as he moved from novice to expert.

Methodology
But what methods do we use to understand the interaction of identity processes across and at multiple timescales – both long-term and short-term – and through multiple contexts? This study does not presume that social identities and practices are entirely incontiguous over time and context. Rather, it understands them to be continuous to a degree, while retaining a varying degree of specificity, dynamism and fluidity. As such, the analysis proposed here will not simply consist of a description of interaction (through language and practice) in these bounded spatial-temporal contexts, but rather attempt to understand it in terms of the overall arc of activity and identity work in the community. Methodologically, it will attempt to accomplish this task by employing an ethnographic perspective on positioning analysis (Gee & Green, 1998; Davies & Harre, 1990; Harre & van Langenhove, 1991). Using the framework of positioning (Davies & Harre, 1990; Harre & van Langenhove, 1991) it examines how one participant’s socially-situated identity is jointly constructed and negotiated in a game-based learning community. However, it extends positioning methods by looking at home the patterns of positioning for one individual change across time, and how modes of positioning appear in one context from others. Positioning analysis examines how a person’s attributes – their character, status and role – and their location within the structure of “rights” of social participation are continually established and negotiated in social interaction (Davies & Harre, 1990; Harre & van Langenhove, 1991). This dialogic negotiation of personhood through interaction speaks volumes about their situated social identity in an affinity space. Identity, then, is repeatedly negotiated as people are re-positioned within different modes of social interaction like conversation and narrative. The specifications of one’s position, and their social identity, are not unilateral or static. Rather a person’s positioning and identity are both fluid and dynamic across social contexts, in conversations with different people, and even within conversations as dynamics change and shift. These ways of locating or specifying a “kind of person” exist in reciprocal relationship with the ongoing dynamic events of an interaction and longer social customs. Wortham (2006) examines broader, sociohistorical models of identification in terms of how they manifest in specific events of identification. Wortham does so by studying how broader identity tropes (e.g. poor student, rowdy student) – more precisely called metapragmatic models of identity – are assigned to students over the course of one full school year. Such an event-driven analysis provides Wortham with the analytic power that Erickson located in close empirical descriptions of events while maintaining an emphasis on changes in trajectories across time. However, Wortham, perhaps wisely, mostly emphasizes the events and trajectories of identification that occur within the classroom in his analysis. How might such an event-driven analysis be extended to consider the trajectories of identification that students bring with them from outside the walls of their learning environment? This paper considers precisely this question, as it attempt to understand how learner’s everyday trajectories of identification, many of which originate outside the learning environment, inform the events of identification – the enactments of identity – that occur within the learning environment. For the purposes of this data analysis, there are two important classes of positioning that are relevant to the social identity work that takes place in the interaction below: first-order and second-order positioning. These two types of positioning refer directly to the moral and power assumptions that are embedded in language. “First-order positioning,” for instance, refers to the assumptions of status in conversation using imperatives. In conversation, commands and other imperatives signal that the addresser feels that the addressee is obliged to him or her in some way. When a person can order another to do a thing without yielding a rationale, such an utterance suggests a certain manner or positioning between the two parties. However, such “first-order” positioning breaks down when an addressee refuses a command or requests an explanation. Once one conversant starts questioning “why” they should perform the requested task, the interaction moves into the domain of second-order positioning, and the moral order has to be justified by the conversant who initiated it. Second-order positioning connotes a more egalitarian status relationship between two conversants, as one must compel the other through rhetoric rather than decree.

Results
Malik’s enthusiasm for the Civilization game series, and the CivWorld program, had gotten Salim interested in the program. Salim faced a two major obstacles related to identification upon entering the program. First, he was by far the youngest player among the participants, and he was entering a social cohort that had mostly coalesced during the summer months. The vast majority of participants in the fall program were working-class African-American young men, who seemed to enjoy exchanging competitive, but mostly good-natured, taunts and boasts. Salim, shy and deferential, was socially ostracized by the other older (ages ten to twelve) male

© ISLS

145

ICLS2012

Volume 1: Full Papers

participants in the program, which often resulted in him being subjected to severe punitive competition, rather than supportive collaboration, by these older youth as he sought to learn the game.. Second, the program organized mostly competitive and multiplayer game sessions, and Salim had to learn the basics of how to play a very hard game that can take hundreds of hours to master while almost all the other participants – his potential in-game rivals – had learned or learn basic game mechanics and strategies over the summer.

Interaction 1: Salim as novice and outsider
Salim’s brother, Malik, entered the computer lab in which the program was housed. Malik was friends with both Michael and Jason, and regularly hung out with them outside of the program. He was also friendly with Darius and Betti, who were both in his grade-level, from school. Tacitly recognized as one of the more advanced players, Malik held significant social status in the CivWorld community. While Malik exchanged hellos with Darius, Jason and Michael, he ignored Salim in line 25, who was clearly anticipating being greeted. He did not seem intentionally or purposely ignoring Salim, but rather just excited to see his friends. Regardless, Salim, who had twisted around entirely in his seat and watched Malik as he entered, presumably in an attempt to catch Malik’s eye and say hello, seemed disappointed that he had been ignored. Salim continued to negotiate with Malik and Salim in an effort to get Malik to join the multiplayer game he was playing, and specifically to determine if Malik could play as America. Malik persisted in declining to join the game, and Lawrence had to inform Salim twice that it was impossible for anyone to join the game as America. Salim quickly became the center of attention, however, when Malik asked who was winning the multiplayer game. 43/ (0:22:15.5) L: [No:::o 44/ M: I'm playing Egypt. 45/ Ma: Who's - who's WINNING?] 46/ (0:22:17.6) M: [U::uh let's see who's winning? 47/ Ma: ((walks to Salim and leans over him to look at screen))] 48/ (0:22:18.9) Ma: >Who are you?< 49/ (0:22:19.6) Ma: You're GREECE? 50/ (0:22:21.3) Ma: [DA:::AANG! see Salim? ((pats Salim on the top of his head affectionately)) 51/ M: ((turns and watches Malik and Salim)) 52/ (0:22:23.5) Ma: Good job Salim! Good job! ((continues patting Salim on the head)) 53/ (0:22:25.2) Ma: You're Greece. 54/ (0:22:26.5) Ma: ((grinning, walks to an adjacent and jostles it excitedly)) 55/ S: ((alternates between grinning and stiffling a grin)) After Malik became interested in who was winning the game, he glanced at the scoredboard on one of the computer screens. Seeing that Greece was in the lead, he asked who was playing as Greece, then looked over at Salim’s screen and saw that it was him. Despite previously having minimal interaction with his brother, and then only interactions that had been initiated by Salim, Malik quickly shifted his attention to Salim and Salim’s game. Getting very excited that Salim was at the top of the scoreboard, Malik excitedly congratulated him on his performance thus far in the game. As Salim grinned widely and Jason and Marcus turned to watch, Malik repeatedly and animatedly praised Salim for being in the lead. Malik emphasized how impressed he was with Salim by raising the volume of his voice, over-stressing words, and over-extending exclamations. While his attempts to enter into conversation earlier were not embraced by his brother, Salim’s in-game performance attracted significant notice and affection. Immediately after Malik congratulated Salim, Jason and Michael, both friends of Malik, conspired to attack Salim. Perhaps they were threatened by the attention Salim was receiving from Malik, or perhaps they were worried that Salim’s in-game successes might translate into broader recognition within the CivWorld community, but the immediacy of their response after Malik’s loud praise of Salim suggests a causal connection between the two. Over the remaining hour and fifteen minutes left in the program session that day, Michael and Jason focused their in-game energies on attacking Salim. During this time, they periodically erupted with another episode of loud taunts directed toward Salim. Salim, conversely, for the most part did not respond to Michael and Jason, but directed his attention toward fending off their in-game attacks. Eventually, with five minutes left in the game, Salim did experience a significant loss – his capital city was captured by Jason. While Salim seemed disappointed by this development, he did not seem frustrated or despondent, and actually appeared to be proud that he had defended his civilization against both Michael and Jason for such a long time.

After interaction: enforcing Salim’s positioning

© ISLS

146

ICLS2012

Volume 1: Full Papers

Malik’s arrival set off a series of changes in the discourse of the space that are important for understanding Salim’s identity relative to the CivWorld program in terms of both his relationship towards the activity of game play in the space as well as the social space around it. Two themes emerged in the continuities and discontinuities of the group’s discursive practices that took place immediately before and immediately after Malik arrived at the program. First, Salim was recognized by his brother in a way that he had not been prior to his arrival. Second, the way in which Michael and Jason positioned Salim changed significantly and swiftly as a result of this recognition. 1) Malik’s recognition of Salim - Sometime, but not immediately, after his arrival, Malik excitedly recognized Salim’s in-game achievements, which had the implicit effect of elevating, or opening up the possibility of elevation, of Salim’s social status role in the group. After ignoring Salim when he entered the program room and began greeting all his friends, Malik later changed course and began excitedly praising Salim and drawing attention to his in-game achievements. Far from the illocutionary marginal status that was implicit in Malik’s greeting ritual that ignored Salim, this sincerely excited and loud recognition of Salim’s achievements put Salim at the center of the group’s attention. In fact, Malik, who was generally recognized as one of the most central members of the group and one its best Civilization players, suddenly appeared to be able to barely restrain his excitement and affection for his brother because of his in-game achievements. He animatedly rubbed Salim’s head, patted his shoulders, and jostled neighboring chairs as he expressed his astonishment and pride that his brother was doing so well in his game play. His praise had the effect of according Salim a significant measure of recognition in this context for his nascent identities as a Civilization player specifically and as a member of the CivWorld community generally. It also had the effect of raising the possibility that Salim’s temporarily elevated positiong in these two local status roles – Salim’s reflexive and context-dependent status of membership in the group – might translate into more permanently raised social status roles – more lasting recognition and status in the group (Kiesling, 2001). 2) Enforcing Salim’s positioning. Immediately following Malik’s recognition of Salim’s in-game achievements, Michael and Jason undertook a campaign to re-enforce Salim’s status as a newer, peripheral member of the group through both in-game attacks and social positioning work outside the game. As Jason and Michael attempted to re-stabilize Salim’s peripheral status in the group, their speech acts and choices in language evidenced their underlying goal. As they gloated to Salim that they were about to attack him, Michael and Jason fell into patterns of discourse that were simultaneously similar and dissimilar from those they had used earlier. As he urged Jason to attack Salim, Michael again adopted an imperative discursive frame, again asserting his authority, confidence and certainty within the social group. Through this positioning work, Michael’s implicit message was that Salim’s identity was that of a social outsider, and a novice to game play.

Interaction 2: Salim and membership
Salim became very involved in the CivWorld program in the four and a half months between Interaction 1 and Interaction 2. Accompanied by his brother, he attended the program every single week and became progressively better and better at playing the game. During the five program weekly sessions after Interaction 1, he chose to play the Greek civilization that he had played during Interaction 1 each time. Fascinated by Greece, he began bringing to the program children’s history books that he checked out from his school library, which kept in his lap or on the desk beside him while playing. He talked regularly to Lawrence, an adult facilitator in the program, about Greece in antiquity. He became particularly interested in hoplites, the citizen-soldiers of ancient Greece, because they were represented in the game as a defensive “special unit” for Greece. He reported that at home, he and Malik took turns watching each other play the game, and Salim often talked to his brother while Malik was making game mods. Salim, in short, became a very engaged and interested participant in the CivWorld club, and took that interest into other parts of his life. The data in Interaction 1 comes from a program session that ran four and a half months after the data in Interaction 2. In this interaction, Salim was waiting with Darius, Malik and Jason to again start playing a multiplayer Civilization III game. Adult facilitators Lawrence, Karl and Brian worked to overcome technical problems to load a saved game session from the previous week that the young participants wanted to play. During a past multiplayer game, Salim, playing as Rome, had suddenly taken the lead. Salim, followed by Malik, walked toward the corner where Darius and Jason were sitting: 24/ (0:16:50.0) M: Salim! ((puts hand on Salim's shoulder)) 25/ (0:16:52.1) M: Rome is gonna die today. 26/ (0:16:54.1) S: No all you [guys are gonna die 27/ (0:16:54.8) J: Yes it is.] 28/ (0:16:55.4) M: Rome is gonna die today. 29/ (0:16:57.0) S: [No:::o? 30/ (0:16:57.3) D: It'll be all of] us put [together versus you, and you're gonna win? 31/ (0:16:58.5) M: Rome- (.) Rome- (.)] ((pats Salim to get attention))

© ISLS

147

ICLS2012

Volume 1: Full Papers

32/ (0:17:00.7) K: WHO's- WHO's winning right now? 33/ (0:17:02.4) M: Right now he's [winning 34/ (0:17:02.8) S: I'm] winning. 35/ (0:17:03.2) M: He's the one who cheats. 36/ (0:17:05.5) D: Yup! 37/ (0:17:07.1) S: NO I'm winning. Again in this interaction, Salim became the target of game-related threats and taunts from his brother’s circle of friends, but this time it was instigated by Malik himself. Salim, however, seemed to become engaged by the taunt in line 26, treating it as a social game itself. As such, he returned the taunt and declared that it was his three interlocutors that were going to lose in the game. Jason and Darius join Malik in taunting Salim, with all three predicting his doom. Darius, however, instead of declaring by fiat that Salim was going to be beaten, he justified his prediction and challenged Salim to provide a rational. In joining the chorus predicting Salim’s doom in line 30, he interrogated Salim’s self-confidence by asking him how Salim was going to defeat his three opponents at the same time. Still, Salim did not seem to mind one bit, and seemed very engaged commanding a war against Brian alongside his brother. Jason, who was presumably allied against Malik, seemed to follow a different trajectory through the game even though he was nominally part of the Malik and Salim’s alliance. While he provided material support to Malik and Salim in-game, and verbal support out-of-game, Jason never directly attacked Brian – it was a rare occurrence that Jason did not participate in a war. Salim and Malik’s war against Brian devolved into a statemate, but Salim seemed to be enjoying himself thoroughly the entire time as he and Malik sought to find a way to use the numerical superiority of their armies to overcome Brian’s defenses.

Analysis 2:
In this social warm-up to the actual multiplayer game that was characterized by braggadocio and smack talk, Salim performed a very different version of himself than he had in prior interactions that speaks to the changes in identification toward the social space surrounding CivWorl that he had undergone in the previous four months. Salim had become much more active and willing to try to define his own positioning in an interaction with older youth, even as they contested just as much as they had in the previous interaction. Throughout the interaction above, Salim contested the way in which Jason, Malik and Darius tried to negative position him in conversation. While Salim showed a growing competency in his ability to reposition himself, Malik, Darius and Jason demonstrated that he was far from mastery beginning in line 24. As they anticipated playing the game, Malik let Salim know that his civilization was going to perish and implied that Malik would be responsible at least in part. While Salim protested in earnest, Jason added his voice in agreement to Malik’s in line 27, declaring that Salim’s civilization would indeed “die.” In these first lines, both Malik and Jason adopted a declarative frame toward Salim – his civilization was inevitably going to die. Salim, in turn, responded in the declarative frame by telling them that it would not in lines 26 and 29. Darius then shifted the interaction into an explanatory frame in line 30 by asking Salim to justify why he thought he could defeat three opponents at once. Perhaps because he thought the older youth were ganging up on Salim unfairly, or perhaps just because he was curious, Karl, an adult facilitator, seized upon this explanatory frame to ask who was currently winning the game in terms of points in line 32. When both Salim and Malik answered that Salim was winning, Salim was provided with an argument to respond to Darius’ rhetorical question – he was winning already, so why could not he maintain that advantage. Salim’s ways of identifying, and negotiating how others identified him, himself socially in the space had changed significantly over the course of a four and a half month period. As the data presented in Interaction 2 demonstrates, Salim had developed some basic competencies in new ways of interacting and positioning himself as well as new forms of talk and identification through language. These new identity-related competencies were linked to larger ongoing project of identification with and identity formation in the CivWorld social space. Salim, in other words, was developing the competencies in order to be considered a full member of the highly social community of practice in the CivWorld program.

Epilogue: Salim’s trajectory of participation
Over the next six months, Salim continued his trajectory toward more full participation in the CivWorld community. Over this time, he became more and more proficient at the social and game-based activities that were valued in the Discourse of the community. Salim became much more knowledgeable and proficient in all aspects of the game, but he became recognized for his ability to achieve a high score in the game without resorting to war or conquest. Specifically, Salim became noted for his ability to achieve “cultural” and “wonder”-based victories in which he advanced his civilization through the production of culture and cultural institutions, or through the construction of “World Wonders” for which the game accords point based on the civilization’s resulting increase in stature. During this time, Salim, following his brother’s lead, began designing rudimentary game scenarios and modifications. Salim was notable for designing game scenarios that

© ISLS

148

ICLS2012

Volume 1: Full Papers

did not revolve around conflict, but rather around diplomatic and cultural acumen. Both Salim and his brother continued their practice of taking turns at home playing the game and spectating as the other played. They began taking part in ad hoc multiplayer games organized on the Internet, and took pride in boasting of their (mostly Malik’s) online victories during the weekly meetings at the CivWorld program.

Discussion
As he moved from peripheral to full participation, Salim developed intermediate identities as a competent (and then expert) Civilization game player and a recognized member of the club’s social community. These intermediate identities were in constant dialogue with both the emergent and reflexive identity positioning work (Davies & Harre, 1990) that occurred at the local level of interaction and the longitudinal identification projects (as a brother, as a masculine pre-adolescent, as a student) that Salim carried with him. Identity at the local level was negotiated between Salim’s positioning at different moments in game play and his concomitant social positioning in the group. At the same time, the social positioning work, in particular, was heavily influenced by long-term identification projects. The form that Salim’s positioning work took changed quite a bit depending upon the long-term positioning project that was foregrounded by the situated and reflexive circumstances of interaction. When Salim was foregrounding his nascent long-term identity as a masculine adolescent, the social work he performed to position himself was far more aggressive, just as the way Jason, Michael, Malik and others tried to position him was far more forceful when they were foregrounding that particular project. Similarly, when Salim placed his “identity-as-a-brother” long-term project at the forefront of his identity performance in Interactions 1 and 2, the way he positioned himself was again very distinct. When acting first as a brother, Salim oriented his actions around his perception of his brother’s needs – trying to help him join the multiplayer game in Interaction 1, and trying to prove his acumen to him in Interaction 2. In those moments, his self-positioning work became supportive and/or inquisitive, instead of the positioning continuum between quiet and aggressive that he inhabited at other times. As such, although the intermediate level was the primary level of identification processes in dialogue with the local positioning work, larger and longitudinal processes were influential as well. To be sure, he underwent powerful identity transformations that were immediately related to the relevant learning game and game-based learning community as he transformed from a user to a designer, from a novice to an expert, and from an outside to a “old-time” participant. However, these more immediate paths of identity formations were inextricably and dialogically connected to both long-term identity trajectories that were not an immediate result of the game and influential short-term identity positioning work that occurred on a local timescale. These different identity processes and trajectories intersected in the nexus of identification that was the CivWorld program. For Salim, they were very tightly bound up in a mesh of identification that resulted in the change in his CivWorld-related identities - his dispositions, capacities and evaluative stances related to his social membership and expertise in the gaming club. There was no one identity that was transferred, unfettered, to Salim by his participation in the program, but rather a number of identities that were enmeshed in Salim’s long-term trajectories of identification.

Conclusion
The central issue in question is the temporal, spatial and social scale at which learning scientists should examine processes of socialization and identification. Researchers favoring a broad, long-term scale argue that research focused only on the short-term, situated contexts fails to relate what happens in those contexts back to historical trajectories and overarching social structures – which in their view is what matters. Conversely, scholars favoring a focus on short-term and situated social contexts argue that a focus on large scales deprive individuals of agency and fail to understand the nuances and fluidity in everyday social life. As an alternative to this bifurcation of both the short-term from the long-term, and enactment from formation, a practical science of identity and learning must not, in an effort to describe the relationship between enactment and formation, fall into the same totalizing trap underlying the bifurcation in the first place. It must eschew the implicit claim that its methods and analysis can describe and represent the totality of a social person, community or ecology, and instead explicitly acknowledge that its methods provide a lens to investigate that relationship that magnifies some parts and occludes others. As Bourdieu (1977) advised, it should embrace the limits of its objective exploration as a means of making clear what it is investigating.

© ISLS

149

ICLS2012

Volume 1: Full Papers

References
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86-107. Barab, S., Dodge, T., Thomas, M., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263-305. Bauman, R., & Sherzer, J. (1975). The ethnography of speaking. Annual Review of Anthropology, 4(1), 95-119. Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge: Cambridge University Press. Côté, J., & Levine, C. (2002). Identity formation, agency, and culture: A social psychological synthesis. New York: Lawrence Erlbaum. Duranti, A. (1985). Sociocultural dimensions of discourse. Handbook of discourse analysis, 1, 193-230. Erickson, F. (1979). Mere ethnography: Some problems in its use in educational practice. Anthropology & Education Quarterly, 10(3), 182-188. Erickson, F., & Schultz, J. (1981). When Is a Context? Some Issues and Methods in the Analysis of Social Competence. Ethnography and language in educational settings, 147. Fairclough, N. (1995). Critical discourse analysis: the critical study of language. London: Longman. Gee, J.P. (2000). Identity as an Analytic Lens for Research in Education. Review of Research in Education, 25, 99-125. Gee, J.P. & Green, J. (1998). Discourse analysis, learning, and social practice: A methodological study. Review of research in education, 23(1), 119-63. Goffman, E. (1959). The presentation of self in everyday life. New York: Doubleday Anchor. Gumperz, J., & Hymes, D. (Eds.). (1972). Directions in Sociolinguistics: The Ethnography of Communication. New York & London: Holt, Rinehart and Winston. Gumperz, J., & Cook-Gumperz, J. (1982). Introduction: Language and the communication of social identity. In J. Gumperz (Ed.), Language and social identity (pp. 1-21). Cambridge: Cambridge University Press. Harré, R., & Van Langenhove, L. (1991). Varieties of positioning. Journal for the Theory of Social Behaviour, 21(4), 393-407. Hymes, D. (1964). Directions in (Ethno') Linguistic Theory. American Anthropologist, 66(3), 6-56. Labov, W. (1972). Language in the Inner City: Studies in the Black English Vernacular. Philadelphia: University of Pennsylvania Press. Mead, G. (1934). Mind, self and society. Chicago: University of Chicago Press. Nasir, N. S., & Hand, V. (2008). From the Court to the Classroom: Opportunities for Engagement, Learning, and Identity in Basketball and Classroom Mathematics. Journal of the Learning Sciences, 17(2), 143179. Scollon, R. (2001). Mediated discourse: The nexus of practice. New York: Routledge. Van Langenhove, L., & Harré, R. (1999). Introducing positioning theory. Positioning theory: Moral contexts of intentional action, 14-31. Wierzbicka, A., & Goddard, C. (1997). Discourse and culture. In T. Van Dijk (Ed.), Discourse as social interaction (pp. 231-57). London: Sage. Wodak, R. (2001). The discourse-historical approach. In R. Wodak & M. Meyer (Eds.), Methods of critical discourse analysis (pp. 63-94). London: Sage. Wortham, S. (2006). Learning identity: The joint emergence of social identification and academic learning. New York: Cambridge University Press.

© ISLS

150

ICLS2012

Volume 1: Full Papers

Working with Teenagers to Design Technology that Supports Learning about Energy in Informal Contexts
Katerina Avramides, Brock Craft, Rosemary Luckin, London Knowledge Lab, University of London, 23-29 Emerald Street, London WC1N 3QS, UK K.Avramides@ioe.ac.uk, B.Craft@ioe.ac.uk, R.Luckin@ioe.ac.uk Janet C Read, University of Central Lancashire, Preston, Lancashire, PR1 2HE, UK, JCRead@uclan.ac.uk Abstract: Energy sustainability is prevalent in political and popular rhetoric and yet energy consumption is rising. Teenagers are an important category of future energy consumers, but little is known about their conceptions. We report on research to design learning technologies that support teenagers’ learning about personal consumption of energy. In this paper we describe our analysis and methodology, which are shaped by the Ecology of Resources design framework (Luckin, 2010). Findings suggest teenagers are aware of energy issues at an abstract level, but do not apply this information in the context of their behaviour. We identify indirect energy use and the relative energy intensity of different behaviours as important areas for learning. For example, behaviours of particular relevance to teenagers are use of electronic devices, and choice of food and personal care products. These findings have implications for the design of technology to support learning about energy in informal contexts.

Introduction
Climate change and resource scarcity have brought energy sustainability to the forefront of environmental concerns. However, contrary to the need for adoption of energy sustainable lifestyles, energy consumption is rising and few people take measures to save energy (for UK trends see Whitmarsh, 2009). Indeed, few studies have found that consumers adopt energy saving behaviours over the long-term even when specifically targeted by intervention studies (see review by Abrahamse, Steg, Vlek & Rothengatter, 2005). Evidence also suggests people are unwilling to adopt behaviour changes that have a high impact on energy demand (Whitmarsh, 2009; Gatersleben, Steg & Vlek, 2002). An important factor in shaping future patterns of energy consumption is the attitudes and willingness to address the problem by younger generations. Little is known about teenagers’ conceptions of the issues around energy use and their attitudes towards the adoption of energy sustainable lifestyles (Wray-Lake, Flanagan & Osgood, 2010). However, the pattern of increase in energy consumption coupled with evidence implicating teenagers in increased energy household bills (Thøgersen & Grønhøj, 2010; BBC, 2006) suggests few are consciously acting to save energy. The reasons for this are likely to be complex. Behaviour is influenced by many factors, including knowledge, attitudes and concerns as well as contextual factors, such as habit and availability of alternatives (Stern, 2000; Whitmarsh, 2009; Gatersleben et al., 2002). Therefore, the lack of conservation behaviour may not necessarily reflect a lack of awareness or understanding. That said, there is evidence that teenagers’ concern about the environment and adoption of conservation behaviours has been declining (Wray-Lake et al., 2010). Also, studies with adults suggest there is both misunderstanding of the energy intensity of different behaviours as well as a more general lack of awareness regarding the connection between environmental issues that arise from energy use and individual consumer choices (Lorenzoni, Nicholson-Cole & Whitmarsh, 2007; Crompton & Thøgersen, 2009; Steg, 2008; Chauhan, Rama das, Haigh & Rita, 2010). This disconnect between individual actions and larger scale problems, and the lack of awareness of how one might act to make a difference suggests the need to link learning about energy and energy related environmental problems with everyday concerns, motivations, and choices. For teenagers to change their energy behaviours they must be motivated to engage with energy related problems in the context of their everyday lives and also helped to develop the skills to explore the impact of their behaviour. In this paper, we report on ongoing work to design learning technologies that support learning about energy in the informal contexts in which teenagers might make energy conscious choices. Our work contributes to research within the learning sciences community to design technologies that support scientific learning in informal contexts (Zimmerman, et al. 2010; Clegg, Gardner & Kolodner, 2011). Specifically, we discuss the application of a methodology for designing learning technology that identifies the learner’s context and draws connections between their circumstances, motivations, attitudes and knowledge.

Consumer Awareness of Energy Consumption
Energy consumption has been increasing over recent years (Abrahamse et al., 2005). It is estimated that households are responsible for between 15 and 20% of national energy consumption in western countries (Steg, 2008; Abrahamse et al., 2005). Hillman (2004, reported in Lorenzoni et al., 2007) reports individuals’ contribution to energy demand in the UK, including household and transport, at 51% of national energy use. Of

© ISLS

151

ICLS2012

Volume 1: Full Papers

this, estimates show around half is due to direct energy consumption, such as house heating, water heating, cooking and lighting, and the other half to indirect energy demand through consumption of products and services (Reinger, Vringer & Blok, 2003). Supporting individuals to adopt energy saving behaviour, therefore, can potentially have a significant impact on national energy demand. The majority of research on attitudes towards energy and adoption of energy saving behaviour has focused on adult consumers. In terms of awareness, these studies suggest people are concerned about the environment, but do not see climate change as an immediate threat and if they do save energy it is mainly for financial reasons (Lorenzoni et al., 2007). There is widespread lack of awareness of the behaviours that impact on energy demand. For example, many people report they adopt recycling to tackle climate change, when in fact recycling has a low impact (Whitmarsh, 2009). Moreover, there is evidence that people are not aware of the energy intensity of different behaviours, such as heating water (Steg, 2008), and tend to attribute greater impact to those behaviours they have adopted over those they have not (Crompton & Thøgersen, 2009). More generally, there appears to be a disconnect in understanding the source of environmental problems in individuals’ behaviour (Lorenzoni et al., 2007). Of greater concern are the findings that people are more willing to adopt changes in their behaviour that do not require much effort and change in lifestyle, such as recycling, but which have small impacts, and much less willing to adopt behaviours that have high impacts but are inconvenient, such as choosing alternatives to car travel (Lorenzoni et al., 2007; Crompton & Thøgersen, 2009; Lindenberg & Steg, 2007). It is also reported that intervention programs that target behaviour changes have been less successful in changing high impact behaviours (Chauhan et al., 2010). We know little about teenagers’ attitudes and behaviours, we can however observe that current trends in energy consumption would suggest that few teenagers are engaging in high impact energy saving behaviour. It is also likely that their understanding of the energy intensity of different behaviours mirrors that of the adult participants as reported in the aforementioned research. Some studies that have involved teenagers have presented a negative picture of their contribution to energy consumption (Thøgersen & Grønhøj, 2010; BBC, 2006), and reported evidence of decline in environmental concerns (Wray-Lake et al., 2010).

Teenagers’ Role in Energy Saving
Teenagers are not the principal decision-makers within the household but they are well placed to have an impact, both by changing their individual behaviour and by acting to influence others within the family, their social network and the public sphere (Larsson, Andersson & Osbeck, 2010). Public campaigning has the potential for greater change, because the adoption of many energy saving behaviours is hindered by contextual factors that must be addressed by organizations and government (Stern, 2000). For example, the availability of alternatives to car travel will facilitate or inhibit their uptake. It is important that teenagers’ behaviour change related to energy use is coupled with knowledge and awareness and not achieved solely through other motivations. Knowledge is an essential addition to other factors, such as the financial and contextual influences that have been found to be strong determinants of energy saving behaviour (Stern, 2000; Abrahamse & Steg, 2011). Change that is based on knowledge, awareness and concern is likely to be more robust than changes in circumstances alone (Lindernber & Steg, 2007). By contrast, if change is based on reward, the moment the reward is removed or is no longer of significance, energy use will return to higher levels. Knowledge and awareness are also important because they are essential for influencing others. In order for teenagers to become effective agents for change, they must be informed about why energy is a relevant issue and about how people can act to mitigate the environmental problems that arise from current energy consumption. However, for the learning experience to be effective, information must be tailored to personal circumstances and made relevant to individuals’ behaviour and choices. The adaptability, flexibility, and portability of learning technologies makes them well placed to create learning experiences that are personalised and that embed information about energy and energy saving into teenagers’ contexts.

Framework for Learner Centred Design
Technology has been used to good effect to support science learning, particularly in collaborative settings and when the learners’ broader circumstances are taken into account (see for example, Pea, 2002; Puntambekar and Kolodner, 2005). A methodology that identifies the multiple influences upon teenage decisions and behaviours, and that draws connections between their circumstances, motivations, attitudes and knowledge is required for the design and use of technology to support learning about energy. This methodology needs to be participatory and to engage the teenagers in the design process in order to ensure that a clear and accurate understanding of the complex influences within their personal contexts is integrated within any resultant design. The Ecology of Resources design framework (EoR, Luckin, 2010) offers a process for working with participants that models and takes account of their context. The framework is inspired by a sociocultural philosophy to understanding and supporting learning (Vygotsky, 1978, 1986) and the notion of scaffolding, which defines the process of providing support that is closely contingent on the learner’s current understanding and skills (Wood, 1976, Luckin at al, 2011; Pea, 2004). The EoR provides a method for designing learning technology and/or learning
© ISLS 152

ICLS2012

Volume 1: Full Papers

technology use that considers the important relationship between the learner in their context and the learning that arises out of their interactions with their context. The EoR conceptualizes a person’s context in terms of their interactions with the multiple resources of their world, for example, with the people, places, books, knowledge concepts and technologies they encounter; and in terms of their personal resources, such as their motivation and existing understanding. In order to support learning the design process seeks to effectively link the world resources that are most appropriate for a specific learning goal and to scaffold interactions between these resources and the learner, taking into account the learner’s personal resources. The EoR provides a method through which we first identify the world resources available to the learner and the processes and relationships that shape the learner’s access to these. We also build an understanding of the learner and what they bring to the learning experience: their personal resources. The EoR also introduces the notion of filters to describe the artefacts that constrain a learner’s access to resources, such as rules, regulations or physical boundaries. Having mapped out the learner’s context we begin an iterative participatory process of design with the aim of developing technology that facilitates access to appropriate resources at appropriate times during the learning process. The EoR has been used with learners and teachers across a range of subjects, including science and language learning to design technology rich learning activities and technology applications, such as smart phone applications (Underwood et al, 2011). The EoR design process offers a 3 Phase structure through which educators and technologists can work together to develop technologies and technology-rich learning activities that take a learner’s wider context into account. In this paper our focus is on Phase 1 (see Luckin, 2010 for framework detail):: Create an Ecology of Resources Model to identify and organize potential resources for learning. Six iterative steps support the modeling process in which some steps will require completion multiple times. Step 2 is of particular importance and can require several iterations through other steps: Step 1 – Brainstorming Potential World Resources Step 2 – Specifying the Focus of Attention Step 3 – Categorizing World Resource Elements into: Knowledge and Skills, People and Tools, Environment. Step 4 – Identify potential Resource Filters Step 5 – Identify the Learner’s Personal Resources Step 6 – Identify potential More Able Partners (MAPs). Phase 2: Identify the relationships within and between the resources produced in Phase 1. Identify the extent to which these relationships meet a learner’s needs and how they might be optimized. Phase 3: Develop the Scaffolds and Adjustments to support learning and enable the negotiation of a Zone of Proximal Adjustment (ZPA) for a learner. Phase 3 of the framework is about identifying the possible ways in which the relationships identified in Phase 2 might best be supported or scaffolded. This support might for example be offered through the manner in which technology is introduced, used or designed.

Research Questions
The study reported here is part of ongoing research to design technologies that will support learning about energy issues and the energy intensity of individual behaviour and choices. The design of these learning technologies will be based on identification of: appropriate areas of focus for learning, an understanding of how to motivate teenagers to engage in learning about energy, and an understanding of how the technology can fit in with the learners’ context in order to make the link between the learning material and the (everyday) context in which it applies. To this end we need to identify and understand the resources available to teenagers: 1. Teenagers’ personal resources: conceptions of energy issues (what are the priority areas for learning?) a. How do they understand energy use and saving in their everyday life? b. How aware are they of their energy use? c. How much knowledge do they have about the energy intensity of their behaviour and choices? 2. Teenagers’ personal resources: concerns and motivations (how do we motivate teenagers to engage?) a. Are they concerned about energy issues? b. What are their attitudes towards energy saving? c. What motivates them to save energy? 3. World resources available to teenagers (how will the technology fit into teenagers’ ecologies of resources?) a. What energy uses are most relevant to them? b. What are the circumstances in which they use energy? c. What are their sources of information about energy and energy saving?

Study Design
Participants
Our participants were a group of 14 teenagers (aged 14-17; 5 female and 9 male) who were taking an IT course at an international school in the UK. Our sample is not representative of the teenage population. However, their
© ISLS 153

ICLS2012

Volume 1: Full Papers

conceptions of energy use and motivations are very relevant. They come from families with higher incomes who are therefore likely to be greater consumers of energy (Abrahamse & Steg, 2011), both now and in the future. In subsequent studies we plan to compare our current findings with those from teenagers from state schools.

Method
The course teacher integrated our research on energy with a course project in which students documented and presented how they use energy and what they could do to save energy. The project was completed shortly after they participated in our study. This allowed us to work with the students over 3 sessions and also to gather data from a presentation of their formally assessed work. We used a combination of photo diaries and focus groups to explore the research questions. Photo diaries In our initial exploration of teenagers’ learning contexts we designed an activity that would be open-ended and not explicitly related to energy. We did not want to prescribe the kinds of things they identified as relevant to energy. Therefore, we gave participants the activity of creating a photo diary of a day in their lives. In preparation for this activity, we dedicated a separate session to the discussion of anonymity in taking photos in line with ethical constraints. We gave the opportunity to participants to practice taking photos that captured an activity but did not identify people. We did not provide reminders, for example to take a photo at set time intervals, because we were interested in what activities the participants would choose to document. Instead we asked them to document any change in activity in order to create a diary of a day in their life with the guideline to aim for 24 photos in a day. Although by necessity the consent forms did mention the energy focus of our research only two participants mentioned energy when discussing their photos and in fact the activities they documented in their diary were not limited to what activities they identified as being relevant to energy use in the focus groups. The session in which we discussed the photo diary activity and gave the participants the digital cameras was on a Friday. They were asked to complete the diary starting from Friday afternoon to Saturday in order to capture activities both in school and at the weekend. Focus groups The photos from the diaries were revisited in a subsequent session in order to translate them into a narrative. This session took place 2 days after the photo diaries were created. Participants worked in groups of 3-4 and took turns in creating their narrative by laying out the photos as a storyboard. Within the group each teenager was assigned a role of either interviewee (the person whose diary was being discussed), interviewer(s) or notetaker. The interviewer(s) were given the following questions to ask the interviewee in order to help create the storyboard: a) What were you doing when you took the photo? b) What else was going on around you? c) How long were you doing this activity for? and d) Why did this activity end? Once the narrative was complete we discussed the activities that were captured in the photos through the lens of energy consumption. In these discussions we explored teenagers’ awareness of energy-related issues, their attitudes and their behaviour. As before, each teenagers within the group was assigned roles of interviewee, interviewer and note-taker. For this discussion we provided the following questions: a) Is what you were doing related to energy use? If yes, then in what way is it related to energy? Where you conscious of how much energy you were using? Could you use less energy? Would you choose to use less energy? b) Are there times when you intentionally use less energy? If yes, when? c) Are you concerned about energy use? Formal presentation and small group discussion After the 3 sessions the students were required to complete a formally assessed project on ways in which they could reduce their energy use. They visited the university department to present their work to our colleagues. After the general post-presentation discussion we divided the participants into groups of 4-5, with one of the researchers leading the discussion in each group. In these small group discussions we used some of their photos from the photo diary activity as prompts to probe into their conceptions further. Specifically we focused on indirect use of energy, which we had identified from our analysis of the photo diaries and focus groups that they appeared to have less awareness of (photos of a bottle of water, filtered water, a zero-sugar soft drink, food (a burger), and clothes). The focus groups, presentation and small group discussions were audio recorded for later analysis.

Findings
We report on the first iteration of Phase 1 of the EoR process of design, which involves identifying the resources that make up the learners’ contexts in relation to energy use and forming an understanding of the learner and their personal resources. We have structured our findings based on the research questions. In the subsequent discussion we reflect on how these findings inform the identification of a focus for learning and motivation for teenagers to engage in learning. We analysed the data by an iterative process of identifying themes. We extracted those conceptions, motivations and context elements that recurred across participants and across data sources (photo diaries, focus group discussions, formal presentation, and small group discussion).

© ISLS

154

ICLS2012

Volume 1: Full Papers

Teenagers’ personal resources: conceptions of energy issues
How do teenagers understand energy use and saving in their everyday life? The discussions of energy mostly concerned electricity and the use of electronic devices. This focus also came through in the participants’ formal presentation, where most focused exclusively on electricity. Only two participants mentioned other uses, that is, amount of hot water used to shower and the energy required to produce plastic water bottles and paper. This focus on direct energy use and lack of awareness of indirect energy use is interesting principally because when prompted about indirect energy use explicitly participants engaged in discussion about it. In the small group discussions (which we engaged in after the formal presentation), we prompted participants with selected photos of food, drink, and clothes, with the aim to elicit their conceptions around indirect energy use (as we had identified this as an area of low awareness from the photo diaries and focus groups). During the discussion participants discussed several issues including production, transportation, refrigeration (in the case of food and drink), packaging, and waste processing of products. When prompted further they also discussed the relative energy intensity of different foods, for example meat versus vegetarian products. This apparent distinction between the energy they use directly and the energy used indirectly in the products they consume also emerged in their focus on energy use that is solely due to their behaviour. In other words, shared use of electricity, for example, in the family’s use of the dishwasher, or the use of classroom computers was only considered after prompting. For example, in the discussion of each photo diary we came across several pictures of food. The participants were prompted to consider what uses of energy are associated with, for example, their dinner, which led to a discussion about the use involved in dishwashers, microwave ovens, and refrigeration. Their ideas for saving energy were also limited to using less but not consuming different products or engaging in different activities. For example, switching off devices when not in use, or using their laptop less. It appears the concept of energy and energy saving is closely tied with not wasting more generally, and it is closely linked with not wasting other resources. For example, during the photo diary discussions many participants mentioned water (though not heating water) and compared not wasting electricity with not wasting water. When asked specifically about food in the small group discussions, the focus for many was again on waste. Similarly with clothes, the focus was on not throwing them away and handing them down to others. From the focus groups it emerged they are prompted by their parents to not waste electricity and water, and many are given their siblings’ clothes, and so it seems natural these are closely linked ideas. How aware are teenagers of their energy use? Almost all participants reported they were generally not aware of their energy use prior to engaging with us. Their teacher also confirmed the difference in their awareness over the course of the study. The only instance they discussed of being aware of energy use and acting to save energy was switching off the lights and music players when they left the room, as prompted to by their parents. We note that while they were able to discuss the different uses of energy, even indirect uses when asked explicitly about it, this information was not translated into awareness of energy use in their everyday life until we prompted them to reflect on it. How much knowledge do teenagers have about the energy intensity of their behaviour? The participants discussions of how they could save the most energy indicated little awareness of the energy intensity of different behaviours. They appeared to base their calculations of the amount of energy used by how long a device had been switched on. For example, because a fridge or phone is continuously in use it was assumed to use more energy. Discussions around indirect energy use suggested that, while they were aware of indirect energy requirements, they considered it negligible in comparison with energy used by lights or electronic devices.

Teenagers’ personal resources: concerns and motivations
Are teenagers concerned about energy issues? There was general agreement that energy related problems are important and relatively urgent issues. The problems discussed included climate change, resource scarcity, and animal habitat destruction. However, the participants did not express great concern. Some suggested problems would be resolved through improved technology, while others simply did not appear to have reflected on the issues and did not elaborate on their concern. What are teenagers’ attitudes towards energy saving? For the majority of participants the responsibility of alleviating the problems around energy use does not lie with them. Some were willing to take responsibility for individual actions and stated that each should lead by example. However, their perception was that this is not something they should be expected to take responsibility for. This finding is not surprising but it highlights an interesting issue in perception of responsibility more generally. For these teenagers the question of whether they should act appeared to be about it being unreasonable for us to expect them to act. It was not seen as a problem that affects them and, consequently, they might choose to do something about.

© ISLS

155

ICLS2012

Volume 1: Full Papers

When we discussed their willingness to change their behaviour in order to save energy, they seemed very willing to act to prevent waste but not to modify their lifestyle. They also doubted that anyone else would be willing to change. Their focus was very much on limiting amount of use rather than finding alternatives. In relation to indirect energy use, as we’ve discussed they did not consider indirect energy use to have as great an impact, so did not discuss purchasing alternative products or services. When queried about what factors would affect their purchases their focus was overwhelmingly on money, but also on the impact on people. For example, they would not buy clothes that involved child labour. What motivates teenagers to save energy? None of the participants reported that they consciously change behaviour to save energy. The only motivation was their parents’ telling them to switch off the lights and electronic devices when not in use.

Teenagers’ contexts
Our focus in addressing this question was on the kind of activities the participants engage in and the choices they make. We analysed all the data (from the photo diaries, focus sessions, formal presentation, and small group discussions) by firstly identifying all the activities, contexts of use, people and products, secondly selecting the ones that occurred with the greatest frequency, and finally grouping these into the EoR categories: Knowledge and Skills, People and Tools, Environment. People and Tools: What energy artefacts are most relevant to teenagers? We identified three groups of energy artefacts that are most relevant to our participants: electronic devices, food, and personal care products. Although there was individual variation within each category, there were some common products across participants, such as laptops, phones, music players, drinks (energy drinks), and water (bottled, filtered, tap water). Environment: What are the environments in which teenagers use energy? The main environment that appeared in the photo diaries and formal presentation was the home. School and restaurants also featured in the photo diaries but were exceptions. Knowledge and Skills: What are teenagers’ sources of information about energy and energy saving? The main source of information the participants mentioned explicitly, and that was brought up in discussions of energy, were their parents. However, this was only in relation to not wasting energy. In terms of facts about energy use and energy intensity of behaviours some mentioned TV programs. For example, one participant mentioned a program about the amount of energy that can be saved through recycling. It is worth noting when information from TV programs was mentioned it was presented with uncertainty as something that they had heard but couldn’t quite recall the details of.

Discussion
The study we report in this paper is part of ongoing research to support teenagers in learning about energy issues and the impact of their individual behaviour and choices. We engaged with teenagers using qualitative methods (photo diary, focus groups and formal project work) to identify important areas of focus for learning, to understand how to motivate teenagers to engage in learning about energy, and to explore how the technology can fit in with the learners’ context in order to make the link between the learning material and the (everyday) context in which it applies. Our work contributes to research within the ICLS community that is concerned with the design of learning technologies to support science learning in informal contexts (for example, Zimmerman et al., 2010). The design of learning technologies must tap into learners’ motivations and identify the multiple influences and resources within the learners’ context. The methodology needs to be participatory to ensure that a clear and accurate understanding of the learners’ personal contexts is integrated within any resultant design. The EoR methodology that we applied offers a process for working with participants that models and takes account of their context. We purposely used open ended tasks that did not focus on energy from the beginning. This was important in understanding the participants’ informal contexts as filtered by what they considered relevant to their lives and also acceptable for us to ‘see’. The data also confirmed the importance of an open ended method for exploring teenagers’ awareness in the context of their everyday life and differentiating awareness and knowledge in this context from the more formal knowledge they have about energy. For example, when we discussed their photo diaries indirect energy use was hardly touched upon. However, when we used some selected photos from the diaries to prompt them explicitly about indirect energy use related to food and water we generated a discussion around energy requirements for production, transportation, storage, and disposal. Had we only conducted a focus group around examples of indirect energy use we might have assumed this information would be readily available at all times. Our focus on the photo diaries also allowed us to centre the discussion around instances of energy use that were personally relevant and, therefore, probably more engaging. The combination of multiple sources of data was also important to our method. The photo diary and subsequent focus sessions allowed us to probe into the learners’ informal contexts and the nature of their conceptions within those contexts. The formal project work around participants’ energy use and ways they

© ISLS

156

ICLS2012

Volume 1: Full Papers

could save energy gave us an insight into their formalized notions of energy use, especially as it was conducted after the diaries and focus groups. When we started the study they had not yet explored their energy use so were still at their pre-study awareness levels. When they began the formal project work they had already engaged in discussion around energy use with us. Therefore, it is reasonable to assume that what was distilled in their formal work represented core ideas (for example, the focus on direct energy over indirect, and their perception of the relative energy intensity of different activities). Motivating teenagers to engage with energy issues will be the biggest challenge. Although they report concerns, it seems the problems around energy use are too abstract for them to make a connection with their lives and the people and things they care about (as appears to be the case with adults, Lorenzoni et al., 2007). It will be critical to foster understanding of the impact of energy issues on people, both those close to them and those further away. Our data suggests that they are concerned about the impact of their choices on people. For example, it is possible the concern they reported about child labour in buying clothes might extend to concern about populations who are more affected by energy related problems. In terms of the knowledge the teenagers would bring to the learning experience, our findings on teenagers’ conceptions of and attitudes to energy use and energy saving largely echo those reported from research with adults (Lorenzoni et al., 2007; Crompton & Thøgersen, 2009; Steg, 2008). Our participants were aware of energy issues at an abstract level, but this did not translate into an awareness of energy use in the context of their individual behaviour. Their conception of the relative energy intensity of different behaviours was not very accurate (for example, the assumption that the fridge uses a lot of energy because it is on all the time), and indirect energy use was largely invisible to them until prompted. It, therefore, appears that what is of critical importance for this group of teenagers is to raise their awareness of the energy intensity of their behaviour and choices. Especially the lack of awareness with regards to indirect energy use which is important given the fact that indirect energy use accounts for around half household energy use. Another reason to focus on indirect energy is that in our analysis we identified it as one of the most relevant to this group of teenagers, specifically in the consumption of food, water and personal care products. Also their focus on personal energy use suggests we need to make their contribution to shared (within the family) and public energy consumption more salient. More generally, the issues around energy might be best conveyed in conjunction with other environmental issues. The participants’ thinking about energy was not clearly differentiated from other resources, such as water. Moreover, the idea of saving was generally interpreted in terms of not wasting rather than changing behaviour to adopt less energy intensive alternatives. This is coupled with a willingness to cut back on waste but a reluctance to make changes that require sacrifice, such as not flying as frequently (which is consistent with adult research, Lorenzoni et al., 2007; Crompton & Thøgersen, 2009; Lindenberg & Steg, 2007; Chauhan et al., 2010). This again suggests we must raise awareness of the impact of behaviours, for teenagers to be in a position to judge whether cutting down on waste is enough. It also suggests that it might be productive to frame energy reduction in general in terms of wastage. In terms of the sources of information in teenagers’ lives that learning technologies could tap into, parents seemed to have the greatest influence but this was limited to not wasting energy by switching off lights and electronic devices. In this group of teenagers at least, there appears to be scope for them to educate their parents. Other information from media was occasionally mentioned, but the uncertainty with which it was presented suggests that the learning technologies we design need to help teenagers capture this information and then actively try to reflect on the implications it has for their actions.

Study Limitations and Ongoing Work
We worked with a small number of participants, which allowed us to collect detailed data but limits the generalisability of our findings. However, the fact that our results largely replicate research reports with adults, suggests that the issues we have identified as those which are an important focus are applicable beyond our sample. We are continuing our work with the same group and engaging with a second group of teenagers. In the second cycle of Phase 1 of the EoR we are presenting and discussing our analysis of the data with the participants. Through this participatory process of involving the teenagers in the design, we will ensure that the learning technologies we develop will focus on the resources that are most relevant and engaging to the learners.

References
Abrahamse, W., & Steg, L. (2011). Factors Related to Household Energy Use and Intention to Reduce It: The Role of Psychological and Socio-Demographic Variables. Human Ecology Review, 18(1), 30-40. Abrahamse, W., Steg, L., Vlek, C., & Rothengatter, T. (2005). A review of intervention studies aimed a household energy conservation. Journal of Environmental Psychology, 25(3), 273-291. BBC (2006). Teenagers are ‘standby villains’. Available online at: http://news.bbc.co.uk/1/hi/scotland/6219862.stm (Accessed 1 November 2011).

© ISLS

157

ICLS2012

Volume 1: Full Papers

Chauhan, S., Rama das, S., Haigh, M., & Rita, N. (2010). Awareness vs intentionality: exploring education for sustainable development in a British Hindu community. Sustainable Development. Available online: http://onlinelibrary.wiley.com/doi/10.1002/sd.494/abstract (accessed 7 October 2011). Clegg, T.L., Gardner, C.M. & Kolodner, J.L (2011). Technology for supporting learning in out-of-school learning environments. Proceedings of the International Conference of Computer Supported Collaborative Learning, Hong Kong. pp 248 – 255. Crompton, T., & Thøgersen, J. (2009). Simple and painless? The limitations of spillover in environmental campaigning. Journal of Consumer Policy, 32, 141-163. Gardner, G. T., & Stern, P. C. (2002). Environmental Problems and Human Behavior. Boston: Pearson. Gatersleben, B., Steg, L., & Vlek, C. (2002). Measurement and Determinants of Environmentally Significant Consumer Behavior. Environment and Behavior, 34(3), 335-362. Larsson, B., Andersson, M., & Osbeck, C. (2010). Bringing Environmentalism Home: Children’s influence on family consumption in the Nordic countries and beyond. Childhood, 17(1), 129-147. Lindenberg, S., & Steg, L. (2007). Normative, Gain and Hedonic Goal Frames Guiding Environmental Behavior. Journal of Social Issues, 63(1), 117-137. Lorenzoni, I., Nicholsoncole, S., & Whitmarsh, L. (2007). Barriers perceived to engaging with climate change among the UK public and their policy implications. Global Environmental Change, 17(3-4), 445-459. Luckin, R. (2010). Re-designing Learning Contexts: Technology-Rich, Learner-Centred Ecologies. London and New York: Routledge. Luckin, R., Looi, C.K., Puntambekar, S., Stanton Fraser, D. Tabak, I. Underwood, J. & Chen W. (2011) Contextualizing the changing face of Scaffolding Research: Are we driving pedagogical theory development or avoiding it? Proceedings of the International Conference of Computer Supported Collaborative Learning, Hong Kong. pp 1037 - 1045. Pea, R.D. (2002) Learning science through collaborative visualization over the Internet. In Proceedings of the Nobel Symposium NS 120, pp. 1-13. Pea, R.D. (2004) The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity Journal of the Learning Sciences, 13, 423-51. Puntambekar, S., & Kolodner, J. L. (2005). Distributed scaffolding: Helping students learn science by design. Journal of Research in Science Teaching, 42, (2), 185-217 Reinders, A. H. M. E., Vringer, K., & Blok, K. (2003). The direct and indirect energy requirement of households in the European Union. Energy Policy, 31(2), 139-153. Steg, L. (2008). Promoting household energy conservation. Energy Policy, 36, 4449-4453. Stern, P. C. (2000). Toward a Coherent Theory of Environmentally Significant Behavior. Journal of Social Issues, 56(3), 407-424. Thøgersen, J., & Grønhøj, A. (2010). Electricity saving in households: A social cognitive approach. Energy Policy, 38(12), 7732-7743 Underwood, J., Luckin, R. & Winters, N. Retelling Stories: Setting Learner Narratives in Resource Ecologies. In Proceedings of the International Conference of Computer Supported Collaborative Learning, Hong Long. pp 611 - 616. Vygotsky, L. S. (1978) Mind in Society: The Development of Higher Psychological Processes. Trans. Cole, M., John-Steiner, V., Scribner, S. & Souberman, E. Cambridge, MA, Harvard University Press. Vygotsky, L. S. (1986) Thought and Language. Cambridge, MA, MIT Press. Wood, D. J., Bruner, J. S. & Ross, G. (1976) The Role of Tutoring in Problem Solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. Whitmarsh, L. (2009). Behavioural responses to climate change: Asymmetry of intentions and impacts. Journal of Environmental Psychology, 29(1), 13-23. Wray-Lake, L., Flanagan, C. A., & Osgood, D. W. (2010). Examining Trends in Adolescent Environmental Attitudes, Beliefs, and Behaviors Across Three Decades. Environment and behavior, 42(1), 61-85. Zimmerman, H.T., Kanter, D.E., Ellenbogen, K., Lyons, L., Zuiker, S.J., Satwicz, T., Martell, S.T., Brown, M., Hsi, S., Smith, B., Phipps, M., Jordan, R., Weible, J., Gamrat, C., Loh, B. & Joyce, M. (2010). Technologies and tools to support informal science learning. Proceedings of the 9th International Conference of the Learning Sciences – Volume 2, Chicago. pp 260 – 266.

Acknowledgements
We thank the students and staff who have worked with us on the Taking in the Teenagers project funded by the Engineering and Physical Sciences Research Council in the UK: Grant reference number EP/I000550/1

© ISLS

158

ICLS2012

Volume 1: Full Papers

Unpacking the Use of Talk and Writing in Argument-based Inquiry: Instruction and Cognition
Ying-Chih Chen, University of Minnesota, 320 LES 1954 Buford Ave, St Paul, MN 55108, chen2719@umn.edu Soonhye Park, University of Iowa, N278 Lindquist Center, Iowa City, IA 52242, soonhye-park@uiowa.edu Brian Hand, University of Iowa, N297A Lindquist Center, Iowa City, IA 52242, brian-hand@uiowa.edu Abstract: The purpose of this study was to unpack the use of talk and writing to support students’ construction of scientific knowledge in argument-based inquiry. Grounded in interactive constructivism, this sixteen-week study utilized qualitative design and was conducted in one argument-based inquiry classroom with participation of 22 fifth-grade students. The results indicated (a) as students had more opportunities to practice, they developed ore sophisticated understanding for oral argumentation, (b) students’ ability to craft a written argument improved over time, (c) when both talk and writing were used, student knowledge construction occurred more than when only one learning tool was used, (d) students’ higher cognitive processes were facilitated more than when talk or writing were used alone, and (e) the more talk and writing were used together, the more student-centered the classroom was.

Introduction
Argumentation is a core practice of scientific inquiry in which students engage in reasoning processes of constructing knowledge claims through interpreting data as sound evidence and debating those claims (Ford, 2008; Hand, 2008). Current research indicates that learning how to engage in productive scientific argumentation to build and propose knowledge is difficult for students (Sampson & Clark, 2009). Many scholars advocate strongly the need to create environments where students can talk in order to construct explanations, models, and theories just as scientists use arguments to relate the evidence they select to their claims (e.g., McNeill, & Pimentel, 2010; Mercer, 2008). Nevertheless, Yore and Treagust (2006) argued that talking is necessary for argumentation, ―but not sufficient to do and learn science‖ (p. 296). They noted that writing also play an important role during argumentative processes. Wallace (2007) succinctly concluded about the effect of talking and writing on argumentation: ―talk is most important for distributing knowledge, while writing is important for manipulating, consolidating, and integrating knowledge‖ (p. 11). Talking and writing are therefore complementary tools for argumentative practice. However, Studies in the field of argumentation have investigated talk and writing separately and thus have tended to disconnect the relationship between the two in terms of the argumentative process due to different theoretical frameworks or methodologies adopted by the researchers. For example, some researchers emphasize the impact of engaging students in talking within groups or as a whole class (e.g. McNeill & Pimentel, 2010), while others focus on the important mechanisms for individually scaffolding the construction of students’ written arguments (e.g. Takao & Kelly, 2003). Those studies that point out the challenges, difficulties, and effects of using talk or writing in argumentation have contributed greatly to the field of science education. However, to learn better how to support students in this argumentative practice and to overcome the challenges, an examination of investigating how talk and writing combined can best support students’ construction of scientific knowledge through argumentation and what kinds of cognitive processes might be facilitated by the combination of talk and writing is needed. So far, only two quasi-experimental studies, conducted by Rivard and Straw (2000) and Rivard (2004), have directly reported interesting findings regarding the impact of both talk and writing on students’ conceptual learning. Rivard and Straw (2000) compared the effects of talk alone, writing alone, the combination with talk and writing, and control condition at a posttest and delayed posttest. They found that talk and writing together were more effective than other three groups in contributing to simple and integrated knowledge at a post-test and delayed posttest. They suggested that science teachers should endeavor to include more writing tasks in the classroom after students have had sufficient opportunities for collaborative exploratory talk while being guided by cognitive engagement in argumentative processes. Yet, their research was quantitative and still focused on the outcome of knowledge acquisition. The patterns of the combination of talk and writing that support students’ knowledge construction and cognitive processes still remain unknown. In response to those research deficits, we try to unpack the use of talk and writing in an argument-based inquiry classroom and set up four research questions to guide this qualitative study: (1) How do students develop an understanding of using talk as a learning tool when participating in an argument-based inquiry classroom during two units over sixteen weeks? (2) How do students craft their written arguments over time in an argument-based inquiry classroom? (3) In what ways do talk and writing support scientific knowledge

© ISLS

159

ICLS2012

Volume 1: Full Papers

construction in an argument-based inquiry classroom? (4) What are the characteristics of the cognitive processes associated with the use of talk and writing in the context?

Theoretical Framework
Grounded in interactive constructivism (Yore, 2001) and using language as a learning tool (Wallace, 2007), argumentation is defined in the current study as both an individual cognitive activity and a negotiated social act for knowledge construction (Hand, 2008). Both the individual and social aspects of argumentation are essential for classroom practice because they enhance students’ ability to reason and justify claims as well as to interact with their teacher and peers in the process of constructing and critiquing their ideas. When students move quickly and effectively between the two landscapes, their knowledge construction can best be facilitated. What makes their moves successful for knowledge construction is the use of language. Stated differently, knowledge construction involves an iterated process of negotiating meaning between a cognitive dynamic within individuals and social interaction in small group and whole class settings while simultaneously utilizing a variety of language forms and modes for representing those meanings. Although other forms of language are used to construct knowledge, talk and writing have been considered two critical and powerful learning tools of knowledge construction in science classrooms by a number of scholars (e.g. Yore & Treagust, 2006). In this study, writing includes not only texts, but also all representations which have the potential to improve students’ conceptual understanding such as graphs, pictures, diagrams, tables, etc. To guide data analysis and interpretation, this study identifies the structure of scientific argument as consisting of three interrelated components: question, claim, and evidence (see Chen, 2011, for descriptions of the framework and the three components).

Research Design
To answer the first and second research question, a basic qualitative approach (Merriam, 1998) was conducted in a fifth grade science classroom taught by one white male teacher in the Midwestern United States. The teacher participating in the study had 10 years of teaching experience. He had been involved in a professional development project for using an argument-based inquiry approach—the Science Writing Heuristic (SWH) approach (Hand, 2008)—for the past three years. This teacher was selected using a purposeful sampling technique because he had incorporated the SWH approach into his classroom at a high level of implementation based upon his score on the Reformed Teaching Observation Protocol (RTOP). In the summer of 2010, he was recruited as an instructor for a SWH professional development project. To understand in depth elementary school students’ perceptions, reasons, and thoughts about their actions and participation in the classroom, six students (two girls, four boys; three high achievement, three medium achievement; three initially talkative, three initially quiet) were selected for targeted interview from the 22 students in the class. To answer the third and fourth research question, a multiple-case study was conducted. Three students were selected from the six target students for careful observation while they discussed, investigated, and wrote in whole class and small group settings over sixteen weeks. The purpose of only choosing three students was to examine the trajectory of students’ use of talk and writing for knowledge construction in a great deal.

Data Collection
To triangulate the findings, data were collected through a variety of sources during two classroom units (Ecosystem & Human Body System), including daily non-participant observations, semi-structured interviews, students’ writing samples, and researcher’s field notes. Interviews about individual students’ writing samples and about classroom observations were conducted in a semi-structured fashion to elucidate the reasons they wrote, drew, and organized datasets in particular ways. The average number of interviews with each student was 11 over sixteen weeks. Students had science class every day. Seventy four classroom videotapes were collected and transcribed.

Data Analysis
Research Question 1&2: Two different analytical approaches were used: (1) the constant comparative method and (2) the enumerative approach (LeCompte & Preissle, 1993). As a result of analysis via the constant comparative method, six core components of oral argumentation related to public negotiation were identified from five rounds of public negotiations over the course of the two learning units: 1) Information Seeking, 2) Elaborating, 3) Challenging (testing, relationship between question and claim, relationship between claim and evidence, the value of a claim, and the value of evidence--sufficiency, validity, and reasoning), 4) Defending (simple, evidence-based), 5) Supporting (simple, evidencebased), and 6) Rejecting (simple, evidence-based). Five core components related to students’ written arguments were also identified: 1) Accuracy of Claim, 2) Sufficiency of Evidence, 3) Reasoning of Evidence, 4) Relationship between Claim and Question, and 5) Relationship between Claim and Evidence. To gain a better
© ISLS 160

ICLS2012

Volume 1: Full Papers

understanding of the change in the quality of students’ written arguments over time, a writing rubric was created based upon the core written argument components. Each component was rated using a five-point scale from 0= ―Lack‖ to 4= ―Proficient‖ for a possible total of 20 points for the instrument as a whole. In addition, six aspects related to students’ perspectives of argumentation emerged from interview and field notes: 1) Meaning of Negotiation, 2) Function of Talk, 3) Function of Writing, 4) Argument Structure, 5) Reason to Shift Ideas, and 6) View of Feedback. Research Question 3&4: To capture individual students’ knowledge construction over time, and the way students use talk and/or writing for the knowledge construction, a purposeful approach was designed, called indepth analysis of Knowledge Construction Trajectory (KCT). This approach consisted of four stages: a) dividing units into classes and identifying activities within a class, b) identifying events within an activity, c) synthesizing events into KCTs, and d) analyzing data related to the KCT by the constant comparative method and the enumerative approach (see Chen, 2011, for descriptions of KCT and the analysis process). To answer research question 3, the data related to each KCT were first analyzed to identify the patterns in the use of talk and writing. As a result of the analysis, each of the 21 KCTs revealed one of the five patterns in the use of talk and writing :a) talk only, b) writing only , c) use of talk and writing in sequence , d) use of both talk and writing simultaneously , and e) a combination of sequential and simultaneous use of talk and writing. Table 1 shows examples of the five patterns. Table 1: Examples of Five Patterns of KCTs Pattern 1: Talk Only (Blair, KCT 1) Pattern 2: Writing Only (Kurt, KCT 1)

Pattern 3: Use of Talk and Writing in Sequence (Blair, KCT 3)

Pattern 4: Use of Talk and Writing Simultaneously (Nolan, KCT 5)

Pattern 5: A Combination of Sequential and Simultaneous Use of Talk and Writing (Kurt, KCT 3)

Writing only Talk Only Talk and Writing

Note: Broken line boxes indicate that a student only used writing in the given event; solid line boxes indicate that a student only used talk in the event; double line boxes indicate that a student used both talk and writing in the event.

© ISLS

161

ICLS2012

Volume 1: Full Papers

To answer research question 4, all data related to the KCT were analyzed using the constant comparative method to identify cognitive processed associated with the use of talk and writing in each KCT. Codes developed through the analysis include express, report, record, describe, elaborate, organize, challenge, compare, reflect, integrate, stimulate alternative ideas, defend, multi-model representation, audience awareness, and analogize. Then those codes were categorized using Bloom’s taxonomy to determine the level of a cognitive process involved in writing, talk, or both (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956). As a result, express, report, record, and describe were categorized in low level (knowledge/comprehension); elaborate, compare, and organize were categorized in medium level (application/analysis); challenge, defend, reflect, stimulate alternative ideas, multi-model representation, audience awareness, and analogize were categorized in high level (synthesis/evaluation). (see Chen, 2011, for descriptions of the coding schemes and procedures).

Findings Finding 1: Increased Understanding of the Nature of Argumentation
First, the frequency of student utterances for each argumentation component increased from the first unit to the second unit (see Figure 1). This result suggests that student participation in argumentative discussions was increasingly encouraged and elicited the more they engaged in negotiation. When students’ oral participation increased and their argumentative practice was encouraged, they were observed to challenge others’ ideas, use evidence to back up their claims, and evaluate explanatory claims in terms of evidence to construct more complete scientific knowledge. Importantly, students gradually linked the use of argumentation components with the negotiated processes. Argumentative skills were not eliminated due to the change of the topic context, as students quickly adapted their argumentative responses to a new situation.

Figure 1. The Frequency of Argumentative Utterances Contributed by Students in Discussing Claim and Evidence in a Whole Class Setting through Sixteen Weeks Second, three main dimensions through which students challenged each others’ ideas were distinguished in the different stages of negotiation. These dimensions were (1) a focus on the test and the accuracy of the claim (beginning stage), (2) a focus on the argument structure (middle stage), and (3) a focus on the quality of the evidence (late stage) (see Figure 2). These results confirmed those from other studies on scientific argumentation that suggest that the ability to understand the structure of argument and the quality of evidence does not come naturally to most individuals, but rather is grown through practice (Hand, 2008).

Figure 2. Frequency, Proportion, and Types of Challenging Responses Contributed by Students over Five Rounds of Negotiations
© ISLS 162

ICLS2012

Volume 1: Full Papers

Third, these students developed a better understanding of how to adopt more evidence to defend, support, and reject an argument after they participated in more rounds of negotiation. The data revealed that the proportion of utterances made by students using evidence to defend, support, and reject an argument were 13%, 0%, and 0% in the first-round negotiation of the first unit. In the second round of the second unit, in contrast, the proportion of students who relied on evidence to defend, support, and reject was substantially increased to 88%, 74%, and 59%. This trend resulted in two characteristics: (1) students were more willing to accept other arguments or shift their original ideas, and (2) the discussion was more effective in terms of core concepts related to that topic.

Finding 2: Increased Quality of Written Arguments
The overall quality of students’ written arguments and their understanding of argument structure and components were improved when they were provided with more opportunities to craft their writing over time (see Figure 3). Students also incorporated multi-model representations to explain their ideas and back themselves up. This improvement in the quality of argument seemed to be due, in large part, to two factors: (1) the engagement in scientific writing embedded within argumentative practice, and (2) an awareness of the usefulness of feedback from peers.

Figure 3. The Average Score for Written Scientific Arguments on Each Component over Two Units Taking the results from Finding 1 and Finding 2 together, the quality of writing was dependent on the way students participated orally in the argumentative practice, and vice versa. In other words, the result, with a parallel shift between oral argumentative practices and the quality of written arguments, indicates that talk and writing are interdependent. Building on these results, Finding 3 will discuss how talk and writing interact with each other and how the different interactions support student knowledge construction in argument-based inquiry. Finding 4 will discuss the characteristics of the cognitive processes associated with the use of talk and writing in the context.

Finding 3: Increased Integration of Using Talk and Writing as Learning Tools
An in-depth analysis of KCTs identified twenty-one KCTs in which students built scientific knowledge using talk and/or writing in an argument-based inquiry classroom during two units. As the semester proceeded, with increasing opportunities to engage in argument-based inquiry, the students came to use different combinations of talk and writing in activities to construct their understanding of core concepts. The number of KCTs in which students used talk and writing simultaneously increased from zero in the first unit to 7 in the second unit, whereas the frequency of the use of talk and writing in sequence decreased from 4 to zero (see Table 2). The use of talk and writing was embedded in the students’ investigations and negotiations and became an integral part of inquiry as students became capable of using the two tools to represent their arguments, analyze data, and debate their ideas. During the interview, students confessed that talk associated with writing helped them to clarify, visualize, and integrate their ideas which brought them to discuss concepts at a deeper level.

Finding 4: Increased Higher Cognitive Processes by Using Talk and Writing in Sequence or Simultaneously
When students perceived and used talk and writing as separated learning tools, their cognitive processes were constrained to lower level, such as reporting, sharing, and recording their ideas. In contrast, when students used those two learning tools together, students usually became involved in higher and more complex cognitive processes, such as integrating, defending, using multi-model representations, and analogizing, etc. The function of writing extended from an individual learning tool supporting personal cognitive activities when used alone to a socially negotiated learning tool through combination with talk. Additionally, writing became both a product-based and process-based approach when students use it with talk simultaneously.

© ISLS

163

ICLS2012

Volume 1: Full Papers

Writing is not simply a matter of translating preconceived ideas into text, but it is a knowledge-constituting process (Galbraith, 1999) in which students synthesize their ideas in written forms to communicate with their peers and external their mental model by using multi-model representation (drawing, tables, figures). The quality of talk was promoted to more evidence-based when writing was used in combination. This trend of using talk and writing together in constructing scientific knowledge over time might be the result of two reasons: (1) students came to take ownership of their learning, and (2) students came to understand the meaning of negotiation. Table 2: Number of KCTs Identified in Each Unit for Each Student Combination of Talk & Writing Writing Combination of Use of Talk and Only Sequential Use Sequence and Writing Simultaneity Simultaneously 1 1 2 0 0 1 1 0 0 2 2 0 1 4 5 0 0 0 1 2 0 0 1 3 0 0 1 2 0 0 3 7 1 4 8 7

Topics

Students Calvin Betty John Calvin Betty John

Talk Only 0 1 0 1 0 0 0 0 1

Unit 1: Ecosystem Sub Total Unit 2: Human Body System Sub Total Total

Summary of Results
Table 3 summarizes the main findings addressing this study’s four research questions in terms of the following five aspects: (1) oral argumentation, (2) ability to craft written arguments, (3) use of talk and/or writing, (4) cognitive processes, and (5) meaning of negotiation. In examining these five major conceptual outcomes arising from this study, the researchers would suggest that the common critical element across these five areas is time. As students encountered more opportunities to obtain ―a grasp of the practice‖ (Ford, 2008), they were observed to develop a more sophisticated understanding of argumentation, use talk and writing as learning tools to construct and negotiate their ideas with peers, engage in more complex and higher-order cognitive processes, and take ownership of their learning in science.

Discussion and Implication for Science Education
One of the major findings of this study indicates that there is a relationship between the way these students participated in oral argumentation and their abilities in crafting written arguments. Examining Finding 1 and Finding 2 together, at the beginning of the semester students focused on challenging the test procedure and accuracy of claims, which was reflected in the improved quality of their written arguments about the accuracy of claims. In the middle of the semester, students’ spoken challenges focused on the structure of the argument, which was also reflected in their improved quality of writing about the relationship between question and claim and the relationship between claim and evidence. At the end of the semester, students moved to challenge the quality of evidence, which was reflected in their improved quality of written arguments regarding reasoning and sufficiency of evidence. These observations, when taken together, indicate that there seems to be a positive relationship between students’ participation in oral argumentation and writing as outcome measures. As such this study would suggest that talk and writing are interdependent in this context. However, it is believed that improved performance in one practice does not necessarily lead to better performance in the other; instead the students seemed to develop better awareness of audience and understanding of claim/ evidence, which guided how they engaged in both practices. This epistemic shift requires a different classroom culture and discourse environment (Cavagnetto, 2010). It is speculated that such a shift requires two conditions to occur: (1) students must be introduced to new criteria or norms for what counts as a claim and what counts as evidence in an explicit fashion in an appropriate context and time, and (2) students need to be encouraged by others to use these new criteria and norms in an appropriate context and time in which they are fruitful and make sense. Berland and Reised (2011) have indicated that students rarely revise their ideas in light of the challenges and questions posed to them. However, this study did find that students at the end of the semester were more willing to revise their ideas if their peers provided evidence to support their opinions. This suggests that a challenge component is necessary for argumentative practice, but it is not enough to generate complete and convincing scientific arguments. Evidence-based defending, supporting, and rejecting components may also

© ISLS

164

ICLS2012

Volume 1: Full Papers

help students to more carefully rethink, reflect on, and compare their ideas to those of others with evidentiary support. The significance here is that only using talk as a learning tool may not produce evidence-based discussion. When students engaged in evidence-based discussion, they usually used writing associated with talk as negotiation tools to help them explain and elaborate upon their ideas. In addition, asking or encouraging students to use multi-model representations may foster their ability to clarify and revisit their ideas instead of relying only on talk or one model (text) for reasoning concepts. Table 3: Matrix of Argumentative Practices in Five Aspects Time Line Week 1-3 Oral Argumentation --Information seeking (great proportion) --Elaborating --Challenging (focus on the procedure, process, and the accuracy of a claim) --Rejecting, supporting, and defending
(simple)

Written Arguments --Focus on the accuracy of a claim --Audience is the teacher --The reason to change ideas is because of teacher’s hint

Use of Talk and/or Writing --Talk or writing alone (writing is a product) --Use talk and writing in sequence (writing is a product, then becomes a process)

Cognitive Processes --Expressing, reporting, recording, describing, and elaborating

Meaning of Negotiation --Teacherdirected --Negotiation is to talk out ideas --Students usually talked over each other

Week 4-12

Week 13-16

--Information seeking (small proportion) --Elaborating --Challenging (focus on the structure of an argument: question, claims, and evidence) --Rejecting, supporting, and defending (more evidence-based) --Information seeking (small proportion) --Elaborating --Challenging (focus on the quality of evidence: sufficiency, validity, and reasoning) --Rejecting, supporting, and defending (evidence-based)

--Focus on the relationship between question, claims and evidence --Awareness of audience shifting from teacher to peers

--Combination of sequence and simultaneity

--Elaborating, organizing, challenging, comparing, reflecting, stimulating alternative ideas, defending, multimodel representing, and audience awareness

--Transition from teacher-directed to studentdirected --Negotiation is to explain ideas and to revise them --Students began to realize the value of critique and listening --Studentdirected --Negotiation is to explain ideas and reach a consensus; provide evidence to support claim --Effective dialogue: students realized the importance of critiquing, constructing, and listening

--Focus on the sufficiency and reasoning of evidence --Audience is both teacher and peers --Willing to change or shift ideas based on peers’ feedback

--Use talk and writing simultaneously in a whole class setting or a small group (writing is both product and process in this context)

--Elaborating, organizing, challenging, comparing, reflecting, stimulating alternative ideas, defending, multimodel representing, audience awareness, and analogizing

The analysis of this study led us to rethink the value of the combination of talk and writing to promote students’ construction of science, as it appears to help students engage in more productive arguments than when using talk or writing alone. This kind of talk associated with writing, as Galbraith noted (1999), required an oscillation between disposition (targeted topic) and linguistic (writing task) knowledge, which led to clarification of conceptual understanding and may lead to the formation of new knowledge. Talk alone simply

© ISLS

165

ICLS2012

Volume 1: Full Papers

deposits ideas that are not recorded. It is difficult to go back to check an idea generated 10 minutes ago. By contrast, the form of talk associated with writing for sharing, challenging and defending is an interaction between students and texts which is represented by someone. Talk associated with writing encourages students to ―freeze‖ their ideas on paper, which records them. Students can frequently collaboratively reflect on their ideas based upon their writing. The role of writing in this case ―serves learning uniquely because writing as process-and-product possesses a cluster of attributes that correspond uniquely to certain powerful learning strategies‖ (Emig, 1977, p. 122). Importantly, this study also found that students engaged in higher-order and more complex cognitive processes when talk and writing were used together either in sequence or simultaneously, rather than when they were used alone. It is clear that teachers have a responsibility to ―design [a] sequence of instruction that provides opportunities‖ for student growth (Duschl & Ellenbogen, 2002, p.3). More importantly, the teacher should create a discourse space in which students can construct an argument as part of an investigation with their peers, clarify their thinking by using the six core argumentation components, monitor their conceptual understandings, as well as learn and use the criteria by which these arguments will be judged or evaluated. Talk and writing can be powerful means by which to foster students to engage in higher cognitive reasoning. Especially, the simultaneous use of talk and writing can be an effective way to help students develop ownership of their learning, participate in more productive argumentative practices, and advance their conceptual growth.

References
Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956) Taxonomy of educational objectives. Handbook 1: Cognitive domain. New York: Longmans, Green. Cavagnetto, A. R. (2010). Argument to foster scientific literacy: A review of argument interventions in K–12 science contexts. Review of Educational Research, 80(3), 336-371. Chen, Y.-C. (2011). Examining the integration of talk and writing for student knowledge construction through argumentation. Unpublished doctoral dissertation, Iowa City, IA. Duschl, R.A., & Ellenbogen, K. (2002). Argumentation processes in science learning. Paper presented at the conference on Philosophical, Psychological, and Linguistic Foundations for Language and Science Literacy Research, University of Victoria, BC, Canada. Emig, J. (1977). Writing as a mode of learning. College Composition and Communication, 28(2), 122-128. Ford, M. J. (2008). Disciplinary authority and accountability in scientific practice and learning. Science Education, 92(3), 404-423. Galbraith, D. (1999). Writing as a knowledge-constituting process. In D. Galbraith, & Torrance, M. (Ed.), Knowing what to write: Conceptual processes in text production (pp. 139-159). Amsterdam: Amsterdam University Press. Hand, B. (2008). Introducing the science writing heuristic approach. In B. Hand (Ed.), Science inquiry, argument and language: A case for the science writing heuristic. Rotterdam, The Netherlands: Sense. LeCompte, M. D., & Preissle, J. (Eds.). (1993). Ethnography and qualitative design in educational research (2nd ed.). San Diego, CA: Academic Press. Mercer, N. (2008). The seeds of time: Why classroom dialogue needs a temporal analysis. The Journal of the Learning Sciences, 17(1), 33-59. Merriam, S. (1998). Qualitative research and case study applications in education. San Francisco: Jossey-Bass. McNeill, K. L., & Pimentel, D. S. (2010). Scientific discourse in three urban classrooms: The role of the teacher in engaging high school students in argumentation. Science Education, 94(2), 203-229. Rivard, L. P. (2004). Are language-based activities in science effective for all students, including low achievers? Science Education, 88(3), 420-442. Rivard, L. P., & Straw, S. B. (2000). The effect of talk and writing on learning science: An exploratory study. Science Education, 84(5), 566-593. Sampson, V., & Clark, D. (2009). The impact of collaboration on the outcomes of scientific argumentation. Science Education, 93(3), 448-484. Takao, A., & Kelly, G. (2003). Assessment of evidence in university students' scientific writing. Science and Education, 12, 341-363. Wallace, C. S. (2007). Evidence from the literature for writing as a mode of science learning. In C. S. Wallace., B. Hand & V. Prain (Ed.), Writing and learning in the science classroom (pp. 9-19). The Netherlands: Springer. Yore, L. D. (2001). What is meant by constructivist science teaching and will the science education community stay the course for meaningful reform? Electronic Journal of Science Education, 5(4), 1-7. Yore, L. D., & Treagust, D. F. (2006). Current Realities and future possibilities: Language and science literacyempowering research and informing instruction. International Journal of Science Education, 28, 291314.

© ISLS

166

ICLS2012

Volume 1: Full Papers

A Data-driven Path Model of Student Attributes, Affect, and Engagement in a Computer-based Science Inquiry Microworld
Arnon Hershkovitz, Ryan S.J.d. Baker, Janice Gobert, Adam Nakama Worcester Polytechnic Institute, Worcester, MA, USA {arnonh,rsbaker,jgobert,nakama}@wpi.edu Abstract: Work in recent years has shown that a student’s goals, attitudes, and beliefs towards learning impact their level of engagement during learning, and that engagement during learning plays a key role in learning outcomes. In this paper, we investigate the mechanisms through which a student’s goals, attitudes, and beliefs impact the student’s engaged and disengaged behaviors. In particular, we study whether affect is a mediating variable between learner attributes and engagement/disengagement during learning. To this end, we conduct exploratory path analysis on data from middle school learners who were conducting inquiry in science a microworld. We find weak but significant relationships between variables related to attitudes and beliefs and variables related to affective states and engagement. We present a path model that highlights boredom as an important mediator between a tendency towards avoiding novelty and off-task behavior during learning.

Introduction
Engagement is a critical factor in accomplishing learning tasks, in both traditional classroom and computerbased learning (Corno & Mandinach, 1983; Pintrich & Schrauben, 1992). Engagement, the extent to which students are involved in learning tasks, is positively linked with desirable learning outcomes such as deep learning and grades (Skinner, Wellborn & Connell, 1990; Wigfield et al., 2008). Individual differences in engagement are often due to differences in students’ goals, beliefs, and attitudes (e.g., Zimmerman & Bandura, 1994). Affect has been hypothesized to be a key potential mediator between these relationships (e.g., Linnenbrink, 2007). In this paper, we investigate how learners’ goals, attitudes, and beliefs influence their degree of engagement during learning, and how this process is mediated by affective states, using data from middle school students completing a computer-based science inquiry activity in the domain of physical science. Although in recent years, research has revealed relationships between student attributes, affective states, and engagement behaviors while using computer-based learning environments, past studies have typically focused only on small subsets of these constructs. We, on the other hand, examine the relationships between a larger set of variables, and we do it using an exploratory path analysis, a method for describing the direct and indirect dependencies among a set of variables (Land, 1969).

Engaged and Disengaged Behaviors
Engagement and disengagement have been characterized as “elusive variables” (Corno & Mandinach, 1993), due to the difficulty of defining these constructs precisely. Our approach towards operationalizing engagement and disengagement is to identify and measure specific behaviors that indicate engagement and/or disengagement. One such “disengaged behavior” is off-task behavior (Karweit & Slavin, 1982), where the student ceases to engage in the learning task, and instead engages in an unrelated activity. Off-task behavior has been shown to lead to poorer learning outcomes in educational software (Baker et al., 2004; Rowe et al., 2009). Another form of disengaged behavior frequently seen in educational software is gaming the system, a behavior in which students engage in systematic guessing or rapid help requests in order to obtain answers rather than thinking through the learning material (Baker et al., 2004). Gaming the system also has been shown to be negatively correlated to learning (Baker et al., 2004).

Student Attributes and Disengagement
There is increasing evidence that semi-stable student attributes such as poor goal orientation (including the goal of avoiding failure – Elliot, McGregor, & Gable, 1999, and the goal of avoiding work – Harackiewicz et al 2002), low self-efficacy (Zimmerman & Bandura, 1994), and low grit (“perseverance and passion for long-term goals” – Duckworth et al., 2007) are associated with disengagement (cf. Elliot, McGregor, & Gable, 1999; Harackiewicz et al., 2000; 2002; Baker et al., 2008). At the same time, there is increasing evidence that student behaviors of various types mediate the relationships between student attributes and learning (cf. Skinner, Wellborn & Connell, 1990; Middleton & Midgley, 1997; Elliot, McGregor, & Gable, 1999; Blackwell, Trzesniewski & Dweck, 2007). For instance, goal orientation was found to be related to cognitive engagement:

© ISLS

167

ICLS2012

Volume 1: Full Papers

students with learning goals tend to be more engaged, and tend to achieve better learning outcomes than students with performance goals (e.g., Wolters & Pintrich, 1996; Dupeyart & Mariné 2005). Another example , is grit, which predicts gaming the system on material the student does not know, in turn leading to poorer learning outcomes (Baker et al., 2008).

Affect and Disengagement
Another factor which plays an important role in student learning and engagement is affect (Craig et al., 2004), the subjective perception of emotional states (Corno, 1986) in context. The affective state of boredom has been shown to be negatively correlated with learning outcomes (Craig et al., 2004), while the affective state of “engaged concentration” (cf. Baker et al., 2010), also termed “flow”, has been shown to be positively correlated with learning outcomes (Craig et al., 2004). The relationship between confusion and learning appears to be complex, with one study suggesting that the context of confusion determines its relationship to learning (Lee et al., 2011). In addition, affect has been found to be related to students’ achievement goals and self-efficacy beliefs (e.g., Dweck & Leggett, 1988; Linnenbrink & Pintrich, 2002). Furthermore, it was shown that affect mediates the relationships between student goals and engagement, with pleasant affect having a positive mediating function and negative affect having a negative effect (Linnenbrink, 2007). To give another example, gaming the system is associated both with high levels of boredom (Baker et al., 2010) and low levels of grit (Baker et al., 2008), and boredom is positively associated with work avoidance (Duda & Nichols, 1992), a construct related to grit. This leads to the question: Does boredom potentially mediate the relationship between grit and gaming the system? And which affective states mediate the relationships between student attributes and other disengaged behaviors?

Methods
Participants
The study participants were 48 sixth-grade students (from 3 different classes), ranging in age from 10-12 years old, from a public middle school in a small city in Central Massachusetts. This school is majority White (68%), but has a substantial Hispanic minority (15%), and over 20% of residents in the city speak a language other than English at home. Overall, the city where the school is located has a median income only moderately lower than the U.S. median, but significantly lower than surrounding communities in the region, with almost 20% of individuals below the poverty level. The relationship between carelessness and student goals was studied in the context of students’ scientific inquiry within a physical science microworld. This microworld, for the domain of phase change, enabled students to explore how a particular substance, such as water, changes phases from solid to liquid to gas as it is heated. The goal of this environment and its associated activities was for students to engage in inquiry in order to develop an understanding that a substance has a melting point and a boiling point, which are both independent of the amount of sample. The phase change environment used, hosted by Science Assistments (www.scienceassistments.org; Gobert et al., 2009; Sao Pedro et al., in press) enabled students to engage in authentic inquiry using a microworld and inquiry support tools. Each task in the learning environment required students to conduct experiments to determine if a particular independent variable (container size, heat level, substance amount, or cover status) affected various outcomes (melting point, boiling point, time to melt, or time to boil).

Variables and Procedure
Within the study, we collected data relative to three categories of research variables, conceptualized as a 3-tier model. The three tiers studied are student attributes, affective states, and engaged and disengaged behaviors. We study the relationships between these tiers under the modeling assumption that the effect of student attributes on engagement is mediated by affective states. After receiving a short introduction to the activity, the students first completed the student attribute surveys online, and then engaged in the phase change learning activities for 20 minutes. Student Attributes (11 variables): The variables are: learning goal orientation, the goal of developing skill or learning; performance-approach goal orientation, the goal of demonstrating competence; performance-avoidance goal orientation, the goal of avoiding demonstrating incompetence; academic efficacy; avoiding novelty; disruptive behavior; self-presentation of low achievement, the desire to prevent peers from knowing how well the student is performing; skepticism about the relevance of school for future success; grit, the perservance and passion for long-term goals; work-avoidance; and self-efficacy for self-regulated learning.

© ISLS

168

ICLS2012

Volume 1: Full Papers

These variables are measured using surveys administered to students online. The Patterns of Adaptive Learning Scales (PALS) survey (Midgley et al., 1997) was used to assess semi-stable student attributes. In specific, two scales were utilized: PALS 1, Personal Achievement Goal Orientation (14 items), and PALS 4, Academic-related Perceptions, Beliefs, and Strategies (28 items), with all items in these scales rated on a 5-point Likert scale. Also, surveys were used to measure grit on a 5-point Likert scale (Duckworth et al., 2007), workavoidance on a 7-point Likert scale (Harackiewicz et al., 2000), and self-efficacy for self-regulated learning on a 7-point Likert scale (Bandura, 1990). Engaged and Disengaged Behaviors (4 variables): The following engaged and disengaged behaviors were coded: on task, on-task conversation, off-task, and gaming the system. These variables were assessed using quantitative field observations during student use of the learning environment (cf. Baker et al. 2004, 2010). For each student and for each observed variable, the percentage of the instances in which each behavior was observed was computed. Observed were conducted by trained observers (in specific, the second and fourth authors), who had previously achieved inter-rater reliability over 0.6 for the coding protocol and both of the coding schemes used in this study, with a different population. This level of inter-rater reliability, while imperfect, is substantially higher than that seen for expert coding of academic emotions and behaviors of this nature in video data (cf. Graesser et al., 2006). Each observation lasted twenty seconds, and was conducted using peripheral vision. That is, the observers stood diagonally behind or in front of the student being observed and avoided looking at the student directly, in order to make it less clear when an observation was occurring (cf. Baker et al. 2004, 2010). If two distinct behaviors were seen during an observation, only the first behavior observed was coded; similarly, if two distinct affective states were seen during an observation, only the first was coded. Only the student currently being observed was coded during each observation period, to avoid bias towards more salient behaviors and affective states. Observations were recorded using an app for the Google Android platform, which enforced the coding procedure and coding schemes. For each student and for each variable, the percentage of the instances in which this variable was observed was computed. Affective States (4 variables): The following affective states were measured: confusion, engaged concentration, boredom, and frustration. These observations were conducted concurrently to the observations of engaged and disengaged behaviors, using the same procedure and coding schemes as in Baker et al. (2010). For each student and for each observed variable, the percentage of the instances in which each affective state was observed were computed. During analysis, the affective state of engaged concentration was treated as representing a different state when students were off-task as opposed to on-task, as being deeply engaged in offtask behavior should not be considered evidence for engagement with the learning material.

Analysis Approach: Path Models
Path models are an extension of multiple regression, in which the relationships between multiple variables are evaluated simultaneously. Path models are a graphical representation of a Structural Equation Model (SEM) describing linear relationships between the variables under consideration (Land, 1969). In recent years, path models have been used to shed light on the relationships between multiple learning variables. In most cases, path models have served as a means to test existing theoretical models of the relationships between variables of different types, e.g., goals, metacognitive, study strategies, and academic achievement (Dupeyart & Marine, 2005; Blackwell, Trzesniewski, & Dweck, 2007). However, in other cases, path models have been built bottomup, to find the best possible model for a given dataset without assuming the model a-priori (cf. Fouad & Smith, 1996; Iverach & Fisher, 2008). In this paper, given the lack of a prior theoretical model linking all of the constructs under investigation, we adopt a bottom-up method, essentially using path models as an educational data mining method (cf. Baker & Yacef, 2009). To this end, we follow a heuristic inspired by Cohen et al. (1993) for building path models in this fashion: potential links are chosen to be added to the path model according to the degree of improvement in the model’s fit to the data. We impose only one type of structure: a 3-tier model, consisting of the following sets of variables: student attributes (tier one), affective states (tier two), and behaviors indicating engagement or disengagement (tier three). Attributes are assumed to influence later affective states, and both attributes and affective states are assumed to influence behavior. The model creation process was initiated using a set of models, each of which consisted of only two variables and one link, based on the set of correlations which were statistically significant under post-hoc correction (listed below). Then, we assessed the impact on model significance from adding additional variables and links to the model. Models were built and assessed using the AMOS 19 software for path modeling (Arbuckle, 2010).

Results
First, in order to understand the frequencies of the research variables, we computed means and standard deviations for each of the research variables. A full list of proportions, means and standard deviations of the variables used is given in Table 1. The average proportion of off-task behavior (13%) is within the range previously observed in studies of students using intelligent tutor software in the U.S. (cf. Baker et al., 2004;

© ISLS

169

ICLS2012

Volume 1: Full Papers

Baker, 2009; Cetintas et al., 2010), and is similar or slightly lower than proportions of off-task behavior seen in traditional classrooms (cf. Karweit & Slavin, 1982). The proportion of gaming the system (1%) is substantially lower than seen in previous studies of intelligent tutoring systems and educational games (Baker et al., 2004, 2010; Walonoski & Heffernan, 2006). Students were observed to be in a state of engaged concentration 59% of the time, a proportion consistent with past research on students using intelligent tutoring systems (Craig et al., 2004; Baker et al., 2010), though higher than seen in some studies (Sabourin et al., 2011). Boredom was substantially more common than in several of these past studies, occurring 20% of the time, while confusion was relatively less common, occurring 8% of the time. Table 1: Mean and standard deviation for the research variables. Student Attributes (all but the last two are on a 5-Likert scale; the last two are on a 7-Likert scale) Variable Mean SD learning goal orientation (N=46) 4.20 0.59 performance-approach goal 3.00 1.19 orientation (N=46) performance-avoid goal 3.09 1.24 orientation (N=46) academic efficacy (N=46) 3.57 0.90 avoiding novelty (N=46) 2.90 1.10 disruptive behavior (N=46) 2.77 1.21 self-presentation of low 2.14 1.00 achievement (N=45) skepticism about the relevance of 2.65 1.12 school for future success (N=45) grit (N=43) 2.86 0.65 work-avoidance (7-Likert scale) 4.06 1.19 (N=47) self-efficacy for self-regulated 4.52 0.88 learning (7-Likert scale) (N=47) Affective States (proportion of observed instances to total number of observations), N=48 Variable frustration confusion engaged concentration Mean 0.06 0.08 0.59 SD 0.06 0.07 0.18

boredom 0.20 0.13 Engagement (proportion of observed instances to total number of observations), N=48 Variable Mean SD on task on-task conversation off-task gaming the system 0.75 0.11 0.13 0.01 0.16 0.10 0.11 0.04

Relationships between the Research Variables
In order to explore the relationships between the research variables, we examined the inter-correlations between the variables of the different tiers. Results are summarized in Table 2. As we ran an overall of 104 bivariate correlations tests, we corrected for multiple comparisons using the post-hoc False Discovery Rate (FDR) method, which produces a q-value that can be interpreted the same way as a p-value. Between student attributes and affective states, only one pair of variables (out of 44) was found to be significantly correlated when controlling for multiple comparisons: disruptive behavior and confusion (r=0.45, q<0.05). This relationship might be explained as the following: as disruptive behavior increases, less learning opportunities are available for the student, hence she or he is more likely to be confused when required to demonstrate their skills. It is surprising that none of the three main goal orientation scales (learning, performance-approach, and performance-avoid) were found to be significantly correlated with any of the affective states, considering previous studies (e.g., Linnenbrink & Pintrich, 2002). Between student attributes and engagement tiers, no significant correlations were found. Again, no correlations were found between the three main goal orientation scales with any of the engagement variables. The strongest correlation (in terms of q-value) was between performance-approach goal orientation and on task conversation (r=0.20, q=0.73). This is consistent with previous studies which have found no relationships between goal orientation and disengaged behaviors within students using educational software (e.g. Baker, 2007; Baker et al., 2008, Rowe et al., 2009). Finally, between the affective states and engaged/disengaged behaviors, four significant correlations were found. Engaged concentration was found to be significantly correlated with on-task (r=0.65, q<0.01) and off-task (-0.54, q<0.01); also, boredom was found to be significantly correlated with on-task (r=0.57, q<0.01) and off-task (r=0.65, q<0.01). One complication in analyzing these relationships is that the affective states and engagement variables are not normally distributed. Testing for multivariate normality, we used Small’s test for skewness (Small, 1980) and Anscome & Glynn’s (1983) variant of Small’s test for kurtosis, and found significant evidence for skew,
© ISLS 170

ICLS2012

Volume 1: Full Papers

Q1=71.28 (df=19), p<0.0001, and for multivariate kurtosis, Q2=56.74 (df=19), at p<0.0001. Hence, nonparametric bootstrapping and Maximum Likelihood estimation were used to generate model parameters and estimates (cf. Kline, 2010), with removal of cases with missing values during bootstrapping. Table 2: Inter-correlations between the research variables of different tiers (significant values after post-hoc corrections bolded). Variables frust. Affective States conf. concent. boredom on task
-0.02 -0.01 0.07 0.15 0.31 -0.16 0.11 0.12 0.24 -0.08 -0.06 -0.19 -0.02 0.65** -0.57**

Engagement ontask conv.
0.00 0.20 0.09 0.00 0.13 0.23 0.00 0.06 0.00 0.24 0.04 0.24 0.14 -0.34 0.18

offtask
0.03 -0.12 -0.15 -0.11 -0.34 0.03 -0.15 -0.23 -0.28 -0.14 -0.01 -0.04 -0.09 -0.54** 0.65**

gaming

learning goal (N=46) perf-approach.goal (N=46) perf.-avoid goal (N=46) academic. efficacy (N=46) avoiding. novelty (N=46) disruptive behavior (N=46) self-presnt. of low achv. (N=46) skepticism (N=45) grit (N=43) work-avoidance (N=47) self-efficacy (N=47) frustration (N=46) confusion (N=46) concentration (N=46) boredom (N=46) * q<0.05, ** q<0.01 Student Attributes Affective States

-0.16 -0.07 -0.16 -0.38 0.01 0.00 -0.13 -0.09 0.17 0.05 -0.16

0.04 0.26 -0.01 -0.15 0.13 0.45* 0.22 0.34 -0.03 0.11 -0.12

0.06 0.12 0.11 0.32 0.21 -0.12 0.07 0.07 0.07 -0.05 0.19

0.09 -0.25 -0.15 -0.17 -0.35 -0.10 -0.19 -0.14 -0.04 -0.02 -0.14

0.00 -0.13 -0.09 -0.24 0.10 -0.05 -0.03 0.01 -0.14 -0.01 0.00 0.23 -0.01 -0.09 -0.11

Path Model
As was previously mentioned, we applied a bottom-up construction heuristic for the path model. Figure 1 presents the best fitting model found, involving only three variables, one of each tier: avoiding novelty, boredom, and off-task. The standardized regression coefficient between avoiding novelty and boredom is -0.28, which is marginally statistically significant, with p=0.06; the coefficient between boredom and off-task is 0.48, which is statistically significant at p<0.01. This model was validated using chi-square, obtaining a value of 1.24 (df=1), p=0.27 (This test compares the best model to a full (saturated) model; a non-significant chi-square is therefore desirable). For this test, a ratio of chi-square to df which is equal to or smaller than 3 is considered acceptable (Schreiber et al., 2006). Three measure of goodness of fit, used for small samples that are not multivariate normal, CFI (comparative fit index), GFI (goodness of fit index), and IFI (incremental fit index), got values of 0.98 each; the agreed cutoff for these measures is 0.95.

© ISLS

171

ICLS2012

Volume 1: Full Papers

Figure 1. The final path model: Rectangles represent observed variables, circles – latent variables This model suggests that the negative relationship between avoiding novelty and off-task behavior is mediated by boredom. In other words, students who avoid novelty are less likely to be bored during learning, and students who are less bored are less likely to become off-task. We can also understand this model as saying that students who desire novelty are more likely to be bored and therefore are more likely to be off-task. The relationship between avoiding novelty and off-task behavior is much more understandable when the mediating factor of boredom is taken into account.

Discussion & Conclusions
Within this paper we have analyzed the relationships between three categories of variables related to students’ characteristics and behavior: a) student attributes, including goal orientation, beliefs regarding learning, and attitudes towards learning; b) affective states during learning; and c) engagement and disengagement during learning. Overall, there were only a few statistically significant correlations in between the different categories. Interestingly, this study failed to replicate previous results that establish relationships between goal orientation and engagement (e.g., Dweck & Leggett, 1988; Elliot & Harackiewicz, 1996), as well as between goal orientation and affective states (e.g., Pekrun, Elliot, & Maier, 2006). A possible explanation is that the domain plays a major role in the lack of these relationships, as inquiry-based learning environments foster better engagement and may lead to a greater probability of mastery goal orientation during learning (cf. Hmelo-Silver, Duncan & Chinn, 2007), hence changing the nature of interactions between variables. Indeed, previously there were no associations found between carelessness (a different measure of disengagement) and goal orientation (Hershkovitz et al., 2011), and no relationships between goal orientation and achievement (Shimoda, White & Frederiksen, 2002) in science inquiry tasks. In general, goal orientation has been a poor predictor of engagement in educational software (Baker, 2007; Baker et al., 2008; Rowe et al., 2009), despite its general importance in other contexts (Wolters & Pintrich, 1996; Dupeyart & Mariné 2005; Blackwell et al., 2007). , We applied a data-driven approach for constructing a path model to describe the relationships between the research variables. Avoiding novelty was found to be related to boredom, which is in turn related to off-task behavior. The model suggests that it will be valuable to proactively scaffold students who tend to avoid novelty in ways that promote interest. It is important to emphasize that the path model by itself does not imply causality, as is generally the case for post-hoc correlational analyses. In general, it will be valuable to replicate this analysis using different datasets from the same domain, and datasets from additional domains, in order to determine how general the patterns are. One other surprising finding was the relative rarity of gaming the system compared to prior research in intelligent tutoring systems and educational games. One explanation is that gaming the system is more difficult in the current learning environment, given the relatively lower degree of feedback than seen in these types of environments. Interestingly, carelessness – another behavior indicating disengagement – has also been found to be less common in Science Assistments than in mathematics tutoring systems (Hershkovitz et al., 2011). In summary, the research presented here suggests that in the context of computer-based science inquiry, there is a substantial relationship between avoiding novelty (a student attribute) and off-task behavior (a disengaged behavior), mediated by boredom (an affective state). In general, it will be valuable to study further how and why boredom mediates this relationship. One method for doing so will be to apply temporal analysis, exploring how engagement and affect change during the learning task.

References
Anscombe, F.J. & Glynn, W.J. (1983). Distribution of kurtosis statistic b2 for normal statistics. Biometrika, 70, 227–234. Arbuckle, J.L. (2010). IBM SPSS Amos 19 User’s Guide. Chicago, IL: IBM. Baker, R.S.J.d. (2007). Modeling and understanding students' off-task behavior in Intelligent Tutoring Systems. Proceedings of ACM CHI 2007: Computer-Human Interaction, 1059-1068. Baker, R.S.J.d. (2009) Differences between intelligent tutor lessons, and the choice to go off-task. Proceedings of the 2nd International Conference on Educational Data Mining, 11-20.

© ISLS

172

ICLS2012

Volume 1: Full Papers

Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-task behavior in the cognitive tutor classroom: When students "game the system". Proceedings of ACM CHI 2004: Computer-Human Interaction, 383-390. Baker, R.S.J.d., D'Mello, S.K., Rodrigo, M.M.T., Graesser, A.C. (2010) Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners' Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments. International Journal of Human-Computer Studies, 68(4), 223-241. Baker, R.S.J.d., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185-224. Baker, R.S.J.d., Yacef, K. (2009) The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3-17. Bandura, A. (1990). Multidimensional Scales of Perceived Self-efficacy. Stanford, CA: Stanford University. Blackwell, L.S., Trzesniewski, K.H., & Dweck, C.S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246-263. Cetintas, S., Si, L., Xin, Y.P., & Hord, C. (2010). Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Transactions on Learning Technologies, 3(3), 228-236. Cohen, P.R., Ballesteros, L., Carlson, A., & Amant, R.St. (1993). Automating path analysis for building causal models from data: First results and open problems. In Proceedings of the Tenth International Conference on Machine Learning, 57-64. Corno, L. (1986). The metacognition control components of self-regulated learning. Contemporary Educational Psychology, 11, 333-346. Corno, L. & Mandinach, E.B. (1983). The role of cognitive engagement in classroom learning and motivation. Educational Psychologist, 18(2), 88-108. Craig, S.D., Graesser, A.C., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241-250. Duckworth, A.L., Peterson, C., Matthews, M.D., & Kelly, D.R. (2007). Grit: Perseverance and passion for longterm goals. Journal of Personality and social Psychology, 92(6), 1087-1101. Duda, J., & Nicholls, J. (1992). Dimensions of achievement motivation in school work and sport. Journal of Educational Psychology, 84, 290-299. Dupeyrat, C. and Mariné C. (2005). Implicit theories of intelligence, goal orientation, cognitive engagement, , and achievement: A test of Dweck’s model with returning to school adults. Contemporary Educational Psychology, 30(1), 43-59. Dweck, C.S. and Leggett, E.L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256–273. Elliot, A.J., & Harackiewicz, J.M. (1996). Approach and avoidance achievement goals and intrinsic motivation: A mediational analysis. Journal of Personality and Social Psychology, 70(3), 461−475 Elliot, A.J., McGregor, H.A., & Gable, S.L. (1999). Achievement goals, study strategies, and exam performance: A mediational analysis. Journal of Educational Psychology, 76, 628–644. Fouad, N.A. & Smith, P.L. (1996). A test of a social cognitive model for middle school students: Math and science. Journal of Counseling Psychology, 43(3), 338-346. Gobert, J., Heffernan, N., Koedinger, K.., & Beck, J. (2009). ASSISTments Meets Science Learning (AMSL). Proposal (R305A090170) funded by the U.S. Department of Education. Graesser, A.C., McDaniel, B., Chipman, P., Witherspoon, A., D’Mello, S., & Gholson, B. (2006). Detection of emotions during learning with AutoTutor. Proceedings of the 28th Annual Meeting of the Cognitive Science Society, 285-290. Harackiewicz, J.M., Barron, K.E., Tauer, J.M., Carter, S.M., & Elliott, A.J. (2000). Short-term and long-term consequences of achievement goals: Predicting interest and performance over time. Journal of Educational Psychology, 92(2), 316-330. Harackiewicz, J.M., Barron, K.E., Tauer, J.M., & Elliot, A.J. (2002). Predicting success in college: A longitudinal study of achievement goals and ability measures as predictors of Interest and performance from freshman year through graduation. Journal of Educational Psychology, 94, 562–575. Hershkovitz, A., Wixon, M., Baker, R.S.J.d., Gobert, J., Sao Pedro, M. (2011) Carelessness and goal orientation in a science microworld. Poster paper. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 462-465. Hmelo-Silver, C.E., Duncan, R.G., & Chinn, C.A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42(2), 99-107.

© ISLS

173

ICLS2012

Volume 1: Full Papers

Iverach, M.R. & Fisher, D.L. (2008). An interdisciplinary investigation of high schools’ approaches to learning sciences: The relations amongst achievement goals, constructivist pedagogical dimensions, motivational beliefs and self-regulated learning. Proceedings of the Fifth International Conference on Science, Mathematics and Technology Education, 233-255. Karweit, N. & Slavin, R.E. (1982). Time-on-task: Issues of timing, sampling and definition. Journal of Educational Psychology, 74, 844-851. Kline, R.B. (2010). Principles and Practice of Structural Equation Modeling (3 rd edition). New York, NY: The Guilford Press. Land, K.C., (1969). Principles of path analysis. Sociological Methodology, 1, 3-37. Lee, D.M.C., Rodrigo, Ma.M.T., Baker, R.S.J.d., Sugay, J.O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. In Proceedings of the 4th Bi-annual International Conference on Affective Computing and Intelligent Interaction. Linnenbrink, E.A. (2007). The role of affect in student learning: A multi-dimensional approach to considering the interaction of affect, motivation, and engagement. In P.A. & P. Reinhard (eds.), Emotion in Education (pp. 107-124). San Diego, CA: Elsevier Academic Press. Linnenbrink, E.A. & Pintrich, P.R. (2002). Achievement goals theory and affect: An asymmetrical bidirectional model. Educational Psychologist, 27(2), 69-78. Middleton, M., & Midgley, C. (1997). Avoiding the demonstration of lack of ability: An underexplored aspect of goal theory. Journal of Educational Psychology, 89, 710–718 Midgley, C., Maehr, M., Hicks, L., Roeser, R., Urdan, T., Anderman, E., & Kaplan, A., Arun-Kumar, R., Middleton, M. (1997). Patterns of Adaptive Learning Survey (PALS). Ann Arbor, MI: University of Michigan. Pekrun, R., Elliot, A.J., & Maier, M.A. (2006). Acheivement goals and discrete achievement emotions: A theoretical model and prospective tests. Journal of Educational Psychology, 98(3), 583-597. Pintrich, P.R. & Schrauben, B. (1992). Students’ motivational beliefs and their cognitive engagement in classroom academic tasks. In D.H. Schunk & Meece, J.L. (eds.), Student Perceptions in the Classroom (pp. 149-183). Hillsdale, NJ: Lawrence Erlbaum Associates. Rowe, J.P., McQuiggan, S.W., Robison, J.L., & Lester, J.C. (2009). Off-task behavior in narrative-centered learning environments. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Education, 99-106. Sabourin, J., Mott, B., Lester, J. (2011) Modeling Learner Affect with Theoretically Grounded Dynamic Bayesian Network. Proceedings of 4th International Conference on Affective Computing and Intelligent Interaction. Sao Pedro, M., Baker, R., Gobert, J., Montalvo, O., & Nakama, A. (in press). Using Machine-Learned Detectors of Systematic Inquiry Behavior to Predict Gains in Inquiry Skills. To appear in User Modeling and UserAdapted Interaction. Schreiber, J.B., Stage, F.K., King, J., Nora, A., & Barlow, E.A. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323-337. Shimoda, T.A., White, B.Y., & Frederiksen, J.R. (2002). Student goal orientation in learning inquiry skills with modifiable software advisors. Science Education, 86(2), 244-263. Skinner, E.A., Wellborn, J.G., & Connell, J.P. (1990). What it takes to do well in school and whether I've got it: A process model of perceived control and children's engagement and achievement in school. Journal of Educational Psychology, 82(1), 22-32. Small, N.J.H. (1980). Marginal skewness and kurtosis in testing multivariate normality. Applied Statistics, 29, 85–87. Walonoski, J. & Heffernan, N.T. (2006). Detection and analysis of off-task gaming behavior in intelligent tutoring systems. Proceedings of the Eight International Conference on Intelligent Tutoring Systems, 382-391. Wigfield, A., Guthrie, J.T., Perencevich, K.C., Taboada, A., Klauda, S.L., McRae, A., & Barbosa, P. (2008). Role of reading engagement in mediating effects of reading comprehension instruction on reading outcomes. Psychology in the Schools, 45(5), 432-445. Wolters, C.W., Yu, S.L., Pintrich, P.R. (1996). The relation between goal orientation and students' motivational beliefs and self-regulated learning. Learning and Individual Differences, 8(3), 211-238. Zimmerman, B.J. & Bandura, A. (1994). Impact of self-regulatory influences on writing course attainment. American Educational Research Journal, 31(4), 845-862.

Acknowledgments
This research was supported by NSF grant DRL-1008649, awarded to Janice Gobert and Ryan Baker. We would also like to thank Michael Sao Pedro, Ermal Toto, Orlando Montalvo, and Juelaila Raziuddin for their role in the development of the physical science inquiry microworld.

© ISLS

174

ICLS2012

Volume 1: Full Papers

Exploring Connectedness: Applying ENA to Teacher Knowledge
Chandra Hawley Orrill, University of Massachusetts Dartmouth, Fairhaven, MA, corrill@umassd.edu David Williamson Shaffer, University of Wisconsin, Madison, WI, dws@education.wisc.edu Abstract: In this study, we consider teacher knowledge of mathematics from the perspective of connectedness. To accomplish this, we adapted Epistemic Network Analysis techniques to characterize the connections between and among pieces of teacher knowledge related to one aspect of proportional reasoning. We discuss the value of this approach as well as directions for further research.

Introduction & Significance
Over the past several years, there has been a growing interest in mathematics teachers’ knowledge (e.g., Ball, Thames, & Phelps, 2008; Hill, Ball, & Schilling, 2008). Despite researchers’ efforts to better understand both the nature of teacher knowledge and how that knowledge might shape learning experiences for students, research connecting teacher knowledge to student learning remains inconclusive. For example, Hill and colleagues have found a correlation between teacher knowledge and student growth (Hill, Rowan, & Ball, 2005). At the same time, Baumert and colleagues (Baumert, et al., 2010) were able to statistically distinguish between pedagogical content knowledge and content knowledge in measurements as well as to determine that teachers’ pedagogical content knowledge seemed more closely related to student learning than did content knowledge. While these studies suggest progress in understanding specialized knowledge teachers need, other studies present less promising results. For example, Shechtman and colleagues (Shechtman, Roschelle, Hertel, & Knudsen, 2010) did not find differences among student learning accounted for by teacher knowledge of mathematics for teaching (the same construct measured in the Ball et al. studies). Similarly, Kersting and colleagues (Kersting, Giviin, Sotelo, & Stigler, 2002) only found one subconstruct of several to be related to student learning despite their efforts to measure the knowledge that teachers use. In short, more research is needed to understand the nature of teacher knowledge. To this end, researchers have begun considering how different statistical models, paired with different conceptions of the domain, might help us better understand teacher knowledge (e.g., Izsák, Orrill, Cohen, & Brown, 2010). To further this line of work, in this exploratory study, we use an emerging understanding of teacher knowledge based on research suggesting that the development of expertise is, in part, related to one’s organization of knowledge (e.g., Bédard & Chi, 2006; Berliner, 1986; Hiebert & Carpenter, 1992; Leinhardt & Greeno, 1991). This is a departure from studies that consider teacher knowledge from the perspective of measuring amounts of information teachers know. We combine this emerging understanding with a relatively new analysis technique designed to highlight connections among “actors” in a system—whether those actors are people, ideas, or concepts. This approach, Epistemic Network Analysis (ENA; Shaffer et al., 2009), allows the discovery of the interconnected nature of expertise. ENA has been used to understand the processes by which students become more expert-like in game situations. In the current study, we consider whether ENA can be used to understand the connections among finegrained ideas about mathematics that teachers use in reasoning about a single proportional reasoning task. We undertake this as a first step toward refining a theory of teacher knowledge grounded in research on expert/novice differences (e.g., Bédard & Chi, 2006; Bransford, Brown, & Cocking, 1999) and related to the knowledge in pieces conceptions of learning (e.g., diSessa, 2006). Our findings highlight the potential benefits of using this approach and we include directions for further investigation.

Theoretical Framework
Expertise & Knowledge in Pieces
Our theoretical framework is grounded in the research on expertise, then introduces Epistemic Frame Theory as the framework of analysis for our study. As mentioned above, we situate this research in prior research on the development of expertise (e.g., Bédard & Chi, 2006) as well as knowledge in pieces (diSessa, 2006; diSessa, et al., 2004). Research on expertise has shown that while experts possess more knowledge of a domain than novices, the quantity of knowledge is not the factor that differentiates them from novices (Bédard & Chi, 2006). Rather, it is the organization of the knowledge that makes them different. For example, experts consider principles rather than surface features in solving problems (Bransford, Brown, & Cocking, 1999). Teachers, for example, differ in their ability to comprehend complex teaching situations in video with the experts more able to attend to important instructional aspects of the class and novices tending to become overwhelmed with all the details of the classroom videos (Sabers, Cushing, & Berliner, 1991).

© ISLS

175

ICLS2012

Volume 1: Full Papers

Developing expertise can be paired with the knowledge in pieces theory (diSessa, 2006; diSessa, et al., 2004), which posits that learners have many fine-grained understandings that are drawn together in different configurations when the learner is presented with a problem. We posit that middle grades mathematics teachers, who are often underprepared in mathematics, may have pieces of knowledge that are present, but not wellconnected to other mathematical understandings. For example, in our work, we have seen teachers fail to invoke particular understandings when posed with mathematics problems or student learning situations despite the fact that they have understanding of particular content. As an example, we interviewed one teacher who lamented the inclusion of a “rate” problem about free throws (e.g., which student would you select to make the next free throw based on given data of the students’ recent free throw attempts) in her fractions unit. Despite the teachers’ guide explaining its alignment to the curriculum in terms of equivalent fractions and decimal comparisons, the teacher was unable to reconcile the presence of this task. She knew about fractions and equivalence and she knew about rate, but she could not use those pieces of knowledge together. Certainly, we recognize that teachers need some amount of particular kinds of knowledge. But, we assert that learning more mathematics is unlikely to be helpful to teachers if they lack connectedness in their understanding (e.g., Hiebert & Carpenter, 1992). If teachers are not building connected networks of understanding, they only have a pool of individual ideas, disconnected, and unable to support them (and their students) in the classroom.

Epistemic Frame Theory
Building from the research on expertise, Epistemic Frame Theory (EFT) suggests that complex thinking can be understood in terms of the connections between frame elements: different skills, knowledge, values, identities, and epistemological rules from a particular domain. In our case, we are considering particular skills and knowledge related to proportional reasoning. EFT argues that expertise can be modeled as a network of connections between specific understandings, which are articulated through discourse. Assessing the development of such expertise, however, is a significant challenge. EFT allows us a means, ENA, to quantify the relationships between epistemic frame elements, and thus assess them, using network analysis (Shaffer et al., 2009; Shaffer & Graesser, 2010). Epistemic network analysis (ENA) extends social network analysis by focusing on the patterns of relations among discourse elements rather than individual ideas within that discourse. By treating epistemic frames as cognitive networks, different frame elements—the particular skills, understandings, identities, values and epistemologies of a profession—become nodes while the patterns of connections constitute the links between these nodes. By quantifying the patterns of connections between the different elements that combine to form epistemic frames, ENA methods provide a new alternative for measuring complex thinking and problem solving. Thus, we wanted to use them to determine whether they provide insight into the organization of teachers’ mathematical knowledge.

Figure 1: Task presented to interview participants

Methods
The data for this study were collected as part of a larger project aimed at understanding how teachers make sense of proportional reasoning. The data were drawn from a single question (Figure 1) that was included in a clinical interview protocol. The question was from a series of prompts considering the relationship of oil to

© ISLS

176

ICLS2012

Volume 1: Full Papers

vinegar in a salad dressing recipe. The videotaped interviews were conducted with middle grades teachers and transcribed verbatim. Those transcripts were coded for this analysis. Note that the researchers had previously coded these data as part of a grounded theory analysis. Therefore, the researchers were familiar with the teachers and their ideas at the outset of this exploratory study allowing coding to be based on the prior coding. We chose three teachers of varying mathematical ability based on their overall interview (not just the task of interest to this study). The strongest teacher, Lynda (all names are pseudonyms), had been a high school teacher while the other two teachers, Susan and Kate, were middle grades trained. Susan was in the middle in terms of overall performance on the interview tasks and Kate was the weakest. However, all three teachers were clear about their thinking making them ideal for exploring the proposed analysis techniques. Table 1: Codes used in the data analysis. Ratio Concepts: R1: Iteration R2: Ideas of Amount/Quantity/ Growth R3: Invariance R4: Iteration without Calculation R5: Ratio-as-Measure R6: Ratio Calculation Fraction Concepts: F1: Fractions (generic) F2: Rules for Operations F3: Equivalence F4: Comparison Representation (Diagram): D1: Interpretation of Diagram D2: Validate Representations Problem Solving: P1: Problem Interpretation P2: Verify Mathematics is Correct Other Mathematical Ideas: M1: Equality M2: Discussion of Fraction/Ratio Relationship M3: Attending to Context

Epistemic Network Analysis
We coded the interviews for important ideas related to connections between fraction and ratio ideas as they arose (Table 1). These codes captured ideas about fractions, ratios, problem solving, reasoning with drawings, and other mathematical topics. For this exploratory analysis, we treated each box in Table 1 as a node and used ENA to help us find the linkages between those nodes. Formally, an epistemic frame can be characterized by individual frame elements, fi, where i = a particular coded element of an epistemic frame. For any participant, p, in any given interview, s, each segment of discourse, Dp,s, provides evidence of whether participant p was using one or more of the epistemic frame elements (nodes). Each segment of coded data was represented as a vector of ones or zeroes representing the presence or absence, respectively, of each of the codes. Links, or relations, between epistemic frame elements were defined as co-occurrences of qualitative codes within the same segment. To calculate these links, each coded vector was then converted into an adjacency matrix, Ap,s, for participant p in a given story assignment, s (1). Ap,s i,j = 1 if fi and fj are both in Dp,s
p,s

(1)

Each coded segment’s adjacency matrix, A i,j, was then converted into an adjacency vector and summed into a single cumulative adjacency vector for each participant p, and each story assignment, Up,s, i.e., for each unit of analysis (2). (2) Up,s = ∑ Ap,s For each participant, p, and each story assignment, s, the cumulative adjacency vector, Up,s , was used to define the location of the segments in a high dimensional vector space defined by the intersections of each the codes. Cumulative adjacency vectors were then normalized to a unit hypersphere to control for the variation in vector length, representing frequencies of co-occurring code pairs, by dividing each value by the square root of the sum of squares of the vector (3). (3) nUp,s = Up,s / √∑(Up,s)2 This was necessary to emphasize similarities in the types of co-occurring code pairs rather than similarities based primarily on the frequencies of co-occurring code pairs. A singular value decomposition (SVD) was then performed to explore the structure of the qualitative code co-occurrences in the dataset. SVD was used to first project the normalized cumulative adjacency vectors into a high dimensional space such that similar patterns of co-occurrences between epistemic frame element codes would be positioned proximately. The SVD analysis then decomposed the structure of the data in this high dimensional space into a set of uncorrelated components, fewer in number than the number of dimensions that still account for as much of the variance in the data as possible. The resulting networks were then visualized by locating the original frame elements in a two dimensional space as follows: For each frame element ei we compute the mean coordinates x! and y! of all of the n2 loading vectors Vi,j for j=1 to n:

© ISLS

177

ICLS2012
!,! ! !!! V! !,! ! !!! V!

Volume 1: Full Papers

x! = y! =

!

This produces a distribution of nodes in the network graph determined by the loading vectors that contain them in the space of adjacency vectors. Nodes are then positioned in the space, and links are constructed according to the adjacency matrix.

!

Findings
We found substantial differences between the teachers’ network graphs. An initial analysis determined that fraction concepts largely fell to the left of the y-axis while ratio concepts fell more to the right. For example, in Kate (Figure 4) we see more attention to thinking about rules for operating on fractions (f2) and finding equivalence (f3). In contrast, Lynda’s map (Figure 2) shows more attention to ratio concepts, like ratio as measure (r5; meaning the measure, “taste”, is the fixed relationship between the vinegar and the oil) and the idea of invariance (r4; meaning the relationship between the two values remains constant). The x-axis appears to divide ideas related to procedural and conceptual understandings. For example, Susan’s graph (Figure 3) shows the relationship of fractions and ratios (m2) as well as problem interpretation (p1) below the x-axis.

Figure 2: Lynda’s map.

Figure 3: Susan’s map.

Figure 4: Kate’s map. The maps also show the teachers’ connectedness of concepts in response to our prompt. To appear as a point, a concept has to have been connected to another concept at least one time. Concepts with lines connecting are related by at least two connections. For example, in Lynda’s graph, ideas of quantity (r2) and attending to the context (m3) co-occurred at least twice, indicating some kind of relationship between those ideas for Lynda. Analysis of these maps suggests to us that Lynda relies on ratio reasoning and problem solving skills (e.g., understanding the problem) to make sense of ratios. In contrast, Kate relies on fraction rules and operations to make sense of these concepts. Susan, interestingly, leaned more toward ratio understandings, but introduced fewer key ideas together as evidenced by the limited number of points in her graph. We posit that this may be a natural step in the progression toward expertise. Kate, our least expert teacher, introduced many ideas that were not connected, while Lynda, the most expert-like teacher, introduced many ideas that were more closely connected. Lynda may have a high number of conceptual resources from which to draw and more connections among those ideas than Kate. Susan did not invoke as many ideas, but the ideas she did invoke were largely connected, suggesting that connectedness of ideas and number of ideas are not necessarily related

© ISLS

178

ICLS2012

Volume 1: Full Papers

in active problem-solving activities, supporting the findings of expertise research that suggest expertise is more than an accumulation of knowledge.

Next Steps
This exploratory study first confirms that fine-grained analysis of teachers’ cognition can lead to interesting and informative network maps. Next steps will include the analysis of larger datasets as well as refinements to the coding scheme to disentangle conceptual and procedural understandings. In short, we see ENA as having promise for making sense of teacher understanding, but more work is needed.

References
Ball, D. L., Thames, M. H., & Phelps, G. (2008). Content knowledge for teaching: What makes it special? Journal of Teacher Education, 59(5), 389-407. Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., …& Tsai, Y. M. (2010). Teachers’ mathematical knowledge, cognitive activation in the classroom, and student progress. American Educational Research Journal, 47(1), 133-180. Bédard, J., & Chi, M. T. H. (1992). Expertise. Current Directions in Psychological Science, 1(4), 135-139. Berliner, D. C. (1986). In pursuit of the expert pedagogue. Educational Researcher, 15(7), 5-13. Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn. Washington, DC: National Academy Press. diSessa, A. A. (2006). A history of conceptual change research: Threads and fault lines. In R. K. Sawyer (Ed.). The Cambridge handbook of the learning sciences (pp. 265-282). New York: Cambridge University Press. diSessa, A. A., Gillespie, N. M., & Esterly, (2004). Coherence versus fragmentation in the development of the concept of force. Cognitive Science, 28, 843-900. Hiebert, J., & Carpenter, T. (1992). Learning and teaching with understanding. In D. A. Grouws (Ed.) Handbook of research on mathematics teaching and learning (pp. 65-97). Reston, VA: National Council of Teachers of Mathematics. Hill, H. C., Ball, D. L., & Schilling, S. G. (2008). Unpacking pedagogical content knowledge: Conceptualizing and measuring teachers’ topic-specific knowledge of students. Journal for Research in Mathematics Education, 39(4), 372-400. Hill, H. C., Rowan, B, & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371-406. Izsák, A., Orrill, C. H., Cohen, A., & Brown, R. E. (2010). Measuring middle grades teachers’ understanding of rational numbers with the mixture Rasch model. Elementary School Journal, 110(3), 279-300. Kersting, N. B., Giviin, K. B., Sotelo, F. L., & Stigler, J. W. (2002). Teachers’ analyses of classroom video predict student learning of mathematics: Further explorations of a novel measure of teacher knowledge. Journal of Teacher Education, 61(1-2), 172-181. Leinhardt, G., & Greeno, J. G. (1991). The cognitive skill of teaching. In P. Goodyear (Ed.) Teaching knowledge and intelligent tutoring (pp. 233-268). Norwood, NJ: Ablex Publishing Company. Sabers, D. S., Cushing, K. S., & Berliner, D. C. (1991). Differences among teachers in a task characterized by simultaneity, multidimensionality, and immediacy. American Educational Research Journal, 28(1), 6388. Shaffer, D.W. & Graesser, A. (2010). Using Quantitative Model of Participation in a Community of Practice to Direct Automated Mentoring in an Ill-Formed Domain. Paper presented at Intelligent Tutoring Systems (ITS), Pittsburgh, PA. Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Frank, K., Rupp, A. A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st century assessment of learning. International Journal of Learning and Media, 1(2), 33-53. Shechtman, N., Roschelle, J., Haertel, G., Knudsen, J. (2010). Investigating links from teacher knowledge, to classroom practice, to student learning in the instructional system of the middle-school mathematics classroom. Cognition and Instruction, 28(3), 317-359.

Acknowledgments
This work was supported by the National Science Foundation through grants REC-034700, DUE-0919347, DRL-0918409, DRL-0946372, and EEC-0938517, DRL-0903411, DRL-1125621, and DRL-1054170 and the MacArthur Foundation. The views expressed here are those of the authors and do not necessarily reflect those of the funding agencies. The authors thank Joanne Lobato and Erik Jacobson for their assistance in collecting the data analyzed in this paper.

© ISLS

179

ICLS2012

Volume 1: Full Papers

Investigating the Effects of Varying Labels as Scaffolds for Visitor Learning
Joyce Wang a, Susan Yoon a, Karen Elinich b, and Jacqueline Van Schooneveld a a University of Pennsylvania, Philadelphia, PA b The Franklin Institute, Philadelphia, PA joycew@gse.upenn.edu, yoonsa@gse.upenn.edu, kelinich@fi.edu, jflicker@dolphin.upenn.edu Abstract: In museum literature, labels have been found to increase visitor learning and contribute to greater cognitive gains. In this study, we seek to understand how various labels support the visitors‘ learning experience, and specifically in regards to conceptual and cognitive learning. We investigated the increasing use of three labels (visual digital augmentations and text-based questions and instructions) in four treatment groups of middle school students. Our analyses indicated that when labels were used, there were significant increases in both conceptual and cognitive understanding. Furthermore, observation notes demonstrated that in the highest treatment group, there were increased instances of problem solving behavior as well as association with prior knowledge.

Introduction
Informal environments, such as science museums, have the potential to offer rich learning experiences (NRC, 2009). Though many studies have determined that museum visits increase visitors‘ curiosity, motivation, and interest in science, less have presented empirical evidence of visitors‘ cognitive or conceptual gains (Allen, 2007; Anderson et al., 2003; Falk et al., 2004). As such, we resonate with this year‘s ICLS conference theme, The Future of Learning, to investigate how museums can improve conceptual and cognitive learning through innovative technological advances, as well as through promising research of new approaches to learning. Most commonly, museums support visitors‘ learning through scaffolds incorporated into exhibit designs. Thus, we examine how various scaffolds affect museum visitors‘ learning experiences (NRC, 2009). This study follows a series of studies that investigate how designed interactives enhance learning in the museum setting (Yoon et al., 2011a, 2011b). In this study, we seek to understand how different forms of labels support visitors‘ learning. We conceive of labels as a type of scaffold that can vary in their representational nature to reveal information visitors would otherwise be unaware of or unable to acquire by themselves. Although labels are traditionally presented as printed text (typically explanatory in nature), we expand this boundary to include any form of a guide that scaffolds a learning experience. We compare the learning outcomes of middle school students placed in several treatment groups that vary in the amount and type of labels as they interact with an exhibit aimed at teaching circuits and electrical conductivity. The research question investigated was ―How does the increasing use of various representations of labels affect students‘ ability to make sense of the information with which they are being presented?‖

Conceptual Framework
In museum literature, labels, originally defined as ―written words used alone or with illustrations in museum exhibitions to provide information for visitors, presented as text on exhibit graphic panels or computer screens‖ (Serrell, 1996, p.239), have been found to impact visitors‘ experiences at the exhibits in various ways. When they are written clearly to convey the goals of the exhibit, more visitors will understand, find meaning in, and enjoy museum exhibitions. Some studies have documented how different types of labels change the way visitors interact with exhibits (e.g. Atkins et al., 2009) while other studies have investigated how labels affect the type of conversations that ensue between group members (e.g., Hohenstein & Tran, 2007). Ultimately, labels seek to increase visitors‘ learning and to contribute to greater cognitive gains (Falk, 1997; NRC 2009). In this study, we want to build upon these claims and examine different ways in which labels can be constructed to elicit deeper understanding. We investigated two representations of three labels: visual digital augmentations and text-based questions and directions. We hypothesized that these labels, when used together, would collectively target and enable students to construct their own knowledge, and would reflect an increase in their level of thinking and learning. In reframing the conception of a museum label more generally as a scaffold, we expand upon the conventional consideration of labels as solely offering descriptive information to support the individual visitor, to include ways in which labels purposefully change visitors‘ behaviors to support each other‘s learning and consequently, their own learning as well.

© ISLS

180

ICLS2012

Volume 1: Full Papers

Digital Augmentation Labels (through Augmented Reality)
At its simplest, Augmented Reality (AR) describes systems that integrate computer-generated virtual elements (objects or environments) or information with the real world environment (Mou et al., 2004). Specifically by superimposing virtual elements and images (which we define as ‗digital augmentations‘) onto the real world environments, AR allows users to experience and perceive the newly incorporated information as part of their present world, thereby enhancing their perception and interaction with the real world (Kirkley & Kirkley, 2004). In this way, AR serves as a scaffold by supporting the user with additional (virtual) information, which might not be directly detected by their senses, to aid in their performance of specific tasks. It is precisely in this scaffolding role that AR offers the unique potential to transform learning at multiple levels. For this reason, we consider these digital augmentations as a type of label in our study. In classrooms, AR has been employed in the form of handheld games to teach mathematical and scientific concepts (e.g. Dunleavy et al., 2009; Klopfer & Squire, 2008) and to cultivate the development of inquiry skills, argumentation skills, and collaborative learning (e.g. Rosenbaum et al., 2006). For example, Klopfer and Squire (2008) developed a Pocket PC game, enabled with augmented reality, to investigate a chemical spill in a watershed that could potentially contaminate ground and surface water. One of the outcomes of the study indicated that students became active problem solvers who learned to critically analyze strategies and negotiate with each other. They were often observed arguing about different investigative strategies that they believed would give them the most reliable information to formulate a solution (Klopfer & Squire, 2008). Although AR has been used in school settings to scaffold learning, in museums, it is still a fairly new phenomenon. While a few studies have investigated the impact of AR devices on visitors‘ experiences (e.g. Szymanski et al., 2008), more have evaluated the design and creation of an AR system and the feasibility of incorporating AR into the museum environment (e.g., Damala et al., 2008). A significant finding of several of these studies reveals that AR increases visitors‘ engagement and interest in museum settings when using these devices (e.g., Hall & Bannon, 2006). While this is an important component of museum experiences, this focuses primarily on engagement and access rather than on learning. In terms of conceptual and cognitive learning, there has been a dearth of research that investigates the impact that AR has on visitors.

Question Labels
Text-based question labels were also incorporated in our investigation as a method to scaffold learning. Questions, as a pedagogical strategy, have the potential to stimulate visitors to think, look, get involved with, and converse with others to ultimately learn about a particular exhibit (Serrell, 1996). In interactive exhibits, question labels can invite visitors to actively participate in an exhibit and thereby integrate their experience with the exhibit‘s main message (Serrell). In some cases, they invite and challenge the visitor to use the exhibit to accomplish a certain task (Gutwill, 2006). Regardless of the method of invitation, questions in labels act to encourage physical engagement of visitors, thereby helping them to anticipate and formulate new meanings (Serrell, 1996). A few studies have evaluated the effectiveness of incorporating questions in exhibit labels to stimulate participation and learning. For example, Hohenstein and Tran (2007) examined the use of guiding questions to stimulate conversations in an object-based science museum. Their findings indicated that in the presence of these questions, visitors were more likely to not only offer explanatory answers to the questions, but to also ask their own open-ended questions. However, it should be noted that the addition of these questions did not have the same effect on all exhibits. Thus, the context of the exhibit must also be considered when creating labels to assist visitors‘ experiences. A different study also investigated the impact that questions had on visitor behavior (their learning and their conversations) in their interaction with objects at a science exhibit (Atkins et al., 2008). After including a hybrid label, a label that involved both questions and suggestions for improved visitor engagement, the exhibit was more likely to be perceived as a lesson with an intentional goal and a particular learning agenda. Specifically, in the presence of the labels, parents were observed to explain the phenomena to their children more often than when the label was not present (Atkins et al., 2008). As such, these studies indicate that text-based questions have the potential to encourage visitors to become active learners and thus engage in deeper levels of critical thinking that develop increases in conceptual knowledge.

Instructional Labels to Promote Collaboration
The last label that our study sought to investigate are labels that instruct visitors on collaboration. Instructional labels have been used to guide visitors in using exhibit devices to produce a particular outcome that illustrates a scientific phenomenon (Gutwill, 2006). In instances like these, labels provided directions that signal to the visitor whether the device worked or not depending on how the visitor operated the device. Unfortunately, the disadvantage of these types of instructional labels is that they may communicate an endpoint or finality that would inhibit further exploration by the visitor (Gutwill, 2006). While these studies have acknowledged how museums have commonly used labels to inform visitors of specific ways to interact with exhibits, there have

© ISLS

181

ICLS2012

Volume 1: Full Papers

been a lack of studies that not only investigate the effects of labels that instruct visitors to collaborate with one another to build knowledge, but that indicate that museums are designing these types of labels. In formal education, collaborative learning has been found to have a positive impact upon student achievement (O‘Donnell, 2006). Regardless of how the groupings are formed, whether homogenous or heterogenous for ability, when students collaborated together, they exhibited increased levels of achievement and motivation (Saleh et al., 2007). A critical factor that impacts students‘ learning in these groups, however, is the extent to which structures are in place to support effective collaboration skills. Many studies have investigated the effect of various scaffolds that support collaboration, including the assignment of student roles, cues, scripts, and rules (Kester & Paas, 2005; Saleh et al., 2007). For example, it was found that when students were verbally cued to summarize, explain, identify errors, or ask for help from one another, they demonstrated higher learning gains. These studies indicate the impact and potential that collaboration, when designed for, can have on learning in the formal environment. Consequently, these results can also offer important considerations for informal environments. As labels are an elemental component within the museum space, we suggest modifying them in a way to encourage visitors to collaborate more. Unfortunately, there has been a dearth of studies that investigate how labels have been designed to encourage visitor collaboration. Thus, we attempt to address this gap in the literature by investigating how instructions for collaborative interaction, incorporated in text-based labels, affect visitors‘ learning and behaviors.

Methods Context & Participants
This study leveraged a currently funded large scale National Science Foundation informal science education project in which the central goal is to design, integrate, and increase the use of educational technologies—and to study their impact—within the science museum learning experience (Yoon et al., 2011a, 2011b). While the larger project focuses on the viability of using augmented reality technologies in informal science settings, the present study extended this focus by investigating the effects of various labels to scaffold understanding. In total, 164 students, from nine diverse schools (including public, charter, urban and suburban) participated. The participants were 6th, 7th, and 8th grade students, chosen because the device addresses electricity and magnetism, topics which are normally covered in grade 4 of the local school curriculum. Thus, all students would already have some prior knowledge. In order to recruit student participants, we enlisted the help of science teachers who were associated with either the research institution or the museum. Students‘ participation in our study was embedded in a larger school field trip. Prior to the day of implementation, researchers went to schools to administer presurveys to the students. On the day of implementation, students were randomly assigned to visit a particular exhibit from which researchers would pull students in pairs or groups of 3 and randomly assign them to one of four conditions (C1–C4) to test the study device. Refer to Table 1 for participant breakdown and a description of the conditions. (The variance in participant numbers for C4 is due to a larger study, consisting of 6 conditions that required greater participant numbers (Yoon et al., 2012). Whereas the larger study sought to investigate the effects of certain structured knowledge building scaffolds on visitor learning, this study analyzes the knowledge building scaffolds as prompts and labels that merely served to complement the device.) The device ―Be The Path‖ consisted of a table with two metal spheres (located at opposite ends of the table), connected in a half loop with wires to a battery and a light bulb. As is, the loop is not fully closed (and is referred to as an ―open circuit‖) so the bulb does not light. When individuals place their hands on the metal spheres, the bulb would light up because their body closed the loop to complete the circuit. In the testing room, students were told to play with the device as if they had come across it on the museum floor while two researchers recorded observation notes of students‘ behavior. After their engagement with the device, the students were led to a different location to answer some questions regarding their interaction and to reflect upon what they learned. (These reflection questions were the same questions on the question labels in conditions 3 and 4.) Following the implementation, researchers administered a post-survey (same as the pre-survey) to the students. Table 1: Description of each condition
Cond. 1 2 N 30 31 Description Control: Device without any additional scaffolds and representative of the current devices out on the exhibit floor. Addition of visual digital augmentation label (DA): When students completed the circuit and made the bulb light up, a projected image (of moving electrons) would appear from overhead and map onto the device and the students‘ bodies. Label s None DA

© ISLS

182

ICLS2012 3 31

Volume 1: Full Papers Addition of question labels (QL): 1. What happened when you touched both metal spheres? 2. What happened when you only touched 1 metal sphere? 3. What happened to make the bulb light up? 4. What does the projection show? 5. What are you supposed to learn by using this device? Addition of instruction labels (IL):  Write down your answer only after your group has come to agreement on it.  Be clear about the difference between what you actually observe happening and what you think about it.  It is important that you listen to each other.  For each question: 1. Brainstorm possible answers. 2. Give reasons to support your answers. 3. Compare your answers and the reasons behind them. 4. Decide on the best possible answer. DA + QL

4

72

DA + QL + IL

Data Sources & Analyses
1. Pre- and post-surveys were gathered to measure changes in students‘ conceptual understanding before and after the implementation. These surveys consisted of 5 multiple choice questions on electricity and magnetism as well as 1 open-ended question (―Think about an electric circuit that supplies electricity to a light bulb. What parts make it work so the bulb lights up? Use words and pictures in your answers.‖) Although we included multiple choice questions (compiled from district curricular materials) to collect some baseline data of students‘ general understanding of electricity, the concept that our device was demonstrating was most related to the open-ended question. A categorization manual was constructed to evaluate this last question and interrater reliability was obtained on 20% of the responses by two independent researchers (α = 0.94). The open-ended responses were coded on a scale from ‗No Understanding‘ (Level 1) to ‗Complete Understanding‘ (Level 5). Paired samples t-tests were conducted for each condition to determine whether there were significant differences in students‘ understandings before and after the intervention. An ANOVA was also conducted to determine which particular label had the greatest impact. Student written reflections were solicited regarding their understanding of the science phenomenon being presented. Although there were five questions asked, only one question (―What were you supposed to learn by using this device?‖) was coded, as it most clearly illustrated their ability to think critically, theorize, and make sense of the phenomenon. Thus, this question intended to evaluate changes in cognitive learning. (These reflection questions were also the same questions on the question labels in C3 and C4). A separate categorization manual was also constructed to code these responses (on a scale from ‗No Understanding‘ (Level 1) to ‗Complete Understanding‘ (Level 5)) and interrater reliability was obtained on 20% of the responses by two independent researchers (α = 0.948). An ANOVA was conducted to determine whether the differences were statistically significant. Researcher observation notes were collected and coded to register different types of critical thinking skills that the students were seen engaging in. The categories, adapted from the Critical Thinking Skills Checklist described in Luke et al. (2007) and refined to reflect the typical experiences of a science museum, were: Participating, Interpreting, Evaluating, Associating with Prior Science Knowledge, Problem Solving, Associating with Other Knowledge, and Engaging in Teamwork. These seven categories were further divided into more refined subcategories (for example, Interpreting included behaviors such as ‗asking questions‘ and ‗exploring how the device operated‘) and were then coded as ‗1‘ or ‗0‘ depending on whether the students exhibited these components. The frequencies of these exhibited skills were then compared across the 4 conditions and a chi-square test was administered to determine if the results were statistically significant.

2.

3.

Results Pre-/Post-Surveys (Measuring Conceptual Learning): Aggregated & Disaggregated for Multiple Choice and Open-Ended Questions
A paired-samples t-test indicated that significant learning gains were made between the pre-surveys (M = 4.22, SD = 1.59) and post-surveys (M = 4.81, SD = 1.60), t(163) = -5.04, p = .01. Specifically, as our device targeted the conceptual understanding in the open-ended response question, we then disaggregated the data to analyze these scores independent of the entire survey score (which was comprised of the multiple choice and openended questions). As shown in Table 2, conceptual knowledge gains were also reflected in students‘ open© ISLS 183

ICLS2012

Volume 1: Full Papers

ended responses between the pre- and post-surveys. Between the conditions on just the post-survey scores, there was a general increase in understanding on the open-ended questions, which were reflected in increases on the entire test as well (see Table 3 for means). However, an ANOVA conducted between the four conditions revealed that these learning gains as reflected in the open-ended question, F(3, 160) = 1.82, p = 0.15, and in the test as a whole, F(3, 160) = 1.25, p = 0.29, were not a direct result of the incorporation of labels. Table 2: Paired-samples t- test results of open-ended responses on pre-/post-surveys
Condition Cond. 1 Cond. 2 Cond. 3 Cond. 4 Test Pre (OE) Post (OE) Pre (OE) Post (OE) Pre (OE) Post (OE) Pre (OE) Post (OE) Mean 2.17 2. 27 2.19 2.65 2.03 2.48 2.25 2.72 SD 1.15 0.78 0.95 0.84 1.08 1.09 0.92 0.97 t value (df) 0.47(29) -2.72(30) -2.04(30) -4.6(71) p 0.64 0.01* 0.05* 0.00*

Table 3: ANOVA results of post-survey scores (aggregated and disaggregated) between conditions
Condition Cond. 1 Cond. 2 Cond. 3 Cond. 4 Post MC Scores (Mean, SD) 2.37, 1.16 2.16, 0.82 2.10, 0.94 2.29, 1.20 Post OE Scores (Mean, SD) 2.27, 0.78 2.65, 0.84 2.48, 1.09 2.72, 0.97 Post-Survey Scores (Mean, SD) 4.63, 1.50 4.74, 1.29 4.58, 1.59 5.01, 1.60

Student Reflections (Measuring Cognitive Learning)
Although there seemed to be a generally increased ability to think and reason at a higher level as more scaffolds were added between C1 (M = 2.23, SD = 0.50), C2 (M = 2.39, SD = 0.76), C3 (M = 2.48, SD = 0.72), and C4 (M = 2.43, SD = 0.80), the main effect of the labels was not significant, F(3, 160) = 1.93, p = 0.13. However, an ANOVA performed between C1 and all other conditions aggregated together, indicated that there was a significant difference F(1, 162) = 5.84, p = 0.02. The observation notes reflected some differences in the students‘ interactions between the four conditions. Although there were seven skills being observed, a chi-square test revealed that there were significant differences in the frequencies of only two categories: Problem Solving (x2 = 9.166, df = 3, p < .027) and Association with Prior Science Content Knowledge (x2 = 12.731, df = 3, p < .005). C4 and C2 students exhibited the greatest instances of demonstrating Problem Solving processes (represented by behaviors that indicated investigation of various configurations in lighting the bulb). Specifically, 96% and 94% of C4 and C2 students respectively reflected these behaviors, as compared to students in C3 (78%) and C1 (84%). Likewise, students in C4 tended to associate their experience with other science knowledge (usually involving recall of similar activities performed in science classes) (26%) more often than students in C3 (9.7%), C2 (13%) and C1 (0%).

Observation Notes

Discussion & Implications
Museum studies have consistently acknowledged the importance of exhibit labels on visitors‘ learning and behaviors (Atkins et al., 2009; Falk, 1997; Gutwill, 2006; Hohenstein & Tran, 2007). In accordance with this, the goal of our study was to evaluate, not only the effects of various types of labels on students‘ learning, but also the effects of increasing the use of these labels on students‘ conceptual and cogntivie understanding of electricity. Through pre- and post-surveys (to evaluate conceptual understanding), written reflections (to evaluate cognitive skills) and observation notes (to evaluate behaviors that would indicate changes in either conceptual or cognitive understanding), we wanted to understand how students made sense of the phenomenon in both a performative and generative context. To this end, we found significant increases in students‘ conceptual understanding after the museum visit, consistent with prior research (NRC, 2009), for students in C2, C3, and C4. Looking within each condition, we found that Condition 2, 3, and 4 students, collectively, after our intervention, significantly understood better the components and operation of an electric circuit. On average, students‘ understanding increased by half a point (which is reflective of half a level of understanding) – a significant result considering
© ISLS 184

ICLS2012

Volume 1: Full Papers

that their interaction with the device was, like most interactions with museum exhibit devices, brief. This positive finding suggests that our device, when scaffolded with various labels, supports students‘ learning of electricity and ciruits. Additionally, at first glance, we noticed that the increase in students‘ conceptual understanding was in proportion to the number of labels they received. For example, it appeared that students in C4, who experienced the most labels, exhibited the most understanding as they had the highest scores. However, an ANOVA conducted across the four conditions indicated that these differences in knowledge growth were not significant. Thus the condition, and subsequently, the labels that comprised each condition, had no significant effect on students‘ understanding. A C4 student would increase their knowledge as much as a C2 student. However, as C1 students who didn‘t receive any labels also did not exhibit any significant knowledge growth, this would suggest that at least one label is needed to significantly impact learning. Thus, this implies that the amount and types of labels that one can use to scaffold a visitor‘s experience does not make a difference. In regards to the impact of labels on cognitive learning, we evaluated students‘ responses on their reflections. Specifically, we hoped that the digital augmentation labels, question labels, and instruction labels on collaborative work, would to varying degrees, affect students‘ ability to theorize about how our bodies can complete a circuit and conduct electricity. Our results indicated that initially, it appeared that as we increased the amount of labels to scaffold their learning, they exhibited greater abilities to think and theorize at a higher level. However, similar to their survey results, an ANOVA on their reflections determined that the increases in their ability to theorize were not significant across the four conditions. Interestingly though, when we aggregated Conditions 2, 3, and 4 together to represent ‗Labeled Conditions‘ and compared this to Condition 1, the differences in their cognitive scores were significant. Thus, this confirms our earlier speculation that the amount and type of labels are not as significant in learning as the presence of just one of these labels. In addition to conceptual and cognitive learning, we were interested in understanding how these three labels affected students‘ behaviors and interactions, that would help us to understand their changes in learning. A chi-square test revealed that there were significant differences in students‘ behaviors in two categories: Problem Solving and Association with Prior Knowledge. Specifically, we saw a marked difference in C4 students‘ interactions; as this label pointedly instructed students on ways to collaborate together, it was encouraging to see more students investigating and attempting additional configurations to light the bulb. For example, a few groups discovered that when the students all linked hands, the bulb would light up. Other students would experiment using different parts of their body including their elbows, fingertips, and the backs of their hands. The discovery of these phenomena often led to expressions of excitement and wonder. Other studies have confirmed that museums can stimulate joy, delight, awe, wonder, appreciation, surprise, etc. (NRC, 2009) and that these affeective reponses are linked to other forms of learning (e.g., Steidl et al., 2006). It was interesting to note that C2 students also demonstrated very high instances of Problem Solving behaviors. This would suggest to us that the novelty of the digital augmentation label by itself, was just as impactful as the instructional label in encouraging this kind of behavior. When the digital augmentation was combined with another label, specifically the question label, however, this exploratory behavior decreased. Likewise, C4 and C2 students were observed making associations with other knowledge in more instances than C3 or C1 students (though the frequency for this behavior is not as similar as that of Problem Solving). The labels in these conditions encouraged students to connect their interaction between our device and other knowledge gained from prior experiences. Some recalled similarities to other school science class activities while others remembered that they had learned about this content previously. In C4, because our instructional labels strongly influenced students to collaborate together, we observed greater amounts of dialogue relating to electricity (though this was not formally measured). Thus, it is not surprising to us that as students are talking about these questions and sharing their observations and hypotheses, their conversations would include more instances of connections to prior knowledge. Though we did not find as many significant results as we had anticipated, we would not rush to conclude that these labels are ineffective. Instead, we suggest revisitng the context and conditions in which our intervention was administered as well as the design of the instruments themselves. First, we posit that the survey question may not have been sensitive enough to measure the smaller differences in students‘ thinking and learning. We worded the question broadly so that students would not feel constrained to answer in a particular manner. However, as a number of responses were too general (i.e., ―We learned about electricity.‖) to offer any significant information, we were not able to accurately assess what students learned and thus, had to code their responses as ―No Understanding‖. Second, the variation in learning results could be explained by the different forms of assessments that were used to measure thinking (hands-on versus paper-pencil) and to the different environments in which the students were assessed. We suggest there may have been a failure of transfer between what was observed and written. Educational researchers have long since theorized that a learning experience may look differently depending on the testing context, as the manner in which the information is learned affects transfer (Bransford & Schwartz, 1999). Thus, while it appeared that students were able to think at a higher level when they were engaging with the device (especially in C4), this aptitude did not transfer once students were removed from the original context and asked to reflect upon it in writing. Lastly,

© ISLS

185

ICLS2012

Volume 1: Full Papers

although we tried to control for this, it was apparent that many students rushed through both the reflections and the post-surveys in an attempt to hurry on to the next activity. For example, some students remarked impatiently that they wanted to finish the reflections quickly so they could return to the main exhibits and continue their exploration. Similarly, during the administration of the post-surveys, many students objected that this was exactly the same thing that they answered previously. Although the researchers explained the rationale for requiring both a pre- and post-survey to the students, informal comparisons (acquired during the collection of the post-surveys) reflected that students had not put as much effort and thought into the post-surveys as they did in the pre-surveys. In the next iteration of this study, we suggest collecting this information in a different format, perhaps embedded in interviews This study sought to investigate the effects of layering three types of scaffolds, in the form of labels, on students‘ conceptual and cognitive learning. While the results didn‘t support all of our hypotheses, we did learn that the addition of at least one of our labels significantly increases conceptual learning. Furthermore, we were encouraged by changes in students‘ interactions to reflect more scientific inquiry behaviors. As we begin to plan for future phases of this study, the findings we‘ve discovered here will guide our decisions as we seek to revisit the context, methods and instruments used to evaluate museum learning.

References
Allen, S. (2007). Secrets of circles summative evaluation report. Report prepared for the Children‘s Discovery Museum of San Jose. Available: http://www.informalscience.org/evaluation/show/115 Anderson, D., Lucas, K., & Ginns, I. (2003). Theoretical perspectives on learning in an informal setting. Journal of Research in Science Teaching, 40(2), 177-199. doi: 10.1002/tea.10071 Atkins, L., Velez, L., Goudy, D., & Dunbar, K. (2009). The unintended effects of interactive objects and labels in the science museum. Science Education, 93, 161-184. doi: 10.1002/sce.20291 Bransford, J. & Schwartz, D. (1999). Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education, 24, 61-100. doi: 10.2307/1167267 Damala, A., Cubaud, P., Bationo, A., Houlier, P., & Marchal, I. (2008). Bridging the gap between the digital and the physical: Design and evaluation of a mobile augmented reality guide for the museum visit. In 3rd ACM International Conference on Digital and Interactive Media in Entertainment and Arts, 120-128. New York: ACM Press. Dunleavy, M., Dede, C., Mitchell, R. (2009). Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. Journal of Science Educational Technology, 18, 7-22. doi:10.1007/s10956-008-9119-1 Falk, J. H. (1997). Testing a museum exhibition design assumption: effect of explicit labeling of exhibit clusters on visitor concept development. Science Education, 81, 679-687. doi: 10.1002/(SICI)1098237X(199711)81:6<679::AID-SCES>3.3.CO;2-7 Falk, J., Scott, C., Dierking, L., Rennie, L., & Jones, M. (2004). Interactives and visitor learning. Curator, 47(2), 171-198. doi: 10.1111/j.2151-6952.2004.tb00116.x Gutwill, J. (2006). Labels for open-ended exhibits: Using questions and suggestions to motivate physical activity. Visitor Studies Today, 9(1), 1-9. Hall, T., & Bannon, L. (2006). Designing ubiquitous computing to enhance children‘s learning in museums. Journal of Computer Assisted Learning, 22(4), 231-243. Hohenstein, J., & Tran, L. U. (2007). Use of questions in exhibit labels to generate explanatory conversation among science museum visitors. International Journal of Science Education, 29(12), 1557-1580. doi: 10.1080/09500690701494068 Kester, L., & Paas, F. (2005). Instructional interventions to enhance collaboration in powerful learning environments. Computers in Human Behavior, 21, 689-696. doi:10.1016/j.chb.2004.11.008 Kirkley, S., & Kirkley, J. (2004). Creating next generation blended learning environments using mixed reality, video games and simulations. TechTrends, 49(3), 42-89. doi: 10.1007/BF02763646 Klopfer, E., & Squire, K. (2008). Environmental Detectives – the development of an augmented reality platform for environmental simulations. Educational Technology Research and Development, 56, 203-228. doi: 10.1007/s11423-007-9037-6 Luke, J., Stein, J., Foutz, S., & Adams, M. (2007). Research to practice: Testing a tool for assessing critical thinking in art museum programs. Journal of Museum Education, 32(2), 123-136. Mou, W., Biocca, F., Owen, C., Tang, A., Xiao, F., & Lim, L. (2004). Frames of reference in mobile augmented reality displays. Journal of Experimental Psychology: Applied, 10(4), 238-244. doi: 10.1037/1076898X.10.4.238 National Research Council (NRC). (2009). Learning Science in Informal Environments: People, Places, and Pursuits. Washington, DC: National Academies Press. O‘Donnell, A. (2006). The role of peers and group learning. In P. H. Winne & P. A. Alexander (Eds.), Handbook of educational psychology (pp. 781-802). Erlbaum: Mahwah, NJ.
© ISLS 186

ICLS2012

Volume 1: Full Papers

Rosenbaum, E., Klopfer, E., & Perry, J. (2006). On location learning: Authentic applied science with networked augmented realities. Journal of Science Education and Technology, 16(1), 31-45. doi:10.1007/s10956006-9036-0 Saleh, M., Lazonder, A., & de Jong, T. (2007). Structuring collaboration in mixed-ability groups to promote verbal interaction, learning, and motivation of average-ability students. Contemporary Educational Psychology, 32, 314-331. doi:10.1016/j.cedpsych.2006.05.001 Serrell, B. (1996). Exhibit labels: An interpretive approach. Walnut Creek, CA: AltaMira Press. Steidl, Sl., Mohi-uddin, S., & Anderson, A. (2006). Effects of emotional arousal on multiple memory systems: Evidence from declarative and procedural learning. Learning & Memory, 13(5), 650-658. Available: http://www.ncbi.nlm.nih.gov/pubmed/17015860 [accessed October 2011]. Szymanski, M., Aoki, P., Grinter, R., Hurst, A., Thomton, J. & Woodruff, A. (2008). Sotto Voce: Facilitating social learning in a historic house. Computer Supported Cooperative Work, 17(1), 5-34. Yoon, S., Elinich, K., Steinmeier, C., Wang, J., & Tucker, S. (2011a). Learning science through knowledgebuilding and augmented reality in museums. In the proceedings of the annual conference of the International Conference on Computer Supported Collaborative Learning, Hong Kong, China. Yoon, S., Elinich, K., Wang, J., Steinmeier, C., & Van Schooneveld, J. (2011b). Fostering critical thinking in science museums through digital augmentations. In the proceedings of the annual conference of the International Conference on Computer Supported Collaborative Learning, Hong Kong, China. Yoon, S., Elinich, K., Wang, J., & Van Schooneveld, J. (2012, April). Augmented reality and knowledge building: To what extent can they enhance learning in a science museum? Paper to be presented at the annual meeting of the American Educational Research Association, Vancouver, British Columbia, Canada.

Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. 0741659. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

© ISLS

187

ICLS2012

Volume 1: Full Papers

Making Technology Visible: Connecting the Learning of Crafts, Circuitry and Coding in Youth e-Textile Designs
Yasmin B. Kafai, Deborah A. Fields, & Kristin A. Searle University of Pennsylvania, 3700 Walnut Street kafai@upenn.edu, stareyes@gmail.com, searle@dolphin.upenn.edu Abstract: We examine high school students‟ designs with the LilyPad Arduino, an electronic textile (e-textile) construction kit used for designing programmable garments. Each kit contains a microcontroller, sensors and LED and other actuators that can be embedded in textiles. We conducted three workshops with 35 high school youth between 14-15 years in a science museum to document, describe and develop a framework for analyzing student learning. The analyses of workshop interactions, students‟ artifacts and reflections indicate how the fabrication of stitches, circuits and code reveals the underlying structures and processes of craft, engineering and programming in tangible and observable ways and renders visible how technology is designed and built. We discuss how this situated nature of e-textiles artifact production provides a promising context for student learning and generates new instructional designs of workshops and computational construction kits.

Introduction
This paper focuses on learning with a new type of tangible, programmable media called electronic textiles (etextiles hereafter). E-textiles include young people's designs of programmable garments, accessories, and costumes. Such designs incorporate elements of embedded computing for controlling the behavior of fabric artifacts, novel materials such as conductive fibers or Velcro, sensors for light and sound, and actuators such as LEDs and speakers, in addition to traditional aspects of fabric crafts using needles, thread, and cloth among others. The development of the LilyPad Arduino (Buechley, 2006) has made e-textiles accessible to novice designers in the same way the design of Lego/Logo (Resnick & Ocko, 1991) did for robotic constructions. Etextiles introduce aesthetic elements that can enrich youth's expressive and intellectual lives and materials that make programming accessible to new groups (Eisenberg, Eisenberg, Buechley & Elumeze, 2006). To date, most of the work around e-textiles has focused on technical developments to test usability and design of materials and activities. We take the next step in documenting, describing and analyzing the learning that takes place as high school youth engage in designing e-textile artifacts. In understanding students‟ design processes and products, we focus on how the craft activities situate and connect engineering and programming in e-textiles, making abstract concepts concrete for learners and making the learning that is (or is not) taking place visible to researchers. This is not just an effort to emphasize interdisciplinary connections present in e-textile designs but also to highlight what Eisenberg et al. (2006; see also Buechley, 2010) called the invisibility principle that “when technology is invisible, it is deliberately placed outside the users‟ awareness; thus there is little reason to communicate how the technology works, and how the user might extend or control it.” (p. 2). It might be productive for expert programmers to know that their black-boxed programs may be taken up exactly and used as they are (Turkle & Papert, 1992), likewise average users might find it appropriate and functional not knowing how computer or software works. However, for learners this invisibility might be counterproductive in some instances because students are unable to grapple with the messiness of the technology: to take things apart and put them back together in a multitude of ways (Resnick, Berg & Eisenberg, 2000). In e-textiles, the fabrication of stitches, circuits and code can reveal the underlying structures and processes of craft, engineering and programming in tangible and observable ways and renders visible how technology is designed and built. Our analyses focus on a series of workshops that took place in a science museum partnering with an urban public high school during the 2010-11 school year. A total of 35 female and male high school freshmen participated in designing e-textiles. While learning how to sew, how to make functional circuits, and how to write code to light up LEDs and control sensors are each important and valuable practices in their own right, learning how to see the connections between these parts is what makes learning to design e-textiles greater than its individual parts. How do young designers see the connections between the stitches they make and the functioning of the circuits they have drawn? How do they understand how writing the code relates to the circuits they have laid out? How can they craft pieces that allow changes in code and vice versa? Finally, what do students‟ conceptualizations of the relationships between craft, circuitry, and code tell us about what they are learning through their engagement with the LilyPad Arduino construction kit? These research questions are at the heart of the analytical framework we developed to understand students‟ learning with and through e-textiles. In the next sections, we review what we know about learning with tangible programmable artifacts at large.

© ISLS

188

ICLS2012

Volume 1: Full Papers

Background
E-textiles are a relatively new development (Post, Orth, Russo, & Gershenfeld, 2000) and only in recent years have e-textile construction kits become available to novice programmers (Buechley, 2006; Ngai, Chan, Cheung, & Lau, 2009; Katterfeldt, Dittmar, & Schelhowe, 2009). Most of what we know about learning with such tangible construction kits comes from research on robotics activities developed and implemented across K-16 settings (Bers, 2009; Druin & Hendler, 2000). In general, these studies find robotics to be engaging activities for students of all ages (and even participating parents) but rarely provide much detail on how and what students learn in constructing a robot. A notable exception is a study by Sullivan (2008) that aligned learning robotics with practices and themes relevant for science literacy. She found that building robotics engages middle school students in (1) the “development of certain thinking skills such as computation and estimation, manipulation and observation, communication, and critical-response skills; (2) the ability to engage in the activity of inquiry” and “(3) the development of an understanding of the common themes of science such as systems, models, constancy and change and scale” (p. 374). Robotics activities are now a prominent part of afterschool programs and have developed over the last 10 years into a nation-wide network of competitions. These findings are relevant for our work because e-textile activities share many common elements with robotics activities but are also distinct in some aspects. Like robotics, e-textiles involve designing and constructing physical artifacts that operate with different sensors and actuators and coordinating interactions via writing of program code. Unlike robotics, the purpose of these artifacts is not to compete but rather is aligned with more personal, decorative goals. Both construction activities are situated in different domains: in the case of robotics the artifacts are associated with the engineering of motors while in the case of e-textiles the artifacts are associated with the crafts of sewing and embroidery. Designing e-textile and robotics artifacts involve an array of different activities, each complex in its own right, that need to be coordinated. Understanding the learning of crafts, circuitry and code then draws on learning in multiple disciplinary contexts that have a longstanding disciplinary tradition, but with e-textiles they venture into newer territory. The crafting involved in e-textiles is the most unusual practice and also the one that has a minimal foothold in traditional K-12 curricula, at least in the United States. Initially part of home economics, crafting involved sewing, embroidery, and knitting for girls and the wood and metal shop and for boys. Many of these crafts activities have been relegated to vocational schooling. Nowadays, the crafts are experiencing a renaissance outside of school in DIY communities as well as in fabrication labs that allow for personal manufacturing (Frauenfelder, 2010; Gauntlett, 2011). On the other hand, learning about circuitry, and by extension about electricity, is an established part of the K-12 science curriculum. Numerous studies have documented students‟ difficulties in understanding the concept of circuits (Engelhardt & Beichner, 2004; Perkins & Grotzer, 2005). Likewise, learning of programming has established the multiple challenges that beginning learners face in writing and implementing code (Guzdial, 2004; Palumbo, 1990; Soloway & Spohrer, 1989). While it is possible to approach the learning with e-textiles from these respective disciplinary foundations, it is also clear that designing and making e-textile artifacts is more than learning discretely about crafts, circuitry or code. In fact, it is the intersection of these three disciplines that makes learning with e-textiles challenging while also providing an authentic context. In particular we examine some of the ways in which the materiality of e-textiles makes aspects of circuitry and programming visible to learners. Previous research has focused on students‟ alternative conceptions and how instructional activities can progressively enrich students‟ models of science (Perkins & Grotzer, 2005). Our research examines in more detail the situated nature of students‟ understanding of electrical circuits and program control flow that overlay and support each other. For instance due to the unusual nature of materials such as the conductive thread, students are challenged to develop understandings of conductivity that draw on some of their informal knowledge about circuits and electricity. Likewise, the less tangible aspect of control flow in programming overlays the coding of sensors and actuators that are connected via circuits. In the production of their e-textile designs, students negotiate and link representations of graphical, textual, symbolic, and material nature and create linkages by literally threading and tying them together. These connections are reminiscent of Blikstein‟s (2010) bifocal modeling that provides explicit links between physical artifacts and their computational counterparts that allow students to design and experiment with both representations. However, in the context of e-textile design, students are the ones constructing the links. Our analyses are a first effort to build a framework to captures the processes involved.

Methods
Participants were 35 freshmen, 14-15 years old, from a public magnet high school focused on science and technology in a large urban school district. The students' self-identified demographic make up was 23% African American, 29% Caucasian, 14% Asian, and 17% mixed race/ethnicity. Five students chose not to identify their race/ethnicity in survey responses. Just under half of the participants were girls (n=15). Overall, the demographics of the workshops reflected the diversity found in the school and district at large. In spite of students‟ interests in science and technology, only a few of our participants had prior programming experience

© ISLS

189

ICLS2012

Volume 1: Full Papers

and none had ever worked with electronic textiles when they elected to participate in the e-textiles workshops. The workshops were conducted as part of a partnership with a local science museum where students spent one afternoon a week at the museum learning about a topic of their choosing. For this study, we organized a series of three workshops, ranging in length from 4 to 6 weeks. Workshop sessions were held once a week and lasted for two hours. During this time period, students learned how to design and create their own e-textiles projects, beginning with aesthetic drawings, followed by circuit schematics, sewing/crafting of designs, and programming. The first workshop had all novices and therefore the concepts built more slowly upon one another; most students finished very simple e-textiles projects with 1-2 LEDs. In contrast, the second workshop had roughly half returning students and all but one of the students in the third workshop had prior experience with etextiles. As students gained knowledge, their projects tended to increase in complexity and they experienced greater success. We also became better instructors and began to set tangible goals for each workshop session, like “sew two LEDs and program them,” so the construction process became more recursive, with students identifying errors as they went rather than just at the end when their completed project did not function as they expected it would.

Materials
All students had access to the LilyPad™ Arduino construction kit (see Figure 1) that enables novice engineers/designers to embed electronic hardware into textiles (Buechley & Eisenberg, 2008; Buechley, Eisenberg, Catchen & Crockett, 2008). In addition, we provided various caps, t-shirts, gloves, cotton bags, fabric and felt pieces on which students could sew their projects in addition clothes or objects student brought themselves. The LilyPad kit is a set of sewable electronic components, including a programmable microcontroller and an assortment of sensors and actuators that allows users to build their own soft wearable computers. Users sew LilyPad modules together with conductive thread instead of traditional tools like insulated wire and soldering irons. To define the behaviors of the project, users employ the popular Arduino or ModKit development environments, enabling them to program the LilyPad microcontroller to manage sensor and output modules (like LEDs) employed in their designs.
The LilyPad™ Arduino kit and components. The microcontroller is in the center, and the other components, clockwise from the top, are: accelerometer, light sensor, tri-color LED, power supply (requires a AAA battery), speaker, and vibrating motor. An FTDI board and USB cable are also part of the introductory kit.

Figure 1. LilyPad™ Arduino.

Data Collection and Analyses
The data collection of all workshops included video recordings (focused on groups working at individual tables), field notes by two independent researchers, photographs of students‟ projects, and final interviews with students. The writings of field notes and interviews were refined in each successive workshop to increasingly focus on the process of design. For instance, in the final workshop two researchers wrote field notes focusing on the individual design decisions made by students with accompanying photographs of their designs at different stages of production (photographs were taken roughly every 60 minutes). This allowed us deeper insight into the ideas, challenges, and decisions that arose in the design process than was available on videos or in interviews about students‟ final projects. We also conducted casual ethnographic interviews with students every week in the final workshop, asking questions like, “What are your goals for this workshop?,” “What parts of e-textiles do you still need help understanding?,” and the more open-ended “tell me about your project.” This allowed us to cull reflections from students in process rather than just at the end. Thus, like students‟ projects, the final workshop was the richest in details about students‟ design decisions and processes. For analysis of workshop activities and case studies, we conducted a two-step open coding (Charmaz, 2000) of all our data (field notes, logged videos and interviews). To wit, we first began by reading a third of the data and listing some of the challenges of learning to design with e-textiles. We then created an initial coding scheme of the learning challenges, categorized by the overlap of crafting, circuitry, and coding, coded one section of the data together to build consensus, and proceeded to independently code two workshops. We then refined our coding scheme to reflect insights from this analysis of the data and re-coded all workshops. We created a thesaurus of codes with definitions and examples and indexed (i.e., counted) all codes, listing them by date with a 1-sentence summary. This allowed us to see which learning challenges were most prominent in

© ISLS

190

ICLS2012

Volume 1: Full Papers

individual workshops or across workshops. Below we discuss the most prevalent codes found across all workshops in regard to students‟ learning in e-textiles.

Results
In analyzing where students faced the greatest challenges in learning how to design e-textiles, we found some of the most significant learning occurred where the material and conceptual aspects of the projects overlapped. First we explain the ways that the physical aspects of crafting illuminated the inner workings of circuits. Then we describe how the materially embodied circuitry shed light on coding. Both of these findings stretch across the breadth of our data from all three workshops.

How Sewing Reveals the Intricacies of Circuits
When designing and creating their own electronic textiles projects, novices must not only sew (a new experience in and of itself for many students) but do so in a way that takes into account the conductivity of the thread they are working with and the fact that what they are sewing must ultimately result in functional electronic circuitry. In our analysis of students‟ design of e-textiles, we found three areas that were particularly non-intuitive: sewing for conductivity, tying knots to end a conductive line, and tidying up loose threads that created accidental short circuits. Each of these areas made qualities of circuits visible to the students in ways that they learned about basic aspects of circuitry.

Sewing for Conductivity
At the most elementary level, students learn that they must sew an electronic component to ensure conductivity as well as stability. If an electronic component like an LED is sewn too loosely or the stitches are too large, the connection between the component and the conductive thread may be lost when the textile bends. Unlike using alligator clips or soldering wires, where electricity is easily conducted on a slight touch between metal and metal, when attaching electrical parts to something more malleable like fabric, one must work (sew) to make sure there is a consistent physical connection. This is the most basic element of conductive sewing but it is not intuitive to novices. Nathaniel described this as a very frustrating part of making his project: RESEARCHER: NATHANIEL: RESEARCHER: NATHANIEL: What was the hardest part about the project? The sewing. The sewing. Why? Like, I know I went through it once, and I had like finished the thing. But it got loose. So I had to unthread it all, go back again. RESEARCHER: Because you hadn't sewn it tight enough? NATHANIEL: Yeah. (Dec. 15, 2010, Video)

Though Nathaniel thought his project was sewn adequately, when he tried to turn his lights on, the connection between the light and the conductive thread was too loose and he had to re-do the project. This happened for many students. We found that despite our instructions from the very beginning to sew each light with three stitches, most students did not do this until they understood the reason for this was to create a conductive connection. Thus students‟ understanding (or lack thereof) of this introductory concept of electrical circuits was made visible in their sewing for conductivity.

The Importance of Tying Knots
Whereas sewing electrical components on fabric made visible the need for conductive connections, tying knots revealed the way electricity takes the path of least resistance in a circuit. For instance, in order to direct electricity to flow through an LED one must consciously tie a knot to end the positive connection on one side of the LED, cut the thread, and start a completely new line of thread on the negative side. This separation between the sides is less visible when one uses alligator clips to connect an LED to a power source because the alligator clips have to be connected at each end anyway. Thus, even after using alligator clips to make and test circuits, in moving to thread many students made the common mistake of using one uncut thread to sew through the sides of an LED, in effect bypassing the LED in the circuit. Mallory described this mistake below. MALLORY: I just now learned like, when I sew here like. When I first, I was sewing through the positive part of [the LED]… [then] through the negative side and then so, it was kind of hard for me cause I was doin' it incorrectly? But now, I know not to. RESEARCHER: Do you understand why that doesn't work? (March 2, 2010, Video) MALLORY: Yeah… Well it's gonna go over it [the LED].

© ISLS

191

ICLS2012

Volume 1: Full Papers

Here Mallory described how she had sewn through both ends of an LED with the same thread, not realizing that she had to cut and separate the sides in order for electricity to flow through the LED rather than thread. As students began to understand conductive pathways, they came up with the phrase “sew like alligator clips” to formalize the concept. Because of the non-trivial effort it took to tie a knot at the end of each positive and negative sewn line, especially for novices sewers, students were anxious to find ways to minimize the number of knots they needed to tie. They were keen to simplify their circuits, which could be done by using one conductive line to connect the negative (or positive) sides of multiple LEDs without the need to tie knots at every end (for an example see the diagram on the right of Figure 3, knots are shown in stars). So, it is okay to sew continuously through the negative ends of multiple LEDs but it is not okay to sew through the negative and positive ends of a single LED without tying knots and cutting the thread to direct electricity through the LED. To our surprise, we found the best way to convey this concept was to describe it in terms of how many knots they needed to tie. We showed this by comparing multiple electric diagrams and having students re-draw the electrical connections for the “fewest knots.” Silas reported proudly about learning to rearrange circuits by connecting multiple sewn lines (i.e., “wires”): My learning experience with the lights has definitely changed--that's one thing. I can like, now, now that I know, um, the electric, um, what is that called? What am I trying to say? Um, how to hook up one wire [thread] from one end to another with positives and negatives, how to connect multiple wires [threads], and create multiple lights on „em. (March, 2011, Video) Many students like Silas said their favorite learning experience was figuring out what could be sewn together and what had to be sewn individually. This learning changed their designs substantially, with almost all students re-orienting their circuits so that either the positive or negative sides of their LEDs could be connected in one line, allowing for many more LEDs to be sewn on with fewer knots and less sewing effort.

Loose Threads Mean Short Circuits
One final circuitry concept that the element of crafting made visible was short circuits. In conductive sewing, crossed threads (which work like non-insulated wires) amount to crossed negative and positive lines, resulting in short circuits. This is non-intuitive from a crafting perspective where we are taught that “the back doesn‟t matter” because no one will see it when the project is finished. Yet in e-textiles, loose threads or too much sewing in a small space can cause the project to fail. Amari and Marcela learned to identify this problem in a project where they were trying to figure out why the LEDs would not turn on: AMARI: ‟Kay, I think I found the problem with this. MARCELA: Okay, what is it? AMARI: Okay, I was looking at the back of it, and I think the problem is that if you look at it closely, these [threads] are all overlapping each other and that could affect the conductivity. ((Pointing to overlapping threads on the back)) MARCELA: "Oh yeah, 'cause negative can't overlap the positive. (May 20, 2011, Video) When Amari pointed to the back of the project where many long ends of threads were touching, Marcela correctly identified this as a problem with short circuits where the negative and positive lines were overlapping. The girls solved the problem by trimming the threads and tying the knots closer to the surface of the fabric where they couldn‟t get loose and touch each other. All students struggled with the issue of short circuits, coming to terms with the idea that the threads conducted electricity even if they were just the loose ends of knots. Yet though it took time to learn this idea, by the end of each workshop, loose threads became one of the first things students looked to when trying to make their LEDs turn on.

How Embodied Circuitry Reveals the Intricacies of Coding
Not only did the materiality of crafting circuits bring out some conceptual issues about circuitry and electronics, but the crafted electronics revealed some concepts about coding and vice versa. At the most basic level, students learned that connecting LEDs to the LilyPad microcomputer was not the same as connecting them to a battery. First, the LilyPad had to be programmed. Kyra demonstrated her partial understanding of this idea when, having finished sewing her project with two LEDs connected to pins 8, 10, and a negative ground, she programmed pin 8 and then wondered why only one LED flashed on. A researcher pointed out to her that she had not yet programmed pin 10 – perhaps an obvious idea to people with programming experience but a novel idea to beginners. A week later Kyra summarized this episode as one of her most important learning experiences when she said “It helped me understand that things – that most things – can‟t work without a code” (Dec. 15, 2010, Video). Below we describe other ways that students learned about coding through circuitry, and circuitry

© ISLS

192

ICLS2012

Volume 1: Full Papers

through coding. We note that in each case, it was not until designs were actually material (sewn circuits) that the relationship between the two became clear to the students.

Why Parallel Circuitry Might Not Be the Best for Blinking Patterns
One challenge that students faced was to learn the affordances of different kinds of circuits. As we mentioned above, students had a vested interest in tying as few knots as possible and for this reason many were interested in making parallel circuits, with one continuous negative line connecting the negative pins of several LEDs and one continuous positive line connecting the positive pins of the LEDs. However, when it came to coding their circuits, they began to realize that parallel circuits did not afford them any opportunities to have their LEDs blink at different times – the shared positive and negative lines meant that all of the lights worked together. For instance, when Aaliyah was drawing out the design of her five-point star with an LED on each point, she decided that some of the LEDs should be sewn in parallel (see left diagram in Figure 2). However, two weeks later after her project was partially sewn and she had made some of the LEDs turn on, she changed her mind as she realized that she would not be able to blink each light independently (see right diagram of Figure 3 for final design). Her old design would not allow her to turn each LED on one by one, rather they would go on in groups that she could never change. This was a common dilemma students faced as they began to understand the relationship between the programming effects they desired and the implications of the design/crafting of their circuitry.

Figure 2: Aaliyah‟s original design (left) with two parallel circuits & one additional circuit and final design (right). with five independent circuits with a shared negative line. Negative lines are solid, positive lines are dashed, knots are shown with small stars (*).

On Hardwired and Programmable Ports
Another challenge that students faced at the intersection of circuitry and programming was learning about the affordances of the LilyPad. Two pins on the LilyPad (the + and the – pins) cannot be programmed and have constant polarization. The remaining (numbered or lettered) pins can be programmed to be either positive or negative. The first realization that students came to was the difference between these two types of pins. One type (the positive and the negative) was unalterable, while the other was completely programmable. Marcela described this as one of her biggest realizations: “I didn‟t get in the beginning why, you know how you connected to negative and then positive? I didn‟t get what does the number represent... And I didn‟t know you could make it anything you want.” (March 2, 2011, Video). Once students realized that they could program any numbered or lettered pin on the LilyPad to be either positive or negative, they had a great deal of freedom to design their circuits. Still, the implications that some pins were hardwired while others were programmable did not come easily to everyone. Below we share how Amari became conscious of this relationship between circuitry design and programming: Amari had finished putting her 5 LEDs on the points of her five point star with the negative ends of the LEDs connected tog