Multimedia at Work

Editor: Tiziana Catarci University of Rome

Does Hypermedia Really Work for Tutoring Children?

Maria Francesca Costabile, Teresa Roselli, Rosa Lanzilotti, and Paola Plantamura University of Bari, Italy
erbal modes of presentation have dominated the way we provide explanations to one another, and verbal learning has dominated education. Similarly, educational research has focused on verbal learning. Nevertheless, teaching is rarely a unimodal activity. Skilled teachers tend to use all available modalities in the communicative context to convey information that can help pupils learn. Instructors often speak with a particular intonation, move their hands and body, draw images on a blackboard, show videos, and so on. The advent of multimedia technology has produced an explosion of research on the use of computers in education. Indeed, the computer is a rich communication tool that instructors can use for effectively illustrating ideas. However, compared to human communication, computers lack several features needed for satisfactory communication. They dont have analogies to eyes, ears, voice, and hands that humans use for acquiring information; neither can they create different information representations. Graphic, video, animation, and sound use in multimedia systems attempts to overcome these limitations to develop systems that can better communicate and cooperate with humans. Several recently developed hypermedia environmentsboth online (on the World Wide Web) or ofine (on CD-ROM)promote knowledge acquisition on a wide range of topics and attempt to teach problem-solving abilities in users. However, the basic question of whether hypermedia learning environments actually promote learning is still debatable, as the Previous Research sidebar (on page 69) indicates.1 Our work here refers to a hypermedia designed for a particular category of users, namely children in a primary school. The hypermedia offers a tutoring component to teach basic logic concepts to children. We call this hypermedia

Logiocando, a name obtained by merging two Italian words (Logica Giocando) and meaning playing with logic. To determine whether this tutoring hypermedia really works (that is, to check its instructional validity), we performed two experiments that evaluated the pedagogical efficiency of the hypermedia in comparison to traditional classroom instruction.

Antonella De Angeli NCR Self-Service Technology & Research, UK

Logiocando hypermedia
The idea of building Logiocando came to us from primary school teachers who wanted a software product to support teaching logic. Because of the special category of users, we designed and developed Logiocando following a strict usercentered methodology. Teachers actively collaborated in the hypermedias design and evaluation, providing suggestions and materials, in accordance with the objectives of the primary school programs as stated by the Italian Educational Council. Primary school students took part in Logiocandos evaluation.2 We organized the hypermedia into four units: sets, set operations, logic operators, and diagrams. Each unit focuses on a specific concept in accordance with its title and is divided into three sections: explanations, logic games, and tests. The explanations section illustrates concepts related to a unit using text, images, and/or sounds. Figure 1 (next page) presents the concept of set union in the set operations unit. The pupil reads concepts related to the topic and might also look at an example illustrating that concept by clicking on the word esempio [example in English], visible at the bottom right of the screen in Figure 1. The logic games section, shown in Figure 2, includes a set of exercises of increasing complexity related to the concepts illustrated in the explanations section of the unit.

1070-986X/03/$17.00 2003 IEEE

Published by the IEEE Computer Society

65

Authorized licensed use limited to: Tec de Monterrey. Downloaded on May 01,2010 at 20:33:12 UTC from IEEE Xplore. Restrictions apply.

Multimedia at Work

Figure 1. Page of the explanations section in the set operations unit.

tions by considering the taxonomies of the educational objectives dened in Bloom et al.,3 while we established the number of games and questions by applying the formula reported in Priore.4 Logiocando permits two kinds of navigation: the guided path and the free one. If the pupil chooses the guided path, the hypermedia provides suggestions on how and what to study. In this way, the tutoring component is active and Logiocandos instructional units prepare students for subsequent units. For example, pupils can navigate a specific unit only when they achieve a preset score in the preceding one. The guided path is activated if a pupil clicks on the image of a virtual teacher on the hypermedia homepage. A first time visit by the pupil to the hypermedia site activates the link of the first unit, sets, in the general index page. After clicking on the link to a unit, a page with three buttons indicating that units three sections appears. Even in the guided path, pupils can decide on which section to visit. A sections general index lists the titles of all pages in the section. By clicking on a title, pupils select a specic page. They can navigate forward or backward via the arrows visible at the bottom right corner of each page, as Figures 1 through 3 show. If pupils choose the free path option by clicking on a bear on the hypermedia homepage, they can freely visit any unit. Teachers suggest this free navigation to pupils already experienced with Logiocando.

Evaluation of Logiocando
To assess its educational efficiency, we compared Logiocando to the work of a teacher in a traditional classroom setting. We describe elsewhere the details of the two experimental studies summarized here.2 This work aimed at answering the following questions:
Figure 2. Example of an exercise in a logic games section.

Can children improve their knowledge of logic using Logiocando? The tests section, illustrated in Figure 3, lets pupils assess their knowledge. It includes 10 questions of various types. For example, in Figure 3 the pupil is asked to classify the four geometric figures at the top of the screen, namely two hexagons and two circles of different colors. Another type of exercise asks the pupil if a statement is true or false. At the end of this section, the system sums the score for each question and computes the pupils knowledge level. We designed the logic games and tests sec Can Logiocando perform as effectively as a teachers lesson for revising logic concepts? As a basic experimental hypothesis, we predicted that Logiocando had the same educational potential as that of a human teacher. Indeed, Logiocando has peculiar advantages over traditional classroom instruction that could compensate for the intrinsic benefits of skilled face-to-face teaching. Logiocandos strengths

IEEE MultiMedia 66

Authorized licensed use limited to: Tec de Monterrey. Downloaded on May 01,2010 at 20:33:12 UTC from IEEE Xplore. Restrictions apply.

mainly rely on its interactivity and exibility, letting pupils organize their own learning paths in a nonlinear, yet controlled way. Moreover, as with every computer-based learning tool, Logiocando relies on a one-to-one relationship, which is seldom possible in a traditional classroom. Hence, pupils can take advantage of Logiocando to accommodate their personal requirements. To test this hypothesis, we ran two controlled user evaluations. These each apply the same experimental paradigm, but present a fundamental motivational difference in how teachers introduce the system to children. The first controlled experiment involved 54 children attending the fourth grade class of a primary school in Bari, Italy. Initially, we administered a pretest to 80 children to evaluate their knowledge of logic. We chose the 54 children whose pretest score was below seven to participate in the experiment. The test scale ranged from zero to 10 (so those scoring below seven didnt have a satisfactory knowledge of logic). We randomly divided them into two groups, one for each teaching method. The teacher-assisted (TA) control group reviewed logic concepts in a classroom, attending two lessons given by the teacher. Pupils assigned to the computer-assisted (CA) experimental group reviewed the same concepts individually using Logiocando. The experiment consisted of two sessions: theory revision and practice. During the theory revision session, children assigned to the CA condition individually reviewed the hypermedias set operations and diagram units. Interacting with Logiocando, pupils reviewed basic operations and the graphical representations of sets. In the TA group, teachers introduced the same instructional material. During the practice sessions, pupils performed a set of exercises based on the presented concepts; the CA group used Logiocando and the TA group a pen and paper. One week after the experiment, we administered a post-test to all participating pupils, as part of their in-class work. They answered a set of questions designed by the teachers to match the pretest ones. Figure 4 (next page) illustrates the average values of the pre- and post-test scores in the two experimental groups. The experiment demonstrated that Logiocando could facilitate learning in a sample of 9- to 10-year-old children. However, contrary to our expectations, we didnt find the hypermedia as effective as the human teacher. Despite this disappointing result, the study was useful because it

Figure 3. Example of a question in a tests section.

let us observe the behavior of children in the two learning settings. The fundamental difference between settings appeared to be the pupils motivation. Children didnt take the work with Logiocando seriously, which they considered a game rather than a learning instrument. Therefore, we ran another experiment in which we carefully controlled childrens attitudes and motivations, making them as responsible as possible for the work they performed. The second experiment followed the same procedure as the first. The only difference was that this time, we devoted great care in motivating pupils to get the most out of their interactive experience with Logiocando. We formally presented the system, casting it as an important learning tool for class activity. Moreover, we clearly explained to the children that the teachers would carefully evaluate their performance as part of their class work. Finally, in contrast to the rst experiment, the teachers monitored the childrens activity throughout the experiment, refraining however from interfering with it. We used the same procedure as the rst experiment to select 40 children for this study. We assigned 20 pupils to the TA group and 20 to the CA group. The behavior of the children who used Logiocando was completely different than what we observed during the first experiment. The pupils were disciplined and worked seriously, precisely executing the assigned tasks. This

AprilJune 2003 67

Authorized licensed use limited to: Tec de Monterrey. Downloaded on May 01,2010 at 20:33:12 UTC from IEEE Xplore. Restrictions apply.

Multimedia at Work
10 9 8 7 6 5 4 3 2 1 0

Average score

Figure 4. Average preand post-test in the rst experiment.

Pretest Post-test

Computer assisted Teacher assisted Teaching method

Average score

Figure 5. Average preand post-test scores in the second experiment.

10 9 8 7 6 5 4 3 2 1 0

Pretest Post-test

Computer assisted

Teacher assisted

Teaching method

behavioral difference led to different results, shown in Figure 5, from the ones obtained in the first experiment. All the children significantly enhanced their knowledge during the experiment, independent of the teaching method. This improvement is equivalent to the one achieved by pupils attending the teacher lesson in experiment one. But once the students were properly motivated, Logiocando reached the same performance as the teacher, letting children improve their knowledge of logic.

teacher, but it can support the learning and reviewing process and deepen knowledge on a topic. Both of our experiments stressed the high acceptability of Logiocando as a learning tool in primary education. Pupils were excited about the system. However, this excitement led to different results according to pupil motivation, which should be carefully controlled when designing systems for pupil instruction. The process of learning is complex due to the myriad of interacting factors involved. Learning isnt an isolated and individual activity but occurs in a social and cultural context. Technology is part of that environmental context. All pupils in our experiments had prior experience with computers but that experience mainly involved video games; these ludic instruments are normally perceived differently from the rigorous activity required by formal learning. We need more research to learn how to embed this rigor in educational software without decreasing the software appeal. However, our experience certainly stresses the need for carefully controlling motivational factors in evaluating educational software, especially that designed for children. Intentional learning requires a motivated individual.5 Our experiments demonstrate that motivation affects learning through hypermedia and can thus help account for the current lack of agreement on the educational value of hypermedia learning environments.1,6 MM

Acknowledgment
This work has been partially supported by the Italian Ministry of University and Research under grants Murst 60% and Con 2000.

References
1. I.I. Suni and S.M. Ross, Adaptive Computer Control in a Hypermedia Materials Science Document, J. Educational Multimedia and Hypermedia, vol. 6, no. 3/4, 1997, pp. 383-393. 2. M.F. Costabile et al., Evaluating the Educational Impact of a Tutoring Hypermedia for Children, J. Information Technology in Childhood Educational Annual, submitted for publication. 3. B.S. Bloom et al., Tassonomie degli obiettivi educativi [Taxonomy of Educational Objectives], Giunti & Lisciani Editori, 1986 (in Italian). 4. F. Priore, Modelli, strumenti e misure della didattica contemporanea [Models, Instruments and Measures of Contemporary Didactics], Mursia, 1995 (in Italian).

Conclusions
By enhancing pupils motivation, the group using hypermedia in our second experiment achieved excellent results equaling that of the pupils who were taught by the teacher. Several factors can explain this result. Interacting with instructional hypermedia, children can personalize their own learning path and follow their own learning rhythms. In this way, children who need a specific period of time for learning a concept can navigate in the hypermedia for as much time as they need. Considering the result of the second experiment, we arent stating that Logiocando can act as a substitute for a human

IEEE MultiMedia 68

Authorized licensed use limited to: Tec de Monterrey. Downloaded on May 01,2010 at 20:33:12 UTC from IEEE Xplore. Restrictions apply.

Previous Research
There are conflicting claims about learning through computer-mediated environments. Bagui refers to several studies showing that computer-based hypermedia can help people learn a greater amount of information more quickly than classroom instruction.1 This author states that multimedia allows interactivity with the computer, multimedia is flexible, multimedia has a rich content, multimedia has motivational effects, and multimedia allows better structured instruction. These claims have validity if we assume a parallelism between multimedia and the natural way people learn, as the information processing theory explains. Other researchers have similar opinions.2 They claim that the hypertext structure reflects a model of learning based on the students semantic memory model, and hypermedia provides interactive mechanisms that let learners manage, manipulate, and organize their lessons. These interactive activities encourage students to play an active role in the learning process, thus supporting the intentional construction of meaning at the basis of learning. Soo and Ngeow showed that multimedia computer-assisted instruction is significantly more effective than teacher-taught classes in increasing the English prociency of students (who were on average 21 years of age).3 Cybulsky and Linden showed that the multimedia-assisted teaching environment (MATE) is a valid complement to traditional teaching based on lectures, tutorials, and practical sessions.4 Other research demonstrated that students favored using a computerbased system for learning the expert practices of engineering.5 Students preferred the computer system to books because it was associated with something on their desks and was immediately accessible, personal, and not rationed. Conversely, they associated paper-based material with something that was remotely stored, not integrated, and not shared, thus not readily accessible. Other authors are more skeptical about hypermedia learning systems. For example, Hegarty et al. reported no differences in learning levels between a group of students using a hypermedia manual for instruction in mechanical systems and a control group using a traditional printed manual.6 Articial intelligence techniques build more efcient hypermedia by guiding the learner through personalized learning paths.7 The pedagogical value of this solution has been demonstrated elsewhere.8 Students using a hypermedia system that incorporated a rule-based tutorial component for learning logic programming acquired more knowledge than a control group using the same system without the tutorial component. In this case, participants were university undergraduate studentsthat is, a special category of users because of their high motivation.

References
1. S. Bagui, Reasons for Increased Learning Using Multimedia, J. Educational Multimedia and Hypermedia, vol. 7, no. 1, 1998, pp. 3-18. 2. P. Aedo et al., Assessing the Utility of an Interactive Electronic Book for Learning the Pascal Programming Language, IEEE Trans. Education, vol. 43, no. 3, Aug. 2000, pp. 403-413. 3. K. Soo, and Y. Ngeow, Effective English as a Second Language (ESL) Instruction with Interactive Multimedia: The MCALL Project, J. Educational Multimedia and Hypermedia, vol. 7, no. 1, 1998, pp. 71-89. 4. J.L. Cybulsky and T. Linden, Learning Systems Design with UML and Patterns, IEEE Trans. Education, vol. 43, no. 4, Nov. 2000, pp. 372-376. 5. J.S. Busby et al., An Evaluation of an Instructional System for Engineering Task Estimation, IEEE Trans. Education, vol. 43, no. 1, Feb. 2000, pp. 30-35. 6. M. Hegarty et al., Multimedia Instruction: Lessons from Evaluation of a Theory-Based Design, J. Educational Multimedia and Hypermedia, vol. 8, no. 2, 1999, pp. 119-150. 7. R.C. Schank, M. Korcuska, and M. Jona, Multimedia Application for Education and Training: Revolution or Red Herring?, ACM Computing Surveys, vol. 27, no. 4, Dec. 1995, pp. 633-635. 8. T. Roselli, Articial Intelligence Can Improve Hypermedia Instructional Technologies for Learning, ACM Computing Surveys, vol. 27, no. 4, Dec. 1995, pp. 624-626.

5. E. Kintsch, M. Franzke, and W. Kintsch, Principles of Learning in Multimedia Educational Systems, tech. report 96-01, Inst. Cognitive Science, Univ. of Colo., Boulder, 1996. 6. A. Lanza and T. Roselli, Effects of the Hypertextual Approach versus the Structured Approach on Students Achievement, J. Computer-Based Instruction, vol. 18, no. 2, 1991, pp. 48-50.

Readers may contact Maria Francesca Costabile at the Dept. of Computer Science, Univ. of Bari, Via Orabona 4, 70125 Bari, Italy; costabile@di.uniba.it.

Readers may contact editor Tiziana Catarci at the Dept. of Information Systems, Univ. of Rome La Sapienza, Via Salara 113, 00198 Rome, Italy; catarci@dis.uniroma1.it.

AprilJune 2003 69

Authorized licensed use limited to: Tec de Monterrey. Downloaded on May 01,2010 at 20:33:12 UTC from IEEE Xplore. Restrictions apply.