Intelligent and Pervasive Multimedia Systems

based on user group calibration (see http:// www.ets.org/gre/);

Interactive Multimedia for Adaptive Online Education

Irene Cheng, Anup Basu, and Randy Goebel University of Alberta
The Computer Reinforced Online Multimedia Education (Crome) system gives students the opportunity to complete a curriculum adjusted to their knowledge level using computer adaptive testing. nteractive multimedia content can help improve learning performance by enhancing user satisfaction and engagement. Multimedia content also can help improve concept representation, which is not possible in conventional multiplechoice and fill-in-the-blank formats. In this article, we present a broader view of multimedia education, focusing on future applications. Our vision is to provide publicly accessible education anywhere, at anytime, and to anyone. To realize this vision, we propose the Computer Reinforced Online Multimedia Education (Crome) framework, which integrates the main components of education, including learning, teaching, and testing, as well as adaptive testing and student modeling. We conducted our research in collaboration with industry and in response to the industrial need and public support for online multimedia education. By presenting our research and development achieved in the last two years, we hope to share our experience with other researchers and bring awareness to the muchneeded outcomes that have not received adequate attention. Our current implementation provides the groundwork upon which more robust, effective, and far-reaching multimedia architectures can be developed. The unique contributions of our system include the following:

adaptive testing using innovative item formats versus using traditional multiple choice items (such as Moodles multiple choice and GMAT);

computer-assisted, automatic, question authoring, rather than manual creation of different questions;1

custom-designed modules for tracking precise concepts that are lacking in recommendation systems;2 and

testing beyond subject knowledge, unlike

other e-learning systems. Multimedia education proves to be more effective than traditional printed material because the dynamic and interactive settings can inspire students to learn and remain interested.3 Furthermore, abstract concepts are expressed more effectively using audio and visual effects, animation, and graphics, including 3D navigation. Humancomputer interaction provides instantaneous feedback to students and helps them feel engaged. Because highspeed computers and networks are commonplace in modern societies, online delivery has become an efficient tool to overcome geographic diversity. The use of a common language in cyberspace can narrow the gap in language and cultural barriers. For this reason, our research and development focus is on delivering online multimedia education.

The mechanism of education

Historically, education was a commodity affordable only by the rich. This would still be the case for high school studies if some governments stopped offering free tuition. However, inequality of opportunities continues to exist between those who can afford private schools and tutors and those who cannot. Those with access to additional guidance outside classrooms have advantages over those who have to rely on public school resources. To hire a qualified private tutor in North America, the regular hourly rate on curriculum-specific subjects currently ranges from $40 to $60. In some municipalities, the difference in expenses between attending a private school and a public school can be thousands of dollars per month.

automatic difficulty-level estimation for

selected subjects, compared to estimates

16

1070-986X/09/$25.00 c 2009 IEEE

Published by the IEEE Computer Society

The demand for affordable private tutors has created a trend in online tutoring, which allows students to reserve specific time slots in advance and, with selected tutors, get help with solving curriculum related questions. The students pay a set monthly fee for a certain number of tutoring hours. Because the service is provided online, the tutors can be remotely located. For example, some service providers take advantage of the lower operating cost overseas and hire tutors in countries like India to answer questions posted online by students in North America. In this kind of outsourcing service, the issues to address include time difference, curriculum diversity, and the possible language and cultural barriers. Moreover, this approach requires little to no multimedia content because the focus is on the conversation between the tutor and the student either in a chat room or through emails. A more widely adopted and effective approach to promote global education is through the use of digital multimedia and e-learning. The format of e-learning can be as general as an open-access Web site such as those supported by the BBC (see http://www.bbc. co.uk/learning/onlinecourses) and Discovery Education (see http://school.discovery.com/ lessonplans), or curriculum-specific programs provided by universities and institutions for high-stakes exams, diplomas, undergraduate testing, and graduate exams. One example is the distance-education programs provided by the University of Phoenix, La Salle, and Athabasca University (see http://www.athabascau.ca). The European Union supported educational research program, New Perspective for Learning, has helped develop and integrate many components for e-learning as well as create learning societies under frameworks FP4 and FP5 (see http://www.pjb.co.uk/npl/index.htm). Instead of making educational material accessible online, tutorial DVDs containing multimedia content are also available. Clearly, online applications have gained broad acceptance in the general public, especially after the launch of the Semantic Web, the social Web, and online applications such as Second Life, YouTube, and Facebook. The increasing acceptance of online applications and products is creating a unique opportunity for online education to make a global impact. There is an abundance of literature in the

educational multimedia area covering a variety of topics from games to learner modeling. Examples of such applications include the following:

computer games, played by an individual or

collaboratively, to teach concepts;4-6

visual effects to assist learner understanding;7,8

artificial intelligence techniques to recommend research papers to learners;9

active learner modeling10 to derive information on a peer group learning together, possibly in a collaborative environment;

telementoring carried out through collaborative agents;11

student performance explored on a multimedia exam program;12 and

open exams set up for MBA/business school

admission (see http://www.syvum.com/ gmat/cat.html). Most of the literature addresses learning applications of multimedia. The use of multimedia in testing has been relatively limited, compared to its use in learning and training. For those systems supporting testing, most of the test questions still adhere to traditional stylesfor example, true or false, multiple choice, and fill in the blankand are not as novel as those proposed in our framework. In addition, our design integrates different aspects of education and is not restricted to a single target component.

Crome examples
To illustrate the pervasive power of using multimedia content online and its impact on improving the quality of education, we will show some examples from our Crome implementation. We refer to a curriculum-specific question as an item following the terminology adopted in the field of educational psychology. In the following, each example item can be used either in learning or testing, depending on the log-on status: practice or testing. Tutorials, hints, and correct answers are available to the students in a practice session.

JanuaryMarch 2009 17

(a)

well as navigate (translate, rotate, and zoom) through the atoms and molecules in 3D space. Atomic and molecular symbols in the formula NaOH HCl ! NaCl H2O are replaced by 3D objects. We can use a similar format to represent biology items. 3D objects have long been used in the entertainment industry and research, but their use in educational applications is still limited. We believe that it will not be long before telesurgery and virtual reality tools become more accessible in education, allowing students to practice dissecting a virtual frog instead of using a real animal, and understanding the human body more effectively by manipulating the different organs in virtual space. Drag and drop items A commonly used interaction design is dragand-drop, which lets students drag text or graphics to the appropriate locations on the screen. Figure 2a shows an example of a geography item requiring students to drag the correct labels onto the map. Drag-and-drop items can be used in other subjects as well and can be either 2D or 3D. Figures 2b, 2c, and 2d show three more examples.

(b)
Figure 1. Chemistry items: (a) item showing a 3D molecule used to test atomic and molecular structure in chemistry, and (b) item that requires a description of the molecular structural changes in a chemical reaction.

3D items In early development, multimedia referred mainly to digital images and videos. 3D objects and scenes became an important component only later. One application of 3D content is in chemistry, which can be categorized into three levels: macroscopic, symbolic, and atomic.13 A chemical phenomenon at the macroscopic level is normally observed in a laboratory and can be shown in educational videos in addition to being shown in the scheduled lab session. The other two levels can be exposed in a more active and interesting manner, instead of writing a formula or drawing a molecule on paper. Molecular structures and the change of structures during a chemical reaction, such as breaking bonds inside a molecule, are best presented in a 3D context. Figure 1 shows screen shots of two chemistry items. Students can use the mouse buttons to pick the required atoms from the periodic table and drag them onto the canvas. They can then create the appropriate bonds between atoms, as

Logical-mathematical items An interactive environment provides more entertainment, and thus more motivation, to students when using and analyzing numbers. In the math item shown in Figure 3a, a student needs to distribute the numbered objects into multiple baskets so that the sum in each basket is the same. Another math item (see Figure 3b) requires the student to rotate the dividing lines on the pie chart so that the sizes of the colored portions correspond to the given ratios. Language items Acoustic and visual effects can be combined to help students relate semantics to spelling and pronunciation. For example, consider different ways of presenting an item to learn and test vocabulary and spelling. Figure 4 (on page 20) shows one example. Note that a student can use the computer keyboard or the alphanumeric, virtual keyboard displayed on the screen. Three components are associated with this item: pronunciation (which can be obtained by pressing a button), spelling, and object recognition.

IEEE MultiMedia 18

Educational games Online games have become commonplace not only among teens but also among adults. Our goal is to present education items in a way similar to what is attracting people to games, so that students can still enjoy playing games, as well as benefit from learning. Students can use education items separately or as batch-like activities grouped together in a game. A final score is awarded at the end of the game or when there is a time-out. The game keeps the top scores to challenge participants. The balloon shooting game (see Figure 5a) tests students math skills with perfect squares, multiples, and prime numbers. A student has to shoot the balloon marked with the target number. The games difficulty is controlled by balloon speed, displayed number range, and distraction amount introduced into the background. The budgeting game (see Figure 5b) is a different way of testing math skills. Here a student is given a budget to buy certain commodities. The student can click any individually priced

(a)

(b)

(c)

(d)

Figure 2. Example items using drag and drop: the student has to (a) drag the correct description to the right location on the map; (b) drag the correct labels into the answer boxes, (c) drag organs to positions on the body, and (d) drag grocery products into correct categories.

(a)

Figure 3. Math item requiring a student to (a) distribute numbers into four baskets so that the sum in each basket is the same and (b) divide the pie chart

(b)

into correct portions.

19

item in the scrollable panel on the right and place the item in the middle panel (shopping cart). The money spent on the selected commodities is displayed at the top of the screen. Crossword puzzles are used to test the vocabulary of a language as well as the semantics associated with a word. An example of such an item is shown in Figure 5c. A student first relates a word with an image. By clicking on the image, the corresponding location of the word in the puzzle area is highlighted. The student then uses the virtual or computer keyboard to spell out the word in the highlighted space.

Crome components
Figure 4. An item requiring a student to spell a word and drag the corresponding image into the answer box.

(a)

(b)

(c)
Figure 5. Educational games: (a) a balloon shooting game to practice and test a students math skills, (b) a budgeting game to test math skills; and (c) a crossword puzzle to test vocabulary and word understanding. The top scores are displayed before the game starts to challenge participants.

We designed Crome for multimedia education, but Web-based multimedia systems can be used in a broad variety of applications, such as hurricane analysis and simulation.14 Crome uses a combination of Web-development tools to optimize the constrained resources and provide user satisfaction. The development kits include Java 2D and 3D applets, JavaScript, Flash, Java 2 Micro Edition, PHP, and MySQL. SQL is an efficient query language used in many database retrieval systems.15 We chose these tools with platform and browser independence in mind. While a JavaScript item is quick to display, its 3D and animation capability is not comparable to Flash and Java3D. Flash animation is appealing, but its 3D capability is not as good as Java3D. However, Java3D items need more programming skills for development, and can be slow during rendering when the content gets too complex. Depending on the complexity of the item and the intended outcome, we choose one of these three tools for a particular design. Figure 6 shows the multimedia data flow in the Crome framework. Among others, an important feature of our design is scalability. We designed a generic template, similar to the multiple-choice format, for each item category that shares certain similarities with other categories. The system can generate multiple questions by altering the content inserted into the template. Furthermore, the formatting of the layoutsuch as the number of objects and baskets shown in Figure 3is controlled by parameters. This approach reduces recoding for individual questions.

20

Automatic difficulty level estimation In the multiple-choice or true-false format, an answer can only be correct or wrong. There is no partial mark awarded. However, when multimedia content, such as 3D items, is used in computer-adaptive testing, we can evaluate a students performance more accurately by considering partial scores. To evaluate the correctness of an answer and award partial marks, we interpret a molecular structure as a graph, where nodes are atoms and edges are bonds. In this way, we can assess an answers correctness by comparing the similarity between two graphs. A number of graph-similaritymatching algorithms can be found in the literature. Among these algorithms, graph edit distance is commonly used.16 In this algorithm, a set of graph edit operations is defined. These edit operations include deletion, insertion, and substitution of a node or an edge. The edit distance of two graphs is the length of the shortest sequence of edit operations required to transform one graph to the other. In our scoring scheme, we extend the edit distance to a weighted version and use a scoring matrix to store the weights. More details on this approach can be found in our earlier work.17 The graph-based strategy for estimating the difficulty level of chemistry questions discussed in Wu and Cheng17 might not be applicable for other subjects. A parameter-based strategy is a more general approach for assigning initial difficulties to items. We can use math problems to illustrate this concept. For example, when solving the problem distribute the numbers so that the sum in each bin is equal (see Figure 3a), we define the problems difficulty level by a function f(nbkt, nnbr), where nbkt is the number of baskets used and nnbr is the number of objects to distribute. The difficulty level increases as nbkt or nnbr increases. We verified the feasibility of our approach by conducting evaluation experiments. We have shown that its possible to predict the difficulty level of the items in Figure 3a by using a two-parameter logistic model following itemresponse theory and estimating the parameters on the basis of time taken to solve a problem by a small group of students using linear regression.18,19 Details on this approach can be found elsewhere.20

Update

Multimedia and item database

Retrieve Passed to multimedia server

Retrieve Processed by server engine

Marking and measurement criteria Update Training set

Handled by communicator

Multimedia item from client browser

Retrieve next item

Pass to communicator

Multimedia item to client browser

Testing beyond subject knowledge Conventional use of multimedia content in education focuses on subject knowledge but not on cognitive skills. According to Gardner, each person has seven intelligence aspects: linguistic, logical-mathematical, spatial, bodily kinesthetic, musical, interpersonal, and intrapersonal.21 By using multimedia content, we are able to test a students broader range of skills. Figure 7 gives an example item to test musical intelligence. The idea in this item is to evaluate how well a student can associate body movements with musical rhythms. The student has to match the videos labeled from one to five

Figure 6. The multimedia data flow in the Crome framework.

Figure 7. (a) An example of a musical item to test a students ability to perceive, express and transform musical forms, and (b) a sequence of video expressing different musical composition.

(a)

(b)

21

Figure 8. A screen shot of our process analyzer showing a chemistry example.

with the corresponding musical clips. Clicking a music button on the left panel plays each clip. Details on this work can be found elsewhere.22 Adaptive testing and student modeling Complex and tedious processes are often encountered in education for physics, chemistry, and mathematics, where the final answer to a particular question or problem is built upon the results of numerous smaller questions or problems. With this progression in mind, we incorporated a process analyzer in our Crome framework.23 The objective is two-fold: to assist students in improving their problem-solving skills using step-by-step hints and instructions, and to assist teachers in monitoring student performance so that proper help can be

Figure 9. An example of the Crome authoring interface.

provided in time. The process analyzer defines the top-level process as the root of a hierarchy composed of simpler and easier steps so that students who have difficulty solving the problem on their own are able to follow the hints and instructions leading to the correct answer. Educators control the hints and the scores in each step. The process analyzer records every response given by a student for performance analysis and student modeling, and is rendered through a GUI (see Figure 8). We designed each composite or simple question in the multimedia item pool for a given grade level and assigned each question a difficulty value. We evaluate students on the basis of their success in obtaining a predefined percentage of correct answers when responding to questions of a particular difficulty. During the process, we can give a student easier or more difficult questions, depending on whether his or her current response is wrong or correct. We have used this approach of adaptively selecting the next item in multiple-choice, true-false, and fill-in-the-blank questions. Using multimedia items in adaptive testing is still a subject of research: there is significant complexity in assigning difficulty values to multimedia items, and in creating a sufficiently large pool of effective items to evaluate student abilities. With an adaptive-testing strategy and a learners step-by-step record, educators can accumulate a training set over time. Teachers can apply machine-learning and artificialintelligence techniques to analyze data in the training set. Because different students require different types of assistance, teachers can customize their guidance on the basis of individual learning patterns. In addition to helping evaluate a students ability level, our design can help monitor a students progress. The training set contains valuable information for curriculum designers to fine-tune education materials. The sense in which educators can model individual students is directly related to the richness of the items being investigated. Multimedia item authoring graphical interface To assist educators who either dont have much time beyond their regular teaching schedule or dont possess the required technical skills to develop new multimedia question items, we created authoring templates. These templates are plug-ins integrated into the basic Crome framework. Figure 9 shows a

22

screen shot of the authoring interface. Educators can run the authoring module online for individual item upload or offline for subsequent upload in a batch. In creating these templates, we faced two main design challenges: the uniqueness of each innovative item type and the need to provide different processing pipelines for online and offline item creation. Uniqueness of each item type means that each template has its own set of properties and its own unique way of processing them. We overcame these challenges by separating the interface implementation into easy-tomanage components (as shown in Figure 9). Although we designed the authoring module for educators, its possible for students to use the tool to create questions, and through this process improve their understanding of the subject. A demo of the Crome framework can be seen at http://crome.cs.ualberta.ca.

(a)

Conclusion
The examples presented in this article represent only a small portion of the ideas motivating the development of Crome. Specific outcomes that cannot be tested purely using paper and pencilsuch as interpersonal skills and remote collaborationcan now be tested in an online computer-based multimedia context. With the design of Crome, we had several goals:

(b)

Portability. The Java-based implementation,

including J2ME on mobile devices, is platformindependent.

Reusability. An object-oriented design makes

individual components reusable.

Although mobile phones have been extensively used for information exchange, its still a research challenge to make multimedia educational items available on them for collaborative learning and adaptive testing, especially given limited mobile resources such as processing speed and bandwidth. MM

Figure 10. Example item on a mobile phone: (a) a student needs to fill in the missing numbers so that the sum in each row and column is the same. (b) Another item type, a word scramble,

Scalability. Users can add new item type

plug-ins without having to modify the basic framework or authoring module.

Acknowledgments
We thank the Crome team for its support. Alberta Informatics Circle of Research Excellence (iCORE) and Natural Sciences and Engineering Research Council (NSERC) supported this work.

showing the process of letter selection to form the correct word.

Interoperability. An authoring tool is an integral part of the Crome system, sharing the item type design as well as the media and item database. Using our framework as an initial step, we intend to bring greater awareness to the multimedia research community and to solicit effort on topics that have not been adequately studied. In future work, we plan to develop collaborative learning environments over mobile phones (see Figure 10). These environments will allow students from anywhere around the globe to collaborate with anyone at anytime.

References
1. S.-F. Chang et al., Next Generation Content Representation, Creation, and Searching for NewMedia Applications in Education, Proc. IEEE, vol. 86, no. 5, 1998, pp. 884-904. 2. G. Adomavicius and A. Tuzhilin, Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions, IEEE Trans. Knowledge and Data Engineering, June 2005, pp. 734-749.

JanuaryMarch 2009 23

3. R. Allen, The Web: Interactive and Multimedia Education, Computer Networks and ISDN Systems, vol. 30, nos. 16-18, 1998, pp. 1717-1727. 4. K.C. Feng et al., Joyce: A Multi-Player Game on One-on-One Digital Classroom Environment for Practicing Fractions, Proc. IEEE Intl Conf. Advanced Learning Technologies (ICALT), IEEE Press, 2005, pp. 543-544. 5. J.R. Galvao et al., Modeling Reality with Simulation Games for a Cooperative Learning, Proc. Winter Simulation Conf., Soc. Computer Simulation Intl, 2000, pp. 1692-1698. 6. A. Mitchel and C. Savill-Smith, The Use of Computer and Videogames for Learning, Learning and Skills Development Agency, 2004. 7. R. F. Lyvers and B. R. Horowitz, A Unique Instructional Tool for Visualizing Equipotentials and Its Use in an Introductory Fields Course, IEEE Trans. Education, May 1993, pp. 237-240. 8. K. Stubbs, Kana no Senshi (Kana Warrior): A New Interface for Learning Japanese Characters, Conf. Human Factors in Computing Systems (CHI) Extended Abstracts on Human Factors in Computing Systems, ACM Press, 2003, pp. 894-895. 9. T. Tang and G. McCalla, Utilizing Artificial Learners to Help Overcome the Cold-Start Problem in a Pedagogically-Oriented Paper Recommendation System, Conf. Adaptive Hypermedia and Adaptive Web-Based Systems, LNCS 3137, Springer, 2004, pp. 245-254. 10. G. McCalla et al., Active Learner Modelling, Conf. Intelligent Tutoring Systems, vol. 1839/2000, 2000, Springer, pp. 53-62. 11. A. Friedman, M. Zibit, and M. Coote, Telementoring as a Collaborative Agent for Change, J. Technology Learning, and Assessment, vol. 3, no. 1, 2004, pp. 1-41. 12. Y. Hao, Exploring Student Performance on a Multimedia Exam Program (Vexam), Proc. World Conf. Educational Multimedia, Hypermedia and Telecommunications, Assoc. Advancement of Computing in Education, 2002, pp. 711-712. 13. A. H. Johnstone, Why is Science Difficult to Learn: Things are Seldom What They Seem, J. Computer Assisted Learning, vol. 7, no. 2, 1991, pp. 75-83. 14. S. Cheng et al., A Reliable Web-Based System for Hurricane Analysis and Simulation, Proc. IEEE Systems, Man, and Cybernetics Conf., IEEE Press, 2004, pp. 5215-5220. 15. N. Rishe et al., Semantic Access: Semantic Interface for Querying Databases, Proc. Intl Conf. Very Large Databases (VLDB), ACM Press, 2000, pp. 591-594.

16. A.R. Kelly and E. Hancock, Graph Edit Distance from Spectral Seriation, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 3, 2005, pp. 365-378. 17. G. Wu and I. Cheng, An Interactive 3D Environment for Computer Based Education, Proc. Intl Conf. Multimedia and Expo, IEEE Press, 2007, pp. 1834-1837. 18. S.E. Embretson and S.P. Reise, Item Response Theory for Psychologists, Lawrence Erlbaum Assoc., 2000. 19. V. Linden and R.K. Hambleton, Handbook of Modern Item Response Theory, Springer Verlag, 1997. 20. I. Cheng, R. Shen, and A. Basu, An Algorithm for Automatic Difficulty Level Estimation of Multimedia Mathematical Test Items, Proc. IEEE Intl Conf. Advanced Learning Technologies, IEEE Press, 2008, pp. 175-179. 21. H. Gardner, Frames of Mind: The Theory of Multiple Intelligences, Basic Books, 1993. 22. I. Cheng, C. Kerr, and W.F. Bischof, Assessing Rhythm Recognition Skills in a Multimedia Environment, Proc. IEEE Intl Conf. Multimedia and Expo, IEEE Press, 2008, pp. 361-364. 23. I. Cheng, N. Rossol, and R.G. Goebel, SelfTutoring, Teaching and Testing: An Intelligent Process Analyzer, Proc. IEEE Intl Conf. Advanced Learning Technologies, IEEE Press, 2008, pp. 746-750. Irene Cheng is the scientific director of the iCORE NSERC Multimedia Research Center in the Department of Computing Science, University of Alberta. Her research interests include incorporating human perceptionjust-noticeable differencewith scalespace analysis based on psychophysical methodology, to improve 3D simplification and transmission techniques. Cheng has a PhD in computing science from the University of Alberta. Contact her at lin@cs.ualberta.ca. Anup Basu is professor and an iCORE-NSERC industrial research chair in multimedia at the University of Alberta. His research interests include the use of foveation in image and video compression and stereo visualization, and panoramic image sensors. Contact him at anup@cs.ualberta.ca. Randy Goebel is the CEO/director of iCORE, Alberta. He is also a professor in the Department of Computing Science, University of Alberta. His research interests include artificial intelligence in medicine, and intelligent data mining. Contact him at goebel@cs. ualberta.ca.

IEEE MultiMedia 24