10 1 1 126

Journal of
Educational Technology &

Society
Published by
International Forum of Educational Technology & Society
Endorsed by
IEEE Technical Committee on Learning Technology
April 2005
Volume 8 Number 2
ISSN: 1436-4522 (online)
ISSN: 1176-3647 (print)
Educational Technology & Society
An International Journal
Aims and Scope
Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology &
Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such
systems. The articles should discuss the perspectives of both communities and their relation to each other:
• Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to
blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by
proactively harnessing the available technologies and how they might be able to influence further developments through systematic
feedback and suggestions.
• Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical
teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the human-
computer interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has
been forgotten.
The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support
each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to
Educational Technology & Society and three months thereafter.
The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal:
Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and
Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance
Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, Human-
Computer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning
Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications,
Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training
Editors
Kinshuk, Massey University, New Zealand; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL
Research & Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany.
Associate editors
Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia
Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture,
Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National
University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Erkki Sutinen, University of Joensuu, Finland;
Vladimir Uskov, Bradley University, USA.
Advisory board
Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Sherman Alpert, IBM T.J. Watson Research Center, USA; Alfred Bork,
University of California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of
British Columbia, Canada; Tak-Wai Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-sen University,
Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, Deakin University, Australia; Roger Hartley, Leeds
University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, National Institute of Information Technology,
India; Akihiro Kashihara, The University of Electro-Communications, Japan; Paul Kirschner, Open University of the Netherlands,
The Netherlands; Ruddy Lelouche, Universite Laval, Canada; David Merrill, Brigham Young University - Hawaii, USA; Marcelo
Milrad, Växjö University, Sweden; Riichiro Mizoguchi, Osaka University, Japan; Hiroaki Ogata, Tokushima University, Japan;
Toshio Okamoto, The University of Electro-Communications, Japan; Yoshiaki Shindo, Nippon Institute of Technology, Japan; Brian
K. Smith, Pennsylvania State University, USA; J. Michael Spector, Syracuse University, USA.
Assistant Editors
Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini,
University of Michigan, USA; Pythagoras Karampiperis, University of Piraeus and ITI-CERTH, Greece; Dorota Mularczyk,
Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad Carlos III de Madrid, Spain; Ali Fawaz Shareef,
Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland.
Executive peer-reviewers
http://www.ifets.info/
Subscription Prices and Ordering Information

Institutions: NZ$ 120 (~ US$ 75) per year (four issues) including postage and handling.
Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling.
Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling.
Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk,
Information Systems Department, Massey University, Private Bag 11-222, Palmerston North, New Zealand. Tel: +64 6 350 5799 ext 2090.
Fax: +64 6 350 5725. E-mail: kinshuk@ieee.org.
Advertisements
Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the
journal, those involved in education and educational technology. Contact the editors at kinshuk@ieee.org.
Abstracting and Indexing

Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences,
ISI Alerting Services, Social Scisearch, ACM Guide to Compg Literature, Australian DEST Register of Refereed Journals, Computing
Reviews, DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier
Bibliographic Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED.
ISSN
ISSN1436-4522
1436-4522. (online)
© International
and 1176-3647Forum (print).
of Educational
© International
Technology
Forum of& Educational
Society (IFETS).
Technology
The authors
& Society
and (IFETS).
the forumThe jointly
authors
retain
andthe
the copyright
forum jointly
of the
retain
articles.
the
copyright
Permissionof the
to make
articles.
digital
Permission
or hard copies
to makeof digital
part or or
allhard
of this
copies
workofforpart
personal
or all of
or this
classroom
work forusepersonal
is grantedor without
classroom
feeuse
provided
is granted
that without
copies are
feenot
provided
made orthat
distributed
copies i
are
fornot
profit
madeor or
commercial
distributedadvantage
for profit and
or commercial
that copies advantage
bear the fullandcitation
that copies
on thebear
firstthe
page.
full Copyrights
citation on the
for components
first page. Copyrights
of this workfor owned
components
by others
of this
than
work
IFETS
owned
mustbybe
others
honoured.
than IFETS
Abstracting
must with
be honoured.
credit is permitted.
AbstractingTowith
copycredit
otherwise,
is permitted.
to republish,
To copy to otherwise,
post on servers,
to republish,
or to redistribute
to post ontoservers,
lists, requires
or to redistribute
prior specific
to lists,
permission
requiresand/or
prior a
specific
fee. Request
permission
permissions
and/or afrom
fee. the
Request
editors
permissions
at kinshuk@massey.ac.nz.
from the editors at kinshuk@ieee.org.
Guidelines for authors
Submissions are invited in the following categories:
• Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words)
• Book reviews
• Software reviews
• Website reviews
All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the
relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review.
• Each peer review submission should have at least following items: title (up to 10 words), complete communication details of ALL
authors , an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, four - five
descriptive keywords, main body of paper (in 10 point font), conclusion, references.
• Submissions should be single spaced.
• Footnotes and endnotes are not accepted, all such information should be included in main text.
• The paragraphs should not be indented. There should be one line space between consecutive paragraphs.
• There should be single space between full stop of previous sentence and first word of next sentence in a paragraph.
• The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example,
Internet based system, Distance learning).
• Do not use 'underline' to highlight text. Use 'italic' instead.
Headings
Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far
as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point
bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this
purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are
consecutive, there should be one blank like between them).
Tables
Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be
provided for every table (below the table) and must be referenced in the text.
Figures
Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be
provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width.
Please also provide all figures separately (besides embedding them in the text).
References
• All references should be listed in alphabetical order at the end of the article under the heading 'References'.
• All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994) or "(authors1, year1; authors2,
year2)" style e.g. (Merrill, 1999; Kommers et al., 1997).
• Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]."
• It is important to provide complete information in references. Please follow the patterns below:
Journal article
Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology,
35 (2), 7-13.
Newspaper article
Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15.
Or
Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, retrieved July 7, 2004, from
http://www.vnunet.co.uk/News/88174.
Book (authored or edited)
Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University.
Chapter in book/proceedings
Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.), Instructional
software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95.
Internet reference
Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2004,
from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html.
Submission procedure
Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest
Editors will advise the authors regarding submission procedure for the final version.
All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible.
The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF
and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text.
Please provide following details with each submission: Author(s) full name(s) including title(s), Name of corresponding author, Job
title(s), Organisation(s), Full contact details of ALL authors including email address, postal address, telephone and fax numbers.
The submissions should be sent via email to (Subject: Submission for Educational Technology & Society journal): kinshuk@ieee.org.
In the email, please state clearly that the manuscript is original material that has not been published, and is not being considered for
publication elsewhere.
ISSN
ISSN1436-4522
1436-4522. (online)
© International
of Educational
© International
Technology
Society (IFETS).
Technology
The authors
& Society
and (IFETS).
authors
retain
andthe
the copyright
forum jointly
of the
retain
articles.
the
copyright
Permissionof the
to make
articles.
digital
Permission
or hard copies
to makeof digital
part or or
allhard
of this
copies
workofforpart
personal
or all of
or this
classroom
work forusepersonal
classroom
feeuse
provided
is granted
that without
copies are
feenot
provided
made orthat
distributed
copies ii
are
fornot
profit
madeor or
commercial
for profit and
or commercial
that copies
on thebear
firstthe
page.
full Copyrights
citation on the
for components
components
by others
of this
than
work
IFETS
owned
mustbybe
others
honoured.
than IFETS
Abstracting
must with
be honoured.
AbstractingTowith
copycredit
otherwise,
is permitted.
to republish,
post on servers,
to republish,
or to redistribute
lists, requires
or to redistribute
prior specific
to lists,
permission
requiresand/or
prior a
specific
fee. Request
permission
permissions
and/or afrom
fee. the
Request
editors
permissions
Journal of Educational Technology & Society
Volume 8 Number 2 2005
Table of contents
Formal discussion summaries
Transitional Developments in Online Courses and Programs: Theory and Practice 1-6
Moderator and Summarizer: Brent Muirhead and Muhammad Betz
Implementing an Innovation Cluster in Educational Settings In Order to Develop 7-15

Constructivist-Based Learning Environments
Moderator and Summarizer: Donna Russell and Art Schneiderheinze
Full length articles
Using discussion webs to Develop an Academic Community of Learners 16-39

Eugene Matusov, Renee Hayes and Mary Jane Pluta
Effects of the Cognitive Level of Thought on Learning Complex Material 40-53

Eshaa M. Alkhalifa
Combining Software Games with Education: Evaluation of its Educational Effectiveness 54-65
Maria Virvou, George Katsionis and Konstantinos Manos
Towards A Web-Based Handbook of Generic, Process-Oriented Learning Designs 66-82

Olivera Marjanovic
Authentic Teaching as the Context for Language Learning 83-93

Ana Deumert and Christine Spratt
ICT-Pedagogy Integration in Teacher Training: Application Cases Worldwide 94-101

Insung Jung
Teachers’ Perceptions on the Roles on Educational Technology 102-106

Raymond A. Zepp
Hypermedia as a Cognitive Tool: Student Teachers’ Experiences in Learning by Doing 107-117

Zahide Yildirim
Designing an open component for the Web-based learning content model 118-124
Xin-hua Zhu
ISSN 1436-4522
1436-4522.(online)
© International
and 1176-3647
Forum (print).
of Educational
© International
Technology
Forum&ofSociety
Educational
(IFETS).
Technology
The authors
& Society
and the(IFETS).
forum Thejointly
authors
retainand
thethecopyright
forum jointly
of theretain
articles.
the
Permissionoftothe
copyright make
articles.
digital
Permission
or hard copies
to make
of part
digital
or all
orof
hard
thiscopies
work for
of part
personal
or allorofclassroom
this work use
for is
personal
grantedorwithout
classroom
fee provided
use is granted
that copies
withoutarefee
notprovided
made or that
distributed
copies iii
for profit
are not made
or commercial
or distributed
advantage
for profitand
or that
commercial
copies bear
advantage
the fulland
citation
that copies
on the bear
first page.
the full
Copyrights
citation onfor
thecomponents
components
by othersof than
this work
IFETS owned
must bybe
honoured.
others thanAbstracting
IFETS mustwith be honoured.
Abstracting
To with
copy credit
otherwise,
is permitted.
to republish,
To copy
to post
otherwise,
on servers,
to republish,
or to redistribute
to post on
to lists,
servers,
requires
or to prior
redistribute
specifictopermission
lists, requires
and/or
priora
specific
fee. Request
permission
permissions
and/orfrom
a fee.
theRequest
editors permissions
A Comparison of Teacher and Student Attitudes Concerning Use and Effectiveness of 125-135
Web-based Course Management Software
Gary H. Jones and Beth H. Jones
Web-based Cognitive Apprenticeship Model for Improving Pre-service Teachers’ 136-149

Performances and Attitudes towards Instructional Planning: Design and Field Experiment
Tzu-Chien Liu
Instructional Uses of Instant Messaging (IM) During Classroom Lectures 150-160

Mable B. Kinzie, Stephen D. Whitaker and Mark J. Hofer
Understanding Teacher Mindsets: IT and Change in Hong Kong Schools 161-169

Robert Fox and James Henri
The Effectiveness of Computer-Assisted Instruction in Teaching Introductory Statistics 170-178

Ramazan Basturk
Student Alienation, Academic Achievement, and WebCT Use 179-189

Genevieve Marie Johnson
Acquisition of simple and complex knowledge; a knowledge gap perspective 190-202

Judy Le Heron and Frank Sligo
Preferences toward Internet-based Learning Environments: High School Students’ 203-213

Perspectives for Science Learning
Chin-Chung Tsai
Web-based Interactive Writing Environment: Development and Evaluation 214-229

Jie Chi Yang, Hwa Wei Ko and I Ling Chung
Software reviews
Time Engineers Software 230-236

Reviewer: Michael Verhaart
Book reviews
Understanding Designers Designing for Understanding 237-237

Reviewer: Taiyu Lin
ISSN
ISSN1436-4522
1436-4522. (online)
© International
of Educational
© International
Technology
Society (IFETS).
Technology
The authors
& Society
and (IFETS).
authors
retain
andthe
the copyright
forum jointly
of the
retain
articles.
the
copyright
Permissionof the
to make
articles.
digital
Permission
or hard copies
to makeof digital
part or or
allhard
of this
copies
workofforpart
personal
or all of
or this
classroom
work forusepersonal
classroom
feeuse
provided
is granted
that without
copies are
feenot
provided
made orthat
distributed
copies iv
are
fornot
profit
madeor or
commercial
for profit and
or commercial
that copies
on thebear
firstthe
page.
full Copyrights
citation on the
for components
components
by others
of this
than
work
IFETS
owned
mustbybe
others
honoured.
than IFETS
Abstracting
must with
be honoured.
AbstractingTowith
copycredit
otherwise,
is permitted.
to republish,
post on servers,
to republish,
or to redistribute
lists, requires
or to redistribute
prior specific
to lists,
permission
requiresand/or
prior a
specific
fee. Request
permission
permissions
and/or afrom
fee. the
Request
editors
permissions
Muirhead, B., & Betz, M. (2005). Transitional Developments in Online Courses and Programs: Theory and Practice.
Educational Technology & Society, 8 (2), 1-6.
Transitional Developments in Online Courses and Programs: Theory and

Practice
Moderator & Sumamrizer:

Brent Muirhead
University of Phoenix, USA
bmuirhead@email.uophx.edu
Muhammad Betz
Southeastern Oklahoma State University and University of Phoenix, USA
mbetz@sosu.edu
Discussion Schedule:
Discussion: January 17-26, 2005
Summing-up: January 27-28, 2005
Pre-Discussion Paper
A chief purpose of this discussion paper is to share information with IFETS readers on the recent advances in the
design of courses and delivery structures of online courses. The online course is an evolving educational offering
and significant progress has been made by online universities and traditional universities offering online courses
in terms of improved formats and protocols. Also, the discussion will strive to engage diverse online educators in
academic debate and discussion of the character of online courses, as evidenced by the distinction between not-
for-profit and for-profit approaches to e-education.
Not-For-Profit Online Courses

1. The Learning Objects Movement
Stephen Downes (2001) has written perhaps the seminal article on Learning Objects, which are as described by
Downes and as currently used, the logical extension of familiar programming concepts used by software
engineers. As Downes stated, “Educators need to apply design techniques learned long ago by the software
industry, and in particular, they need to learn a concept called Rapid Application Design” (Downes, 2001, p. 9).
Downes asks that an online course be viewed as a piece of software with a need to develop reusable content in
the form of self-contained objects of content, marked by generic identification monikers or metatags, that can be
openly shared, searched and used by course designers. The immediate rationale for learning objects, as defined
here, is that they would save developmental time and money.
Ultimately, their success relies on course designers’ willingness to share content, free of charge, with other
designers and on the feasibility of creating sharable components of content that would have relevance and
applicability. The use of learning objects for online courses was to be made viable through the use of IMS
[Instructional Management Systems] and SCORM [Sharable Courseware Object Reference Model] standards for
learning objects. As Downes, stated for example, “In order for this to work, the atlas in Turkey and the course in
the United States must define similar objects in a similar manner” (Downes, 2001). Many writers have attested to
the difficulty of the use and reuse of learning objects (Friesen, 2004; Sicilia & Garcia, 2003; Bratina, Hayes, &
Blumsack, 2002). Further, to date, there has not been a large scale implementation of learning objects as a basis
for online course creation.
2. U.K. eUniversity: A Contemporary E-Learning Case Study
It will be informative to briefly review the rise and fall of the online university known as United Kingdom e-
University (UKeU). The British government invested 62 million pounds ($113 million) to develop their
commercial venture which began in 2000 and eventually failed in 2004. UKeU worked with over 20 universities
in the United Kingdom that offered approximately 40 degree programs and had 900 students by the fall of 2004.
(Garett, 2004).
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
1
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.
An analysis of UKeU does reveal how a promising online educational plan can fall apart even when it has strong
financial resources. Garrett (2004) cites five major problems with UKeU’s plans:
¾ Timing- it began in February 2000 based on a fear that American online universities would capture the
international student market and they must act quickly to prevent this from happening. The dot-com crash
drove away potential organizations which might have invested in their educational venture.
¾ Focus- the UKeU business model stressed using an online delivery model but there was not adequate
evidence of a stable student market for their product. It is estimated that over one million students take
online classes in the US but the international market projections are much smaller.
¾ Branding- there were marketing problems that created confusion over value of the UkeU’s degrees because
advertisements focused on convenience which is not a central feature of higher education in the United
Kingdom .
¾ Platform-UKeU decided to create their own platform and not to use either Blackboard or WebCT which are
popular course management systems. This drained valuable financial resources from the company which
limited their ability to recruit new students.
¾ Impatience- it did take three years to actually launch their product which restricted recruitment of students
and hindered attracting private financial investments. UKeU operated only a year which is not adequate time
to build an elearning organization.
The platform was built in-house and was promoted as the best course management system in the world. Garett
(2004) observes that
“There is generally a significant gap between the pedagogic conception of instructional designers
and those of the average faculty member, to say nothing of the average student (even at the
graduate level). This suggests a tension between the amount of investment in the new platform
(said to be around £20/$35 million) and the rather limited marketing pay-off in terms of student
recruitment” (Buy or Build, paragraph #3).
The UKeU’s failure does raise serious questions about the feasibility of government sponsored eLearning
initiatives that can be competitively developed and implemented. UKeU does join a growing list of organizations
that have failed at their efforts in having online degree programs such as NYU Online and Scottish Knowledge.
Garrett (2004) relates that “massive up-front investments, lack of private sector cash, low enrollments, brand
confusion, and incomplete platform meant that by 2004 UKeU was doomed” (Impatience, paragraph #3).
For-Profit
3. Blackboard, WebCT, et al!
At the forefront of the for-profit advent of online courses are integrated course delivery packages and integrated
learning management systems. For example in a recent article four designated integrated course delivery
packages, Ed2Go, Elluminate, LearnLinc, and Wimba, were reviewed and evaluated on several criteria: cost,
complexity, control, clarity, common technical framework and features (Annis, Hensel, Lunstrom, & Jones,
2003). Likewise, four integrated learning management systems, CentraOne, IntraLearn, Lyceum, and Silicon
Chalk, were recently reviewed and evaluated (Clark, Cossarin, Doxsee, & Schwartz, 2004). Nevertheless, the
two major course delivery systems in the current online market are the well known Blackboard and WebCT.
Both of these platform giants have made significant developmental progress in online course capabilities and
formats.
World Wide Web Course Tools, WebCT, has produced a newly released version called WebCT Vista Academic
Enterprise edition in 2004 (Morningstar, Schubert, & Thibeault, 2004). This new version expands the flexibility
and features of WebCT by adding more elaborate text uses, personal file storage, shared online folders, and
shared curriculum content capabilities. It allows the import of learning objects from multiple sources and
numerous tools for course creation and conduct. The costs of WebCT have risen to a level of hundred(s) of
thousands of dollars (US), based on the many needs associated with licensing and hosting the platform
(Morningstar, et al.).
Probably the most popular course delivery platform, Blackboard, has created a new and more extensive program.
The basic Blackboard package has had a start-up cost of around $30,000 for a host server, along with an annual
charge of under $10,000 for yearly operations. Blackboard has offered course cartridges for several years in two
formats: standard and open access. The cartridge method supplies course content created by a supplier to course
managers, and the cost is passed on to students in the form of textbook-like fees, ranging from $10 to $25 per
2
student. Upon receipt of the cartridge fee, the student receives a “key” code that allows the cartridge to be
downloaded into the Blackboard shell (Blackboard, 2001).
As of August, 2004, Blackboard has added a Content System that, like WebCT, expands its capability to offer
new tools and content services. The stated purpose of the Content System is to allow Blackboard users to make
use of the world’s geometric expansion of digital content and to facilitate the sharing of it. Unlike the not-for-
profit view of “sharing,” Blackboard’s comes at a cost. The Core Content Repository allows Blackboard users to
create, import and export content throughout the entire owning organization. The Repository contains the
following components: Virtual Hard Drive, Learning Content Management, e-Portfolios, and Library Digital
Asset Management (Blackboard, 2004). The Content System contains a Learning Objects Catalog, such as
pictures, Flash movies, and PowerPoint presentations that can be shared among different classes. The use of the
Blackboard Content System includes an additional cost of approximately $35,000 a year, however; thus
elevating the overall costs of using the product.
4. University of Phoenix Innovations: rEsource, E-CaD, and Virtual Organizations
A majority of the University of Phoenix Online courses have utilized rEsource which is a collection of
electronically delivered learning resources which are designed to support individual course objectives. Students
and teachers have access to the resources which contain a syllabus or Unimodule, course readings or articles,
Power Point Presentations, Web based links to specific resources for a class and links to other student support
materials such as the University online library which includes 14,000 journals and over 20 million full text
articles (UOP Fact Book, 2004). Students can access UOP’s ebook collection of entire textbooks involving titles
from a diversity of academic disciplines Electronic textbooks that can be read online or students can print the
material. Additionally, UOP has been integrating simulations into their business classes which provide students
with opportunities to study business practices and decisions. ( Muirhead, McAuliffe & La Rue, 2001).
Recently, the University of Phoenix (UOP) has launched a new online educational design model know as E-CaD
(Enhanced Curriculum & Delivery Model. The University of Phoenix strives to be innovative in their online
design and delivery of online education. UOP educators and administrators have been studying how to make
their delivery model more efficient without sacrificing academic rigor. A real issue became one of scalability
because the institution had to find a better way to accommodate more students. The student population involves
over 227,000 students who participate in on-ground and online classes. As a for profit organization it does have
to be sensitive to stockholder concerns about the potential for future growth in their online degree programs
(Muirhead, 2004a).
UOP has frequently promoted their small class sizes in their literature. Swenson (2001) states “the low
student/faculty ratio and class sizes that average 13 students facilitate active learning and collaboration,
encourage time-on-task, and foster high student-faculty interaction” (p. 5). University officials relate that the
majority of today’s major distance education schools often have at least 20 or more students per class. Therefore,
the curriculum changes represent a major response to market factors which have help to prompt these changes.
E-CaD is an instructional format that has been created to enable instructors to facilitate a class size of 20 students
(Muirhead, 2004a).
The University has been testing various online delivery systems and E-CaD represents the culmination of their
research and pilot studies. It is a creative design that has retains an emphasis on essential student skills and
subject knowledge but enables instructors to handle larger classes. E-CaD has the following key features:
Student Academic Expectations

¾ students actively participate with substantive remarks in online discussions 4/7 days a week (previously 5/7
days)
¾ final week of class has optional student participation in online discussions (previously students participated
all weeks of course)
¾ weekly summaries are optional ( previously these were required)
Faculty Academic Expectations

¾ provide detailed syllabus (change only in specific E-CaD details)
¾ share two weekly online discussion questions (previously 3-6 questions)
¾ freedom to assign weekly online discussion questions to learning teams (previously dialog questions created
only for individual students)
3
¾ share weekly lectures can be optional if course has weekly overview of material in rEsource
¾ respond to student comments 5/7 days in online discussions (no change)
¾ share weekly grade reports with students (no change) (E-CaD, 2004; Muirhead, 2004).
The E-CaD model is currently being phased into the various online classes which will require careful
modification of the curriculum to fit this new format. UOP facilitators are naturally a little anxious about
increased class sizes and the impact that it will have on their work load. The author has taught several online
classes (US history & film studies) under the new model and has found that students are actually sharing at least
five days a week. Also, grading papers and responding to students online has been quite manageable and no
more time consuming. Instructors must be careful to sustain good online presence with more students in their
classes. The key is to daily share relevant messages in the main newsgroups and relate to all of the students
during each week of class (Muirhead, 2004a).
An upcoming innovation to be released by UOP online is the establishment of Virtual Organizations designed by
subject matter experts and practicing professionals to create realistic work environments, in a clinical, online
setting. UOP’s Virtual Organizations will target four areas: business, schools, healthcare and government. The
main thrust of UOP’s progressive development, in addition to adjustments in its course format, is a wide ranging
and increasingly sophisticated course presence on associated web sites.
5. Online Instructor Concerns
Teaching online is becoming more common with larger numbers individuals working in distance education
institutions or teaching online classes at traditional universities. Muirhead (2004b) notes the informal
observations from the University of Phoenix teachers about the various academic challenges that they and their
students face in their online classes:
Teacher challenges
¾ Student plagiarism of assignments
¾ Students with weak writing skills
¾ Providing adequate feedback for papers
¾ Managing learning team problems
¾ English is not their first language
¾ Unrealistic grade expectations
Student challenges
¾ Lack of basic computer skills
¾ Inconsistent grading of papers
¾ Writing quality papers
¾ Effectively handling the action research project
Conclusion
There are important questions that need to be addressed about online education that challenge today’s educators.
Distance education is an evolving entity that continues to change as technological advances provide more
sophisticated delivery of information to their students and as providers learn more about the market realities that
they face.
Suggested Questions for Discussion and Debate

1. What role do market forces play in the design and delivery of online courses? Why?
2. Can governmentally funded projects compete with market driven products in providing usable platforms for
online courses?
3. Will Learning Objects achieve the vision of uniform codification and widespread sharability? Why or Why
not?
4. Will for-profit systems, like Blackboard and WebCT continue to dominate the market for online courses?
5. How are web sites used to supplement online courses?
4
6. What innovations for online courses, other than those mentioned in this paper, are currently available?
Where are they available and why?
7. In what direction should the progressive development of online courses evolve in the next five years?
[This list of questions should not be considered an exclusive list of topics for this discussion.]
Post-discussion summary
1. Learning Objects
While the original intentions of this discussion were focused on the seven questions listed in the pre-discussion
paper, the dominant influence that captured the thoughts of most readers related to learning objects. Learning
objects were seen to be at the heart of the topic of online courses, and contributors were very diverse in their
interests related to them. Mark Nichols countered the theme of the pre-discussion paper that the case of learning
objects involved much more than the concept of sharing of content, pointing to issues related to building
consensus among developers to promote their use. Terry Anderson picked up the theme of a need for
developmental work with learning objects with respect to standards and accessibility, as well. Stephen Downes
similarly cited several reasons that developers were moving slowly in producing learning objects, due to a wide
diversity of new and existing tools for online courses and to an unfavorable environment for the reuse of learning
objects. To the contrary, Downes asserted that the current situation favors creating new learning objects, not the
reuse of existing learning objects.
Mitchell Weisburgh supported the notion that market forces are not favorable to the learning objects movement.
He stated, “First, there are low barriers to entry, both in terms of creating software tools and also in terms of
creating content.” The existence of many competent producers of online course content and the competition of
the market are seen as inhibitory. H. von Brevern (George) saw the success of learning objects as being tied to
the challenge of matching them with course activities, indicating issues of course design. Further, George stated
that researchers must find common standards and nomenclature for the learning objects movement to succeed.
James Kariuki introduced the topic of alternatives to learning objects in current developmental initiatives of
online courses. He believes that Expert Systems incorporated within larger Courseware Management Systems
offer more stable and reliable ways of presenting content. Muhammad Betz added to the theme of alternatives to
learning objects by suggesting that repositories of general learning materials hosted at web sites for general
accessibility, like the MERLOT project at http://www.merlot.org/home/MaterialView.po. Fridolin Wild
supported the existence of alternatives to the learning objects view with mention of the resource project,
EducaNext, that he described as, “a multilingual, academic exchange portal, where members of higher education,
research organizations, and professional communities can share, retrieve, and reuse learning resources.” Fridolin
qualified his comments by stating that repositories would not replace learning objects, but that learning objects
would generate repositories for their dissemination, in support of the importance of carefully marking and
tagging of learning objects for reuse. He suggested two strategies for reuse: whole and partial, related to the
perceived fit for the teacher-student situation.
A theme that emerged in this discussion of learning objects is that they were not fully understood in a general
sense and that they represented a line of development that did not have a promising future. Alfred Bork, for
instance, asserted that learning objects were not a correct direction for inquiry, and called for full course analyses
in which approaches to learning are carefully evaluated. Muhammad Betz drew an analogy between the learning
objects movement and shareware, that genre of software that is not quite ready for the market, but which is
passed around by a small clientele of enthusiasts. Richard Dillman concurred that given the enormous diversity
in the overarching e-environments, the learning objects movement is unstable.
2. Closing
What is remarkable about this discussion on transitional developments in online courses is that for all practical
purposes the IFETS forum had little to say except on the topic of learning objects. Why? Learning objects are
nebulous entities that professionals feel they need a good knowledge of. Yet, under scrutiny, they seem to hold
less promise than existing trends in research imply. The theme of the authors of the pre-discussion paper that the
market was overwhelming the learning objects movement held up. Another emergent theme of the discussion is
that there is great diversity in contemporary online courses, a diversity that apparently does not indicate clear
norms of application.
5
References
Blackboard: Course cartridge overview (2001). Blackboard, Inc, retrieved December 7, 2004 from
http://www.blackboard.com.
Blackboard content system (2004). Blackboard, Inc, retrieved December 7, 2004 from
http://www.blackboard.com.
Bratina, T., Hayes, D., & Blumsack, S. (2002). Preparing teachers to use learning objects. Technology Source,
November-December, retrieved December 7, 2004 from
http://www.wisc.edu/depd/html/TSarticles/Preparing_Teachers.htm.
Downes, S. (2001). Learning objects: Resources for distance education worldwide. International Review of
Research in Open and Distance Education, 2 (1), retrieved December 7, 2004 from
http://www.irrodl.org/content/v2.1/downes.html.
E-CaD (2004). University of Phoenix faculty materials, Phoenix, AZ, USA: University of Phoenix.
Friesen, N. (2004). The international learning object metadata survey. International Review of Research in Open
and Distance Learning, 5 (3), retrieved December 7, 2004 from
http://www.irrodl.org/content/v5.3/technote5.html.
Garrett, R. (2004). The real story behind the failure of U.K. eUniversity. Educause Quarterly, 27 (4), retrieved
December 7, 2004 from http://www.educause.edu/apps/eq/eqm04/eqm0440.asp?bhcp=1.
Muirhead, B. (2004a). E-CaD: A new curriculum model at the University of Phoenix. Learning Technology, 6
(4), 27-28, retrieved December 7, 2004 from
http://lttf.ieee.org/learn_tech/issues/october2004/learn_tech_october2004.pdf.
Muirhead, B. (2004b). Online teacher education at the University of Phoenix. Paper presented at the Annual
Conference of the Association of Information Technology for Teacher Education (ITTE 2004), July 5-7, 2004,
Chester, England.
Muirhead, B., McAuliffe, J., & La Rue, M. (2001). Online Resource Page: Using Technology to Enhance the
Teaching and Learning Process. Educational Technology & Society, 4 (4), retrieved December 7, 2004 from
http://ifets.ieee.org/periodical/vol_4_2001/discuss_summary_august2001.html.
Principles of virtual organizations: A primer for students and faculty (2004). University of Phoenix, retrieved
December 7, 2004 from https://ecampus.phoenix.edu.
Sicilia, M., & Garcia, E. (2003). On the concepts of usability and reusability of learning objects. International
Review of Research in Open and Distance Education, 4 (2), retrieved December 7, 2004 from
http://www.irrodl.org/content/v4.2/sicilia-garcia.html.
Swenson, C. (2001). New models for higher education: Creating an adult-centered institution. Globalization:
What issues are at stake for universities? retrieved December 7, 2004 from
http://www.bi.ulaval.ca/Globalisation-Universities/pages/actes/Craig-Swenson.pdf.
UOP Fact Book (2004). University of Phoenix, Phoenix, AZ, USA: University of Phoenix.
6
Russell, D., & Schneiderheinze, A. (2005). Implementing an Innovation Cluster in Educational Settings In Order to Develop
Constructivist-Based Learning Environments. Educational Technology & Society, 8 (2), 7-15.
Implementing an Innovation Cluster in Educational Settings In Order to

Develop Constructivist-Based Learning Environments
Moderator & Sumamrizer:

Donna Russell, Ph.D.
Assistant Professor
Dept. of Curriculum and Instructional Leadership
School of Education, University of Missouri-Kansas City
USA
russelldl@umkc.edu
Art Schneiderheinze, Ph.D.

Project Construct National Center
ASchneid@COLUMBIA.K12.MO.US
Discussion Schedule:
Discussion: February 10-23, 2005
Summing-up: February 24-25, 2005
Pre-Discussion Paper
Abstract
This forum will initiate with a description of the implementation of a constructivist-based reform effort in k-12
classrooms in the US. The reform effort was an online collaboration among four teachers in 4th and 5th grades
classrooms to implement a design problem-based unit meant to develop higher-order thinking responses in
students. The unit design principles are correlated to current research in the cognitive sciences and the teachers’
efforts at implementing reform are correlated to current research in theories of innovation. The initial
description will also briefly characterize aspects of the implementation process that led to productive or less
productive reform results. The questions that will be developed in this forum include:
1. How can teachers’ professional development programs to increase the effectiveness of innovation in
education?
2. Do students have a stake in educational reform including emerging technologies-specifically constructivist-
based learning principles-, as in are these learning principles important and valid for students?
3. How can educational systems encourage innovation in education?
4. How can communities support efforts to change in education?
5. Are reforms, such as the integration of advanced technologies within a constructivist-based learning
environment meant to develop higher-order cognition valued in today’s society?
6. How can educational research aid efforts at change in education?
Introduction
This dialog will focus on the design and the goals for implementation of an innovation cluster which includes an
online workspace for students and an authentic problem-based unit designed to develop higher-order cognition
processes in k-12 students. The topics of consideration in this dialog include the identification of:
1. the productive aspects of collaborative professional development for innovative educators,

2. the characteristics of the online technology used to develop the constructivist-based learning environment,
3. the types and qualities of communications in the teacher’s local context that supported innovation.
7
The Innovation Cluster
The innovation cluster included an emerging technology, Shadow NetWorkspace™ (SNS), and a unit design
framework, “Improving Interstate 70”, based on constructivist-learning principles. Rogers (1995) suggests that
some innovations, usually technology innovations, are better viewed as a cluster, in which the innovations within
the cluster share a complementary relationship.
Shadow NetWorkspace™
The 4th and 5th grade teachers used SNS to collaborately design and implement the unit. Their students, as well,
collaborated throughout the unit and interacted with experts in related domains using SNS. SNS (Laffey, Musser,
& Espinosa, 2000), designed and produced by the Center for Technology Innovations in Education (CITE) at the
University of Missouri at Columbia, is a web-based work environment designed and developed specifically for
use in K-12 schools to support schools and learning. Shadow was developed at CTIE with support provided by
the SBC Foundation, the Missouri Research and Education Network (MOREnet), the University of Missouri
System, and the U.S. Department of Education. The operating system for SNS is Red Hat Linux. Since SNS is
distributed with an open source license, it is free to all schools. MOREnet served the middleware tool from its
server to the schools of the participating teachers.
The goal of the developers of SNS was to develop advanced online tools that make a collaborative learning
environment possible for any school. SNS provides shared workspaces for collaboration; discussion boards, chat,
email, and messenger systems for communication; and web-based document editors and viewers for
representation. A user of SNS gets a personal desktop that allows him or her to use the tools available in SNS
including a calendar, homework tool, profile, and word processing tool called Shadowdoc. Any user can create
different workgroups, see Figure 3, including a review panel group which allows outside experts to dialog with
students through a structured response forum. The students invite the experts in through an email template and
then the experts respond through chat or discussion board dialogs. Students and teachers can engage in
synchronous dialogs using the chat function in SNS.
The file management system enables a user to upload and store files and media objects in a folder system. Users
can create new folders, move and delete folders and files, and search for files within their file system. Group
structures enable distribution, and sharing of file objects by its members. Groups are designed around social
structures in learning communities and provide a new functionality for the roles within the group. In one
particular type of group, review panels, the owner can place items and the other group members (including
people from outside the community of users) can view them and leave notes. This creates a potential to offer
authentic interactions with experts in the field with whom the students would otherwise not have contact.
SNS served several purposes in the development of the unit. First, the teachers in this study collaborated to
design the unit via professional development activities provided by the both Art and me. They took advantage of
SNS tools such as a discussion board, file management system, messaging, and chat rooms. Additionally, the
students used SNS to facilitate collaboration, communication, and knowledge sharing during the unit. SNS is a
closed-community of members that have the ability to form different types of groups. Therefore, the students are
capable of working collaboratively online within a safe environment.
The CBLE unit
The unit, “Improving Interstate 70”, engages students in tackling a complex, open-ended problem taken from a
real-world context. Interstate 70, stretching across 251 miles of Missouri and intersecting almost every major
highway that traverses the state, is the most widely used highway in Missouri. The newest stretch of I-70 is
nearly 34 years old, and the oldest part is over 43 years old. The Missouri Department of Transportation reports
that nearly 200 people are killed in accidents on Missouri interstates each year. Furthermore, much of Interstate
70 is carrying more traffic than it was designed to accommodate. By 2020, with no changes made, traffic on the
entire route will far exceed its capacity. The design problem for this unit is the development of the I-70 corridor
throughout the state of Missouri.
In this problem-based unit, students from four Missouri schools collaborated to work as engineers to tackle this
problem and propose an effective and efficient way to ultimately improve the highway border-to-border in
Missouri. Appendix A is the Instructional Design Template for the I-70 Unit distributed to the teachers prior to
8
the initiation of the unit. The goals of the unit were based on the Missouri Show-Me Standards, a set of academic
standards that school districts can use to align curriculum and serve as the basis for state-wide assessment on the
Missouri Assessment Program (MAP). The standards are based on the idea that “the success of Missouri’s
students depends on both a solid foundation of knowledge and skills and the ability of students to apply their
knowledge and skills to the kinds of problems and decisions they will likely encounter after they graduate”
(Missouri Department of Elementary and Secondary Education, ¶1).
The standards are grouped into two sections: performance (process) standards, which identify thinking processes
and skills students need in order to acquire, organize, and apply knowledge, and knowledge (content) standards,
which identify important content students should learn from their studies in the traditional subject areas.
Although the instructional unit in this study incorporates a set of knowledge standards in communication arts,
math, science, and social studies, the goals for the unit align closely with the performance standards. The goals
for the unit included:
¾ Working as a researcher exploring transportation systems in Missouri, the student will gather, organize,
analyze, and apply information and ideas (Show-Me Performance Standard #1).
¾ While analyzing potential problems and solutions, the student will communicate effectively within and
beyond the classroom (Show-Me Performance Standard #2).
¾ Using the tools of inquiry to develop a highway improvement plan, the student will recognize and solve
problems (Show-Me Performance Standard #3).
¾ While considering the interdependence of community and state needs, the student will use critical thinking
to defend and support his or her decisions (Show-Me Performance Standard #4).
¾ As an engineer working with other engineers, the student will be a responsible group member and
demonstrate positive leadership skills (Show-Me Performance Standard #4).
The unit in this study draws upon the theoretical and practical applications of constructivist learning principles.
For the purposes of this study, the researchers identified important concepts based on a constructivist theoretical
framework that contributes to the design of this unit. These concepts, within an ill-structured problem-based
instructional design method (Jonassen, 2000; Savery & Duffy, 1996) include: scaffolding and mediation
(Vygotsky, 1986; Wertsch, 1998), goal-directed (Schank, 1994), meaningful context and inquiry (Lave &
Wenger, 1991), and collaboration (Salomon, 1993).
A template of the unit Phases is shown below as Figure 1. The three phases were designed to be developmental
and build on previous learning experiences. The design problem is the development of the I-70 corridor
throughout the state of Missouri. The ill-structured problem was used as a design topic because the interstate
runs across the state and is used by all the communities involved in the study. Phase 1 was designed to aid the
students in developing the local issues in the problem. Phase 2 allows the students to develop expertise by
communicating with experts. During Phase 3 the students develop solutions and present these solutions to an
audience.
Design Problem Solving Model

Drawing upon literature in problem solving, specifically design problem solving, the unit engaged students in
authentic processes similar to actual designers. Students can be actively involved in practices directly related to
practices of a particular domain rather than passively reading about, hearing about, or merely thinking about
those practices as something outside of school (Barab & Duffy, 2000; Lave & Wenger, 1991). First, as suggested
by Banathy (1996), design problem solving initially involves a transcending process – a realization that the
existing system must be changed as a whole or that a new system must be created. If a system cannot be changed
or a system does not already exist, the designer begins a series of transformations from genesis to a model of the
future system. Although Banathy (1996) defines the “genesis” as recognizing that a new system is needed, other
literature in engineering and graphic design suggests that the designer does not always initiate this process. In
fact, a client, who hires the designer to do something, starts this process. This leads to a series of constraints
identified that fall on a continuum from most flexible with more degrees of freedom (constraints identified by the
designer) to the most rigid (constraints imparted on the designer by a legislative body) with mandatory
requirements (Lawson, 1984).While doing this, the design problem solver negotiates the design inquiry
boundaries.
9
Scaffolding
Throughout Phase 1 of the unit, students gathered scientific, historical, and physical evidence from a variety of
sources in order to determine why the Interstate 70 problem is important to their community. In the process,
students developed a vision of how the Interstate 70 system should meet the needs of people in their community.
Working with students from other participating schools, students negotiated design boundaries by understanding
concepts important to the design of an interstate highway system (i.e., traffic flow, socioeconomics, design and
engineering, public affairs, natural environment, human environment, financing, and constructability). This in-
depth research enabled students to activate relevant prior knowledge, identify designer, client, and legislative
constraints, and recognize system interactions. With expertise distributed among students from among
10
participating classrooms, students returned to their original groups and began to outline the specifications of the
future system, based on their reconstruction of the knowledge acquired from the expert groups and their
conceptions of their community and the state’s needs for transportation. Depending on the goals of the
participating teachers, students might develop a model of the future system to communicate their ideas and
defend the idea’s feasibility to experts in the field.
Technology
In a problem solving experience, students actively confront ill-structured problems that closely resemble real-
world problems. Throughout the experiences, students assume the goal of responding to the problem with a
feasible solution based on the knowledge they constructed throughout the unit. Jonassen et al. (1999) suggest
that the learning environment includes a problem manipulation space in which students can experiment with the
problem and see the results of their experimentation. In addition, students need access to information such as text
documents, video, sound, graphics, and so forth to begin making meaning about the problem as well as related
cases to represent the complexity of the problem from multiple perspectives. This instructional method requires
students to demonstrate a set of skills that extend beyond the mere recitation of facts or correct responses.
Fortunately, teachers can support student learning by taking advantage of technology tools that foster cognitive
processing and collaboration.
Goal-directed
Throughout the instructional unit, students will work collaboratively to respond to the critical question in each
phase of the unit --- in Phase 1, “Why is the Interstate 70 problem important to your community?”, in Phase 2,
“How can we use our expertise in an area related to Interstate 70 to better understand the problem and develop a
feasible solution?”, and in Phase 3, “How can we use the knowledge and skills from Phase 1 and Phase 2 to
develop a feasible solution to the Interstate 70 problem?”. As students work to develop responses to these
questions, the teachers will provide ways for students to control the environment in which they learn giving them
the opportunity to adapt what is presented to them to their needs, to make choices to direct their learning, and to
construct their own understanding of the information. Therefore, according to Schank, Fano, Bell, and Jonassen
(1994), the students must find the goals of the unit authentic, and the skills necessary to accomplish those goals
must be skills that the teacher wants the students to learn.
Meaningful context
Petraglia (1998) questions teachers’ attempts to pre-authenticate constructivist-based learning experiences when
inherent in constructivist theory are the ideas that meaning-making is in the mind of the knower and reality is a
product of the mind. While making a parallel between rhetoric and constructivist learning, Petraglia suggests
“argument as the quintessential constructive process” (p. 128). Consequently, within this framework arguments
can draw on prior experiences, prior reasoning frameworks, knowledge of social convictions, and the gathering
of relevant information to serve as evidence. (p. 128)
While persuading students to accept the authenticity of an educational task, the teacher can create a sense of the
significance of the issue using a variety of data, theoretical frameworks, and lines of reasoning. Embedding skills
and knowledge in holistic and realistic contexts also enable students to generate new knowledge and sub-
problems as they determine how and when knowledge is used (Cognition and Technology Group at Vanderbilt,
1992). When students conceptualize the learning process as a means of finding a solution or eliciting a response
predetermined by the teacher, they will not assume ownership of the learning (Savery & Duffy, 1996;
Schoenfield, 1996). The students spend the first phase of the instructional unit determining why the problem in
the unit is important to their community. The connection may be initially ambiguous, but the teachers can help to
establish the meaningful context by providing students with opportunities to gather information and questioning
the relevance of that information to their community and the unit problem. Integrated into this model for design
problem solving is also the model used by expert designers and engineers at the Missouri Department of
Transportation to propose alternative solutions to the public. Their model created an authentic context for
students by closely integrating the design processes used by experts.
Hinrichs (1992) suggests that more experienced designers employ an associated mechanism for drawing on prior
experiences and look for similarities between these prior experiences and the current problem. This process,
11
known as case-based reasoning, helps explain a portion of the cognitive processes that designers undergo to
propose solutions to problems. Designers store previous experiences with a problem in memory as chunked
information known as cases. Cases do not only come from a designer’s personal experience but also from
documented experiences of others (Garza & Mayer, 1996). When a new problem arises, the designers retrieve
elements of cases from prior experience and determine the adaptability of that experience to the new problem.
By drawing on previously stored cases, the designer can avoid spending time necessary to derive solutions from
scratch and use prior experiences as a means to rationalize the quality of a solution when no algorithmic method
for assessment is available (Kolodner & Leake, 1996). Consequently, teachers will provide opportunities for
students to analyze case studies (Shulman, 1992) about other projects related to design problem-solving in order
to enrich the context for students to apply expertise and identify interrelationships among those areas of
expertise.
Inquiry
Through the process of inquiry, students construct much of their understanding of the natural and human-
designed worlds. Inquiry implies a "need or want to know" premise. Inquiry is not so much seeking the right
answer -- because often there is none -- but rather seeking appropriate resolutions to questions and issues.
Although the students are working towards responding to the same critical question in Phase 1, students will
generate a series of questions about their community and about the Interstate 70 based on their prior knowledge
or naïve theories (Clement, 1993; Resnick, 1983; Mestre, 1987). At the end of Phase 1, the teacher provides the
opportunity for students to reflect on the importance of the Interstate 70 problem and determine what areas of
expertise should be formed in order to develop a feasible solution to the problem. Throughout the process, the
students will access a variety of information resources, including digital resources available on the World Wide
Web.
Collaboration
Students develop responses to the critical questions in the unit through collaboration that enables them to
distribute the cognitive load among members of the group, take advantage of the groups’ distributed expertise,
and support individuals to resolve diverse points of view (Evenson & Hmelo, 2000; Brown, 1995; Pea, 1993;
Salomon, 1993). Within the unit, students will participate in three different types of collaborative efforts. First,
in Phase 1, students worked in groups of two or three students to gather and analyze information about the
importance of the Interstate 70 problem to their community. Each small group of students presented their
perspective to the other groups in the class, while the teachers facilitated students negotiating multiple
perspectives and shared understanding (Bednar et al., 1992; Resnick et al., 1991).
In Phase 2, students formed cross-classroom workgroups based on the areas of expertise they identify as
important to solving the Interstate 70 problem. After the four participating classrooms determined the number of
areas of expertise to investigate, the teachers divided her students (based on their interest) into the determined
number of expert groups. The students, working with students from the other classrooms in the same expert
group, collaborated using the SNS network technologies to develop their understanding of the expert area and
determine the importance of that expert area to solving the Interstate 70 problem.
In Phase 3, students created collaborative solution groups within their classrooms that consist of students from
all of the areas of expertise. The students, who took on various roles in the knowledge building process in Phase
2, will worked with others in a jigsaw format (Aronson et al., 1978) to develop group understanding of each area
and of the interdependence of the expert areas. In order for the solution group to develop a feasible solution to
the Interstate 70 problem, all of the areas of expertise are important to assess the group’s proposed solution.
Conclusions
Constructivist learning theory serves as an epistemology of learning and understanding and suggests that
knowledge and meaning are not fixed but instead constructed by the individual within the context of meaningful
learning. A constructivist-based learning environment, such as an extended unit focusing on a particular open-
ended problem, engages students in meaningful learning through complex, novel, and authentic tasks.
12
In the Report to the President on the Use of Technology to Strengthen K-12 Education in the United States
(1997), the committee of advisors recognized the benefits of a constructivist theoretical framework to student
learning and recommended looking more closely at the constructivist pedagogic model and the role of
technology within a constructivist curriculum. Instances of constructivist-based learning theories include situated
theories of learning (Brown, Collins, & Druguid, 1989: Greeno, 1997; Lave & Wenger, 1991; Roth & Bowen,
1995) and distributed cognition theory (Pea, 1993; Resnick, Levine, & Teasley, 1991; Salomon, 1993), which
emphasize how the responses of the learner and the design of the learning environment affect the cognitive
development of knowledge in students.
Problem-Based Learning (PBL) is an instructional method that addresses the complex knowledge and skill
applications that students will face in the future by participating as problems solvers to tackle complex, ill-
structured problems that resemble if not mirror real world problems. PBL proposes that learning experiences that
build on the interdependent attributes of meaningful learning including authentic, intentional, active,
constructive, and cooperative learning (Jonassen et al., 1999) and involve meaningful application of knowledge
and skills. The advanced learning potential incorporated in the design of these types of units with the infusion of
technology to support the development of these skills has been shown to be an effective form of innovation in
educational research. However, if the development of cble’s has been identified as a force of change in education
and if new technologies have been shown to develop these types of learning environments, then why are there so
few examples of the successful pairing of these innovations?
Questions for Online Forum:

1. How can teachers’ professional development programs to increase the effectiveness of innovation in
education?
2. Do students have a stake in educational reform including emerging technologies?
3. How can educational systems change to encourage innovation in education?
4. Can communities support efforts to change in education?
5. Are reforms, such as the integration of advanced technologies within a constructivist-based learning
environment meant to develop higher-order cognition considered valuable in today’s society?
6. How can educational research aid efforts at change in education?
Post-discussion summary
The IFETS dialog that responded to the topic of constructivist learning environments and the impact of
technology on the development of these innovative learning environments focused on two main themes:
innovation and creativity and the use of technology in educational settings and the societal issues related to
reform in education.
Innovation and Technology in Education

In response to the initial questions on innovation in education, Alfred posted with a discussion of past innovators
and how technology can develop creativity in students. Brent posted an overview of studies on creativity and
asked whether the current educational system can support innovative teachers and the development of creative
uses for technology. Dawn posted on the learning benefits of using these technologies to solve real-world
problems by allowing students to respond to and structure information. Diane commented on the types of
educational contexts that develop innovation and how technologies can impact that process in formal and
informal learning environments including a question or whether or not innovative principles mediated by
technology are unique to different learning environments. Liz and Niahm dialoged about the potential of
multimedia enhanced learning environments to develop creative responses. Bill posted with examples of
creativity in several settings including the U.S. Army.
Change in Educational Settings

Several ideas were discussed related to understanding how changes were and could occur in educational settings.
August and Mitchell both noted how difficult it is to change educational settings using game theory to discuss
13
the issues of risk and potential inherent in change. Rien posted on how important it is to be aware of how the
human and non-human aspects impact the development of innovative learning environments with advanced
technologies. Dr. De Hoyos reviewed the collaborative research project with the University of Texas at Austin
and the Tecnologico de Monterrey in Mexico City which is studying systemic change processes in higher
educational settings. Chew mentioned the difficulties of change in educational settings where creativity is
encouraged as an explicit policy by the education ministry but the assessment process is still a rigid and
summative program. Art dialoged on the professional development needs of innovative teachers and how these
teachers need the ability to resolve conflicts in their work activity that result from their efforts to implement
change. Marylu also posted on the support systems need by innovative educators implementing change. Oyeteju
and Kenn noted the differences in the ways learning is viewed and knowledge is used in online learning
environments.
Conclusions
The common thread that emerged in the postings was a realization of how powerful these new technologies can
be when used to develop constructivist learning and how radical the changes must be for them to be used in their
most productive ways in educational settings. So much of the system has to be questioned before it can change.
Questions such as how do emerging technologies mediate the learning processes, what are the pre-existing
concepts of what productive learning is and how it occurs in educational settings and what are the conflicts
between them, what is the teacher’s role in the change process, and what are the social and cultural norms and
expectations for education in relation to change? All of these questions presuppose an open and informed dialog
on the process of changing education in response to a changing world. Dialogs such as this IFETS forum which
brought together educational experts from all over the world to discuss the potential for new learning
technologies in constructivist settings is an important part of the process of conceptualizing the potential.
Andrews quote from Gibran is an excellent final thought for this dialog on reform in education, “Progress lies
not in enhancing what is, but in advancing toward what will be.
References
Aronson, E., Blaney, N., Stephan, C., Sikes, J., & Snapp, M. (1978). The Jigsaw classroom, Beverly Hills, CA:
Sage Publications.
Barab, S. A., & Duffy, T. (2000). From practice fields to communities of practice. In D. Jonassen & S. M. Land
(Eds.), Theoretical Foundations of Learning Environments (pp. 25-56), Mahwah, NJ: Lawrence Erlbaum.
Brown, A. (1995). The advancement of learning. Educational Researcher, 23 (8), 4-12.
Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading,
writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert
Glaser (pp. 453-494), Hillsdale, NJ: Erlbaum.
Clement, J. (1993). Using bridging analogies and anchoring intuition to deal with students' preconceptions in
physics. Journal of Research in Science Teaching, 30 (10), 1241-1257.
Greeno, J. G. (1997). On claims that answer the wrong question. Educational Researcher, 26 (1), 5-17.
Hinrichs, T. R. (1992). Problem solving in open worlds: A case study in design, Hillsdale, NJ: Lawrence
Erlbaum.
Jonassen, D. H., Peck, K. L., & Wilson, B. G. (1999). Learning with technology: A constructivist perspective,
Upper Saddle River, New Jersey: Merrill-Prentice Hall.
Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and
Development, 48 (4), 63-85.
Kolodner, J., & Leake, D. (1996). A Tutorial Introduction to Case-Based Reasoning. In Leake, D. (Ed.), Case-
Based Reasoning: Experiences, Lessons and Future Directions (pp. 31-65), Menlo Park, CA: AAAI Press.
14
Laffey, J., Musser, D., & Espinosa, L. (2000). Shadow netWorkspace™ Learning Systems Project. In Kinshuk,
Jesshope C., & Okamoto T. (Eds.), Proceedings of the International Workshop on Advanced Learning
Technologies (pp. 188-189), Los Alamitos, CA: IEEE Computer Society.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation, Cambridge, UK:
Cambridge University Press.
Lawson, B. R. (1984). Cognitive studies in architectural design. In N. Cross (Ed.), Developments in design
methodology (pp. 209-220), NY: Wiley.
Mestre, J. (1987). Why should mathematics and science teachers be interested in cognitive research findings?
Academic Connections, Summer, 3-5, 8-11.
Missouri Department of Elementary and Secondary Education (1996). The Missouri Show-Me Standards,
retrieved April 2, 2005, from http://www.dese.state.mo.us/standards/process.html.
Pea, R. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed.), Distributed
cognitions: Psychological and educational considerations (pp 47-87). Cambridge, UK: Cambridge University
Press.
Petraglia, J. (1998). Reality by design: The rhetoric and technology of authenticity in education, Mahwah, NJ:
Lawrence Erlbaum.
President's Committee of Advisors on Science and Technology (1997). Report to the President on the use of
technology to strengthen K-12 education in the United States, Washington, DC.
Resnick, L. B. (1983). Mathematics and Science Learning: A New Conception. Science, 220, 477-478.
Resnick, L., Levine, J., & Teasley, S. (Eds.) (1991). Perspectives on socially shared cognition, Washington, DC:
American Psychological Association.
Rogers, E. M. (1995). Diffusion of innovations (4th Ed.), NY: The Free Press.
Roschelle, J. M., Pea, R. D., Hoadley, C. M., Gordin, D. N., & Means, B. M. (2000). Changing how and what
children learn in school with computer-based technologies. Children and Computer Technology, 10 (2), 76-101.
Roth, W. M., & Bowne, G. M. (1995). Knowing and interacting: A study of culture, practices, and resources in a
grade 8 open-inquiry science classroom guided by a cognitive apprenticeship metaphor. Cognition and
Instruction. 13 (1), 73-128.
Salomon, G. (1993). No distribution without individuals’ cognition: A dynamic interactional view. In G.

Salomon (Ed.), Distributed cognitions: Psychological and educational considerations, NY: Cambridge
University Press.
Savery, J. R., & Duffy, T. M. (1996). Problem based learning: An instructional model and its constructivist
framework. Educational Technology, 35 (5), 31-38.
Schank, R. C. (1994). Goal-based scenarios. In R. C. Schank, R. P. Abelson, & E. Langer (Ed.), Beliefs,
reasoning, and decision making: Psycho-logic in honor of Bob Abelson (pp. 1-32), Hillsdale, NJ: Lawrence
Erlbaum Associates.
Schank, R. C., Fano, A., Bell, B., & Jona, M. (1994). The design of goal-based scenarios. The Journal of the
Learning Sciences, 3 (4), 305-345.
Shulman, L. (1992). Toward a pedagogy of cases. In J. H. Shulman (Ed.), Case methods in teacher education,
NY: Teachers College Press.
Vygotsky, L. (1986). Thought and language, Cambridge: The MIT Press.
Wertsch, J. V. (1998). Mind as action, NY: Oxford University Press.

15
Matusov, E., Hayes, R., & Pluta, M. J. (2005). Using discussion webs to Develop an Academic Community of Learners.
Using discussion webs to Develop an Academic Community of Learners

Eugene Matusov, Ph.D.
Associate Professor, School of Education
University of Delaware, Newark, DE 19716, USA
Tel: +1 302 831-1266
Fax: +1 302 831-4445
ematusov@udel.edu
Renee Hayes, Ph.D.

Visiting Professor, School of Education
University of Delaware, Newark, DE 19716, USA
Tel: +1 302 831-3096
Fax: +1 302 831-4445
rhayes@udel.edu
Mary Jane Pluta

Former University of Delaware student
pluta@udel.edu
Abstract
The World Wide Web has made possible an entirely new form of communication in the classroom:
asynchronous, public, non-sequential, and selective (Windschitl, 1998). However, it is unclear how
discussion webs can contribute to educational processes. Our research investigates the role of instructional
interactive webs in promoting among preservice teachers an "academic community of learners," defined as
an academic community that grounds inquiries and dilemmas emerging in their practice in an academic
discourse based on considering alternatives and providing argumentation and evidence for their claims
(Wells, Chang, & Maher, 1990). Based on this definition of a community of learners and concerns raised by
fellow instructors, we created categories and analyzed one class discussion web, coding a total of 1,124 web
entries of undergraduate students and their instructor to examine references they used, topics, genres, and
relationships with other messages. Our findings suggest that students' web postings were mostly very
sophisticated in that students were able to integrate outside references with new and enriching discussion
topics, thereby providing viewpoints alternative to and sometimes critical of those expressed by the
instructor and other students. These findings suggest that instructional interactive webs can be a useful tool
for promoting and building an academic community of learners.
Key words
Academic community of learners, Web discussion
Historical Background
Internet-based classroom applications have been enthusiastically adopted in various educational settings. By
2003 the majority of community colleges were offering on-line courses (Bagnato, 2004), 81% of all post-
secondary institutions offered some on-line courses, and a third offered entire on-line programs (Conhaim,
2003). Deakin University in Australia has recently begun to require that all graduates complete at least one on-
line course (Woodhead, 2004). To what extent and in what ways will this internet-technology boom transform
higher education? According to Windschitl (1998), the Internet may enhance classroom learning in one of two
ways: it may improve access to information or expand potential for communication. This expanded potential,
however, is not simply increased efficiency. Wertsch points out that the internet, as a tool mediating
communication, will not simply extend or improve existing forms, but will necessarily make qualitative changes
to the nature of communication (Wertsch, 2002). Bonk, Appleman, and Hay (1996) pose important questions
concerning the use of internet-based tools: “How might these tools encourage learners to explore and
accommodate alternative viewpoints?” and, “How does one gain a greater sense of intersubjectivity and common
ground through computer technologies?”
While web-based technology offers an opportunity to change the nature of classroom communication, often this
new technology is used only for facilitating low level tasks such as posting the syllabus, distributing lecture
notes, or administering multiple-choice exams (Jack A. Cummings, Curtis Jay Bonk, & F. Robert Jacobs, 2002;
Marx, Blumenfeld, Krajcik, & Soloway, 1998; Means, 1994; Putnam & Borko, 2000). In some cases, the
16
technology is used for “disseminating conceptual information and testing conceptual knowledge” (Kidwell,
Freeman, Smith, & Zarcone, 2004, p. 149) in much the same way as in face to face instructor-student
interactions, but across greater distances. In some cases, print materials (such as textbook and lecture notes) are
presented multi-modally to accommodate different cognitive styles (Brown & Liedholm, 2004) or to provide
easier access to these materials (Jensen-Lee & Falahey, 2002). Nevertheless, a Power Point presentation
available on-line is no different from a passive classroom lecture (Boylan, 2004). This “pre-packaged print-based
component” of much on-line learning may very well derive from the institution context (Kuboni & Martin, 2004,
p. 28). Rather than focus on the “business as usual” aspect of current practice, however, it is much more useful to
focus on the innovative “outliers,” which may provide guidance for how web-based teaching may provide new
kinds of interactions among students and teachers in the classroom (Jack A. Cummings, Curtis J. Bonk, & F.
Robert Jacobs, 2002) Specifically, the purpose of this study was to examine whether and how interactive
discussion webs can support the process of developing an academic community of learners and, thus, enhance
students’ learning.
Koschman (1996) argues that the use of computer-mediated communication (CMC) in education has been
guided by diverse psychological paradigms such behaviorist, information processing, Piagetian, and
sociocultural. Each of this paradigm replies to the question of the role of internet-technology in education
differently. Our framework stemmed from a sociocultural approach assuming that learning involves a
transformation of participation in a community of practice (Lave & Wenger, 1991). The theoretical roots of this
investigation, however, began with instructors’ concerns about using discussion webs. At the time of the writing
of this article, the first author has used discussion webs in 32 classes, which he has taught at three different
universities 24 of which classes targeted preservice teachers. The second author has used discussion webs in 8
classes. The first author has also given many workshops to faculty about the effective use of web discussions in
college courses. While giving these workshops, the first author of this paper compiled a collection of the
practical concerns expressed by university faculty about ways in which the discussion web might functionally
fail to meet instructional goals. We analyzed these concerns to extract the implicit assumptions of what defines a
successful classroom discussion web. We found that these assumptions are reflected in Wells, Chang, and
Maher’s definition of “an academic community of learners” as an academic community of inquiry that grounds
inquiries and dilemmas emerging in their practice in an academic discourse based on considering alternatives
and providing argumentation and evidence for their claims (1990). This notion is very compatible with Putnam
and Borko’s (2000) notion of “a discourse community for preservice teachers” that focuses on the development
of a “teaching culture” (Stigler & Hiebert, 1998) where means of teaching are “polished” by efforts of all
community and not left to individual teachers. Discussion-webs explicitly based on the notion of creating a
virtual teaching community have reported success in terms of providing a common discourse as well as
discussing issues emerging directly from practice (Wiske, Sick, & Wirsig, 2001). Salmon argues that the web-
based discussion can facilitate reflection-in-action, a collaborative reflection that take places “in the midst of
practice”(2002, p. 380). In one survey of preservice teachers, over 90% reported that participation on an online
web discussion allowed them to benefit from peer discussions while engaged in their practicum (Khine &
Lourdusamy, 2003). Goodfellow (2004) cautions that this notion of a virtual learning community is not
completely unproblematic, since socialization into this community, as in any institutional setting, may involve
some degree of social conflict and institutional resistance. With this in mind, our analysis focuses on the nature
of the student interactions as well as the content of the postings.
Research questions
We extracted from Wells, Chang, and Maher’s definition for an academic community of learners four key
defining aspects, which corresponded to the four major concerns expressed by instructors. We then recast these
as research questions, which in turn led to coding topics from which emerged our coding categories for data
analysis. These coding categories will be explained in more detail in the coding section of this paper. Below we
describe the four main aspects of a community of learners, research questions based on these, and the resulting
coding topics (see Table 1).
Table 1. Research questions and coding topics

Aspect of a community of learners Research question Coding topics and variables
Intersubjectivity: Shared focus on How did the topics of students web Type of topics brought to web by
academic themes postings relate to course topics? students
Ontology: Integration of Did students refer to their lives Type of explicit reference in
participants’ lives into community outside the class in their web student postings
practice postings?
17
Problematicity: Defining/addressing Did students both define and address Genre of problems defined/genre
professional inquiries issues, concerns, and dilemmas on of problems addressed
the web?
Dialogicity: Academic/Dialogic Did students both support and Type of relationship of student
relations among web postings of challenge each others’ and posting to previous posting
different authors instructors views?
Motivation: Ownership of the web Did the students go beyond extrinsic Number of postings per week and
discussion motivation of fulfilling the word length of their postings
classroom requirements while
participating in the web discussions?
As mentioned earlier, we also spoke with colleagues to understand what they considered to be potential sources
of web talk failure. We took these into account, and for each aspect of a community of learners, we used these
related instructor concerns to define potential web failure as well as web success (see Table 2).
Table 2. Web discussion functional failures and successes

Aspect of a Web discussion failure Web discussion success
community of learners
Intersubjectivity: Chit-chat Shared focus on the academic themes
Shared focus on
academic themes
Ontology: Integration Assignment board Integration of the community practice and other (non-
of participants’ lives community) aspects of participants’ lives in an
into community academic discourse
practice
Problematicity: Shallow contributions Defining and addressing professional/academic
Defining/addressing inquires in individual web postings
professional inquiries
Dialogicity: Collective monologue Academic dialogic relations among web postings of
Academic/Dialogic different authors
relations among web
postings of different
authors
Motivation: Ownership Minimum efforts Ownership of the web discussion
of the web discussion
In the following section we discuss in more detail the four aspects of a community of learners, the potential web
success and failure associated with each, and some theoretical grounding for the relevance of these issues.
1. Intersubjectivity: Shared focus on academic themes
What were the themes of students’ web discussions? How did the topics of the students’ web postings relate to
the courses’ topics? Did the class web discussions successfully create “intersubjectivity in a shared social space”
(Bonk, 1998; Schrage, 1990) and shared collective reality (Gallimore & Tharp, 1992) relevant to the course foci?
Intersubjectivity is referred to herein as shared themes, problems, and approaches to solve them coordinated by
the participants (Matusov, 1996). During the workshops, some instructors expressed concern that the web
discussion could deteriorate into "chit chat," wherein students socialize through conversation having nothing to
do with the class focus. Coding topic: “Type of Topics Brought to Web by Students”.
Some web research has claimed that the professor not permit off-topic postings (Lim & Cheah, 2003), and that
webtalk success depends on strict structure and guidance provided by the professor (Lobry de Bruyn, 2004);
“student participation and learning benefited as the professor actively charted a preliminary course for the
lessons based on prior discussions and the objectives of the course” (King, 2002, p. 241). We wanted to see if the
students would themselves provide this focus on course objectives without the instructor’s explicit constraint. It
is important to notice that although the instructor may not impose any explicit constraint for the classroom
shared focus on web discussions, it may be placed implicitly in the minds of the students who may actively
18
interpret the institutional norms and the instructor’s intentions and may be concerned about their potential
consequences for the students’ well-being and professional future.
2. Ontology: Integration of the community practice and other (non-community) aspects of participants’
lives in an academic discourse
Did the students refer to dilemmas and issues coming from their lives outside of class? Did the students provide
references to the literature (both read and non-read in the class) and other classes? Instructors expressed concerns
that the class might be reduced to an assignment board wherein students, worrying mostly about grades, might
try to guess what the instructor wants from them and narrowly address the class agenda defined by the instructor
without initiating their own topics and integrating them with a broader context. Coding topic: “Type of Explicit
Reference in Student Postings.”
Windschitl (1998, p. 31) makes a point that “communication scholars have studied how students learn from
various arrangements of text and graphics on a computer, but these studies have focused on communication
whose purpose is to instruct users, not to relate to them on a personal level.” Traditionally, institutionalized
learning occurring within institutional academic settings has typically involved a two-stage sequence. During the
first learning stage, a student has to learn decontextualized skills and knowledge; during the second stage, the
student should learn how to apply those skills and knowledge in different practical contexts. This model of
learning has posed the issue of “transfer” -- what the student has learned in one context (e.g., abstract,
decontextualized) has to be applied in another context (e.g., concrete, practical). Unfortunately, this “transfer”
often does not occur in schools (Anderson, Reder, & Simon, 1997).
Alternatively, a sociocultural view of learning is a holistic approach and involves a process in which students’
experiences and reflections are woven within a theoretical framework which promotes reflections in conjunction
with one another. The sociocultural notion of learning defines it as a whole-person-learning-in-activities
penetrating all aspects of a student’s life (Cole, 1996; Lave, 1992; Palmer, 1998). This understanding of learning
is very compatible with Wells, Chang and Maher’s notion of an academic community of learners.
Many discussion webs analyzed in the literature have as their goal a specific task of tasks assigned by the
instructor, that is, either coming to an understanding target concepts (Lobry de Bruyn, 2004; Orvisa, Wisher,
Bonk, & Olsond, 2002), completing a task (Kidwell et al., 2004), or solving a problem (Orvisa et al., 2002; Rose,
2004). By contrast, we partially defined success of the classroom web discussions by the ontological nature of
the students’ web contributions -- the degree to which students were able to introduce their own extra-curricular
references into the web discussions. In the case of preservice teachers, it is especially beneficial to read about
what other students are experiencing in their teaching practicum, as well as how teachers in the field address
classroom situations that they may encounter themselves someday. The totality of a student’s learning
experiences is much greater and far more expansive and comprehensive when integrated with issues from
outside the class, particularly pedagogical issues encountered in one’s teaching practicum. Learning is enhanced
and a natural byproduct of such a situation is that course content is expanded beyond that which the text and
lecture alone are able to provide (Bonk, Daytner, Daytner, Dennen, & Malikowski, 1999).
3. Problematicity: Defining and addressing professional/academic inquires in individual web postings
Did the students bring issues, concerns, and dilemmas to the web discussions and try to address them? Teacher
electronic networks have been criticized for their lack of critical reflections (Putnam & Borko, 2000). Instructors
expressed concerns that students might provide only shallow contributions, being unable either to define
important and relevant issues or to address these issues in depth. Coding topic: “Genre of Problems
Defined/Genre of Problems Addressed in Student Postings.”
In the literature on innovative education, the notion of “community of learners” is often defined as shared
ownership for both problem-defining and problem-addressing processes (Bonk & K.S.King, 1998; Dewey,
1966; Lave, 1988; Rogoff, Matusov, & White, 1996; Wells et al., 1990). In Moss (1998), there were concerns
that practicing teachers did not develop an appreciation for “reflecting on their practice through a problem-
solving model.” Web-based learning communities, have the potential to strengthen connections between
classroom and real world learning (Johnson, 2001). Unlike case-based teaching, where students are assigned
cases for discussion and success is defined by the degree to which student discussion remains oriented to these
assigned tasks (Schellens & Valcke, 2005), we wanted our students to define for themselves problems that were
19
important to them as developing members of the teaching community. In our analysis we hoped to find evidence
that students were able to both define problems in their field and engage in meaningful attempts to solve them.
4. Dialogicity: Academic dialogic relations among web postings of different authors
Did the students provide support and challenge to each other’s and the instructor’s statements on the web? Did
they present supporting examples, evidence, and personal experiences for their claims? Did they provide
alternative views? Piaget (Piaget, 2002) defined a conversation consisting of unrelated or loosely-related
statements as a "collective monologue." In contrast, we hoped to find the type of engagement described by
Bakhtin (1990) as "dialogic relations." Dialogic relations between participants’ utterances bounded by dialogic
turns involve participants’ reactions toward what has been said before by the others. During the workshops,
some instructors expressed concerns that students would post messages that are either unrelated or loosely
related to each other. Coding topic: “Type of Relationship of Student Posting to Previous Posting.”
The meaning making process involves a response to the questions of others. Meaning is a dialogic relationship
between the question and the response. As Bakhtin wrote, “Understanding comes to fruition only in the
response; understanding and response are dialectically merged and mutually condition one another; one is
impossible with the other” (Bakhtin, 1991, p. 282). Yang and Lui’s analysis of a professional development web
discussion for teachers found that messages were more like board postings than true conversations (Yang & Liu,
2004). Pena-Shaff and Nicholls found that students tended to agreeing or disagreeing with a previous posting
and then engaged in an extensive monolog where they elaborated their view, and that they very rarely returned to
defend or elaborate on a statement which had been challenged by another (Pena-Shaff & Nicholls, 2004).
However, Bonk (1998) found that students provided mentoring, questioning, scaffolding, feedback and task
structuring for one another and that they were quite adept at exploring and articulating ideas for each other in
their class web discussions. These characteristics mirror Latour’s notion of changing “modality,” a core
characteristic of scientific discourse (Latour, 1987). Studying scientific discourse, Latour (1987) found that one
of the most important functions of the discourse is to change the modality of statements that scientists make.
According to Latour, modality of a statement is elevated when the statement is supported by the author such a
way that the statement sounds more like a “fact” by de-emphasizing the statement’s authorship and by
objectivizing its content (e.g., “The Earth rotates around the Sun” – we are not informed by the statement about
who is making this claim and on what grounds). In contrast, modality is lowered when a scientific statement is
undermined by the author in such a way that the statement sounds like an “artifact” because it emphasizes its
authorship shaped by a false subjectivity (e.g., “Aristotle confused force and movement”). These changing
modalities, a requisite of dialogic exchange, are an important aspect of an academic community of learners that
we hoped to find in our web discourse.
5. Motivation: Ownership of the web discussion
Did the students go beyond the extrinsic motivation of fulfilling the classroom requirements while participating
in the web discussions? During our workshops about the use of webs in classrooms, some instructors raised a
concern that the students will try to minimize their efforts and do the minimum required or even less so. To
examine this issue, we coded number the students’ actual postings per week and length of the postings.
One of the core features of a community of learners is intrinsic motivation and shared ownership for learning
activities (Bonk & K.S.King, 1998; Dewey, 1966; Lave, 1988; Rogoff et al., 1996; Wells et al., 1990). A
sociocultural approach to motivation stresses the social, participatory nature of motivation. It values the
development of intrinsic motivation that provides dynamic access to changeable sociocultural practices.
Motivation implies openness, making choices, taking the risk and responsibility for choices, cooperation, sharing
interests, and quality of participation as the outcome of learning. These feature are manifested in students’ doing
beyond what is required by the instructor.
Method and Procedures

Participants and Contexts
We analyzed the discussion web created during a 3-credit course entitled “Instructional Strategies and Reflective
Practices” (referred to as EDST390) taught by the first author in the Spring Semester of 1998. This course was
20
conducted over a 16-week period, including a week of spring break and a final exam week (although there were
no exams for this class). As stated in the syllabus, the instructor’s purpose for the class was:
“to examine our perception and priorities regarding teaching and learning. I hope that such
reflection will promote experiential knowledge that can help you to become a schoolteacher, who
fosters meaningful education for all your future students. In this class, we will focus on how to
develop teaching goals and priorities for instruction and ‘in flight’ decision making in the
classroom to promote active learning in students.” (EDUC 390 course syllabus)
The course was a part of the Elementary Teacher Education Program and was a mandatory core class for
preservice teachers. The course was also connected with a methods class on teaching literacy (referred to as
EDDV305). As part of the class, students were involved in an 8-week teaching practicum in an elementary
school, wherein the students were required to make observations on teaching and to prepare lessons on teaching
literacy (reading and writing). The class and web discussion during practicum time was an important way for
students to meet together to reflect up experiences they had in different settings; providing a common
community of practice for students actually engaging physically distant practices (Buckingham, 2003).
There were 23 students from several eastern states, most of whom were and white middle-class females in their
early twenties. The class had one African-American student, one Latino student, two male students, and two
returning adult students who had returned to college after having had professional careers in other areas. The
instructor (first author of this paper) was a white male Russian immigrant in his thirties. The third author of this
paper was a student in the class. None of the students had prior experience with web discussions.
The class had a seminar format in which class time was devoted to the instructor’s presentations, group
discussions of readings, preparation and presentation of group projects, and open class discussion. The course
also required participation on the class web, used for posting and completing assignments as well as a free
discussion forum called "WebTalk." It is this WebTalk, the unstructured discussion web created by the students
and teacher during the course, which is analyzed in this investigation.
The Interactive Discussion Web (WebTalk)

Although the instructor provided immense freedom to the students in initiating and supporting as many
discussion threads as they deemed appropriate and necessary, he also designated the minimum number of web
postings that students had to write on the web. He required each student to read all postings and to contribute at
least two messages on the web per week (either an initiation of or a reply to a discussion thread). The required
number of postings was comparable with the average number of postings occurring in web discussions in which
there is no requirement of mandatory participation. Thus, Dysthe (2002) reported that during her 2-week class
web discussion 9 out of 10 students generated 27 postings. In our studied class, there were also four structured
reading groups and each group was required to make one posting per week concerning a topic discussed during
class. The instructor monitored each student’s progress in terms of total number of web postings and provided
the class with a confidential means of viewing a progress report indicating whether or not the student had
fulfilled the course requirements, including the number of weekly postings to the Web Talk. Students behind in
their postings could fulfill the requirements of the course by doing additional web postings. There were no web
postings required during the following weeks: week 1 (orientation to the class), week 8 (Spring Break), week 15
(last week of class), and week 16 (the exam week).
The instructor asked that the messages be broadly relevant to the class and encouraged students to provide
reflections on the following activities: class presentations, practicum experiences, literature, feedback on the
class, personal experience relevant to the class discussions, information sources that facilitate other students’
learning, and personal and career development. Students had an opportunity either to initiate a new topic or reply
to somebody else’s posting and, thus, to continue (or alternate) an existing discussion thread. Each message was
indexed in the hypertext content table. Each reply was automatically indented as reflected in Figure 1.
The hypertext content table provided information about the message title, author, and date and time of the
posting. The privacy of the students’ Internet web discussions was protected by a password available only to the
members of the class. The instructor used Microsoft Front Page 98 to design the class web.
21
Figure 1. Class discussion web
Similar to Dysthe’s (2002) pedagogical design, postings were not graded or evaluated for content. We decided
that grading the content would restrict the freedom of postings and change the medium from student-oriented
discussion to instructor-oriented assignment. In the words of one instructor, her initial choice to grade web
postings in her preservice teacher discussion forum turned out to be “counterproductive to reflectivity”
(Wickstrom, 2003). There was no requirement concerning the length of a posted message; however, the
instructor advised the students to keep postings brief to save other people time in reading. The genre of the
participants’ writing was similar to Dysthe’s (2002) definition of a “hybrid genre” (p.341) which involves a
combination of “freewriting” (Elbow, 1973) or “writing-to-think” (Dysthe, 2002) and presentational writing
aimed at the audience of the classroom community. Students were guided to focus on clarity of communication
rather than grammatical correctness. During the web orientation on the second day of class, the instructor
“modeled” how making a grammatical mistake in his own posting did not alter the content of the posting and
thus, an informal atmosphere of web participation was promoted (Wegerif, 1998). This pedagogical design was
aimed to develop “relaxed norms of coherence” that are important for the successful functioning of computer-
mediated communication (Danet, Ruedenberg-Wright, & Rosenbaum-Tamari, 1998; Herring, 1999). Students
tend to assume they are constantly being evaluated, and it is up to the instructor to eliminate this assumption of
quality control (Schallert & Reed, 2003-2004).
Web Talk Pedagogical Design: Theoretical Underpinnings

While many class discussion webs are designed to enable students to participate in teacher-initiated problems-
solving (Orvisa et al., 2002; Rose, 2004), our particular Web Talk was designed to promote student-initiated
discussions of preservice teachers’ class and practicum experiences focusing on how they themselves view and
learn new ways of teaching (Putnam & Borko, 2000). Following Bakhtin (1999), we believe that truth only
exists in dialogue rather than that truth emerges from a dialogue. In contrast to some educators who view
engagement of the learners in interactions with others as an enhancement of their learning processes (see, for
example, Laurillard, 1993), we consider social interaction as the necessary medium in which learning occurs and
knowledge exists. Our pedagogical goal fits a recent trend of the use of electronic discussions to promote
collective reflections among preservice teachers wherein they are guided by the instructor about topics of interest
or problems seen in the schools (Admiraal, Lockhorst, Wubbels, Korthagen, & Veen, 1997; Bonk, Malikowski,
22
Angeli, & Supplee, 1998; Whipp, 2003). For preservice teachers, electronic conferencing has been implemented
in an attempt to alleviate "(1) the isolation students feel when in the field; (2) the lack of community and
dialogue among teacher education participants; (3) the disconnectedness between classroom knowledge and field
experiences; (4) the limited reflective practices observed among novice teachers; and (5) the need to appreciate
multiple perspectives and diverse cultures.” (Bonk et al., 1999). Our Web discussion is part of a multi-modal
design sometimes referred to as “blended learning” (Garrison & Kanuka, 2004; Khine & Lourdusamy, 2003) or
a “hybrid course” (Brown & Liedholm, 2004) where the advantages of on-line asynchronous discussion (such as
freedom from space and time constraints) are coupled with the advantages of classroom discussion (such as
spontaneity). Furthermore, the immediate physical availability of the instructor during class meetings and office
hours also helped to alleviate the technical frustrations and feelings of alienation identified in some exclusively
web-base distance learning seminars (Hara, 2000). In this sense, our web design most closely mirrors the BBS
(electronic asynchronous bulletin board system) described by Pena-Shaff and Nicholls, where professor and
teaching assistant participate in the discussion largely sustained by students (Pena-Shaff & Nicholls, 2004). This
design was primarily based on the accumulated teaching experiences of the first author, as well as on his
conversations with other instructors and students, including reflections on the nature of communication desired
in the classroom and the potential and restrictions inherent in various forms of communication.
The instructor had a special role on the web discussion. This role involved guiding the preservice students in
how to analyze pedagogical actions regarding classroom instruction, classroom management, communication
with parents, and so forth by considering and inviting the students to consider the short- and long-term teaching
goals and priorities, the complexity of desired and undesired consequences, educational philosophies, cultural
values and assumptions about education, and institutional and historical contexts and constraints. The instructor
also pushed the students to provide justification, clarification and examples for their ideas so that other class
members would follow and visualize the situations and problems discussed. Finally, the instructor modeled for
the students how to create a web community through providing supportive and respectful criticism by: inviting
other class members to join and continue the discussion (e.g., all instructor’s postings were ended with the
question, “What do you think?” – a practice that many students eventually adopted); bringing examples of
successful and unsuccessful teaching from his own personal teaching experiences; supporting “non-popular”
dissent and minority views; inviting and supporting students’ criticism of his own ideas; considering pros and
cons for each idea; and supporting students who had not received replies from other students at the beginning of
the semester. While other instructors have chosen not to participate in the discussion with the goal of
encouraging student-student interaction rather than student-teacher interaction (Pinch & Graves, 2000; Rose,
2004), our experience has been that a careful instructor presence could enhance, rather than inhibit, student
dialogue.
Data Coding and Analysis

It is only recently that web discussions have begun to be analyzed using qualitative methodology, and early
attempts to create coding categories have been vaguely defined and difficult to work with (Pena-Shaff &
Nicholls, 2003). In our study, we chose to derive our research questions and coding categories from our own
pedagogical experiences and concern with the development of an academic community of learners. To address
the main research question of whether or not web discussions may support the development of a community of
learners, the primary focus of our analysis centered on 859 student postings. The excluded messages, such as
assignments posted by the instructor and assignments completed by the students, were deemed to be "assignment
board" postings and not considered for purposes of the analysis. Many studies of web discussions have focused
implicitly on characteristics of individual postings as the unit of analysis, such as degree of critical thinking
(Henri, 1992), cognitive complexity (Christopher, Thomas, & Tallent-Runnels, 2004), or task orientation
(Nicholson, 2002). Nevertheless, recent studies focusing on the web’s potential to foster community have begun
to consider the relations among postings (Dringus & Ellis, 2005; Makitalo, Hakkinen, Leinonen, & Jarvela,
2002). In this spirit, we have noted an emerging focus on the nature of interactions, using inter-posting
relationships as a unit of analysis (Pena-Shaff & Nicholls, 2004). Since we were concerned primarily with
community development, we used two units of analysis: individual postings and the relationship between
postings written by different authors. As described earlier in this paper, research questions reflecting both
characteristics of an academic community of learners and potential failures were recast as coding topics for
analysis. Based on these topics, postings were coded according to the following nine categories, and the results
were compiled in a database:
1. date
This is the date that the posting was submitted to the Web Talk.
2. word length
23
This is the total number of words in a single posting, excluding the title.
3. author
Students were required to sign their postings so that class members were aware of who they were addressing.
4. thread position
This refers to the position of the posting in the discussion thread (web discussions around a certain topic).
Participants chose whether to start a new thread or respond to a posting within an existing thread. Postings were
organized visually into threads in such a way that postings were situated directly underneath the posting to which
it responded, and indented.
5. number of replies to the posting
This includes all postings that either replied directly or replied to another posting that replied to the initial
posting. In other words, the total number of replies includes all postings situated beneath the posting in its
discussion thread.
6. topics discussed
This category indicates the thematic foci of the students’ web postings and includes (but is not limited to) the
following: discussion of educational philosophies and instructional strategies, classroom management and
discipline, co-operating practicum teachers, aspects of self-concern, self-confidence, self-management, career
and what students should do in specific situations, issues of diversity and special education, professional issues
of how to become a teacher, issues of testing and learning assessment, and so forth. For example,
I would just like to comment on the issues of disciplining. Through all of my elementary
education, the teacher was the one who made the rules of his/her classroom and it was the student's
job to obey them or who knows what would happen, your name on the board, stay inside with the
teacher at recess, etc. These are some examples of what would happen in my previous elementary
school and they are also the rules of my cooperating teacher. These are the rules that I have really
only seen in my past and in field placement; In my opinion, they are not that bad of rules.
…(student posting coded as classroom management).
This category can help to address the question of whether topics discussed by the students on the web
discussions were relevant to the class (i.e., the chit-chat concern).
7. explicit references
This category includes other postings, the teaching practicum, class discussions, literature experiences, out-of-
class life experiences and so forth. For example,
I just read Melanie’s message and I too got an excellent evaluation from my cooperating teacher.
(coded as having references to both another student’s posting and to the teaching practicum).
This category was designed to address the question of whether the students brought broader issues and topics
from outside of the class to the web discussions (i.e., the assignment board concern).
8. posting genres
This category includes students expressing concern, surprise, dilemma, complaint, statement, sharing positive or
negative feelings, asking for advice, providing suggestions, raising or addressing a question and so forth. For
example,
The teachers that I worked with were not even aware of their responsibilities to us…We had a
thirty second discussion prior to my teaching, and then NO discussion following. I am basically
disgusted with this situation and I can only pray that my cooperating teacher for student teaching is
enthusiastic about helping out and getting involved. …(coded as complaint).
Another example is,
The article on rewards really surprised me. To me rewards were a good idea. I thought that
children enjoyed getting them I was surprised to hear that when children were told that they were
going to get a reward that they did not work as hard as the kids who were not going to get a
reward….(coded as surprise).
This category was aimed to address the question of whether the students brought problematicity on the web
discussions (i.e., the shallow contributions concern).
9. relationships to the previous posting
Postings were considered to be either supporting or challenging the original author’s message. This category
includes explicit agreement, explicit disagreement, aligning with the author’s idea or experience, elaboration,
explicit encouragement, acknowledgement and appreciation of the author, presenting alternative ideas or
opposite case, request for clarification of ideas, and conflict. Consider the following exchange between two
students:
The article on rewards really surprised me. To me rewards were a good idea. … I still think that
some kids like being motivated this way. If it gets them to learn what's the problem? I guess the
teacher has to monitor this for long term effects. This is just my opinion.
Four hours later (7:45pm), a student replies:
24
I think the problem they had with rewards is that the students aren't retaining the information they
should be learning. They only try to get the reward, and are rushing through the work to get
pleasure from the reward, and not from learning. Were you pleased because you were learning or
were you pleased because you were acting as you were expected? This is the question they are
trying to attack in the article. Does this help? (coded as disagreement and providing alternative
views).
This category can help to address the question of whether the students relate to each other’s postings on the web
discussions (i.e., the collective monologue concern).
The coding was not mutually exclusive within or between categories. For example, the posting above was coded
as both "explicit disagreement" and "providing alternative views," both of which are options within the category
relationships to previous posting. In addition, this particular posting was also coded across all other categories,
for example as "providing suggestion" in the category of posting genre.
To access students’ “intrinsic motivation” for posting through how lengthy students’ posting were, we randomly
selected 100 messages from the XMCA e-mail list of professional academic researchers interested in issues of
learning, education, and human development archived at http://communication.ucsd.edu/MCA/Mail/index.html.
We measured the number of words in these messages and found the median. Student postings were considered
“extra long” when the number of words exceeding this median.
All data was coded by a coder “blind” to the research questions. About 20% of the students’ postings (170
postings) were also coded by the first author. The inter-coder agreement on the categories ranged between 89%
and 95%. The database created thereby has enabled us to compile research data in order to assess the ability of
this particular class to promote asynchronous communication and interactivity by, between and among the
students and the instructor. It is for this reason that only four of the nine coding categories (topics discussed,
explicit references, posting genres, and relationships to previous postings) are used for analysis in this paper. It is
expected that this paper will provide a framework for discussion that will lead to questions for further analysis.
Findings and their Discussion

A: Intersubjectivity: Did the Web’s topics focus on academic educational issues central to the course?
The students had a great deal of freedom in terms of the themes and topics encompassed by their web postings.
The overwhelming majority of the discussion threads was initiated by the students (365 postings or 97% of all
postings’ initiations), indicating that nearly all topics were brought to the web discussion by the students and not
by the instructor. Since there was nothing in the instructional design, as articulated by the instructor, that
prevented the students from freely defining themes and topics, we wondered if this freedom might allow students
to stray away from the focus of the course and use the web discussion for their chitchat socialization.
We sought to determine what topics were discussed by students on the web and whether these topics were
relevant to the purpose of this course as described in the EDST 390 course syllabus, “In this class, we will focus
on how to develop teaching goals and priorities for instruction and ‘in flight’ decision-making in the classroom
to promote active learning in students” (Course syllabus, p.1). To this end, we coded each posting within the
topics discussed category described above, and graphed the type of topic by frequency. We then compared the
topics brought to the Web Talk by the students to the focus of the course as described in the course syllabus by
the instructor.
By far, the most frequent topic was “educational philosophy and instructional strategies” (58%). The following
student web posting illustrates this topic:
I just had a thought about what to do about the advanced students in the class - how do I handle
them when I'm conducting a lesson and they know every answer to every question, and they don't
let other students have a chance to answer? I have a handful of students in my 1st grade classroom
that are on a 5th grade reading level. They always finish their work ahead of time, and it seems
that they are always the first to call out the answers to questions I have for the whole class. This
makes the other children want to participate less because they know that the smart kids will come
up with the answers first. I did my read - aloud lesson on Friday morning and I prepared a series of
pre-reading and post-reading questions for the students. Right away, 2 of the advanced kids called
out telling me that they had already read the book and spoiled the ending to my story. This
bothered some of the other students who really wanted to hear the story. I didn't know weather to
25
say something to them or not, seeing how I had only been in the class 3 times. I kind of shhhhh-ed
those 2 students and continued with my story, even thought I was a little discouraged. I know that
things won't run smoothly all the time, but I was hoping that my first experience teaching this class
would have been a bit better.
The Web as a Medium for Preservice Teachers to Discuss Educational Issues:

Type of Topic by Frequency
70%
60%
50%
40%
30%
20%
10%
0%
rs
e
t
t
er
ns
rn
ns
s
n
d.
en
en
em
us
io
in
he
.E
ce
ch
io
io
m
em
ct
ck
Ab
at
at
ac
bl
ea
on
ec
ru
lve
ra
rs
ro
el
ag
Te
st
ld
C
/T
R
vo
ve
lP
/S
l f-
In
hi
an
ts
p
s'
In
on
C
Se
ty
y/
a
e
-O
M
en
id
t
m
ph
si
nt
C
lK
Co
m
m
of
ci
co
er
re
so
c
So
oo
ss
iv
na
pi
Be
s
Pa
lo
ct
To
se
sr
so
hi
pe
to
of
as
As
ff-
lP
er
As
ow
es
Cl
O
rp
s/
na
su
st
te
H
io
Te
In
Is
at
uc
Ed
Figure 2. Web as a Medium for Students' Classroom Discussion
Other topics ranging in frequency from 26% (classroom management) to 3% (child abuse) constitute legitimate
subtopics of the primary course focus. Thus, the frequency distribution of students’ web themes reflects the
instructional focus as articulated in the course syllabus, that is, the topics most frequently brought to the Web
Talk by students reflect the course purpose, suggesting that students’ and instructor’s shared ownership for the
class thematic focus was supported by the Web Talk discussions.
By contrast, only 3% of all students’ postings could be categorized as “off-topic,” which is consistent with
similar research (Bonk et al., 1998) that concluded that “the off-task behaviors were essentially nonexistent.”
Curiously, similar research into a synchronous chat among military trainees found that up to a third of all
postings were social rather than task oriented; and that the patterns of off-task chat followed the patterns typical
of face-to-face collaborative interactions (Orvisa et al., 2002).
Here are three examples of “off-topic” posting:

1. Good job. Nice message girls!
2. I just wanted to wish everyone luck next week on teaching their special education units. It is definitely a
stressful period, because so much is going on. Keep your heads up and remember that this semester is
almost over.
3. Hi everyone, I just wanted to tell you all to have an awesome summer -- we DESERVE it!!! This
has definitely been a hectic year and we all have worked so hard, so live it up for the next few
months while you can! I will be student teaching next semester, then I'm all finished. But I'm sure I will
see you sometime in the near future. Good job on the presentations and the party on the last
day was great! Good luck to everyone and I hope all goes well for you guys in Fall.
The issue of “off-topic” web postings is ambiguous and complex. Like these three postings cited above, many of
the web messages that we coded as “off-topic” involved students’ encouragement and emotional support of each
other. While some researchers on class electronic discussions express a negative attitude to students’ expressions
of “social acknowledgments” on the class web (Bonk et al., 1998), Salmon considers online socialization to be
an important aspect of the discussion, and that early in the discussion the professor should encourage
socialization through “sensitive and appropriate” design and participation (Salmon, 2000, p. 28). We agree that
students’ socialization with each other is often very important for developing a feeling of community that may be
an especially important part of socialization for female students – the overwhelming majority of the class which
26
is the subject matter of this investigation. Similar points are made in feminist psychological literature (Gilligan,
1993), social psychology (Forsyth, 1998; Gillette & McCollom, 1990), and literature on conversation analysis
(Jefferson, Sacks, & Schegloff, 1987). One researcher seemed surprised that adult students surveyed valued
highly these affective messages, “It is easy to assume that most of the learners in online university-level courses
are adults, and therefore do not need to be affectively validated… However, our interviews with learners
provided data that this is not true.” (Blignaut & Trollip, 2003, p. 161). Other studies have further supported the
importance of internet-based communication channels solely intended to provide social support (Herring, 1999;
Nicholson, 2002). Thus, it may even be undesirable to have zero “off-topic” web postings.
B: Ontology: Did the web allow students to bring and to integrate in-class experience with out-of-class
experience?
Being able to bring and use observations, experience, information, readings, and opinions from different and
even unassigned sources is a hallmark of an authentic academic community of learners, as defined and
developed within the sociocultural theoretical framework described above. We found that the students referenced
diverse sources in their writings to the web discussions. We divided the references into explicitly in-class
references (other web postings, teaching practicum, and the class itself) and explicitly out-of-class references
(other classes, life experiences outside the class, non-class references, and discussions outside the class).
According to our analysis, 85% of all students’ web postings had explicit in-class references (such as to other
students’ and instructor’s postings, classroom discussions, assigned readings, class assignments, and practicum
experiences); 40% contained explicit out-of-class references (such as other classes, life experiences, readings
outside the class, discussion with friends and relatives about the class topics, and so on); and 6% had no explicit
references. (Please note that these are not mutually exclusive categories, it is quite possible for a posting to
contain both in-class and out-of-class references).
Our coding of the references expressed in the students’ web postings was conservative – we coded only explicit
references when the students clearly marked the source of their message. It seems safe to assume that the
messages had more tacit references than we coded. Figure 3 shows the frequency distribution of in-class and out-
of-class references.
Web as an Integrator of Students' In-class and Out-of-class Experiences: Type of Explicit Reference by Frequency
50%
45%
In-Class Out-of-Class
40%
35%
30%
25%
20%
15%
10%
5%
0%
Other Postings Teaching Practicum Class References Other Classes Life Experiences Non-class references Discussion Outside
Outside The Class Class
Figure 3. How Web Integrates Students' In-Class and Out-of-Class Experiences
At times students made explicit connections between in-class and out-of-class references, but we feel that even
without explicit connection, these in-class and out-of-class references were integrated by the nature of being
27
placed in the web conversation. For example, one student referenced a local political event that she discussed
with a roommate outside the class:
The other day my roommate asked me if any of my classes were talking about the Governor and
what he did during an assembly that he was speaking at. I had no clue what she was talking about.
She told me that it was this big controversy, her teachers were talking about it and it was also in
the newspaper. I am not always up to date (like I should be) on my current events. But she told me
that the Governor was a guest speaker at some school in Delaware As he was giving his speech,
the students were not paying attention at all, so I guess the Governor got a little mad and told the
kids to SHUT UP! I am not exactly sure what happened, all I know is that the students were told to
shut up by the Governor. (either he said "SHUT UP" or "Would you SHUT UP!") either way he
told them to shut up. I may be getting my info. wrong, but from what I heard I thought it was kind
of funny. I could not help to laugh when she told me-it could have been the way she said, but... I
mean he should not have done that, but I do not think it was that big of a deal. I think that the
children should have shown him more respect. The students should have been told prior to his
appearance on how to behave-maybe they were but it does not seem that way. Just wanted to share
this with you (all my facts may not be entirely true, I did not get to hear about this from a
newsman or anything like that.) But I thought it was interesting.
The student explicitly refers in this posting to an out of class event (discussion with her friend), which has a very
clear but implicit reference to an in-class topic (classroom management). It is clear that this posting was inspired
by the extensive in-class discussion of classroom management, and since her audience consisted of classroom
peers and the instructor, who were involved in this discussion, it was not necessary to explicitly connect the story
of the governor’s classroom management problems with the in-class ongoing discussion of this topic. In this
case, the student integrated relevant information from outside sources into the web discussion on classroom
management issues in much the same way as she would have done in any FTF conversation, without explicit
connection or introduction.
The students were motivated by diverse sources in their postings to the web discussions. The graph shows that
students’ practicum experiences (43%) and other participants’ web postings (40%) were the highest motivators
of students’ writing to the web discussions followed by students’ relevant life experiences outside the class
(27%) and issues emerging during classroom meetings and assignments (26%).
The web involved conversations that established dialogic relations among the messages and participants and was
not simply a static space registering students’ ideas. Often class material and experiences were understood,
reflected, and analyzed by the students using out-of-class references and vice versa. Through this process,
students’ experiences, reflections, and analytical tools were integrated and woven together (Putnam & Borko,
2000). Consider, for example, the following exchange among three students:
(Student 1) I just have a question that I just don't understand. WHY ARE SOME OF THE TEACHERS SO
JEALOUS OF SOME PRACTICUM STUDENTS, WHO AREN'T EVEN CERTIFIED?
(Caps in original)
(Student 2) L, I am not quite sure that I know exactly what you mean, but I hope that I have an idea. If you
mean that some of the teachers are jealous of the relationships that the practicum students have
with the children, then it could be simple. Some teachers seem to have a bond with their class
and when someone new comes in and creates a new relationship with the students, friction may
be felt. I definitely don't agree with that at all because I feel that the more positive influences in
a student's life at school, the better. Students need all of the help and support that they can get
and I think that teachers need to put the best interests of their students first. What do you think?
(Student 3, replying to "M" above) I agree with you on this one. Teachers do build up their own relationships
with their students and even grow to love them like their own kids because of all the time they
spent with each other and when someone else comes into the classroom and your students start
building up a relationship with these new people and doing thing to please them, it just starts to
hurt your feelings. You know your students still feel the same for you, but having someone
new in the class is exciting and gets students curious so naturally they start bonding with the
new person. I'm sure the teacher knows this, but it does still hurt you to see your class turning
their attention to the new person. My sisters had a student teacher in their class this year and
they as well as the other students were so excited, especially since the student teacher was a
male which is rare over there. Kids were always around him, they seemed to have fun in
everything they did and they absolutely loved him. One day the real classroom teacher went
over to check on them on she saw the kids laughing it up and giving the student teacher hi fives
for correct answers and she just broke down. She was over in my mom’s class crying about
28
how much the kids love him and how she was losing them. I know she felt the kids were
learning and that he was doing a great job and she was in this program to help him graduate
and it was in everyone's best interest to see new teaching methods and all, but teachers do have
feeling too, and when this sort of "take-over" of their kids comes up, of course their going to be
a little jealous or upset about their kids suddenly turning all their attention to someone else.
This is clearly a dialogic exchange where the students’ postings are intended to respond to and elicit the response
from each other (note the comments like "feel free to write" and "what do you think?"). In this sense, the Web
Talk is a dialogic space where the discussion by the first two students of their practicum and in-class experiences
elicits the third student’s description of her sister’s story, an out-of-class experience.
It is important to note that the practicum and out-of-class experiences were student-initiated references and not
examples elicited by the professor to illustrate the instructor’s curriculum. This provides evidence of the
students’ active learning and shared ownership for the class being supported by the Web Talk discussions.
Moreover, these student-initiated references helped to guide the instructor’s class instruction by revealing
students’ practicum and life experiences relevant to the class and the “hot” issues of the students’ immediate
concerns. In this way, the students guided and owned the curriculum on the web rather than treating it as an
“assignment board” for narrow replies to the instructor’s questions.
C: Problematicity: Did the students use the web both to define and to address problems?
We found that both processes were present in the web discussion. According to our analysis, 47% of all student
web postings involved problem-defining issues and 94% of the postings included attempts to address issues or
solve problems, while only 3% of all student postings did neither. We coded different degrees of expression of
problem-defining and problem-addressing processes in the students’ web postings as shown on Figure .
Students' Use of Web as a Problem-Defining and Problem-Solving Medium:

Type of Posting Genre by Frequency
80%
Problematizing/Problem Defining Addressing/Problem Solving
70%
60%
50%
40%
30%
20%
10%
0%
Explicit Observation
Evidence/Example
Sharing Negative
Sharing Positive
Advice/Prescription
Explanation/Argument
Concern
Question/Inquiry/Ask
Surprise
Educational Practice
Intellectual Reflection
Statement
Dilemma
Complaint about
Feelings
Supportive
of an Event
Feelings
for Help
Figure 4. Web as Problem-Defining and Problem-Solving Medium
We think that both problem-defining and problem-solving processes support the professional nature of this
learning community and address the concern about the possibility of students’ individual web postings making
“shallow contributions.” Problem-defining most frequently took the form of students’ complaints (34%) and
concerns (28%) about educational practices they witnessed and/or experienced in their teaching practicum or as
students (in their past and present). Once defined, these problems provided opportunities for reflection and
critical thinking initiated by the students, evidenced by the high frequency of problem-solving postings and
exemplified by intellectual reflection (67%) and statements (67%) about educational practices, supportive
evidence or examples (55%), observation of a practicum event (46%), and explanation and argumentation (41%).
The interplay of student-initiated problem-solving and problem-defining postings seems to reflect students’ deep
29
intellectual engagement and constructive and critical thinking about educational practices – the core component
of the discourse responsible for development of capable educators and a key component of a community of
learners.
D: Dialogicity: Did students both support and challenge ideas of other students?
We analyzed the relations among students’ web postings to see if we could find dialogic relations of supporting
and challenging each other’s ideas that characterize an academic discourse. Out of all students’ web postings,
60% involved providing support to the original author’s messages, 15% challenged, and 37% neither supported
nor challenged. However, if we take out students’ initiations (42% or 365 postings) and focus on only students’
replies (494 postings), the percentages are 94%, 24%, and 4%, respectively.
As indicated in figure 5, we identified several different methods of support as well as challenge. The most
frequent means of support was through elaboration (32% of all students’ postings) and aligning with author’s
experiences or ideas (22%) (see figure 5).
The Web as a Medium for Changing Modality of Other Students'

Contributions:
Type of Relationship to the Previous Posting by Frequency
35%
30%
Elevating Lowering
25%
20%
15%
10%
5%
0%
ng
n
n
t
t
t
l
n
ria
en
en
en
ew
io
io
as
io
ni
at
st
sa
em
em
at
m
C
Vi
ig
ue
or
ic
dg
er
ite
Al
re
re
rif
e
ab
Q
dv
le
iv
Ag
ag
os
la
w
El
at
e
t/A
rC
pp
is
th
no
rn
flic
O
fo
to
te
ck
Al
on
t/A
st
ly
ep
ue
C
en
R
eq
em
R
ag
ur
co
En
Figure 5. Web as a Medium for Supporting and Challenging Other Students' Ideas
It is important to note that elaboration on the original author’s posting is “a borderline” case of supporting the
original author because an elaborating reply often transcended the original author’s ideas and, thus, may
problematize it as is evident in the following exchange among three students:
(Student 1) I have noticed that alot of the students in my class tell on each other immediately about
everything. Even the most trivial things. They are in first grade and I have been telling them
that nobody likes a tattletale and they should work things out themselves. Have I been handling
this OK? I never know what to do.
(Student 2, replying to student 1) L, I think you are absolutely correct in telling your kids that no one likes a
tattletale. My cooperating teacher deals with tattletaling in one quick and easy way -- she has
been telling the kids from the beginning of school, "I don't want to hear about it unless
someone is hurt or there's a fire." This narrows it down a bit for them, and now they won't dare
go up to her and say something like, "Johnny knocked my book on the floor!" I hope this helps
you out a little.
Although Student 2 seemed to think that Student 1 was “absolutely correct,” her elaboration of the first student’s
approach leads to an indirect critique of the approach. Blind forbidding of kids’ reporting on each other can be
30
dangerous. She points out that her practicum teacher legitimately reserved students’ reporting on each other for
some serious cases. This hidden critique became explicit in another student’s later reply to Student 1 (it is
difficult to say how much Student 3 was influenced by Student 2’s message):
(Student 3 in reply to Student 1) I think that you definitely handled the situation correctly, but next time a
situation arises you might want to explain a little more. In other words, I feel that the student
should know that if it is serious issue or problem that it is then okay to tell. That is the only
thing I would encourage, but other than that you handled it great.
The most frequent ways of challenging the original messages were by the students’ introducing alternative ideas
(10%), bringing opposite cases (6%), and openly disagreeing (6%). Often students both increase and decrease
the modality of the original message as in the following example, where students write advice to future EDST
390 students:
(Student 1) This class can be overwhelming, only if you let it get that way. From day one, students should
be reading the syllabus so they know what is due and when. What I did, which worked out
great, is planning ahead. I figured out when things were due and worked on them ahead of time
so I wouldn't get bogged down at the last minute. Another bit of advice, that is kinda obvious,
is to really do the readings. I got so much out of the readings and I think the people who didn't
learn a lot from this class didn't do the readings.
(Student 2 in reply to Student 1) I think what you said about actually doing the readings holds true in most
cases. The reason why I think most people don't do the required readings is because a
combination of the assigned readings in all of your classes is way to overwhelming to even
consider. In the perfect world, we would have nothing tempting us from reading 10 or more
chapters a week. But let's face it, in most cases this does not happen. The best thing to do
would be to keep most of your education textbooks for future reference. You may not think so
now, but I bet they definitely come in handy later. Just an idea for those who aren't broke right
now and in desperate need of selling books back!
Although Student 2 formally agreed with Student 1 about the importance of reading assigned class literature on
time, she actually undermined the original statement by arguing that in the real world reading on time can’t be
possible because of the amount of literature assigned by different courses.
Students not only challenged each other but also the instructor, as is evident from the following example:
(Instructor) I really like the discussion K. is initiated (although some people raised similar questions
earlier). I want just comment on one point that E. made, "If I don't do my work then I may have
to miss recess to get it done." One of the problems with the strategy you described is that it
makes the student feel the learning and activity almost as punishment (i.e., negative). In the
innovative school I participated and observed in Utah (OC), this was a major concern of the
teachers and parents -- how to make consequences favorable for learning. For example, after
all-school program "Inventions and Inventors" all children were asked to invent something and
write patents. Then, there was an invention convention when kids from all school come to see
the inventions presented by the authors. Those kids who didn't put enough efforts have
opportunities to observe how little audience their invention gathered. In my view, this type of
"natural consequences" supports learning and learning activities because as I observed many
kids learned to put more efforts next type in their projects. What do you think?
Student 1 (in reply to Instructor) Eugene, although I agree that what you described can translate into learning
for some individuals, I do not think that all students will benefit from this. Nor do I think all
students will benefit from "staying in at recess." What you described, though (students not
receiving an audience for the work they did) can only be applied in a situation where an
audience would be a factor. This does not always happen in normal classroom structure. Take
for example, a student whom I know. He was not doing his math homework and did not know
the lesson once he went to school. At school, he felt bad about himself--because he didn't know
the answers when called on. His parents took away privileges in order to get him to do his
homework. He had a set amount of time to do math homework. He also had the support and
help of his parents. What this student will now tell you is that he feels good about himself; he
is surprised to find out that he is not stupid, and that he is not "bad at math." He learned
through CONSEQUENCE that it is important to study and that the result of that can lead to the
intrinsic desire to continue learning.
Based on the presence of both supportive and challenging relationships among postings, we concluded that the
students were in dialogic relations with each other. In a similar analysis of a web-based discussion, Dysthe
31
(2002) also found a high degree of dialogicity among participants, and argued that this dialogicity was at least in
part due to the nature of the discussion prompt provided by the instructor, an open-ended question about one of
the class readings. Dysthe argues that “it is crucial to set an interesting and challenging initial assignment, so
there a certain curiosity about the input from different voices” (Dysthe, 2002, p. 346). Wickstrom also concluded
from her own preservice teacher discussion forum that the most interesting postings occurred when students
brought their own genuine concerns to the web (2003). We agree, but in our own case we believe that the student
motivation to engage in dialogic interaction came from the lived and unpredictable practicum experience.
Theoretical and hypothetical questions, no matter how engaging, are no match for reality.
The process of collective reasoning and reflection sometimes, but not always, led to students’ declarations of
changing their mind or initial positions, as has been described in other research (Lampert & Ball, 1998).
Although less frequent than support and agreement, challenging and disagreeing was a part of the class web
discussion. The students clearly agreed more with each other than they disagreed, which is consistent with
findings from similar research (Bonk et al., 1999). One study has linked feedback, particularly supportive
feedback, with deeper level discussions, suggesting that this process fosters interactions by creating a
“sympathetic sense of community” (Makitalo et al., 2002). It may be a “natural” process of any academic
discourse that participants have a higher percentage of postings supporting rather than challenging each other’s
ideas. Or, it may be that the students tried to be “nice” to their peers and not challenge them too much (possibly
to avoid spoiling relations over disagreements). Indeed, one study found that students in an online discussion
were more likely to make negative comments out previous postings if they were able to do so anonymously
(Freeman & Bamford., 2004).
It is important to recall that our web participants were a strikingly non-diverse group in terms of ethnicity and
class, as mentioned earlier. It has been claimed that web-based discussions allow discussants more freedom to
express themselves because markers race, gender, class, and ethnicity are not visually available (Lenert & Harris,
1994). Nevertheless, at least one study has found majority-minority conflict in web-based discussion involving
majority (Anglo) and minority (Hispanic and Navajo) students, with the troubling tendency of the minority
voices to be silenced and diminished (Sujo de Montes, Oran, & Willis, 2002). In this light, we suspect that the
relatively homogenous nature of our students might have further contributed to their tendency to support each
other, and that minority perspectives that really challenged the students’ majority world view tended to come
from the situations arising in the practicum and sometimes from the instructor, further emphasizing both the
importance of the diversity practicum and the key role of the instructor, as described earlier.
Motivation: Were students motivated beyond the extrinsic motivation of course requirements to
participate in the web discussion?
We made an indirect analysis of students’ motivation to participate on the web. Our results suggest evidence of
mixed motivation for many students. We assessed the number of students writing additional and extra-long
postings. Additional number of postings was defined as exceeding the minimum number of 2 per week as
required by the instructor. Extra-long postings were defined as posting having more than 215 words (see Figure
6). This threshold came from the median of number of words in the XMCA professional academic e-mail list as
described above. Bonk and his colleagues (1999), whose class web discussions were organized as response to
cases, found that the average number of words per student posting varied between 110 and 140 words per class.
Dysthe (2002), whose web design involved postgraduate students’ response to the instructor questions, reported
on average 300 words per student posting.
Almost all students exceeded requirements at some point of the class. For the semester, web postings generated
intensive writing from the students with totals ranging from 11 to 84 300-word double spaced pages with a
median of 19 pages. Of course, there could be alternative explanations of why students posted additional web
postings per week rather than intrinsic motivation (e.g., possible informal competition among students for being
“the best student”, expecting extra credit from the instructor despite the fact that the instructor did not promise
any extra credit). However, it is difficult to find an alternative explanation of why the students wrote extra-long
postings. It is also important to remark that to some degree, our assessment of intrinsic motivation is indirect and
“conservative.”
The graph also suggests that the number of students exceeding the number of required postings per week and/or
writing extra-long postings both increased in the first part of the semester. In the second part of the semester, the
number of students posting extra-long messages decreased somewhat, probably because of an increase of
demands from other classes and because of the end of the practicum in week 12.
32
Figure 6. Number of Students Writing Additional and/or Extra-long Postings
The fact that some students did not write additional or extra-long postings per week does not suggest that they
did not have intrinsic motivation. We also cannot conclude that the students were intrinsically motivated to
participate in their web discussions all the time during the semester. The students’ anonymous course evaluations
showed that all students liked participation on the web very much and considered it as the one of the most
successful features of the class, although some students also indicated initial reluctance to participate due to
problems with computers, initial lack of computer skills, negative attitudes to computers, initial disruption of life
habits associated with systematic checking the class web, and, finally, the mandatory nature of their participation
on the web discussions (not knowing what to write). However, they unanimously insisted that the success of the
web discussions was associated with mandatory participation. This indicates some complex intertwining of
extrinsic and intrinsic motivation, at least for some students. More research is needed to investigate this issue. In
sum, the findings suggest that at least for some students at some time, participation on the web discussion was
intrinsic and authentic beyond the extrinsic motivation of the class requirements.
Conclusions
Our findings suggest that discussion webs can support the development of an academic community of learners.
Our web-based discussions demonstrate that:
1. Students comments demonstrated a shared focus on the academic themes defined by the instructor in
the syllabus, despite the fact that he did not assign topics for web discussion or penalize off-topic
comments.
2. Students integrated their lives outside the classroom and outside the teaching practicum with the issues
discussed, suggesting they participated deeply as “whole-person-in-leaning” rather than disembodied
information receptacles.
3. Students defined their own problems and addressed them; this is especially important to prepare
students for teaching, since problems and issues defined by instructors may not be relevant for them in
their future practice.
4. Students were able to enter into dialogic discussions with each other, both supporting and challenging
the statements of others in various and often subtle ways.
5. Students seemed to have shared ownership of and genuine interest in their class web discussions.
As we mentioned in the introduction, we believe that new media, as any tool, has the potential to fundamentally
change the nature of communication, both in ways that we hope for and in perhaps less desirable ways. In this
33
paper we have demonstrated one way that on-line discussion can help us to achieve our goal of creating a
community of learners, but how might this happen? The nature of web discussions is different from both face-
to-face communication and e-mail exchanges. Unlike face-to-face communication (FTFC), web discussion is
asynchronous, non-sequential, and selective (i.e., participants can ignore other persons’ messages without any
negative consequences). The discussion web creates and stores its own history and provides a shared space for
communication; “Unlike a live classroom, where conversations disappear, the Web allows every thought to be
captured for future examination, elaboration, and extension” (Owston, 1997, p. 29). In this sense, web
discussions can provide opportunities for deeper reflection and promote more access from different students to
classroom communication. The non-linear graphically represented discussion threads help students follow and
reflect on the ongoing web discussions because they preserve “turn adjacency” -- messages are posted with
regard for what they are responding to (Herring, 1999).
We do not suggest replacing traditional ways of class interaction with a class web discussion but rather
supplementing them (Bonk, 1998). A discussion web can supplement other ways of communication and
instruction, providing additional channels of comfort for some students, and thus facilitating pedagogical
innovations (Bonk et al., 1999). Jaffee (1998) contrasts a traditional college classroom with a classroom
involving Asynchronous Learning Network (ALN):
The classroom institution has historically centralized power and influence in the hands of the
instructor. When faculty walk into the classroom the learning begins; faculty are the source of
knowledge; faculty communicate information and influence the students; faculty determine what
will be taught, who will speak and when; faculty determine the correct or incorrect answer; and
faculty determine when it is time for students to "stop learning" and leave the classroom. ALNs, in
contrast, shift a considerable amount of power, authority, and control from the faculty to the
students.
Implications
We do not assume that web discussions are either a necessary or sufficient pedagogical tool to determine
development of a community of learners. There are many different ways of developing an academic community
of learners around subject matter with and without using web discussions. Our study was designed to investigate
whether a class discussion web in and of itself can support the development of an “academic community of
learners.” We argue against “technological determinism” according to which ALN by itself, without a specific
pedagogical design of its use provided by the instructor and supported by the students, leads to a reform of class
instruction (Bonk et al., 1998). As Putnam and Borko put it, “Simply providing new media and access to
communication with a much wider circle of colleagues and experts is, in itself, unlikely to change the nature or
form of teachers’ professional interactions” (Putnam & Borko, 2000, p. 11).
Pena-Shaff and Nicholls, in their discussion of the largely monologic nature of the web discussions they
analyzed, concluded that the technology alone is not enough to produce dialogic interaction, but that the
teacher’s integration of these discussions into the course is a crucial aspect of the success of these discussions
(2004). In our class, where students met virtually on the Web and also physically in class every week,
discussions initiated on the web often continued in the physical classroom space and vice versa, and the
instructor often mediated and encouraged this process. Therefore, we agree with the argument that teachers are
responsible for facilitating the development of dialogic interactions as they emerge on the web. The instructor’s
role in this practicum class, like the moderator’s role in any on-line learning community, consists of
“encouraging critical reflection on workplace practices and group identity” (Gray, 2004). One study based on
simply counting the number and length of postings and threads determined that instructors who post more
frequently and initiate threads more frequently actually tend to elicit fewer and briefer students postings
(Mazzolini & Maddison, 2003). We think these results point to the importance of further qualitative analysis of
the strategies of instructor participation on the web.
One instructor/researcher working in a class similar to ours describes that her students’ internet-based class
discussions achieved a new degree of critical reflection on their practicum experiences when she brought these
virtual free discussions into the classroom. In class she helped them to frame questions around their reported
experiences based on classroom readings and encouraged them to take these critically-framed discussions back
to the web for further discussion (Whipp, 2003). We refer to our own similar strategy as bring “hot topics” that
emerge in web-based discussion into the class and structuring our curriculum loosely with these emerging hot
topics always taking priority over previously scheduled activities (Matusov, St. Julien, & Hayes, 2005).
34
As we begin the next step of our investigation of instructional contributions of class web discussions, it is
important to examine institutional constraints, students’ backgrounds, overall classroom pedagogical design, and
properties of the webs to define conditions that allow interactive webs to support the development of an
academic community of learners. It is also important to address limitations, drawbacks, and expenses of using
discussion webs for the students and instructors as well as to examine their attitudes and perceptions of the class
web discussion. Finally, many instructors considering using web discussions in their classes are interested in
defining “developmental phases” of unfolding a “typical web” to distinguish a normal development of a
discussion web from web failures. They are also interested in the instructor’s role in supporting a class web
discussion. This study also did not focus on distinguishing characteristics of individual students as evidenced in
the nature of their participation on the Web Talk, but again this is a direction for further research. Finally, we
have been collecting data from students concerning their attitudes toward using the Web Talk, in an effort to
advance future research in this direction.
We see our research as a beginning, not a conclusion. We feel that one of the most important contributions of
this study was in developing a new methodology (e.g., deriving coding categories from the theory and from
instructors’ concerns; defining and operationalizing concepts like Piaget’s "collective monolog" (Piaget, 2002);
and Latour’s concepts of raising and lowering modalities inherent in scientific academic discourse). We designed
these coding schemes and raised these particular questions based on our experiences in practice, as part of an
ongoing attempt to design and redesign our own practice and share ideas with other researchers.
Acknowledgements
Parts of the paper were presented at the International Conference on the Learning Sciences, December 1998,
Atlanta, Georgia. We would like to thank David Blacker and John St. Julien for discussion of the research and
providing feedback on the paper. This research was supported by a Spencer Small Grant.
References
Admiraal, W., Lockhorst, D., Wubbels, T., Korthagen, F., & Veen, W. (1997). Computer-Mediated
Communication in Teacher Education: Computer Conferencing and the Supervision of Student Teachers. Paper
presented at the 7th Biannual Meeting of the European Association for Research on Learning and Instruction,
August 26-30, Athens, Greece.
Anderson, J. R., Reder, L. M., & Simon, H. A. (1997). Situated versus cognitive perspective: Form versus
substance. Educational Researcher, 26 (1), 18-21.
Bagnato, K. (2004). Learning virtually: harried commuters, working mothers and traveling soldiers are all
pursuing higher education online, a trend prompting some educators to wonder if traditional teaching methods
will one day be history. Community College Week, 16 (20), 6-8.
Bakhtin, M. M., Holquist, M., & Liapunov, V. (1990). Art and answerability: Early philosophical essays (1st
Ed.), Austin: University of Texas Press.
Blignaut, S., & Trollip, S. R. (2003). Developing a taxonomy of faculty participation in asynchronous learning
environments - an exploratory investigation. Computers & Education, 41, 149-172.
Bonk, C. J. (1998). Pedagogical activities on the "Smartweb": Electronically mentoring undergraduate

educational psychology students. Paper presented at the American Educational Research Association annual
meeting, April 13-17, San Diego, CA, USA.
Bonk, C. J., Appleman, R., & Hay, K. E. (1996). Electronic conferencing tools for student apprenticeship and
perspective-taking. Educational Technology, 36 (5), 8-18.
Bonk, C. J., Daytner, K., Daytner, G., Dennen, V., & Malikowski, S. (1999, April). Online mentoring of
preservice teachers with web-based cases, conversations, and collaborations: Two years in review. Paper
presented at the American Educational Research Association annual meeting, April 19-23, Montreal, Canada.
35
Bonk, C. J., & King, K. S. (1998). Electronic collaborators: Learner-centered technologies for literacy,
apprenticeship, and discourse, Mahwah, NJ: L. Erlbaum Associates.
Bonk, C. J., & Kim, K. A. (1998). Extending sociocultural theory to adult learning. In M. C. Smith & T.
Pourchot (Eds.), Adult learning and development: Perspectives from educational psychology (pp. 67-68),
Mahwah, NJ: L. Erlbaum Assosiates.
Bonk, C. J., Malikowski, S., Angeli, C., & Supplee, L. (1998). Holy COW: Scaffolding case-based
"conferencing on the web" with preservice teachers. Paper presented at the American Educational Research
Association annual meeting, April 13-17, San Diego, CA, USA.
Boylan, M. (2004). What have we learned from 15 years of supporting the development of innovative teaching
technology. Social Science Computer Review, Vol. 22 (2), 405-425.
Brown, B. W., & Liedholm, C. E. (2004). Student preferences in using online learning resources. Social Science
Computer Review, 22 (4), 479-492.
Buckingham, S. (2003). Perspectives on the experience of the learning community through online discussions.
Journal of Distance Education, 18 (2), 74-91.
Christopher, M. M., Thomas, J., & Tallent-Runnels, M. K. (2004). Raising the bar: Encouraging high-level
thinking in online discussion forums. Roeper Review, 26 (3), 166-171.
Cole, M. (1996). Cultural psychology: A once and future discipline, Cambridge, MA: Harvard University Press.
Conhaim, W. W. (2003). Education ain't what it used to be: eighty-one percent of all higher education
institutions offer at least one fully online or blended course, and 34 percent offer one or more complete online
degree programs. Information Today, 20 (11), 37-38.
Cummings, J. A., Bonk, C. J., & Jacobs, F. R. (2002). Twenty-first century college syllabi: Options for online
communication and interactivity. Internet and Higher Education, 5, 1-19.
Cummings, J. A., Bonk, C. J., & Jacobs, F. R. (2002). Twenty-first century syllabi: Dynamic tools for promoting
interactivity. Internet and Higher Education, 5 (1), 1-19.
Danet, B., Ruedenberg-Wright, L., & Rosenbaum-Tamari, Y. (1998). Smoking dope at a virtual party: Writing,
play and performance on Internet Relay Chat. In F. Sudweeks, M. L. McLaughlin & S. Rafaeli (Eds.), Network
and netplay: Virtual groups on the Internet, Cambridge, MA: MIT Press.
Dewey, J. (1966). Democracy and education: An introduction to the philosophy of education, New York: Free
Press.
Dringus, L. P., & Ellis, T. (2005). Using data mining as a strategy for assessing asynchronous discussion forums.
Computers & Education, 45 (1), 141-160.
Dysthe, O. (2002). The learning potential of a web-mediated discussion in a university course. Studies in Higher
Education, 27 (3), 339-352.
Forsyth, D. R. (1998). Group dynamics (3rd Ed.), Belmont, CA: Wadsworth Publishing.
Freeman, M., & Bamford., A. (2004). Student choice of anonymity for learner identity in online learning
discussion forums. International Journal on E-Learning, 3 (3), 45-49.
Gallimore, R., & Tharp, R. G. (1992). Teaching mind in society: Teaching, schooling, and literate discourse. In
L. C. Moll (Ed.), Vygotsky and education: Instructional implications and applications of sociohistorical
psychology (pp. 175-205), Cambridge, MA: Cambridge University Press.
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher
education. Internet and Higher Education, 7, 95-105.
36
Gillette, J., & McCollom, M. (1990). Groups in context: A new perspective on group dynamics, Reading, MA:
Addison-Wesley.
Gilligan, C. (1993). In a different voice: Psychological theory and women's development, Cambridge, MA:
Harvard University Press.
Goodfellow, R. (2004). Online literacies and learning: operational, cultural and critical dimensions. Language
and Education, 18 (5), 379-399.
Gray, B. (2004). Informal learning in an online community of practice. Journal of Distance Education, 19 (1),
20-35.
Hara, N. ( 2000). Student distress in a web-based distance education course. Information, Communication &
Society, 3 (4), 557-579.
Henri, F. (Ed.). (1992). Computer conferencing and content analysis (Vol. 90), Berlin: Springer-Verlag.
Herring, S. (1999). Interactional Coherence in CMC, Retrieved April 2, 2005, from

http://www.ascusc.org/jcmc/vol4/issue4/herring.html.
Jaffee, D. (1998). Institutionalized resistance to asynchronous learning networks, Retrieved April 2, 2005, from
http://www.aln.org/publications/jaln/v2n2/pdf/v2n2_jaffee.pdf.
Jefferson, G., Sacks, H., & Schegloff, E. A. (1987). Notes on laughter in the pursuit of intimacy. In G. Button &
J. R. E. Lee (Eds.), Talk and social organisation (pp. 152-205), Clevedon, England: Multilingual Matters.
Jensen-Lee, C., & Falahey, A. (2002). Using the Web: a discussion of some of the non-technical factors to be
taken into account when designing web sites intended to support courses taught primarily in face-to-face mode.
Journal of Sociology, 38 (3), 291-301.
Johnson, C. M. (2001). A survey of current research on online communities of practice. Internet and Higher
Education, 4, 45-60.
Khine, M. S., & Lourdusamy, A. (2003). Blended learning approach in teacher education: combining face-to-
face instruction, multimedia viewing and online discussion. British Journal of Educational Technology, 34 (5),
671-675.
Kidwell, P. K., Freeman, R., Smith, C., & Zarcone, J. (2004). Integrating online instruction with active
mentoring to support professionals in applied settings. Internet and Higher Education, 7, 141-150.
King, K. P. (2002). Identifying success in online teacher education and professional development. Internet and
Higher Education, 5, 231-246.
Kuboni, O., & Martin, A. (2004). An assessment of support strategies used to facilitate distance students’
participation in a web-based learning environment in the University of the West Indies. Distance Education, 25
(1), 7-24.
Lampert, M., & Ball, D. (1998). Teaching, multimedia, and mathematics: Investigations of real practice, New
York: Teachers College Press.
Latour, B. (1987). Science in action: How to follow scientists and engineers through society, Cambridge, MA:
Harvard University Press.
Laurillard, D. (1993). Rethinking university teaching: A framework for the effective use of educational
technology, London: Routledge.
Lave, J. (1988). Cognition in practice: Mind, mathematics, and culture in everyday life, Cambridge, UK:
Cambridge University Press.
37
Lave, J. (1992). Word problems: A microcosm of theories of learning. In P. Light & G. Butterworth (Eds.),
Context and cognition: Ways of learning and knowing (pp. 74-92), Hillsdale, NJ: Lawrence Erlbaum.
Lenert, K. F., & Harris, J. B. (1994). Redefining expertise and reallocating roles in text-based asynchronous
teaching/learning environments. Machine-Mediated Learning, 4 (3), 129-148.
Lim, C. P., & Cheah, P. T. (2003). The role of the tutor in asychronous discussion boards: A case sudy of a
preservice teacher course. Education Media International, 40 (1/2), 33-47.
Lobry de Bruyn, L. A. (2004). Monitoring online communication: Can the development of convergence and
social presence indicate an interactive learning environment? Distance Education, 25 (1), 67-81.
Makitalo, K., Hakkinen, P., Leinonen, P., & Jarvela, S. (2002). Mechanisms of common ground in case-based
web discussions in teacher education. Internet and Higher Education, 5, 247-265.
Marx, R. W., Blumenfeld, P. C., Krajcik, J., & Soloway, E. (1998). New technologies for teacher professional
development. Teaching and Teacher Education, 14, 33-52.
Matusov, E. (1996). Intersubjectivity without agreement. Mind, Culture, and Activity, 3 (1), 25-45.
Matusov, E., St. Julien, J., & Hayes, R. (2005). Building a creole educational community as the goal of
multicultural education for preservice teachers. In L. V. Barnes (Ed.), Contemporary teaching and teacher
issues, Hauppauge, NY: Nova Science.
Mazzolini, M., & Maddison, S. (2003). Sage, guide or ghost? The effect of instructor intervention on student
participation in online discussion forums. Computers & Education, 40, 237–253.
Means, B. (1994). Technology and education reform: The reality behind the promise (1st ed.), San Francisco,
CA: Jossey-Bass.
Moss, C. M. (1998, April). Using principles of educational psychology to foster web-based collaboration among
practicing and aspiring teachers. Paper presented at the American Educational Research Association annual
meeting, April 13-17, San Diago CA, USA.
Nicholson, S. (2002). Socialization in the ‘‘virtual hallway’’: Instant messaging in the asynchronous Web-based
distance education classroom. Internet and Higher Education, 5, 363–372.
Orvisa, K. L., Wisher, R. A., Bonk, C. J., & Olsond, T. M. (2002). Communication patterns during synchronous:
Web-based military training in problem solving. Computers in Human Behavior, 18, 783–795.
Owston, R. D. (1997). The World Wide Web: A technology to enhance teaching and learning? Educational
Researcher, 26(2), 27-33.
Palmer, P. J. (1998). The courage to teach: Exploring the inner landscape of a teacher's life (1st Ed.), San
Francisco, CA: Jossey-Bass.
Pena-Shaff, J. B., & Nicholls, C. (2003). Analyzing student interactions and meaning construction in computer
bulletin board discussions. Computers & Education, 42, 243–265.
Pena-Shaff, J. B., & Nicholls, C. (2004). Analyzing student interactions and meaning construction in computer
bulletin board discussions. Computers & Education, 42, 243–265.
Piaget, J. (2002). The language and thought of the child (3rd Ed.), London: Routledge.
Pinch, W. J. E., & Graves, J. K. (2000). Using web-based discussion as a teaching strategy: Bioethics as an
exemplar. Journal of Advanced Nursing, 32 (3), 704-712.
Putnam, R. T., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research
on teacher learning? Educational Researcher, 29 (1), 4-16.
38
Rogoff, B., Matusov, E., & White, C. (1996). Models of teaching and learning: Participation in a community of
learners. In D. R. Olson & N. Torrance (Eds.), The handbook of education and human development: New models
of learning, teaching and schooling. (pp. 388-414), Malden, MA, US: Blackwell Publishers Inc.
Rose, M. A. (2004). Comparing productive online dialogue in two group styles: Cooperative and collaborative.
The American Journal of Distance Education, 18 (2), 73–88.
Salmon, G. (2000). E-moderating: The key to teaching and learning online, London: Kogan Page.
Salmon, G. (2002). Mirror, mirror, on my screen...Exploring online reflections. British Journal of Educational
Technology, 33 (4), 379-391.
Schallert, D. L., & Reed, J. H. (2003-2004). Intellectual, motivational, textual, and cultural considerations in
teaching and learning with computer-mediated discussion. Journal of Research on Technology in Education, 36
(2), 103-118.
Schellens, T., & Valcke, M. (2005). Collaborative learning in asynchronous discussion groups: What about the
impact on cognitive processing? Computers in Human Behavior, 21 (6), 957-975.
Schrage, M. (1990). Shared minds: The new technologies of collaboration, New York: Random House.
Stigler, J. W., & Hiebert, J. (1998). Teaching is a cultural activity. American Educator, Winter, 4-11.
Sujo de Montes, L. E., Oran, S. M., & Willis, E. M. (2002). Power, language, and identity: Voices from an
online course. Computers and Composition, 19, 251-271.
Wegerif, W. (1998). The social dimension of asynchronous learning networks, Retrieved April 2, 2005, from
http://www.sloan-c.org/publications/jaln/v2n1/v2n1_wegerif.asp.
Wells, G., Chang, G. L. M., & Maher, A. (1990). Creating classroom communities of literate thinkers. In S.
Sharan (Ed.), Cooperative learning: Theory and research. (pp. 95-121), New York, NY: Praeger Publishers.
Wertsch, J. V. (2002). Computer mediation, PBL, and dialogicality. Distance Education, 23 (1), 105-108.
Whipp, J. L. (2003). scaffolding critical reflection in online discussions: Helping prospective teachers think
deeply about field experiences in urban schools. Journal of Teacher Education, 54 (4), 321-333.
Wickstrom, C. D. (2003). A funny thing happened on the way to the forum. Journal of Adult and Adolescent
Literacy, 46 (5), 414-423.
Windschitl, M. (1998). The WWW and classroom research: What path should we take? Educational Researcher,
27 (1), 28-33.
Wiske, M. S., Sick, M., & Wirsig, S. (2001). New technologies to support teaching for understanding.
International Journal of Educational Research, 35, 483-501.
Woodhead, B. (2004). Deakin University expands online learning system. Australasian Business Intelligence,
Sept. 7, 2004.
Yang, S. C., & Liu, S. F. (2004). Case study of online workshop for the professional development of teachers.
Computers in Human Behavior, 20 (6), 733-761.
39
Alkhalifa, E. M. (2005). Effects of the Cognitive Level of Thought on Learning Complex Material. Educational Technology
& Society, 8 (2), 40-53.
Effects of the Cognitive Level of Thought on Learning Complex Material

Eshaa M. Alkhalifa
Department of Computer Science,
College of Information Technology, University of Bahrain,
P. O. Box 32038, Isa Town, Bahrain
ealkhalifa@itc.uob.bh
ABSTRACT
The main goal here is to introduce a new perspective through which cognitive learning theory plays an
active role in instructional hypermedia design and evaluation through testing educational mediums that
elicit two distinct levels of cognitive processing for materials of different levels of complexity. Results
indicate that if the cognitive level required is high and the materials are more complex, then a retardation
effect occurs to learning, while a lower cognitive level requirement achieves better results will all types of
materials.
This highlights the importance of taking cognitive requirements into consideration during the design of
instructional hypermedia to produce “Cognitively Informed Systems”. This perspective will allow a
designer to analyze the same system from the perspective of how the presentation style and medium are
likely to interact with students’ cognitive processes during learning. This perspective is predicted to help
lower the cognitive load demands of various instructional hypermedia systems in order to increase the
educational impact of these systems and avoid any learning inhibitors to arise.
Keywords
Cognitive learning theory, Educational tutoring systems, Instructional hypermedia design, Mathematical
educational systems, Interactive tutoring systems
Background
Cognitive learning theory represents here the part of the field of Cognitive Science that focuses on the study of
how people learn and remember the information presented to them. In addition to being concerned with the
transfer process from the presentation of the material to memory, it concerns the mental representation of
concepts in memory as well as the cognitive load these concepts impose onto the cognitive system during the
learning process. These aims qualify this theory as an ideal source of knowledge that is capable of enriching
instructional hypermedia design. Oddly, this promising potential has only been identified by a few researchers
including Jonassen (1991), van Jooligan (1999) as well as Albacete and VanLehn ( 2000a, 2000b).
Jonassen (1991) advocates the constructivist approach to learning where students are given several tools to
relieve them from repetitive computation or to externally represent text they are required to recall as is usually
done when writing on a paper, in order to allow them to focus on the learning task at hand. He adopts the
assumption originally proposed by Lajoie and Derry (1993, Lajoie, 1990) that computers fill the role of cognitive
extensions, by performing tasks to support basic thinking requirements like calculating or holding text in
memory which caused them to label computers as “Cognitive Tools”. Jonassen’s (1991) central claim is that
these tools are offered to students to lower the cognitive load imposed during the learning process which in turn
allows them to learn by experimentation and discovery. However, no experimental evidence was presented to
support these claims where students achieved more learning with these designs.
Wouter van Jooligan (1999) takes this concept a step further by proposing an environment that allows students to
hypothesize and pursue the consequences of their hypotheses. They presented two systems; the first supports the
hypothesis formation step by providing several windows that help students form their hypotheses and the second
provides a formatted presentation of experiments already tested and their results in a structured manner. They
also added intelligent support to the system by providing feedback to students to guide their hypothesis
formation approach. Yet again the work was lacking of a proper comparative evaluation.
Albacete and VanLehn (2000a, 2000b) by contrast recognized the cognitive anomaly that exists between the
naïve students’ ill-structured knowledge of conceptual physics and the highly structured knowledge of experts in
the field. Consequently their presented system concentrates on teaching students how the various concepts relate
to each other. The evaluation of results exhibited no significant differences between the learning outcomes of the
control group when compared to the learning outcomes of the experimental group. Albacete and VanLehn
(2000b) then utilized alternative means of analysis to highlight various differences in learning between the
40
groups. The first was through measuring the effect size as done by Bloom (1984) while the second was to
compare results to the nationwide score on a standardized test. The third was to consider how much students who
have different pretest scores learned when compared to each other.
Thus, a central goal of this paper is to present “Cognitively Informed Systems” as an approach to the design of
instructional hypermedia systems by questioning the cognitive impact of the various modules within the system
utilized to present information to learners. The answers to these questions would then be utilized to set a scheme
of evaluation that is specific to this design in order to avoid the “no significant difference” trap that researchers
fall into. If the evaluation pre and post tests are analyzed from a cognitive perspective then, different types of
questions may elicit different levels of cognitive processing so simply adding the results of all types of questions
is not likely to be sufficient to distinguish where student learning is facilitated versus where it is not.
Introduction to Cognitively Informed Systems

Instructional systems that take cognitive research findings into account qualify to be described as Cognitively
Informed Systems by virtue of the cognitive information they carry within their design and evaluation stages.
The justification for this type of system lies in the fact that learning is a cognitive task and this in turn implies
that the designer of the learning environment should be acquainted with some of the findings known to date that
concern the process and the mental representation of the topic to be learned. For example, if the topic to be
taught is mathematical series, then the designer of the teaching system should be acquainted with concepts such
as “cognitive load” which is defined as the amount of cognitive processing required to perform an operation. For
example, the cognitive load associated with performing an addition of two numbers is less than that required for
learning how to add.
A formalization of the main areas of contribution to qualify a system to be described as a “Cognitively Informed
System” is as follows:
1. Perception and Recognition: In fields like Medicine, it is extremely important and difficult to alert student
attention to particular aspects of images and scans that are critical to a patient’s health which caused several
theories to emerge. These include Marr and Nishihara’s theory (1978) which argues that the main axes of
the objects shown are utilized to recognize the object while Biederman (1987) contests that recognition can
also be made by breaking the image into its primary components. Yet both theories unite in several basic
principles including the coding of edges, grouping features to recognize higher order features and matching
what is seen to structures stored in memory as well as accessing semantic knowledge about these shapes.
Bruce and Younge (1986) proposed an influential model where they argued that familiar and unfamiliar
faces are processed in different ways while a revised version of the model exhibits that recalling the name
associated with a face differs from recalling information about the depicted person. Pane, Corbett and John
(1996) designed an instructional system that exposed subjects to images that alter with time and through that
may have caused them to focus their attention onto the particular points in the images that students required
training in. However, their research question was to compare the learning effects of animation to carefully
selected still images. So when they tested student learning with declarative questions they did not examine
the “visual skills” students may have gained from the animated system that was not present in the still
images even though these skills may prove priceless in a medical setting. Their conclusion was that
animation and carefully selected images produce similar learning outcomes.
2. Attention and Memory: A central concern of instructional hypermedia is to attract students’ attention to the
points of importance in presented knowledge in order to promote recall of this knowledge at a later point in
time. Theories of attention focus on limitations in the cognitive capacity to attend to a particular input by
explaining possible causes for these limitations through various approaches (Broadbent, 1958; Treisman,
1964; Deutsch and Deutsch, 1963). In addition to this, a central division has been established between short-
term and long term memory (James, 1890) where the term “short term memory” was eventually replaced by
Baddeley and Hitch, (1974) to be “working memory”. Atkinson and Shiffrin (1968) indicate that the
working memory model is of relevance to activities such as mental arithmetic (Hitch, 1978) verbal
reasoning (Hitch and Baddeley, 1976) and comprehension (Baddeley and Hitch, 1974) in addition to the
task of recalling things from memory. One of the basic assumptions is that a student learning a skill has to
recall the instructions as well execute them by recalling the given information. For example, someone
learning how to drive has to recall how to drive in addition to paying attention to the road and the other cars
there. Once this person acquires the skill of driving recall is reduced to the road situation because the driving
task turns into a motor activity.
3. Mental Representation of Concepts: Albacete & VanLehn (2000a) attempted to utilize the findings on the
structure of mental representation in Physics. They based the teaching strategy of the “Conceptual Helper”
41
by comparing the unstructured mental representation of students of conceptual Physics as compared to the
highly structured mental representation of experts. The system, therefore, concentrated on helping students
find the “links” that connect the domain concepts to each other. They defined these links as associations that
are classically used in semantics to describe a relationship such as that between the concepts “parrot” and
“birds” because the first belongs to the category of the second. By doing this, they assumed a symbolic
mental representation which is primarily propositional (see e.g. Collins & Quillian, 1969; Rumelhart &
Ortony, 1977). Issues of interference that adversely affects recall rose within this domain as is exemplified
by the finding made by Baddeley, Grant, Wight and Thomson (1975). Subjects were informed of the
locations of digits on a matrix verbally while they were visually tracking a light moving along a circular
track and they were then asked to reproduce the matrix. Results showed that verbal messages that can be
easily visualized are adversely affected while complex messages that cannot be visualized remained
unaffected. This finding informs the design of multimedia systems to properly align them such that one does
not retard learning in the other. There are many other findings within this domain capable of informing
instructional hypermedia design.
4. Natural Language Comprehension and Generation: Communication in an educational setting can only be
achieved if both parties arrive at a common interpretation of the written text in a hypermedia setting. Frazier
and Rayner (1982) proposed a garden path model which earned its name because it can “lead up the garden
path” by ambiguous sentences formed with correct grammar as in; “The horse raced past the barn fell”,
“When Fred eats food gets thrown”, “Mary gave the child the dog bit a band-aid”, and “I convinced her
children are noisy”. Additionally, there is a great deal of work on story comprehension of which one of the
most successful theories was proposed by Kintsch and van Dijk (1978) where they indicated that story
processing occurs at two levels; the micro structure where the details of the story are considered at the level
of propositions and the macro structure level where the edited version of the micro structure is formed. The
generalization that occurs is of particular interest to learning as some students tend to overlook important
details when they generalize learned texts. Text generation by converse, involves generating language in
forms as close to “natural languages” as possible and this is subject to various theories. The goals are
usually to guide subjects towards self reflection and defending their own arguments. The setting usually
involves an intelligent tutoring system that generates the text according to specific points it notices as in
remediation of common errors. In short, this domain is vast, as it incorporates all the findings made in the
study of “effective communication” and many findings could be used as a guide.
5. Reasoning and Deduction: Johnson-Laird and Byrne (1993) indicate that deductive reasoning is a central
intellectual ability which is essential: “in order to formulate plans; to evaluate alternative actions; to
determine the consequences of assumptions and hypotheses; to interpret and formulate instructions, rules
and general principles; to pursue arguments and negotiations; to weigh evidence and to assess data; to
decide between competing theories and to solve problems.” It is these domains that are affected by attaining
an understanding of reasoning. For example, the mental models theory(Johnson-Laird and Byrne, 1991)
assumes that models are formed according to present criteria such that “truth” is reinforced. Students
therefore dislike assuming false facts unless they are explicitly stated. In the learning domain this implies
that students are likely to accept presented materials at face value rather than question what if an exception
emerges whenever instruction does not include concrete examples. The theory of Interpretation (Stenning
and van Lambalgen, 2004) is based on the assumption that all tasks presented to students in natural language
are subject to a number of possible interpretations as dictated by the semantics of the language. Following
that it is quite possible for reasoning to occur in a logical fashion. This allows different learners to associate
different interpretations with the same presentation materials if any ambiguities exist and the range of these
presentations can be predicted by the logic of the presented materials. An example perhaps is the work done
by Suthers, et al., (1995) which attempts to impose a particular “ideal” reasoning structure onto student
reasoning.
6. Cognition and Emotion: Freud (1915, 1943) argued that very threatening or anxiety-provoking material is
repressed from gaining access to conscious awareness and in turn cannot be remembered. Based upon this
Gilligan and Bower (1984) indicate that recall is best when the mood of the student at recall matches that at
the time of learning. Eysenck (1992) also argued that the main function of anxiety is to detect an
environmental threat and as a consequence it may affect how widely focused a student’s attention is. Images
in a topic like medicine should be informed of these results.
7. Cognitive Learner Differences: Jonassen and Grabowski (1993) give a detailed account of basic learner
differences that are embedded into cognitive learning theory. These include the differences between a
visualizing learner who likes to imagine concepts versus the verbalizing learner who likes to learn through
verbal communication. Albalooshi and Alkhalifa (2002) utilized this division to exhibit that a multimedia
presentation of a concept that has both animation for the visualizer and verbal representation for the
verbalizer exhibits reinforcement of one modality by the other. Ignorance of these differences may result in
an ineffective design of the educational system.
42
This list contains all major areas of influence where cognitive learning theory can inform instructional
hypermedia design and evaluation. To support and justify this framework, this paper presents a study of the
interaction between the cognitive levels of thought that is elicited through the approach selected to present
educational materials and two levels of complexity of educational materials.
The Interaction between the Cognitive Level of learning and the Complexity of the
Learned Material
There are two basic dimensions of change that are at play whenever students are presented with educational
materials. The first dimension concerns cognitive processes that are elicited during the learning task. It can be
primarily visual, overloads working memory, requires complex forms of mental representation, is a reasoning
task or can be affected by emotional states or individual differences. The second dimension of change that can
affect learning is associated with the level of complexity of the presented materials where some concepts may be
defined in terms of other basic concepts.
The study presented here concerns teaching students how to solve two basic types of questions commonly
associated with mathematical series. The first is to evaluate the series by applying the various operations in order
to attain the resulting value. The second type of question is when students are given the expanded version of the
series and asked to give the expression. Both of these types can be taught for the addition, multiplication,
division and power operations. Notice that the power operation is defined as repetitive multiplication which
qualifies it to be more complex than multiplication. This instructional hypermedia system will examine the
interaction between the two dimensions described above.
Determining the Cognitive Levels of Thought
The first dimension of change requires a careful analysis of various areas of influence that are extracted from the
presented formalization. Undoubtedly, if we restrict the number of variables that are allowed to change then the
reliability of the conclusions made increases. Consequently, the instructional system attempts to standardize all
factors that affect cognition by unifying the colors and schemas of presentation except for the factor under
analysis. This is the cognitive level of thought that is elicited by the presentation style and is based on Bloom’s
taxonomy of cognitive objectives (Bloom et al., 1956).
The lowest level is the knowledge level at which learning involves only the recall of facts, terminology, and
methodology without any requirement to understand what is recalled. The next level is Comprehension at which
elementary understanding is required as well as some use of the knowledge as in translating what was presented
in the student’s own words or interpreting it. This is followed with the level of Application which requires
students to be able to generalize from given abstract rules to a form they can apply. Example questions include;
determine the hypoteneous of a right triangle if one of the sides is of length 3 and the other of length 4.
The next level that concerns us here is the Analysis level which requires the extraction of features from a
knowledge domain that can describe it. In addition, it also identifies the relationship between these elements. For
example, if we consider mathematical series requires that students must gain the ability to distinguish some
central features between the various operations. For example, if the main operation is subtraction, then the terms
will be in decreasing order by a fixed amount. If on the other hand, the main operation is addition, they will be
increasing by a fixed amount that is added to each. Recognizing such features is usually done at the Analysis
Level of cognitive processing which is higher than that required for the evaluation of the various expressions.
The highest level of cognitive processing that is defined by Bloom’s taxonomy is the evaluation level where
judgments can be made about some of the content of the knowledge based upon criteria that is generated by the
person or adopted from external sources. In order to raise the elicited level of cognitive processing to this level
requires altering the basic approach adopted by the tutoring system when presenting information to students. A
possible approach is to inform students of the solution patterns they are following so that they recognize their
errors.
“When a learner is engaged in a discussion about the learner model, he is reflecting upon his
domain knowledge and experience re-calling and re-considering ideas of which he is
aware.”(Dimitrova et al, 2000)
43
A student therefore is offered the perspective of an instructor that considers the work of a student (in this case
themselves) with the aim of evaluating it. The existing approaches for involving the learner in the modeling
process include open learner models (Paiva and Self, 1995), collaborative student models (Bull et al, 1995) and
interactive diagnosis (Dimitrova et al.,2000).
TAGUS is a workbench for dynamic learner modeling (Paiva and Self, 1995) aimed at externalizing and
dynamically changing the learner model. The model itself is represented by Prolog clauses and learners can
manipulate the model presented by selecting options and typing Prolog clauses in the control panel. In this
system, both the educational system and the learner are external agents and interact with the model. Even though
the system externalizes the student model, students find difficulty in understanding and interacting with this
representation.
Mr Collins (Bull et al, 1995) is a student model that is open for inspection and negotiation with the student. The
system and learner are allowed to have separate and possibly even different perspectives of the students
knowledge and argue when they disagree with each other. Here the student model is externalized in tables, which
contain domain rules, and the system and learner’s estimate of levels concerning how much a student has
learned. The communication environment is text based as selections are made from menu options. This may
prove itself to offer a limited number of choices that a learner can reflect upon.
STYLE-OLM (Dimitrova et al.,2000) is an interactive diagnosis based modeler in an environment that teaches
scientific terminology. A dialogue game model is suited for maintaining an interactive diagnostics dialogue and
diagrammatic communication language provides a graphical externalization of the learner’s beliefs.
Communication is organized as an exchange of speech acts where dialogue moves are extracted from a
framework for analyzing educational dialogues. So a student selects a question word from a list of possibilities
and so on till the question is fully formed. However, the system itself has not been fully evaluated.
Determining Cognitive Load During Learning
The level of cognitive learning objectives was altered by altering the design of the educational module but this
does not imply that while utilizing the same module that all types of series will impose comparative cognitive
loads during the learning process.
In the field of mathematics, the basic operations are addition and subtraction. Multiplication can then be defined
as repeated addition while division can then be defined as repeated subtraction. Power comes at an even higher
level of processing as it is defined as repeated multiplication which in turn is defined in terms of addition.
These levels of processing suffer from the burden of interaction between the elements as it is defined by John
Sweller (1994). The task students perform involves considering a series of terms of the form:
Series 1: 3 + 6 + 9 + 12 + 15
Series 2: 3 + 9 + 27 + 81 + 243
They are expected to dissect each number into its components such that they would comprehend the relationship
that is preserved between them. One possibility is as follows:
Series 1: 3x1+3x2+3x3+3x4+3x5
Series 2: 3x1+3x3+3x3x3+3x3x3x3+3x3x3x3x3
The result of applying similar transformations to the first and second is just a step towards identifying what the
summation notation is. For the first, it is immediately clear that the terms are multiples of 3 and the index that
alters goes from 1 to 5. In the case of the second, we find that the index of the series has to be counted as it is
represented as the number of times the number 3 is multiplied by itself. This places it at a higher level of
complexity than that of series 1.
First of all, according to the definitions presented by Sweller (1994) both series satisfy the requirements of a
complex task as resolving them requires a series of steps that are related or interact with each other. As complex
tasks, these tasks are affected by varying cognitive load requirements of the task. Sweller (1994) defines the
intrinsic cognitive load as that which is dependant on the task as directly dependant on the level of interaction
necessary to complete the task. Series 2 shown above requires additional transformations when compared to
series 1 because the index of the series notation cannot be readily extracted from what is shown as the number of
44
repetitions must be counted. This leads to the conclusion that series 2 should require an intrinsic cognitive load
that is higher than series 1 according to the rationale offered by Sweller (1994).
A Two Module System
In order to vary the cognitive levels of thought versus the cognitive load that a student is faced with can be done
through a two module system. First of all, operations including multiplication and the power operation occur at
the same cognitive level even though they impose two different levels of cognitive load onto a student’s memory
provided they are all presented by the same type of module.
The main teaching module is the interactive part of the system that allows students to insert key values and
operations to evaluate the series in real time or give them examples of the reverse process if they insert some key
values. This module helps them achieve the level of Analysis where they would recognize through the
interaction with the system the features of each type of series to enable them to detect it.
The second main module is the “Mirror Modeler” and this part cannot exist in isolation of a teaching medium
that also tests students. The results of the tests are taken and analyzed according to a table of errors that will be
isolated from a specifically dedicated experiment. The system displays the probabilities associated with these
errors to the student who made them and then solves three new questions in real time in front of these students
while mimicking each student’s errors according to the probabilities associated with them. This externalizes the
whole solution process by displaying it live in front of the students. It allows them to process the information at
the evaluation cognitive level where they would evaluate their way of working by comparing it to an ideal that is
also displayed and calculated in real time in a window adjacent to the one displaying their way of solving the
problem.
Consequently, the instructional hypermedia system teaches materials that require two different levels of
cognitive load, through two modules that elicit two levels of cognitive processing to study the interaction
between these variables. The goal is to exhibit that cognitive processing characteristics described in the
formalization should influence instructional design by providing the means to lower the cognitive load during
learning to avoid learning to be hindered by cognitive overload.
Design of the Instructional Hypermedia System

The model was developed using IBM’s Java Visual Age, which is an integrated visual development environment
that facilitates the generation of complex functions. Its main features include the ability to import Graphical User
Interfaces (GUIs) and Java Beans that could be constant throughout several applications. The tool generates java
applets as in the case of this project or Java Servlets as is required.
The instructional hypermedia system is composed of two main modules; an interactive tutoring module that
elicits the analysis level of cognitive processing and a model generation and comparison module that elicits the
evaluation level of cognitive processing as they were defined by Bloom’s taxonomy.
Interactive Module
This module is composed of a tutorial section and a practice test section. The tutorial section of the system is
composed of two main parts that introduce students to the concept of mathematical series by taking them through
three examples where they generate the series from the summation notation and seven examples where they are
shown how the summation notation is derived from the series. The system is interactive because it allows
students to select some of the variable values and generates the series accordingly whenever possible. This
problem is a complex one composed of three terms and is broken up into several parts that are calculated
dynamically. Students are allowed to specify the starting and ending terms indicating the length of the resultant
series and to be able to recognize how the series can change based on different starting and ending numbers. The
second part of the tutorial is composed of seven examples of the more difficult task of extracting the notation
from the series. The tutorial includes the steps to first select the starting and ending points followed by finding a
common divisor and then the generation of the terms of the series to check that is the response is correct. It is
difficult here to allow extensive interaction flexibility because the problems given are set problems and the
system is primarily a teaching tool at this stage where all calculations are done online.
45
The practice test section is concerned with a more interactive practice session where students write the
summation notation they believe to be the answer and are shown the resulting generated series. They can then
compare this series to the original and practice any number of times they wish. At this stage students can select
from the different given notations and are allowed to practice and see the result of each selection. They are also
given advice of the probable cause of error based on the errors made. The number of options vary from one
problem to another to test student learning and to expose students to more than one possible option.
The Model Generation and Comparison Module
This module is composed of two sections; a test section and a model comparison section. The test section is
similar to the one described above because test questions are given to students with the exception that here,
students are not shown the resulting series so they are not aware of whether or not their answers are correct.
Students are showed three problems and they have to fill in several slots with the answers they believe to be true.
In a sense, they break up the notation into a starting number, an ending number, etc. This allows the system to
dynamically evaluate their responses. Student responses are then analyzed using an expert system that was
specifically designed based on the errors revealed by experiment one.
The model comparison section utilizes simple Bayesian rules to extract the probability of that student makes
each type of error and it generates a descriptive verbal model of the results. Note, however, that some of the
defined errors depend on previously defined errors. This implies that they are not completely independent and
the rules the modeler utilizes reflects that dependence. The modeler then shows students the ideal solution of
several new sample problems while regenerating how they would solve the same problems using student models
as a guide. The idea is to allow students compare their behavior to that of the ideal and allow them to reflect on
the causes of their errors.
Evaluating the modules
In order to evaluate the modules three separate experiments were run on second and third year students at the
University of Bahrain. The first experiment was run with the aim of categorizing common student errors in the
task which was then used as a guide for the next two experiments. One group was tested before and after using
the interactive tutorial module in order to detect any differences in their knowledge levels when exposed to this
module. Another group was exposed to both the interactive tutorial module and the mirror modeler and tested in
a pre and post test fashion. Do notice that it is not possible to test student learning before and after the mirror
modeler as it cannot exist in isolation without the support of a tutoring system.
Experiment One
This experiment was conducted with the aim of categorizing common student errors in this task, and in order to
identify if any cognitive load differences exist between the different operations. Its results were used as a basis
for the expert system rules used by the modeler. 8 second and third year students from the University of Bahrain
solved 6 summation questions each.
Materials
The questions used were specifically selected such that they relate to each other in a way that could be later
compared for further analysis. Each student was given the expanded forms of the summations shown below.
46
10 10
Q. 1 ∑ 3(m+1) Q. 4 ∑ 2i + 3
m=1 i=1
10 15
Q. 2 ∑2 m Q. 5 ∑ 3i
m=1 i=5
20 10
Q. 3 ∑ m2 Q. 6 ∑ i/4
m=10 i=1
The summations shown include formats that target testing of the different operations. For example, the variable
and constant positions are reversed for the power operation from 2m to m2 and for the multiplication operation
from 3i to i/4. Some of the digits were altered during these transformations to avoid making the questions
predictable to students.
The aim of this design is to be able to highlight all different types of student errors with respect to the variations
of the question format as well as to highlight the dependence of particular errors on particular question formats,
in order to identify the existence of any variances in cognitive load requirements.
Results
A list of all possible errors was compiled and categorized as shown in table 1 based on student responses. This
list is based on partitioning and an analysis of the surface structure of obtained responses to ensure that every
possible error type is considered.
Table 1. The definition of each error type based on student responses

Error 1 The arithmetic operation in the chosen notation is incorrect.
Error 2 The integer number in the notation is incorrect.
Error 3 The starting number of the chosen notation is incorrect.
Error 4 The ending number of the chosen notation is incorrect.
Error 5 The number of terms in the resultant series of the chosen notation is less than the number of
terms in the problem’s series.
Error 6 The number of terms in the resultant series of the chosen notation is more than the number
of terms in the problem’s series.
The frequency of each type of error that students made per question is shown in table 2 with a maximum
frequency of 8 for each cell.
Table 2. Number of errors students made in each question as classified by error type
Error 1 Error 2 Error 3 Error 4 Error 5 Error 6
Q1 7 7 6 6 1 2
Q2 8 5 6 6 0 7
Q3 3 3 3 3 0 3
Q4 3 7 7 5 3 1
Q5 3 2 4 5 2 3
Q6 2 1 3 2 0 1
By definition Errors 5 and 6 are dependent on errors 3 and 4, consequently, if they are taken out of a Chi test
calculation in order to have independent frequencies then results shown no significant differences with respect to
subject behaviors between the errors, nor in general behavior with respect to the question. However, taking the
data through a more detailed level of analysis using the Fisher exact test, reveals a significant difference with
respect to the frequency of Error 1 between Questions 2 and 3 with p < 0.013 and Questions 1 and 6 with p <
0.02 and for Error 2 between questions 1 and 5 with p < 0.02 and questions 4 and 5 with p < 0.02.
47
Discussion
These results first and foremost highlight the main types of common errors that can exist in student responses
some of which are compound errors such as errors 5 and 6 that are dependent on other factors as those
contributing to errors 3 and 4.
The second issue is that student reactions of the differences in the complexity of questions were reflected in the
results. Including multiple operations in the same series notation as in power plus addition or multiplication plus
addition, offer the most challenge to students who performed this task as is evident above in Question 1 and
Question 4 results in particular with error types 2 and 3. Following that in the ranks, comes the power operation
which caused a difference in Errors 1 and 2 especially when compared to the multiplication and division
operations.
These results confirm the predictions made that the added complexity offered by the power operation seems
more cognitively taxing to students which in turn causes a higher frequency of student errors. At the same time,
the level of overall difficulty is not affected as the results of the table as a whole does not show any significant
differences through a Chi Test. This implies that the increased cognitive load, is not accompanied with added
difficulty of the material, instead the task requires more thought processes, not thought processes that follow a
completely different approach to solve the problem. In other words, the two tasks seem to unite in the way they
can be solved and differ in their intrinsic cognitive load requirements. This is not surprising as the power
operation is defined as in terms of repetitive multiplication so results support the basic assumption.
Now that the main types of error in this task have been defined and sufficient justification for the difference in
cognitive load has been presented. The interaction between these two tasks and the two modules that elicit
different cognitive levels of learning objectives can be studied through a specially designed system.
Experiment Two
The aim here is understand the effectiveness of having an interactive user interface to teach materials that require
different levels of cognitive load. Students were, therefore, given a paper and pen test that is composed of three
questions that test for the multiplication, power and division operation ahead and following their use the
Interactive Tutorial Module.
Subjects
21 students from the University of Bahrain participated as volunteers in return for course credit.
Materials
The questions used were specifically selected such that they relate to each other in a way that could be later
compared for further analysis while ensuring they are not identical to avoid allowing students to simply repeat
the answers they gave in the pretest. Students were given the series of numbers shown below and asked to
reproduce the summation Notation that is to the left of each series shown.
Table 3. he pre test series given to students is shown to the right and they must reproduce the notation shown to
the left
10 S=1/4 + 1/2 + 3/4 + 1 + 5/4 + 3/2 + 7/4 + 2 + 9/4 + 5/2
∑ i/4
i=1
10
∑ 2i S = 2 + 4 + 8 + 16+ 32 + 64 + 128 + 256 + 512 + 1024
i=1
11 S= 6 + 9 + 12 + 15 + 18 + 21 + 24 + 27 + 30 + 33
∑ 3i
i=2
48
Table 4. he post test series given to students is shown to the right and they must reproduce the notation shown to
the left
12 S=33 + 44 + 55 + 66 + 77 + 88 + 110 + 121 + 132
∑ 11m
m=3
11 S = 9 + 27 + 81 + 243 + 729 + 2187 + 6561 + 19683 + 59049 + 177147
∑ 3m
m=2
10 S= 1/7 + 2/7 + 3/7 + 4/7 + 5/7 + 6/7 + 1 + 8/7 + 9/7 + 10/7

∑ m/7
m=1
The mapping between questions types is as follows:

Question Type One is Q1 in the Pretest and Q3 in the Post-test
Question Type Two is Q2 in the Pretest and Question 2 in the Post-test
Question Type Three is Q3 in the Pretest and Question 1 in the Post-test.
Results
The number of errors produced by question type in the pre and post tests in addition to the percentage
improvement is shown in table 5.
Table 5. he number of errors in the three operations in the pre and post tests
Division Multiplication Power
Pre-test 56 70 54
Post-test 14 25 28
Percentage Improvement from total 33.3% 35.7% 20.6%
Chi Test Significance p < 0.0000 0.0000 0.0007
If the number of errors in each column and the number of correct question parts are compared for the pre-test,
then no significant differences emerge. This implies that the three types of questions do not differ in their
difficulty. Running the same test on the post-test data gives a Chi Value of 5.914 with p < 0.05 so student
learning different from one operation to the next.
A Chi Yates value of 7.299 with p < 0.007 emerges upon more detailed testing between the division operation
and the power operation. A large difference also exists between the multiplication operation and the power
operation but it is not a significant one.
Additionally, the significant results of a comparison of the pre and post tests according to error type are shown in
table 6.
Table 6. umber of Errors made by students classified according to error type in the three operations
Classified by Error type
Interactive Chi Yates
Correct
Tutorial Pretest No Correct question Interactive Tutorial Posttest Significance
question
of errors of this parts No of errors of this type
parts
type
Question 1-Multiplication
E1 15 6 6 15 0.014
E3 17 4 5 16 0.001
E4 17 4 6 15 0.001
E6 16 5 5 16 0.002
Question 2-Power
E1 14 7 6 15 0.031
49
E6 12 9 5 16 0.059
Question 3-Division
E1 13 8 3 18 0.004
E3 12 9 3 18 0.010
E4 13 8 3 18 0.004
E5 13 8 2 19 0.001
Discussion
Results obtained in table 5 show that no significant differences in difficulty exist as students start the learning
process but differences do exist when we compare the amount of learning they achieve for each operation while
using the same interactive instructional system. The only remaining identifiable cause for this difference is when
the division operation is compared with the power operation and this supports the assumption that dictates the
existence of cognitive load differences between the two concepts.
It is glaringly obvious in table 6 that the improvement students achieve in multiplication and division spanned a
larger subset of error types than in the power operation. Therefore, one may safely conclude that although
learning occurs for all three operations while using the interactive module, the total gain and nature of this
learning differs from one operation to the next, in a way that consistent with the implications of the cognitive
load theory.
All this occurs when the instructional hypermedia module elicits the analysis level of cognitive learning
objectives, so what may occur if it demands a higher level of cognitive processing? This is what the next
experiment will investigate.
Experiment Three
The mirror modeler works by extracting errors from student responses to a set of questions and identifying
common error types in order to mimic the way that students solve this type of question. A student learns by
observing the process and evaluating it by comparing it to an ideal solution process.
Consequently, this module does not offer novices any basic information about the subject matter and this means
that testing it in isolation of other learning modules is meaningless. This leaves two possible options; to
introduce a simple textual explanation of the subject matter to the student prior to exposure to the mirror modeler
or to expose them to the interactive module tested in the previous section. The first option involves introducing
new material that has not been properly tested for its learning impact, while the other involves the use of a very
well assessed module that is the one to be compared to this one.
Consequently, both modules were presented to the students in a consecutive fashion while testing is done prior to
the first and following the second utilizing the same questions used in experiment two.
Subjects
12 students from the University of Bahrain participated as volunteers in return for course credit.
Materials
The materials were identical to those given to students in experiment two.
50
Results
Analysis of student responses showed in general that the number of errors made in the Pretest were 37 and the
number of errors made in the Post-test were 17 with a probability of p <.001 of this happening by chance. Table
7 shows the number of errors according to question type.
Table 7. Number of errors in the three operations in the pre and post tests
Division Multiplication Power
Pre-test 6 21 10
Post-test 0 1 17
Percentage Improvement from total 8.3% 27.8% -9.7%
Chi Test Significance p < 0.037 0.000 0.200
Discussion
The results of using the interactive tutoring module followed by the mirror modeler shows a clear difference
between the division, multiplication and power operations. The division and multiplication operations both
recorded significant improvements in student levels while the power operation was not significantly affected by
the modules that are presented. This is further evidence to support the assumption that the difference between the
cognitive load requirements of the multiplication and division operations when compared to the power operation
caused a serious difference in the amount of learning achieved as students utilized these two modes of learning.
It is therefore, not wise to assume that any educational system that is successful in teaching a particular concept
will be equally successful in others without running a comparative analysis of the cognitive load requirements.
Further results, however, can only be delineated from the data when it is compared to the results of experiment
two.
If this is done, then we find that students learned from the interactive hypermedia system in all operations, but
learning was to a higher degree in the division and multiplication operations which require a lower cognitive
load than in the power operation. Consequently, the results obtained in experiment three for the power operation
can only be obtained if the mirror modeler hindered learning for the power operation.
General Discussion
This paper shows that students mentally process the power operation in mathematical series in a way that is
different to the way they process the multiplication operation. The difference does not seem to exist in how the
operation is carried out because the first experiment showed no distinction with respect to difficulty. Instead,
results indicate that the cognitive effort required by one exceeds the cognitive effort required by the other.
Both operations were taught by two modules that require different cognitive levels. The first requires students to
recognize features that distinguish the behavior of the two operations by offering interaction. The second places
a student in a teacher’s shoes and expects that student to compare their own behavior to the ideal solution
process.
If cognitive processing characteristics have no effect on learning, then both operations of differing complexity
would be learned at the same rate in the same module. However, the fact that learning of the power operation
was hindered by the mirror modeler while that of the multiplication operation was fortified gives a strong
indication that cognitive processing overload may inhibit learning.
Reducing cognitive load therefore required an understanding of the relevant cognitive areas that were presented
earlier in this paper which results in the development of what may be described as Cognitively Informed
Systems.
Acknowledgements
We would like to thank God for the insight reflected in this paper. I would also like to thank the reviewers of this
paper, for their illuminating comments and to thank last but not least Dr Amjad Mahmood for his comments on
the final draft and questions that open new avenues for future papers.
51
References
Albacete, P. L., & VanLehn, K. (2000a). The Conceptual Helper: An intelligent tutoring system for teaching
fundamental physics concepts. Lecture Notes in Computer Science, 1839, 564-573.
Albacete, P. L., & VanLehn, K. (2000b). Evaluating the effectiveness of a cognitive tutor for fundamental
physics concepts. In L. R. Gleitman & A. K. Joshi (Eds.), Proceedings of the 22nd Annual Meeting of the
Cognitive Science Society. Mahwah, NJ: Erlbaum. 25-30.
AlBalooshi, F., & Alkhalifa, E. M. (2002), Multi-modality as a cognitive tool. Educational Technology and
Society, 5 (4), 49-55.
Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: A proposed system and its control processes. In K.
W. Spence & J. T. Spence (Eds.), The Psychology of Learning and Motivation (vol. 2), London: Academic
Press, 89-195.
Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In G. H. Bower (Ed.), The Psychology of Learning
and Motivation (vol. 8), London: Academic Press, 47-90.
Baddeley, A. D., Grant, S., Wight, E., & Thomson, N. (1975). Imagery and visual working memory. In P. M. A.
Rabbitt & S. Dornic (Eds.), Attention and performance V, London: Academic Press, 205-217.
Biederman, I. (1987) Recognition by components: A theory of human image understanding, Psychological

Review, 94, 115-147.
Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-
one tutoring. Educational Researcher, 13, 4-16.
Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational
objectives. The Classification of Educational Goals: Handbook 1. Cognitive Domain, NY: Longmans, Green.
Broadbent, D. E. (1958). Perception and communication, Oxford: Pergamon.
Bruce, V., & Young, A. W. (1986). Understanding face recognition. British Journal of Psychology, 77, 305-327.
Bull, S., Brna, P., & Pain, H. (1995), Extending the Scope of Student Models. User Modeling and User Adapted
Interaction, 5 (1), 45-65.
Collins, A. M., & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of Verbal Learning and
Verbal behavior, 8, 240-248.
Deutsch, J. A., & Deutsch, D. (1963). Attention: Some theoretical considerations. Psychological Review, 70, 80-
90.
Dimitrova, V., Self, J. A., & Brna, P. (2000). Involving the Learner in Diagnosis – Potentials and Problems.
Tutorial presented at the Web Information Technologies : Research, Education and Commerce, 2-5 May,
Montpellier, France.
Eysenck, M. W. (1992) Anxiety: The cognitive perspective, Hove, UK: Lawrence Erlbaum Associates.
Frazier, L., & Rayner, K. (1982) Making and correcting errors during sentence comprehension: Eye movements
in the analysis of structurally ambiguous sentences. Cognitive psychology, 14, 178-210.
Freud, S. (1915). Repression. Freud’s collected papers, 4, London: Hogarth.
Freud, S. (1943). A general introduction to psychoanalysis, New York: Garden City.
Gilligan, S. G., & Bower, G. H. (1984). Cognitive consequences of emotional arousal. In C. Izard, J. Kagen & R.
Zajonc (Eds.), Emotions, Cognition, and Behaviour. New York: Cambridge University Press, 547-588.
52
Hitch, G. J., & Baddeley, A. D. (1976). Verbal reasoning and working memory. Quarterly Journal of
Experimental Psychology, 28, 603-621.
Hitch, G. J. (1978). The role of short-term working memory in mental arithmetic. Cognitive Psychology, 10,
302-323.
James, W. (1890). Principles of Psychology, New York: Holt.
Johnson-Laird, P. N., & Byrne R. M. J. (1993). Models and deductive rationality. In K. I. Manktelow & D. E.
Over (Eds.), Rationality: Psychological and philosophical perspectives, London: Routledge, 177-210.
Johnson-Laird, P. N., & Byrne, R. M. J. (1991). Deduction, Hove, UK: Lawrence Erlbaum.
Jonassen, D. H., & Grabowski, B. L. (1993). Handbook of Individual Differences: Learning and Instruction,
Lawrence Erlbaum.
Jonassen, D. H. (1991) Objectivism vs. Constructivism: Do we need a new philosophical paradigm shift?
Educational Technology: Research & Development, 39 (3), 5-14.
Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text comprehension and production, Psychological
Review, 85, 363-394.
La Joie, S., & Derry, S. (1993) Computers as Cognitive Tools, Hillsdale, N.J.: Erlbaum.
Lajoie, S. P. (1990) Computer environments as cognitive tools for enhancing mental models. Paper presented at
the annual meeting of the American Educational Research Association, April 16-20, Boston, MA, USA.
Marr, D., & Nishihara, K. (1978) Representation and recognition of the spatial organisation of three dimensional
shapes. Philosophical Transactions of the Royal Society (London), B200, 269-294.
Paiva, A., & Self, J. A. (1995). TAGUS - A User and Learner modeling workbench. User Modeling and User-
Adapted Interaction, 4, 197-226.
Pane, J. F., Corbett, A. T., & John B. E. (1996), Assessing Dynamics in Computer-Based Instruction. In
Proceedings of the 1996 ACM SIGCHI Conference on Human Factors in Computing Systems. Vancouver, B.C.
Canada, April 1996, 197-204.
Rumelhart, D. E., & Ortony, A. (1977). The representation of knowledge in memory. In R. C. Anderson, R. J.
Spiro & W. E. Montague (Eds.), Schooling and the Acquisition of Knowledge, Hillsdale, NJ: Lawrence Erlbaum,
99-136.
Stenning, K., & van Lambalgen, M. (2004) A little logic goes a long way: basing experiment on semantic theory
in the cognitive science of conditional reasoning. Cognitive Science, 28 (4), 481-529.
Suthers, D., Weiner, A., Connely, J., & Paolucci, M. (1995). Belvedere: Engaging students in critical discussion
of science and public policy issues. Paper presented at the 7th World Conference on Artificial Intelligence in
Education, August 16-19, Washington D.C., USA.
Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design, Learning and Instruction,
4, 295-312.
Treisman, A. M. (1964). Verbal cues, language, and meaning in selective attention, American Journal of
Psychology, 77, 206-219.
Van Joolingen, W. (1999) Cognitive Tools for Discovery Learning, International Journal of Artificial
Intelligence, 10, 385-397.
53
Virvou, M., Katsionis, G., & Manos, K. (2005). Combining Software Games with Education: Evaluation of its
Educational Effectiveness. Educational Technology & Society, 8 (2), 54-65.
Combining Software Games with Education: Evaluation of its Educational

Effectiveness
Maria Virvou, George Katsionis and Konstantinos Manos
Department of Informatics
University of Piraeus, Piraeus 18534, Greece
mvirvou@unipi.gr
gkatsion@kman.gr
konstantinos@kman.gr
Abstract
Computer games are very popular among children and adolescents. In this respect, they could be exploited
by educational software designers to render educational software more attractive and motivating. However,
it remains to be explored what the educational scope of educational software games is. In this paper, we
explore several issues concerning the educational effectiveness, appeal and scope of educational software
games through an evaluation study of an Intelligent Tutoring System (ITS) that operates as a virtual reality
educational game. The results of the evaluation show that educational virtual reality games can be very
motivating while retaining or even improving the educational effects on students. Moreover, one important
finding of the study was that the educational effectiveness of the game was particularly high for students
who used to have poor performance in the domain taught prior to their learning experience with the game.
Keywords
Educational software games, Virtual reality, Evaluation, Intelligent tutoring systems, Student model
Introduction
The process of learning is a very complex cognitive task that can be very imposing on students since it requires a
lot of effort from them. Consequently, they need a lot of motivation to cope with it. In view of this, it is within
the benefit of education to create educational software that is interesting and stimulating for students. On the
other hand, there is a fast growing area of computer technology, that of computer games, that is extremely
appealing to children and adolescents. Indeed, anyone who interacts with children and adolescents in every-day
life can easily observe that they like computer games. This is also a view that has been supported by many
researchers who have conducted empirical studies (e.g. Mumtaz 2001). Thus the computer games technology
could be used to render educational software more motivating and engaging. In this respect, the difficult process
of learning could become more amusing.
Indeed, there are many researchers and educators that advocate the use of software games for the purposes of
education. Papert (1993) notes that software games teach children that some forms of learning are fast-paced,
immensely compelling and rewarding whereas by comparison school strikes many young people as slow and
boring. Boyle (1997) points out that games can produce engagement and delight in learning; they thus offer a
powerful format for educational environments. Moreover, there are studies that have shown that the use of
carefully selected computer games may improve thinking (Aliya 2002). As a result, many researchers have
developed games for educational purposes (e.g. Conati & Zhou 2002).
However, the attempts to create educational games have not reached schools yet. There are several reasons for
this. At first, not all educators and parents are convinced that educational games can be beneficial to students.
Second there are criticisms about the quality of the existing educational games. For example, Brody (1993)
points out that the marriage of education and game-like entertainment has produced some not-very-educational
games and some not very-entertaining learning activities.
Given the motivational advantages of software games as well as the criticisms that have been made on
educational games, there has to be further investigation on the advantages and limitations of software games for
education. Such investigation may lead to useful guidelines for the design of effective educational software
games. Indeed, educational software games should be designed in such a way that they are educationally
beneficial for all students, even those that are not familiar with computer games.
In view of the above, we have conducted an evaluation study on a virtual reality educational game that we have
developed, which has been briefly described in (Virvou et al. 2002). The game is called VR-ENGAGE and
54
teaches students geography. VR-ENGAGE aims at increasing students’ engagement by providing a popular and
motivating virtual reality environment. In this way, it aims at being more effective in teaching students than
other educational software and traditional media of education. The main focus of the research described in this
paper is to measure the educational effectiveness of an educational VR-game as compared to educational
software that does not incorporate the gaming aspect. The main aim of this comparison is to find out whether the
gaming environment may improve education.
The virtual reality educational game

VR-ENGAGE is an Intelligent Tutoring System (ITS) that operates through a virtual reality game. In common
with most ITSs it has the main components of an ITS. It has been widely agreed that an ITS, should consist of
four components, namely the domain knowledge, the student modelling component, the tutoring component and
the user interface (Self, 1999; Wenger, 1987). In the case of VR-ENGAGE, the student modelling component
models the student’s knowledge and his/her ability to reason plausibly about knowledge acquired on the domain
of geography. In this way, while playing, students may practice both their factual knowledge on geography and
their reasoning ability and thus they are led to “enjoyable” mastering of knowledge. Domain knowledge is
represented in the form of hierarchies that capture the relations between domain concepts. The tutoring
component generates advice tailored to the needs of individual students. Finally, the user interface consists of the
virtual reality game environment and its gaming features.
The environment of VR-ENGAGE is similar to that of the popular game called “DOOM” (ID-Software 1993)
which has many virtual theme worlds with castles and dragons that the player has to navigate through and
achieve the goal of reaching the exit. VR-ENGAGE has also many virtual worlds where the student has to
navigate through. There are mediaeval castles in foreign lands, castles under the water, corridors and passages
through the fire, temples hiding secrets, dungeons and dragons. The main similarity of VR-ENGAGE with
computer games like DOOM lies in their use of a 3D-engine. However, VR-ENGAGE unlike DOOM and other
computer games of this kind is not violent at all and is connected to an educational application.
VR-ENGAGE communicates its messages to students through animated agents that use speech synthesisers or
through windows that display text. When a student is asked a question s/he may type the answer in a dialogue
box. The user interface employs two types of animated agent, the dragon, which is the virtual opponent of the
player and the angel, which is the virtual companion of the player. Both types of animated agent use synthesised
voice as well as written messages. However, their voices are different so that the player may distinguish between
them.
The story of VR-ENGAGE incorporates a lot of elements from adventure games. The ultimate goal of a player is
to navigate through a virtual world and find the missing pages of the book of wisdom, which is hidden. To
achieve the ultimate goal, the player has to be able to go through all the passages of the virtual world that are
guarded by dragons and to obtain a score of points, which is higher than a predefined threshold. The total score
is the sum of the points that the player has obtained by answering questions.
In particular, while the player is navigating through the virtual world, s/he finds closed doors, which are guarded
by dragons as illustrated in the example of Figure 1. A guard dragon poses a question to the player from the
domain of geography. If players give a correct answer then they receive full points for this question and the
dragon allows them to continue their way through the door, which leads them closer to the “book of wisdom”.
However, if a player is not certain about the correct answer, s/he is allowed to ask the dragon for a “negotiation”.
The student modelling capabilities needed for the negotiation mode of the game are based on a cognitive theory,
called “Human Plausible Reasoning theory” (Collins & Michalski 1989). This theory formalises the plausible
inferences based on similarities, dissimilarities, generalisations and specialisations that people often use to make
plausible guesses about matters that they know partially. Important inference patterns in the theory are the
statement transforms. These inferences may lead to either correct or incorrect guesses; in any case these guesses
are plausible. The theory is used to simulate the reasoning of students when they give an erroneous answer. If
the student is found to have used a known pattern from the theory for his/her answer then this answer is
considered as a “plausible” error. Thus, in the case of negotiation, the student is allowed to make a guess for
which s/he has to provide a justification. The amount of points that the student is going to receive in the
negotiation mode, depends on how close the student’s answer is to the correct answer and/or how plausible the
reasoning that s/he has used is. The results of the error diagnosis that the system performs, are communicated to
the student through the virtual companion agent that appears in situations where the student needs help.
55
Figure 1. An example of a virtual world of the game
For example, the student may have been asked the following question: “What is the capital town/city of the
geographical compartment called Achaia (in Greece)?” While being in the negotiation mode, the student admits
that s/he does not know the correct answer and wishes to make a plausible guess such as: “My guess is that Rio
is the capital of Achaia. I know that Rio is an important town in Achaia. Therefore, it is likely that Rio is the
capital of Achaia.” This kind of answer is not actually given in natural language and the student is not allowed
unlimited selection of relevant pieces knowledge. The student is allowed to select several patterns of reasoning
and fill-in the names of cities, towns, mountains, etc. from a list of the domain knowledge of VR-ENGAGE. In
this way, it is ensured that the student’s answer will be within the “limits” of the domain knowledge encoded in
the ITS-game.
For the example above the student would be able to select the city of “Rio” from a list of known Greek cities.
The student also provides a justification for his/her guess such as declaring that Rio is an important city of
Achaia, and it could be its capital. The student’s guess may be correct or incorrect; in the case of the example, it
is incorrect because Patras is the correct answer. However, the reasoning that s/he has used may reveal whether
the student has a good knowledge of geography and whether s/he is able to use it plausibly. In the case of the
example, the student has shown that s/he knows some relevant pieces of knowledge, such as that Rio is an
important town of Achaia, which is very close to Patras. In this respect, the above answer exhibits some
plausible reasoning as compared to an answer that would be absolutely irrelevant. This kind of interaction helps
the students to reason about the domain being taught because even if they do not know the correct answer they
can use their knowledge on the domain to make a plausible guess about the correct answer.
In this sense the game provides an environment where there is opportunity for a negotiating teaching-learning
dialogue between the ITS and the students. Collaborative discourse is an issue that has attracted a lot of research
energy in the recent years (e.g. Moore 2000; Baker 1994). The process of becoming an expert in a certain
domain should no longer be solely viewed as the acquisition of a representation of correct knowledge; the
knowledge to be acquired should flexibly manage open problems (Andriessen & Sandberg 1999). In the case of
VR-ENGAGE, the reasoning in the diagnostic process of the system is part of the game’s plot and interactivity.
56
It aims at immersing the student in a way that is educationally beneficial through the teaching learning dialogue
between the actors of the game (student and dragon).
As part of the adventure of the game the player may also come across certain objects or animated agents. These
objects or animated agents appear at random and give hints to students or guide them to tutoring places
respectively. In tutoring places, students are encouraged to read a new part of the domain being taught. However,
these hints or the parts of the theory read, are not immediately usable by the students since they refer to questions
that the students will have to answer at a location of the virtual world other than the one they are currently at.
Hence, the students will have to remember these hints or parts of the theory so that they may use them when the
time comes. Educationally, these objects or animated agents motivate students to read and memorise important
parts of the theory.
Evaluation aims and experiment

The evaluation that was conducted on VR-ENGAGE focused primarily on evaluating the educational
effectiveness of the gaming aspect of the educational software. One could argue that the greatest advantage of
games is the motivation provided to students by the game environment whereas one possible disadvantage for
the learning process could be the students’ distraction by this game environment. However, even the
motivational advantage of educational games may be questioned since in a classroom there may be students who
do not like games or students who find it difficult to navigate through the virtual world and thus may not be able
to benefit to the full from the educational content of the software.
A common theme found in the literature for educational games, both electronic and non-electronic, is that these
games and software are considered successful only if they are as effective as traditional classroom education (Mc
Grenere 1996). However, this kind of comparison implies that games are not meant to be included in traditional
classroom education but rather they are meant to replace it. In our view, games should be used to supplement
traditional classroom education. Human teachers still have more abilities in explaining domain issues and
diagnosing students’ problems than any kind of software irrespective of its sophistication. This view is
reinforced by empirical studies that show that no matter how successful an ITS may be, students still prefer the
human teacher (e.g. Tsiriga & Virvou 2004). Therefore, in the present evaluation, we did not consider
conducting a comparison between human teaching and tutoring through the game.
Thus, to find out whether the game environment is in fact motivating and educationally beneficial to students
and not distractive we conducted an experiment where the game-ITS could be compared to an ITS that had a
conventional user interface without any virtual reality game. Both educational software applications had the
same underlying reasoning mechanisms with respect to student modelling as well as the same help and theory
functionalities. The main difference between the two educational software applications (game-ITS and ITS with
a simple user interface) was that one had a gaming approach whereas the other one did not have any gaming
approach at all. In fact, the software with the simple user interface had a hypertext display of domain theory and
exercises that were communicated to students through forms, dialogue boxes, buttons, drop-down menus etc.
However, these exercises were not part of any story as in the gaming approach. Moreover, there was no virtual
reality environment and no animated-speaking agents. For example, the way that the exam question, “Ethiopia is
in Africa. Right or Wrong?”, is presented to the user of VR-ENGAGE is illustrated in Figure 2, and the way the
same question is presented to the user of the software with a simple user interface is illustrated in Figure 3.
The evaluation experiment was connected to the underlying rationale of the educational game, which was to
engage students in learning the domain concepts that were taught to them. Thus the aim of the experiment was to
find out whether the educational game was in fact more motivating while it was at least as effective with respect
to students’ learning as the educational software with the conventional interface. Moreover, one of the primary
aims of the experiment was to reveal the degree of educational effectiveness (if any) for students whose
performance was considered good, mediocre or poor respectively by their human teachers.
This experiment took place in classrooms. School children usually have a preconception of educational means as
being totally different from entertainment. In this respect, the experiment aimed at finding out how school
children would react to an educational game in the settings of a real classroom where an entertaining aspect of
education would be rather unexpected. This was the main reason why the experiment took place in school-
classrooms. Human tutors were present and they were asked to observe their students while they interacted with
the computer but were not actively involved in the evaluation. There were, however, lab assistants that helped
students with the interaction with the game if the students needed help.
57
Figure 2. A question posed in the VR-ENGAGE
Figure 3. A question posed in the simple UI application
58
The experiment consisted of four parts. All four parts were similarly set up and involved a comparison between
VR-ENGAGE and the ITS with the simple User Interface (UI) in terms of the educational effectiveness and
motivation. All four parts were conducted in parallel. All of the children who participated in all four parts of the
experiment were of 9-10 years old and attended the fourth grade of elementary schools in Greece. They had been
taught the same syllabus on geography and they had a similar background on the use of computers. More
specifically, all of them were computer-literate and had been trained in their respective schools in the use of
Windows, the Internet and other popular software packages such as word-processors etc.
Each part of the experiment was different from the other parts in the type of the school-children that participated
in it. Specifically, the first part of the experiment involved all the students of 5 classes of school children of the
fourth grade of an elementary school, 90 children altogether, and their respective geography teachers.
For the second, third and fourth part of the experiment, the students who participated, were also of 9-10 years old
(fourth grade of the elementary school), but the selection of them was based on the mark that these students had
received by their respective human teachers in geography in the previous term. The term-marks that the students
of the fourth grade of elementary schools receive usually range from A to C. “A” is given to students with good
performance, “B” is given to students with mediocre performance and “C” is given to students with poor
performance. The participants were selected from the total students of 7 classes (127 students) of the fourth
grade of an elementary school, which was different from the one that had been used for the first part of the
experiment. From the total of the 127 students, 30 students were selected for the second part of the experiment
based on the criterion of their having received the mark “A” in the previous term, 30 students were selected for
the third part of the experiment based on the criterion of their having received the mark “B” in the previous term
and finally 30 students were selected for the third part of the experiment based on the criterion of their having
received the mark “C” in the previous term. The number 30 was selected so that we could have equal numbers of
students participating in each of the remaining three parts of the experiment.
Each group of children that were selected to participate in each part of the experiment was randomly divided into
two independent sub-groups of the same number of children. Thus, there were two independent sub-groups of 45
students for the first part of the experiment, two independent sub-groups of 15 students for the second part of the
experiment, two independent sub-groups of 15 students for the third part of the experiment and two independent
sub-groups of 15 students for the fourth part of the experiment. The first sub-group of each group would use
VR-ENGAGE and the second sub-group of each group would use the ITS with the simple UI (User Interface).
Before using their respective version of educational software, the students of both sub-groups of all the groups,
were asked to work on a pre-test using paper and pencil. This pre-test was an ordinary classroom test in which
every student had to answer 100 questions by filling in a test paper. The students’ performance in the pre-test
was compared to the students’ performance in a post-test that was given to the students after the use of their
respective software. The post-test was of a similar level of difficulty as the pre-test and consisted of the same
number of questions (100). The comparison of students’ results in the pre-test and the post-test was used to draw
conclusions about the educational effectiveness of VR-ENGAGE as compared to the simple ITS. In particular,
the school teachers were asked to count the number of erroneous answers of each student in the pre-test and the
post-test.
The students’ pre-test and post-test performance was compared using t-test statistics. In particular, the
educational effect of VR-ENGAGE was compared to that of the simple ITS by comparing the number of
mistakes of the students of the VR-ENGAGE sub-groups with the number of mistakes of the students of the
respective sub-groups that had used the simple ITS. It was expected that the number of mistakes that the students
would make after the use of either of the software versions would be reduced in comparison with the pre-test
because both applications provided quite sophisticated tutoring from the adaptive presentation of the theory and
the reasoning and student modelling of both applications. However, the post-test would reveal the degree to
which students who had used VR-ENGAGE exhibited greater or less improvement than those who had used the
ITS with the standardised user interface.
After the post-test, all the students who had participated in the experiment were also interviewed about their
experiences using their respective educational software. Moreover, the teachers of the school classes who had
participated in the experiment were also interviewed concerning their students’ performance and behaviour
during the experiment. Teachers were also asked to give their comments on their students’ performance on the
pre-tests and post-tests.
59
Evaluation results
First part of the evaluation
As mentioned earlier, the first part of the evaluation involved 90 students of the fourth grade of an elementary
school who were separated into two sub-groups of 45 children that would use the VR-ENGAGE and the simple
ITS respectively. The results showed a greater improvement of the VR-ENGAGE users over the users of the
other software. In particular, in the post-test, the players of VR-ENGAGE made 43.15% less mistakes than in
the pre-test. The other sub-group of students that had used the simple ITS resulted in 32.48% less mistakes of
their answers in total, as compared to the pre-test. Thus the players of VR-ENGAGE resulted in a higher
improvement of 10.67% in terms of their mistakes than the users of the simple ITS. This showed that VR-
ENGAGE had achieved its aim of being at least as effective as conventional educational software in the learning
outcomes and was in fact better in this respect.
In more detail, the total questions that were asked to the total number of students of each sub-group were 4500:
45 students x 100 questions = 4500 questions. In total, the students who had worked with VR-ENGAGE failed
during the pre-test in 1599 questions. The mean value of errors per student was 35.53 and the standard deviation
18.51. In total, the students who had worked with the simple ITS failed during the pre-test in 1647 questions.
The mean value of errors per student was 36.6 and the standard deviation 19.23. An initial analysis concerning
the comparison of the number of mistakes of each sub-group in the pre-test was not statistically significant
showing that the two sub-groups had similar background knowledge on geography. Indeed, there was a t-test
performed for the pre-test of the students of the two sub-groups. The null hypothesis (H0) was that there was no
difference between the mistakes of the two sub-groups and the research hypothesis, (H1) was that there was a
difference between the mistakes of the two sub-groups. The t-value result of 0.27 was smaller than its critical
value 2.00. This showed that the students of the two sub-groups had similar prior knowledge of the domain of
geography.
Then, after the students had completed their interactions with the two applications, they were given the post-test.
The players of VR-ENGAGE made 909 mistakes in the post-test. This number of mistakes as compared to the
1599 of the pre-test constituted an improvement of 43.15%. The students who had worked with the other
educational software made 1112 mistakes, which constituted an improvement of 32.48% in the number of
erroneous answers. The users of both educational applications showed an improvement in their post-test
performance. This was expected since both applications provided quite sophisticated tutoring and reasoning
mechanisms. However, the main aim of our experiment was to compare the improvement of the respective sub-
groups that had used the two applications.
Thus, the second statistical analysis compared the improvement between VR-ENGAGE and the simple ITS
users, on the number of mistakes for each sub-group between the pre-test and the post-test. The comparison
concerned the improvement that the users of VR-ENGAGE had vs the improvement that the users of the other
educational application had. There was a t-test performed. The null hypothesis (H0) was that there was no
difference in the improvement on the number of mistakes for the two sub-groups (VR-ENGAGE and simple
ITS) and the research hypothesis, (H1) was that there was a difference in the improvement on the number of
mistakes for the two sub-groups. As a result, the t-value of 4.52 was significantly greater than its critical value
2.00. This showed that the difference was statistically significant for the first sub-group in comparison with the
difference of the second sub-group, leading to the result that the 45 students who had used VR-ENGAGE had a
higher educational benefit than the 45 students who had used the simple ITS. All the statistical results are
summarised in Table 1.
Specifically, Table 1 illustrates the mean value of the errors made during the pre-test, the mean value of the
errors made during the post-test and the mean value of the percentage improvement on mistakes between the two
tests, for both of the sub-groups, the first who had used VR-ENGAGE and the other who had used the simple
ITS. Additionally it includes the respective results of the t-tests. These are the results of the first t-test, after the
pre-test, which show that there was no significant difference on the background knowledge on geography for the
two sub-groups of VR-ENGAGE and the simple ITS, and the results of the second t-test, after the post-test,
which show that there was a greater improvement on the number of mistakes for the sub-group of VR-ENGAGE
users over the other group. These results involve the standard deviations, the t- values (Tv) and the critical values
(Cv) of the t-tests.
60
Table 1. Results of the analysis of the students’ mistakes
VR-ENGAGE Simple ITS
Sub-Group (n=45) Sub-Group (n=45)
Variable Tv; Cv
Mean Standard Mean Standard
Value Deviation Value Deviation
Pre-test errors Tv = 0.27;
35.53 18.51 36.60 19.23
(Between 0 and 100) Cv = 2.00
Post-test errors
20.20 10.21 24.71 14.07
(Between 0 and 100)
Improvement percentage
Tv = 4.52;
on mistakes between pre- 43.15% 12.57 32.48% 9.26
Cv = 2.00
test and post-test
In the above t-tests the t-value of each t-test is calculated by performing a t-test for independent samples for each
of the null and research hypotheses (Voelker, 2001). The critical value for each t-test is the value taken from
Table T for a two-tailed research hypothesis depending on the sample number. The t-test results show that there
is a statistically significant difference in favour of the educational benefits of VR-ENGAGE over the simple ITS.
Second, third and fourth part of the evaluation

The second, third and fourth part of the evaluation involved 90 students of the fourth grade of an elementary
school, which was different from the first part of the evaluation, separated into three groups of 30 children
having poor, mediocre and good performance in geography. Every group of 30 children was then separated into
two sub-groups of 15 children that would use the VR-ENGAGE and the simple ITS respectively.
In the post-test, the VR-ENGAGE students who used to be poor and mediocre performers made 48.97% and
38.5% less mistakes respectively than in the pre-test. The students of poor and mediocre academic performance
that had used the simple ITS resulted in 31.57% and 31.64% less mistakes respectively, as compared to the pre-
test. Thus the sub-groups of students, of previous poor and mediocre academic performance that had used VR-
ENGAGE resulted in a higher improvement of 17.4% and 6.86% respectively in terms of their mistakes than the
students of the respective sub-groups that had used the other application. Moreover, the good students who had
used VR-ENGAGE resulted in a 33.8% improvement while the good students who had used the other
application resulted in a 32.84% improvement. This showed, that for the two sub-groups of the good students
there was also a small difference in favour of VR-ENGAGE but this difference was not statistically significant.
In particular, the total questions that were asked to the total number of students of each sub-group were 1500: 15
students x 100 questions = 1500 questions. In total, the sub-groups of students of previous poor, mediocre and
good academic performance who had worked with VR-ENGAGE failed during the pre-test in 921, 535 and 213
questions respectively. In total, the sub-groups of students of previous poor, mediocre and good academic
performance who worked with the conventional educational software failed during the pre-test in 906, 493 and
201 questions respectively. An initial analysis concerning the number of mistakes of each sub-group in the pre-
test involved 3 t-tests for the students of poor, mediocre and good academic performance respectively. For each
of the 3 t-tests the null hypothesis (H0) was that there was no difference between the mistakes of the VR-
ENGAGE sub-group and the simple ITS sub-group. The research hypothesis, (H1) was that there was a
difference between the mistakes of the two sub-groups. The t-value results of 0.53 for the students of poor
previous performance, 1.27 for the students of mediocre previous performance and 0.56 for the good students
were smaller than their critical values of 2.05, 2.05 and 2.05 respectively. This led to the acceptance of the null
hypothesis (H0), which showed that the respective sub-groups, that had used VR-ENGAGE and the simple ITS,
for each of the three categories of students, had similar prior knowledge of the domain of geography.
Then, after the students had completed their interactions with the two applications and answered the questions of
the post-test, we came up with the following results. The players of VR-ENGAGE (poor, mediocre and good
previous performance) made 470, 329 and 141 mistakes respectively. These mistakes compared to the 921, 535
and 213 in the pre-test constituted an improvement of 48.97%, 38.5%, and 33.8% respectively. The students who
had worked with the non-game ITS made 620, 337 and 135 mistakes, which constituted an improvement of
31.57%, 31.64% and 32.84% respectively in the number of answers failed. The statistical analysis which took
61
place after the post-test, compared the improvement on the number of mistakes for each sub-group between the
pre-test and the post-test. The comparison concerned the improvement of the users of each of the three categories
of students of VR-ENGAGE vs the improvement of the users of each of the three categories of students of the
other educational application.
There were 3 t-tests performed concerning the comparison of the improvement on the number of mistakes for the
students of poor, mediocre and good academic performance respectively. For each of the 3 t-tests the null
hypothesis (H0) was that there was no difference in the improvement on the number of mistakes for the
respective sub-groups. The research hypothesis, (H1) was that there was a difference in the improvement on the
number of mistakes for the respective sub-groups. The t-value result of 4.86 for the poor performing students
was significantly greater than its critical value of 2.05. The t-value result of 2.28 for the average students was
adequately greater than its critical value of 2.05. This showed that the difference in the improvement on the
number of mistakes for the respective sub-groups of both poor and mediocre performing students was
statistically significant. However, the difference on the improvement was mostly evident for the case of poor
performing students. On the other hand, the t-value result of 0.27 for the good students was significantly smaller
than its critical value of 2.05. This showed that the difference in the improvement on the number of mistakes,
was not statistically significant for the respective sub-groups of good students, leading to the result that good
students who had used VR-ENGAGE benefited in a similar way with the good students that had used the non-
game ITS.
The mean values of the pre-tests and the post-tests of the students, and the results of the above t-tests are
summarised in Table 2. In particular, Table 2 illustrates the mean values of the errors made during the pre-tests,
the mean values of the errors made during the post-tests and the mean value of the percentage improvement on
mistakes between the test pairs, for all of the sub-groups pairs, the first who had used VR-ENGAGE and the
other who had used the simple UI application. Additionally it includes the respective results of the t-tests. These
are the results of the three t-tests, after the pre-tests, which showed that there was no significant difference on the
background knowledge on geography for the sub-group pairs, and the results of the three t-tests, after the post-
tests, which showed the results of the comparison between the improvement on the number of mistakes for VR-
ENGAGE users and the users of the simple ITS. These results involve the standard deviations, the t-values (Tv)
and the critical values (Cv) of the t-tests.
Table 2. Results of the analysis of the students’ mistakes

VR-ENGAGE Sub-Group Simple ITS Sub-
(n=45) Group (n=45)
Variable Tv; Cv
Mean Standard Mean Standard
Value Deviation Value Deviation
Pre-test errors of students of
previously poor academic Tv = 0.53;
61.40 7.70 60.40 5.84
performance Cv = 2.05
(Between 0 and 100)
Tv = 1.27;
previously mediocre academic 35.67 9.43 32.87 8.62
Cv = 2.05
performance (Between 0 and 100)
previously good academic Tv = 0.56;
14.20 4.36 13.40 3.40
performance Cv = 2.05
(Between 0 and 100)
Post-test errors of students of

previously poor academic
31.33 8.04 41.33 6.00
performance
(Between 0 and 100)
previously mediocre academic
21.93 8.18 22.47 6.93
performance
(Between 0 and 100)
9.40 3.02 9.00 1.93
previously good academic
62
performance
(Between 0 and 100)
Improvement percentage on
mistakes between pre-test and post- Tv = 4.86;
48.97% 10.94 31.57% 7.72
test for students of previously poor Cv = 2.05
academic performance
38.50% 10.06 31.64% 5.08
test for students of previously Cv = 2.05
mediocre academic performance
33.80% 9.66 32.84% 9.67
test for students of previously good Cv = 2.05
academic performance
In the above t-tests the t-value of each t-test is calculated by performing a t-test for independent samples for each
of the null and research hypotheses. The samples were independent (and not correlated) because the experiment
aimed at comparing the improvement of two independent groups of students, the group of VR-ENGAGE users
and the group of the simple ITS users. The critical value for each t-test is the value taken from Table T for a two-
tailed research hypothesis depending on the sample number.
In summary, the above results showed that the sub-group of students of previous poor performance, which had
used VR-ENGAGE, benefited the most of all the sub-groups from the educational game. In addition, the sub-
group of VR-ENGAGE students of previous average performance had also benefited more than the respective
sub-group that had used the simple ITS since they made fewer mistakes (and the difference was statistically
significant). On the other hand, good students who had used VR-ENGAGE benefited in a similar way with the
good students that had used the non-game ITS.
Interviews of students and teachers

All of the students who had participated in the experiment were interviewed concerning the software they had
used. These interviews revealed that the players of VR-ENGAGE were fascinated by the idea of a game in the
classroom and they were certainly more enthusiastic about the software that they had used than the other group
of students. However, despite the fact that all students had liked the game in the context of their classroom work,
a large part of them criticised the game in comparison with other commercial games and said that they would
like VR-ENGAGE to have more virtual objects, a more sophisticated environment, more adventure and more
action. Students who were experienced game-players mainly made these comments. Such students had high
expectations from VR-games.
As to the teachers, most of them were particularly impressed by the effect that the game had on students who
were previously poor performers on geography. This category of students included quite a lot of those students
who the teachers thought were not easily disciplined in class. The teachers reported that these students seemed
absolutely absorbed by the game environment and kept working in piece and quiet without talking to anyone and
without disturbing anyone. To some extent, this comment was also made for the same category of students who
were given the other educational application to work with. In general, the teachers thought that the use of
computers was very good for the students who they used to consider as non-disciplined in class. However, they
thought that those who had used the game seemed so immersed that their behaviour in class had changed
completely and they had appeared to be very satisfied and interested in the educational content. The teachers
were very happy with their students’ performance on the post-test and most of them said that they would
certainly wish to include educational games of this kind in classroom. Some of them suggested that they might
even use the game on their own laptop in classroom and show the action of the game through a projector to their
whole class so that the whole class could participate in a single game play.
63
Discussion and conclusions
The results from the evaluation showed that students would benefit from educational games in classrooms and
would be quite happy to work with a computer game, which represents a more amusing teaching fashion than
that of conventional educational software. Moreover, one important finding that should be noted from the t-tests
of the second, third and fourth part of the evaluation is that when the subgroups of students who previously had
good, average and poor performance respectively were compared separately, it was revealed that the subgroup of
students who used to be poor performers had benefited the most from the game environment whereas the
subgroup of good students had benefited the least from the game environment. This coincides with findings
about the benefits of multimedia in general (Mayer, 2001).
The above finding may be explained by the fact that good students usually perform well under any
circumstances, whereas the rest of the students and particularly those who perform poorly may do so because of
lack of interest for their lessons and tests. Thus, students with little interest in their courses may benefit from
extra motivating environments such as those of VR-educational games. This finding was also confirmed by the
teachers’ impression about the students who they thought they were not easily disciplined in class. These
students were reported to have been absorbed by the game and they did not seem willing to take time out to talk
to other students or to try to cheat on the test and so on. This is probably due to the fact that games are able to
attract the attention of students who do not concentrate easily on their assignments due to boredom or other
distractions.
The students who used to have good academic performance did not have any significant difference in their
improvement through the use of the game or the use of the other software. However, one important finding is
that the performance of previously good students has not deteriorated by the use of the educational game due to
possible usability problems in the VR-environment or their possible distraction through the game. It seems that
good academic performers can keep their good academic record despite the fact that some of them were not
experienced virtual reality game players. From the interviews it was evident that they too had enjoyed the
learning experience through the game to a large extent.
However, it must be noted that during the experiment all students had as much help as they needed from lab
instructors concerning their interaction with the VR environment of the game. If the students had used the
software on their own at home, then perhaps they might have had more usability problems, especially those who
were not sufficiently experienced in virtual reality game playing. Such problems might have resulted in less good
educational results. Therefore, in future versions of VR-ENGAGE we aim to improve the usability of the game
environment and incorporate more on-line help. Finally, the game environment of the educational game has to be
very competitive with commercial games to attract a high degree of interest from students. This is so because
children are quite familiar with commercial games and therefore they have high expectations from game
environments.
References
Aliya, S. K. (2002). The role of computer games in the development of theoretical analysis, flexibility and
reflective thinking in children: A longitudinal study. International Journal of Psychophysiology, 45, 149.
Andriessen J., & Sandberg, J. (1999). Where is Education Heading and How about AI? International Journal of
Artificial Intelligence in Education, 10, 130-150.
Baker, M. (1994). A Model for Negotiation in Teaching-Learning Dialogues. International Journal of Artificial
Intelligence in Education, 5 (2), 199-254.
Boyle, T. (1997). Design for Multimedia Learning, London: Prentice Hall.
Brody, H. (1993). Video Games that Teach? Technology Review, November/December, 51-57.
Collins, A. & Michalski, R. (1989). The Logic of Plausible Reasoning: A core Theory. Cognitive Science, 13, 1-
49.
64
Conati, C., & Zhou, X. (2002). Modeling students’ emotions from cognitive appraisal in educational games. In
S. A. Cerri, G. Gouarderes, & F. Paraguacu (Eds.), Proceedings of the Intelligent Tutoring Systems 2002,
Lecture Notes in Computer Science, 2363, Berlin Heidelberg: Springer, 944-954.
Hasebrook, J. P., & Gremm, M. (1999). Multimedia for Vocational Guidance: Effects of Indiviualised Testing,
Videos and Photography on Acceptance and Recall. Journal of Educational Multimedia and Hypermedia, 8 (4),
377-400.
Id Software (1993). Doom - Virtual Reality Computer Game, Id Software Company, Texas, USA.
Mc Grenere, J. L. (1996). Design: Educational Electronic Multi-Player Games. A Literature Review Technical
Report 96-12, Vancouver, Canada, University of British Columbia.
Mayer, R. E. (2001). Multimedia Learning, New York: Cambridge University Press.
Moore, D. (2000). A framework for using multimedia within argumentation systems. Journal of Educational
Multimedia and Hypermedia, 9 (2), 83-98.
Mumtaz, S. (2001). Children’s enjoyment and perception of computer use in the home and the school.
Computers and Education, 36, 347-362.
Papert, S. (1993). The Children’s Machine: Rethinking School in the Age of the Computers, New York: Basic
Books.
Tsiriga, V. & Virvou, M. (2004). Evaluating the intelligent features of a web-based intelligent computer assisted
language learning. International Journal of Artificial Intelligence Tools, 13 (2), 411-425.
Virvou, M., Manos, C., Katsionis, G., & Tourtoglou, K. (2002). VR-ENGAGE: A Virtual Reality Educational
Game that Incorporates Intelligence. Paper presented at the IEEE International Conference on Advanced
Learning Technologies (2002), September 16-19, 2002, Kazan, Russia.
Voelker, D. (2001). Statistics, New York: Wiley.
Wenger, E. (1987). Artificial Intelligence and Tutoring Systems, Los Altos, CA, USA: Morgan Kaufmann.
65
Marjanovic, O. (2005). Towards A Web-Based Handbook of Generic, Process-Oriented Learning Designs. Educational
Technology & Society, 8 (2), 66-82.
Towards A Web-Based Handbook of Generic, Process-Oriented Learning

Designs
Olivera Marjanovic
School of Information Systems, Technology and Management
The University of New South Wales
Sydney, Australia
Tel: +61-2-9385-4473
Fax: +61-2-9662-4061
o.marjanovic@unsw.edu.au
ABSTRACT
Process-oriented learning designs are innovative learning activities that include a set of inter-related
learning tasks and are generic (could be used across disciplines). An example includes a problem-solving
process widely used in problem-based learning today. Most of the existing process-oriented learning
designs are not documented, let alone analysed, in any systematic way because they are tacit knowledge
gained through years of experience and reflection.
The paper investigates the problems of creation, sharing and IT support of process-oriented learning designs
and proposes a new type of process-oriented, knowledge management educational technology called the
web-based handbook of learning designs. It is envisaged that this technology will enable teachers to
assemble, share, reuse and execute process-oriented learning designs without any programming involved.
To design this technology, the paper proposes a multidisciplinary framework that integrates research in six
different areas: educational theories, educational technologies, knowledge management, software
engineering, process management and web-services. The paper then uses this multidisciplinary framework
to identify the main research challenges that need to be solved before this technology can be adopted on a
larger scale.
Keywords
Process oriented learning designs, Innovative teaching practices, Educational technologies, Knowledge
management, Web services
Introduction
One of the key challenges to innovative, computer-supported education today, is development of a framework
based on sound pedagogical principles that will promote the exchange and interoperability of learning concepts,
materials and teaching strategies (IMS Global Learning Consortium http://www.imsproject.org), (Oliver, Harper,
Hedberg and Willis, 2002), (LOM Standard http://ltsc.ieee.org/wg12/index.html). The new term learning design
has been proposed to describe, at the conceptual level, various components of learning experiences including, for
example, teaching resources, tools and innovative teaching strategies. So far, learning designs have been mostly
resource-oriented. Thus, the main emphasis has been on educational content and tools for content presentation,
delivery and management.
In recent years, several standards have emerged (see for example (IMS Global Learning Consortium), (LOM
Standard), (Learning technology standard committee)) to enable specification of various learning objects (again
mainly content or resource oriented) so they could be easily exchanged among various educational tools and
platforms. Furthermore, the latest educational technology standard SCORM (Sharable Content Object Reference
Model) (ADL, 2004a) goes one step further and enables sequencing and dynamic presentation of learning
content to suit the needs of a particular learner. However, sharing the innovative teaching practices and activities
(not only content-based) among teachers is even more important, but yet to be adequately supported by the
emerging standards and existing educational tools and platforms.
This paper concentrates on process-oriented learning designs, that are innovative teaching/learning activities
that consist of a set of inter-related learning tasks and are generic (rather than discipline specific). In order to
promote active learning individual tasks need to include collaboration, critical thinking, problem solving and
authentic interactions with real-world problems. Therefore, process-oriented learning designs are at the core of
socio-constructivist theory of learning (Bostock, 1997). It is important to note that they are not necessarily fully
on-line activities and could incorporate both face-to-face as well as technology supported tasks. Consequently,
they could be used both in the mixed (blended) mode of learning as well in e-learning.
66
Some process-oriented learning designs have became widely known and accepted across different disciplines,
for example the problem-solving process that forms a foundation of problem-based learning. Although there are
many variations of the problem-solving process, in essence, it includes tasks such as: identification of a problem,
alternative evaluation and selection of the course of actions for the chosen alternative. There are many more
processes, hidden within boundaries of individual disciplines, which could be generalised and used in other
disciplines (e.g. the balanced scorecard method).
However, the analysis and implementation of process-oriented learning designs is very challenging because they
are difficult to (1) describe, (2) share among teachers and consequently (3) support by information technology.
1. Difficulties in descriptions arise because the designs are in fact, tacit knowledge gained through years of
practical experience of looking for the answers for a simple question: “How do I help my students to learn?”
This is the main reason why, most of the existing process-oriented learning designs are not described or
analysed in any systematic way.
2. Because process-oriented learning designs are hard to describe, they are very hard to share among teachers
(especially across disciplinary boundaries). Benefits of this knowledge sharing are obvious (especially for
the less experienced teachers) and even more important in the world where advanced information
technologies open up new possibilities for innovative learning designs.
3. In order to support process-oriented learning designs by Information Technology (IT) (if such a support is
required), it is necessary to have process-oriented educational technology. However, currently popular
educational technologies are still quite limited when it comes to process support. If one ignores different
“look and feel” of the leading commercial learning environments (for example WebCT and Blackboard),
one can observe that they provide support for various aspects of course and student management, online
quizzes in different forms, but in essence, they are still task-oriented rather than process-oriented. In order to
support processes, in addition to communication and collaboration, it is necessary to adequately support
coordination of different tasks.
The main objectives of this paper are to further analyse process-oriented learning designs in order to find out
what type of IT support can be offered to teachers to create, share and reuse their innovative designs. More
precisely, the paper aims to:
4. critically analyse the problems of description, sharing and support of process-oriented learning designs from
six different perspectives (disciplines).
5. propose a multidisciplinary framework and use this framework to identify the main research challenges
related to process-oriented learning designs
6. define the main requirements for a new type of knowledge-management, process-oriented educational
technology called the web-based handbook of process-oriented learning designs. It is envisaged that this
technology will be used by teachers to create, store, retrieve, modify, re-use and assemble process-oriented
learning designs without any programming involved.
This paper argues that the creation, sharing and IT support of process-oriented learning designs is a complex,
multidisciplinary research problem. The following sections analyse, in more details, the identified problems of
creation of learning-designs, knowledge sharing and possible IT support. This analysis then leads to a
multidisciplinary research framework that will be used to define the requirements for the web-based handbook.
Motivating examples
The following simplified examples illustrate typical process-oriented activities that many teachers are already
using. Suppose that both teachers A and B are interested in implementing problem-based learning in an
assignment they are about to give to their students. Recall that in problem-based learning going through the
process is as important as the final outcome itself. Obviously, this has to be clearly stated within the learning
objectives for this activity.
The first example comes from Oliver and Omari (2001) who used the following scenario to illustrate how
technology can be used to support problem based learning: Teacher A posts an authentic problem on the course
web site along with links to various learning resources and questions for self reflection. He also specifies the
deadline for submission of a solution that students are required to post on the web site. In the next step students
use web-based discussion tools (forum) to discuss a possible solution. Then, they post a short summary of their
solution on the web by the given deadline. The teacher then marks each summary. Finally, all summaries are
made available to all students on the course web site. A similar example of problem-based learning can be found
in (Phillips, 2000) where the individual tasks of this process are supported by WebCT.
67
It is important to observe that in both examples, integration and coordination of individual tasks into a process
are not supported by technology. This is because technology used (e.g. WebCT) is task-oriented rather than
process-oriented.
In the second motivating example, suppose that teacher B is in charge of a postgraduate course. In this course,
students are required to solve a complex unstructured problem for their assignment. This teacher is interested not
only in the final solution but also in finding out how students work together to solve the given problem. In other
words, she is interested not only in the outcome but in the steps students take to solve the given problem. To start
their assignments, students need to register their groups (via e-mail) and allocate roles to individual members.
Then, they are required to submit a project plan within two weeks from the day the assignment was issued. In
order to complete the given assignment, students need to find and assemble the relevant learning resources (e.g.
journal articles, web sites etc.) and share them within their own group. To support their work, students also have
access to a number of communication and collaboration tools (e-mail, chat, forums etc.).
Furthermore, teacher B wants to implement a form of peer-review because she believes that students can also
learn from reading and commenting on other completed assignments. So, after all assignments are submitted, the
teacher marks each assignment. Then, for each assignment she allocates another group of students that will play
the role of reviewers. The reviewers are not supposed to mark the allocated assignment but are required to
comment on the proposed solution. After they complete their reviews and send back the allocated assignment,
teacher B will give the reviewers additional marks based on the quality of their comments.
To support this particular process-oriented learning design by existing educational technologies is not as simple
as it may appear. This is because the existing, widely available tools do not support the coordination aspect and
do not provide simple integration of various tools and resources (especially across different software platforms).
In terms of integration support, the exception is the latest standard SCORM. However, this standard is yet to
become widely accepted and incorporated into the popular Learning Management Systems (LMS). Furthermore,
teacher B will find it really hard to monitor students’ progress especially in terms of problem solving stages. For
example, she could analyse the messages students post on the topic forum but they may not correspond to the
phases of students’ progress.
In terms of knowledge sharing, unless these two teachers have a chance to exchange their experience related to
the practical implementation of problem-based learning, their practices will stay within their individual subjects
or disciplines. This means that it may never occur to teacher A to use peer-review in his course. Even if he
decides to use it, A could also benefit from B’s experience e.g. learn about difficulties in supporting this learning
design with the existing educational technologies. The following sections will analyse creation, sharing and IT
support for process-oriented designs from a multi-disciplinary perspective.
Creation and sharing of process-oriented learning designs

The problem of creation of process-oriented learning designs and their sharing can be analysed from the
perspectives of educational theories as well as knowledge management, as described in this section.
Educational theories
It has been widely recognised that one of the most important skills that students should acquire during their
university study today, is the ability to learn how to learn. To help students build these skills, it is necessary to
engage them in carefully planned learning activities (i.e. processes) rather than isolated learning tasks. Learning
activities should include the integrated learning tasks that promote active learning through collaboration, critical
thinking, problem solving and authentic interactions with the real-world problems. Also to help students become
more “self-regulated”, reflective learners it is necessary to develop student’s awareness of the process itself.
Even more, in many cases, going through the process is equally important as the process outcome itself (for
example in problem-based learning).
In order to describe process-oriented learning designs from the educational perspective, this paper adopts the
activity-centered approach to design of student learning experience. This approach differs from both resource-
centered and technology-centered approaches to instructional design. For example, when the resource-centered
approach is adopted, the main emphasis is on learning resources. Consequently, student learning is organised
around the available resources (e.g. around textbook’s chapters, web resources etc.). On the other hand, in the
68
technology-centered approach, the main emphasis is placed on the available technology (educational tools). This
approach is best described by the following question: “How should I design student learning experience so I can
use a particular tool”? (e.g. bulletin board, chat tool etc)
On the other hand, according to the activity-centered approach, the first step is to identify learning objectives
(what students need to learn) and then design learning activities to help students meet those objectives. These
activities will make use of the available resources and educational tools. Finally, it is necessary to design the
assessment tasks that will measure the intended outcomes (how well the students have achieved the intended
objectives).
The activity centered approach to learning complements the design framework described in (Oliver, Harper,
Hedberg and Willis, 2002) and (Agostinho, Oliver, Harper, Hedberg and Wills, 2002). According to (Oliver et.
al, 2002), a learning design comprises three key elements: the task or activities learners are required to perform;
the content or resources learners interact with; and the support mechanisms provided to assist learners to engage
with the tasks and resources.
In order to support process-oriented learning designs, it is necessary to extend this framework and introduce one
more key element of a learning design - the process. Figure 1 illustrates the extended activity-centered approach
proposed in this paper. This extension acknowledges that tasks are not performed in isolation and the actual
process (e.g. in problem-based learning) is as important as the final outcome.
Figure 1. The proposed extension of the activity - centered approach to learning
Knowledge management
Design and sharing of process-oriented learning designs is also a knowledge-management challenge. This is
because, learning designs are, in fact, tacit knowledge that is gained/invented by teachers and refined through
years of practical experience and reflective practice. Therefore, they represent experiental rather than widely
available external knowledge that can be easily shared. In fact, one could claim that process-oriented learning
designs have a very long-history (probably as long as formal education) as teachers have always invented and
used innovative learning activities to help their students to learn.
On the other hand, this type of knowledge sharing is becoming increasingly important not only to help the less
experienced teachers. More and more teachers need to share their experiences simply to stay up-to-date with ever
changing technologies as well as to meet higher student expectations. This is especially the case in the area of
IS/IT/CS education where most students come to expect their teachers to use very sophisticated educational
technology. Furthermore, some universities are in the process of implementing various institutional strategies to
encourage knowledge sharing among teachers that should, over time, result in much better student learning
experiences (see for example UNSW’s teaching and learning guidelines (UNSW, 2004)).
The relevant literature in the area of knowledge management also confirms the need for research on knowledge
management in educational systems. However, current research efforts in this area concentrate more on
69
knowledge management from the learner’s (student’s) perspective rather than teacher’s perspective. For
example, (Kayama and Okamoto, 2001) define knowledge management in the educational context as the
systematic process of finding, selecting, organising and presenting information in a way that improves learner’s
comprehension and/or ability to fulfill his/her current learning objectives. This paper argues that knowledge
management is relevant for both teachers and learners. In fact, better knowledge management support on
teacher’s side should, in turn, result in a better learning experience on student’s side.
In order to offer better knowledge management support to teachers, it is necessary to better understand the
knowledge processes they are involved in when dealing with process-oriented learning designs. In this respect,
we see the universities as organisational “knowledge systems” (where the term “system” refers to human and
organizational capital rather than technology systems) and adopt the framework developed by Alavi and Leinder
(2002). According to this framework, organizations as knowledge systems consist of four sets of socially enacted
“knowledge processes”: (1) creation (also referred to as construction) of knowledge, (2) storage/retrieval, (3)
transfer and (4) application of knowledge. Alavi & Leinder’s view of organizations as knowledge systems
represents both the cognitive and social nature of organisational knowledge and its embodiment in the
individual’s cognition and practices as well as the collective (i.e. organizational) practices and culture.
Also relevant for the knowledge management perspective are the well-known educational projects such as
MERLOT (http://www.merlot.org/Home.po) and ARIADNE (http://www.ariadne-eu.org/). These projects
concentrate on sharing of various static educational resources and providing tools for teachers to search and
create educational content (such as lecture notes, curriculum design etc.). Another example of a similar static
repository of educational resources is the educational clearinghouse called ERIC (http://ericeece.org/) that was
recently closed after 36 years of existence.
From the current experience with MERLOT project, one can observe that teachers are willing to share their
learning resources. However, sharing of static resources is only one, very limited aspect of knowledge
management that in some cases can easily lead to resource-centered teaching. Certainly, a more challenging
problem is sharing and exchange of teaching strategies and innovative practices. In terms of knowledge
management, this problem requires support for: creation, storage/retrieval, transfer and application of knowledge
as identified by Alavi and Leinder’ framework.
One of the objectives of this paper is to investigate how to use technology to support all four knowledge
processes and enable teachers to create new process-oriented learning designs, store them, share and apply
within their own discipline and learning setting. The following section describes in more details what is currently
available and, more importantly, what is required to create such technology.
Supporting process-oriented learning designs with IT

Educational technologies
To support composition, storage and execution of process-oriented learning designs, it is critical to have process-
oriented educational technology that is capable to support the coordination component of a process (that is
scheduling and coordination of individual tasks). However, as already pointed out, currently popular educational
technologies (such as WebCT, Blackboard et.) are still quite limited when it comes to coordination support.
At this point, it is also important to mention the growing number of educational technology standards such as
Learning Object Model (LOM) etc. As more and more researchers and practitioners are adopting the emerging
standards in this area, there is a growing number of learning objects (e.g. educational resources) that could be
shared across different educational platforms. At the same time, currently available modeling languages,
invented and used to specify various, interoperable learning designs are far too complex for the non-IT
specialists (e.g. XML based). This is not surprising, as these standards were not invented to enable teachers to
model and share learning designs, but to enable exchange and reuse of educational materials by different
educational environments (software packages). These languages are intended for software engineers who are
developing educational tools and environments.
Furthermore, the emerging standards mostly concentrate on static resources (educational materials). An
exception is certainly the latest version of SCORM (see ADL, 2004a). SCORM supports the notion of learning
content composed from relatively small, reusable content objects aggregated together to form units of instruction
such as courses, modules, chapters etc. Thus, a learning management system (LMS) based on SCORM standard,
70
offers a set of functionalities designed to deliver, track, report and manage learning content as the learner moves
through the assigned content. This dynamic presentation of learning content is based on learner needs and is
enabled by the “Sequencing and Navigation Model” (see ADL, 2004b). Here, sequencing rules and navigation
behaviour are specified independently from the content objects by the instructional designer. Finally, SCORM
standard aims to enable interoperability of reusable, sharable objects (instructional components).
Compared to other standards, SCORM goes one step further towards process support. It enables sequencing of a
learning content in a form of so called “activity tree” to suit the needs of a particular learner. However, it is
important to note that in many aspects this process support is very similar to what is already available in the field
of process technologies (as explained in the next session). Nevertheless, sequencing of content further confirms
the need to support process-oriented learning activities by educational technology.
Process-Management Technologies
As already pointed out, process-oriented learning designs require technologies that are capable to support
processes rather than individual tasks. In the area of business technologies, such support is already available in
the form of workflow technology that has been widely recognized as the leading process-oriented technology
(WFMS). In essence, workflows are designed to specify, execute, manage, monitor and streamline business
processes by allocating the right task to the right person at the right point of time along with the resources
needed to perform the assigned task. Workflow technology enables coordination of different tasks as well as
integration of tools and technologies used to support individual tasks. Because of these features, workflow is
considered to be the leading coordination and integration technology (WFMS).
Workflow models are designed by workflow analysts and stored in a workflow repository during design (or
build) time. Then, during run-time, a process instance is created and enacted (executed) based on the stored, pre-
defined model. The actual enactment of different tasks and their coordination are performed by the workflow
engine on the basis of different events (e.g. task completion event). For example, when one task is completed for
a particular instance, workflow engine will “read” the corresponding workflow model and activate the
subsequent tasks if their activation conditions are satisfied.
At this point, it is important to draw parallels between SCORM’s Sequencing and Navigation model on one side
and workflow models and workflow engine on the other side. Although they are designed for different domains
(education vs. business), in terms of their coordination (sequencing) mechanism they offer very similar
functionality. From the modeling perspective, workflow models are much richer models in terms of coordination
and sequencing constructs and options. Therefore, they could be used to express a relatively simple sequencing
structure as in SCORM. In this case, sequencing and navigation could be achieved through workflow engine that
could activate individual tasks through an event (e.g. completion of a learning module). In fact, a similar project
was described by (Marjanovic, Orlowska, 2000). The result of this project was a proprietary, workflow-based,
flexible-learning environment called Flex-eL. In Flex-eL, a single workflow model was designed for a course (as
depicted by Figure 2) to include a number of content modules and corresponding fully supervised quizzes. Then
workflow engine was used for sequencing and navigation through a set of learning modules. The main objective
of this system was to make the right module available to the right student at the right point of time along with the
required educational resources and tools. It is important to note that a workflow model was designed by a
workflow analyst (not end-user). Also one process instance was created for each individual student enabling
them to progress through the content in a variety of ways. For example, after completing Module 1, student
could do Modules 2 and 3 (and the corresponding quizzes) in any order. Then after completing the first three
modules, a student could progress through the remaining modules in any preferred order e.g. complete Module 8
and the assignment, then Modules 6 and 7 and Quiz 4 and finish off with Modules 4 and 5 and Quiz 3. A student
had to complete all modules and quizzes before attempting the final exam. This model also contained the
associate temporal constraints making sure that all components are completed by the required deadlines.
Another example of process-oriented educational technology can be found in (Van der Veen, Jones and Collis,
1998) where they use workflow technology to track and manage student projects. It is important to observe that
both projects use a single process model, predefined by a workflow analyst and imbedded into the workflow
system. The same model is then executed many times, once for each student. The same limitation of a pre-
defined model applies to SCORM (this will be discussed in more details in later sections of this paper).
71
Figure 2. A workflow model in Flex-eL by (Marjanovic and Orlowska, 2000)
However, in spite of its process-orientation, workflow technology is not directly applicable to process-oriented
learning designs for several reasons. It is primarily business rather than educational technology and therefore, its
main objective is to improve process efficiency through task coordination. Models of business processes are very
different from process-oriented designs. Among other things, it is easier to create models of business processes
than process-oriented learning designs because the standard descriptions of various business process models are
becoming widely available and accepted. In fact MIT’s handbook of organisational processes already exists (see
Malone, 1999) and is available to users to store and review models of common organisational business
processes. The purpose of MIT’s handbook is not to provide a repository of executable models but to enable
users to see the benchmark examples of various business process models that they may adopt in their
organizations. As already pointed out, such a collection of models of process-oriented learning designs does not
exist due to the fact that process-oriented learning designs are tacit (experiential) rather than explicit (widely
available and documented) knowledge. Furthermore, while business process models (as the ones used in
workflows) are designed by expert process analysts, it is envisaged that modeling of process-oriented learning
designs should be done by domain experts - teachers. Therefore, they will require different type of support than
workflow specialists.
Finally, models of process-oriented learning designs need to be much more flexible than the currently available
workflow models as well as activity models in SCORM. For example, some learning designs even evolve
(emerge) during the learning experience (unstructured problem-solving processes is one such example). Support
of emergent processes is a very recent research area of process management and technologies and modeling
frameworks are currently under investigation and not yet widely available (Marjanovic, 2005), (Carsen and
Jorgensen, 1998).
Software Engineering
Another area related to design and possible IT support of process-oriented learning designs is software
engineering. Thus, when analysed from the software engineering perspective, learning designs are related to the
work on patterns. For more details on patterns see (Hillside Patterns Home at http://hillside.net/patterns/). In
general, patterns are used to describe various repetitive activities at a very high level so they could be reused.
There are several different categories of patterns that are applicable to different modeling and implementation
levels such as design patterns, software architecture patterns etc. It is important to observe that the use of the
pattern theory in educational systems is still limited to design of educational software rather than learning
experience. For example, (DiGiano and Roscelle 2000) use software architecture pattern to create so called
componentware for education. However, they do not consider learning processes at all and concentrate on stand-
alone learning tasks.
Furthermore, (Crista and Garzotto, 2004) focus on design patters for adaptive/adaptable educational hypermedia
(AEH). AEH are designed to provide learning experiences that are dynamically tuned according to learner’s
characteristics. Although, AEH patterns are not directly applicable to research presented in this paper (at least at
this stage), they nevertheless illustrate the growing importance of patterns and their reuse in educational
technologies.
Another special category of patterns also related to process-oriented learning designs are the so called
pedagogical patterns (http://www.pedagogicalpatterns.org). These are high-level (natural language) descriptions
of various educational experiences, components and activities (where some of them represent examples of
learning activities) designed by software engineering community interested in teaching software engineering
concepts.
72
Although this high-level specification is not precise enough to be used as a model of process-oriented learning
designs, nevertheless it provides an excellent collection of examples of various learning tasks and activities that
can be further generalised.
Enabling new process-oriented learning designs

The latest developments in IT open new possibilities for more flexible process support as well as integration and
coordination of individual tasks and education tools. We argue that IT can be used not only to support the
existing process-oriented learning designs (partially or in full) but also to enable new learning designs never
before possible. This section gives a brief overview of web services – an emerging technology that can be used
to enable more flexible process-oriented learning designs.
Web services are Internet-based, modular applications, possibly offered by different providers that have standard
interface to enable efficient integration of business applications across organisational boundaries. Recent reports
by various leading industry analysts and practitioners claim that web-services will revolutionise the existing IT
applications as they enable easy integration of different platforms, tools and resources (Zhang, L-J, 2004),
(Yang, 2003). Web-services are platform-neutral and they enable building of composite services by using the
existing elementary or complex services. So far, web services have been developed mainly for various business
applications, while in the education domain their wide application is yet to be seen (especially beyond simple
integration of service infrastructure). Currently, there are several exemplary projects that deal with web service
support for learning processes at the technical level (see IEEE Learning Technology Newsletter, 2004). These
projects use the term “learning process” to describe a model of a composite web-service. (Note that the same
term is used in educational theories to denote cognitive processes of human learning rather than learning
activities).
We argue that it is necessary to take the top-down approach i.e. to design (model) learning activities to achieve
the intended learning objectives and then map the conceptual design into a composite service. From the
educational perspective this approach follows the activity-centered approach to learning where composite web
services are used for integration of tasks, tools and resources into a process (learning activity) at the technical
level. However, modeling of a composite service at the conceptual level (by non-specialists) is a real research
challenge. In fact, the current experience with conceptual modeling of a composite service in the business
domain illustrates that, as technologies and infrastructures are becoming available, the gap is widening between
the available technology and our current understanding of process modeling, verification, orchestration and
monitoring. In other words, modeling of the process (made of composite web services) and its context are
currently the major research problem in the area of web-services (Zhang, L-J, 2004), (Marjanovic, 2004).
A web-based handbook of process-oriented learning designs

As already pointed out, it is important to start from understanding and modeling of learning designs – not by IT
experts, but by the domain experts – teachers – who are involved in creation and sharing of learning design in
their everyday work. Consequently, knowledge management processes (creation, sharing, transferring,
application and modification of learning designs) are not happening in IT labs. They occur while teaching is in
progress and while teachers are looking for the best way to help their students to achieve the intended learning
objectives at that particular point in time.
A multidisciplinary framework for design of a web-based handbook
In order to design a web-based handbook, it is necessary to integrate the expertise in innovative teaching on one
side, and currently available and emerging technologies and standards on the other side, in a much better way
than what is currently offered by the commercial learning management systems.
To address this integration challenge, this paper proposes a new type of knowledge-management, process-
oriented educational technology called the web-based handbook of process-oriented learning designs. The main
motivation behind development of this technology is to enable teachers to retrieve, modify, re-use, share and
assemble the executable components to support their learning designs. These components are self-contained
modules used to implement one or more process tasks. By combining individual components, teachers can create
and implement various process-oriented learning designs without any programming involved. Recall that these
73
technology-supported tasks could be also combined with various tasks not supported by technology (e.g. face-to-
face activities).
Design and implementation of the web-based handbook is a complex multidisciplinary research and
development problem that requires integration of knowledge, tools and methods from several different
disciplines (as depicted by Figure 3).
In terms of Alavi and Leinder’s framework (2000), the proposed web-based handbook of learning designs
represents a knowledge repository that will enable teachers to perform all four knowledge processes:
¾ create learning designs by assembling different executable components
¾ store them in, and retrieve from the knowledge repository
¾ transfer their knowledge and experience to other teachers and
¾ apply this knowledge within their own teaching discipline to further improve their teaching practices.
Obviously, it would be unrealistic to expect teachers to implement (develop code for) their own executable
components. Instead, teachers’ role is to create a process-oriented learning design and select and assemble the
most appropriate executable components that could support parts of, or the complete process. However, in order
to enable development of the new executable components and provide ongoing support, it is necessary to
establish a feedback loop (knowledge sharing) between domain experts (teachers) and IT specialists (knowledge
engineers) as described in the next section.
Figure 3. Process-oriented, generic learning designs – a multidisciplinary framework
Web-based handbook – basic functionality
In terms of its functionality, the web-based handbook provides a knowledge repository (used to store executable
components, process models and instances) and a set of tools for its main users: teachers and knowledge
engineers. Before the handbook can be made available to teachers it is necessary to populate the knowledge
repository with executable components designed and developed by knowledge engineers. The system
development lifecycle of the new executable components involve the following steps:
(1) Externalisation of tacit knowledge

This step starts from understanding of the intended process-related learning designs. Thus, teacher’s role here is
to describe the intended learning design to a knowledge engineer by using the activity-centered approach (as
depicted by Figure 1). This means that they need to start from the intended learning objectives and then describe
the series of inter-related learning tasks designed to achieve these objectives along with educational resources
and tools that could be made available to students.
74
(2) Modeling of innovative practices
In this step, knowledge engineers document the described learning design in a more formal way, by using a
process-modeling language. The formal model enables capturing of the details of individual tasks as well as the
intended coordination mechanisms and process constraints (e.g. how long does it take to complete the given
process).
(3) Identification of possible generic patterns

In order to identify possible generic patterns that could be turned into executable components, knowledge
engineers need to analyse and compare the available process models. This highly creative task requires a very
good understanding of the capabilities of the existing educational technologies and standards as well as the
ability to identify new components that could be created to support individual tasks. Formal models provide
consistency in process description that, in turn, enables easier comparison of different models and detection of
possible similarities and patterns.
(4) Design and development of executable components

The identified patters are then used to design and implement new executable components. When completed and
tested these components are then stored in the knowledge repository and made available to teachers.
(5) Ongoing management (maintenance) of the knowledge repository

After they develop the initial set of components, knowledge engineers will be involved in ongoing management
(maintenance) of the knowledge repository. This involves storage and retrieval of executable components and
possible improvement (modification) of the existing components. Note that, in addition to individual
components, the knowledge repository could also store complete or incomplete models pre-assembled and used
by other teachers.
When components are made available, teachers can use the web-based handbook to perform the following
process-related activities:
(1) Selection and configuration of components and process models

Again starting from the intended learning objectives and activities designed to meet these objectives, teachers
can select the executable components to support individual tasks, parts of a process or a complete process. The
real challenge here is to decide which part of the intended process can and should be supported by technology
and then find the most appropriate components to support it. However, the match between the individual
learning tasks and executable components is not one to one, as a single task may require more than one
component to support it and vice versa. Therefore, especially for non-IT teachers, it is necessary to involve
knowledge engineers to help with possible selection of the most appropriate components. It is envisaged that
over the time, as teachers gain more and more experience this help will be less and less required. Individual
components should be configured (i.e. different values assigned to different parameters) to make sure they fit the
intended process and comply with process constraints. As the knowledge repository also contains process
models, teachers should be able to retrieve and configure complete models rather than individual components.
(2) Composition, verification and simulation of process models

When the individual components are selected, the next step is to assemble the selected components into a
process model (learning activity). This involves specification of the various process constraints including both
temporal and structural constraints. This process model is also stored in the knowledge repository and will be
used during run-time for activation of process instances. Again, teachers may need help from knowledge
engineers to compose a process model and decide how to coordinate the selected components. In addition to
individual components, teachers should be able to reuse a pre-assembled process that had been composed and
tested by other teachers. Note that all process components may not be identified in advance. Therefore, teachers
should be able to add or remove components during the actual execution of the process. Finally, before the
assembled process could be used it is necessary to verify its consistency and validity. For example, teachers
should be able to check if it is possible to complete the whole process within the required time (e.g. within 6
weeks) based on the duration of individual components and well as to verify if the components are assembled in
the right logical order (e.g. peer marking comes after assignment submission). Recall that the assembled process
could also include specification of non-IT tasks (face-to-face discussion) and these tasks should also be included
in the process model so it can be properly verified.
(3) Initiation and monitoring of process instances

After the process model has been verified, it is ready to be used by the teacher and his/her students. The same
process model can be reused many times. The actual execution of the given model is called a process instance. A
75
process instance is initiated by an event (e.g. teacher’s action, deadline constraint etc.). In general, a single
process instance is designed for a single course (subject) and shared by all people in that course (teachers and
their students). However, different courses can reuse the same learning design (model) at the same time. In that
case, they will have separate process instances. Obviously, the same student enrolled in two different courses
may participate in more than one instance of the same process at the same time. Obviously, these process
instances will have separate contexts.
The key component here is the coordination support that can be implemented via to-do lists in shared and private
workspaces. In essence, as teachers and students progress through a particular process instance, new tasks will
appear on their to-do lists and the required tools will be available in their shared and private workspaces at the
right point of time. For example, once the assignment is ready for peer review, the task will appear on the to do
list of the student that has been selected to peer review the assignment. Once all reviews are completed a
message will appear on teacher’s to do list to inform him/her that this particular task has been completed.
Although, in most cases, process instances will follow the corresponding process model, teachers should be able
to change the pre-defined model and add new components or add and remove the selected ones during run-time.
However, this flexibility cannot be adequately provided by the existing workflow technology. The same problem
remains if coordination is implemented though SCORM’s navigation and sequencing mechanism. (ADL,
2004b).
Finally, it is important to observe that the instances of the running and completed processes are also stored in the
knowledge repository. This enables reuse of the same experience (model) by the same or different teachers and
learning from completed instances.
(4) Analysis of the accumulated knowledge and experience

In addition to the process models and experiences, teachers should be able to store their experience with the
particular model in the form of comments or their own personal reflections. This should be made available to
other teachers so they could learn from the accumulated experience to create better process models or avoid
repeating the same mistakes.
Figure 4. Web-based handbook – the architecture
A high-level architecture of the web-based handbook is depicted by Figure 4. Development of its prototype is
currently in progress. The current version of the prototype is implemented in Java programming language
combined with Oracle DBMS. This proprietary system is based on a client-server architecture that uses the
Model View Controller (MVC) design pattern. Knowledge repository is stored in Oracle database. At the
moment, it contains a small number of components for problem based learning along with process models
76
(assembled out of these components) and their corresponding instances. The Model tier is designed to
encapsulate all processing logic and knowledge repository maintenance. This level also integrates a special
purpose coordination mechanism that is designed to handle declarative models of learning designs (also invented
for the purposes of this project). The core software within the model tier has been implemented in Java. The
View tier encapsulates the interface modules enabling two different categories of users (teachers and knowledge
engineers) to use the system. This tier has been implemented using Java Server Pages (JSP). The Controller tier
manages the communication between model and view tiers. It has been implemented by using Java servlets. The
initial version of the prototype has been developed to test the underlying theoretical concepts and frameworks. It
is envisaged that the next version of the prototype will adopt the web service architecture where individual
components will be implemented as web services. This solution should provide much better flexibility and
interoperability between components than what is currently the case. At the same time, the objective is to move
from the proprietary system to a system that will be compliant with the emerging educational technology
standards to the extent that is possible without limiting its required flexibility.
Using the web-based handbook
Going back to the motivating examples introduced previously in this paper, suppose that both teachers A and B
are given access to the web-based handbook. The following are some examples of the possible components that
could be stored in the knowledge repository:
¾ Problem registration – This component enables teachers to post the assignment on the web. Students will be
automatically notified that the assignment is ready via e-mails. This will also generate a temporal constraint
(submission deadline) that is used for verification purposes during model composition as well as process
instance.
¾ Registration of groups – When this component is used, students will receive an invitation to register their
groups by the given date. If configured in that way by the teacher, this component may, for example,
automatically form groups of all students who fail to register by the deadline.
¾ Problem-solving component – For example this component may include a number of subcomponents both
generic and subject specific such as electronic brainstorming, mind-mapping, alternative analysis or
collaborative design of ER diagrams.
¾ Assignment submission component – This component can be designed to work similarly to conference paper
submission system or simply to remind the students of the approaching deadline and issue a confirmation
that the assignment has been submitted.
¾ Peer-review component – This component could be configured by the teacher in a variety of ways to support
different roles and coordinate tasks accordingly. For example, the same item can be marked only by the
lecturer or by two different groups of students or even by an external marker.
Note that when stored in the knowledge repository these components are not pre-ordered in any way (e.g. in a
form of a activity tree). Therefore, they could be selected independently as required by a particular learning
design.
Even with this small number of components teachers can create several different combinations of learning
designs. For example, teacher B may assemble all the above components into a process, giving her students a
choice between various problem-solving subcomponents. At the same time teacher A may decide to use only
“problem registration” and “assignment submission” components and supplement them with traditional face-to-
face problem solving activities. Teacher C may request development of a new problem-solving component that
will simulate De Bono’s “Six thinking hats” activity (De Bono, 2001). Teacher D may decide to use electronic
debate instead of the electronic brainstorming tool etc.
Furthermore, when creating learning designs some teachers may decide to assemble a complete model during
design phase and then use it without any changes. Others may decide to use semi-structured or unstructured
models and then complete these models during run-time depending on students’ progress.
In essence, as these teachers gain more and more experience in using the web-based handbook, they will be able
to invent further components and delegate their implementation to knowledge engineers. Once components are
77
developed and stored in the repository, all users (with valid access privileges) will be able to share and reuse the
stored components, process models as well as process-related experience.
Web-based handbook and SCORM Standard

The main purpose of this section is to discuss a possible relationship between the web-based handbook as new
educational technology and SCORM as an emerging educational technology standard. More precisely, this
section explains to what extend the web-based handbook can benefit from the current developments in SCORM.
It also defines new research and development challenges that are yet to be addressed by this or other emerging
standards. For more details on SCORM see (ADL, 2004a and 2004b).
A possible relationship between the web-based handbook and SCORM can be analysed along several important
dimensions as follows:
¾ Instructional design approach
Although the term “content” may have different broad interpretations, SCORM standard may be more suitable
for a Learning Management System designed to support the “resource (or content) – centered” approach to
instructional design. SCORM enables a content to be dynamically sequenced for each learner. Learners are then
monitored as they progress though the content. A learning activity designed for each student is also content-
based. Moreover, learning objectives are narrowly defined and based on student’s progress though the assigned
unit of content.
On the other hand the web-based handbook technology adopts the activity-centered approach to instructional
design. Recall that design of a process-oriented learning design starts from the intended learning objectives.
Here, learning objectives are defined for the whole process. However, in many instances, they may not be
directly associated with student’s progress through the assigned content and therefore, cannot be easily
monitored and measured by technology.
¾ Mode of learning
SCORM standard is designed primarily for an e-learning environment. On the other hand, the web-based
handbook incorporates process-oriented learning designs that have both technology supported tasks as well as
the tasks that could be, but are not necessarily supported by technology (face-to-face discussion, lecture
presentation etc.).
¾ Modeling of a learning activity
In SCORM, a conceptual structure of a learning activity for each learner is called an activity tree. The current
model of an activity tree is not flexible enough to support the requirements of the web-based handbook. When
designing process-oriented learning designs, teachers need more flexibility as they could change the sequence
during run-time as well as introduce new tasks or skip some of the existing tasks. In this way, teachers should be
allowed to interact with the model to make sure that students are using the available tools and resources in the
best possible way and more importantly that they are meeting the intended learning objectives. Furthermore, the
nature of individual tasks may vary. They may not be directly linked to the content and completion of a content
module (as in SCORM).
An activity tree in SCORM is designed for individual learners. In the web-based handbook, models are designed
and components selected for a cohort of students as well as their teachers. Students may progress through some
tasks individually and as a group through the other tasks. To enable collaborative work in some instances,
students have to be aware of each other’s tasks. Then, some tasks are designed for teachers too. Students’ and
teacher’s activities are then coordinated to achieve the intended objective.
¾ Navigation and sequencing (coordination mechanism)
As already pointed out, navigation and sequencing mechanism provided by SCORM could be, to larger extent,
provided also by workflows. Therefore, compared to workflows, the main advantage of the current version of
SCORM is not in the functionality of its sequencing mechanism. Rather, it is in its semantic richness as it is
designed for the educational domain.
78
However, both workflow and SCORM based navigation and sequencing are not flexible enough for the
requirements of the web-based handbook. The current version of the web-based handbook is using a special-
purpose, component-based, declarative model of a process model and proprietary component-based coordination
engine that is more flexible than traditional workflow engine. In the future, a more flexible solution could be
provided by web services.
¾ Integration
In terms of integration, SCORM enables integration and reusability of instruction components. For the purposes
of the web-based handbook it is necessary to consider integration at several different levels. At the technical
level, it is necessary to ensure integration of content as well as various existing applications and tools. Both
SCORM standard and web services deal with integration issues. However, it is more important to enable
integration at the conceptual level (including non-IT tasks) to enable consistent, well-designed and educationally
sound process-oriented learning design.
Finally, design of the web-based handbook supports the vision that the real value of this new type of educational
technology is not in coordination and interoperability mechanisms that support a particular process at the
technical level. Rather, the real value is in the wealth of externalised models of tacit knowledge stored in the
knowledge repository in a form of models of process-related learning designs and their enabling components.
Therefore, the main objective of the web-based handbook is to enable teachers and capture, store, share and
reuse learning experiences.
Web-based handbook – the main research challenges

Design and implementation of the web-based handbook is a complex research and development problem that
cannot be solved by a simple integration of the existing methods and tools available in the identified related
disciplines. This section describes how design and implementation of the web-based handbook creates a set of
new interesting challenges for all related disciplines used in the framework.
¾ Educational theories:
The main research challenge here is how to create an instructionally sound model of process-oriented learning
designs that will match the identified learning objectives. Then, once the model is created, the next challenge is
to support teachers to identify/select, configure and assemble a set of executable components that will support/
enable the intended learning design. These research challenges are likely to result in further extensions of the
activity-centered approach to design of student experience.
Another equally important educational challenge is related to evaluation of process-oriented learning designs.
Therefore, it is necessary to design new instruments that will evaluate the proposed learning designs to make
sure they meet the intended learning objectives. Furthermore, it is also necessary to design evaluation
instruments to evaluate the use of the web-based handbook by teachers and consequently possible effects of
improved knowledge sharing on student learning.
¾ Educational technology:
When designing and implementing the web-based handbook, in addition to content integration, it is necessary to
integrate the existing tools and applications (for example collaboration tools such as chat and bulletin board) as
they could be used in individual tasks. As already pointed out, the integration problem has been solved to some
extend by the emerging SCORM standard (ADL, 2004a) that enables integration of content as well as web
services that enable integration of applications and tools. However, web services need to be designed to comply
with the existing educational standards. This, in turn, may require a possible extension of the existing web
service standards (such as Web Service Description Language WSDL) or creation of new cross-disciplinary
standards (between web services and educational technologies).
In order to support more flexible learning designs it is necessary to extend SCORM even further than what is
currently possible. In that respect, some important lessons could be taken from the field of process-related
technologies. There are more than 100 commercial workflow products, each using a workflow engine to enable
scheduling and coordination of tasks in a much more complex way that what is currently possible with SCORM.
79
Yet, more than decade of research and practical use of this technology has proven that this technology has been
successful only when applied to highly structured, repetitive processes. One of the current challenges in this area
is a possible support for highly flexible emergent processes. On the other hand, SCORM’s navigation and
sequencing mechanism, at least at this stage, appears to be very similar in terms of its functionality.
Consequently, it may prove to be as inflexible as workflow models and their coordination mechanism.
Most importantly, when designing learning designs it is important to concentrate on their instructional value
rather than technology and coordination mechanism. Obviously, the emerging standards are important in order to
create solid and stable technical infrastructure, however their adoption could easily result in the technology-
centered learning. “…Unless all learning specification turn the focus from infrastructure to pedagogical
soundness, they are in danger of becoming instructionally irrelevant…Despite the progress being made on the
interoperability front, that doesn’t necessarily guarantee that what actually runs on SCORM systems will be
worthwhile instructionally.”(Welsch, 2004, pg. 2).
Teaching is highly creative process and should not be restricted by inflexible technology. At the same time,
capturing of this creativity could be made possible, to some extend, with the help of new educational technology
such as the web-based handbook. But at the same time, another standards body should emerge that would create
e-learning instructional standards separately from basic e-learning infrastructure standard (Merill as cited by
Welsh, 2004).
¾ Knowledge-management:
The main research challenge here is to design a simple methodology that will enable teachers (helped by
knowledge engineers) to externalise their tacit knowledge and create process-oriented learning designs in a form
that could be stored, shared, combined and reused. This also includes mapping between the process model and
executable components. Ultimately, this transfer of domain knowledge is necessary for the on-going
development of new executable components.
Another interesting knowledge management challenge relates to various organisational strategies that need to be
developed to encourage sharing of learning designs between practitioners (teachers). The current body of
knowledge in the area of professional communities of practice could also benefit from this research on sharing of
best innovative practices among teachers.
¾ Software Engineering:
The main research challenges here are related to component-based software engineering methods especially in
the area of specification and verification of re-useable (process-oriented) components in the educational domain.
What makes this challenge very interesting is the fact that the end-users, helped by knowledge engineers, should
be able to select, assemble and verify components as well as execute (enact) the resulting processes without any
programming involved.
¾ Process management and process-oriented technologies:
Recall that in the existing process-oriented technologies (such as workflows), process-specialists are in charge of
process modeling. In the case of the web-based handbook, the main process-related research challenges are: end-
user process modeling and support for emergent processes. To tackle the first challenge, it is necessary to design
methodologies and tools for end-user support (e.g. the simulation and verification of process designs). So far,
process-related technologies do not adequately support end-user modeling as tools are designed for professional
process analysts.
Another equally challenging research problem includes support for emergent learning designs. These are
process-oriented learning designs that evolve with experience (for example in the case of problem-based
learning). Currently available solutions in the area of emergent business processes are still immature. However,
design of technology support for emergent learning designs is, in many aspects, much more challenging than
support for emergent business processes. The main emphasis is not on process effectiveness as in business but on
student learning.
80
¾ Web-services:
The area of web service research and implementation is currently emerging with many standards, models and
methodologies yet to be widely adopted. So far, the main driver as well as the application domain has been
business rather than education.
However, the needs of these two domains are very different. Before this promising technology can be adopted
more effectively in education, further investigation of the issues related to web service composition and
deployment in the educational domain is required. This includes the educational, technical, organisational and
social implications of offering learning designs as web services. At the same time, exchange of ideas between
web service and educational technology communities should be further encouraged as they are many concepts
related to process support that could be shared.
Conclusions and future work

“Process-oriented learning designs” is a new term used to describe innovative learning activities that include a
set of inter-related learning tasks (thus, they are process-oriented) and are generic (can be used across different
educational disciplines). Process-oriented learning designs are at the core of socio-constructivist theory of
learning and there is strong evidence that students learn more then they are actively involved in a process rather
than isolated learning tasks. Consequently, there is a need to create and share process-oriented learning designs
among teachers, especially in the world where information technologies constantly change what is possible.
The paper investigates the problems of creation, sharing and IT support of process-oriented learning designs and
proposes a new type of process-oriented, knowledge management educational technology called the web-based
handbook. It is envisaged that this technology will enable teachers to assemble, share, reuse and execute process-
oriented learning designs. Design of this technology is based on the multidisciplinary research framework that
integrates up-to-date research in six different area of research: the existing educational theories, educational
technologies, knowledge management, software engineering, process management and web services. The paper
then uses this multidisciplinary framework to identify the main research challenges that need to be solved before
this technology can be adopted on a larger scale.
In summary, the web-based handbook enables teachers to: (i) reuse not only educational resources but innovative
teaching practices; (ii) combine the same components in different ways (where possible) and create different
learning experiences for their students (iii) share their experience with other teachers and learn from their
successful and less successful experiences.
Current and future work in this area includes further investigation of the identified research challenges related to
both implementation and deployment of this new type of educational technology.
Acknowledgement
This research was partially supported by the UNSW Vice-Chancellor (Education)’s Innovative Teaching and
Educational Technology Fellowship, ITET4, 2003.
References
ADL (2004a). SCORM- 2nd Edition Overview, retrieved November 1, 2004, from http://www.adlnet.org.
ADL (2004b). SCORM- Sequencing and Navigation (SN), Version 1.3.1, retrieved November 1, 2004, from
http://www.adlnet.org.
Agostinho, S, Oliver, R., Harper, B., Hedberg, H., & Wills, S. (2002). A tool to evaluate the potential for an
ICT-based learning design to foster "high-quality learning". In A. Williams, C. Gunn, A. Young & T. Clear
(Eds.), Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in
Tertiary Education, (pp. 29-38). Auckland, New Zealand: UNITEC.
Alavi, M., & Leinder, D. E. (2001). Review: Knowledge Management and Knowledge-Management Systems:
Conceptual Foundations and research issues. MIS Quarterly, 25 (1), 107-136.
81
Bostok, S. J. (1997). Designing Web-based Instructions for Active learning. In Khan B. H. (Ed.), Web-based
Instruction (pp. 225-230), Educational Technology Publications, New Jersey.
Carlsen, S., & Jorgensen, H. (1998). Emergent Workflows: The AIS Workware Demonstrator. Paper presented
at the CSCW-98 Workshop on Towards Adaptive Workflow Systems, November 14, Seattle, USA.
Cristea A., & Garzotto, F. (2004). Design Patterns for Adaptive or Adaptable Educational Hypermedia: a
Taxonomy. Paper presented at the ED-MEDIA’04 Conference, June 21-26, Lugano, Switzerland.
De Bono, E. (2001). Six Thinking Hats, UK: Penguin.
DiGiano, C., & Roschelle, J. (2000). Rapid-Assembly Componentware for Education. Paper presented at the
IEEE International Workshop on advanced Learning technologies, December 4-6, Palmerston North, New
Zealand.
IEEE Learning Technology Newsletter (2004). Special issue on “Learning Communities and Web Service
Technologies, 6 (1), retrieved April 2, 2005, from
http://lttf.ieee.org/learn_tech/issues/january2004/learn_tech_january2004.pdf.
Kayama, M., & Okamoto, T. (2001). The Knowledge Management for Collaborative Learning Support in the
Internet Learning Space. Paper presented at the IEEE International Conference on advanced Learning
Technologies, Issues, Advances and Challenges, August 6-8, Wisconsin, USA.
Malhotra, Y. (2002). Why Knowledge Management Systems Fail? - Enablers and Constraints of Knowledge
Management in Human Enterprises. In Holsapple, C.W. (Ed.), Handbook on Knowledge Management 1:
Knowledge Matters (pp. 577-599), Berlin: Springer-Verlag.
Malone, T. W., Crowson, K., & O’Donnell, E. (1999). Tools for Inventing Organisations: Toward a Handbook
of Organisational processes. Management Science, 45 (3), 425-443.
Marjanovic, O. (2005). Towards IS supported Coordination in Emergent Business Processes. To appear in the
Business Process Management Journal (accepted on 20.1.2004).
Marjanovic, O. (2004). Web Service Business Context - the Normative Perspective. International Journal of
Web Service Research, 1 (2), 16-36.
Marjanovic, O., & Orlowska, M. (2000). Making the flexible learning more flexible. Paper presented at the
IEEE International Workshop on advanced Learning technologies, December 4-6, Palmerston North, New
Zealand.
Oliver, R., Harper, B., Hedberg, J., & Willis, S. (2002). Information and Communication Technologies and their
role in Flexible learning. AUTC project report, retrieved August 25, 2004, from
http://www.learningdesigns.uow.edu.au.
Phillips, R. (2000). Issues involved in developing a project based online unit which enhances teamwork and
collaboration. Australian Journal of Educational Technology, 16 (2), 147-160.
UNSW (2004). Welcome to the Guidelines on Learning that Inform Teaching at UNSW, retrieved April 2, 2005,
from http://www.guidelinesonlearning.unsw.edu.au/.
Van der Veen, J., Jones, V., & Collis, B. (1998). Workflow applied to projects in higher education. Paper
presented at the 3rd International Conference on Cooperative Information Systems (COOPIS’98), August 20-22,
New York, USA.
Welsch, E. (2004). SCORM: Clarity or Calamity? Online Learning Magazine, 29 November 2004.
Yang, J. (2003). Web Service Componentization. Communications of the ACM, 46 (10), 35-40.
Zhang, L.-J. (2004). Challenges and Opportunities for Web Service Research. International Journal of Web
Service Research, 1 (1), 41-57.
82
Deumert, A., & Spratt, C. (2005). Authentic Teaching as the Context for Language Learning. Educational Technology &
Society, 8 (2), 83-93.
Authentic Teaching as the Context for Language Learning

Ana Deumert
School of Languages, Cultures and Linguistics
Research Director ‘Language and Society Centre’
Faculty of Arts, Monash University
3800 Victoria, Australia
Tel: +61-3-9905 9830
Fax: +61-3-9905 5437
Ana.Deumert@arts.monash.edu.au
Christine Spratt
School of Nursing and Midwifery
Faculty of Health Sciences
University of Tasmania
Launceston, Australia
Tel: +61-3-6324 3423
Fax: +61-6324 3297
Christine.Spratt@utas.edu.au
ABSTRACT
This paper reports the redevelopment and subsequent evaluation of a unit in dialectology within a foreign
language curriculum (German). In doing so it is a case study which serves to offer insight into the student
experience of studying linguistics within a foreign language curriculum, the potential of online/electronic
pedagogies for the teaching of dialectology and the way in which creative, authentic teaching builds a
context for learning. The paper begins with a brief overview of the teaching of linguistics in general and
dialectology or language variation studies in particular. It presents the evaluation methodology and
findings, and raises implications for further research. It also provides a summary of some aspects of the re-
development of the unit.
Keywords
Language learning, Dialectology, Online learning, Curriculum evaluation, Authentic teaching
Introduction: The Institutional Context

Monash University is one of Australia’s major research-intensive universities. It is a large multi-campus
university with offshore campuses, a substantial distance education student cohort and a strong on-campus
pedagogical culture at its two major metropolitan campuses in Melbourne. To sustain its strategic imperatives,
the University has supported a number of centrally funded initiatives to encourage Faculty in the use of
technology supported learning, one of which has been the Unit Innovation Grant Scheme (UIGS). In 2003, the
UIGS awards were made to academic applicants who could demonstrate the way in which technology-supported
learning using the University’s corporate learning management system (WebCT) might potentially enhance
students’ on-campus learning experience.
The Faculty of Arts awarded a UIG to the German linguistics unit ‘Dialects and Dialectology’ in 2003. The grant
application proposed to integrate on-line and face-to-face teaching by developing a WebCT unit as a resource
centre, which would (a) allow students flexible access to a range of digital resources in dialect geography (e.g.
the Digital Wenker Atlas, various audio data files), and (b) provide a virtual environment (a ‘meeting place’) for
student discussion in-between classes (through the use of the discussion board module).
The grant application also outlined the intention to develop a general model for online learning within a foreign
language curriculum that could usefully be applied in other pedagogical settings. It was intended that the grant
funds support the development of an online learning environment, and the development of prototypical
electronic interactive activities to support student learning. Centrally, the unit was to be redeveloped to integrate
an academic focus with the fact that the student cohort were language students and that the unit was also
intended to support their process of second language acquisition (in this case German).
83
Teaching Dialectology within a Foreign Language Curriculum
One of the key challenges for tertiary language departments is to ensure that students get sufficient opportunities
for both their language learning and their academic development. Students necessarily spend many of their
contact hours in language classes, i.e. in a teaching and learning mode which does not contribute directly to the
acquisition of research or academic writing skills. Whereas a linguistics student will have three years of
discipline-specific lectures (48 points in the Monash system), a language student specializing in linguistics at
Monash will only attend a maximum of three linguistic electives (18 points; the minimum requirements are 2
electives, 12 points). It is thus important to ensure that the available class time is optimally used, giving the
student a firm grounding, not only in the target language but also in the wider academic field of study (literary
studies, social/cultural studies or linguistics).
The study of language variation, traditionally known as dialectology, is one of the key areas of sociolinguistic
research. Following Britain’s (2002) Good Practice Guide any unit in dialectology should:
¾ provide students with a critical overview of the research tradition which has its roots in nineteenth-century
historical linguistics (‘traditional dialectology’);
¾ introduce students to new approaches to the study of language variation as formulated by sociolinguists
(following the seminal work of William Labov in the US) from the 1960s onwards, including an emerging
focus on social dialects (in addition to regional dialects; ‘sociolinguistic dialectology’); and
¾ give students opportunities for original data collection and analysis.
In other words, students of dialectology should not only be able to understand, explain and critically evaluate
these various research traditions (in particular the contrast between traditional and modern/sociolinguistic
dialectology), but also acquire the skills to ‘do’ dialectology, i.e. to design research projects, conduct interviews
and analyse the thus obtained primary linguistic data. This will help students to develop practical research skills
and to engage directly in processes of discovery, analysis and integration of knowledge which are central goals
of university teaching (Elton, 2001). Dialectology units can also serve to introduce students to questions of
language attitudes (opinions people have about ‘good’ and ‘bad’ language usage) as well as ideological concepts
such as standard language, correctness and language norms (cf. in this context the study by McKinney & Swann,
2001). This can be described as the Research or Academic Focus of the unit.
At the same time the unit is intended to contribute to the students’ acquisition of their second language (in this
case German). This aspect of the curriculum can be called the Acquisition Focus. The incorporation of first-hand
language material and the collection of new data, in particular, allow foreign language students direct and
authentic access and exposure to a range of linguistic data in the target language, including historical texts,
audio recordings, ‘raw’, i.e. not yet interpreted, dialect maps.
The notion of authenticity is commonly used in second language pedagogy to describe learning interactions
which are not artificially constructed, but which are socially and contextually meaningful to the students
(Kramsch, 1993; Van Lier, 1996). Much dialogue in second-language classrooms falls into the category of
‘pretend’ conversations and tend to involve rehearsed and predictable question-answer sequences. Authentic
conversation, on the other hand, ‘refers to students’ active and creative engagement with the foreign language’
(Swann et al., 2004, p. 19).
In addition, careful presentation and discussion of the primary language material, based on the well-established
principles of contrastive linguistic analysis, supports the students’ understanding of dialect forms and the
dialect/standard opposition, and helps to develop their metalinguistic knowledge about the target language
(Tulaswicz & Zajda, 1998 discuss the role of language awareness in language instruction and literacy teaching;
as does Fairclough, 1992; cf. appendix 1 as an example).
The challenge of the unit re-development was to maintain a balance between these two learning and teaching
goals through the pedagogically-sound selection and incorporation of off-line and online teaching material into
the unit. In this context the teacher plays a central role; he or she frequently mediates between the different
learning goals and re-focuses the students’ attention as appropriate.
An exemplary teaching unit which combines dialectology and language acquisition in an online teaching
environment is ‘Dialekt in Deutschland’ (‘Dialect in Germany’), which was developed by staff and students at
the University of Mainz (see http://www.daf.uni-mainz.de/landeskunde/2002_1/Dialekt/inhalt.htm). The unit
‘Dialekt in Deutschland’ consists of three 90 minute modules and introduces the student to a number of issues
and debates surrounding the structure and role dialects in Germany (dialect use and dialect loss, media
84
representations of dialect, attitudes). However, unlike the current project, ‘Dialekt in Deutschland’ is situated
firmly within the curriculum of Deutsch als Fremdsprache (DaF, German as a Foreign Language), and does not
form part of an academic linguistics curriculum at a research-focused university. Nevertheless, ‘Dialekt in
Deutschland’ is highly innovative in that various online resources are used and the student is exposed to a wide
range of authentic language material. It was this unit’s focus on electronic resources which inspired the re-
development of ‘Dialects and Dialectology’ which this paper discusses.
Innovations in the Assessment of ‘Dialects and Dialectology’

Today, the production of hard-copy dialectological maps as well as the publication of linguistic corpora is no
longer economically viable for most academic publishers. Researchers in the field of regional and social
dialectology therefore make extensive use of online technologies to present their work to the wider research
community. This includes maps and (written or acoustic) samples of language data (Deumert, 2001 for an
overview).
Maps: Detailed geolinguistic information and data files are available, for example, for the North American
Linguistic Atlas (http://us.eglish.uga.edu). Some of the maps of the Phonological Atlas of North America, which
was produced under the leadership of William Labov, are also available online
(http://www.ling.upenn.edu/phono_atlas/home.html). In the context of German dialectology, the Research
Institute Deutscher Sprachatlas (http://www.uni-marburg.de/dsa/) allows online access to its digitalized German
dialect maps (DIWA, Digitaler Wenkeratlas; see below).
Corpora: Moreover, a wide range of linguistic (dialectal, non-standard as well as standard language) corpora are
available online. The language data is either presented in its raw form or annotated (i.e. tagged for grammar or
semantics). Michael Barlow’s Corpus Linguistics page at Rice University
(http://www.ruf.rice.edu/~barlow/corpus.html)
provides links to corpora in several languages (including, for example, Chinese, Gaelic, Malay, Russian etc.), as
well as links to sites providing parsing software. Linguistic data bases and parsing software can also be found at
the Linguistic Data Consortium (http://ldc.upenn.edu). The Child Language Data Exchange System (CHILDES;
http://childes.psy.cmu.edu/) gives access to its database of transcripts of first and second language learners (of
English, German and other languages), as well as to software for data analysis. The Institut für Deutsche Sprache
(Mannheim) and the Max Kade Institute (University of Wisconsin) allow access to their extensive German
language corpora.
Given the wealth of linguistic and, in particular dialectological material online, it is highly desirable to
incorporate aspects this material systematically into university teaching and research training. One of the re-
development aims of the unit was to combine available public-domain internet resources in the area of
dialectological research and corpus linguistics with specially developed linguistic exercises (available online and
off-line), discussion groups and notice boards using the WebCT environment. The development of a multi-
layered on-line component allowed a flexible approach to course delivery and self-directed learning, and was
combined with small-group tutorial work.
The assessment of Dialects and Dialectology included several components: two practical homework exercises
(20%, discussed below), a class presentation and a research essay based on the presentation (15% for the
presentation, 40% for the essay), and a final exam (25%).
Analysis and Integration of Knowledge (Assessment 1) - In one of the assessments (Appendix 2) students were
asked to work with original and unedited dialect maps. In the lecture they were introduced to a range dialect-
geographical concepts and the assessment task allowed students to apply this knowledge in a practical research
exercise.
This exercise was made possible by the online availability of Georg Wenker's ‘Linguistic Atlas of the German
Empire’ (Sprachatlas des Deutschen Reichs; data collected between 1876 and 1887). The so-called Wenker-
Atlas shows language data for approximately 50 000 different locations in the German-speaking areas. The
original maps were hand-drawn and are unparalleled in their attention to detail and clarity. The original atlas has
never been published and until recently has been available only to senior researchers at the University of
Marburg. The situation changed when the German Research Foundation (Deutsche Forschungsgemeinschaft)
funded the digitalization of the original maps through a major research grant (as part of its “Retrospective
85
Digitalization of Library Holdings” program). The Digital Wenker Atlas is currently in the developmental phase;
however, many of the maps are already available online and are thus accessible to Australian university students.
The dialect maps of the Winker Atlas include all the detail of the original questionnaires and need to be
interpreted carefully. These ‘raw’ dialect maps contrast strongly with the simplified, interpretative maps
provided in textbooks (e.g. Barbour and Stevenson, 1990), and give students an opportunity to engage with the
‘real-life’ complexities of linguistic analysis. Data reduction and pattern recognition are skills which students
will employ in the completion of this assessment.
Collecting and Analysing New Data (Assessment 2) – Authenticity was not only achieved by providing students
with authentic research material (unedited maps or conversational transcripts), but students also engaged in data
collection themselves. Since the unit is taught in Australia the opportunities for German language data collection
and fieldwork is limited. In order to allow students to experience dialects first-hand, two research assistants were
appointed to accompany the course. Both research assistants had grown up with a German dialect (Swabian and
Swiss German respectively) as their first language.
Students were given a short introduction to the methods and approaches of linguistic fieldwork (including
vocabulary collection, establishing morphological paradigms and syntactic structures), and were provided with a
comprehensive reader which introduced them to the practicalities and ethics of linguistic fieldwork. They were
then divided into groups of three and were given 60 to 90 minutes with each of the native speaker consultants.
Prior to their meeting with the native speaker, students had the opportunity to develop and discuss their
collective fieldwork design, using the WebCT discussion board module (in addition to face-to-face meetings
which were scheduled during a study week).
Students were allowed to record their data collection session and were asked to focus either on phonology or
well-defined aspects of grammar (e.g. verbal paradigms). Students submitted their analyses of the collected data
as part of their assessment. In addition, students were asked to reflect critically on the fieldwork session and to
make suggestions for improvement in their research design.
The fieldwork session allowed students to interact with native dialect speakers, collect (record and transcribe)
authentic data for further analysis, demonstrate their linguistic skills and abilities in the analysis, and learn about
linguistic methodology (elicitation techniques).
We developed the assessment strategies with the hope that they would be motivating for students’ learning (after
James, 2002) as well as measures of successful student learning outcomes, the usual purpose of assessment
tasks. This is discussed in more detail in the Evaluation Findings section.
The Evaluation Method

The unit coordinator (the first author) believed that the unit required a systematic and thorough evaluation
integrated with the re-development. The unit coordinator collaborated with the second author, then a senior
academic in the University’s Centre for Learning and Teaching Support, in conceiving a qualitatively focussed
evaluation strategy. The unit evaluation in this case report is ‘illuminative’ (Fetterman, 1989) and provides
insight into the current students’ experience of the pedagogical strategies adopted especially the seminar
approach to unit ‘delivery’, the assessment tasks and the student use of an embryonic online learning support site
developed in WebCT. It was anticipated that the outcomes of the evaluation would inform the way in which the
unit re-development could be aligned with the intentions of the UIGS award proposal outlined above.
The co-authors have a shared understanding that unit evaluation is iterative, fundamental to curriculum renewal,
and requires academic teachers to ‘catch themselves in the act’ of teaching so to speak, that is to be critics of
their own practice. We also hold central the view that students ought to be active participants in curriculum
evaluation. Student participation in the evaluation was voluntary. As the student cohort was small and as we
were eager to integrate evaluation with the re-development, formatively, a primarily qualitative evaluation
method (focus group interview) was used. Monash University does support a standardised approach to the
collection of student evaluation data through survey questionnaires. We asked students to complete the standard
survey questionnaire: too few were returned to be of value. However, anecdotally, students told co-authors that
they felt that the opportunity to be ‘heard’ through the focus group interviews and their day-to-day interaction
and feedback with the unit coordinator made them feel confident that they had ‘said what we wanted to say’ and
they were disinclined to fill in an institutional form (they claimed ‘evaluation fatigue’!).
86
Over the course of the semester, two focus group interviews were conducted with the students, one at Week 4
(15 of 20 students participated as did the Faculty’s Associate Dean of Teaching) and again at Week 10 (11
students of 20 participated). In the following discussion, direct quotes from the qualitative data drawn from the
focus group interviews are in quotes and italicized. The co-authors collaborated on developing the semi-
structured interview schedules, which were based on identified areas of pedagogical interest, especially the
assessment strategy, the seminar approach and the use of the WebCT site. Qualitative data from the focus group
interviews was collated and analysed and key themes emerged which are discussed subsequently.
Evaluation Findings
Student’s conception of the unit – Students anticipated the unit would assist them ‘broaden their knowledge of
dialectology’. Students understood that the unit’s primary aim was to develop their skills in German dialectology
and they believed that the best way for them to do this was to be ‘doing practical things’. The cohort had varied
backgrounds and experience in the field; three students were native German speakers, some perceived they had a
good background in linguistics, others did not. In the Faculty of Arts at Monash University linguistics is studied
formally at Year 1 as part of a core of German language units. Students generally agreed that the Year 1
linguistics unit was ‘overloaded’. Many of them ‘didn’t remember a thing about it’ [Year 1 linguistics]. They
did not believe that their limited experience and recollection of linguistics would affect their learning
detrimentally in this unit.
Perceptions of successful learning in the language classroom – Students clearly enjoyed the unit and they
uniformly attributed this to the ‘fantastic teacher’. This group of students would not change the seminar
approach. They liked the three-hour face-to-face sessions and saw the online activities as ‘complementary’. They
enjoyed participating in extra-curricular activities organised by the unit coordinator as part of the pedagogical
program. Students stated they ‘had no specific goals’ for themselves when they began the unit aside from a
general aspiration to ‘learn more about dialectology’. The students were generally pleased that they enjoyed the
unit and learnt more than they anticipated. They argued that the general unit goals articulated in the formal
course documentation were a means of ‘laying a foundation’ and ‘providing stepping stones’ for their study.
They commented that in other units they have studied the unit outcomes or synopsis are ‘often rubbish’ and
bears little resemblance to the way the unit is taught and structured. They liked descriptive terms used in
understandable language to help them move forward. They were uniformly supportive of the approachable
teaching style. Once they started the unit, they ‘did not want to turn back’ and they claimed they ‘learnt a lot’
and ‘it was enjoyable’. Students valued clarity in knowing what was expected of them and a clear articulation of
how the unit activities were intended to help them learn.
Students believed that this unit has given them a ‘broader experience than just learning’. They considered the
unit to be ‘diverse’ in its approaches such as the use of media through WebCT contributing to their perception
that the unit was ‘well rounded’ and gave them a ‘comprehensive’ exploration of dialectology. They attributed
their success to the way in which the unit coordinator set her expectations clearly, they believed strongly that
‘she wanted us to succeed’ and this contributed to their motivation for learning in the unit. The students
perceived ‘personable’ and ‘responsive’ teaching as integral to their success as learners.
Using WebCT to support language learning – The participants had used WebCT in other courses they have
studied at Monash. They appreciated that accessibility issues for students are very important to consider when
designing online learning environments. They discussed this briefly in relation to accessibility to the Internet and
also in terms of students having software capability appropriate to online learning activities e.g. file size of
audio/video and having appropriate download capacity. The WebCT site was highly valued as a single point of
access to resources: they liked the fact that ‘there was one point of reference for unit material’. They also
appreciated that the WebCT unit site meant that they could ‘access it from home’ or ‘anywhere they wanted,
even at 2am’.
Participants liked the online discussion forum and used it purposefully when they thought they needed to. They
believed that online learning environments ought to be ‘relevant to the purposes of the unit’ and interesting to
them intellectually. Participants recognized that the success of any online discussion lies in its’ ‘responsiveness’.
In this context particularly they spoke highly of the unit coordinator and appreciated the consistency of her
approachable teaching approach to their work in the face-to-face classroom and the shared online spaces.
87
Perceptions of ‘valuable’ assessment tasks – Participants agreed that unit assessment contributes to helping
them gauge ‘workload and effort’ required by any particular unit. Generally, they believed that success in any
unit assessment was a good measure of their learning. They believed that assessment at tertiary level ought to be
cohesively integrated continuously throughout a semester and they believed that this unit tried to achieve this
where many of their other units did not.
They debated the value of the end-of-semester examination as a measure of successful learning as they felt that
they had demonstrated their expertise enough in the other assessment tasks. They also argued that ‘weighting’ is
often problematic where the demands of some assessment tasks meant that ‘a lot of real work goes in to some
assignment that might only be worth 20%’.
The use of group assessment tasks is often problematic for students and indeed teachers. Participants believed
that all assessment ought to have ‘clear, unambiguous criteria’. They felt strongly that they ‘don’t need group
work in every subject’ and they did not seem confident that ‘individual marks from group assignments are
always realistic of an individual’s effort and learning’. The students in this unit believed that ‘different students
have different goals and expectations of group assessment’. They felt strongly that good teaching ought to assist
them succeed in group learning and peer assessment by helping them understand its purposes and processes.
Summary findings
In summary, the illuminative evaluation reported in this case study provided data about the students’ experiences
of learning in a particular setting. The key issues raised are summarised here.
A well designed WebCT site can enhance learning – Participants saw the WebCT site evolving as a valuable
‘adjunct’ to their face-to-face sessions. They considered it a ‘meeting place’ which gave them increased
flexibility; as a central point where they could easily access print and other online resources and discussions they
needed for their learning development. Having experienced this unit with its use of the tools of WebCT they
agreed that there was potential for language units such as this to be available to a wider audience of off-campus
learners. The associated potential of multi-media applications in the pedagogy of language learning was also
obvious to them. Importantly however, while students can see the value and potential of educational technologies
such as WebCT, they were quick to point out that ‘nothing can replace face-to-face teaching with a teacher like
[the unit coordinator]’.
Assessment tasks ought to be relevant, pragmatic and challenging – It is widely accepted in tertiary teaching
that assessment largely defines the curriculum for today’s busy students. Participants in the evaluation reported
here clearly articulated the way in which assessment tasks assist them in gauging both workload and anticipated
effort of a particular unit of study. As Schell (1986: 411) articulated in a key paper two decades ago, if students
‘are to learn desired outcomes in a reasonably effective manner, the teacher’s fundamental task is to get students
engaged in learning activities that are likely to result in achieving outcomes’.
The re-development of the unit, and especially the design of assessment tasks, demonstrates a dynamic and
innovative approach to the development of pedagogical strategies to meet the outcomes outlined in the first
section of this paper. The authors believe that it offers a potentially exemplary model for academic teaching in
foreign language departments. Since the (language) acquisition focus of this and other linguistics units is
cumulative and contributes to the incremental development of meta-linguistic knowledge, it was not practical to
assess actual improvements in the language learning of the students at the end of the semester. Moreover,
students had different levels of German and were distributed over a number of classes (from intermediate to
advanced). The acquisition effects of the unit are believed to be gradual, stretching across the three year
undergraduate course of these students. However, the student cohort showed progressively higher marks across
the semester, in many cases moving from marks in the mid 50s or low 60s to distinctions in the exam and final
essay. It would certainly be a worthwhile longitudinal research project to compare the language performance of
students specializing in linguistics (where all academic seminars incorporate an explicitly articulated acquisition
focus and aim to contribute to the development of meta-linguistic skills) with that of students specializing in
literature (where language is primarily a tool for accessing the national literary tradition and the focus is on
meaning rather than form). Other linguistic seminars offered in the German program at Monash University
include Second Language Acquisition and Language and Society.
Assessment ought to see students motivated, engaged and active. While students often take a strategic and
pragmatic approach to assessment as our findings have indicated, we believe that assessment ought to refocus
88
students’ attention so it is conceived as central to learning; that is, assessment tasks ought to ‘capture learner
attention, they ought to promote motivation, encourage students to harness their time and other personally-held
resources towards defined outcome goals, and as such encourage the prioritisation of these resources’ (James
2002, p. 10) In other words as James (2002) argues, assessment ought to be ‘performative’: students actively
engaged in assessment tasks that can both promote and indeed measure learning.
Students recognise and value ‘authentic teaching’ – Promoting authenticity in tertiary teaching needs to
recognise the ‘creativity’ inherent in it. The unit coordinator and her ‘pedagogy’ was seen by her students as
personal, unique and exemplary. She was perceived by her students as an inventive, critical thinker, a scholar
whose open, responsive manner demonstrated explicitly her student-centred teaching philosophy, her innovative
approaches to pedagogy and curriculum development and her recognition that student learning is the crux of
pedagogy; that is teaching as the ‘context’ for learning (Green 1998). Teaching ‘style’ is central to students’
understanding of good practice and successful learning in face-to-face and online settings.
Participants were clearly able to differentiate what they believed contributed to ‘good’ teaching practice in the
face-to-face classroom and online, as ‘being responsive and encouraging’, with what they perceived as ‘bad’
online teaching practice which several had experienced in other units; ‘where lecturers just can’t be bothered’ to
develop meaningful and relevant learning and teaching strategies or to interact in ways participants believed
would assist their learning.
O’Dea (1992) has argued that being an authentic teacher implicates those so recognised with a responsibility to
live in the knowledge that they are ‘self-creating, self-legislating beings’ and that acting ‘authentically’ is to act
sincerely and honestly in a manner that is true to oneself as such a being’ (p. 44). In this case study, participants
regarded the unit coordinator highly. They clearly respected her expertise and appreciated her experience as an
academic teacher and researcher and the contribution this made directly to their development as learners in this
unit is evident. An open, responsive and engaging approach on the part of the teacher is central to developing a
very positive learning experience for students. In this case participating in the evaluation demonstrated to
students that the unit coordinator took her teaching seriously and evaluation seriously; they appreciated ‘being
heard’. They felt confident that their contribution to the evaluation would result in demonstrable change and
ongoing improvement in this unit. In this case study the unit coordinator articulated and demonstrated in her
practice the deep connections between the ethics and morality of teaching and pedagogical behaviours. As she
has come to create such forms of excellence in pedagogy and pedagogy for transformative change she
exemplifies the ‘cultivation and exercise of the virtue of authenticity’ as a teacher (O’Dea 1992, p. 48).
Concluding comments and implications for further research

As a case report this paper allows us to present an ‘evidentiary boundary around which we can discuss
educational problems’ (Walker, 2002: 10) and propose areas of interest for further research. For us, the insights
drawn from the evaluation have allowed us to propose a curriculum redevelopment which will be conceived and
designed so that online strategies can both add flexibility and learning enhancement to the face-to-face seminar
series and to develop the program for online/off-campus delivery.
Principally, the evaluation has reinforced for us that the nature of online interactive exercises and assessment
ought to be driven firmly by the intended educational purposes and that the online as well as the classroom
environment needs to be interesting, challenging and, importantly, responsive. The evaluation in conjunction
with the unit coordinators creative pedagogy has established an imperative for unit redevelopment which will
include:
¾ The strategic use of electronic linguistic data and the development of specific online and offline exercises to
develop students’ linguistic research and data collection/analysis skills.
¾ The further development and refinement of the existing WebCT site so that it becomes reflective of a
learning environment that will support the pedagogical strategies, the student learning experience and act as
a ‘resource repository’.
¾ The development of a CD-Rom which will accompany the learning process and which allow students to
work off-line.
The evaluation has helped us recognise that assessment is a learning strategy not merely a summative ‘measure’.
Moreover, we have become more aware that student perception of assessment and their beliefs about the impact
of assessment on their learning can be powerful tools to assist academic staff develop relevant and valid
assessment tasks. James, McInnis & Devlin (2002) conceded in their recent major study ‘Assessing Learning in
89
Australian Universities’, that students value ‘unambiguous expectations, authentic tasks and choice and
flexibility’ in assessment.
Finally, the evaluation has served to reveal in this case, that students hold very firm beliefs about what
constitutes ‘good teaching’ and that it is essentially about creativity and authenticity. Spratt (2002: 157) has
argued that creative teachers as early adopters of technology ‘look like’ Getzels & Csikszentmihalyi’s (1976:
183) creative problem-posing artists:
who approach their work with personal commitment yet without stereotyped problems in mind,
not only produce drawings that are rated more original and of greater aesthetic value, but persist in
art longer. It would seem that problem finding is an integral part of a person’s cognitive style: it is
a reliable characteristic. A problem finding orientation seems necessary for creative work.
Getzels & Csikszentmihalyi (1976: 247) conclude that ‘a creative problem cannot be fully visualised in the
“mind’s eye”; it must be discovered in the interaction with the elements that constitute it’. Perhaps ‘interaction
with the elements that constitute it’ in the context of creative teachers as early adopters of technology equates to
the intellectual, cognitive and emotional engagement with practice that so characterises the way in which
academics ‘become’ creative teachers. The capacity to become creative, reflective and problem-posing may
force creative academic teachers engaged in technology and pedagogical innovations, to take risks in their
practice, to avoid oversimplifying the curriculum, and to look for ways to engage learners in authentic learning
environments which are more deeply concerned with creating intellectual transformative change in students
(Spratt 2002). The paper has also supported the growing literature exploring collaboration between academic
colleagues and their students to develop quality learning environments.
There are several areas for further research which the work has highlighted for us. Importantly these include
issues related to the pedagogical discussion laid out in the first section of the paper. In this way the evaluation
and its findings has served to support in an iterative and reflective way the pedagogical thinking of the unit
coordinator about the redevelopment of the unit as the evaluation unfolded. We are led therefore to potential
investigations of the way in which online environments ought to be structured so that students are engaged in a
rigorous pedagogy that prepares them to work in a field that requires advanced skills of linguistic analysis and
research. It has also raised research interests for us in ways in which language teachers may develop peer
assessment tasks and group learning opportunities that demonstrate valid and reliable measures of individual
learning outcomes from shared experiences. Exploring the way in which the undergraduate foreign language
curriculum can best support growth over time towards the development of advanced language skills, discipline
knowledge and practical research skills remains a challenge as does characterising the most effective ‘teaching’
discourses for the language classroom to prepare students for the challenges of their chosen discipline. Finally,
investigating teaching as the ‘context’ for learning may illuminate ‘authenticity’ in teaching and the impact this
has on the experience of learning for students.
References
Barbour, S., & Stevenson, P. (1990). Variation in German, Cambridge: Cambridge University Press.
Britain, D. (2002). Dialectology. (A Guide to Good Practice for Learning and Teaching in Languages,
Linguistics and Area Studies, retrieved July 30, 2004 from
http://www.lang.ltsn.ac.uk/resources/goodpractice.aspx?resourceid=964.
Deumert, A. (2001). Sociolinguistic Internet Resources. In Mesthrie. R. (Ed.) Pergamon Encyclopaedia of

Sociolinguistics, Oxford: Pergamon Press, 830-834.
Elton, L. (2001). Research and Teaching: conditions for a positive link. Teaching in Higher Education, 6, 43-56.
Fairclough, N. (Ed.) (1992). Critical Language Awareness, London: Longman.
Fetterman, D. (1989). Qualitative Approaches to Evaluation in Education: The Silent Scientific Revolution, New
York: Praeger.
Getzels, G., & Csikszentmihalyi, M. (1976). The Creative Vision: A Longitudinal Study of Problem Finding in
Art, New York: John Wiley and Sons.
90
James, R., McInnis, C., & Devlin, M. (2002). Assessing Learning in Australian Universities, Centre for the
Study of Higher Education, University of Melbourne, retrieved April 23, 2004 from
http://www.cshe.unimelb.edu.au/assessinglearning/.
Kramsch, C. (1993). Context and Culture in Language Teaching, Oxford: Oxford University Press.
McKinney, C., & Swann, J. (2001). Developing a sociolinguistic voice? Students and linguistic descriptivism.
Journal of Sociolinguistics, 5, 576-590.
Shuell, T. (1986). Cognitive conceptions of learning. Review of Educational Research, 56, 411-436.
Spratt, C. (2002). Critical Teachers as Early Adopters of Technology: Agency and Structures of Indifference in
Higher Education, Unpublished Doctoral Thesis, Geelong: Deakin University.
Swann, J., Deumert, A., Mesthrie, R., & Lillis, T. (2004). A Dictionary of Sociolinguistics, Edinburgh:
Edinburgh University Press.
Tulaswicz, W., & Zajda, J. (Eds.) (1998). Language Awareness in the School Curriculum, Melbourne: James
Nicholas.
Van Lier, L. (1996). Interaction in the Language Curriculum: Awareness, Autonomy and Authenticity, London:
Longman.
Walker, R. (2001). Case study, case records and multimedia, Centre for Applied Research in Education,
Norwich, University of East Anglia, retrieved April 4, 2004 from
http://www.uea.ac.uk/care/people/RW_recent_writing/final_14_09.html.
91
Appendix 1: In-Class Group Exercise
In this exercise students were given a short dialogue from an Asterix cartoon (the popular Asterix cartoons have
been translated into a number of German dialects). The students were asked (a) to translate the dialect text
provided in the cartoon into standard German, (b) to identify phonological and grammatical features of the
Cologne dialect.
---------------------------------------------------------------------
Asterix dialogue (available at http://www.cologneweb.com/asterix/)
The opening dialogue between Asterix and Obelix from the volume Däm Asterix singe Jung, translated by Alice
Herrwegen, Ingeborg Nitt, Volker Gröbe, Gerd Schumacher (members of the Akademie för uns kölsche Sproch
‘Academy for our Cologne Language’).
Picture 1: Wie alle Dääch, su jeit och hück de Sonn üwer däm kleine jotbekannte Dörp vum Asterix op,
wo luuter Freud und Fridden es ...
Picture 2: ... bloss jestört vun däm einzije jallische Hahn, dä schnorks
Picture 3: Kock, Kock, wat läuf Kock?

Kick Kick es et ald widder esu wick?
Picture 4: Erus us der Lappenkess! Dat weed secher ne herrlije Daach!

Huah!
Picture 5: Dis Naach han ich janz jet jeckes jedräump, Asterix.
Picture 6: Ich han jedräump, de Stösch wören üwer uns Dörp jefloge, för de bestellte Quös afzeliwere.
Einer vun denne hät sich ävver verdon un eine Ditz he bei uns afjesatz.
Picture 7: No saach bloss, do jläuvs noch aan de Klapperstorch, dä de klen Kindercher brengk?
Woröm dann nit? Ich liwere jo och de Hinkelstein, oder?
Picture 8: Eines Daachs muss ich deer een ‘n paar Saache usenanderpusamenteere, Obelix!
---------------------------------------------------------------------
The exercise allowed students to discover a range of phonological and grammatical features of Kölsch, an urban
dialect spoken in Cologne and surroundings.
Phonological features include e.g.

Standard German f > p (dorp instead of dorf ‘village’)
Standard German g > j (word-initial; jeit instead of geht ‘goes’)
Standard German g> ch [x] (word-final; daach instead of tag ‘day’)
Consonant cluster simplification (jedräump instead of geträumpt, ‘dreamed’), etc.
Grammatical features include e.g. plural formation through vowel change rather than suffixation:
Standard German Tag (‘day’) – Tage (‘days’)
Kölsch Daach (‘day’) – Dääch (‘days’)
The phonological features discussed in this exercise then form the basis for the following lecture in which
students are introduced to the Hochdeutsche Lautverschiebung (‘High German Sound Change’) which is of
central importance to the linguistic classification of the German dialects.
92
Appendix 2: Dialect Map Interpretation Exercise (accessible to students via WebCT)
Instructions
Go to the website of the Forschungsinstitut für deutsche Sprache (‘Research Institute for the German Language’,
Marburg; http://www.uni-marburg.de/dsa/) and follow the link to the Digital Wenkeratlas
(http://www.diwa.info/). Click on the link Kartenverzeichnis.
It is possible that you will need to install additional software (plugin). This software is free of charge and can be
downloaded from the webpages of the Digital Wenkeratlas.
Look at the following maps:

Dorf (‘village’, http://137.248.81.135/DiWA/ECW.asp?ID1=221&V=0&S=0)
machen (‘to make’, http://137.248.81.135/DiWA/ECW.asp?ID1=417&V=0&S=0).
Carefully describe the linguistic variants which can be found in the area around Düsseldorf, Köln and Bonn.
Compare the detailed map with the interpretative map given in Barbour & Stevenson (1990). Critically evaluate
the differences between the detailed map (also called display map) and the interpretative map.
93
Jung, I. (2005). ICT-Pedagogy Integration in Teacher Training: Application Cases Worldwide. Educational Technology &
Society, 8 (2), 94-101.
ICT-Pedagogy Integration in Teacher Training: Application Cases

Worldwide
Insung Jung
Educational Technology and Communications
Division of Education, International Christian University
3-10-2 Osawa, Mitaka-shi, Tokyo 181-8585, Japan
Tel: +81 422-33-3125
isjung@icu.ac.jp
ABSTRACT
Teaching is becoming one of the most challenging professions in our society where knowledge is
expanding rapidly and modern technologies are demanding teachers to learn how to use these technologies
in their teaching. While new technologies increase teachers’ training needs, they also offer part of the
solution. Information and communication technology (ICT) can provide more flexible and effective ways
for professional development for teachers, improve pre- and in-service teacher training, and connect
teachers to the global teacher community. This paper analyses and organizes a variety of approaches
found in ICT uses in teacher training into a four-cell matrix. Based on the analysis of those approaches, it
discusses new possibilities and challenges that ICT has brought to teacher training and professional
development. It concludes with discussion of emerging research issues with respect to ICT integration into
teacher training and networking.
Keywords
ICT teacher training, ICT use, online training, teacher training, training approaches
Introduction
While information and communication technology (ICT) is not a panacea for all educational problems, today’s
technologies are essential tools for teaching and learning. To use these tools effectively and efficiently, teachers
need visions of the technologies’ potential, opportunities to apply them, training and just-in-time support, and
time to experiment. Only then can teachers be informed and confident in their use of new technologies (Bowes,
2003).
Teaching is becoming one of the most challenging professions in our society where knowledge is expanding
rapidly and much of it is available to students as well as teachers at the same time (Perraton, Robinson, & Creed,
2001). As new concepts of learning have evolved, teachers are expected to facilitate learning and make it
meaningful to individual learners rather than just to provide knowledge and skills. Modern developments of
innovative technologies have provided new possibilities to teaching professions, but at the same time have
placed more demands on teachers to learn how to use these new technologies in their teaching (Robinson &
Latchem, 2003). These challenges ask teachers to continuously retrain themselves and acquire new knowledge
and skills while maintaining their jobs (Carlson & Gadio, 2002). Then what can be done to help teachers meet
these challenges?
Today, a variety of ICT can facilitate not only delivery of instruction, but also learning process itself. Moreover,
ICT can promote international collaboration and networking in education and professional development. There's
a range of ICT options – from videoconferencing through multimedia delivery to web sites - which can be used
to meet the challenges teachers face today. In fact, there has been increasing evidence that ICT may be able to
provide more flexible and effective ways for lifelong professional development for today’s teachers.
Because of rapid development in ICT, especially the Internet, traditional initial teacher training as well as in-
service continued training institutions worldwide are undergoing a rapid change in the structure and content of
their training and delivery methods of their courses. However, combining new technologies with effective
pedagogy has become a daunting task for both initial teacher training and in-service training institutions.
This paper looks at a variety of approaches in ICT-Pedagogy integration in teacher training. Those approaches
are organized into a four-cell matrix and elaborated with the collection of cases in an international context. Via a
variety of examples, the author tries to show that ICT use is not only a matter of new possibilities but that it also
brings with it new implications and new challenges. The paper concludes with a discussion of emerging research
94
issues involved in ICT applications in teacher training. Methods employed in this study include desk analyses of
various ICT teacher training cases and approaches published in selected articles or websites and face-to-face or
email interviews with experts for verification.
Teacher Training Approaches

Research indicates that ICT can change the way teachers teach and that it is especially useful in supporting more
student-centered approaches to instruction and in developing the higher order skills and promoting collaborative
activities (Haddad, 2003). Recognizing the importance of ICT in teaching and learning, a majority of the
countries in the world have provided ICT teacher training in a variety of forms and degrees. Even though many
teachers report that they have not had adequate training to prepare themselves to use technology effectively in
teaching and learning, there seem to be several efforts around the world in which countries are effectively using
technology to train teachers, and/or are training teachers to use technology as tools for enhancing teaching and
learning.
ICT teacher training can take many forms. Teachers can be trained to learn HOW to use ICT or teachers can be
trained VIA ICT. ICT can be used as a core or a complementary means to the teacher training process (Collis &
Jung, 2003). This paper organizes various ICT teacher training efforts found in different countries into four
categories using the framework of Figure 1.
Figure 1. Categories for ICT in teacher training (adapted from Collis & Jung, 2003, p.176)
ICT use as main content focus of teacher training

Most of the early ICT teacher training programs in the 1990’s focused on ICT use as the main training content.
This approach has an emphasis on teacher training in how to use ICT in the classroom. It addresses issues such
as selecting appropriate ICT tools and supporting students in the use of those tools, using ICT to promote
learning activities, developing new methods of facilitating learning and evaluating student performance, and so
on. One example case from Singapore’s initial teacher training is provided below.
95
As Singapore’s only pre-service teacher training institute, the National Institute of Education (NIE) was
entrusted with the responsibility for integrating ICT into initial teacher training programs based on the nation’s
Masterplan for IT in Education. Accordingly, the NIE developed and began implementing a new ICT plan in
1998, which identified four main areas that needed change: curriculum; physical and technological
infrastructure; human resource infrastructure; and R & D in the use of ICT in education. For the purpose of this
chapter, I will focus on how NIE has revised its curriculum to promote ICT use in the classroom for future
teachers (Jung, 2001).
The curriculum was revised to include three kinds of ICT courses for student teachers: basic ICT-skill
workshops, a 30-hour ICT foundation course, and a 26-hour elective course. In addition, the 6 to 12 hours of
ICT integration into each curricular subject class was recommended. Basic ICT skill workshops, paid for by
students, are provided by external organizations and cover word processing, PowerPoint, Internet literacy, and
other technical skills. A 26-hour elective course covers the design and production of computer-based instruction.
A 30-hour ICT foundation course is entitled “Instructional Technology” and covers: “learning, thinking and the
effective use of instructional technologies in the classroom; instructional planning models; selecting, creating,
evaluating, and integrating instructional technologies and resource materials; promoting creativity and complex
thinking through ICT project work activities; and organizing and managing instructional activities with
appropriate ICT resources in the classroom.” Besides taking these courses, NIE students pursuing a Diploma in
Education must have five weeks of practicum during the first year of their pre-service training and ten weeks
during the second year. The trainee is expected to use ICT while teaching, depending on the school’s ICT
infrastructure.
As shown in the NIE’s case, this approach of using ICT as the main content focus of teacher training emphasizes
the development of basic ICT skills, design and development skills, and pedagogical strategies. However, the
basic ICT skill development, rather than the ICT-pedagogical integration, has been the major concern of teacher
training. When interviewed about the new teacher training curriculum, student teachers at NIE agreed that the
foundation course provided useful pedagogical strategies for the use of ICT in classroom teaching. However,
they also reported that the 30 hours of instruction was not enough time to gain proficiency in ICT-pedagogy
integration, and some wanted more ICT-pedagogy integration in the practicum. The next approach provides a
more effective way of ICT-pedagogy integration in teacher training programs.
ICT use as part of teaching methods

This approach integrates ICT into teacher training to facilitate some aspects of training. Two cases below show
how a variety of ICT are adopted as part of effective training methods. In these cases, teachers are provided with
examples of ICT-pedagogy integration in their training process.
Captured Wisdom (http://www.ncrel.org/cw/) is a resource developed by the federally-funded (USA) North

Central Technology in Education Consortium for K-12 teachers, school administrators and extended to adult
literacy educators. It uses videotape and CD-ROM to help US teachers to see how technology can be integrated
into their work. The Captured Wisdom (tm) CD-ROM Library is made up of stories about teachers who are
making meaningful and creative uses of technology in their instruction. These CD-ROMs contain video
descriptions and demonstrations of how technology is used in teachers' classrooms. They provide “examples of
real educators and learners using successful practices of technology to support instruction and learning in their
classrooms.” Video sequences are viewed by teachers' focus groups who then discuss the strategies and
techniques of classroom management, assessment, etc. In this specific case, teachers learn how to use ICT in
their classrooms by actually being engaged in the process of ICT-integrated training.
Another example of this approach can be found in the School Administrators’ Technology Integration Resource
(http://www.satir-ritas.org) project. It is a bilingual Canadian initiative which provides tools and resources to
help school administrators successfully integrate ICT into curriculum in their school. It includes the National
Center for Technology Planning clearinghouse of school district ICT plans, advice on how to provide
technology, successful practices in introducing ICT, perspectives on staff development, a beginners’ guide to the
Internet, etc. The focus of this project is not on the basic skill development but on the development of ICT-
pedagogy integration skills of educators by sharing successful cases and practical ideas.
UNICEF’s Teachers Talking About Learning (http://www.unicef.org/teachers/) also illustrates the application of
this approach to ICT teacher training. It is designed for international collaboration between teachers in
96
developing countries using the Internet and television. It provides access to teacher training materials and useful
links and promotes discussions among teachers.
All the cases discussed above use ICT as part of training methods and promote teachers’ ICT-pedagogy
integration in the classroom by demonstrating examples and allowing discussions among teachers throughout the
whole training process. Participants of the training are asked to actually use ICT to learn about ICT skills and
develop ICT-integrated pedagogies. These training strategies seem to be supported by previous research that
argues that teachers are likely to benefit by actively experiencing ICT skills as a learner (Jung, 2003).
ICT as core technology for delivering teacher training

In this approach, ICT is used as the major way of providing the learning experience of teacher training. The
content of this approach does not necessarily focus on ICT skill itself but rather covers a variety of ICT
applications. As you will see in the two examples below, the digital technology is frequently becoming the core
technology of ICT teacher training.
The case of the Virtual High School (VHS: http://www.govhs.org/website.nsf) in the USA provides an example
of the Internet-based ICT teacher training. VHS is a non-profit organization that facilitates a collaborative of
participating secondary schools; for every semester a participating school offers a VHS NetCourse that school
can enroll up to 20 students in VHS courses. A limited number of student-only schools are allowed to enroll
students (10 per semester) on a trial basis, for a single year, after which they must train a teacher and join VHS
as a fully participating school. The VHS has developed two graduate-level online professional development
courses for teachers of participating high schools: a 26-week Teachers Learning Conference (TLC) course which
trains teachers to develop and teach a NetCourse for VHS and a 15-week Netcourse Instructional Methodologies
(NIM) which trains teachers to teach an existing online VHS course.
The TLC is designed to train teachers to become online course instructors and course developers whereas the
NIM is designed to prepare classroom teachers to become online course instructors only. The TLC provides
instruction on the pedagogy and methodology that each teacher will need to develop an effective NetCourse to
be offered to the VHS students. A facilitator, a veteran VHS teacher, is assigned to each TLC participant to
ensure that they have the correct resources to achieve training objectives. The focus the NIM is on content and
curriculum, as well as good online course delivery. Experienced facilitators are assigned to help NIM
participants access the correct resources and monitor each participant's progress.
Both of the teacher training courses at VHS use the Internet as the main delivery technology and focus on ICT-
pedagogy integration in an online learning environment. Support given by facilitators of these courses is known
as an important factor which helps teachers have positive experiences with technology and integrate technology
into their own teaching (Freeman, 1997).
Another case of adopting ICT as the core delivery means of teacher training can be found in the LearnLink
project (http://www.aed.org/learnlink) supported by USAID and AED. The project has implemented computer-
mediated professional development programs to improve training and support services for teachers in several
developing countries (Fontaine, 2000; Collis & Jung, 2003). For example, in Guatemala, the project includes the
development of culturally appropriate Mayan language instructional materials, and improvement of teacher’s
professional skills in Mayan languages. Necessary equipment and multimedia computer labs have been installed
in several teacher training schools in the Quiche region and instructional materials for bilingual teacher
preparation, including an interactive multimedia system on CD-ROM to train teachers in oral and written
languages have been developed. In Morocco and Namibia, the Computer Assisted Teacher Training project has
started to develop computer-assisted teacher training courses and construct communications network to facilitate
interaction among teacher trainees, teacher trainers, and inspectors. Moreover, collaboration and information
sharing among peers across the provinces have been emphasized. Similarly, the Connectivity for Educator
Development project in Uganda has been designed to improve professional development for primary school
teachers, with a focus on multimedia-assisted teacher training and digital library resources. The US-Brazil
Learning Technologies Network is an Internet-based learning environment and clearinghouse on the role of ICT
in education and promoting interactive collaboration between teachers in the two countries.
The LearnLink project is still under implementation. Some of the expected outcomes include: increased
collaboration and interactions among educators in each country or among countries, institutionalization of
97
support for learning technology in each country, greater ICT access for teachers and students, ICT-based
curriculum reform, and enhancement of pedagogy.
An Internet-based online teacher training is recently introduced and has been found to provide a flexible and
interactive training environment for teachers (Jung, 2003). However, costs related to the online training cannot
be ignored in most parts of the world and effective online training pedagogies for ICT teacher training have yet
to be explored.
ICT used to facilitate professional development and networking

Whereas the use of ICT as core technology for delivering teacher training can be found in limited contexts, there
are many examples of ICT, particularly Internet and Web-based communication technologies, being used to
support teachers' on-going professional development and networking. Many countries have developed a
website or websites to provide online resources for teachers and facilitate teachers’ networking based on the
assumption that professional development should be an integral part of daily practice for all teachers and the use
of the Internet would enhance continuous professional development activities of teachers, connecting teachers to
larger teaching communities and allowing for interaction with expert groups. Specific examples are discussed
below.
The UK Virtual Teacher Centre (http://vtc.ngfl.gov.uk) website provides a "Career Development" area which
provides a variety of learning and teaching resources and links to support teachers’ continuing professional
development. Under "Support Providers", for example, teachers can find a range of resources for professional
development, such as the ICT Support Network Directory which provides easy access to ICT provision and
training. Teachers also find a link to the New Opportunities Fund (NOF), which is currently providing ICT
training for teachers and librarians. “International Professional Development” helps teachers learn from and
contribute to educational ideas and best practice throughout the world. TeacherNet UK
(http://www.teachernetuk.org.uk), an independent professional association for teachers, also supports teachers’
professional development and national and international teacher networking.
The Korea’s EduNet (http://www.edunet4u.net/) is an integrated educational internet services for K-12 students
and teachers managed by the Korea Education and Research Information Services. Through the EduNet, teachers
can search the materials according to training institution, content, instructor, year of publication and type of
training, and download them for self-training. These online materials can be also used for individual study in
conjunction with face-to-face courses, or as learning resources for online teacher training courses offered by
educational institutions.
Similar to the cases above, the US Teachers Network (http://www.teachnet.org), a nationwide, educational non-
profit organization, identifies and connects innovative teachers exemplifying professionalism and creativity
within public school systems. This network promotes interactive collaboration among teachers and educators to
improve teaching and student achievement, provides resources for designing their own professional
development, disseminates the work of outstanding classroom teachers, and attempts to provide teachers with the
knowledge and skills needed for good teachers. At the international level, the World Bank’s World Links for
Development (WorLD) (http://www.worldbank.org/worldlinks/english/index.html) program provides Internet
connectivity and training for teachers, teacher trainers and students in developing countries in the use of ICT and
other technologies in education. WorLD then links students and teachers in secondary schools in developing
countries with schools in industrialized countries for collaborative learning via the Internet.
Other examples include: SchoolNet SA (http://www.school.za), a South African organization providing supports
to educators and learners who wish to use ICT in education; Singapore’ Clearinghouse
(http://www1.moe.edu.sg/iteducation/resources/welcome.htm), a website created by Ministry of Education to
provide ICT resources and internet educational resources including lesson plans for various content areas for
teachers; Swedish Schoolnet (http://www.skolutveckling.se/skolnet/english/index.html), a website to stimulate
the use of ICT in schools; and European Schoolnet (http://www.eun.org/eun.org2/eun/en/index.html), the
European framework for the co-operation between the European Ministries of Education on ICT use in
education.
One of the best ways to develop teachers’ ICT skills and promote ICT-pedagogy integration in their teaching is
the provision of ICT-based training environments where on-demand access to materials, peers, and networks of
experts where expertise and advices can be obtained and active discussion can take place in relation to
98
technology or pedagogy. In this regard, the approach of using ICT to support teachers’ on-going professional
development and networking can be very effective as long as organized support is provided (Pacey, 1999).
Discussions and Conclusions

This analysis of approaches in ICT teacher training indicates that there are possibilities and challenges in
adopting ICT in teacher training and professional development. Some possibilities are discussed below.
Overall, governments and teacher training institutions seem to recognize the importance of integrating ICT in
education and teacher training. In many cases, the national vision for ICT use in education has been integrated
into teacher training. For example, Singapore’s teacher training institute has successfully integrated the national
vision toward ICT use in education into its ICT plan. Other countries such as UK, USA, South Africa, Sweden
and Korea have developed extensive online resources and encouraged active exchanges of new pedagogical
ideas to upgrade teachers’ knowledge and skills at the national or international level. In addition, the LearnLink
project in several developing countries is being implemented with close relationship with each country’s
government to integrate its activities into the nation’s educational vision and policies.
It is also observed in the analysis that a variety of ICT-integrated training environments have been created to
provide more effective ICT training. As indicated above, teachers tend to integrate ICT in their teaching if they
experience ICT skills as a learner (Collis & Jung, 2003). Teacher training approaches in this paper show that
many cases adopt ICT into their training process not just as content of the training but rather as an integrated
training environment and thus allow teachers to experience ICT-based pedagogies. The cases of VHS and
Captured Wisdom are those examples. One UK site has compiled cases of technology-integrated pedagogical
strategies for teachers (http://www.educ.cam.ac.uk/tips/reports.html) and made suggestions in incorporating ICT
into the curricula. More hands-on experiences that relate ICT to the achievement of wider pedagogical
objectives are suggested at the initial training level and at the advanced level, the provision of opportunities for
teachers to produce and disseminate ICT-based instructional materials is recommended.
Another possibility with the use of ICT in teacher training is that it connects teachers to a larger international
teaching community. Several cases analyzed above operate the Internet-based teachers’ learning community and
support teachers to interact with peer teachers as well as teachers in other countries. Moreover, they invite
experts to provide expertise to teachers through online forums or emails. Best practices in using ICT in teaching
and learning and successful pedagogies are now being shared among teachers scattered around the world.
While these possibilities are observed in ICT teacher training approaches, those experiences also impose
challenges to teachers, teacher training institutions, and nations. Some of the challenges are presented below.
First, teacher training approaches need to adopt cost-effective strategies. Most nations have limited resources for
teacher training and must make decisions based on cost-effectiveness. The teacher training experiences provide
several cost-saving strategies (Collis and Jung, 2003):
¾ Maximize use of computer facilities in training centers to lower user contact hour costs through efficient
scheduling. Outside training hours, open computer labs to the public for a small fee (as Uganda has planned
in the Connect-ED project).
¾ Standardize on hardware and software and negotiate best prices with vendors. Complementary peripheral
devices can mean savings in hardware costs and free, public-domain software lowers costs. Some vendors
include ICT skills training in the purchase price.
¾ Share Web-based resources and training materials with other training institutions.
Second, support and investment in teacher trainer training is important for the adoption of ICT for teacher
training. The experiences of NIE, VHS, and LearnLink indicate the importance of providing a variety of both
formal and informal teacher trainer training systems so that trainers could take advantage of the methods which
suit them best. Experience shows that to enlist staff support and involvement, it is useful to:
¾ Employ a variety of teacher trainer training methods, ranging from face-to-face workshops to online self-
study programs depending on training objectives and environments.
¾ Integrate informal support into the formal teacher trainer training system so that the less experienced teacher
trainers can obtain timely assistance.
¾ Plan to provide multiple incentives such as workload reduction, recognition and reward in faculty
evaluations, increased research allocations to encourage use of ICT in teaching, and compensation for those
providing educational or technological assistance to others.
99
Finally, national and international partnerships across public and private sectors need to be formed to share
resources, knowledge, and experiences in providing effective and efficient ICT teacher training. ICT teacher
training efforts made by organizations have shown training advantages of international collaborations and
benefits of using ICT for teacher training. One of such advantages of international collaboration is to bridge the
gap between ICT haves and have-nots. Governments or teacher training institutions seeking to promote national
and international partnerships should:
¾ Provide incentives for private and public participation and investment in ICT teacher training.
¾ Remove legal barriers – for example, classroom attendance requirement – to online training courses shared
by several countries or institutions.
¾ Incorporate a plan to lesson the digital divide that may exist in participating countries or training
institutions.
A well-designed teacher training program is essential to meet the demand of today’s teachers who want to learn
how to use ICT effectively for their teaching. This paper was an attempt to report popular approaches of using
ICT in teacher training programs and organized them into four categories. However, to provide proven
strategies to design effective ICT teacher training programs, we need empirical research in the use of ICT for
teacher training and professional development.
First, we need more studies comparing effectiveness and cost-effectiveness of different training approaches. A
few attempts have been made to investigate effectiveness or cost-effectiveness of online ICT teacher training
over traditional face-to-face ICT training based on empirical data (Jung & Rha, 2000; Jung, 2003). One of those
studies reports that ICT training, regardless the modes of the training, has contributed to increasing the quantity
of ICT use in teaching and the Internet-based online training tends to encourage teachers’ Internet use in
teaching (Jung, 2003). While these studies are useful in providing overall effectiveness or cost-effectiveness of
different training modes, not much research has been conducted to assess effectiveness or cost-effectiveness of
specific pedagogical approaches to ICT teacher training which have been discussed in this paper. We need
future studies that include the investigation of an application level of long-term effectiveness or cost-
effectiveness of different ICT teacher training approaches using more elaborated criteria.
We also need more empirical studies focusing on factors affecting learning process, satisfaction and achievement
in different teacher training approaches. Given the fast development of ICT, we can expect that ICT will bring
changes in forms of teacher training throughout the world. It is thus important for teacher trainers and policy
makers to understand the factors affecting effectiveness and cost-effectiveness of different approaches to ICT
use in teacher training so training strategies can be appropriately explored to make such changes viable to all.
Finally, more attention should be paid to specific roles of ICT in offering multimedia simulations of good
teaching practices, delivering individualized training courses, helping overcome teachers’ isolation, connecting
individual teachers to a larger teaching community on a continuous basis, and promoting teacher-to-teacher
collaboration. Intended outcomes as well as unintended results of using ICT for teacher professional
development need to be explored.
References
Bowes, J. (2003). The emerging repertoire demanded of teachers of the future: Surviving the transition,
retrieved, September 1, 2004, from http://crpit.com/confpapers/CRPITV23Bowes.pdf.
Carlson, S., & Gadio, C. T. (2002). Teacher professional development in the use of technology. In W.D. Haddad
& A. Draxler (Eds.), Technologies for education: potential, parameters, and prospects, Paris and Washington,
D.C.: UNESCO and AED, retrieved April 25, 2005 from
http://www.schoolnetafrica.net/fileadmin/resources/Teacher_Professional_Development_In_the_use_of_Techno
logy.pdf.
Collis, B., & Jung, I. S. (2003). Uses of information and communication technologies in teacher education. In B.
Robinson & C. Latchem (Eds.), Teacher education through open and distance learning, London:
RoutledgeFalmer, 171-192.
Fontaine, M. (2000). Teacher training with technology: Experience in five country programs. TechKnowLogia,
November/December, 69-71.
100
Freeman, M. (1997). Flexibility in access, interaction and assessment: the case for web-based teaching programs.
Australian Journal of Educational Technology, 13 (1), 23-39.
Haddad, W. D. (2003). Is instructional technology a must for learning? Techknowlogi.org, retrieved, September
23, 2004, from
http://www.techknowlogia.org/TKL_active_pages2/CurrentArticles/main.asp?IssueNumber=19&FileType=HT
ML&ArticleID=455.
Jung, I. S., & Rha, I. (2000). Effectiveness and cost-effectiveness of online education: A review of literature.
Education Technology, July-August, 57-60.
Jung, I. S. (2001). Singapore's approach to preparing new teachers to use technology in the classroom. In J.
Capper (Ed.), Case studies of innovations in teacher training and technology. Washington, DC: The World
Bank, retrieved, September 20, 2004, from http://www.the3tconnection.org/SingaporePrintingVersion.pdf/.
Jung, I. S. (2003). A comparative study on the cost-effectiveness of three approaches to ICT teacher training.
Journal of Korean Association of Educational Information and Broadcasting, 9 (2). 39-70.
Pacey, L. (1999). Integration of information and communication technologies (ICTs) through teacher
professional development: comparative analysis of issues and trends in seven APEC economies, Canada: Judy
Roberts & Associates Inc.
Perraton, H., Robinson, B., & Creed, C. (2001). Teacher education through distance learning: technology,
curriculum, evaluation, cost, Paris: UNESCO.
Robinson, B., & Latchem, C. (2003). Teacher education: challenges and change. In B. Robinson, & C. Latchem
(Eds.), Teacher education through open and distance learning, London: RoutledgeFalmer, 1-27.
101
Zepp, R. A. (2005). Teachers’ Perceptions on the Roles on Educational Technology. Educational Technology & Society, 8
(2), 102-106.
Teachers’ Perceptions on the Roles on Educational Technology

Raymond A. Zepp
Department of Educational Sciences
Eastern Mediterranean University
Gazimagusa, TRNC, Via Mersin 10, Turkey
raymond.zepp@emu.edu.tr
zepp@rocketmail.com
Tel: +90 533 875 4998
Fax: +90 392 630 4038
ABSTRACT
Entering graduate students were asked whether technology would soon eliminate the need for teachers.
Responses (all negative) were clearly separable into ‘modernist’ responses, which indicated that the role of
education was the transmission of information and skills, but that teachers can do a better job of this; and
‘postmodernist’ responses, which indicated other roles for educators. Many incoming students displayed a
lack of insight into the possibilities for technology in education. The study concludes that teachers should
be made aware of the various goals of education, and which goals can best be achieved using technology. A
second recommendation is that potential educators be made more aware of the various roles of technology
in education, and how teacher-technology synergies can be tapped.
Keywords
Technology, Postmodernism, Teacher roles, Teacher training
Introduction: ‘Death of the Professor’

Technology can now perform many of the traditional roles of teachers. In recent years, some ‘postmodernist’
educators have lamented the possible replacement of teachers by computers and other technological devices.
Notable among these is Jean-Francois Lyotard (1984), who has written about the ‘death of the professor’: as a
result of the emphasis placed by society on the efficiency of the transfer of information or skills. If computers
can transfer more information to more students in a shorter time span and at lower cost, then the efficiency of
technology will force teachers out of business. The revolt against ‘modernism’ bemoans the relegation of
education to an input-output model subject to economic cost-benefit analysis.
Of course, wise teachers have always understood that they serve as role models, motivators, and coaches in
critical and creative thinking, in addition to being organizers and dispensers of information. Lyotard and others
stress, however, that, in real-world practice, the twentieth century single-minded preoccupation with economic
growth and information transfer has led teachers away from these other goals of education. In many instances,
technology can outperform the teacher in terms of sheer efficiency – more information can be transmitted to
more students in a shorter time span by technology than by teachers.
Influential authors like Rorty (1979) argue that even in our information and economic skills-oriented age, the
teacher should have many roles in addition to relaying information and skills. In Rorty’s view, teachers need “…
to keep space open for the sense of wonder … to make students thrill to the same things they themselves thrill.”
(p. 42). He distinguishes between systematic philosophers and edifying philosophers:
Great systematic philosophers are constructive and offer arguments. Great edifying philosophers
are reactive and offer satires, parodies, aphorisms. (Rorty, p. 369.)
The edifying teacher, for example, inculcates in students an appreciation for the beauty of a
mathematical theory better than the computer can.
Nuyen (1995) describes Rorty’s argument concisely: “The role of education is not to pass on the truth but to
edify… What we need is not a battery of computer terminals, but a whole range of Professors who not only
purvey truth, but who excite student’s imagination.”
Besides edification, according to many educational philosophers, there are other human values inherent in the
educational process. Bertrand Russell (1994), in his famous treatise On Education devotes more than the first
half of the book to discussions of the building of ‘character’, and only later does he treat the ‘intellect’, while
102
deprecating the mere transfer of information. Broudy (1962) reflects this view, although he eschews words like
‘character’ and uses instead ‘attitude formation’ and ‘value climate’. “In attitude formation, human models and
the value climate of the school as a whole are important, perhaps crucially so. Clearly the machine cannot serve
as such a model.” (p. 152.)
Thus, the postmodernist revolt emphasizes a re-focus on these non-informational roles of teachers. Teachers are
to serve as role models and transmitters of values. In addition to being a source of knowledge, a good explainer,
or an expert motivator, the new teacher is expected to be a human role model, in order to guide the students’
characters in the right direction.
Theory into Practice: Students at Eastern Mediterranean University

What do today’s educators feel about the arguments expressed above? If teachers feel that transmission of
information is the only goal of education, then perhaps most of their work can and should be done by
technology. If, however, they disagree with the information-only view, then they can go on to define and focus
on additional roles for teachers.
Thirty-eight students applying for admission to postgraduate programs in Education at Eastern Mediterranean
University were asked the following question as part of their admissions examination:
“In the near future, it will be possible for technology (including books, videos, computers, etc.) to
transmit any information or skill more effectively than teachers. Therefore there will be no need
for teachers.” Do you agree or disagree with this statement? Discuss.
The question was deliberately worded in order that the respondent might focus on either the premise or the
conclusion. One can argue with the premise that it will be possible for technology to transmit information more
effectively, or on the other hand, even if the premise is true, whether the conclusion follows that there will be no
need for teachers.
Now these 38 educators or future educators were not about to agree with a statement implying that they will no
longer be needed. However, disagreement can follow two lines, which might be termed the modernist line and
the postmodernist line.
The modernist line buys into the present-day view of education solely as transmitting information and skills, and
that its success is measured by its efficiency in such transmission. A student could disagree with the premise of
the statement in the examination along modernist lines by stating that in the long run, teachers are better at
transmitting information.
This modernist disagreeing argument runs that teachers are better motivators than machines. They can better
interact with students than can machines. They can better intuit students’ learning difficulties, especially when
these involve emotional problems. So even if these arguments run along humanistic lines, they still see
interactive teacher interventions as efficient means to the modernist end of transmitting information.
The postmodernist line, in accordance with Rorty, emphasizes education as much more than the transmission of
information. A postmodernist disagreement with the conclusion to the examination question might focus, as does
Rorty, on the non-informational role of the teacher, more as a guidance counselor or a societal role model.
Attention to students’ personal growth is seen not as a means to information acquisition, but as an end in itself.
In most cases, the student responses were very clear as to which line of reasoning the student followed. The
following are excerpts from typical ‘modernist’ responses:
Computers cannot give the information more clear than teachers. Teachers give the information
more clear than books.
In order to transmit information or skills effectively, we need people who can …..
Computers cannot express a motivational feedback or reinforcement.
Teachers know how to teach specific information to a learner.
103
Nothing can transmit any information or skill more effectively than teachers, who can organize,
motivate, show warmth, humour, and flexibility.
Notice that while such responses may refer to ‘human qualities’ such as motivation or care, the emphasis is on
using those qualities as means to the end of acquiring information. Thus, the ‘modernist’ response may
emphasize humanistic methods, but the goal is still the purely cognitive acquisition of information and skills.
On the other hand, clear examples of postmodernist responses were the following:
…to guide the student socially and psychologically, and to develop interpersonal relationships.
What about the soul? We can transmit any information or skill by means of technology but
unfortunately it is not enough .. to touch the human magic.
Children also need love and care, and to acquire discipline.
…security, belonging, self-esteem, emotional as well as academic problems.
In these latter examples, teachers are seen as ministering to the personal needs of children as ends in themselves.
Four of the 38 responses were judged as ambiguous and so were rejected from the tabulations below. But even
those ambiguous responses provided some insights. One important line of thought attempted to cut between the
modernist and postmodernist views. This was the reference to multiple intelligences and learning styles. Three
responses argued that all children were different, but it was not clear whether the responses meant that children
require different interventions in order to achieve the (modernist) goal of acquiring information, or that children
have different needs.
Other expressions that could be interpreted as either modernist or postmodernist include the words ‘guide’,
‘human touch’, and ‘interaction’. In such cases it was usually clear from the context which line was taken. For
example, teachers may guide students to acquire information, or they may guide them to solve family problems.
A human touch may be used to motivate a student in acquiring information, or it may be used to assist the child
in his/her social adjustment. Interpersonal interaction may be a means towards acquiring information, but it may
also be an end in itself. For example, one response referred to a girl who is shy in class, and the teacher may be
able to help overcome that shyness with other students.
Tabulations of the responses into Modernist or Postmodernist lines of reasoning, are shown in Table 1 below.
Table 1. Tabulation of Responses by Student Level

Modernist Postmodernist Ambiguous
Master’s applicants 13 10 3
PhD applicants 5 6 1
All applicants 18 16 4
It appears that the master’s applicants leaned more heavily towards the modernist line of argument, while the
Ph.D. applicants leaned towards the postmodernist. A chi-square analysis of the 34 unambiguous responses,
however, yielded a chi-square of only 0.37 and revealed no statistically significant differences between the two
groups.
Another question arising in this context is whether applicants with previous degrees in education tend to respond
differently from applicants from other disciplines. Students with a B.A. or M.A. in education might have been
exposed to the postmodernist view, and thus might be expected to fall into the Postmodernist category more
often than their non-education counterparts. Table 2 below shows the breakdown, combining both master’s and
doctoral applicants:
Table 2. Tabulation of Responses by Education Background

Modernist Postmodernist Ambiguous
Previous Education Degree 13 10 2
No Education Degree 5 6 2
Total 18 16 4
104
Once again, a chi-square test performed on the unambiguous responses yielded a chi-square of only 0.37 and
revealed no significant differences between the previous-education and the no-education groups, with total
modernist and postmodernist responses nearly equal in number. (The fact that the split was the same as the
Master’s/PhD split reported above is probably just coincidental.) Thus there is no evidence that applicants with
education degrees were more or less likely to give postmodernist responses.
Thus, the students separated themselves into two clear camps, with 18 of the 34 unambiguous responses taking
the modernist line (53%) and the other 16 (47%) taking the postmodernist line.
Discussion and Conclusion

It was surprising that most of the ‘modernist’ responses incorrectly claimed that teachers have certain traits but
technology does not. Six responses stated that only teachers can convey humor. But a computer can tell a joke as
well, even though it does not have a sense of humour per se. It would be one thing to state that teachers can use
humor to better advantage than computers, but it is quite wrong to state that computers are incapable of using
humor. Similarly, seven responses stated that only teachers can interact with students. But interactive computer
programs have come a long way in the past twenty years.
Several responses stated that only teachers can answer questions or give detailed explanations. But in many
instances, the writers of educational software are well aware of the various questions posed by students, and
incorporate FAQ sections into their interactive software. And it is quite possible that a well-thought-out
explanation offered by a computer can be much clearer than a spontaneous explanation by a teacher. Most
students mentioned motivation as unique to teachers, but a good interactive program can give immediate
feedback using phrases like, “That’s wonderful!”, or “Good job, Kathy!” A yellow smiley face or a frown can
convey nearly the same approval or disapproval as can a teacher.
These mistaken judgments came in many instances from applicants who already had a bachelor’s or master’s
degree in education. Somewhere along the line, their training should have included an assessment of what
technology can or cannot do, such as interacting with students, answering questions, and offering clear
explanations. A major recommendation of this study is that teacher training programs include discussions about
the possibilities and limitations of technology, such as the discussion in the above paragraphs.
The responses came from students in Northern Cyprus. Many of them have taken education courses in student-
centered education, philosophy of education, the role of education, et al, and still appear to view education solely
as the transfer of information and skills. This bias could of course be cultural. It might be interesting to compare
a larger sample size in Cypriot culture (a mixture of Western thought and Islamic thought) with a more
Westernized audience, as well as with a more traditional Islamic audience.
Cypriot culture lies at the crossroads of East and West. The sample used in this study, while all nominally
Muslims, cannot be classified as devoutly religious. Students from a more strict Muslim culture might be
expected to emphasize the role of the teacher as spiritual guide to a greater extent than Western teachers.
(Ozdemir and Frank, 2000, p. 16). Indeed, Islam’s whole concept of knowledge and the role of education is quite
different from the secular view. In Islam, knowledge is to a large extent ‘revealed’ knowledge as disseminated
by the Prophet, and the role of education is to instill this body of unquestioned revelation to the students. For
example, the skill of critical thinking might have lower value in Islamic education than in Western education.
It is perhaps significant that this Cypriot sample split almost evenly between the two camps. Perhaps a more
Western audience would fall more into the ‘modernist’ camp, while a more devoutly Islamic audience would fall
more into the ‘postmodernist’ camp.
Despite the relatively small sample size, it was clear from the responses that both the modernist and the
postmodernist view of education and the role of the teacher are prevalent among both present and future
educators at Eastern Mediterranean University, and that these views are distributed among doctoral as well as
master’s level students, as well as among former educators and those with little or no teaching experience.
However, none of the respondents mentioned both viewpoints or drew any distinction between them. This is
evidence that the respondents, many of whom have bachelor’s and master’s degrees in education, have not really
thought deeply about the various goals of education. A major recommendation of this study is therefore that
teacher training programs stress these various goals and discuss which goals are best amenable to technological
105
interventions. A second major question addressed in any education program should be, “Technology – for
what?”
Many discussions of the teacher-technology interface treat the matter as an exclusive ‘either-or’ question, that is,
which is better, teachers or technology. In fact, the more appropriate question should be, “how can teachers and
technology interact to the optimal advantage?” For example, teachers, videos, and interactive computer
programs might join together to improve a child’s self-esteem, or to promote interpersonal social skills. There
are many synergies to be explored which exploit the advantages of the information processing of technology
along with the more human aspects of the teacher. Awareness of these aspects should also be included in teacher
training courses.
Are potential teachers in today’s teacher training programs being well-versed in how technology relates to
educational philosophy? A survey of teacher training programs might shed light on how institutions are
preparing teachers in the uses of technology.
Further research might also investigate whether followers of the postmodernist argument (accepting both the
acquisition of information and the pursuit of human objectives) value one teacher role more than the other. A
traditional Muslim might strongly prefer a more value-based education rather than the acquisition of
economically useful skills. The present study serves only to illustrate that at least two lines of thought exist and
are clearly identifiable; further research may identify other lines of thought and may address questions of
differences in attitudes among various social, religious, or national groups.
References
Broudy, H. S. (1962). Teaching machines: Threats and Promise, Educational Theory, 12, 151-156.
Broudy, H. S., & Palmer, J. (1965). Examples of Teaching Methods, Chicago: Rand McNally.
Lyotard, J.-F. (1984). The Postmodern Condition: A Report on Knowledge (trans. Benningtion, G. & Massumi,
B.), Minneapolis: University of Minnesota Press.
Nuyen, A. T. (1992). Lyotard on the Death of the Professor, Educational Theory, 42 (1), 25-38.
Nuyen, A. T. (1995). Lyotard and Rorty on the Role of the Professor. In Peters, M. (Ed.), Education and the
Postmodern Condition, Westport, Connecticut & London: Bergin and Garvey, 41-57.
Peters, M. (Ed.) (1995). Introduction to Education and the Postmodern Condition, Westport, Connecticut &
London: Bergin & Garvey.
Rorty, R. (1979). Philosophy and the Mirror of Nature, Princeton, N.J.: Princeton University Press.
Rorty, R. (1990). The Dangers of Over-Philosophication. Educational Theory, 40 (1), 41-44.
Russell, B. (1994). On Education, London & New York: Routledge.
106
Yildirim, Z. (2005). Hypermedia as a Cognitive Tool: Student Teachers' Experiences in Learning by Doing. Educational
Hypermedia as a Cognitive Tool: Student Teachers’ Experiences in

Learning by Doing
Zahide Yildirim
Middle East Technical University, Faculty of Education
Department of Computer Education and Instructional Technology
Inonu Bulvari, 06531 Ankara, Turkey
Tel: +90 312 210 3679
Fax: +90 312 210 1006
zahidey@metu.edu.tr
ABSTRACT
This study investigates student teachers’ opinions about learning “instructional technology and material
preparation” subjects in a hypermedia-based constructivist learning context. A qualitative case study design
was employed. The students of one classroom were the focus of an in-depth investigation by means of
interviews, which were designed to elicit these students’ perceptions concerning the use of hypermedia as a
cognitive tool in the learning process. The study sample consisted of twenty-eight second-year students who
enrolled in the Instructional Technology and Material Preparation Course in the Fall of 2001 at Middle East
Technical University in Ankara, Turkey. The study lasted fourteen weeks (two hours per week). In the
group, high, average, and low achievers (based on their GPAs), and both males and females were
represented. An interview schedule consisting of sixteen questions was designed to obtain the students’
opinions about developing hypermedia as a cognitive tool to learn the subject. The data collected through
the interviews were subjected to content analysis. The results indicate that the majority of the students
preferred the hypermedia-based constructivist learning environment in the course to a traditional learning
environment. The students thought that the hypermedia-based constructivist learning environment helped
them learn the subject matter more effectively.
Keywords
Hypermedia, Cognitive tools, Constructivism, Project-based learning
Introduction
Though many educators in the 20th century advocated the learner-centered approach in education, currently, the
transmission of knowledge rather than the construction of knowledge is favored in educational settings. With
technological advances, it was thought that computers would provide solutions to many educational problems.
However, as in the case of educational systems, computers are used merely to transmit knowledge most of the
time, or simply to perform traditional instruction more efficiently and effectively (Pea, 1985). Perkins (1986)
once said, “How we think of knowledge could influence considerably how we go about teaching and learning”
(p.2). If knowledge is considered to be information, as Perkins also argued, it is isolated from its context
resulting rote learning. But if, on the other hand, knowledge is seen as a sort of design, then knowledge is related
to understanding, and to critical and creative thinking. It can then be structured meaningfully, and adapted or
revised for different purposes. The value of design experience, as Carver et al. (1992) emphasized, comes from
the opportunity provided to students to develop a variety of complex thinking skills.
One way of promoting knowledge as design is to use computers as cognitive tools. With cognitive tools,
information is not presented in the educational material to be learned. Students’ roles are those of designers and
problem-solvers. The students use the computers as cognitive tools to access, analyze, interpret, and organize
their personal knowledge. This approach enables students to think deeply and to perform their learning tasks
effectively (Pea, 1985; Kozma, 1992; Mayes, 1992; Reeves, 1999; Jonassen, 2000).
To be able to use technology as cognitive tool, it is important to provide an appropriate learning context for the
students. Jonassen and Reeves (1996) have stated that learning with technology requires a constructivist learning
context. The constructivist view emphasizes that learning occurs when it is supported by different perspectives
within meaningful contexts; social interactions are also critical in this learning process (Oliver & Herrington,
2003). Duffy and Cunningham (1996) highlighted some pedagogical goals in the constructivist learning context.
These include the provision of experience in the knowledge construction process, the provision of realistic and
relevant contexts, the encouragement of ownership, embedded learning in a social context, the use of multiple
modes of representation, the appreciation of multiple perspectives, and the development of self-awareness in the
knowledge construction process (Oliver and Herrington, 2003).
107
Jonassen, Mayes, and McAlessi (1993) categorized the learning process as three learning phases: introductory,
advanced, and expert. They argued that when the student lacks complicated schemata and has limited
transferable prior knowledge in the subject area to be learned, introductory learning occurs. In the advanced
learning phase, students are more mature and are able solve more complex and context-specific problems; they
also acquire more advanced knowledge. In the expert learning phase, students have complex and meaningfully
linked knowledge structures relating to the subject. Jonassen et al. (1993) state that the introductory phase is
better served by classical instructional design, while constructivist learning environments are generally more
viable for the advanced knowledge acquisition phase. Universities and secondary education institutions are the
appropriate places in which to encourage advanced knowledge acquisition in a constructivist environment.
Duffy and Cunningham (1996) highlighted another characteristic of constructivist learning. They emphasized the
importance of language and dialogue in human culture and cognition. In order to facilitate dialogue among
students, group work should be practiced to share different viewpoints, and to challenge and develop alternative
perspectives. So, rather than sticking to only one way of thinking, in a constructivist learning context, the student
should be able to benefit from other individuals’ thinking/ideas by means of engaging in group work (Steketee,
2002).
In a constructivist environment, learning is arranged around a project. The two essential components of an
appropriate project are the product and the problem. The product component is the starting point, which allows
the students to learn the essential and critical points of the topic while developing the end product (Liu, 2003).
The problem component is related to conducting investigations, searching for solutions to problems, solving
problems, and making decisions in order to produce the product. In this process, the students come to understand
the main principles and concepts related to the problem, and they should feel a great sense of achievement
(Blumenfeld et al., 1991; Liu, 2003). To benefit from a project-based learning environment, the students’
engagement over an extended period of time is required, and they must concentrate on complex, integrated
modules of long-term instruction (Blumenfeld et al., 1991; Krajcik et al., 1994, cited in Jonassen, 1998)
Blumenfeld et al. (1991) indicated that “technology facilitated, project-based learning has great potential to
enhance students’ motivation, and supports information gathering and presentation” (cited in Liu, 2003, p.25).
Having students design hypermedia for the subject to be learned is one type of project-based learning, which
focuses on the development of design skills (Lui, 2003). In this approach, hypermedia supports knowledge
construction, exploration, learning by doing, learning by conversing, and learning by reflection (Jonassen, 2000;
Jonassen & Reeves, 1996; Lehrer, 1993). Designing hypermedia as an author promotes not only deep learning of
the subject matter, but also social skills like collaboration, project management, decision-making, research,
organization and presentation, and reflection (Carver et al., 1992). Kafai, Ching, and Marshall (1997) stated that
“the production of technological artifacts as a promising context for students to learn the coordination of
multiple demands needed to design and implement such project” (p. 117). They commented that “creating
multimedia appears to be a good context for students to learn about collaboration and project management” (p.
118).
Today, we experience various problems when attempting to integrate technology into teaching and the learning
process. Technology is used either as a management tool to help teachers and administrators in their
management tasks, or it takes the form of educational software, in which content is encoded. Technology has
been integrated into the educational system to a large extent; however, all of the potentials embodied in the goals
“learning with technology” and “using technology as a cognitive tool” have yet to be fully realized. It is clear
that the role of teachers will be integral for future development along these lines.
The influence of a hypermedia learning environment on learning processes has mostly been investigated under
the condition where students learn from the hypermedia. In addition, though there are research studies that
investigate the implementation of various cognitive tools with a wide range of student groups, student teachers’
use of hypermedia as a cognitive tool within a constructivist learning context has not been studied much. How
do student teachers perceive hypermedia as a cognitive tool? How do they interact within such a learning
environment? In what ways can they effectively learn with hypermedia, when it is used as a cognitive tool?
Questions like these need to be explored in order to understand what processes the student teachers go through as
they construct their own knowledge through the use of hypermedia. Therefore the purpose of this study is to
explore how student teachers perceive learning through the use of a hypermedia cognitive tool in a constructivist
learning environment. The specific focus is student teachers’ opinions about using hypermedia as a cognitive
tool to learn the course content of “Instructional Technology and Material Preparation.”
108
Methods
Overall Design
A qualitative case study design was used in this study. The learning experiences of students in a teacher training
course where hypermedia was used as cognitive tool were examined in depth through interviews. The purpose
was to understand student teachers’ perceptions of the use of hypermedia as a cognitive tool in their learning
process. The study included twenty-eight second-year Computer Education and Instructional Technology
students at Middle East Technical University in Ankara, Turkey. The students possessed basic knowledge
concerning the design of hypermedia along with associated computer applications. In the group, high, average,
and low achievers (based on GPAs) were all represented. Before the study was initiated, the students were
informed that the course would proceed within a project-based constructivist learning context, and that
hypermedia would be used as a cognitive tool. Throughout the semester, the students were required to work in
groups to develop hypermedia relating to the course content. The procedures that the student teachers followed
during the course are presented in Figure 1.
Figure 1. The procedures student designers followed
At the beginning of the semester, the students formed their project groups, which consisted of two to five
students each. There were eight groups in the study. Throughout the semester, the instructor avoided presenting
the course content to the students didactically. Instead, she established a constructivist learning context in which
she facilitated, coached, and guided the learning experiences of the students. In order to help the student teachers
to use their time more effectively and efficiently, the content was divided into sub-components. These sub-
components were scheduled for eight weeks, so that the students could spend sufficient time investigating and
forming the content of the hypermedia to be developed. To establish the content of the hypermedia, the students
investigated the sub-components of the topic, and wrote weekly reflective journals in which they reported the
results of their investigation. Each week, they received feedback from the instructor on their journals, and
participated in group discussions and learning activities held in the class (two hours per week). Two computer
laboratories in the department served as (two hours per week for fourteen weeks) hypermedia development
facilities for the students. While developing their hypermedia learning environment templates, each group
presented their work in class and received feedback from the instructor and their classmates. By the end of the
semester, the students finished their hypermedia learning environments. Examples from the students’ projects are
presented in Figures 2 and 3.
109
Figure 2. Example 1 from students’ projects
Figure 3. Example 2 from students’ projects
Participants
The study included a total of twenty-eight second-year Computer Education and Instructional Technology
students (twenty-four male and four female), who enrolled in the Instructional Technology and Material
Preparation Course in the Fall of 2001-2002, at Middle East Technical University in Ankara, Turkey. This
course lasted for fourteen weeks. The majority of the students in this department (and also in the study)
graduated from computer departments in vocational or technical high schools. Since most of the students
graduated from computer departments in vocational or technical high schools, they already possessed basic
computer knowledge and skills in a variety of computer applications, and some hypermedia design experience
prior to this case study. For this reason, they did not require any orientation to designing hypermedia at the
beginning of the study.
Data Collection
An interview schedule was developed to collect qualitative data on the students’ opinions and perceptions of the
use of hypermedia as a cognitive tool in a constructivist learning environment. The schedule included sixteen
questions on the following areas: “how the constructivist learning environment contributes to both the students’
learning of instructional technology and to their material preparation of the course content; advantages and
limitations of the use of hypermedia as a cognitive tool in a constructivist learning context, in comparison to
traditional instruction; perceptions of the procedures followed during the hypermedia development process;
opinions about the role of the instructor in a constructivist learning context; suggestions concerning the effective
application of hypermedia as a cognitive tool in a constructivist learning environment; and finally, perceptions
and suggestions concerning the assessment of student performance in such a learning environment.”
110
All of the students who took part in the study were interviewed at the end of the semester in their project groups.
Thus, a total of eight group interviews were carried out. There were several reasons for preferring group
interviews to individual interviews. First, throughout the study, the students worked in their project groups and
experienced this process together. Second, group interviews can be both effective and efficient as a qualitative
data collection technique, since they allow the interviewees to consider what others in the group say, and then
may build upon those shared opinions. Third, this method provides quality control in data collection, since
multiple perspectives on the same issue may provide a basis for validation. Fourth, the group dynamics typically
contribute to focusing on the most important topics and issues relevant to the study. Finally, group interviews
can be used to identify a program's strengths, weaknesses, and needed improvements (Patton, 1987). For these
reasons, I chose to carry out group interviews, which produced a rich data base to answer the research questions
in this study.
Before each interview took place, the students were informed of the purpose of the interview. Each interview
lasted about one and a half hours, and all of the interviews were tape-recorded with the permission of the
participants.
Data Analysis
The data collected from the interviews were subjected to content analysis. As Miles and Huberman (1994) have
stated, content analysis involves searching for meaningful phenomena in the data and assigning descriptive codes
to describe the data as a meaningful whole. For this purpose, the interview data were first transcribed into eight
interview files. Second, the transcribed data were coded using a predetermined set of terms and concepts based
on the literature (Duffy and Cunningham, 1996; Jonassen, 2000; Lehrer, R., 1993; and Pea, 1985). Coding was
conducted to classify and organize the data. Third, themes were identified in order to arrange the codes in a
meaningful and consistent way, and then the codes were grouped under these themes (such as the constructivist
learning environment, writing weekly reflective journals, hypermedia development, group work, learning
activities, and assessment). Finally, the coded data were presented and described under these main themes, and
then the interpretation and discussion of the results was offered.
Results
The analysis of the interviews produced certain themes in relation to the use of hypermedia as a cognitive tool.
The results of this study are presented under these themes, along with problems/difficulties that the students
faced in relation to these themes.
Overall assessment of the constructivist learning environment
Most of the students (N=25) mentioned that they preferred using hypermedia as a cognitive tool in a
constructivist learning environment to traditional classroom instruction. They stated that in traditional
instruction, while the instructor was presenting the content, the students could not focus their attention on the
subject for a long time. Only three students preferred traditional settings to constructivist settings. One of them
reported (in contrast to his teammates), “I learn social subjects better in a traditional setting, and I prefer the
instructor to present the content. However, I learn technical subjects by myself, and I do not need group work.”
Another said, “I can learn better by listening to the instructor.” The third student commented, “It was a new
method, and I had difficulty getting used to it.”
Those who were in favor of constructivist learning indicated that the constructivist learning environment led
them to be an active participant in the learning process, to learn and retain the course content more effectively,
and to pay more attention to their project. One student stated, “It is not possible to notice if students are listening
to the instructor or day-dreaming in traditional instruction. But in this environment, we had tasks to finish by due
dates, so we had to work.” One of the groups suggested, “It would be better if a combination of the two
strategies, traditional and constructivist, were used together.” Two groups commented that a fourteen week
period was too long, and that the project was too large. One student from these groups said, “I learned the subject
better with this approach, but dealing with the project’s long duration was tiring and boring.” Another student
said, “the content covered in the project was too much; it would be better to keep the content less, or divide the
subject into sub-subjects, and after finishing the first, it would be better to move forward.” They suggested that it
would be more helpful if such a large project consisted of several smaller projects or modules.
111
One group reported that the constructivist approach conducted in this course was “a very good approach;
however, it cannot be applied in all courses.” This group thought that they would have difficulty applying this
approach in all courses, because that would require far more work by the students than they could possibly do.
This would place too much pressure on the students and would result in poor performance at least in some
courses. One student from the group explained that “this course took too much of my time. If the other courses
required that much time and effort, I could not handle them all together.”
For the learning purpose, most of the students (N=25) argued that this method was better than the traditional
method. Since they learned actively, they felt more involved, and they retained knowledge better, even though
the course created “a huge workload” for them. In the traditional method, they studied only for the exams; then
they forgot what they learned. One student said, “I, and my friends too, study the course material just before the
exam to get a good grade. One week or two days—it depends on the subject. After the exam, I forget most of
what I have studied. But in this approach, I had to focus on the subject several times to present it better. It is
better for learning.”
Writing weekly journals to form the content of the hypermedia
To determine the content of the hypermedia that they were to develop, the students searched for information.
Based on their search results, as groups they wrote weekly journals. They received feedback on their journals
from the instructor and made revisions in their hypermedia content.
The majority of the students (N=23) stated that, even though it was their first experience, writing weekly
reflective journals and receiving weekly feedback from the instructor made a significant contribution both to
their learning of the subject matter and to their efforts in determining the content of the hypermedia that they
developed. One student commented that “it was nice to receive weekly positive and informative feedback; it
helped us in the process of content formation.” Another said, “Investigating the subject, writing the results of my
investigation, receiving feedback, and making revisions contributed much to my understanding of the subject.”
Many of the students perceived the reflective journal as an effective tool which helped them to learn the subject,
to retain the knowledge, and to internalize it in their long-term memory. The majority (N=19) of the students
stated that the process of searching for knowledge and transferring it into the content of the hypermedia required
a detailed examination of the subject, and that this resulted in a deeper understanding of the content. At the same
time, they mentioned that it was a difficult and time-consuming process, since it involved examining the
information, holding a group discussion on main themes, organizing the main themes based on their importance
for the subject area, and writing the journal. Though it was a demanding process for the students, most of them
found the journal writing process very beneficial for learning.
However, one group (five students) did not find the journal writing process to be very effective. One of this
group’s members said, “It was difficult to get together to discuss the journal, even through e-mail.” They stated
that this process would be more beneficial if there were fewer group members. Two of the group members said,
“Each week, two or three of us were responsible for journal writing. The members who wrote the journal
examined the subject deeply.”
Even though most of the students complained about the workload, they found it very beneficial for learning the
subject matter and for helping them prepare for hypermedia development. One student said, “If we did not write
the journal beforehand and go with hypermedia development, it would be too difficult to organize the content,
design intended hypermedia, and learn the subject.”
One of the problems in writing the journal was attributed to group structure and functioning. First of all, it was
difficult to get up to five members together to write the journals. Second, the students had difficulty distributing
responsibility and managing the tasks of the journal writing, since this was their first experience with such a
project. Third, the duration of the journal writing period (eight weeks) was long for the students in a self-
regulated learning process.
Developing hypermedia
All of the students (N=28) stated that in order to present the content in hypermedia, they first had to examine the
content, search for information, and learn the content themselves. They indicated that their specific
investigations into the subject matter, their journal writing to form the content of the hypermedia, and their
112
receipt of feedback about the journals all helped them in hypermedia development. Most of their learning
occurred while investigating and writing the reflective journals. They indicated that developing the hypermedia
required a preliminary examination of the content, and that the journal writing met this requirement in this
implementation. While determining their strategy for the presentation, they examined the structure of the content
and possible ways to present it. This process helped them review the materials and resulted in long-lasting
learning. One student said, “Preparation for the hypermedia development, rather than coding the hypermedia,
helped me learn the content.” The majority of the students in the groups stated that the hypermedia development
exercise was an important stage of this process. They mentioned that “if we were not required to write journals
about the content, we would not spend much time with the content, and the benefit of developing hypermedia
would be limited.”
Two groups (eight students) mentioned that the content covered in the hypermedia was too dense, and this
affected their learning in a negative way. One student suggested, “rather than covering a lot of content in one
hypermedia development project, it would be more beneficial to cover only one subtopic from the whole content.
This would give us time to make a deeper investigation into the topic, and would result in better hypermedia
design and learning.”
Three students from one group indicated that they learned some technical knowledge and skills related to
hypermedia development in this process. One of the students said, “One of our group members was
knowledgeable about the programming of hypermedia, and he taught us how to perform certain tasks. With his
guidance, we developed our parts, and then we combined all of our work. I learned a lot of new technical
knowledge and skills from my group-mates and class-mates in this process.”
However, some groups had problems at this stage, concerning the distribution of the tasks. One student
commented, “Since I was the most skillful one in the group, I performed most of the development tasks, and it
was too much work for me. I would not prefer such work.” His group members also confirmed his statements
about his contribution to the project. This same student suggested, “In the beginning of the semester, the
responsibilities of group members should be determined clearly in a contract, and assessment of the performance
should be based on this contract.”
At the beginning of the study, the students were asked if they needed any orientation relating to the knowledge
and skills they would need to use to develop the hypermedia. The students had replied that they did not require
any orientation, because they all had prior knowledge and experience in preparing hypermedia, though this was
of varying degrees. In accordance with this variance, some students stated in the interviews that they faced some
technical difficulties while developing the hypermedia. To overcome these difficulties, they received help from
their peers. The interview results show that students who had more knowledge and skills in this process helped
their classmates. One of the students said, “I did not have all the necessary knowledge and skills to develop a
good hypermedia project before we started. One of our group members had the necessary skills and knowledge,
and he taught me how to carry out some tasks. With his help, I overcame this problem and contributed to the
project equally.” All of the students stated that they used the Internet as a source of help in dealing with the
technical issues.
Group work
Most of the students (N=18) said that working in groups improved their problem-solving skills, their ability to
work collaboratively, and their ability to think critically in regard to the content and its presentation. They
learned from each other while preparing the content of the hypermedia and while they were deciding how to
present it. In addition, they learned a lot from each other (in groups and between groups) while solving the
technical problems they faced during the hypermedia development. They benefited from both content-based and
skill-based reflections, within the groups and between groups.
The majority (N=16) stated that they shared responsibilities in the groups based on their abilities and skills.
Social involvement was encouraged in group work. They learned to respect each other’s ideas and to reach a
consensus. Even though they complained about the extend of journal writing and self-study process, they said
that the long duration of the group project (fourteen weeks) helped them learn and produce even more. If they
had worked individually, they would have had simpler products, and in turn, they would have ended up with less
learning. Group interdependence made them work harder to achieve the course objectives.
113
However, the interview results show that two members from one group had some personal problems while they
were working on their project. These students did not indicate whether this problem resulted from the project or
from other factors. However, they mentioned that they solved their problems in a professional way. Their
friendship seems to have suffered, but they still managed to work on their project professionally. One of the
students from this group suggested, “This may be due to the long period of the project time.” They may have
realized that solving this conflict could contribute to their professional cooperation skills. However that may be,
it seems that the students were able to develop their conflict resolution skills in this process as well.
The groups which had five students (two of the groups) stated that their group size was not appropriate, because
five students were too many to work together while writing journals and developing the hypermedia. One
student said, “We expected other members to perform the determined tasks, so not all of us were involved in the
process fully.” Another problem was to arrange the meeting times, since students had different course schedules.
One student reported, “Since we had difficulty arranging meetings, we distributed the tasks. Even though we
examined what other people did in the project, I did not feel fully involved in the project.” Therefore, they
suggested that the number of students in each group should be three to four, in order to have more effective
collaboration and communication among the group members.
The groups also came up with some further suggestions to make the group work more effective. Though they
made group contracts by themselves in the beginning of the semester, they suggested that group rules should be
established at the beginning of the semester, with the guidance of the instructor. These rules should be observed
throughout the semester, and they should be monitored closely by both the group members and the instructor.
Students from three groups indicated that it was difficult to perform group activities in the computer laboratories
at the department. If they did not have their own facilities at home or in the dormitory, it would have been
difficult for them to perform this task. They suggested that “for such an implementation to take place, logistical
support and appropriate facilities should be provided, such as computers, related software, and audio-visual
processing technology. These facilities should be scheduled for the groups according to their time.”
Learning activities conducted in the classroom
All of the students (N=28) indicated that the instructional activities conducted by the instructor were effective for
helping them learn the content better. The learning activities carried out in class helped them acquire a broader
understanding of the subject matter. They discussed the problems which arose or the materials provided in class,
and each person in the activity group shared ideas with other group members. They indicated that this resulted in
their examination of the issues from different perspectives. One of the students said, “The learning activities
conducted in class made me focus on the most important parts of the subject. In the individual group discussion
and through whole group sharing, I learned both from my group members and other groups.” Another student
said, “The discussions about the activities made me see the problem from different dimensions, and this resulted
in better retention of the subject.” One student from the group mentioned that “the use of humor while working
on the activities made the knowledge more lasting.” These results indicate that the learning activities helped the
students learn abstract concepts more effectively. It also helped them improve their knowledge retention and
knowledge-transfer skills, and they became more critical about what they were learning.
In addition to the learning activities, all of the students (N=28) stated that they benefited from discussing each
group’s project in terms of design issues, which were included in the course content. The feedback given by the
instructor and the other students was valuable in terms of learning the design issues better and refining their
projects.
At the same time, all of the students (N=28) stated that the number of learning activities conducted in the class
was insufficient. They suggested that immediately after writing the journal, there should be a learning activity
related to the journal’s content. In this case, the course could have been more successful in terms of helping the
students to learn and apply the course content more effectively.
Assessment of the students’ performance
In this study, the students’ performance assessment was based on their reflective journals (15%), participation in
the class activities (10%), their hypermedia development (35%), and their final exam (40%). A majority of the
students (N=26) reflected that more weight should be given to journal writing and the term project than to the
114
final exam and participation. They preferred an assessment based on the tasks they performed throughout the
semester. One of the students stated, “40% for the exam is too much. We worked too much for the whole
semester, and it was not fair to get 50% (journal writing + hypermedia development) for that much work done
for the whole semester.” Another said, “We worked very hard on our projects throughout the semester, but we
worked for the exam for one week or one night. Therefore, our work for the project is worth more.”
All of the students (N=28) suggested a midterm exam to allow them to see the degree of their understanding of
the essential concepts and processes related to the course objectives. One student said, “I would prefer a midterm
exam to see my performance and the way the instructor asks questions in the final exam.” Another student said,
“Rather than having one exam for the whole content at the end of the semester, there could be two exams, and
we could divide the content into two, so the exams would not be so threatening for us.”
Three students criticized the evaluation process, noting that only academic performance was considered. One of
them said, “It is not fair to measure only academic performance, since we worked in groups so long. It would be
better to evaluate our behaviors in groups.” They suggested that group-members’ contributions to the projects
and participation in the project management process should also be taken into consideration. From these
statements, it is clear that the students felt the assessment process should be more directly related to the learning
process which the students went through to complete the course objectives.
Discussion
The results of this study reveal that the students found the strategy of using hypermedia as a cognitive tool to be
effective for constructing an understanding of the content. All but three of the students indicated that they prefer
this approach to traditional instruction, because they are more active and can construct their own knowledge. The
students stated that they learned a lot while writing their reflective journals to form the content of the
hypermedia. This result is consistent with propositions that Jonassen (2000) has made. He stated that students
should make their own decisions about searching for and gathering relevant information, and that they should
select the most important parts to represent. According to Jonassen, “this is the heart of the learning process” (p.
222). In line with this statement, journal writing for the purpose of establishing the content of the hypermedia
appears to be a valuable learning activity. Mayes (1993), and Turner and Handler (1997) argued that a student
who develops hypermedia material learns the topic better than one who is merely exposed to the finished
product. In another study by Yildirim (2004), in which students implemented visual design principles from their
focal subject into hypermedia products which they were developing as cognitive tools, the author indicated that
the students’ hypermedia products reflected that students both learn and implement what they learn in this active
process.
Overall, the students did perceive the use of hypermedia as a cognitive tool to be effective for learning.
However, their suggestions about the amount of the content to be covered in such a learning environment need to
be considered. As they suggested, rather then having too much content, it is better to allow the students to focus
on one specific subject in depth.
Another result of this study is that most of the students benefited from the group work. Petraglia (1998) stated
that “the presence of other learners provides students with the means to gauge their own progress which, in turn,
assists them in identifying their relative strengths and weaknesses and permits them the insight necessary to
improve their own learning” (p. 55). In this study, the students not only learned the content from each other but
also the technical skills to design the hypermedia, and some social skills, such as project management, reaching a
consensus, and respecting each other’s ideas. The long period of collaborative group work was effective in terms
of helping them learn the content and build social skills. This result is consistent with Krajcik et al.’s (1994)
statement that, through project-based learning, students concentrate on complex, integrated modules of long-term
instruction (cited in Jonassen, 1998). In keeping with Krajcik et al.’s suggestion, this study was based on group
projects and lasted for a reasonably long period of time. This teaching method was found to be effective by most
of the students. To help the students benefit more from the group work, distribution of the tasks at the beginning
of the project and close monitoring of the group members’ performances throughout the process should also be
considered.
The students felt that the number of learning activities conducted in class was insufficient. To make this strategy
more effective, the instructor should provide more learning activities. Providing a contextually rich learning
environment that supports the student designers seems to be important for knowledge acquisition.
115
All of the students in the study felt that the assessment of their performance was problematic. They were not
happy with the high emphasis on the final exam, and suggested that it would be better to assess their
performance based on the tasks they performed throughout the semester. As Jonassen, Mayes, and McAlesse
(1993) mentioned, assessment of constructivist learning experiences should reflect the knowledge construction
process. The reflections of student teachers in this study seem to confirm this principle as well.
In their study, Kafai, Ching, and Marshall (1997) concluded that, in collaborative activities, students need more
support if the activities are intended to offer an effective learning context for each student in the group. Having
students design hypermedia to learn the subject area is a challenging activity and may not result in sufficient
learning when it is used alone. To benefit more from using hypermedia as a cognitive tool, students should be
guided throughout the process. Moreover, they should be supported by additional learning activities to keep
them on track, and to help them focus their attention on the key aspects of the subject area. Finally, they should
be assessed in a way that reflects the learning processes they go through.
The results of this study offer important insights into how learners progress through the use of hypermedia as
cognitive tools in their knowledge construction processes, and how they compare this learning context with a
traditional one. In addition, the results may help instructional material designers/developers and practitioners in
schools to better understand the potential contributions and limitations of hypermedia as a cognitive tool in
classroom settings. Though it requires commitment from both the students and the instructor, using hypermedia
as a cognitive tool is a valuable learning experience. The results of this study show mostly a positive outcomes
arising from the use of hypermedia as cognitive tool. However, further research studies are needed to examine
the full effects of hypermedia as a cognitive tool for knowledge acquisition, in comparison to traditional
classroom instruction and other computer-based cognitive tools.
References
Blumenfeld, P. C., Soloway, E., Marx, W. R., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating
project-based learning: Sustaining the doing, supporting the learning. Educational Psychologist, 26 (3/4), 369-
398.
Carver, S. M., Lehrer, R., Connel, T., & Erickson, J. (1992). Learning by hypermedia design: Issues of
assessment and implementation. Educational Psychologist, 27 (3), 385-404.
Duffy, M. D., & Cunningham, D. J. (1996). Constructivism: Implications for design and delivery of instruction.
In Jonassen, D. H. (Ed.), Handbook of research for educational communications and technology, New York:
Simon & Schuster Macmillan, 170-198.
Jonassen, D. H., & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools. In
Jonassen, D. H. (Ed.), Handbook of research for educational communications and technology, New York:
Simon & Schuster Macmillan, 693-719.
Jonassen, D. H. (1998). Designing constructivist learning environments. In Reigeluth, C. M. (Ed.), Instructional

theories and models (2nd Ed.), New Jersey: Lawrence Erlbaum, 219-239.
Jonassen, D. H. (2000). Computers as mindtools for schools (2nd Ed.), New Jersey: Prentice-Hall.
Jonassen, D. H., Mayes, T., & McAleese, R. (1993). A manifesto for a constructivist approach to uses of
technology in higher education. In Duffy, T. M., Lowyck, J. & Jonassen, D. H. (Eds.), Designing environments
for constructivist learning, Berlin: Springer-Verlag, 231-247.
Kafai, Y. B., Ching, C. C., & Marshall, S. (1997). Children as designers of multimedia software. Computer
Education, 29 (2/3), 117-126.
Kozma, R. B. (1992). Constructing knowledge with learning tool. In Komers, P. A. M., Jonassen, D. H. &
Mayes, J. T. (Eds.), Cognitive tools for learning, Berlin: Springer-Verlag, 23-32.
Lehrer, R. (1993). Authors of knowledge: Patterns of hypermedia design. In Lajoie, S. P. & Derry, S. J. (Eds.),
Computers as cognitive tools, Hillsdale, NJ: Lawrence Erlbaum Associates, 197-228.
116
Liu, M. (2003). Enhancing learners’ cognitive skills through multimedia design. Interactive Learning
Environments, 11 (1), 23-39.
Mayes, J. T. (1992). Cognitive tools: A suitable case for learning. In Komers, P. A. M., Jonassen, D. H. &
Mayes, J. T. (Eds.), Cognitive tools for learning, Berlin: Springer-Verlag, 7-18.
Mayes, J. T. (1994). Hypermedia and cognitive tools, retrieved May 10, 2004 from
http://penta.ufrgs.br/edu/telelab/11/paper9.htm.
Miles, M. B., & Huberman, M. (1994). Qualitative data analysis: An expanded sourcebook (2nd Ed.), Thousands
Oak, CA: Sage.
Oliver, R., & Herrington, J. (2003). Exploring technology-mediated learning from a pedagogical perspective.
Interactive Learning Environments, 1 (2), 111-126.
Patton, M. Q. (1987). How to use qualitative methods in evaluation, Newbury Park, California: Sage.
Pea, R. D. (1985). Beyond amplification: Using the computer to reorganize mental functioning. Educational
Psychologist, 20 (4), 167-182.
Perkins, D. N. (1986). Knowledge as design, Hillsdale, New Jersey: Lawrence Erlbaum.
Petraglia. J. (1998). The real world on a short leash: The (mis)application of constructivism to the design of
educational technology. Educational Technology Research and Development, 46 (3), 53-65.
Reeves, T. C. (1999). A research agenda for interactive learning in the new millennium, retrieved February 18,
2005 from http://it.coe.uga.edu/~treeves/EM99Key.html.
Steketee, C. (2002). Students’ perceptions of cognitive tools and distributed learning environments. In Goody,
A., Herrington, J. & Northcote, M. (Eds.), HERDSA Proceedings, Higher Education Research and Development
Society of Australasia, 626-633, retrieved October 10, 2004 from
http://www.ecu.edu.au/conferences/herdsa/main/papers/ref/pdf/Steketee.pdf.
Turner, S. V., & Handler, M. G. (1997). Hypermedia in education: Children as audience or authors? Journal of
Information Technology for Teacher Education, 6 (1), 25-35.
Yildirim, Z. (2004). Outcomes of constructivist learning environment: How learners apply visual design
principles. Education and Science, 29 (132) 78-84.
117
Zhu, X. (2005). Designing an open component for the Web-based learning content model. Educational Technology &
Society, 8 (2), 118-124.
Designing an open component for the Web-based learning content model

Xin-hua Zhu
Department of Computer Science
Guangxi Normal University
Guilin 541004, China
Tel: +86-773-5848991
zxh429@263.net
ABSTRACT
On the basis of analyzing the characteristics of content components in the current distance education
technology specifications, this paper puts forward an Open Content Object model for the Web-based
learning content by extending the Sharable Content Object (SCO) of the Sharable Content Object Reference
Model (SCORM) which was established by the Advanced Distributed Learning (ADL) of the Office of the
Secretary of Defense (OSD). In this components model, the Open Content Object has the function of
requesting services and providing services by the messages passing mechanism which takes the Learning
Management System(LMS) as scheduling center of messages, thus the content components can not only be
aggregated to compose higher-level units of instruction, but also can form the interoperable associational
relationship, thereby it can provide a more flexible and effective approach to the design of the interoperable
aggregation and sequence of the Web-based learning content.
Keywords
Distance education technology specification, Content components, Sharable Content Object, Open Content
Object
1. Introduction
Nowadays, in order to accelerate the development of the distance education based on WWW and to implement
the share and reuse of the learning resources, many international organizations are researching on and
establishing the standards and specifications of the distance education. Notably, among these initiatives are the
Aviation Industry CBT Committees (AICC,1998),the IMS Global Learning Consortium (IMS,1997), the IEEE
Learning Technology Standers Committee (IEEE/LTSC,1997), the W3C (1994), the Advanced Distributed
Learning Initiative/ Sharable Content Object Reference Model(ADL /SCORM,1999) and the European
Committee for Standardization/Information Society Standardization System (CEN/ISSS,1997). Each established
specification of the distance education has a Content Model that describes the components used to build a
learning experience from reusable learning resources; the Content Model also defines how these lower-level
sharable, reusable learning resources are aggregated to compose higher-level units of instruction(ADL,2004a).
Essentially there are two types of the Web-based content components in all Content Models supported by
various specifications.
One is pure web page content component that doesn’t need the Learning Management System(LMS) to track.
This type of content component is a kind of basic learning content, and it can be supported by all the e-learning
specifications, such as IMS’s “webcontent” (IMS,2003a), ADL’s “Assets” (ADL,2004a) and CEN/ISSS’s
“Information Resource” (CEN/ISSS,2001). Formally, this type of content components can be electronic
representations of media, text, images, sound, web pages, assessment objects or other pieces of data that can be
delivered to a Web client. All pure web page content components are only encoded in JavaScript and HTML or
JavaScript and XML, and they do not need to communicate, using the API and Data Model, back to the Learning
Management System (LMS), so LMS can launch these components by using the HTTP protocol directly.
Another type is advanced content components that can be tracked by the LMS. Advanced content components
can communicate with an LMS when running, so that it can design different learning contents, approaches and
styles for different learners according to their abilities and performances. Additionally, it gives the access to the
new instructional technologies such as intelligent instruction and real-time instruction. This kind of content
components can only be supported by several specifications, such as ADL/SCORM’s “SCO” and AICC/CMI’s
“Lesson” (AICC,2000). The advanced learning components need to communicate, using the API and Data
Model, back to the LMS, so LMS must launch them in a browser window that is a child window or a child frame
of the LMS window that exposes the API Adapter as a Document Object Model (DOM) Object(ADL,2004b).
The API Adapter must be provided by the LMS.
118
Being different from pure web page content components, advanced content components need a mechanism that
enables them to communicate with an LMS in the run-time environment, but they can only communicate with
LMS ,not with each other. Thus, both of the two types of content components are all self-contained in the
function and are all made to be independent units, and they can not link with each other in any way; neither
references nor messages are allowed between them (ADL,2004a;IMS2003a). The structure of the independent
components is closed; they can be used even though breaking away from the context of learning content, so they
are convenient to be aggregated, shared and reused. However, just because of the closed structure of the sharable
content components, there exist some problems:
1. Content components only have complemental relationships with each other in function, and they can be
aggregated to compose higher-level units, but they don’t have helpful relationships with each other in
running process, namely a content component cannot receive another’s support when running.
2. In a typical object-oriented system, there should be a relationship of aggregation and association between
objects (Rumbaugh,et al.,1991;Booch ,1993). But In the current specifications, as an instructional object, the
content components are forbidden to establish the associational relationship by the messages passing
mechanism. For this reason, the object-oriented characters of the learning system is not integrated.
3. The learning sequence cannot be implemented by the association of content components, so at present the
simple interoperable learning sequence can only be organized by the structure of content aggregation, while
the complex interoperable learning sequence like the flow of branched instruction must be designed in
content package outside of content components, and the design should conform to the rule of Sequence
Specifications based on XML(IMS,2003b; ADL,2004c). the XML (W3C, 1998) is a markup language that
implements data exchange. Its strength is data describing(Birbeck,M.,et al., 2001); using it to control the
learning flow is inefficient and complex.
To improve the above problems that are caused by the content components independent of learning content, this
paper puts forward a new content components model on the basis of the SCORM’s SCO (Sharable Content
Object). The model breaks the closing of content components. Its basic idea is to extend the component-units of
the learning content to be Open Content Objects(OCOs) that have the function of requesting services and
providing services by the message passing mechanism. This open components model allows OCOs to form the
interoperable associational relationship ,and makes the object-oriented characters of the learning systems more
comprehensive, thereby it can provide a more flexible and effective approach to the design of the interoperable
aggregation and sequence of Web-based learning content.
2. Open Content Object

Open Content Object (OCO) is an open content component that has the function of requesting services and
providing services. The open characteristic of the OCO is evolved from the characteristic that ADL/SCORM’s
SCO can communicate with an LMS in the run-time environment. As the same as the program objects in C++
and Java programming, an OCO object can request other objects to provide services by sending messages to the
LMS, thus the associational relationship between OCO objects is formed. OCO object may request other OCO
objects to provide services when it offers services to others, and the learning experience can be completed in the
process of continual object requesting and object serving.
To enable that OCO object can be moved independently between different LMSs, but not be moved together
with the requested OCO object, the representation of the associational relationship between OCO object and its
requested OCO object must be picked up from the OCO object’s internal design, thus replacing a message that
sends service request to the LMS. The message mapping about the requested OCO object’s appointment is put
off, and it is not expressed as an interoperable standardized manner in the content packages outside of OCO
object until OCO is aggregated, ultimately the service request and response in a learning experience will be
dynamically accomplished via the LMS . The figure 1 illustrates how an OCO object and its requested OCO are
dynamically linked by the messages passing mechanism which takes the LMS as scheduling center of messages.
When issuing, an OCO must bind related meta-data to describe the listing of request services and the interface of
service functions. Depending on the meta-data, the OCO requesting services and the OCO providing services in
a content package can be designed by different developers, and an OCO object's services request is allowed to be
responded by different OCO objects in different content packages. Thereby OCO object can be independently
moved with a package from one LMS environment to another, and the 0associational relationship that is formed
between OCOs by the messages passing mechanism is interoperable in different LMSs (shown in the figure 2).
119
Figure 1. The process of dynamic link between an OCO object and its requested OCO
Figure 2. The interoperable associational relationship between OCOs
Being based on the Run-Time Environment in ADL/SCORM specification, the Run-Time Environment of OCO
can be formed by extending SCORM's API adapter methods and communication data model, and OCO can be
formed by using some properly extended API calls that send service request messages to the LMS in SCORM's
SCO object. In the structure of OCO based on SCO shown in the figure 3, when instructional events that trigger
the service request happened, OCO sends a message about service request to the LMS through extended API
calls LMSSetValue (“cmi.core.message.name”,“messagename”), and meanwhile it requests the LMS to launch
the service object which is dynamically appointed by the LMS through extended API calls
LMSLaunch(“responder”).
Figure 3. OCO
120
3. Instances of the Application of OCO in the Design of Content Aggregation and
Learning Sequence
In the present specifications, to absolutely insure the reusability of content components, external learning
resources should not be referenced within components in direct or indirect manner and all components in content
packages are equal and independent, and there is no container that contains other components. For these reasons,
the object-oriented characters of the learning system are not integrated ,so the aggregation of learning content
only can be designed outside of the content components according to the content packaging specification (IMS,
2003a; ADL,2004a), and the learning sequence must be designed outside of the content components by using the
rules of sequence specifications (IMS,2003b; ADL,2004c). This causes the design of the content aggregation and
the learning sequence to be complicated and ineffective. After designing an open component OCO for the Web-
based learning content model, the learning contents can be aggregated through OCO containers and the learning
sequences can be organized through the associational relationship between OCOs. This particular approach to
design content aggregation and learning sequence has not been offered by any present specification.
In the following text, there are application instances of the use of OCO in the content aggregation and the
learning sequence design.
3.1. An Instance of Aggregating Learning Contents through OCO Container
Suppose there are three knowledge points in the same section of a course and their learning content components
respectively are OCO1, OCO2 and OCO3. Now those learning content components are required to be aggregated
to compose a bigger sectional learning unit. This can be accomplished by designing another container OCO4 that
contains these three components. It is shown in the figure 4.
Figure 4. The structure of OCOs’ container
The detail design steps are:

(1) Three learning events, which can be triggered by the action of some links, menu items or command buttons,
are designed in the OCO4 container to control the navigational sequence of the three contained components.
(2) In the program processing Learning events within the OCO4 container, indirectly linked the three contained
components by sending messages to the LMS.
(3) In the manifest file of content packages, only OCO4 will directly be the item of the aggregation structure,
while OCO1, OCO2, OCO3 will only be three responders of OCO4’s three messages, but not be the items
of the aggregation structure. Therefore, the design of aggregation structure will be simplified.
3.2. An Instance of Organizing Branched Instruction Through the Associational Relationship between
OCOs
After the learning process of an instructional unit is finished, different subsequent instructional units will be
provided to the learners according to their grades. This is the so-called branched instruction, which is a basic and
common instructional manner in Computer-Based Instruction (Kearsley, 1983). In the content design that
conforms to IMS and ADL specifications, the interoperable flow of branched instruction can only be organized
in content package according to the rules of sequence specifications, and the implementing process is very
complex. After OCO is imported, the interoperable flow of branched instruction can be organized through the
121
associational relationship between OCOs, and the implementing process is simple and perspicuous. For example,
as for the branched instruction shown in the figure 5, it can be organized through the associational relationship
shown between OCOs shown in the figure 6.
Figure 5. A flow of branched instruction
Figure 6. The OCOs’ association relationships which are used for organizing branched instruction
The detail design steps are:

(1) The preceding learning unit and two succeeding learning units are respectively designed to be OCO1, OCO2
and OCO3.
(2) When the preceding learning object OCO1 is being designed, the following branched controlling sentences
will be added at the end of OCO1’s interior:
if (score>75) then
send message1 to the LMS
else
send message2 to the LMS
(3) In the content package, the responder of message1 will be mapped to be OCO2, and the responder of
message2 will be mapped to be OCO3 by an extended element MessageMapping that is required for OCOs’
packaging in a content package:
<MessageMapping>
<mapping message=”MESSAGE1” responder=” OCO2” />
<mapping message=”MESSAGE2” responder=” OCO3” />
</MessageMapping>
4. Conclusions
Designing content components to be open ones which have the associational relationship makes the
interrelationship between content components more comprehensive, thereby the learning contents can be
aggregated through OCO containers and the learning sequences can be organized through the associational
relationships between OCOs. This particular approach to design the interoperable aggregation and sequence of
Web-based learning content is flexible and effective and has not been offered by any present specification. It has
two characteristics:
122
The design of content aggregation and learning sequence is partly implemented witin the OCO object. The
reason is: When content aggregation and learning sequence are designed through OCO containers and
associational relationships, the message sending, the navigational control of learning event and the control of
learning flow are implemented within an OCO object through the advanced programming language like
JavaApplet, JavaScript. The advanced programming language is inherently smart in flow control, so this
approach is flexible and effective in the design of learning content aggregation and sequence.
The content aggregation and the learning sequence designed through OCO containers and associational
relationships are interoperable. The reason is that the associational relationship between OCOs is formed by
OCO sending service request message to the LMS， but not formed by OCOs’ referencing directly with each
other inside of the OCO, and the corresponding message mapping is described as an interoperable standardized
manner in the content packages outside of the OCO objects
The Open Content Object is only a conceptual model yet.As for the application of the Open Content Object , it is
prerequisite that a correlative reference model, which will be composed of content aggregation model and run-
time environment, should be designed. Fortunately, the OCO Reference Model can be established based on
ADL/SCORM. Now, the author is engaged in the related research work.
Acknowledgements
This work has been supported by the Guangxi Science Foundation of China under the contract number 0447034.
References
ADL (1999). Sharable Content Object Reference Model (SCORM), retrieved September 23, 2004, from
ADL (2004a). SCORM Content Aggregation Model (CAM) Version 1.3, retrieved September 23, 2004, from
ADL (2004b). SCORM Run-Time Environment (RTE) Version 1.3, retrieved September 23, 2004, from
ADL (2004c). SCORM Sequencing and Navigation (SN) Version 1.3, retrieved September 13, 2004, from
AICC (1988). Aviation industry CBT committees, retrieved September 13, 2004, from http://www.aicc.org.
AICC (2000). CMI001 Guidelines for Interoperability Version 3.4, retrieved September 13, 2004, from
http://www.aicc.org/.
Birbeck, M., Duckett, J., Gudmundsson, O. G., Kobak, P., Lenz, E., Livingstone, S., Marcus, D., Mohr, S.,
Pinnock, J., Visco., K., Watt, A., Williams, K., Zaev, Z., & Ozu, N. (2001). Professional XML (Programmer to
Programmer) (2nd Ed.), Indianapolis: Wrox Press.
Booch , G.(1993). Object-oriented analysis and design with applications, Massachusetts: Addison Wesley.
CEN/ISSS (1997). CEN Information Society Standardization System, retrieved September 13, 2004, from
http://www.cenorm.be/ISSS.
CEN/ISSS (2001). CEN Workshop Agreement (CWA) 14356, retrieved September 13, 2004, from
http://www.cenorm.be/ISSS.
IEEE/LTSC (1997). IEEE Learning Technology Standers Committee, retrieved September 13, 2004, from
http://ltsc.ieee.org.
IMS (1997). IMS Global Learning Consortium, Inc, retrieved September 13, 2004, from
http://www.imsglobal.org.
123
IMS (2003a). IMS Content Packaging Specification, Version 1.1.3 Final Specification, retrieved September 23,
2004, from http://www.imsglobal.org/content/packaging.
IMS (2003b). IMS Simple Sequencing Information and Behavior Model, Version 1.0 Final Specification,
retrieved September 23, 2004, from http://www.imsglobal.org/simplesequencing.
Kearsley, G. (1983). Computer-based training: a guide to selection and implementation Reading, Massachusetts:
Addison-Wesley.
Rumbaugh, J. R., Blaha, M. R., Premerlani, W., Eddy, F., & Lorensen, W. (1991). Object-Oriented Modeling
and Design, New Jersey: Prentice Hall.
W3C (1998). XML: eXtensible Markup Language Version 1.0, retrieved September 13, 2004, from
http://www.w3.org/XML.
124
Jones, G. H., & Jones, B. H. (2005). A Comparison of Teacher and Student Attitudes Concerning Use and
Effectiveness of Web-based Course Management Software. Educational Technology & Society, 8 (2), 125-135.
A Comparison of Teacher and Student Attitudes Concerning Use and

Effectiveness of Web-based Course Management Software
Gary H. Jones, Ph.D.
College of Business
Western Carolina University
Cullowhee, NC 28723 U.S.A.
Ph: (828) 227-3615
Fax: (828) 227-7414
gjones@email.wcu.edu
Beth H. Jones, Ph.D.

College of Business
Western Carolina University
Cullowhee, NC 28723 U.S.A.
Ph: (828) 227-3465
Fax: (828) 227-7414
bjones@email.wcu.edu
ABSTRACT
The purpose of this study was to assess the perceived effectiveness of the CourseInfo instructional
technology package implemented at a regional Midwestern university. An online survey was administered,
with 971 students and 44 faculty responding. Results indicate attitudes of both faculty and students were
positive – both believed that the web, and specifically Blackboard's CourseInfo, were beneficial educational
tools. They felt confident in their ability to utilize both CourseInfo and the Web and considered themselves
computer literate. Faculty, significantly more than students, believed both teacher-student and student-
student communication was facilitated. Students, much more than faculty felt that CourseInfo had improved
student learning.
Keywords
Course management software, CourseInfo, WebCT, Online learning environment
Introduction
Many educational institutions are making significant investments in Web-based course management software
such as WebCT and Blackboard (and its subset, known until recently as CourseInfo). Such packages allow for a
standard Web interface—for both faculty and students—across a networked environment. When installed by a
system administrator, software templates within these packages make available a variety of interactive features
that would otherwise require fairly sophisticated programming to install. The market for such software is highly
lucrative, with thousands of colleges and an ever-growing number of high schools considering the purchase of
such programs to be the first step towards Internet-classroom integration. The educational technology research
company MDR stated in its 2002-2003 College Technology Review: "Contributing to the increasing use of
electronic communication on campus, course management systems are present in virtually all (94%) of
colleges.” Among schools reporting a single course management system, Blackboard recorded the largest market
share at 46% of colleges, followed by WebTV at 35% and eCollege at 4% of schools. (MDR, 2002-2003, p. 3).
Blackboard announced its revenues of $25.5 million in Collegiate Presswire (2003) and pointed out this
constitutes an 11,046% revenue growth from 1998 to 2002. Business Wire, July 14, 2004, summarized the
results of a WebCT survey conducted by Boston-based Atlantic Research & Consulting in April, 2004 as
follows: "Thirty-seven percent of the survey's 416 respondents say they have implemented e-learning institution-
wide, up from 25 percent in 2002… Student participation in e-learning is growing at a 31 percent clip, and
faculty members are catching up to the demand with a 44 percent aggregate growth rate in e-learning
participation, according to the survey. The survey results indicate e-learning is no longer a peripheral part of
education at colleges and universities around the world" (WebCT User Survey, 2004).
Considering the vast amount of money, time, and energy being invested in Web-based teaching programs across
the country, a greater understanding of their perceived effectiveness is needed. To this end, both sets of users –
teacher and student – at a four-year regional Midwestern university were surveyed.
125
Literature Review
While appropriate warnings against the uncritical introduction of elaborate instructional technologies into the
curriculum have been voiced (Daniel, 1997; Ehrmann, 1991), there is widespread acceptance of the proposition
that new technologies generally—and the Web specifically—will continue to exert major influence on the ways
that knowledge is retrieved, stored and shared in educational settings (Brazell, 1998; Chickering & Ehrmann,
1997; Dugan, 1997; Ehrmann, 1991; Katz, 1999; Rowley, Lujan, & Dolence, 1998).
To date, much of the empirical research in this field has focused on the effectiveness of the strictly traditional
classroom versus the strictly virtual classroom. Tom Russell (1999) has compiled a comparative research
annotated bibliography that explores 355 research reports, summaries, and papers on technology for distance
education. Updates are referenced on his website as well: http://www.nosignificantdifference.org/. (Additional
studies in this area include Carr, 2000; Irons, et al., 2002; Jones, 1999; Koorey, 2003; Schutte, 1997; Spooner,
1999; White, 1999.). While data from studies examining virtual environments are important to schools when
deciding whether or not to add distance education courses to their curriculum, they are less helpful when
analyzing the Web’s most likely role in the future of mainstream education—the integration of Web components
into a standard classroom.
Several studies have explored the effect of course management software systems on student performance and
attitudes. In Rivera, McAlister and Rice’s 2002 study, one section of an introductory management information
systems course was offered almost exclusively via the web, another was taught in the traditional classroom
setting and a third was a hybrid of traditional supported by the course management system WebCT. Class
enrollment averaged 45 per section. No differences in student performance (as measured by exam scores) were
found. Students were generally satisfied with the traditional and hybrid classes; less so with the online course. A
similar study was conducted by Lim (2002) who also compared three different instructional formats. One was a
traditional classroom with eight students, one was Web-based with six students, and five students participated in
the satellite based section (distance learning with video and communication systems). No significant differences
in learning were identified. Wernet, Olliges and Delicath (2000) reported on a survey of 39 social work students’
satisfaction with, and perceptions of, WebCT in social work education. All students found course materials on
the course Web site helpful. In a study where WebCT was used as a primary means of delivery in a Community
Information class with an enrollment of 46, student surveys reported high student satisfaction and good student
performance (Kendall, 2001). Sherman (1998) introduced Web assignments into an advanced course in social
psychology using activities that entailed collaboration and knowledge sharing. Web components of the course
engendered an overall positive response from the 10 students in this class. Murphy and Lidner (2001) surveyed
111 students enrolled in one Texas A & M undergraduate course on technological change which used WebCT
through the semester. They asked students five questions about their perceptions of the use of WebCT. Overall,
students agreed that WebCT had “contributed to their success in the course” (72%), had a positive perception of
the use of WebCT in the course (89%), and most reported that they had accessed their grades via WebCT
(76 %). Figueroa and Huie (2001) anecdotally reported students in computer information system courses “liked
using Blackboard as a record of their grades, course outlines, and other pertinent course information” (no survey,
no “n” reported.)
Fewer studies have assessed teachers’ attitudes towards the effectiveness of course management software. Witt
(2003) surveyed 36 instructors using supplemental Web sites for a classroom course (24 responded), asking
open-ended questions designed basically to assess “Are course Web sites worth the trouble?” Most instructors
indicated that their goals for the Web sites have been entirely or largely achieved, and most consider the sites to
be essential to successful course design. A number of the respondents felt that course Web sites promoted “more
and better communication.” In a study by Crooks, Yang and Duemer (2003), 127 faculty answered eight survey
items designed to measure views about the effectiveness, reliability, and usability of Web-based resources for
instruction purposes. The questions referred to Web-based resources in general rather than Web-based course
management software tools. There were no differences found in the six demographics tested: current position,
gender, years in academia, public vs. private institution, institution’s student population, and university
classification (doctorate, masters, etc.) Daugherty and Funke (1998) conducted a study of faculty and student
perceptions towards Web-assisted learning at Georgia College and State University. They surveyed 76 faculty
members recruited from listserves devoted to combining education and technology and required 36
undergraduate students enrolled in a health science course to complete several Web-based assignments during
the semester. The surveys consisted entirely of open-ended questions regarding the problems, advantages, and
general issues of Web-assisted education. Overall, both students and professors were upbeat about the effects of
the Internet on the learning process. Fifty-five percent of the students stated that communication was effective
during the Internet portion of the course. In fact, many students claimed that teacher-student communication
126
improved due to Internet interaction. Both faculty (38%) and students (58%) believed that the biggest advantage
of Web-based education was that it taught students the computer skills necessary to be competitive in the modern
workforce.
The problem with these studies is their limited scope. None has large sample sizes, and a few have such small
samples that no generalizations can be made. Every study of students cited above only used one class, or at best
several sections of one class. Of the three studies that surveyed faculty, two only used open-ended questions, the
third did not ask about course management software, and none of the three directly compared faculty and student
opinions.
The study at hand adds to previous research by surveying both students and teachers within all academic
divisions at one university. It reports on attitudes of over 1,000 respondents. A Likert-scale questionnaire was
used in order to facilitate comparisons of student-teacher perceptions. The large sample size, variety of users,
and use of the same questionnaire for faculty and students are particular strengths of this study.
Research question and methodology

The primary objective of this project was the measurement of faculty and student opinions regarding
CourseInfo’s effectiveness. The general research question is: “What is the perceived utility of CourseInfo by
involved faculty and students, and how do those perceptions compare?”
Based on prior research, we expect both faculty and students who use CourseInfo to have a positive attitude
about CourseInfo’s capabilities and educational effects. We have no a priori reason to expect student and teacher
attitudes to differ significantly and we found no prior research addressing this question. Therefore, we
hypothesize no differences in perceptions between these two groups.
A Web-based survey was used for data collection. The survey used the software package “Web Survey
Assistant,” available online (http://www.surveyassistant.com). This survey, in two versions, was formatted for
the Web and posted on the principal author’s university Website. The two versions—one for faculty and one for
students—consisted of a set of identical questions; however, each incorporated a few questions unique to their
respective target respondents (Appendix A). A link to these surveys, with brief cover letter and permission
request, was e-mailed to all professors using CourseInfo. Instructors were requested to forward by e-mail the link
to the student version to all of their students enrolled in CourseInfo-assisted classes. The target populations were
thus easily identified and easily reached.
The institution where the survey was administered is a four-year regional comprehensive Midwest public
institution with a declared liberal arts mission. ACT scores of entering freshman average 27; nine percent of the
student population is minority. Virtually 100% of the student body consists the traditional 18-22 year-old. The
student body size is approximately 6,000 undergraduate students, with a male-to-female ratio of 40%-to-60%.
Over 93% of faculty are full-time, numbering 360. Eighty-five percent of full-time faculty hold terminal degrees
in their discipline.
The questions given to both students and faculty focused on the usefulness of the program itself, its integration
into the classroom environment, and its effects on the student/instructor’s learning experience. Basic
demographic questions regarding gender, academic division/discipline and, in the case of students, year in
school, were also included. The survey resided on the Web for two weeks; most data were collected the final
week of classes. In order to attract student responses, a random drawing for $50 was announced. Entry in the
drawing was optional, but did require that interested respondents to leave their name and e-mail address. Faculty
were offered $5 for taking the survey.
Results
Forty-four faculty and 971 students responded to the Web-based electronic survey. All student respondents were
undergraduate students. Student respondents across grade level were roughly proportionate to actual enrollment
numbers. Responses by academic division were more disparate: Majors in the academic division of Business
and Accountancy represented nearly 25% of all student responses, followed by Language and Literature, 17%;
Human Potential and Performance, 16%; Social Science, 15%; Science, 14%; with various other divisions
adding to 13%. Thirty-four percent of the student respondents were male; 66% were female, reflecting a slightly
127
higher female response than university percentage (60% female). Responses were generally evenly split among
grade levels (30% freshmen, 21% sophomores, 25% juniors, 24% seniors). Faculty responses also came from
throughout the university, with faculty from 9 of the 11 divisions responding. The majority of faculty were from
Language and Literature (45%), followed by Science (20%) and Human Potential and Performance (14%).
Respondents answered nine Likert-scale questions which asked about attitudes toward the World Wide Web and
CourseInfo specifically. Responses were scaled from +3 (strongly agree) to -3 (strongly disagree). A problem
with the survey form was the accidental omission of the "disagree" category, forcing respondents who disagreed
to choose between the milder "tend to disagree" or the more forceful "strongly disagree" category. However,
there was very little disagreement with the survey questions (and the polarity of the response set was always
consistent). So to compensate, every “tend to disagree” (-1) response was recoded as a “disagree” (-2) response,
realizing that if the response to the question still differed significantly from zero in the positive direction, this
difference was meaningful. In other words, the coding adjustment overcompensated for potential data error.
Even with this scaling, responses on every faculty survey question and every student survey question but one
were significantly positive at the p < .01 level. That is, respondents of this survey felt positively about their
computer skills and about CourseInfo’s enhancement of communication and its educational value (see Table 1:
Mean Scaled Survey Responses of Faculty and Students). Specifically, both faculty and students felt they were
computer literate and confident in their ability to utilize both the World Wide Web and CourseInfo. Both groups
of respondents, on average, considered themselves active CourseInfo users, believed both the Web and
CourseInfo were “beneficial educational tools’ and that CourseInfo “improved student learning”. They also felt
CourseInfo facilitated teacher-student communication and students liked it when professors communicated via
email. Teachers agreed that CourInfo facilitated student-to-student communication while students did not agree.
To test for significant differences between student and faculty means, a multivariate model was run using SPSS.
The nine questions were entered as dependent variables, the student-teacher variable was the independent
variable, with gender and division entered as covariates. Test results are shown in Table 2 (Table 2: Tests of
Between-Subject Effects). Significant differences between teacher and student attitudes were detected at the p
<.05 level on three of the nine ranking questions tested. Two of these dealt with student/faculty perception
regarding CourseInfo’s ability to facilitate electronic communication. Faculty (mean .68) were significantly
more likely than students (.03) to agree with the statement “CourseInfo substantially facilitates student-to-
student communication” (p < .02). This is the one question where the student response was not significantly
different from zero. In other words, while faculty believed the course management software was enhancing
student-to-student communication, this was not the case according to students. Similarly, teachers felt more
affirmative in response to the question “CourseInfo substantially facilitates teacher/student communication”
(faculty mean 1.84, student mean 1.08.)
Table 1. Mean Scaled Survey Responses of Faculty and Students

Mean Std Deviation n Agree***
I consider myself computer Literate
Student 2.11* 1.05 971
Faculty 2.30* .95 44
I am confident in my ability to utilize the WWW
Student 2.27* .93 971
Faculty 2.55* .59 44
Web is beneficial educational tool
Student 2.35* .91 971
Faculty 2.20* .73 44
I am an active CourseInfo user
Student 1.52* 1.45 971
Faculty 1.84* 1.24 44
I am confident in my ability to use CInfo
Student 2.19* .98 971
Faculty 1.98* 1.21 44
CInfo is beneficial educational tool
Student 1.72* 1.36 971
Faculty 2.07* .82 44
CInfo facilitates teacher-student communication
Student 1.34* 1.36 971
Faculty 1.84* 1.08 44
128
CInfo facilitates student-to-student communication
Student .03 1.75 971
Faculty .68* 1.20 44
CInfo has improved student learning
Student 1.93* 1.20 971
Faculty 1.27* .90 44
I like it when professors communicate via e-mail
Student 2.66* 1.23 971
Faculty N/A N/A N/A
(Where 3=Strongly Agree, -3=Strongly Disagree)
* Mean response significantly differs from zero (p <.01, one-sample t-test), i.e., respondents agree with these
statements.
Table 2. Tests of Between-Subject Effects

User Type Gender ± Academic Division
(Teacher vs. Student) (Male vs. Female) (11 Areas of Study)
Dependent Variable F Sig. F Sig. F Sig.
I consider myself computer literate 0.847 0.358 16.702 0.000* 0.103 0.748
I am confident in my ability to utilize the WWW 2.658 0.103 28.840 0.000* 0.119 0.730
Web is a beneficial educational tool 1.200 0.274 1.923 0.166 5.076 0.024*
I am an active CourseInfo user 2.480 0.116 0.593 0.441 7.551 0.006*
I am confident in my ability to use CourseInfo 1.843 0.175 0.003 0.954 1.383 0.240
CInfo is a beneficial educational tool 3.048 0.081 0.112 0.738 2.719 0.099
CInfo facilitates teacher/student communication 6.433 0.011* 1.342 0.247 4.536 0.033*
CInfo facilitates student/teacher communication 5.990 0.015* 0.253 0.615 3.235 0.072
CInfo has improved student learning 13.553 0.000* 1.254 0.263 0.005 0.943
(* significant at the p <.05 level)
(± See Table 4)
Another interesting difference between student and teacher perceptions was in their response to the question “All
things considered, CourseInfo has improved student learning”. Both sets of respondents agreed with this
statement; specifically, 80% of faculty and 93% of students marked ‘strongly agree’, ‘agree’, or ‘tend to agree’.
The more marked contrast exists in the level of agreement. Seventy-seven percent of students either ‘strongly
agreed’ or ‘agreed’ with this statement, while only 39% of faculty answered so positively (see Table 3:
Responses to “CourseInfo Has Improved Student Learning”).
Table 3. Responses to “CourseInfo Improved Student Learning”

Students Faculty
Strongly Agree 358 37% 4 9%
Agree 388 40% 13 30%
Tend to Agree 154 16% 18 41%
Not Sure 26 2% 9 20%
Tend to Disagree 0 0% 0 0%
Disagree 37 4% 0 0%
Strongly Disagree 8 1% 0 0%
Total 971 100% 44 100%
There were two questions where there were significant differences between genders: “I consider myself
computer literate” and “I am confident in my ability to navigate the WWW”. Further analyses were performed
on responses to these two questions. We compared male teachers and male students on both questions; no
significant difference. The test was rerun comparing female teachers and female students; again, no significance.
We examined only teachers – male vs. female – and again no significance on either question. The significance
differences on both questions occurred between male students and female students (p <.001 on both questions).
See Table 4: Questions Where Gender Was Significant.
129
Table 4. Questions Where Gender Was Significant
Teachers Students
n mean std. dev. n mean std. dev.
Males 21 2.52 0.68 332 2.29* 1.05
Females 23 2.09 1.12 639 2.02* 1.04
“I consider myself basically computer literate”
Teachers Students
n mean std. dev. n mean std. dev.
Males 21 2.62 0.50 332 2.49* 0.75
Females 23 2.48 0.67 639 2.16* 1.00
“I am confident in my ability to utilize the WWW”
(* Difference between male and female students significant at p <.001)
It is very interesting to note the direction of the means. Male students were clearly more self-confident about
their computer ability than female students. The same tendency, although not statistically significant (p = .13 and
.43 on the questions, respectively) can be observed in faculty responses. The large student n made their
difference of 2.29 vs. 2.02 (male-female, respectively) on the “computer literate” question statistically significant
while the numerically larger faculty differences of 2.52 vs. 2.09 were not statistically significant.
The other intervening variable deserved further examination as well. On three questions, responses differed by
academic division. These questions were: “The WEB is a beneficial educational tool”, “I am an active
CourseInfo user” and “CourseInfo facilitates teacher/student communication”. Tables 5a-5c show means, sample
size and standard deviation by division. The tables are in descending order by degree of agreement with the
question. Teachers and students were combined as the number of teachers by division was so small (44 teachers
in 10 divisions, or an average of less than 5 teachers per division).
Table 5a. “CourseInfo Substantially Facilitates Teacher/Student Communication”

Responses by Division (Where 3=Strongly Agree, -3=Strongly Disagree)
Division Mean N Std. Dev.
Education 1.73 15 0.96
Business & Accountancy 1.55 238 1.16
Undeclared 1.54 54 1.09
Math & Computer Science 1.53 53 1.32
Human Potential & Performance 1.33 160 1.25
Science 1.25 145 1.56
Language & Literature 1.23 176 1.47
Social Science 1.23 146 1.38
Fine Arts 1.14 22 1.25
Justice 1.00 6 1.55
Total or Average 1.36 1015 1.35
Table 5b. “I Consider Myself an Active CourseInfo User”

Fine Arts 1.55 22 0.91
Science 1.46 145 1.70
Justice -0.17 6 2.04
130
Table 5c. “The World Wide Web Is a Beneficial Educational Tool for Students”
Justice 2.33 6 0.52
Science 2.27 145 0.91
Fine Arts 1.86 22 0.94
It can be seen from Tables 5a-5c that each division’s mean response was positive on these three questions with
the exception of the six students in justice who did not consider themselves active CourseInfo users. Degree of
agreement differed, however. Consistently, the most positive responses came from the Business & Accountancy
division. Other than that, there was very little consistency among divisions.
One additional t-test was run. Respondents were split between “active users” and “non-active users” of
CourseInfo, based on their response to the question “I am an active CourseInfo user”. The “users” were those
who answered “Strongly agree”, “Agree”, or “Tend to Agree”; the non-users were all others. Comparing survey
results revealed a statistically significant difference (p < .01) on every one of the remaining eight questions. As
one would expect, user responses were more positive than nonusers. Cause-and-effect cannot be determined.
That is, the positive response from users may be due to self-selection, where students and faculty who like the
course-management software use it, or it may be that those students required to use the technology grew to
appreciate its benefits.
To compare student and faculty opinions on which CourseInfo features they found most useful, respondents
were asked to rank the top three (only) features of the CourseInfo program. The features listed were: ability to
link to external web-based resources, ability to link to internal course documents, calendar, digital drop box,
formulaic student Web page construction, online grade sheet, student survey/quiz, student/student
communication, and teacher/student communication. An “other” box was also provided. Students and faculty
both ranked "links to internal course documents" and "teacher-to-student email" as 1 and 2, respectively. They
agreed that the three features next in importance were "online grade sheets", "links to external web resources"
and "student-to-student email", though their rank order differed. The rankings are presented in Table 6 (Table 6:
Ranking of CourseInfo Features on Importance). The rankings of the “other” features were as follows:
Student: Calendar (6), Drop Box (7), Quiz (8), Other (9), Web Construction (10)
Faculty: Calendar (8), Drop Box (5 [tie]), Quiz (7), Calendar (8), Other (9) Web Construction (9
[tie]).
Table 6. Ranking of CourseInfo Features on Importance

Links to Internal E-mail (Teacher- Online Links to External E-mail (Student-
Course Documents to-Student Grade Sheet Web Resources to-Student)
Student 1 2 3 4 5
Rank
Faculty 1 2 5 3 4
Rank
A few questions of a descriptive nature were unique to the faculty version of the questionnaire. When asked the
most important factor in choosing what CourseInfo features were used in their courses, 22 faculty (or 50%)
indicated “general usefulness,” and 34% marked “relevance/contribution to course content.” The remaining 16%
of responses were about equally split between “knowledge of how to use the feature” and “time required to set
up/maintain the feature.”
131
Conclusion
There was overwhelming agreement between both faculty and student respondents that the Web is a beneficial
educational tool, and very substantial agreement that CourseInfo specifically is a beneficial educational tool
which improves student learning. Overall, both students and faculty respondents considered themselves
computer literate, were confident in their ability to utilize the World Wide Web, and considered themselves
active and competent CourseInfo users. The overall positive responses from our large survey of faculty and
students certainly gives the indication that web management software is perceived as beneficial.
One significant difference between faculty and student opinions was that faculty agreed CourseInfo “facilitated
student-to-student communication,” while students did not agree. This is understandable given that students
often use personal email accounts (Hotmail, Yahoo, etc.). Also, at this particular university they had access to an
email account they could throughout their student career. In contrast, the CourseInfo email accounts were
associated with a particular course and would run out at the end of each semester. Faculty also agreed more
strongly that CourseInfo “facilitated faculty/student communication.” The indication here is that faculty made
use of CourseInfo’s email feature. Given that one of the benefits of course management software is the ease of
sending email to the entire class, faculty appreciation of this feature makes sense. Students agreed that
CourseInfo facilitated faculty/student communication. They just did not agree as strongly with that statement as
faculty. A logical conclusion would be that they did not need the mass email feature. To students, shooting off an
email from their personal account was probably just as easy as sending one from their CourseInfo account. The
last significant difference between students and faculty was that students, more than faculty, agreed CourseInfo
had improved student learning. At first glance, this result appears surprising. Upon more reflection, it is
consistent with the experience of many faculty who find their students incautiously optimistic about their
performance and their grade. Or perhaps students were referring to the benefit of learning the web tools, while
faculty were thinking of course content.
Faculty and student ratings of CourseInfo features were similar. Both students and faculty rated “links to course
documents” and “teacher-to-student e-mail communication” first and second respectively. Among the top five
features only the importance of the online grade sheet differed by more than one position in the rankings, with
students ranking the feature 3rd and faculty ranking it 5th. Faculty might want to consider the importance of fast
and accessible feedback on student performance not only as it relates to CourseInfo classes, but generally.
When respondents were separated into those who agreed with “I am an active CourseInfo user” and those who
were neutral/disagreed, marked differences appeared. Those who agreed answered every one of the other eight
survey questions more positively than those who did not agree. Cause and effect cannot be determined;
however, there is certainly a correlation between use and positive opinion toward the course management
software.
Limitations and key assumptions

There are several limitations in the scope and accuracy of the data collected. First, the sample was not random—
the population consisted of students and faculty using CourseInfo. Further, participation among members of that
population was voluntary, introducing a self-selection bias. As a result, it is highly likely that both the student
and faculty samples were biased in the direction of computer literacy. Additionally, because the responsibility of
informing the student sample of the survey’s existence fell to the professors who maintained CourseInfo pages,
instructors who were more enthusiastic about the software may have more actively encouraged their students to
take the survey. The $50 random drawing may also have posed a threat to student respondents’ sense of
anonymity. Although entry into the drawing was voluntary and participant contact information was disassociated
from other data, the responses of participants who were particularly sensitive to the anonymity issue may have
been affected by the inclusion of their name and e-mail address.
This study also did not collect the opinions of teachers and students who do not use CourseInfo, including the
reasons why they may not. For example, the more computer-literate instructors may find it easier and more
efficient to create and maintain a VAX server page than a CourseInfo site.
This study focused upon perceptions rather than actual effectiveness. It does not attempt to provide an
assessment of the actual benefits or effectiveness of the CourseInfo program. One cannot assume that simply
because a particular feature is most frequently used or highly regarded by students or faculty members that it is,
in fact, the most relevant, important, or useful feature on the system. Popularity does not always equate to utility.
132
Suggestions for future research
Although not a formal aspect of the original set of research questions, some interesting gender differences were
noted in the data analysis. Twenty percent more males than females (54% vs. 34%) strongly agreed with the
statement “I consider myself basically computer literate.” Considering the large sample size, this is a very
noteworthy difference in self-perception. Similarly, 61% of the males, compared with 42% of the females,
strongly agreed with the statement regarding self-confidence of World Wide Web utilization. Although these
differences largely disappear when the two categories of strongest agreement are combined, the difference
between sexes at the strongest level of agreement is still surprising. This is made even more striking when
compared with the distribution of responses by gender to statements concerning activity on CourseInfo and
ability to utilize CourseInfo: there the differences disappear. This difference in self-perceived general ability
(computer literacy, Web utilization) vs. self-perceived specific abilities of program activity and utilization is
worthy of further examination.
Habits in each academic discipline can be different regarding the usage and perception of e-learning facilities. As
can be seen from this study, user attitudes differed by division on several questions. Future research is needed to
examine more specifically how web management software is being used in different disciplines and how this
usage affects attitudes and perceptions.
Finally, aspects of CourseInfo that did not feature prominently in this study could be more thoroughly examined.
A 1999 study by Romine on the use of “discussion boards” to foster student interaction with practicing
professionals is one promising avenue of exploration. Other communication features, such as facilitation of
group communication, could also be pursued. As the incorporation of external Web links into CourseInfo, and
the course, was a particularly popular use of the software a closer examination of the types of Web pages linked
and the manner of their elaboration is called for. Greater study of less frequently used but still potentially
worthwhile features like the "digital dropbox," online grade sheet, and student quiz could also be explored.
References
Brazell, J. (1998). Rewiring the way we learn. Bradley [University] Hilltropics, 4 (4), 10-13.
Carr, S. (2000). Online psychology instruction is effective, but not satisfying, study finds. Chronicle of Higher
Education, 46 (27), A48.
Chickering, A. W., & Ehrmann, S. C. (1997). Implementing the seven principles: Technology as lever (with
2003 update), retrieved November 6, 2004, from http://www.tltgroup.org/programs/seven.html.
Collegiate Presswire (2003). Blackboard announces record company revenue of $25.5 million, retrieved
November 12, 2004, from http://www.cpwire.com/archive/2003/11/4/1430.asp.
Crooks, S. M., Yang, Y., & Duemer, L. S. (2003). Faculty perceptions of Web-based resources in higher
education. Journal of Educational Technology Systems, 31 (2), 103-13.
Daniel, J. S. (1997). Why universities need technology strategies. Change, 29 (4), 10-17.
Daugherty, M., & Funke, B. (1998). University faculty and student perceptions of Web-based instruction.
Journal of Distance Education, 13 (1), 21-39.
Duggan, E. S. (1997). Higher education and the new media age. Liberal Education, 83 (2), 20-25.
Ehrmann, S. C. (1991). Gauging the value of a college’s investment in technology. Educom Review, 26 (3/4), 24-
28.
Figueroa, S., & Huie, C. (2001). The use of Blackboard in computer information systems courses. Paper
presented at the Teaching in the Community College Online Conference, April 17-19, 2001, ERIC Document
Reproduction Service No. ED 469883.
Irons, L. R., Keel, R., & Bielema, C. L. (2002). Blended learning and learner satisfaction: keys to user
acceptance? Retrieved November 6, 2004, from http://www.usdla.org/html/journal/DEC02_Issue/article04.html.
133
Jones, E. (1999). A comparison of an all-web-based class to a traditional class. Paper presented at the 10th
Society for Information Technology & Teacher Education International Conference, February 28 - March 4,
1999, San Antonio, TX, USA.
Katz, R. N. (1999). Dancing with the devil: Information technology and the new competition in higher
education, New York: Jossey-Bass.
Kendall, M. (2001). Teaching online to campus-based students: the experience of using WebCT for the
community information module at Manchester Metropolitan University. Education for Information, 19 (4), 325-
346.
Koorey, M. A. (2003). Differences in learning outcomes for the online and F2F versions of an 'Introduction to
Shakespeare, retrieved September 6, 2004, from http://www.sloan-c.org/publications/jaln/v7n2/v7n2_koory.asp.
Lim, D. H. (2002). Perceived differences between classroom and distance education: Seeking instructional
strategies for learning applications. International Journal of Educational Technology, 3 (1), retrieved April 2,
2005, from http://www.ao.uiuc.edu/ijet/v3n1/d-lim/.
MDR (Market Data Retrieval, a Company of D&B) (2002-2003). The College Technology Review, 2002-2003
Academic Year, retrieved September 9, 2004, from http://www.blackboard.com/docs/MDR2003.pdf.
Murphy, T. H., & Lidner, J. R. (2002) Building and supporting online learning environments through Web
course tools: It is whippy, but does it work? Paper presented at the 97th Annual Meeting of the Southern
Association of Agricultural Scientists: Agricultural Communication Section, January 27-31, Fort Worth, TX,
USA.
Rivera, J. C., McAlister, M. K., & Rice, M. L. (2002). A comparison of student outcomes & satisfaction between
traditional & Web-based course offerings, retrieved November 3, 2004, from
http://www.westga.edu/~distance/ojdla/fall53/rivera53.html.
Rowley, D., Lujan, H. L., & Dolence, M. (1998). Strategic choices for the academy: How demand for lifelong
learning will re-create higher education, New York: Jossey-Bass.
Russell, T. (1999). The no significant differences phenomenon, Montgomery, AL: International Distance
Education Certification Center (IDEC).
Schutte, J. G. (1997). Virtual teaching in higher education: The new intellectual superhighway or just another
traffic jam? Retrieved September 6, 2004, from http://www.csun.edu/sociology/virexp.htm.
Sherman, R. C. (1998). Using the World Wide Web to teach applications of social psychology to everyday life.
Teaching of Psychology, 25 (3), 212-216.
Spooner, F., Jordan, L., Algozzine, B., & Spooner, M. (1999). Journal of Educational Research, 92 (3), 132-
141.
WebCT User Survey. Business Wire (2004). WebCT user survey shows significant increase in enterprise-wide
implementations, usage and user satisfaction, Retrieved September 9, 2004,
http://www.internetadsales.com/modules/news/article.php?storyid=2369.
Wernet, S. P., Olliges, R. H., & Delicath, T. A. (2000) Post-course evaluations of WebCT (Web Course Tools)
classes by social work students. Research on Social Work Practice, 10 (4), 487-505.
White, S. (1999). The effectiveness of web-based instruction: A case study. Paper presented at the Joint Meeting
of the Central States Communication Association and the Southern States Communication Association, April 7-
11, 1999, St. Louis, MO, USA.
Witt, P. L. (2003). Enhancing classroom courses with internet technology: Are course web sites worth the
trouble? Community College Journal of Research and Practice, 27 (5), 429-438.
134
Appendix A
FACULTY AND STUDENT SURVEY QUESTIONS PERTAINING TO THIS STUDY
A. Questions on both faculty and student survey instruments
Both questionnaires asked gender, academic division, academic discipline, and the following Likert-scale
questions (Strongly Agree, Agree, Tend to Agree, Not Sure, Tend to Disagree, Strongly Disagree):
¾ I consider myself basically computer literate.
¾ I am confident in my ability to utilize the World Wide Web (e.g., search, navigate, bookmark)
¾ The World Wide Web is a beneficial educational tool for students.
¾ I consider myself an active CourseInfo user.
¾ I am confident in my ability to utilize CourseInfo.
¾ CourseInfo is a beneficial educational tool for students.
¾ CourseInfo substantially facilitates teacher/student communication.
¾ CourseInfo substantially facilitates student-to-student communication.
¾ All things considered, CourseInfo has improved student learning
Both questionnaires also asked:

¾ With respect to achievement of course learning objectives, please RANK the following features of
CourseInfo in order of importance to you by typing a number in the appropriate box [top THREE
only, with ‘1’ being most important]
Ability to link to external Web-based resources
Ability to link to internal course documents
Calendar
Digital Drop Box
Formulaic student Web page construction
On-line grade sheet
Student survey/quiz
Student/student communication
Teacher/student communication
Other
B. Additional question on faculty questionnaire:
¾ What is the most important factor in choosing what features you use on CourseInfo?
Knowledge of how to use the feature
Time required to setup/maintain the feature
General usefulness of the feature for you
Relevance/contribution to course content
Other
C. Additional questions on student questionnaire:

¾ Year in school (Freshman, Sophomore, Junior, Senior)
And the additional Likert-scale question:

¾ I like it when professors communicate with the class via email.
135
Liu, T.-C. (2005). Web-based Cognitive Apprenticeship Model for Improving Pre-service Teachers’ Performances and
Attitudes towards Instructional Planning: Design and Field Experiment. Educational Technology & Society, 8 (2), 136-149.
Web-based Cognitive Apprenticeship Model for Improving Pre-service

Teachers’ Performances and Attitudes towards Instructional Planning:
Design and Field Experiment
Tzu-Chien Liu
Assistant Professor
Graduate Institute of Learning & Instruction
National Central University, No. 300, Jung-da Rd.
Jung-li City, Taoyuan, Taiwan 320, R. O. C.
Tel: +886-3-4227151 ext. 33851
Fax: +886-3-4273371
ltc@cc.ncu.edu.tw
ABSTRACT
Instructional planning is an essential professional activity often used by teachers. However, some
characteristics of existing university-based teacher education programs may hamper pre-service teachers’
learning of instructional planning. Thus, this study adopts the cognitive apprenticeship as a theoretical
foundation to construct a web-based learning model that integrates expert teachers and Internet technologies
(web-based multimedia, performance support system, and electronic conferencing). To examine the
effectiveness of this model, a seven-week web-based course was designed and a field experiment was
conducted. Experimental results reveal that the course based on the web-based cognitive apprenticeship
model improves pre-service teachers’ performance and attitudes on instructional planning more effectively
than the traditional training course. Furthermore, the study discusses possible factors based on qualitative
data and provides recommendations for future studies and web-based teacher education instruction.
Keywords
Cognitive Apprenticeship, Instructional Planning, Web-based Learning, Teacher Education, Field Experiment
Introduction
Instructional planning plays a pivotal role in connecting curriculum to instruction (Byra, & Could, 1994; Clark &
Yinger, 1987), developing effective learning environments (Clark & Dunn, 1991; Reiser & Dick, 1996), and
effecting what occurs in the classroom context (Byra, & Could, 1994; Clark & Dunn, 1991; Clark & Peterson,
1986; Clark & Yinger, 1987). Even experienced teachers rely on it to ensure the direction of their teaching and
bolster their confidence (McCutcheon, 1980). The ability to plan instruction effectively can affect not only a
teacher’s success (Arnold, 1988; Borko & Niles, 1987) but also the results of education reform (Hoogveld, Paas,
Jochems, & Van Merriёnboer, 2002). Byra and Coulon (1994) pointed out that pre-service teachers must have
sufficient opportunity to learn instructional planning. Thus virtually every teacher education program allots
considerable time and effort in teaching pre-service teachers how to write instructional plans (Kagan & Tippins,
1992).
However, some features of university-based teacher education programs generally impede the learning of
practical knowledge and thinking skills about instructional planning. In the first place, pre-service teacher
education is often regarded as “overly theoretical, fragmented, and unconnected to practice” (Beck, & Kosnik,
2002, p. 420). For example, teacher educators often simplify and systematize the instructional planning process
and fail to link individual instructional plans with real classroom contexts (Clark & Yinger, 1987; Kagan &
Tippins, 1992; Neely, 1986). Secondly, teacher training in the university cannot sufficiently support pre-service
teachers (Furlong et al., 2002). When writing instructional plans, pre-service teachers need to deal with at least
five general characteristics involved in practical and dynamic situations, including “complexity, uncertainty,
instability, uniqueness, and value conflict” (Clark & Yinger, 1987, p. 97). During instructional planning, pre-
service teachers may become discouraged if they are not given gradual guidance and enough support. Finally,
from a psychological perspective, to grasp what expert teachers are thinking during their instructional planning is
crucial in clarifying how teachers comprehend, interpret, judge and transform knowledge; then formulate
intentions; and finally perform from that knowledge and those intentions (Clark, 1988). Through observing,
conversing, imitating, reflecting and modifying, pre-service teachers can learn high-level cognitive and meta-
cognitive skills which expert teachers use during instructional planning. However, during the university-based
teacher education program, pre-service teachers lack contact opportunities with experienced teachers and so few
of them have the opportunity to learn how an expert thinks (Schrader, et al., 2003).
136
These problems can also be found in the Taiwanese teacher education program. To improve the effectiveness of
pre-service teachers’ learning of instructional planning, the current study first constructs a web-based cognitive
apprenticeship model. In this model, cognitive apprenticeship is the theoretical foundation, network
technologies are the supporting teaching and learning tools, pre-service teachers are the learners, expert teachers
are the major instructors, and the Internet is the main learning environment. The second purpose of this study is
to test the effectiveness of this web-based model on the performance and attitudes of pre-service teachers with
respect to instructional planning, and to explore possible explanations for the test results.
Rationale for the Web-based Cognitive Apprenticeship Model

Cognitive Apprenticeship
Cognitive apprenticeship is based on Vygotsky’s research and is also related to other studies of conventional
apprenticeship (Brown, Collin, & Duguid, 1989; Lave, 1993; Lave & Wenger, 1991). Cognitive apprenticeship
is viewed as an “instructional tool” (LeGrand, Farmer, & Buckmaster, 1993) that is aimed at acquiring thinking
skills such as cognitive skills and metacognitive skills resulting in sustained participation within a community
(Brown et al., 1989; Collins, Brown, & Newman, 1989) and applicable to solving future problems (LeGrand, et
al., 1993). Collins et al. (1989) pointed out that in cognitive apprenticeship, learners can observe how experts
deal with problems in an authentic context, and they learn to solve the same or similar problems by “learning-
through-guided-experience” in authentic activities (p. 457). Farmer, Buckmaster, & LeGrand (1992) interviewed
450 practitioners in different professions to discover which instructional style could best help them learn and
understand how to solve complex and ill-defined problems. The interview results reveal that an instruction style
similar to cognitive apprenticeship is viewed as the most helpful one.
Some cognitive apprenticeship models have been constructed to enhance learning and instruction (For example:
LeGrand et al., 1993; Collins et al., 1989; Farmer et al., 1992). The model that Collins et al. (1989) proposed
listed six major steps: (a) Modeling: the experts demonstrate and explain their way of thinking for students to
observe and understand. (b) Coaching: the learners practice the methods, while the experts advise and correct.
(c) Scaffolding: through increasing the complexity of problems and decreasing the level of assistance according
to the learners’ progress, the experts progressively help the learners successively approximate the objective--
accomplishing a task independently. (d) Articulation: the learners are given opportunities to articulate and
clarify their own way of thinking. (e) Reflection: the learners compare their own thoughts with those of experts
and peers. (f) Exploration: the learners manipulate and explore the learned skills or knowledge to promote their
true understanding.
Furthermore, considering that few scholars have formally applied cognitive apprenticeship to different
professional areas, Farmer et al. propose an instructional model of cognitive apprenticeship which can be
employed in professional education (LeGrand et al., 1993; Farmer et al., 1992). This model comprises five
stages: (a) Modeling: an expert demonstrates the process of solving problems in realistic contexts and clarifies
the employed thinking skills. (b) Approximating: in authentic activities, the learners imitate the expert’s
behavior and thinking skills to cope with the same or similar tasks and express their ideas when solving
problems. (c) Fading: when the learners improve, the expert gradually reduces coaching and scaffolding. (d)
Self-directed learning: as the learners internalize the expert’s thinking skills, they accomplish tasks on their
own; at this time, the instructor assists only when a learner requests it. (e) Generalizing: the expert and all the
learners discuss how to generalize the learned behaviors and thinking skills that are appropriate to similar tasks
or problems.
Although these cognitive apprenticeship models offer conceptual frameworks and implementation methods,
some difficulties still exist when these frameworks and methods are applied to improving teacher education
programs. First of all, given the large number of pre-service teachers, expert teachers cannot offer in-depth
guidance. Secondly, since expert teachers take a long time to design, implement, and evaluate instructional
plans, pre-service teachers cannot simply be waiting and watching on the sidelines. Even if this were possible,
the practice would interfere with the normal teaching of expert teachers. Finally, expert teachers are scattered, so
pre-service teachers would have to travel far to discuss matters with expert teachers.
137
Web-based Technologies
Applying web-based technologies can help overcome these obstacles to implement cognitive apprenticeship in
teacher education. The following sections review three technologies which were applied in this study.
Multimedia programs allow “simulated apprenticeship as well as a wealth of learning support activities”
(Reeves, 1993, P. 107) and anchoring contexts for learning practical knowledge according to the situated
learning model (McLellan, 1994). Through multimedia case studies, teacher educators can lead focused
discussions and promote multiple perspectives as pre-service teachers watch and reflect on the same multimedia
fragments (Lambdin, Duffy, & Moore, 1997), and pre-service teachers can compare the instructional plan,
teaching demonstration, and teacher’s reflections to further understand the challenges in the classroom context
and the methods in hand (Barron & Goldman, 1994). “Web-based” multimedia can free the above-mentioned
merits from time and space limitations, and can effectively combine other network functions (Barnett, Keating,
Harwood, and Saan, 2002).
Moreover, performance support systems, such as Electronic Performance Support System (EPSS), Support for
Teachers Enhancing Performance in Schools (STEPS), and Instructional Planning Assisting SyStem (IPASS)
that were developed by this research team can help teachers in instructional planning. The advantages of these
web-based systems are: (a) helping inexperienced teachers visualize how a lesson works and providing them
useful information, (b) saving teachers’ time by providing well-developed lessons on specific topic areas, (c)
lowering teachers’ cognitive load through step-by-step guidance, (d) helping teachers understand the just-in-time
approach to learning how to plan using specific standards, and (e) preparing pre-service teachers to enter the
workforce with their existing knowledge, skills, and abilities (Koszalka, Breman, & Moore, 1999; Liu & Juang,
2002; Taylor & Janet, 1998).
Finally, web-based conferencing is viewed as a tool to help pre-service teachers to be reflective practitioners and
to give them sufficient knowledge to confront reform-oriented teaching practices (Devlin-Scherer & Daly, 2001).
According to Barnett et al. (2002), when pre-service teachers learn how to be good teachers in university-based
courses, web-based conferencing provides them with the opportunities to interact and reflect with in-service
teachers. In-service teachers participating in web-based conferencing with pre-service teachers become a
motivational factor for pre-service teachers and enhance discussions about the events in a real classroom context
(Barnett et al., 2002; Bonk, Malikowski, Angeli, East, 1998).
Web-based Cognitive Apprenticeship Model

This study refers to some cognitive apprenticeship models (Collins et al., 1989; LeGrand et al., 1993; Farmer et
al., 1992) and considers the characteristics of web-based learning to design a three-phase web-based cognitive
apprenticeship model (See Figure 1) supported by the Internet technologies mentioned above. Each phase is
named after the tasks of the expert teacher and pre-service teachers in that phase.
Modelling-Observing Scaffolding-Practicing Guiding-Generalizing

Expert teachers Expert teachers
demonstrating Expert teachers
in authentic contexts
Scaffolding
Interacting
Coaching
Articulating
Interacting
Learning through
Guding
modelling
Cognitive
Interacting
observing
Pre-service teachers Pre-service teachers

practicing generalizing
Pre-service teachers in authentic activities what they learned
Web-based
IPASS
multimedia
Web-based conferencing tools
Figure 1. Web-based Cognitive Apprenticeship Model

138
The “Modeling-Observing” phase. Through web-based conferencing, the expert teacher leads pre-service
teachers to observe his/her cognitive modeling displayed by web-based multimedia and guides them to
constructing initial conceptual models of how to write and implement an instructional plan. During this
procedure, web-based multimedia is used to demonstrate the cognitive modeling of an expert teacher in a real
classroom context by simultaneously presenting the instructional plan and the video case for the expert teacher’s
articulation of why and how to write the instructional plan, or the instructional plan and the video case about
teaching demos based on the instructional plan. Moreover, through issues posed in discussion forums, expert
teachers guide pre-service teachers to focus on key points of cognitive modeling displayed by web-based
multimedia and to share ideas with others. Finally, by interacting with expert teachers and peers in the internet
chat room, pre-service teachers construct their own personal conceptual models through sharing, debating,
modifying, and discussing.
The “Scaffolding-Practicing” phase. Supported by the expert teacher and network technologies, pre-service
teachers write, implement, review, and revise their own instructional plans considering the real classroom
context, and then articulate and reflect on the knowledge and thinking skills they employed. This process aims
not only to enhance pre-service teachers’ hands-on experiences and meta-cognitive (self-monitoring/self-
checking) abilities, but also to allow them to modify their conceptual model constructed in the first phase. When
writing instructional plans, IPASS is used to help them process both complex and trivial tasks and so focus their
efforts towards learning. Through online interaction, the expert teachers give guidance, feedback, and
suggestions regarding pre-service teacher performance. If necessary, expert teachers can lead pre-service
teachers back to the previous phase to observe again the cognitive modeling of the expert teachers displayed
through multimedia.
The “Guiding-Generalizing” phase. Through web-based conferencing, the expert teacher guides pre-service
teachers to generalize principles of instructional planning from the thinking skills and practical knowledge that
they had just learned. This phase aims to make the conceptual models of pre-service teachers more flexible and
useful than those which were constructed in the previous phases.
Research Questions
This work designed a course based on the web-based cognitive apprenticeship model mentioned above as an
experiment to examine whether this course benefits pre-service teachers’ learning of instructional planning, and
to explore possible causes of the results. There were two sets of research questions as follows:
¾ Compared with the traditional course, can this web-based course effectively improve pre-service teachers’
instructional planning performance? And why?
¾ Compared with the traditional course, can this web-based course effectively promote pre-service teachers’
positive attitudes toward instructional planning? And why?
Methods
Design
In this study, a field experiment was conducted in the real-life setting. The study used a 2×2 mixed design, with
“Course” as a between-subjects factor (web-based and traditional) and “Measuring phase” (pre-test and post-test)
as a within-subjects factor. Measures were taken of pre-service teachers’ instructional planning performance and
their attitudes toward instructional planning. The subjects were randomly assigned to either the web-based
course group or the traditional course group.
Courses (7 weeks)
The course based on the web-based cognitive apprenticeship model, web-based course for short, was taken by
the web-based course group. Details of the web-based course are shown in Table 1. This course also included
three phases: Modeling-Observing (2 weeks), Scaffolding-Practicing (4 weeks) and Guiding-Generalizing (1
week). Aside from performing a teaching demonstration in the expert teachers’ elementary school class in the
sixth week of the course, pre-service teachers learned through the web for the remainder of the time. In addition,
there were two types of online discussion for each phase: asynchronous communication and synchronous
communication. Asynchronous communication refers to the expert teacher posting discussion topics on the
139
discussion forum, and inviting pre-service teachers to join together in the discussion. Synchronous
communication refers to expert teachers and pre-service teachers using chat rooms for discussion for two hours
each week at a fixed time, normally at night.
Table 1. Contents of the Web-based Course

Phase Week Assignment and Activity Contents
Modeling- 1st Assignment:
Observing The pre-service teachers read the specified chapters in the textbook assigned by the
2nd teacher educator.
The pre-service teachers observe the web-based multimedia and post their reactions
and ideas on the discussion forum.
Learning activity: Expert teachers lead the pre-service teachers in discussing the web-
based multimedia content.
Scaffolding- 3rd Instructional planning
Practicing Assignment: Pre-service teachers, with the help of the IPASS, design an instructional
4th plan that will be implemented in the 5th and 6th week.
Learning activity: Expert teachers, using the IPASS, get to know each pre-service
teacher’s state of instructional planning; and using electronic conferencing, lead the
pre-service teachers to discuss and modify their own instructional plan.
5th Teaching demo
The pre-service teachers use the modified instructional plan for the teaching demo.
6th The pre-service teachers observe actual teaching demos of the other pre-service
teachers.
Expert teachers observe the pre-service teacher demo and gives on-the-spot
suggestions.
Guiding- 7th Assignment: The pre-service teachers record their reflections about the instructional
Generalizing plan design and practice.
Learning activity: The expert teachers use electronic conferencing to lead the pre-
service teachers to generalize principles of instructional planning from the thinking
skills and the practical knowledge that they learned in the course.
The course taken by the traditional course group is often employed in teacher education and it includs four
phases: Preparing (2 weeks), Designing (2 weeks), Implementing (2 weeks) and Reflecting (1 week). Details of
the traditional course are displayed in Table 2. In the preparation phase of this course, pre-service teachers in the
university learn the knowledge and skills related to instructional planning. Moreover, they watch the expert
teachers’ teaching demo in the classroom together (once). During the design phase, the pre-service teachers plan
their own teaching demonstrations. In the implementation phase, the pre-service teacher, based on his/her own
written instructional plan, does the actual teaching. Finally, in the reflecting phase, the pre-service teachers
report on their own instructional plan and micro-teaching. The expert teacher reads the report and give
suggestions. For each phase, if problems occur during the trials, pre-service teachers can query the expert
teachers using face-to-face discussion or e-mail.
Table 2. Contents of Traditional Course

Phase Week Assignment and Activity Contents
Preparing 1st Assignment:
The pre-service teachers read the specified chapters in the textbook assigned by the
2nd qualified educator.
After the pre-service teacher observes the expert teacher’s teaching demo, the pre-
service teacher writes a report.
Learning activity: Pre-service teachers (12 persons) observe the expert teachers’
teaching demo and discuss it with the expert teachers (2 hours).
Designing 3rd Assignment: Pre-service teachers use computers to design an instructional plan that will
be implemented in the 5th and 6th week..
4th Learning activity: The pre-service teachers, using actual visits or e-mail, ask questions
of the expert teachers and give them the completed instructional plan. The expert
teachers give the pre-service teachers suggestions on how to modify their instructional
plans.
140
Implementing 5th Teaching demo
The pre-service teachers use the modified instructional plan for the demo.
6th The pre-service teachers observe actual teaching demos of the other pre-service
teachers.
Expert teachers observe the pre-service teacher demo and give on-the-spot suggestions.
Reflecting 7th Assignment: The pre-service teachers write their own reflections about the instructional
plan they designed and the results of the demo.
Learning activity:
Expert teachers read the pre-service teacher’s reactions and give suggestions.
Participants
Expert teachers. Four expert teachers were the major instructors to teach and guide both the web-based course
and the traditional course. They had similar backgrounds as current primary school teachers, with a master’s
degree or being enrolled in a master’s degree program, with significant experience in helping pre-service
teachers, with experience in authoring primary school textbooks, and with network teaching experience.
Subjects. One junior class with 24 students in the elementary education department of a teachers college in
Taiwan was chosen and randomly assigned into two groups. The web-based course group had one male and 11
females, while the traditional course group had three males and nine females. The average age of each group was
21. Before joining the research, all subjects had the experience of writing two instructional plans in other
courses.
Tools for Collecting Data
Instructional planning performance (rating scales). This tool consisted of six sub-areas: (a) activity (3 items),
(b) goals and objectives (2 items), (c) method and procedure (6 items), (d) material resources (7 items), (e)
assessment (4 items) and (f) holistic (6 items). The first five sub-areas were linked with corresponding major
components of instructional planning and the last one was related to the reasonability and effectiveness of the
overall plan. Each item was a statement about the features of a good instructional plan. Each subject’s
instructional planning performance was graded according to whether the plan that the subject designed matched
each statement (using a five-point Likert scale, where 1= poor and 5=excellent). The overall performance score
was the average of the total subscores (28 items).
Before conducting the experiment, two reliability analyses regarding this scale were executed: inter-rater
reliability and internal-consistency. For analyzing inter-rater reliability, two expert teachers used this scale on ten
instructional plans. The correlation coefficients of the 28 items ranged from .84 to .95, with an average of .87.
The results showed that the items in this scale are clear and definite, giving good scoring agreement between
raters. Moreover, for analyzing internal-consistency, 36 instructional plans were graded by one expert teacher
based on this scale and Cronbach’s α, an index of homogeneity (Gregory, 1996, p. 96), was calculated. The
analytical results showed that the Cronbach’s α of six sub-areas ranged from .65 to .90, indicating that the scale
had acceptable internal-consistency.
During the experiment, one expert teacher was responsible for using this rating scale to evaluate all instructional
plans that the subjects designed in the pre-test and the post-test. During the evaluation process, all instructional
plans used the same format but without the designer’s name, so the grader did not know whether the
instructional plans were designed by the web-based course group or by the traditional course group.
Attitudes toward instructional planning (self-reporting scale). This tool consisted of three parts:
identification with the importance of instructional planning (9 items), identification with the functions of
instructional planning (12 items), and willingness to design and implement instructional plans in the future (7
items). Each item was a statement. Subjects rated each item according to the degree of their agreement with the
statement with a score from 1–4, where 1=strongly disagree, 2=disagree, 3=agree, and 4=strongly agree. The
reason why this tool did not use the five-point scale but rather the four-point scale was to prevent the subjects
from answering “no comment” when filling up the scale. The overall attitude score was the average of the total
subscores (28 items). Before the experiment, reliability analysis for measuring internal-consistency was
executed, in which 36 pre-service teachers who were not participants of the experiment answered the scale. The
141
analytical results showed that the Cronbach’s α for each part ranged from .94 to .97, indicating that the scale had
good internal-consistency.
Other tools. Web-conferencing tools, the chat room and the discussion forum for the web-based course group
and the e-mail for the traditional course group, were used for collecting the interaction text between pre-service
teachers and expert teachers or between peers. The interview questionnaire in a semi-structured form was used to
collect the participants’ viewpoints. All of these data were collected to interpret the experimental results.
Procedures
Preparing materials. Before the experiment, the researcher and expert teachers met regularly to discuss how the
web-based course and the traditional course would be carried out and to produce the instructional materials,
including multimedia and assignments.
Practicing web-based technologies. One week before the experiment, a workshop (six hours) was carried out so
that the participants of the web-based course could familiarize themselves with each web-based technology that
will be used in the experiment.
Pre-testing. Before the experiment all subjects during the class answered the scale regarding attitudes toward
instructional planning. Moreover, the researcher collected the instructional plans that subjects had designed
before the experiment and the expert teacher rated these plans in accordance with the rating scale for
instructional planning performance. These scores were used as pre-testing scores.
Proceeding courses (interventions). Both courses, web-based and traditional, were carried out in seven weeks.
The actual procedure and activities for the two courses are shown in Tables 1 and 2.
Post-testing. When the courses were concluded (7th week), all subjects answered the same attitudes scale again.
During the next two weeks (8th and 9th week), each subject wrote a new instructional plan. These plans were
rated using the scale for instructional planning performance. These scores were used as post-testing scores.
Results
The 2×2 mixed ANOVA was executed as follows. Firstly, the F test was performed to determine the interaction
between “Group” and “Measuring phase”. When the significant interaction of “Group” and “Measuring phase”
was confirmed, four main effect analyses were executed. These main effect analyses included the following:
comparing the pre-test means between groups, comparing the post-test means between groups, comparing means
between pre- and post-test within the web-based course group, and comparing means between pre- and post-test
within the traditional course group.
Instructional Planning Performance
Instructional planning performance of the subjects was analyzed with the overall performance score, the average
of total subscores, as the dependent variable. The analytical results revealed that significant “group” ×
“measuring phase” was present (F(1,22)=11.87, MSE=.3, p<.05) (see Figure 2). Four main effect analyses
showed (1) the pre-test means between the web-based course group (M=2.88, SD=.65) and the traditional course
group (M=2.96, SD=.82) were not significantly different (F(1,44)=0.09, MSE=.48, p>.05); (2) the difference
between post-test means of the web-based course group (M=4.44, SD=.40) and of the traditional course group
(M=3.44, SD=.49) were significant (F(1,44)=12.57, MSE=.48, p<.05); (3) the means between pre-test (M=2.88,
SD=.65) and post-test (M=4.44. SD=.40) for the web-based course group were significantly different
(F(1,22)=49.83, MSE=.30, p<.05), and (4) the means between pre-test (M=2.96, SD=.82) and post-test (M=3.44,
SD=.78) for the traditional course group were statistically different (F(1,22)=4.78, MSE=.3, p<.05).
142
5
4.5
4
3.5
3
Score
2.5
2
1.5
1 web-based course group
0.5 traditional course group
0
pre-test post-test
Measuring phase
Figure 2. Interaction of “Group” and “Measuring Phase” on “Instructional Planning Performance”
Figure 2 shows that before the two courses commenced, the two groups had similar and medium level
instructional planning performance (full score is 5 points; the pre-test average score of the web-based course
group was 2.88; the pre-test average score of the traditional course group was 2.96). After seven weeks of
instruction, the instructional planning performance of both groups, although showing significant improvement,
indicated that the web-based course group clearly progressed more rapidly than the traditional course group (the
post-test average score of the web-based course group was 4.44; the post-test average score of the traditional
course group was 3.44).
Attitudes toward Instructional Planning
Subjects’ attitudes toward instructional planning were analyzed with the overall attitude score, the average of
total subscores, as the dependent variable. According to analytical results, significant “group” × “measuring
phase” was present (F(1,12)=6.82, MSE=.13, p<.05) (see Figure 3). The four main effect analyses showed: (1)
the pre-test means between the web-based course group (M=2.57, SD=.59) and the traditional course group
(M=2.62 SD=.44) were not significantly different (F(1,44)=0.06, MSE=.26, p>.05); (2) the post-test means of
the web-based course group (M=3.33, SD=.38) and the traditional course group (M=2.83, SD=.59) were
significantly different (F(1,44)=5.65, MSE=.26, p<.05); (3) the means between pre-test (M=2.57, SD=.59) and
post-test (M=3.33, SD=.38) for the web-based course group were significantly different (F(1,22)=22.66,
MSE=.13, p<.05), and (4) the means between pre-test (M=2.62 SD=.44) and post-test (M=2.83, SD=.59) for the
traditional course group were not significantly different (F(1,22)=2.16, MSE=.13, p>.05).
4
3.5
3
2.5
Score
2
1.5
1 web-based course group
0.5 traditional course group
0
pre-test post-test
Measuring phase
Figure 3. Interaction of “Group” and “Measuring Phase” on “Attitude toward Instructional Planning”
In Figure 3, a higher average score means more positive attitude toward instructional planning. Before taking the
courses, the attitude of the two groups regarding instructional planning were similar and tend to be positive (full
143
score is 4 points; the pre-test average score of the web-based course group was 2.57; the pre-test average score of
the traditional course group was 2.62). After seven weeks of the course, the attitude of the traditional course
group regarding instructional planning was not significantly altered (the post-test average score of the traditional
course group was 2.83); the attitude of the web-based course group regarding instructional planning became
significantly more positive (the post-test average score of the web-based course group was 3.33). Since the
attitude of the web-based course group became more positive, the post-test score of this group was therefore
higher than that of the traditional course group.
Discussion
Basically, the web-based course group and the traditional course group had the same teachers (four expert
teachers) and the same class duration (seven weeks), as well as similar assignments and activity contents. The
biggest difference between the two groups was in how and where the learning activities were carried out. The
web-based course group was supported by web technologies where the expert teacher led the pre-service
teachers in cognitive apprenticeship learning activities. The learning activities for the traditional group, on the
other hand, consisted mainly of pre-service teachers going to the classroom of the expert teachers for observation
and discussion. According to the analytical results, the web-based course group and the traditional course group
had the same performance and attitudes about the instructional planning before the experiment was carried out.
However, after having taken different courses, both groups had significant progress in the instructional planning
performance, but only the web-based course group had a significant improvement in terms of attitude. Moreover,
the web-based course group was significantly better than the traditional course group not only in having positive
attitudes toward instructional planning but also in the instructional planning performance.
In summary, the course that was based on web-based cognitive apprenticeship model have the potential to more
effectively enhance pre-service teachers’ performance and attitudes towards instructional planning than the
traditional course. The following sections explore and discuss the possible reasons for these results based on the
qualitative data.
Reasons That the Web-based Course More Effectively Improved Pre-service Teachers’ Performance
The first possible reason is that the web-based multimedia provided clear and effective cognitive modeling
as a response to the needs of the pre-service teachers. Cognitive modeling is seen as the heart of the cognitive
apprenticeship model (LeGrand et al., 1993; Farmer et al., 1992). Colins et al. (1989) point out that effective
cognitive modeling refers to the expert who can clearly externalize his/her practical knowledge and thinking
skills which are used to deal with complex tasks, according to the learner’s needs. Although some studies
indicate that web-based multimedia can overcome time limits so that the learner can repeatedly observe the
cognitive modeling of expert teachers (e.g. Barnett, et al., 2002), there must be proper design to make the web-
based multimedia effectively show the cognitive modeling.
In order for web-based multimedia to effectively show the cognitive modeling of expert teachers in an actual
environment, the researcher first collected the expert teachers’ instructional plans and made a video of their
instructions following those plans, and utilized these instructional plans and teaching demo videos to lead the
expert teachers to articulate their obscure ideas and tacit knowledge when writing and implementing these plans.
The entire interview process was also recorded by video. In addition, the instructional plan samples, teaching
demo videos, and the interview videos were edited into an instructional multimedia regarding the requirements
of the pre-service teachers based on the teacher educators’ experiences. This process allowed the rich practical
knowledge and thinking skills of the expert teachers regarding instructional planning to be in an external form
that met the needs of the pre-service teachers. An expert teacher said during the interview after the experiment
that “Previously, I did not really clarify in depth what I think and how I think during writing and implementing
of instructional plans. When someone touched on these issues, I often responded in heuristics and without
careful consideration . . . These processes [of interviewing] made me deliberate, reflect, and articulate these
issues according to what I do [displayed by the instructional plan samples and teaching demo videos]. I think
these processes [of interviewing] also could facilitate my own professional development.” A pre-service teacher
of the web-course group noted during the interview after the experiment that “the instructional multimedia made
me realize what really happens during expert teacher writing and implementing an instructional plan and
understand what the key points are at the same time.”
144
In contrast, the traditional course group could observe the expert teacher’s demonstration in the classroom and
also discuss directly with them. Nevertheless, since the expert teachers do not have an ordered way of direction
or enough time to clarify and organize their own ideas, during the discussions they are less able to systematically
explain their practical knowledge and thinking skills used in designing and implementing instructional plans. An
expert teacher said in an interview regarding the live teaching demo, “I don’t like a group of people observing
how I teach in class…my students and I were affected. We easily get nervous and distracted. [After the teaching
demo,] I can’t think straight during discussion [with pre-service teachers]…You know, I just came from a
scuffle. It’s difficult to calm down and slowly explain my ideas [during the discussion]. In reality, I think that I
am not even clear about my ideas…The pre-service teachers raised some questions. These questions were few
but very strange. Some questions had to deal with details such as the background of students. Since the pre-
service teachers do not understand this class, and the time [that can be used to answer] is short, I cannot just
explain things right there…”
The next possible reason is that the integrated application of web-based multimedia and Internet
conferences benefited the pre-service teachers’ learning. First of all, this integrated application could
stimulate the participants’ discussion. For instance, the frequently discussed issues in the “Modeling-Observing”
phase, not only by the expert teachers initiating but also by the pre-service teachers, were related to the matters
displayed by web-based multimedia, which facilitated a lively discussion. Barnett et al. (2002) present similar
results, showing that a combination of online multimedia and online conferences supported “a number of
discussion threads,” in which learners explore their ideas about learning topics (p. 310). Additionally, the
integrated application of web-based multimedia and conferencing allowed expert teachers, based on their
instructional considerations, to have flexibility in guiding the pre-service teachers to review of specific sections
of the multimedia material and then to discuss them. For example, in the “Scaffolding-Practicing” phase, a pre-
service teacher during online interaction asked the expert teacher “how to design an activity to stimulate the
students’ motivation to learn.” The expert teacher asked pre-service teachers to “first observe the expert
teacher’s teaching method from the multimedia and then discuss it.” This approach easily gave the discussion a
focus and allowed the pre-service teacher, according to his/her own learning progress, to repeatedly perceive and
interpret the expert teachers’ cognitive modeling.
In contrast to the web-based group, after the traditional course group watched the teaching demo of the expert
teacher in a classroom setting, there was less chance to interact with the expert teacher on the spot. Once pre-
service teachers had left the classroom, the traditional course group resorted to sending e-mail to the expert
teacher. However, because neither party was able to focus on the same details of the teaching demo, discussions
were generally shallow.
Moreover, the web-based course seemed to offer the pre-service teachers more sophisticated and timely
support than that from the traditional course. For example, through online discussion in the “Modeling-
Observing” phase, the expert teachers guided pre-service teachers to co-explore the knowledge and thinking
skills that the expert teacher used in the instructional planning, as shown in the multimedia. In this way, pre-
service teachers constructed comprehensive conceptual models as they were starting to learn. One subject of the
web-based course group reviewed his learning experiences, “The multimedia could display what they [expert
teachers] think and do [during writing and implement instructional plans]; however, it is expert teachers’
guidance in the forum that helped me build a framework to integrate these many details.” Another pre-service
teacher said, “the [online] discussion provided me with multiple perspectives that made me reflect and modify
my ideas.”
The performance support system, IPASS, also gave timely assistance. For instance, in the “Scaffolding-
Practicing” phase, IPASS offered several types of support so that the pre-service teachers could concentrate on
learning. The subjects of the web-based course group pointed out that using IPASS to plan lessons could save a
lot of time in dealing with trivial things. It can lead them, step-by-step, to design instructional plans and support
them to find rich and useful instructional resources. The results of this study are similar to that of Liu & Jung
(2002), who surveyed in-service teachers’ opinions about IPASS.
In contrast with the web-based course group, the pre-service teachers in the traditional course group had to write
their own complete instructional plan after observing the demo given by the expert teacher. Although the pre-
service teachers in the traditional course group can choose actual visits or use e-mail to ask the expert teachers
questions on how to design instructional plans, most pre-service teachers preferred to use e-mail. They thought
that actual visits to expert teachers could not be done because of scheduling problems and discussions would be
interrupted by sudden events. For example, one pre-service teacher said during the interview, “I first used e-mail
to make an appointment with the [expert] teacher. However, I discovered that for most of the time slots, it’s
145
difficult to find a common time… When we met together, after talking for less than ten minutes, the [expert]
teacher’s colleague visited her to talk about official business.” In addition, when pre-service teachers used e-mail
to ask questions of expert teachers (for example, what is the format of the instructional plan), they often did not
touch on key points since pre-service teachers lacked practical experience. Given this situation, the traditional
course group had limited opportunities to revise their conceptual model to plan lessons.
Finally, the last phase of the web-based course, “Guiding-Generalizing”, helped to elaborate and extend
the conceptual model of the pre-service teachers. Faced with complicated and ill-structured instructional
planning, even if the pre-service teachers had constructed and modified their conceptual models in the previous
phases, they still did not really know how to apply the skills and knowledge that they had acquired. Through the
discussion in the forum, expert teachers assisted pre-service teachers to put what they had learned into practical
principles. Expert teachers often brought up certain questions to extend pre-service teachers’ thinking, such as
“If you are faced with certain . . . situations, what do you do?” In so doing, pre-service teachers, when faced with
different classroom situations and teaching contents, could use these principles to plan lessons. Although the
traditional course group had to do assignments at the same time, most pre-service teachers concentrated on
reflecting on the effects of their own instructional planning but not on the broad rules which can be applied in the
future.
Reasons That the Web-based Course More effectively Enhanced Pre-service Teachers’ Attitudes
As described above, the web-based multimedia was designed to display the expert teachers’ ideas during
designing, implementing, and reflecting on instructional plans. Through repeatedly observing the web-based
multimedia and online interactions with expert teachers and peers, the pre-service teachers of the web-based
course group had many chances to experience the complexity of actual teaching situations and to consider the
importance of the teacher designing, implementing, and reflecting on instructional plans as helping improve
teaching. During the interview, one pre-service teacher said “although the textbooks and teacher educators had
told us how important the instructional planning is, not until watching multimedia and interacting with expert
teachers did I really sense the significance of instructional planning in actual classroom context.” In contrast, the
pre-service teachers in the traditional course group had a chance to observe the expert teachers’ teaching demo in
the classroom and discuss with the expert teacher face-to-face. However, because the expert teachers cannot
immediately and systematically clarify their ideas, pre-service teachers may have few chances to obtain the
complete image about the relationship between instructional planning and instruction improvement. Moreover,
after leaving the classroom, the pre-service teachers of the traditional course group may not be able to repeatedly
explore and think of the importance of the instructional plan. Thus, the traditional course may not effectively
improve the positive attitude of the pre-service teachers regarding the instructional plan.
Furthermore, dialogues similar to the following excerpts often appeared in the online discussions: One pre-
service teacher said, “why does the student’s multiple intelligence have to be considered when writing an
instructional plan? It’s too bothersome;” and one expert teacher responded, “that is an interesting question,
there’s no harm in trying….” This extended discussion arising from the pre-service teacher’s question provided
them a way of exploring what points to keep in mind during the instructional planning and also allowed them to
consider the reasons for using a particular method. Subsequently, the implementation of the instructional plan
allowed the pre-service teachers to further confirm these reasons. In this way, pre-service teachers progressively
increased their identification with the functions of instructional planning. The traditional course group, in
contrast, lacked such an interactive procedure.
According to Riesbeck (1996), electronic conferences allow pre-service teachers to construct new knowledge
with peers under the support of practitioners or experts. In this study, through web-based conferencing, expert
teachers gave appropriate and progressive support. Thus the pre-service teachers in the web-based course group
have a greater likelihood of success than the traditional course group. In the last phase (Guiding-Generalizing),
the expert teachers guided pre-service teachers to summarize practical principles for later instruction planning.
The above learning processes and learning results enabled the pre-service teachers in the web-based group to be
more willing to make further instructional plans.
Conclusions and Recommendations

In the general effort to enhance teacher education by finding means to overcome some of the limitations of
current university-based teacher education programs, the present study constructed a web-based cognitive
146
apprenticeship model that integrates cognitive apprenticeship theory, expert teachers, and web technologies
(web-based multimedia, performance support system, and electronic conference) to guide pre-service teachers to
learn instructional planning. The experimental results showed that the course based on this model is significantly
more effective than the course based on current university-based programs to improve pre-service teachers’
performance and attitudes regarding instructional planning.
In summary, compared with the teaching method conventionally used in teacher education programs, the web-
based cognitive apprenticeship model has the following features which can enhance the instructional planning
performance of pre-service teachers:
¾ Well designed web-based multimedia can provide clear and useful cognitive modeling that can help pre-
service teachers to observe and understand what and how expert teachers do and think when designing and
implementing instruction plans based on actual situations.
¾ Integrated application of online multimedia and online conferencing provides an effective way to stimulate
participants’ discussion and allows expert teachers to flexibly guide the pre-service teachers to review the
expert teachers’ cognitive modeling.
¾ Expert teachers and web technologies offer opportune and timely assistance to support pre-service teachers
to construct, modify, and elaborate their conceptual models.
¾ Pre-service teachers have opportunities to elaborate and to extend their conceptual models under the expert
teacher’s guidance to conclude and arrange the practical principles of instructional planning that can be
applied in the future.
In addition, the web-based cognitive apprenticeship model has the following characteristics, which give the pre-
service teachers more positive attitudes toward instructional planning:
¾ Pre-service teachers have many chances to explore the reasons for the instructional planning done by expert
teachers and compare the results with actual instructional plan implementation.
¾ Through discussions with expert teachers, pre-service teachers can re-consider what matters should be noted
and their motivations when planning instruction.
¾ Through progress guidance by the expert teachers and support by web-based technologies, pre-service
teachers can become more confident when writing instructional plans.
Based on the results of this study, we recommend the following aspects for future research. In the first place,
considering that the field experiment and the small sample size may affect the validity of this study’s results,
more studies with larger size samples are recommended. Moreover, even if the current web-based cognitive
apprenticeship model is only applied to learning how to plan lessons, its characteristics seem to permit it to be
used in other practical teacher education courses that teach pre-service teachers to deal with complex and ill-
structured situations, such as class management. Future study can explore its suitability. Furthermore, the results
of this study reveal that the interaction between pre-service teachers and expert teachers significantly affects
web-based learning and teaching. The large amount of web-interactive text collected by this study is currently
being analyzed, and the research results can help researchers in understanding the relationship between the types
of online interaction and learning results. Finally, the research results imply that teacher educators, pre-service
teachers, and in-service teachers can be closely linked with the proper use of network technologies. Future
research can investigate the possible relationships or cooperation among the three.
Acknowledgment
The author would like to thank the National Science Council of Taiwan for financially supporting this research
under Contract No. NSC-91-2520-S-008-009. The author would also like to thank the teacher educators, the
expert teachers and the pre-service teachers who participated in the experimental study. Finally, the author would
like to thank the editor of ET&S and the anonymous reviewers of this paper for their kind assistance and helpful
suggestions.
References
Arnold, V. D. (1988). Planning for effective instruction. Teacher Education, 24 (3), 10-12.
Barnett, M., Keating, T., Harwood, W., & Saam, J. (2002). Using emerging technologies to help bridge the gap
between university theory and classroom practice: Challenges and successes. School Science and Mathematics,
102 (6), 299-313.
147
Barron, L. C., & Goldman, E. S. (1994). Integrating technology with teacher preparation. In Means, B. (Ed.)
Technology and education reform, San Francisco: Jossey-Bass, 81-110.
Beck, C., & Kosnik, C. (2002). The importance of the university campus program in preservice teacher
education. Journal of Teacher Education, 53 (5), 420-432.
Bonk, J. C., Malikowski, S., Angeli, C., & East, J. (1998). Web-based case conferencing for prospective teacher
education: Electronic discourse from the field. Journal of Educational Computing Research, 19 (3), 269-306.
Borko, H., & Niles, J. A. (1987). Descriptions of teacher planning: Ideas for teachers and researchers. In
Richardson-Koehler, V. (Ed.), Educators’ handbook: A research perspective, New York: Longman, 167-187.
Brown, J. S., Collin, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational
Researcher, 18 (1), 32-42.
Byra, M., & Coulon, S. C. (1994). The effect of planning on the instructional behaviors of preservice teachers.
Journal of Teaching in Physical Education,13 (3), 123-139.
Clark, C. M., & Peterson, P. L. (1986). Teachers’ thought processes. In Wittrock, M. C. (Ed.), Handbook of
research on teaching, New York: MacMillan Publishing Company, 255-296.
Clark, C. M., & Yinger, R. J. (1987). Teacher planning. In Berliner, D. C. & Rosenshine B. (Eds.), Talks to
teachers, New York: Lane Akers, 84-103.
Clark, C. M. (1988). Asking right questions about teacher preparation: Contributions of research on teacher
thinking. Educational Research, 17 (2), 5-12.
Clark, C. M, & Dunn, S. (1991). Second-generation research on teacher planning. In Waxman, H. C. & Walberg,
H. J. (Eds.), Effective teaching: Current research, Berkeley, CA: McCuthan, 183-201.
Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: teaching the crafts of reading,
writing, and Mathematics. In Resnick, L. B. (Ed.), Knowing, learning, and instruction: Essays in honor of
Robert Glaser, Hillsdale, N. J.: Erlbaum, 453-494.
Devlin-Scherer, R., & Daly, J. (2001). Living in the present tense: Student teaching telecommunications connect
theory and practice. Journal of Technology and Teacher Education, 9 (4), 617-634.
Farmer, J. A., Jr., Buckmaster, A., & LeGrand, B. (1992). Cognitive apprenticeship: Implications for continuing
professional education. New Directions for Adult and Continuing Education, 55, 41-49.
Gregory, R. J. (1996). Psychological testing: History, principles, and applications (2nd Ed.), Boston, MA: Allyn
and Bacon.
Hoogveld, A. W. M., Paas, F., Jochems, W. M. G., & Van Merriënboer, J. J. G. (2002). Exploring teachers'
instructional design practices: implications for improving teacher training. Instructional Science, 30, 291-305.
Kagan, D., & Tippins, D. J. (1992). How U.S. preservice teachers "read" classroom performances. Journal of
Education in Teaching, 18 (2), 149-158.
Koszalka, T. A., Breman, J., & Moore, M. K. (1999). Sharing lesson planning over the World Wide Web:
Important components. Journal of Education and Information Technologies, 4 (2), 143-151.
Lambdin, D., Duffy, T., & Moore, J. (1997). Using an interactive information system to expand preservice
teachers’ visions of effective mathematics teaching. Journal of Technology and Education, 5, 171-202.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation, New York: Cambridge
University Press.
148
Lave, J. (1993). Situating learning in communities of practice. In Resnick, L. B., Levine, J. M., & Teasley, S. D.
(Eds.), Perspectives on socially shared cognition, Washington, DC: American Psychological Association, 17-36.
LeGrand, B., Farmer, J. A., Jr., & Buckmaster, A. (1993). Cognitive apprenticeship approach to help adults
learn. New Directions for Adult and Continuing Education, 59, 69-78.
Liu, T. C., & Juang, Y. R. (2002). IPASS -Teacher’s knowledge management platform for teachers’ professional
development. Paper presented at the International Conference on Engineering Education, August 18-22, 2002,
Manchester, England.
McCutcheon, G. (1980). How do elementary school teachers plan? The nature of planning and influences on it.
The Elementary School Journal, 81 (1), 4-23.
McLellan, H. (1993). Evaluation in a situated learning environment. Educational Technology, 33 (3), 39-45.
Neely, A. M. (1986). Planning and problem solving in teacher education. Journal of Teacher Education, 37 (3),
29-33.
Reeves, T. C. (1993). Evaluating interactive multimedia. In Gayeski, D. M. (Ed.) Multimedia for learning:
Development, application, evaluation, Englewood Cliffs, NJ: Educational Technology Publication, 97-112.
Reiser, R. A., & Dick, W. (1996). Instructional planning: A guide for teachers (2nd Ed.), Boston: Allyn and
Bacon.
Riesbeck, C. K. (1996). Case-based teaching and constructivism: Carpenters and tools. In Wilson, B. G. (Ed.),
Constructivist learning environments: Case studies in instructional design, Englewood Cliffs, NJ: Educational
Technology Publications, 49-61.
Schrader, P. G., Leu, D. J., Kinzer, C. K., Ataya, R., Teale, W. H., Labbo, L. D., & Cammack, D. (2003). Using
Internet delivered video cases, to support pre-service teachers' understanding of effective early literacy
instruction: An exploratory study. Instructional Science, 31 (4-5), 317-340.
Taylor, N. P., & Pilcher, J. K. (1998). STEPS: An EPSS tool for instructional planning, East Lansing, MI:
National Center for Research on Teacher Learning, ERIC Document Reproduction Service No. ED423853.
149
Kinzie, M. B., Whitaker, S. D., & Hofer, M. J. (2005). Instructional Uses of Instant Messaging (IM) During Classroom
Lectures. Educational Technology & Society, 8 (2), 150-160.
Instructional Uses of Instant Messaging (IM) During Classroom Lectures

Mable B. Kinzie
Department of Leadership, Foundations & Policy
Curry School of Education
University of Virginia
Charlottesville, VA 22904-4277 USA
kinzie@virgina.edu
Stephen D. Whitaker
Department of Human Services
Curry School of Education
University of Virginia
Charlottesville, VA 22904-4270 USA
whitaker@virginia.edu
Mark J. Hofer
Department of Reading, special Education, and Instructional Technology
Towson University
Towson, MD 21252 USA
mhofer@towson.edu
ABSTRACT
Can “Information Age” learners effectively multi-task in the classroom? Can synchronous classroom
activities be designed around conceptually related tasks, to encourage deeper processing and greater
learning of classroom content? This research was undertaken to begin to address these questions. In this
study, we explored the use of instructionally-related instant messaging (IM) discussions during
undergraduate university lectures. Over the course of three weeks, students practiced with and then
employed hand-held computers for brief, synchronous class discussions in response to assigned questions
related to the lectures. Students were observed during these sessions, and students and the instructors were
interviewed separately afterwards. The contents of students’ discussions indicate that they were capable of
engaging in on-task discussions and of expressing opinions and exploring instructionally relevant topics.
However, even though students routinely multi-task in classrooms as they attend to lectures, process the
contents, and record notes for later study, both students and the instructors expressed some discomfort with
discussion occurring synchronously with classroom lectures. In this paper, we describe student discussion
behavior, and reflect on possible applications of multiple sequential (rather than simultaneous) tasks during
classroom instruction.
Keywords
Computer-mediated communication, Instant messaging (IM), Interactive learning environments, Multi-tasking,
Classroom discussion
Introduction
Frand (2000) suggests that today’s “Information Age” students have come to expect constant connectivity and
real-time dialogue from anywhere, and that they are used to multi-tasking (performing multiple tasks
simultaneously), particularly multi-tasking involving chat or instant messaging (IM) sessions. Use of IM is a
preferred on-line activity for teens, with 76% of high school-age respondents to a recent NetDay survey (2004)
reporting use of at least one IM screen name. Findings presented in a recent Pew Report mirror these numbers:
74% of teens use IM, with 69% using IM at least several times a week (Lenhart, Rainie, & Lewis, 2001). Teens
use IM for homework support as well as for socializing, and often engage in multiple simultaneous
conversations (Grinter & Palen, 2002). As the information age mindset becomes the norm among our students,
we have the opportunity to adapt our teaching and learning pedagogies to fit students’ characteristics and help to
encourage an active community of learners (Frand, 2000). Can the connectivity and real-time discussion
facilitated by Chat or IM sessions and the multi-tasking capabilities of students contribute to new learning
opportunities in the classroom? We designed and undertook a research inquiry to begin this exploration.
Past research on “divided attention” in cognitive psychology suggests that splitting one’s attention between
simultaneous tasks results in decreased performance on one or both tasks (Hembrooke & Gay, 2003). While the
150
tasks employed in this line of research generally appear to be unrelated (e.g, reading a list of words while
listening to a string of digits), Hembrooke and Gay obtained similar results when their range of multi-tasking
included content-related web browsing in addition to un-related web browsing, e-mail, and IM. However, if
student multi-tasking was conceptually related and specifically aimed at encouraging reflection on the lecture
contents, it is possible that deeper processing and greater learning of classroom content might occur, in the same
way that “dual processing” (e.g., employing both verbal and visual media) (Clark & Paivio, 1991) and active
learning (Grabinger, 1996; CTGV, 1992) can positively influence learning.
In the typical college classroom, lecture prevails as an instructional medium. While class discussion can
encourage thoughtful reflection and deeper student processing, often there isn’t time for everyone in the class to
participate, and there certainly isn’t time to discuss everything that could productively be explored. In larger
classes, separate “discussion sections” meet to perform at least part of this function. With the advent of on-line
learning technologies, some of these discussions are moving to asynchronous venues such as threaded discussion
boards, and even to synchronous chat-based discussions. A downside to these discussions, at least those that are
intended to build on a classroom lecture, demonstration, or other activity, is that are that they are generally
removed in time and place from the classroom events. Concepts, topics, and questions that students might want
to explore can be lost in the process.
Synchronous interactions while still in the learning environment have clear appeal for students. For instance,
Guernsey (2003) describes college students at the University of Maryland who requested IM access during
lectures. More than a dozen students participated at a time, and they used the opportunity to ask questions about
the lecture, to support or refute points being made, and to provide relevant links to their classmates.
We were curious about students’ ability to engage in meaningful discussions that took place concurrently with
the instructor’s presentation. We wanted to encourage students to more deeply process the contents of what was
presented to them by engaging in synchronous on-line discussions with assigned chat partners.
Methods
Participants and Procedures
The setting for this research was a university course for preservice teachers on the use of instructional
technology. The class was held for three separate sections each week; students in all sections were offered the
chance to participate; 36 students elected to participate and 3 students declined (students declining participation
engaged in the same activities but their chat participation was not analyzed and they were not observed during
class sessions). A preliminary survey of the students indicated that 76% of the students used IM every day, while
another 11% used IM weekly (5% reported only one or two uses of IM and 8% had never used IM). Of the
students who used IM, session lengths ran from 5 minutes to “all the time, literally.” Conversely, no students
reported daily use of a hand-held computer or personal digital assistant (PDA); only 5% reported weekly use,
another 5% having used these devices only once or twice, and 89% reported no experience in using them.
Outside of the weeks reserved for this research, a typical class meeting consisted of several components. First, a
brief model lesson was presented to students, asking them to think of themselves as K-12 students in a content
area such as social studies or language arts. The lessons all incorporated technology to teach content. After the
lessons were finished, students would be asked to reflect upon the role of technology in the lessons, and to
critically evaluate the role it played. After this discussion, time was reserved for teaching the skills necessary to
use the technology demonstrated (software, for example). The class focused upon teaching practice rather
specific technology competencies.
Students participating in the project were assigned a chat partner for each week of the study (to accommodate an
odd number of students, several three-person groups were used over the duration of the study). During week one,
a practice chat was held for a portion of the class session, to enable participants to become familiar with the
hand-held computers they would be using (students all reported familiarity with the on-line chat environment).
During weeks two and three, students engaged in on-line discussions during class sessions: As a
lecture/demonstration was presented to the students, they were asked to respond to two specific questions as the
focus of their online conversations: “What are possible instructional applications of the technology being
demonstrated?” and “What concerns do you have about use of the technology?” The chat sessions lasted for an
average of 16 minutes during both weeks. During the chat sessions, we observed students at two minute intervals
151
and recorded the numbers of students looking up at the instructor, and the number of students looking down at
the handheld computers.
After each of the chat sessions, students saved transcripts of their discussions and we collected the hand-held
computers and transferred the transcript files to a single location. Students’ chat performance did not contribute
to their grade for the class (and instructors would not review the chat transcripts until after the end of the
semester, when the transcripts were anonymized).
Following the chat session in week three, students were broken into small groups for a follow-up interview, with
chat partners separated and placed into different groups where possible. The same interview questions were
asked in all groups and these questions served as a springboard for discussion of the experience. Interview
questions included:
Did you find it useful to be able to discuss the instructor’s presentation while it was underway?
Was it distracting to have multiple tasks to accomplish (listen to the instructor; chat with your
partner)? and
In what ways could this activity be more effective, as a way for you to reflect on what is being
presented?
These interviews were tape recorded and then transcribed. The instructors each prepared a written reflection
about teaching these class sessions while students multi-tasked, and also participated in a follow-up interview.
Data Analysis
To preserve the anonymity of the participants, each was assigned a four-character pseudonym. Any occurrence
of a student’s real name within the transcripts was replaced with the corresponding pseudonym. After the
transcripts were anonymized, instructors were allowed to review students’ chat discussions and the data analysis
began.
Analysis of the transcripts began with a qualitative approach. We read through the transcripts and sought to
identify the broad groupings into which participant comments could be sorted. This method, based in Glaser and
Strauss’ grounded theory (1967), allows researchers to sort data “into categories and then, through constantly
comparing the content of them, [define] the properties of the categories until they have taken on an abstract
form” (Tesch, 1991). Such a process helps ensure that identified themes are grounded in, and defined by, data.
Four main categories of chat comments were identified through the analysis:
Greetings and directions,

On-task conversation about the class,
On-task conversation directly related to the assigned discussion questions, and
Off-task remarks
We then began the process of assigning each of the participants’ comments to one of these categories. Within
each of the two “On-Task” categories, we further attempted to identify whether comments built upon one
another, taking the conversation to a more reflective level (as opposed to comments made more spontaneously,
without regard to the prior comments made in the context of the conversation). As we assigned comments to
either of the On-Task categories, we assigned it a “level,” with “1” assigned to an initial comment, a “2”
awarded when a comment built upon a “1” comment by adding something new to the discussion, and a “3”
assigned when a comment built upon a “2” comment by adding something new to the discussion. Perhaps due in
part to the short duration of the discussion, there were no comments that took the discussion beyond a level 3.
Comments that added nothing new to the discussion were assigned the same number as the comment to which
they were responding. This analysis was performed initially by all three researchers working individually to code
all of the transcripts. Although many comments fell squarely into one category or another, not all were as easy to
assign. To achieve consensus and to try to increase the reliability of our assignments, the researchers then met as
a group to examine all of the coded transcripts. During these meetings, “controversial” items were examined, and
discussions were held concerning the best way to code these items. As a result, a number of guidelines were
developed that served to guide coding strategies and to maintain consistency. Examples of these guidelines
included the “same speaker rule,” in which separate but immediately sequential comments by the same person
were treated as continuations of the initial comment for the purposes of depth analysis; and the principle that
when a speaker expressed two contradictory points of view in one comments (e.g., “I don’t like this software, but
152
I can see how beneficial it would be to a classroom”), the last opinion would set the tone for the comment’s
categorization (needed for the “positive, negative, neutral, or question” follow-up analysis described below).
Although the results and discussion below include a brief analysis of the non-task-related comments (such as
greetings), our primary focus of analysis was upon the on-task conversation related to the assigned discussion
questions. Specifically, we were interested in examining the contents of the conversation. Repeated readings of
this portion of the chat transcripts yielded three content-related categories:
Reflections on teaching practice,

Discussion of possible applications of the technology being demonstrated, and
Opinions about the software being demonstrated.
Comments in the “on-task, assignment-related” category were analyzed and assigned to one of these three
categories. In a follow-up analysis, each of these assignment-related comments was studied to determine if it
expressed a positive, negative, or neutral comment, or posed a question. We also explored a possible analysis
method to examine the impact of interpersonal differences on chat behavior.
As noted, every two minutes during these on-line chat sessions, counts were made of the numbers of students
looking up at the instructor or looking down at the hand-held computer. Overall percentages were computed for
each week.
To analyze the transcripts from the follow-up interviews, we used a qualitative approach to look for themes
across subjects. Participants’ comments focused upon their comfort with using the handheld computers, their
ability to focus on the lesson being delivered by the instructor, and their suggestions on the future use of
synchronous on-line discussions in the classroom. Transcripts were coded for these themes, with favorable and
unfavorable views identified.
Instructor reactions were analyzed in a similar way, with transcripts read first to identify themes and then
reviewed to find supporting or refuting evidence for these themes. Instructor comments focused upon their
comfort levels with teaching while students were engaged in on-line discussions and on their perceptions of the
usefulness and appropriateness of the discussion activity.
Results
Chat Transcript Analysis
Table 1 displays the average numbers of the different kinds of comments made in each chat session, and the
average numbers of words expressed.
Table 1. Categories of Student Chat Activity

Greetings/Instructions Off-Task On-Task On-Task
About Class On Assigned Topics
Comments Words Comments Words Comments Words Comments Words
Week 2
number 3.1a 8.2 13.5 84.3 17.5 82.1 17.8 198.5
% 5.9% 2.2% 26.0% 22.6% 33.8% 22.0% 34.3% 53.2%
Week 3
number 3.2 7.5 26.8 175.6 13.2 62.2 22.4 237.9
% 4.9% 1.5 40.8% 36.3 20.1% 12.9% 34.2% 49.2%
a
Average for each chat session.
Greetings and Off-task Conversation
During week two of the study, the smallest proportion of student comments was devoted to greetings and
instructions (M = 5.9% of all comments; M = 2.2% of all words expressed). A similar small proportion was
devoted to greetings and instructions during week three (M = 4.9% of comments, M = 1.5% of words). Typical
comments were simple, one- to three-word phrases such as “Hi!” “How you doing,” and “Hey.”
153
A much larger percentage of the conversation was made up of off-task remarks. In week two, over a quarter of
comments (M = 26% of comments; M = 22.6% of words) made by students were off task. For week three, this
number increased (M = 40.8% of comments; M = 36.3% of words). Off-task conversations ranged greatly in
content; no “typical” off-task content was detected. Topics included students’ boyfriends, girlfriends and other
social considerations (“I've been getting so many random guys lately on AOL”); post-graduation plans (“I want
to teach in Atlanta”); catching up on personal news (“How is your leg/ankle?”); and other courses (“I hate that
stupid class”).
On-Task Conversation About Class (Not Directly Related to the Discussion Assignment)
A sizeable portion of student comments was related to what was occurring in the classroom, but not directly
related to the two discussion questions meant to frame the chat assignment. During the second week, these
comments made up 33.8 percent of the comments made during the chats (M = 22.0% of words). In week three,
on-task, non-assignment related comments dropped to 20.1 percent of the overall chats (M = 12.9% of words.
This decrease seems to correlate with the increase in students’ off-task conversation from weeks two to three
(see above).
Topics of these conversational strands varied from comments about how a particular instructor was teaching a
lesson (“Steve couldn’t…do this effectively without Mark working the computer”) to the functionality of
equipment (“Good… [my handheld computer] is working now”) to considerations of the actual content of the
lesson being presented (“When he is talking about…[concept mapping for] fairy tales, [are] they supposed to be
like classic fairy tales and not like Disney movies or what?”).
On-task Conversation Related to the Assigned Discussion Questions
Overall, students devoted about one-third of their chat comments to the assigned discussion questions during
weeks two (M = 34.3%) and three (M = 34.2%), but these categories encompassed about one-half of overall
words (week two M = 53.2%, week three M = 49.2%). These on-task, assignment-related questions were the
focus of our further analysis.
To help us explore individual student chat participation, we next considered average student behavior (in
addition to the previously reported sums and percentages reported across students). During week two, there was
an average of 7.7 comments related to the chat assignment made by each student (out of an average of 24
comments made per student). There was a wide range in the amount of participation, however (range = 2.0 - 18.3
comments/student made in this category). During week three, students averaged 11 assignment-related
comments each (out of an average of 30 comments made per student), with an even wider range in participation
(range= 2.3 – 40.5 comments/student). On average, student comments were made at a depth of 1.3 (1=initial
comment, 3=maximum depth) during week two and a depth of 1.7 during week three.
During week two (when the instructors demonstrated instructional use of concept mapping software), students
tended to devote more of their on-task, assignment-related comments to opinions about software (49.1% of these
comments), as opposed to discussion of possible applications of technology (M = 30.0%) and reflections on
teaching practice (M = 21.3%). During week three (when the instructors demonstrated instruction using
spreadsheets), this focus shifted, with more attention paid to reflections on teaching practice (M = 44.0%) and
discussion of possible applications of technology (M = 42.8%) than to opinions about software (12.7%).
Reflections on Teaching Practice
During the second week’s discussion (accompanying demonstration of instruction with concept mapping
software) students made an average of 2.5 comments related to teaching practice (range = 0 – 6
comments/student on this topic). The comments were at an average depth of 1.3. A typical comment at the “1”
level was “I'm not used to this flow chart concept thing” with “Yeah, me too - i am still not seeing the overall
point, unless it is just an organizer” following at level “2.”
The discussion related to teaching practice increased during week three (when the instructors demonstrated
integration of spreadsheet software in teaching practice), with students averaging 5.6 related comments (average
range for most students was between 0 and 7.5 comments, however in one chat pair the average number of
154
student comments on this topic was 23). These discussions also tended to reflect more depth (M = 1.7). Typical
level “1” comments such as “What are your feelings of technology in classrooms?” began the related exchanges,
with responses such as “They will need more attention and one-on-one, than time with computers” made at level
“2” in response. Some of the exchanges built to level “3,” continuing, “It's not that i don't think computers have
their place…I think the kids need to use them just so they're not afraid of them.”
Discussion of Possible Applications of Technology
Students contributed an average of 2.7 comments (range = 0 – 7.7) related to possible applications of the
technology being demonstrated during week two (concept mapping software), with an average depth of 1.4.
Comments such “See, i'm doing elementary school. How are they supposed to use this thing?” were made at
level “1,” followed by “It would be like the ‘webs’ that you used to do in elementary school” at level “2.” While
comments related to this content area did not often rise to the “3” level there were several, including, “high
schoolers think more in depth, and have to for their papers and such, so this would guide them a lot better than a
2nd grader who likes simple thoughts.”
As was the case for reflections on teaching practice (above), the discussion related to technology applications in
teaching increased during week three (spreadsheet software). During this class session, students averaged 3.8
related comments (average range = 0.5 – 8.0 for most students, but in two chat pairs, students averaged 16.5 and
22.5 comments each on this topic). These week three discussions also tended to reflect slightly more depth than
during week two (M = 1.6). Chat groups began these exchanges with comments like “Ok, 3 fun ways to use this
in teaching…,” followed quickly by ” comments such as “Height of Presidents past and present vs. People in the
class… To predict who might be president one day” and “You could figure out the height of mountains in
different places in the world to determine where you want to go hiking…”
Opinions About the Software Demonstrated
During the second week’s discussion (demonstration of concept mapping software in teaching practice) students
averaged 3.8 comments related to the software being demonstrated (range = 0.5 – 10). The comments were at an
average depth of 1.2. Typical comments at the “1” level included, “So this would be hard to use in a classroom
sometimes” followed by “Yeah, you need several people (one to type and keep up and mark and one to initiate
conversation within the classroom)” at level “2.” As the discussion built, level “3” comments included those
such as, “It might be good in showing how to plan a paper - but it would be hard to do this and also ensure that
the class would help.”
Unlike the pattern noted for the discussions related to teaching practice and to applications of technology, the
discussion related to the software being demonstrated decreased during week three (demonstration of
instructional use of spreadsheet software). During week three, students averaged 1.8 related comments (range =
0 - 4.5 comments). These discussions reflected slightly more depth (M = 1.4) than during week two. Typical
level “1” comments included, “Ok- concerns w/using spreadsheets?” and “Why would there be concerns- it's not
like they are that controversial.” Comments such as “It takes me like 10 times to get the right graph when i use
them” emerged at level “2” in the discussion, while infrequent comments such as “It would take a whole lesson
just to teach the kids how to use it” were made when the discussion evolved to level “3.”
Positive, Negative, Neutral, and Questioning Comments
During week two (when the instructors demonstrated instructional use of concept mapping software), students
tended to devote more of their on-task, assignment-related discussion to Positive (42.5% of the comments made)
and Neutral (40%) comments, as opposed to Questions (10%) and Negative Comments (7.7%). While there were
a higher proportion of Negative comments during week three (when the instructors demonstrated instruction
using spreadsheets), there again were more Positive (37.5.%) and Neutral (34.2%) comments being made than
Negative Comments (19.2%) or Questions (9.1%).
155
Individual/Group Dynamics
During our many readings of the chat transcripts, we observed some interesting differences in chat behavior
between individuals, and wondered if assignment of partner might influence others’ behavior. To explore how
such an analysis might be undertaken, we conducted a sample analysis of the chat behavior of two individuals,
one who exhibited high levels of on-task behavior, and the other who exhibited high levels of off-task behavior.
On-Task Student
During week two, “Cham” and her two chat partners focused 45% of their comments toward the assigned
discussion questions, compared to the overall average of 38% for all groups. Cham herself contributed 26% of
the group’s comments, and over half of these (58.6%) were related to the assignment. This was a higher degree
of on-task performance than her partners, who each devoted 40% of their comments to the assignment.
During week three, Cham and her partner spent 51% of their comments on the assigned discussion questions,
and Cham continued to show her task focus, with 39.1% of her comments focusing on the assigned discussion
questions. Her partner focused 57.8% of comments on the assigned topics, up 7.4% from her performance in
week 2. Meanwhile, one of Cham partners from week two, when placed with a new partner during week three,
evidenced a very low proportion of on-task comments in week three (3.6%) when placed with a different partner.
Cham uses the chat to pose questions related to the assignment and raises issues concerning implications of the
class discussion on an elementary classroom.
Off-Task Student
Conversely, “Hunz’s” chat participation reflected the off-task end of the spectrum. Hunz and her chat partners
devoted very little attention to the discussion questions during both weeks, with only 2.3% related comments in
week two and 10.2% in week 3, compared to the 38% and 40.75% averages overall. Hunz uses the chat to tell
jokes and engage in small talk.
Analysis of Interview Transcripts and Classroom Observation Data
Student Comfort with Technology
An examination of responses from the students about the experience provides further insight into their
experience. The majority of students reported feeling comfortable using the handheld computers to engage in the
on-line discussions, despite minor technical difficulties. “Instant messaging” (IM’ing) is an important part of
their culture:
“I can’t imagine what life is like without it.”
“Because I usually IM all the time, it was so natural.”
Some students mentioned difficulties such as typing with the foldable keyboards perched awkwardly on their
laps, or the different navigation structure of the Pocket PC interface, but most adjusted quickly:
“I think the first day I was kind of uncomfortable because I didn’t know where all the menus were
– they were weird and stuff – but by the third time I was pretty comfortable using it.”
Distractions of Multitasking
Despite their comfort levels, students felt the instant messaging distracted them from the content of the
instructor’s lesson:
“I have a finite capacity for attention. As it gets divided up between more and more tasks, there’s
less given to any individual thing. So then, I’m not grasping any single thing. The attention needs
to go one place or the other for me to get something out of it.”
156
The instructors mirrored the students in this regard. Both commented that they felt some discomfort while they
were teaching. They attributed this to not receiving the usual levels of attention and non-verbal feedback (facial
expressions, head nods, etc.) from the students, making it more difficult to assess students’ understanding.
Interestingly, both instructors reported their impression that, most of the time, the majority of students’ attention
was focused on the handheld computers and their on-line discussions and not on the instructors or the lesson.
While classroom observation of student behavior indicates some division of attention, the observation data show
that students were looking up and attending to the instructor more often (65.9% of the time during week two, and
56.1% of the time during week two) than not. Nonetheless, the division of students’ attention was troubling to
both the instructors and the students. A student summed up the nature of the problem:
“I [am] definitely one who likes to multi-task, but as far as reading and listening simultaneously, I
don’t feel like I can do either one well unless I am giving one or the other my full attention.”
Suggestions for On-Line Chats in the Classroom
While 68% of students reported that they would not recommend synchronous on-line discussions in the
classroom, many of these students (50% of students overall) did offer suggestions for how to use on-line
discussion to support classroom instruction. They proposed interspersing short “discussion periods” that would
alternate with “lecture periods,” or incorporating a discussion period at the conclusion of a lecture. Others
offered a variant on this theme, endorsing synchronous on-line chats but only for asking short clarifying
questions (“I only use it [IM] to communicate very brief thoughts”). A smaller, 12.5% of the participants voiced
only negative opinions on their use in the classroom.
The instructor comments reflected similar thinking about intervals of instruction/demonstration and discussion.
They also provided suggestions for use of on-line discussions for connecting students in multiple classrooms, or
for interacting with a “guest speaker” unable to travel to the class in person.
Discussion
This study’s small sample and limited time frame do not permit broad conclusions about the potential of online,
synchronous classroom chats to enhance student learning during class time. However we can make some
observations about students’ effectiveness at on-line chat discussions, summarize student and instructor
perceptions of the experience, and make some recommendations for future research and instructional practice.
Students’ Effectiveness at On-line Chat Discussions
Students reported comfort with the use of the handheld computers and with the on-line chat medium (though
they felt discomfort with the synchronous nature of the discussions, a finding we’ll return to). While there was
some off-task behavior both weeks, students were able to engage in effective discussions in response to the
assigned questions, devoting a third of their comments (and half of the words they wrote) to these aspects of
their chat participation. The depth of these assignment-related discussions (the degree to which their comments
reflected and built on those of others) increased from weeks two to three. The focus of this assignment-related
discussion shifted more towards issues of teaching practice and instructional applications of technology (the
assignment for their chat discussions), and away from expressing opinions about the software being
demonstrated. Whether these changes were due to greater comfort and facility with the chat activity or to the
different technology application being demonstrated, or both, is not possible to say.
Students tended to make more positive and neutral comments than negative comments during both chat sessions,
though students were more negative in week three than in week two, when they indicated that the software being
demonstrated in week three (spreadsheets) was difficult to use, and that they saw little application for the
software in their classrooms. Further research will be needed to examine the development of students’ discussion
skills and the evolution of their discussions over time. For instance, it is possible the individuals involved can
significantly affect the quality and focus of the chat discussions they are engaged in, encouraging more or less
task focus in their chat partners – as the performance of “Cham” and “Hunz” hints at. It will be interesting to
explore this over an entire semester when such patterns may be followed and characteristics of on- and off-task
individuals may be identified.
157
Perceptions of Synchronous Discussion During Lecture
After their experiences in this pilot study, both students and instructors recommended against the use of
synchronous discussions during classroom instruction. Both reported difficulties related to students’ divided
attentions. What was at the root of the problem? We suspect that the difficulties did not emerge from asking
students to synthesize the instructor’s presentation and make related observations—research on note-taking
indicates that students can engage in this synchronous activity effectively and usefully (Beecher, 1988). Instead,
it seems likely that the difficulties were the result of the task requirements to process input simultaneously from
two sources: the instructor and the discussion partner(s). While research suggests that teens regularly do multi-
task with IM, teens also report possession of a “personal threshold” for the amount of attention they can devote
to keeping track of multiple threads of conversation and thought (Grinter & Palen, 2002) and college students
indicate some “disorientation” when returning to a lecture after following a related IM discussion thread
(Guernsey, 2003).
It may also be that multi-tasking IM behavior is more natural for younger cohorts of students than our college
undergraduates. Our college undergraduates indicated frequent use of IM, a behavior underscored by a recent
Pew Report (Jones, 2002) in which college students were often observed using IM while multi-tasking in
computer labs. But it may be an even more natural habit for younger individuals. Alan Kay (Jones, 1983)
proposes that technology is that which happened after you were born. For teens in high school, IM may be less a
technology and more “a fundamental way … to interact and relate with their peers” (NetDay, 2004, p. 21). Based
on their extensive survey results, NetDay researchers suggest that the ninth-grade students of 2003 may be at the
“leading edge in terms of acceptance and assimilation of IM technology” (p. 21). Still, students’ predominant use
of IM while multi-tasking (multiple IM windows open while also engaged in other tasks) may approximate
asynchronous communication more than the synchronous communications employed in our current study--there
is normally a lag time associated with typing and other on-line activity such as Web surfing (Reynolds, 2003),
allowing time for cognitive processing and reflection.
Students and instructors participating in our study did endorse the use of on-line discussions in the classroom,
but not at the same time as a lecture or other instructional presentation. Their suggestions for distinct periods of
instruction and discussion, for instance, would reduce the channels of information students would have to attend
to while still offering the opportunity for reflection and exploration of the content to be learned.
Recommendations for Instructional Practice
Our preliminary findings suggest that students can engage in productive on-line discussions during class time,
that the depth of their discussions may develop over time, and that engaged students may have a positive effect
on their discussion partners. Overall, participants in this pilot study used their discussion opportunities to explore
their perceptions of the instructional content, making important observations that might not otherwise have been
shared in the classroom setting. While on-line discussions during lecture presentation may not be advisable,
participating students and instructors did feel that use of on-line discussion in the classroom was worth pursuing
as a way to encourage deeper student engagement and processing. The question remains, however: Why not just
break into student groups for the benefits of discussion and avoid the complications of a technology-assisted
discussion? We can see several reasons for an on-line discussion:
Students can engage in on-line discussions without needing to leave their seats, making efficient
use of instructional time,
Communication within large lecture classes may be enhanced without the noise associated with
many students speaking aloud and, perhaps most importantly,
Students would have a written record of their discussions which could be used for study notes or
other instructional products.
There is one case in which we might recommend use of a simultaneous discussion during a classroom lecture:
The asking of clarifying questions about a part of the instruction students don’t understand or did not hear. These
questions might be directed to a study partner, a teaching assistant who could respond immediately or collect
questions for instructor review, or to the instructor, who might structure his/her classes to allow for periodic
review of accumulated questions. In this way, students’ questions would not be “lost” while they wait for an
opportunity to ask them, and their understanding of subsequent instruction might be enhanced. There may be
other similar kinds of simultaneous multi-tasking that could enhance learning while still being concluded
rapidly, thus minimizing distraction: Recent research by Gay and colleagues (Grace-martin and Gay, 2001, and
158
Hembrooke and Gay, 2003) suggest that shorter, “staccato-like” tasks can be a more effective form of
instructional multi-tasking than fewer but more focused simultaneous tasks.
Recommendations for Future Research
The methods we employed here for examination of students’ chat transcripts (the qualitative analyses and the
considerations of conversation depth and partner performance) should prove useful for future research. The
obvious aspect of student performance that we did not pursue was the effect of students’ discussions on the
learning that took place. Examination of student performance on subsequent measures such as examinations or
student projects would help us understand some of the benefits of this discussion activity.
Research inquiries over a longer time period would help illustrate development of students’ on-line discussion
abilities over time, and enable study of variation in student reactions to instructional content. Research by
Spelke, Hirst, and Neisser (1976, cited by Hembrooke & Gay, 2003) suggests that practice can improve
performance on simultaneous directed activity. In such future research, it may be useful to incorporate on-going
instructor review and perhaps grading of student chat transcripts, as a way of encouraging greater levels of on-
task behavior. It would also be of benefit to examine student discussions for evidence of multiple coding of the
learning underway.
To understand how individual participants affect their on-line discussions, an examination of personality types
might prove useful. For example, McCrae & Costa's five-factor model of personality may be useful to explore an
individual's capability for on-line interaction (McCrae & Costa, 1999). A similar study with more chat sessions
would enable us to examine how an individual affects his/her group’s discussion performance, and how group
size might moderate behavior.
Conclusion
To return to the opening assertion about “Information Age” learners and their multi-tasking ability, the results of
this pilot research suggest that our university undergraduates were comfortable with the medium of instant
messaging. They were capable of engaging in this synchronous discussion productively (half of the words they
expressed were related to on-task instructional exploration) but they felt uncomfortable doing so. While it may
be that future cohorts of students will be more comfortable engaging in this form of instructional multi-tasking,
the students and instructors participating in this research recommended multi-tasking involving focused
sequential attention to different related activities, rather than simultaneous attention. Use of brief alternating time
periods for lecture and on-line discussion, for instance, would help us consider a sequential definition of multi-
tasking. In addition, it may be that skills in instructional multi-tasking can be developed for individuals over
time; related inquiries could productively explore patterns of student discussion activity over time and across
types of content and student personality.
While we do know that this activity enabled students to engage in meaningful in-class discussion that might
otherwise not have been possible, we cannot yet say whether instant messaging in the classroom can lead to
greater learning. Follow-up research will be needed to examine the effects and not just the processes involved.
References
Beecher, J. (1988, December). Note-taking: What do we know about the benefits? ERIC Digests, 37, EDC-CS-
88-12.
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3, 149-
210.
CTGV (1992). The Jasper Experiment: An exploration of issues in learning and instructional design.
Educational Technology Research and Development, 40 (1), 65- 80.
Frand, J. (2000). The information-age mindset: Changes in students and implications for higher education.
Educause, 35 (5), 14-19,22,24.
159
Guernsey, L. (2003). In the lecture hall, a geek chorus. The New York Times, July 24.
Glaser, B. G., & Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research,
Chicago: Aldine.
Grabinger, R. S. (1996). Rich environments for active learning. In D. H. Jonassen (Ed.), Handbook of Research
for Educational Communications and Technology (pp. 665-692), New York: Macmillan.
Grace-Martin, M., & Gay, G. (2001). Web browsing, mobile computing and academic performance. Educational
Grinter, R. E., & Palen, L. (2002, November). Instant messaging in teen life. Proceedings of the 2002 ACM
Computer Supported Collaborative Work (CSCW) conference, New Orleans, 21-30.
Hembrooke, H., & Gay, G. (2003). The laptop and the lecture: The effects of multitasking in learning
environments. Journal of Computing in Higher Education, 15 (1), retrieved April 2, 2005, from
http://www.hci.cornell.edu/LabArticles/Multitasking_Hembrooke.pdf.
Jones, S. (2002, September). The Internet goes to college: How students are living in the future with today’s
technology. Pew Internet & American Life Project Report, Washington, DC: Pew. http://www.perinternet.org.
Jones, T. (1983). The Grand Old Man - Interview with Alan Kay. Psychology Today, December, 50-56.
NetDay (2004). Voices and views from today’s tech-savvy students, retrieved September 29, 2004, from:
http://www.netday.org/speakupday2003_report.htm.
McCrae R. R., & Costa, P. T. (1999). A Five-Factor Theory of personality. In L. A. Pervin & O. P. John (Eds.),
Handbook of Personality Psychology (pp. 139-153), New York: Guilford.
Reynolds, B. (2003, April). Making IM work for education, retrieved September 29, 2004, from
http://www.xplanazine.com/archives/2003/04/making_im_work.php.
Tesch, R. (1991). Qualitative Research: Analysis Types and Software Tools, London: Falmer Press.
160
Fox, R., & Henri, J. (2005). Understanding Teacher Mindsets: IT and Change in Hong Kong Schools. Educational
Understanding Teacher Mindsets: IT and Change in Hong Kong Schools

Robert Fox
Faculty of Education, The University of Hong Kong, Pokfulam Road
Hong Kong, Special Administrative Region of the People’s Republic of China
Tel: +852-2859 8014
Fax: +852-2517 7194
bobfox@hku.hk
James Henri
Faculty of Education, The University of Hong Kong, Pokfulam Road
Hong Kong, Special Administrative Region of the People’s Republic of China
Tel: +852-2859 2849
Fax: +852-2517 7194
james@cite.hku.hk
ABSTRACT
In 1998, the Hong Kong government introduced a new policy to use information technology (IT) as a
“catalyst” for change in classroom practices. The desired effect was to create a shift from a predominantly
teacher-centred approach to more learner-centred teaching. This investigation explores the impact of IT on
teaching practices in Hong Kong schools from the perspective of teachers, through the collection of data
from surveys and reports into individual school practices, group discussions and follow-up individual
interviews. The data was analysed using Fullan’s 2001 framework for understanding leadership within a
culture of change. This framework was chosen to facilitate a better understanding of the problems facing
classroom teachers. The analysis revealed that while the teachers in this study had indeed taken up the
challenge to use IT in their teaching, their practices were little changed. IT was used mainly to support the
existing teacher-centred approach. This study identified that the lack of clear and systematic leadership
support, the inflexibility of the curriculum and assessment processes, time constraints, and limited
appropriate professional development were the main reasons for the restricted impact of IT on the teachers’
practices in the classroom.
Keywords
Teacher development, IT and teacher understandings and practices, Change in schools
Introduction
This paper explores the use of information technology (IT) in classrooms and teacher readiness to use IT to take
up the challenge to use IT as a catalyst for change and reform in education. Through a study of primary and
secondary school teachers, this study aims to reveal the degree of success in the implementation of the policy at
the end of the five year period (1998-2003) and discusses the factors that need to be taken into account before
significant change can occur. The study identifies enabling factors and inhibitors of change as perceived by
teachers who were technically equipped to affect change. Fullan’s model was chosen as the conceptual
framework that would enable the researchers to understand the success or failure of the Hong Kong IT initiative.
Teachers involved in this study were taking a masters course in IT in education. They identified their present
uses of IT in the classroom and offered a vision of where IT could be used to support student learning. They
were asked how best their schools could move forward to take up the challenge laid down by the Hong Kong
government to use IT as a “catalyst” to change from a predominantly teacher-centred approach to a more learner-
centred approach (EMB, 1998). All references in this article are from the perspective of the teachers. In the
study, this research examines teacher readiness and their ability to take up the challenge outlined above by the
government. It explores the pressures teachers face that impede making significant changes in their work
practices. An assumption embedded in the government policy (EMB, 1998) was that IT could act as a catalyst
for change in classroom practices; that the use of IT per se is good for education; that IT without other major
changes and substantial support can lead to significant change in education. In this paper we argue that without
significant changes and incentives in other areas of education, for example curriculum reform, changes to the
examination system, major staff development initiatives, this change is unlikely to happen and that the use of IT
in classrooms alone will not effect the desired change.
In 1998 “HK$B3.05 (about US$M391)” was made available by the new government in Hong Kong to equip
schools with hardware and software, to provide IT infrastructure and technical support as well as five years of
161
technical competency training for teachers (Plomp et al. 2003, p. 25). The initiative reflected the government’s
perception that the adoption of IT into the education system would be the “catalyst” for a much needed paradigm
shift “from a largely text-book based, teacher centred approach to a more interactive and learner-centred
approach” (EMB, 1998, p.1). This assumption was in line with IT and education reform agendas elsewhere in
much of the industrialised world (e.g., DFEE, UK, 1997; MOE, Singapore, 2000; Ministerio de Educación,
Republica de Chile, 1998; MOE, Research, and Church Affairs, Norway, 2000). These reform agendas are all
concerned with the adoption and use of IT in schools to increase learning opportunities and student motivation
and achievement.
According to Cuban (2001) these kinds of educational reform agendas have three distinct goals for the infusion
of IT into education:
Goal 1. Make schools more efficient and productive than they currently are. … Goal 2. Transform
teaching and learning into an engaging and active process connected to real life. … Goal 3.
Prepare the current generation of young people for the future workplace (pp.13-15).
Becker (2001) conducted a school-based study where he highlighted the limitations of teacher and student uptake
of IT for educational purposes. He cited existing limitations on appropriate staff development as a primary
obstacle to any meaningful change in educational practices as a result of introducing IT into schools. Despite the
findings in studies such as those of Cuban and Becker pressure on schools to computerise and join the IT
revolution continued and continues to accelerate. There appears to be a strong belief internationally as well as
locally in Hong Kong that IT wields a kind of magic and once installed it will transform an education system
from a cash strapped, overstretched, transmission mode ugly duckling into the white swan of cost effective,
student centred mass education.
International studies into successful IT adoption in schools (e.g. Rowand, 2000; Williams; 2000; Fox, 2003;
Ofsted, 2004) tell us that teachers need considerable additional support to make significant changes in their roles
and pedagogic work practices and that IT by itself plays a very minor role in transforming teachers and teaching
approaches in schools. It is not enough to provide hardware and software, technical support, Internet
connections, technical training programs for teachers and students (Pelgrum & Law, 2003; Yuen, Law, & Fox,
2004; Trinidad & Pearson, 2005). Successful integration of IT into the classroom must involve many factors
including the integration of and interrelation between government policy, individual institutional vision, history
and established culture, teaching philosophy and approaches, curriculum and examination issues, restrictions and
limitations, change management strategies, community support and practices of individuals, and teacher-student
motivation and abilities and most significantly, clear and committed leadership at all levels.
Regional context and IT initiatives

The injection of funds into the Hong Kong education system followed on from similar initiatives in the region,
notably Singapore, which launched a five-year master plan in 1997. In the same year Hong Kong’s fledgling
government unfolded its own initiative which the new Chief Executive announced in his first policy address:
We will launch a five-year IT education strategy to promote the use of IT to enhance teaching and
learning. … Within five years, we are aiming to have teaching in at least 25% of the curriculum
supported through IT. Within ten years, we aim to see IT being applied comprehensively in school
life, and all our teachers and Secondary 5 graduates being able to work competently with IT tools
(Tung, 1997, paras 46-47).
This address signified a commitment to IT in education based on the assumption of the power of IT to play a
dominant role in effecting educational change and reform. In the five years since the injection of funding, money
has been spent on the installation of multimedia laboratories, computers, technical staff support, infrastructure
and networking within schools and to the Internet, the development of educational software and technical
programs for teachers. The stage is set for educational change; so what has happened in Hong Kong schools?
Regional research into IT in education

Cultural context is an important issue for reformers and the work of Ladao (1999) and Kwo (2000) provide a
useful lens into the Hong Kong context through their discussion of the “small potato mentality”. The notion of
162
“small potato” describes the passive role of the teacher who identifies her role as limited to classroom instruction
and staffroom marking. Kwo (2000) argues that the result of this perceived passive role on the part of the
teachers, is a lack of participation in policy making outside the classroom. Consequently teacher influence
beyond individual classrooms is limited as is their recognition of the need for broader professional development.
Effecting change in the way people do things and in particular, in the way teachers teach, takes time (Fullan,
2001). Various longitudinal studies, for example ACOT’s report on ten years of research (1995), note that
teachers normally go through stages in the change process and likewise with the uptake of new technology. The
main stages identified by the ACOT study are outlined below:
1. Entry - learn the basics of using the new technology

2. Adoption - use of new technology to support traditional instruction eg replace OHP and blackboard with
PowerPoint instruction
3. Adaptation - integrate new technology into traditional classroom practice. Here, they often focus on
increased student productivity and engagement by using word processors, spreadsheet and graphics tools
4. Appropriation - focus on cooperation, project-based and interdisciplinary work - incorporating technology
as needed and as one of many tools
5. Invention – discover new uses for technology tools for example, developing spreadsheet macros for teaching
algebra or designing projects that combine multiple technologies (ACOT, 1995, p. 16).
The ACOT study notes that this journey through the various stages “is enhanced when teachers and students
have unlimited access to technology in the classroom and are able to look at different approaches to teaching and
learning” (Newhouse, Trinidad & Clarkson, 2002, p.23). In Hong Kong, teachers and students certainly have
more access to technology now than ever before, but are they able to adopt different approaches to teaching and
learning – the hoped for shift that was the basis for the government initiatives (EMB, 1998)?
Law et al. (2001) reviewed the first few years of the HKSAR Government initiatives for the integration of IT
into education. This study found that most teachers in Hong Kong perceived their role as providers of
knowledge. Few saw themselves as people who help their students to identify their own knowledge needs and to
search for and evaluate information for themselves as advocated by the vision of the Government. Law et al.
(ibid) found that although there was no noticeable change in teaching practice across the general population of
Hong Kong schools, innovative pedagogies had successfully emerged in some schools. And further that the
innovative pedagogical practices found in Hong Kong were on a par with similar practices that were very much
valued elsewhere in the world as identified later by Kozma (2003). Law et al. (ibid) found that the emergence of
such innovative practices was more to do with strong leadership within schools, where principals have a clear
vision for the role of IT in their schools than solely with the presence of IT in the schools.
Lam and Lee’s (2000) research into Hong Kong schools also found that IT was predominantly used simply to
transmit information via PowerPoint as an alternative to the overhead projector, to prepare lesson materials and
for school administration purposes. Clearly at the time that this and Law et al.’s studies were conducted, the
teachers in the study could be seen as predominantly reaching Stage 2 of the ACOT (1995) stages of change in
the uptake of new technology, outlined previously in this article.
Methodology; participants and context of the study

The school teachers involved in this study were all Cantonese speakers, undertaking the Master of Science in
Information Technology in Education (MSc[ITE]) in the Faculty of Education at the University of Hong Kong.
Their reasons for undertaking this course were varied, but mainly they were concerned with finding out how IT
could be used to enhance teaching and learning and how to manage IT for sustainable development within their
schools.
The MSc[ITE] course itself, is designed to meet the needs of those in the educational community, who are
involved in IT implementation and use. The course aims to enable participants to become critically informed
about IT issues and practices in education and to prepare teachers and educators to design, develop, evaluate,
implement, and manage the use of IT at various educational levels and settings. The course offers an opportunity
for those who seek to develop leadership capacities in the use of IT in education to pursue advanced study in the
field of IT in education and provides a comprehensive introduction to cultural, administrative and technological
implications of new technologies in and for education. The course is part-time study and offers a flexible modular
structure enabling participants to progress according to their own pace, while still teaching in their schools.
163
An assumption made in this study was that, those who participate in an IT in education masters program are
likely to hold a positive attitude towards the use of IT in education. It was not assumed, however, that course
participants would necessarily see the need to do things differently in teaching and learning in order to meet the
government’s challenge to use IT as a catalyst for the educational reform agenda.
Teachers were asked in class to identify current practices in their schools as well as their own use of IT in the
classroom. They were asked to describe examples of their experiences and to share success stories with using IT
in their schools to support learning, where they felt that learning was advanced through the use of IT. Teachers
were then asked to upload their responses to an online course room and use these comments as a basis to discuss
similarities and differences between their schools and individual views and experiences. A number of classes
were involved in undertaking this activity in 2001 and 2002 and a total of over 200 teachers took part.
Teachers were then selected for interviews based on sampling a spectrum of experiences and positions in
schools. In particular age, gender, years of teaching experience and school responsibilities. Initially interviews
were conducted in English to gain insights into the participants’ experiences and views of the role of IT in
schools. Although the interviews did highlight some issues that built on data collected from the online teacher
comments, the responses were considered rather superficial in nature. The researchers felt that more in-depth
discussions were necessary to elicit subtle and more deep-seated concerns about IT in education and to build on
data already collected. A research assistant was therefore employed to conduct another round of interviews in
Cantonese, the first language of the participating teachers. In total, 36 teachers were interviewed.
Findings and discussion

In broad terms the accounts given by the teachers reflected the first four ACOT (1995) stages of progress
namely: entry, adoption, adaptation and appropriation outlined previously. However the great majority of
accounts from teachers related to the earlier stages of technology adoption, indicating that although these
teachers were taking a masters level course in IT in education, most had limited experience of using IT up to this
stage in their professional lives.
Most of the participating teachers concurred that changes, when they were noticeable, were mostly limited to
lower secondary school and primary school classes. Tight curriculum requirements, lessons of typically 33
minutes, and examination pressures gave little time to try out new ideas during later secondary school classes.
The teachers felt constrained by what they referred to as the overfilled curriculum and believed that curriculum
change needed to occur in order to make room for new opportunities and to explore different ways of working.
They felt their priority was to ensure that their students attained the academic level required to gain a good pass
in the existing examination system. The washback effect of the current examination system created a tidal wave
of pressures for rote learning from Grade Three to Six in primary school, to meet the intensive competition to get
into the right secondary school, followed by two sets of exams at secondary level, one at the end of Form Five
and another at the end of Form Seven for those wishing to enter university. There was agreement that unless or
until the examination system was changed in Hong Kong, teachers would be pressured by the community, the
principal, parents and students to focus on helping students achieve good examination results. As one teacher
remarked “… if you cannot make good use of IT to help improve student examination results, it is better not to
use IT at all.”
These findings suggest that a shift to learner centred approaches to teaching and learning is dependent not on the
introduction of IT but on changing the curriculum and the exam orientated educational culture that still exists
here. Curriculum and assessment reform in Hong Kong is slowly taking place, but it is very much a top down
process.
Key issues discussed at length with teachers included their views and understandings of their school’s
development and future planning. On the issue of their school’s development, few saw this as an important
concern for them. “It is up to the principal to define the school’s vision and mission and to then inform us of
what direction he wants us to go” was a typical comment. With regard to the issue of leadership, the majority of
teachers identified themselves as “small potatoes” and believed that leadership was the role of the principal only.
Nevertheless, the participating teachers strongly suggested that principals who were perceived by their staff as
“strong leaders” with a clear vision for the school and a good understanding of how to integrate IT into the
curriculum were generally enthusiastically followed. Unfortunately, of the 36 teachers interviewed only four
considered that their principal had such a vision or such an understanding of how IT could be successfully
integrated into the school and into the curriculum.
164
Making sense of where we are and moving forward
The participants in the study identified the “small potato-large potato” divide as critical to school change and
school improvement. They were small potatoes; their principals were large potatoes. This divide was reinforced
by a few respondents who indicated that if they were to become principals they would do things quite differently
than their current principals who they felt, did not really understand technology and its relevance to the
government’s proposed shift to more learner centred activities. These respondents felt that they could not lead
significant change from their position as “small potatoes”.
The challenge for the researchers was to find a way of understanding why a change mindset was uncommon
even among these respondents who had the technical expertise, had been exposed to diverse pedagogical
practices, had taken courses in educational leadership and change, and were living and working in a society that
was very actively promoting a culture of change. What were the shortcomings and how could they be identified?
The researchers turned to the work of Fullan and in particular to his text, Leading in a culture of change (2001)
in an attempt to unpack the key ingredients that underscore a change mindset. Fullan has articulated a framework
for leadership that underpins the adoption of change (Fig 1.1) and the researchers were interested to see whether
the mapping of this framework to the interview data would provide a clue or clues to explain the passive attitude
towards individual and personal change that was exposed by the data.
Figure 1. A framework for leadership, adapted from Fullan (2001)
Understanding the framework

The outer ring comprises three forces namely: enthusiasm, energy and hope (Fig 1.1). These elements are the
fuel that enables activation of the mindsets that are the heart of the framework: moral purpose, coherence
making, knowledge creation and sharing, and relationship building.
There is ample evidence that the participants had enthusiasm and energy. This is demonstrated from the
interview data and from their lifestyles. Evidence of hope is more problematic. A small minority of respondents
indicated that their principals understood the need for change, realized that harnessing new pedagogies to
technology was important, and were moving forward. The majority of respondents, however, saw the inability of
the principal to grapple with the change issues coupled with his/her big potato status to be critical impediments
165
to change. What hope they had was invested in the unlikely possibility that a new, enlightened principal could
lead change.
Moral purpose is about means and ends. Teachers must see that it is important that they make a difference in the
present and future lives of their students. They must also be interested in the adoption of appropriate, fair and
ethical means to achieve these ends. Teachers cannot put on moral purpose each morning as they arrive at
school. Moral purpose is a central component of professional (and even personal) practice. It is what allows
teachers to see the preferred and probable future and to assemble resources to equip students to adapt to that
future.
While some respondents demonstrated some levels of personal moral purpose, there was a distinct lack of
application of that to the corporate purpose and future of schooling. This deficit is explained in part by the “small
potato” mentality that promotes a view of the world that a mere classroom teacher does not need to consider such
matters. The teacher’s role is perceived to be largely instrumental in that they must follow the will of the
government, the Education and Manpower Bureau (EMB), and the principal. Teaching is perceived as an
occupation rather than a profession. The implementation of IT is discussed as a series of challenges rather than
part of a personal vision. Examples of good IT practice are often not personal accounts but rather grapevine
accounts attributed to colleagues. The idea of professional responsibility is subservient to the will of the principal
even when teachers recognise that the principal is out of step with the directives of the EMB.
Fullan argues that it is essential that leaders understand change. By this he means those who understand change
are concerned less about particular innovations and more about a state of mind he calls innovativeness.
Understanding change is about equipping teachers with a capacity and commitment to solve complex problems
and situations. Understanding change requires an appreciation that the introduction of new materials, behaviours,
or beliefs, will generate levels of anxiety and even chaos and will almost certainly lead to a dip in performance
before the value of the innovation can be seen and evaluated.
Despite the fact that the respondents had taken modules in leadership and change in the masters course, they did
not articulate an understanding of change as described by Fullan. Indeed while the barriers to change were
clearly understood, and while the frustrations of implementation had been experienced, there was a general lack
of understanding of the role that they could play as an agent of school change. (This was so even though some
respondents were able to point to changes that they had introduced within their classrooms.)
The data reveals that the respondents believe that building relationships is important and they indicate that this is
a feature of their schools. Fostering a caring culture is a key objective both among teachers and among students.
All schools have small groupings where bonding is a hallmark. However, respondents indicated that relationship
building was often task orientated and bonds were formed out of necessity to produce a desired result rather than
out of a desire to get to know a larger circle of colleagues. Bonding was most likely to occur within subject
teams or among teachers with similar responsibilities. Sometimes relationships were skin deep and power plays
often broke out between competing groups. Indeed some respondents indicated that they tended to ignore other
groups, and even individuals, because that was the easiest way to avoid messy disputes and ongoing conflicts.
There was scant evidence that schools adopted a strategic approach to team building.
Respondents used the term “no sharing culture” to explain the paucity of creative solutions to knowledge
creation and sharing. That is not to say that evidence of sharing was totally lacking, however, this tended to be
couched in terms of personal friendship, and ad hoc in nature, rather than systematic and extensive. Heavy
workloads and therefore lack of time were often proffered as reasons for a lack of collaboration even though the
application of IT could have facilitated knowledge sharing.
A culture of change requires constant choices among several possible innovations and Fullan argues that
leadership demands that the test of coherence be applied as the determining factor in making choices. Schooling
can be characterised by episodic, piecemeal, and disconnected projects and plans that compete for attention and
for resources. Leadership is about making sense of these disturbing ideas, projects and plans, firstly through the
lens of moral purpose and then through the lens of coherence.
Purpose is what binds the ethical elements of moral purpose with the decision-making tools that are associated
with coherence making. But it seems the work-world of these respondents is only indirectly determined by
purpose. The two factors that determine their choices are: urgency and student affairs. The idea of coherence
does not seem to be a factor in decision-making. For example one common response when asked how the teacher
prioritised her work stated: “I follow the principal’s instructions. I do what I am asked”.
166
Framework and amplification
The Fullan framework does illuminate some of the cultural factors that make the application of theory from one
culture to another problematic. While western cultures purport to be built around open communication and
exchange of information, this is certainly not a feature of the culture of Hong Kong or of China generally. Here,
the culture is much more attuned to respecting age and the wisdom of age. The question arises then, is leadership
in this culture to be understood differently than it is by Fullan? Is the leader’s toolkit different or must leaders
break out of the cultural norms?
The framework certainly provides an excellent set of lenses that can be employed to better understand the
obstacles that confront reformers and the diffusion of innovations throughout schools in Hong Kong. The reform
movement is about matching the world trend to identify schools as places that equip students to learn how to
learn and where students are guided to become life-long problem solvers. But the reform has been top-down and
has ignored the possibility that teachers themselves are products of an educational culture which did not equip
them to become independent decision makers. Or perhaps reform is only possible if teachers adopt reforms
unquestioningly because they are required so to do.
The evidence from this study is that these teachers, despite technical competence, are not equipped to lead. They
do not hold a consistent view about what it means to be a teacher, nor do they have the means to judge whether
or not a new innovation compliments the existing culture. Indeed teachers protect themselves and are protected
from such matters which are seen to be entirely the domain of the principal. Teachers do not therefore exhibit
personal ownership of what happens at school or where the school is going; that is someone else’s area of
expertise and responsibility. Teachers are very much orientated towards student success (as measured by
examination results) and they will therefore rather blindly adopt any gimmick that may increase student
motivation, and consequently, student success-even if only in the short term.
Teacher involvement beyond the classroom is likely to occur only on the request of the principal. This ensures
that teachers have fragments of knowledge about schools and they are therefore unable to discern linkages that
could decrease workloads and provide synergies for consistent and long lasting innovative practice.
Paradox of cross cultural implementation

The authors suggest that this small study provides some evidence to support the argument that innovations from
one culture cannot be easily transplanted into another culture and that innovations that can be linked to existing
norms are more likely to succeed. In the case of Hong Kong the educational reforms have generally mirrored
reforms in the “western world” and it has been assumed that such transplantation is possible and desirable.
Western educational reform has embraced the devolution of authority to the school level and this translates into
forms of democratic action among teachers to form a consistent mission and adoption of selected interventions.
In the western world the principal delegates responsibility to enable fusion of ownership and it could be argued
that other reforms are dependent upon this. Delegation by principals is not a common practice within Hong Kong
schools and this creates a significant challenge for teacher-lead reform.
Addressing this paradox is perhaps the key to success. Addressing the day-to-day concerns of the small potatoes;
heavy workloads, high student-teacher ratios, and limited opportunity for collaboration, in tandem with other
reforms would create the time and opportunity for the reflective practices that are fundamental to changing
traditional mindsets.
Conclusion and recommendations

This small scale study indicates that very limited change has occurred in Hong Kong schools due to the
introduction of IT, in terms of moving from a teacher centred approach to a more learner centred one as outlined
in the government’s plan (EMB, 1998). This study concludes that the community cannot realistically expect
major pedagogical change to occur in Hong Kong schools unless a more holistic and systematic approach to
facilitate change is adopted. In particular, the reform needs to involve changes in the curriculum, assessment and
the professional development of staff (not limited to just technical training in how to use new technology). There
needs to be administrative support as well as incentives for teachers to engage in the change process. From the
principals down, all staff at all levels should be given a stakehold and a leadership role in the reform process in
schools as well as incentives for stronger partnerships between schools, homes and the community. Any
167
sustainable IT implementation should therefore only be seen in the context of holistic and systematic reform and
unless other changes are coordinated with the introduction of IT in schools, the expectations for reform will fail.
Teachers “believe” that they are small potatoes, and therefore they are small potatoes. Teachers are extremely
busy, often overloaded with administrative trivia and other busy-work. They do not have sufficient time for the
reflective practices that facilitate the development of some of Fullan’s key measures of the change agent. They
have little time or opportunity for communication and relationship building let alone opportunity for deliberating
moral purpose or for identifying coherence across practices. Furthermore, the present practices are reinforced by
a system that still imposes examinations as the key measure of learning.
Teachers who want to empower their students, in addition to ensuring that they obtain good grades, must provide
learning space, both physical and mental, in which they can mature. Caught in the maelstrom of everyday school,
Hong Kong teachers do well to survive, let alone provide leadership. Government and school authorities must
find ways of providing time for teachers to engage in deep reflection. Without this time teachers cannot harness
the skills that they have learned as part of their ongoing professional development.
This study tells us that a good conceptual model enables understanding across cultural and socio-political
boundaries. The study also warns us that transplanting educational reforms across these same boundaries
requires extensive preparation and understanding of the broader school context. Providing teachers with
technical competence for change, while necessary, is not sufficient to achieve change. Understanding the culture
and energising that culture to accommodate change is also a necessary condition for successful implementation.
This study has also reinforced our understanding that change requires time. That is, it requires a length of time,
but more than that it requires ongoing “space’ for reflection and discussion.
Acknowledgements
This project was funded by the University of Hong Kong Research Initiation Grants #10203556.
References
ACOT (1995). Changing the conversations about teaching, learning and technology: a report on the 10 years of
ACOT research, Frenchs Forest, NSW: Apple Computer Pty Ltd.
Becker, H. (2001, April). How are teachers using computers in instruction? Paper presented at the meeting of the
American Educational Research Association, Center for Research on Information Technology & Organizations,
University of California, Irvine, retrieved January 7, 2005 from
http://www.crito.uci.edu/tlc/FINDINGS/special3/.
Cuban, L. (2001). Oversold & underused: Computers in the classroom, Cambridge, Mass.: Harvard University
Press.
Department for Education and Employment (1997). Connecting the Learning Society: National Grid for
Learning [on-line], retrieved November 22, 2004 from
http://www.dfes.gov.uk/consultations/conResults.cfm?consultationId=1104.
Education and Manpower Bureau (1998). Information Technology for Learning in a New Era, Five-Year
Strategy 1998/1998 to 2002/03. Printing Department: Hong Kong Government.
Fox, R. (2003). Technology and change. An examination of staff beliefs and use of ICT. Staff and Educational
Development International, 7 (1), 85-94.
Fullan, M. (2001). Leading in a culture of change, San Francisco: Jossey-Bass.
Kwo, O. (2000). Hierarchy and the small-potato mentality: Constraints and opportunities for teacher leadership.
Paper presented to 13th International Congress for School Effectiveness and Improvement, January 4-8, 2000,
Hong Kong.
168
Kozma, R. B. (Ed.) (2003). Technology innovation and educational change: a global perspective, Danvers:
ISTE.
Ladao, M. (1999). Problems inhibiting teacher development. Paper presented to the Symposium on Creating a
Changing Force for Educational Improvement, November, Hong Kong.
Lam, C. C., & Lee, F. L. (2000). A caveat to researchers on the implementation of information technology.
Educational Research Journal, 15 (2), 241-256.
Law, N., Yuen, H. K., Ki, W. W., Li, S. C., & Lee, Y. (1999). Second International Information Technology in
Education Study: Hong Kong SAR Report, Hong Kong: Centre for Information Technology in School and
Teacher Education, The University of Hong Kong.
Law, N., Yuen, H. K., Ki, W. W., Li, S. C., Lee, Y., & Chow, Y. (2000). (Eds.). Changing Classroom and
Changing Schools: Study of Good Practices in Using ICT in Hong Kong Schools, Hong Kong: Centre for
Information Technology in School and Teacher Education, The University of Hong Kong.
Law, N., Yuen, H. K., & Wong, K. C. (2001). Preliminary Review and Evaluation of IT in Education Initiatives,
Research report submitted to the Education Department of the Hong Kong SAR Government, 214pp.
(Publication No. : 69868).
Ministerio de Educación, Republica de Chile (1998). Reform in progress: Quality education for all, Santiago,
Chile: Ministerio de Educación.
Ministry of Education, Research, and Church Affairs, Norway (2000). ICT in Norwegian Education: Plan for
2000-2003, retreived January 7, 2005 from http://odin.dep.no/archive/kufbilder/01/03/IKTiu005.pdf.
Ministry of Education, Singapore (2000). Mission with a passion: Making a difference, Singapore: Ministry of
Education.
Newhouse, C. P., Trinidad, S., & Clarkson, B. (2002). Quality pedagogy and effective learning with information
and communication technology (ICT): a review of literature, retrieved April 2, 2005 from
http://www.eddept.wa.edu.au/cmis/eval/downloads/pd/litreview.pdf.
Ofsted (2004). Impact of government initiatives five years on, Ofsted publication No. 2050, Office for Standards
in Education, UK, retrieved January, 2005, from
http://www.ofsted.gov.uk/publications/index.cfm?fuseaction=pubs.displayfile&id=3652&type=pdf.
Rowland, C. (2000). Teacher use of computers and the Internet in public schools, National Center for
Educational Statistics, U.S. Department of Education, retrieved March 27, 2004 from
http://nces.ed.gov/pubs2000/2000090.pdf.
Pelgrum, W., & Law, N. (2003). ICT in education around the world: trends, problems and prospects, Paris:
UNESCO: International Institute for Educational Planning.
Plomp, T., Anderson, R. E., Law, N., & Quale, A. (Eds.). (2003). Cross-national information and
communication technology policies and practices in education, Greenwich: Information Age Publishing.
Trinidad, S., & Pearson, J. (2005). Using information and communication technologies in education. Effective
leadership, change and models of good practice, Singapore: Pearson Prentice Hall.
Tung, C. H. (1997). Building Hong Kong for a new era, Address by the Chief Executive at the Provisional
Legislative Council Meeting on 8 October, 1997. Printing Department: Hong Kong Government.
Williams, C. (2000). Internet access in U.S. public schools and classrooms: 1994-99, National Center for
Educational Statistics, U.S. Department of Education, retrieved January 7, 2005 from
http://nces.ed.gov/pubs2000/2000086.pdf.
Yuen, H. K., Fox, B., & Law, N. (2004). Curriculum innovations and multi-level e-leadership requirements:
putting research into practice, Asia-Pacific Cybereducation Journal, 1 (1), 1-12.
169
Basturk, R. (2005). The Effectiveness of Computer-Assisted Instruction in Teaching Introductory Statistics. Educational
The Effectiveness of Computer-Assisted Instruction in Teaching

Introductory Statistics
Ramazan Basturk, Ph.D.
Pamukkale University, Egitim Fakultesi
Incilipınar, Denizli 20020, Turkey
Tel: +90 258 2125555 / 282
Fax: +90 258 2125524
rbasturk@pamukkale.edu.tr
rbasturk@hotmail.com
ABSTRACT
The focus of this study is to demonstrate and discuss the educational advantages of Computer Assisted
Instruction (CAI). A quasi-experimental design compared learning outcomes of participants in an
introductory statistics course that integrated CAI to participants in a Lecture-only introductory statistics
course. Reviews of participants’ identical midterm and final exams scores demonstrated that participants in
Lecture-plus-CAI section obtained higher averages on midterm and final exams than participants in the
Lecture-only sections and these higher averages likely were because of their better performance on
concepts and practices that were taught in both regular lecture and CAI course. In addition, when the topics
of the introductory statistics course moved from descriptive statistics to inferential statistics, the learning
gap between Lecture-only and Lecture-plus-CAI is increased. Findings suggest participants’ learning
capacity of the introductory statistics could be improved successfully when CAI used as a supplement to
regular lecture in teaching introductory statistics course.
Keywords
Computer assisted instruction, Teaching introductory statistics, Teaching in higher education
Introduction
Introduction to statistics courses serve as a general introduction to descriptive and inferential statistics theory and
practice. In traditional classroom-based statistics courses, much of the learning comes from reading the selected
particular textbook, attending lectures and taking notes regularly. Recent technological developments, however,
offer instructors an additional method for teaching introduction statistics’ content and practice. Computer-
assisted instruction (CAI) continues to increase, eventually offering several advantages. Some of the benefits of
using CAI include emphasis on active learning, enrichment of collaborative learning, encouragement of greater
students independence and task-based teaching (Worthington et al., 1996; Spinelli, 2001; Prvan et al., 2002).
According to Worthington et al. (1996), computerized study guides can impact and improve students’ overall
level of mastery. Also, they emphasize that testing may be improved if students complete tests on computer
screens and receive immediate feedback about their performance.
Learning Statistics
According to Prvan et al., (2002), the range of learning and assessment activities used in statistics classes has
been extended to include group discussions, ‘real life’ simulations, problem solving and worksheets. They
emphasize that this has take place because of recent discussion about the critical role of assessment in statistics
education (Garfield and Gal, 1999; Prvan et al., 2002). They believe that “assessment activities usually focus on
the task or statistical idea and learning activities also need to include an emphasis on the students’ understanding
of learning statistics” (Prvan et al., 2002, p. 68). They also argue that “ focusing on the students’ knowledge,
rather than on the lecturer’s ideas of important content, is the characteristic of a flexible learning environment
that encourages students to develop higher conceptions of learning” (Prvan et al., 2002, p. 68).
Petocz and Reid (2001) showed that students experience learning in statistics in six qualitatively different ways.
They pointed out that these ways of experiencing learning in statistics move from a disjointed conception such as
“doing required activities in order to pass or do well in testing” towards more holistic conceptions for example
“using statistical concepts in order to understand areas beyond statistics” or “using statistical concepts in order to
change students’ views”. Using in-depth interviews, Petocz and Reid (2001) have found these very different
170
ways of experiencing learning in statistics in students at both the first and third years of tertiary study. They
strongly believe that statistics is not a ‘spectator sport’. They concluded, “Students learn statistics only if they
actually practice statistics through a whole range of statistical activity supported by an appropriate computer
package and discussion” (Petocz and Reid, 2001, p. 69). What they found is, obviously, not a new discovery, and
the recent statistics education literature contains many references to the use of laboratory and group activities in
statistical learning (Spinelli, 2001; Harrington, 1999; Nicholson 1998; Rossman and Chance 1998; Steinhorst
and Keeler 1995; Prvan et al., 2002).
Computer Assisted Instruction
Several researchers have studied CAI in teaching statistics, for example: Mathematical statistics, biostatistics,
social statistics and even business statistics with different level (e.g., Spinelli, 2001; Harrington, 1999; Warner
and Meehan, 2001; Thyer et al., 1997; Oswald, 1996). Some of the aspects of CAI were also studied in the
investigations, including teaching statistics with laboratories (Prvan et al., 2002), and using spreadsheets, for
example EXCEL, instead of particular software programs such as Minitab or SAS (Spinelli, 2001; Warner and
Meehan, 2001). Recently, researchers have begun to combine and compare CAI with programmed
instruction/distance learning approaches (e.g., Harrington, 1999; Thyer et al., 1998).
On the other hand, literature review shows that only a few studies have particularly investigated CAI in teaching
introductory statistics (Nicholson, 1998; Roiter and Petocz, 1996; Steinhorst and Keeler, 1995). These studies
illustrate that investigating CAI in teaching introductory statistics is very important because after taking
introductory statistics courses, most of undergraduate and graduate students do not take any further statistics
courses because of negative feelings and anxiety towards statistics (Garfield and Ahlgren, 1988; Peterson, 1991;
Rosenthal, 1992). Students frequently consider that their quantitative methods and statistics courses are more
difficult than their major subjects (Murtonen and Lehtinen, 1999). It is a well-known fact that one of the
common nicknames of the statistics course is “sadistics” (Forte, 1995; Rosenthal, 1992). Perney and Ravid
(1991, p. 2) stated in their research “Statistics courses are viewed by most college students as an obstacle
standing in the way of attaining their desired degree. It is not uncommon to see students who delay taking the
statistics courses until just before graduation. . . College professors who teach the research and statistics course
are all too familiar with the high level of anxiety exhibited by the students on the first day of the term.” Students
often have low motivation about introductory statistics courses, particularly “if they cannot see the direct
relevance of the course to their own real interests” (Wild, 1995, p.57). A survey of heads of biology departments
in universities in the UK (A'Brook and Weyers, 1996) cited lack of motivation and an inability to see the
relevance of statistics to biology as a factor limiting students' ability to learn statistics. In many universities
around the world, introductory statistics courses are taught to a very large class from a range of discipline
backgrounds, so examples cannot be made specific to an individual's subject area.
According to Worthington et al., (1996), many design issues arise when evaluating the efficacy of CAI. They
pointed out that one of the most pernicious is possible selection bias when comparing two classes that receive
different treatments. After a thorough review of literature, Harrington (1999) emphasized the quality of
relationship between student and instructor during the instruction needs to be observed and accounted. Duncan,
(1993) recommended that some participant variables that should be controlled: Interest in the subject, prior
knowledge of an area, generalized anxiety (Tobias, 1987), and computer anxiety (Lambert and Lenthall, 1989).
Liefeld and Herrmann (1990) controlled academic major, number of previous courses in the major, score on an
English aptitude test, and semester grade point average in their relational research. Literature shows that some
researchers have matched groups for equality on critical variables. For example, Underwood and Underwood,
(1987) matched groups on IQ scores, pretest of ability to classify objects and reading ability. Trowbridge (1987)
took into account, grade point average, gender, age and family income in his research.
This research responds to Duncan’s (1993) and Worthington et al., (1996) methodological suggestion by
examining the major impact of CAI through the use of quasi-experimental design (Compbell and Stanley, 1966;
Cook and Campbell, 1979). Campbell and Stanley stated that quasi-experimental studies are “well worth
employing where more efficient probes are unavailable” (1966, p. 205). To determine if students who completed
one credit hour CAI session in addition to attending traditional lectures outperform students who only attend
regular lectures, two different sections of introductory statistics courses were matched on confounding variables
recommended by Tobias (1987); Trowbridge (1987); Liefeld and Herrmann (1990); Duncan (1993) and
Worthington et al., (1996).
171
The impact of CAI participation was assessed with identical Midterm and Final exams for both groups. All items
in Midterm and Final examination were covered in lecture and text. However, some items in both exams were
also addressed by CAI experiences. As a result, both testing scores produced two indices of student outcomes:
General examination score and examination score for CAI-supplemented items. It is predicted that students with
the Lecture-plus-CAI section would outperform students in the Lecture-only section.
Methods and Techniques

Participants
Participants were graduate level students during all six quarters of the last two academic years at a Carnegie I
Research University. In the first meeting of the course, a survey included five questions distributed to the
participants. Questions in the survey were: Gender, age, academic major, degree pursued and number of statistics
courses taken. During the first meeting of the lecture portion of the course, the professor explained that a one
credit hour lab was scheduled and recommended. The students were then registered for the introduction
statistics course without Computer-assisted learning (Lecture - only; n = 140) or with Computer-assisted learning
(Lecture-plus-CAI; n = 65).
Procedure
Quasi-experimental design was used in this research. Both courses had several features in common. All were
taught by the same instructor and had the same objectives, content, and homework assignments. The instructor
was an experienced professor who has been teaching the three credit hour introductory statistics courses since
1970. Different editions of the same textbook (Gravetter and Wallnau, 2000) have been used in this class for
more than eight years. The instructor worked with two doctoral level assistants across seven different offerings
of the course that were trained to teach the CAI portion. With the exception of the two graduate assistants, the
CAI components were identical from quarter to quarter. Course content included descriptive statistics, frequency
distribution, central tendency and variability, hypothesis testing, t tests, correlation, regression and non-
parametric statistics (chi-square).
Students in all courses took the same multiple-choice Midterm and Final exams. Both exams consisted of 62
multiple-choice questions. Of the 62 questions of the Midterm and Final examinations, 50 items tested
generalized learning (Hannafin and Carney, 1991; Worthington et al., 1996), however, 12 items (those questions
reflecting in the CAI session) tested domain-specific learning (Worthington et al., 1996). While Midterm
examination was administered after the 7th week of the course (middle of the quarter) to each session, Final
examination was administered to each section at the end of the quarter, (15th week). Generalized learning items
included definitions, interpretations and discriminate of terms and concepts, calculations of statistics, and
interpretations of results.
Students in the Lecture-plus-CAI section attended 40-minute class each week and completed systematically
computerized exercises and tutorials. After learning concept and theory in Lecture-only part of the course,
students who choose Lecture-plus-CAI section came to the computer lab and lab instructors show them how to
make practice on real data set. For example, students learn and understand theoretical base of the measure of
central tendency and what it means in the Lecture-only class. And then in Lab, students learn how to run
measure of central tendency, get computer outputs, analyze and interpret them appropriately. Software used to
provide these exercises was a data analysis package, SPSS (Statistical Package for Social Sciences).
SPSS as a Laboratory Software Program
SPSS introduced the first mainframe statistical software package to appear on a personal computer. In addition,
SPSS was the first package released of statistical products for the Microsoft Windows personal computer
operating system. SPSS recently received the 2002 Illinois High Tech Award for statistical software
innovation. It is used in statistical education particularly for social science courses in areas such as psychology,
sociology and education. According to Prvan et al., (2002), SPSS will carry out almost all statistical analyses
required at a professional level, and certainly covers all that would be needed in a first statistics course. It is
particularly good for analysis of questionnaire data.
172
Constructivism and Relational Learning Theory
Additional one-credit hour computer laboratory was developed to help students engage with statistical ideas
supported by constructivist and relational learning theories. The constructivist view, the path of Dewey, Piaget
and Vigotsky among others, suggests that there is no such thing as knowledge "out there" independent of the
knower, but only knowledge we construct for ourselves as we learn (Dewey, 1938). Learning is not
understanding the "true" nature of things, nor is it remembering weakly perceived perfect ideas, but rather a
personal and social construction of meaning out of the mystifying array of sensations that have no order or
structure besides the explanation that we fabricate for them. According to Prvan et al., (2002), the relational view
suggests that students understand learning only in relation to their perception of their learning situation and the
subject area. Relational learning theory agrees that learning can be done through assimilation and
accommodation where old information can be adapted to create new experience by facilitating learning through
the arrangement of information. Since learners have different personalities, general aptitudes and knowledge of
a subject area, they will progress at different rates. Therefore, effective learning can occur when students engage
their interest with the content. When interest is associated to learning, the information will be remembered and
applied in real life experiences (Marton and Saljo, 1979; Petocz and Reid, 2001; Prosser and Trigwell 1999;
Ramsden, 1992).
The application of learning statistics using SPSS may benefit students by empowering them to develop their own
understanding of statistics concepts. Students will have the opportunity to learn by constructing their own ideas
and knowledge from the statistical software experiences, with supportive direction from the lab instructor.
According to Packard et al., (1993), students who are actively involved in their own learning usually become
more independent learners and problem solvers.
Results
Comparing for Matched Sections
We used to control same confounding variables that may effect why students choose one course over another.
Demographic variables used to identify any systematic differences between two different sections included age,
gender (Trowbridge, 1987; Worthington et al., 1996), prior knowledge of an area (Tobias, 1987), academic
major (Liefeld and Herrmann, 1990) and degree pursued. Mean age for Lecture-only was 27 and Lecture-plus-
CAI was 28. Standard deviations for both sections were 3. The 140 Lecture-only students consisted of 91 Male
and 35 Female students. Of the 65 Lecture-plus-CAI students, 33 were Male and 32 were Female. Specific data
for gender, students’ educational major areas and degree pursued are presented in Table 1. There was no
difference in gender between Lecture-plus-CAI and Lecture-only groups, χ2 (1, N = 205) = 1.76; p > .05. For
prior knowledge of statistics, survey distributed during the first meeting of the course was analyzed. All
participants indicated that this course is the first course about statistics and any of them did not take any statistics
courses in their academic background before.
For academic major, The 140 Lecture-only students consisted of 22 Educational Policy & Leadership (P&L), 7
School of Teaching & Learning (T&L), 67 School of Physical Activity and Educational Services (PAES), 4
Social Sciences (SOCIAL), 26 Health Sciences (HEALTH) and 14 Others (OTHERS). Of the 65 Lecture-plus-
CAI students, 9 were majoring (P&L), 9 (T&L), 14 (PAES), 11 (SOCIAL), 8 (HEALTH) and 14 (OTHERS)
students. Similarly, students’ academic major did not differ between two groups; χ2 (5, N = 205) = 10.66; p >
0.05.
Table 1. Gender, Major, and Degree Pursued comparisons between two Groups
Groups
Lecture-plus-CAI Lecture-only
n % n % χ2
Gender
Male 33 51 91 65 1.76*
Female 32 49 49 35
Major
P&L 9 14 22 15 10.66*
T&L 9 14 7 5
PAES 14 22 67 48
173
SOCIAL 11 17 4 2
HEALTH 8 11 26 19
OTHERS 14 22 14 10
Degree Pursued
Master 41 63 97 69 0.78*
Ph.D. 24 37 43 31
* Not significant, p > 0.05
Since this course is the graduate level course, participants were divided into those who pursued Master or Ph.D.
degree. For degree pursued, The 140 Lecture-only students consisted of 97 Master and 43 Ph.D. students. Of the
65 Lecture-plus-CAI students, 41 were master and 24 were Ph.D. students. There was no difference in degree
pursued between Lecture-plus-CAI and Lecture-only groups, χ2 (1, N = 205) = 0.78; p > 0.05.
Performance
In order to compare the Lecture-plus-CAI and Lecture-only groups’ Midterm and Final exam performance,
results were analyzed using independent-samples t tests. These analyses revealed a significant difference
between the two groups, t (204) = 5,09; p < .05, for midterm and t (204) = 5.83; p < .05 for final exam
performance. The sample means are displayed in Table 2, which shows that subjects in the Lecture-plus-CAI
scored significantly higher on Midterm and Final exam performance than subjects in the Lecture-only (for
Subject in the Lecture-plus-CAI, M = 43.40, SD = 7.47 for midterm and M = 48.76, SD = 6.89 for final and
Subject in the Lecture-only, M = 38.03, SD = 7.62 for midterm and M = 33.28, SD = 7.45 for final exam
performance).
Table 2. Comparisons of Course Performance by Lecture Types and Testing Periods

Testing Periods
Midterm Exam Final Exam
Lecture Types n M SD t n M SD t
Lecture-plus-CAI 65 43.40 7.47 65 48.76 6.89
5.09* 5.83*
Lecture only 140 38.03 7.62 140 33.28 7.45
* Significant, p < 0.05.
Midterm and Final exams contained 12 critical items related to domain-specific learning, which were covered in
the CAI exercises as well as in the book and lecture. These questions were analyzed separately from questions
covered only in the generalized learning (reading textbook and attending the lecture). These analysis revealed a
significant difference between the two groups, t (204) = 6,39; p < .05, for Midterm and t (204) = 17.46; p < .05
for Final exam performance. The sample means are displayed in Table 3, which shows that subjects in the
Lecture-plus-CAI scored significantly higher on Midterm and Final exam performance than subjects in the
Lecture-only (for Subject in the Lecture-plus-CAI, M = 7.16, SD = 1.45 for Midterm and M = 9.00, SD = 1.68
for Final and Subject in the Lecture-only, M = 5.83, SD = 1.25 for Midterm and M = 4.93, SD = 1.23 for Final
exam performance).
Table 3. Comparisons of Domain-Specific Items’ Performance by Lecture Types and Testing Periods
Testing Periods
Midterm Exam Final Exam
Lecture Types n M SD t n M SD t
Lecture-plus-CAI 65 7.16 1.45 65 9.00 1.68
6.39* 17.46*
Lecture only 140 5.83 1.25 140 4.93 1.23
* Significant, p < 0.05.
174
Figure 1 specifically shows that subjects in the Lecture-plus-CAI increased their scores from Midterm to Final
exam performance. On the other hand, subjects in the Lecture-only decreased their scores from Midterm to Final
exam performance. These results can be explained by the topics covered on Midterm and Final exam. Midterm
exam topics include descriptive statistics, specifically introduction to statistics, scales of measurement,
frequency distribution, measure of central tendency and measure of dispersion. However, Final exam topics
include foundations of inferential statistics, specifically normal distribution, standard distribution, probability
and samples, the distribution of sample means, hypothesis testing, t statistics, correlation, regression and
nonparametric tests (Chi-Square). This result shows that when topics moved from Midterm to Final, in other
words, from descriptive statistics to inferential statistics, the learning gap between Lecture-only and Lecture-
plus-CAI is increased.
54 Lecture-plus-CAI
Lecture-only
51
48
45
Exam scores
42
39
36
33
30
Midterm Final
Figure 1. Participants’ Midterm and Final Exam Performance by Lecture Types
Discussion
The purpose of this paper was to gather evidence that might support further investigation in the use of CAI to
teaching introductory statistics course. According to Yilmaz (1996), traditional methods of teaching introductory
statistics are generally viewed as being ineffective because they fail to establish a clear link between statistics
and its uses in the real world. To be more effective, using computers with software programs in the introductory
statistics course would be one of the important ways to improve student knowledge about statistics and its
usefulness in real life. It is a fact that emphasis on real-world applications with the computers is becoming more
prevalent in introduction statistics courses at many colleges and universities, including this course at Carnegie I
Research University.
According to Hornby (1995), the opportunity to use computers in teaching statistics provides hands on activities,
supports cooperative learning, provides active/constructive learning experiences and produces greater peer
interaction. Moreover, one-hour laboratory provides opportunities for students to engage with statistical concepts
within a learning environment supported through problem-based learning and exchange of ideas, irrespective of
the particular package that is being used.
In testing the effectiveness of CAI, several variables were controlled in this research. We inspected student
characteristics (age, gender), academic major, degree pursued and number of statistics courses taken (see
Duncan, 1993; Liefeld and Herrmann, 1990; Tobias, 1987; Trowbridge, 1987; Worthington et al., 1996).
Analyzes indicated that all variables concerning reasons for taking the class did not differ between the two
sections.
One of the important and new findings of this research in the literature is that when introductory statistics’ topics
moved from descriptive statistics (specifically, introduction to statistics, scales of measurement, frequency
175
distribution, measure of central tendency and measure of dispersion) to inferential statistics (normal
distribution, standard distribution, probability and samples, the distribution of sample means, hypothesis testing,
t statistics, correlation, regression and nonparametric tests), the learning gap between Lecture-only and Lecture-
plus-CAI is increased.
Additional time spend in the computer lab might have accounted for better performance of participants in the
Lecture-plus-CAI session on the Midterm and Final examination, but this is unlikely. The items in the Midterm
and Final exams were divided to two groups. One group was a general question, generalized-learning (Hannafin
and Carney, 1991; Worthington et al., 1996) and the other group was a CAI specific question, domain-specific
learning (Worthington et al., 1996). We were able to see whether additional class time spent in computer lab
improved generalized learning or domain-specific learning. Basically, all of the increased performance was
attributable to domain-specific performance on the items covered in Midterm and Final exams. Participants in
the Lecture-plus-CAI session, on average, answered five more items correctly than did students in the Lecture-
only session however, students in the Lecture-plus-CAI session correctly answered two more of the critical items
covered in the Lecture-plus-CAI questions on Midterm. For Final, participants in the Lecture-plus-CAI class, on
average, answered thirteen more items correctly than did students in the Lecture-only class however, students in
the Lecture-plus-CAI session correctly answered four more of the critical items covered in the Lecture-plus-CAI
questions. In addition, number of corrected items was increase from Midterm to Final examination in the
Lecture-plus-CAI session. This result suggested that it was not additional time spent that affected performance.
Rather, additional time spent in computer lab interpreted into gains in learning that component part of the course;
this results is matching with Castellan (1993) and Worthington et al., (1996) study.
This research shows that SPSS is a useful tool for teaching introductory statistics course. SPSS could be used as
a first statistics package, especially for psychology, social science or education students. Prvan et al., (2002)
indicated that SPSS is particularly good with questionnaire data and produces high quality output (e.g. in cross
tabulations), so it is attractive for these particular groups of students. This findings support the result of Prvan et
al., 2002 study that examined practical comparison of Minitab, Excel and SPSS.
Finally, Lecture-plus-CAI demonstrated in this paper was part of a Total Learning Environment for the students,
and was intended to help students develop their understanding of statistical concepts and ideas (Reid and Petocz,
2001). According to Prvan et al., (2002), statistical laboratories need to be used as part of an environment that
supports student dialogue, investigation and judgment. They emphasized that “their strength lies in their close
connection with the “experience” of a statistician, that is, working as a numerical detective with “messy” data to
solve real problems in a collegial environment” (Prvan et al., 2002, p. 74).
References
A’Brook, R., & Weyers, J. (1996). Teaching of statistics to UK undergraduate biology students in 1995. Journal
of Biological Education, 30 (4), 281-288.
Castellan, N. J., Jr. (1993). Evaluating information technology in teaching and learning. Behavior Research
Methods, Instruments & Computers, 25, 233-237.
Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research, Boston:
Houghton Mifflin.
Cook, D. D., & Campbell, D. T. (1979). Quise-experientation design, and analysis issues for field settings,
Chicago: Rand McNally.
Dewey, J. (1938). Experience and Education, New York: Kappa Delta Pi.
Duncan, N. C. (1993). Evaluation of instructional software: Design considerations and recommendations.

Behavioral Research Methods, Instruments & Computers, 23, 324-327.
Forte, J. (1995). Teaching statistics without sadistics. Journal of Social Work Education, 31 (2), 204-308.
Garfield, J., & Ahlgren, A. (1988). Difficulties in learning basic concepts in probability and statistics:
Implications for research. Journal for Research in Mathematics Education, 19 (1), 44-63.
176
Garfield, J. B., & Gal, I. (1999). Assessment and statistics education: Current challenges and directions.
International Statistical Review, 67 (1), 1-12.
Gravetter, F. J., & Wallnau, L. B. (2000). Statistics for the behavioral sciences (5th Ed.), Minneapolis-St. Paul,
MN: West Publishing Company.
Hannafin, M. J., & Carney, B. W. (1991). Effects of elaboration strategies on learning and depth of processing
during computer-based instructions. Journal of Computer-Based Instruction, 18, 77-82.
Harrington, D. (1999). Teaching statistics: A comparison of traditional classroom and programmed

instruction/distance learning approaches. Journal of Social Work Education, 35, (3), 343-352.
Hornby, P. E. (1995). Using a computerized laboratory as a springboard for transforming a traditional lecture
course. Paper presented at the 9th annual conference on undergraduate teaching of psychology, March 22-24,
Ellenville, NY, USA.
Lambert, M. E., & Lenthall, G. (1989). Effects of psychology courseware use on computer anxiety in students.
Computers in Human Behavior, 5, 207-214.
Liefeld, J. P., & Herrmann, T. F. (1990). Learning consequences for university students using computerized
mastery testing. Educational Technology, Research & Development, 38, 19-25.
Marton, F., & Saljo, R. (1979). On qualitative differences in learning: I. outcome and process. British Journal of
Educational Psychology, 46, 4-11.
Murtonen, M., & Lehtinen, E. (1999). Difficulties in research methodology studies experienced by social science
students. Unpublished manuscript.
Nicholson, J. (1998). Extending the use of statistical packages in an elementary statistics course to enhance
understanding. Paper presented at the 5th International Conference on Teaching Statistics ICOTS5, June 21-26,
Singapore.
Oswald, P. A. (1996). Classroom use of the personal computer to teach statistics. Teaching of Psychology, 23,
124-26.
Packard, A. L., Holmes, G. A., & Fortune, J. C. (1993). A Comparison of Three Presentation Methods of
Teaching Statistics. ERIC Document Reproduction Service No. 365 696, Chicago, IL.
Perney, J., & Ravid, R. (1991). The Relationship Between Attitudes Towards Statistics, Math Self-Concept, Test
Anxiety and Graduate Students' Achievement in an Introductory Statistics Course. Unpublished manuscript,
National College of Education, Evanston, IL.
Peterson, I. (1991). Pick a sample. Science News, 140, 56-58.
Petocz, P., & Reid, A. (2001). Students’ experience of learning in statistics. Quaestiones Mathematicae,
Supplement 1, 37-45.
Prosser, M., & Trigwell, K. (1999). Understanding Learning and Teaching: The Experience in Higher
Education, UK: Society for Research into Higher Education
Prvan, T., Reid, A., & Petocz, P. (2002). Statistical laboratories using Minitab, SPSS and Excel: A practical
comparison. Teaching Statistics, 24 (2), 68-75.
Ramsden, P. (1992). Learning to teach in higher education, London: Routledge.
Reid, A., & Petocz, P. (2001). Using professional development to improve the quality of assessment tasks and
student learning environments. In. C. Rust (Ed.), Improving student learning strategically, Oxford: Oxford
Brookes University, 161–167.
177
Roiter, K., & Petocz, P. (1996). Introductory statistics courses - a new way of thinking. Journal of Statistics
Education, 4 (2), retrieved April 2, 2005, from http://www.amstat.org/publications/jse/v4n2/roiter.html.
Rosenthal, B. (1992). No More Sadistics, No More Sadists, No More Victims. UMAP Journal, 13, 281-290.
Rossman, A. J., & Chance, B. L. (1998). Workshop statistics: Dissemination and assessment. Paper presented at
the 5th International Conference on Teaching Statistics ICOTS5, June 21-26, Singapore.
Spinelli, M. A. (2001). The use of technology in teaching business statistics. Journal of Education for Business,
77 (1), 41-45.
Steinhorst, R. K., & Keeler, C. M. (1995). Developing material for introductory statistics courses from a
conceptual, active learning viewpoint. Journal of Statistics Education, 3 (3), retrieved April 2, 2005, from
http://www.amstat.org/publications/jse/v3n3/steinhorst.html.
Tobias, S. (1987). Learner characteristics. In R. Gagne (Eds.), Instructional technology: Foundations, Hillsdale,
NJ: Lawrance Erlbaoum, 207-231.
Thyer, B. A., Polk, G., & Gaudin, J. G. (1997). Distance learning in social work education: A preliminary
evaluation. Journal of Social Work Education, 33, 363-367.
Thyer, B. A., Artelt, T., Markward, M. K., & Dozier, C. D. (1998). Evaluating distance learning in social work
education: A replication study. Journal of Social Work Education, 34, 291-295.
Trowbridge, D. (1987). An investigation of groups working at the computer. In Berger, D. E., Pezdek, K. &
Banks, W. (Eds.), Applications of cognitive psychology: Problem solving, education and computing, Hillsdale,
NJ: Lawrance Erlbaum, 47-57.
Underwood, G., & Underwood, J. D. M. (1987). The computer in education: A force for change? In Blacker, F.
& Osborne, D. (Eds.), Information technology and people: Designing for the future, Letcworth, England: British
Psychological Society, 167-190.
Warner, C. B., & Meehan, A. M. (2001). Microsoft Excel as a tool for teaching basic statistics. Teaching of
Psychology, 28 (4), 295-298.
Wild, C. J. (1995). Continuous improvement of teaching: A case study in a large statistics course. International
Statistical Review, 63, 49-68.
Worthington, E. L. Jr., Welsh, J. A., Archer, C. R., Mindes, E. J., & Forsyth, D. R. (1996). Computer-Assisted
Instruction as a supplement to lectures in an introductory psychology class. Teaching of Psychology, 23, 175-
181.
Yilmaz, M. R. (1996). The challenge of teaching statistics to non-specialist. Journal of Statistics Education, 4
(1), 1-9.
178
Johnson, G. M. (2005). Student Alienation, Academic Achievement, and WebCT Use. Educational Technology & Society, 8
(2), 179-189.
Student Alienation, Academic Achievement, and WebCT Use

Genevieve Marie Johnson, Ph.D.
Department of Psychology and Sociology
Grant MacEwan College, 10700 – 104 Avenue
Edmonton, Canada T5J 4S2
Tel: +1 780 497 4541
Fax: +1 780 497 5308
johnsong@macewan.ca
ABSTRACT
The current investigation sought to understand the relationships between college student alienation,
academic achievement, and use of WebCT. Fifty-three students enrolled in an undergraduate educational
psychology course provided three types of data: 1) self-rating of eight Likert scale alienation items, 2)
academic achievement measured with four types of multiple choice questions evaluating mastery of course
content, and 3) use of WebCT defined as total number of Hits, Articles Posted, and Articles Read. Findings
suggest that peer alienation was associated with increased WebCT use; learning alienation and course
alienation were associated with low WebCT use. Learning alienation demonstrated an inverse relation to
academic achievement. In most cases, significant predictive relationships between academic achievement
and student use of WebCT were curvilinear.
Keywords
WebCT, Student alienation, College student achievement, Online learning
Introduction
Alienation is a term used to describe student estrangement in the learning process (Brown, Higgins, & Paulsen,
2003). Mann (2001) defined alienation as “the state or experience of being isolated from a group or an activity to
which one should belong or in which one should be involved” (p. 7). Newmann (1981) identified four
fundamental aspects of student alienation; powerlessness, normlessness, meaninglessness, and social isolation.
Powerlessness refers to student perception of absence of personal control in learning. Normlessness reflects lack
of appropriate rule-governed behavior (e.g., academic dishonesty). Meaninglessness describes alienated
students’ interpretation of curriculum as irrelevant to their current and future needs. Loneliness and separation
from peers and teachers characterizes social isolation. Alienation is a useful construct for understanding the
mechanisms associated with undesirable learner outcomes and in developing strategies to circumvent student
academic failure (Redden, 2002; Taylor, 2000; Thorpe, 2003).
The causes of student alienation are multifaceted including curricular, institutional, and socio-cultural factors
(Brown et al., 2003; Huffman, 2001; Redden, 2002; Rokach, Bauer, & Oreck, 2003; Taylor, 2001; Trusty &
Dooley-Dickey, 1993). Alienated students feel incongruent with curricula and devoid of opportunities to
establish meaningful connections. Such disconnection results in apathy in the learning process (Parish & Parish,
2000). According to Mann (2001), alienation is caused by a teaching-learning process characterized by
compliance and bereft of creativity. In higher education, the learner is largely removed from the content to be
learned; individual opinion is devalued and reliance on personal perception is dismissed as unscientific. Frosh
(1991) argued that the very ethos of universities and colleges alienates students by excessive focus on
utilitarianism, instrumentalism, measurable performance indicators, and standardized competencies. As Barnett
(1994) cautioned:
To reduce human action to a constellation of terms such as “performance,” “competence,” “doing”
and “skill” is not just to resort to a hopelessly crude language with which to describe serious
human endeavors. In the end, it is to obliterate the humanness in human action. (p. 178)
Ross (2000) argued that contemporary society breeds alienation and disconnection among people. The increasing
presence of information and communication technologies has been identified as a catalyst of alienation in human
learning and social exchange (Cooper, 1995; Rintala, 1998). Knapp (1998) summarized the popular sentiment
that “computer-based information technologies separate and alienate people from direct experience with nature
and community, pollute the environment, disrupt ecosystems, and lead to inadequate curricula” (p. 7). Some
educators/researchers paint a picture of alienated youth surfing the net in chronic social isolation (Tell, 2000) or
gravitating toward violent and alienating computer games and Websites (Slater, 2003). Cadieux (2002) reported
that college students in face-to-face learning groups had stronger feelings of trust and interaction effectiveness
179
than did students in online learning groups, although “no significant relationship was found between sense of
community and course grades” (p. 1). Social and learning applications of information technology are, to some
extent and in some cases, interpreted as mechanisms of student alienation (Muse, 2003).
In contrast to the complex and sometimes controversial causes of student alienation, the consequences of
academic disconnection are straightforward. Withdrawing from post-secondary study prior to program
completion has been attributed to student alienation (Cadieux, 2002; Muse, 2003). Slater (2003) identified
aggression, manifest in a variety of forms, as a consequence of student alienation. A range of psychological and
emotional problems characterize students who are alienated from the learning process and estranged from the
instructional environment (Redden, 2002). According to Brown and her colleagues (2003), the consequences of
student alienation are “gang activity, violence, vandalism, absenteeism, truancy, and other forms of deviant
behavior” (p. 3). Alienated students experience an inability to cope with unfulfilled social and learning
expectations. Mann (2001) interpreted alienation as a strategy of self-preservation. By refusing to engage in the
processes of learning and by abandoning personal attempts to connect with curricula and with others, “the sense
of self is not threatened, safety is maintained and unity is preserved” (p. 10). The consequences to the learner,
unfortunately, are absence of vitality and abandonment of the desire to learn.
Student alienation has been measured in a variety of ways with a number of different instruments (Rokach et al.,
2003; Slater, 2003; Thorpe, 2003). The Chilly Classroom Climate Scale and its variations (Janz & Pyke, 2000)
have been popular in determining college student alienation associated with gender and non-traditional areas of
study (Allan & Madden, 2003; Seifried, 2000; Serex, 1997). One approach to assessing student alienation is the
Classroom Life Instrument (Johnson & Johnson, 1983) which includes 11 items that measure alienation
including items that require inverse scoring (e.g., Schoolwork is fairly easy for me). Ghaith (2003), for example,
used a modified version of the Classroom Life Instrument to investigate the relationship between perceptions of
classroom support, feelings of alienation from school, and the academic achievement of university-bound
learners of English-as-a-Foreign-Language.
Student Alienation in E-Learning Environments

During the past few decades, instructional applications of computer and Internet technologies have resulted in
range of e-learning environments and online learning events (McArthur, Parker, & Giersch, 2003). While
considerable research effort has attempted to apprehend the mechanisms of e-learning and establish effective
instructional technology practices (Al-Bataineh & Brooks, 2003; Hedberg, 2003; Johnson, Howell, & Code, in
press; McNabb, 2001; van Eijl & Pilot, 2003; Valdez et al., 1999), many questions remain. The extent to which
research findings based on traditional classrooms can be applied to e-learning is unclear. On the one hand, it
could be that psycho-educational constructs such as alienation transcend the real – virtual instructional
dichotomy. On the other hand, it may be that differences exist between real and virtual learning environments
that radically alter the pattern of variables previously implicated in student alienation and academic achievement.
In e-learning environments, are some students estranged from curricula and disconnected from e-instructional
processes?
A particularly popular post-secondary application of Internet learning technology is WebCT (Curtin, 2002;
Johnson & Howell, in press; Kendall, 2001; Linder & Murphy, 2001). WebCT is a course management system
comprised of an integrated set of educational tools for constructing and managing online course environments.
For teaching faculty, these tools fall into four categories: 1) educational tools that facilitate learning,
communication, and collaboration; 2) content building utilities for organizing course material; 3) administrative
utilities for managing courses; and 4) design utilities for constructing courses (WebCT, 2004). WebCT provides
a variety of tools and features that can be added to courses including a conferencing system, on-line chat, student
progress tracking, grade maintenance and distribution, access control, navigation tools, auto-marked quizzes,
electronic mail, course calendar, and student homepages. Most typically, instructors tailor courses by selective
use of WebCT tools (Willett, 2002).
The current investigation sought to understand the relationships between college student alienation, academic
achievement, and use of WebCT. How is student alienation manifest in hybrid e-learning environments? Is
student use of WebCT predictive of, or predicted by, academic achievement? What are the paths of influence
between student alienation, academic achievement, and WebCT use?
180
Method
Participants
All students enrolled in two sections (40 students per section) of an undergraduate educational psychology
course offered at a Western Canadian college were invited to participate in the study. Fifty-three students
satisfied research requirements and were included in the study (i.e., many students were absent or late the day
that participation was solicited). Both course sections were taught by the same instructor in the same way (i.e.,
utilized the same textbook and supplementary materials, PowerPoint slides, WebCT tools, course assignments,
examinations, and evaluative weightings). All students were required to engage in regular online group
discussion of case studies. Student online discussion postings were individually marked (i.e., independent of
other group members’ postings) and contributed 20% to the final course grade. All students had access to
optional cooperative online study groups of which approximately 9% chose to make postings for bonus marks.
Students also had WebCT access to a range of optional learning events such as online practice quizzes, course
material, and grades. Thus, the course is best described as hybrid e-learning (Willett, 2002).
The 53 research participants ranged in age from 18 to 43 years (M = 21.5, SD = 4.91). Approximately 90% of the
sample was female which is typical in the context of the participating college. Subjects reported, on average, 21
college credits complete (range 0 to 145, SD 26.1) where 3 credits correspond to a typical one-semester course.
With data missing for one participant, 49.1% of the sample indicated their intention to teach elementary school,
43.4% reported the intention to teach secondary school, and 5.7% were undecided.
Measures
Student Alienation
A self-report rating scale was developed that measured student alienation. Eight items, based on the alienation
subscale of Johnson and Johnson’s (1983) Classroom Life Instrument and adapted for a college sample, were
rated by participating students on a 7-point Likert scale ranging from 1 (not at all) to 7 (absolutely). Table 1
describes the pattern of ratings on these alienation items for the group of participating college students. For
seven items alienation was reflected by high student ratings (e.g., I feel upset at school) and for one item
alienation was reflected by low student ratings (i.e., I am a good student).
Table 1. Descriptive Statistics for Student Rating of Alienation Items

Student Self-Rated Alienation Item Mean Rating Range SD
In this course the marking system is fair. 4.6 1–7 1.8
The workload in this course is excessive. 2.6 1–4 1.0
I should get along with others better than I do. 2.7 1–7 1.6
I get discouraged in school. 3.8 1–7 1.6
I feel upset at school. 3.1 1–7 1.7
I am not doing as well in college as I would like. 4.4 1–7 1.7
I am a good student. 5.7 3–7 1.0
When I take a test, I am afraid that I will fail. 4.3 1–7 1.9
Note: In the rating scale, 1 = not at all, 7 = absolutely.
Student Use of WebCT
Student use of WebCT was measured using the WebCT Track Students function. Track Students maintains a
record of the number of times each student accesses course features (WebCT, 2004). Three measures of student
use of WebCT were obtained via Track Students:
1. WebCT Hits: The number of times each student accessed the Homepage (first page following sign on), any
tool (from the options available), or a Content Module page.
2. Articles Read: The number of articles each student opened in the Discussions tool. This included both
required reading of online case study postings and optional reading of online study group postings.
3. Articles Posted: The number of articles each student posted in the Discussions tool. This included both
required online case study postings and optional online study group postings.
181
Table 2 describes these three measures of WebCT use for the group of participating college student. There is
considerable variability in participants’ use of WebCT. One student, for example, scored 104 on WebCT Hits;
another student scored 684 on the same metric during the same four month period. One student made no WebCT
Discussions postings (i.e., achieved a mark of zero on required online case study postings and made no online
study group postings for bonus marks); another student made 39 postings which included several optional online
study group postings and postings beyond the recommended number for the case study assignments (i.e.,
minimum requirement of 16 online postings, four for each of four case studies).
Table 2. Descriptive Statistics for Measures of Student Use of WebCT

Measure of Student Use of WebCT Mean Range SD
WebCT Hits (Homepage, any tool, Content page) 289.0 104 – 684 120.0
Articles Read (Discussions articles accessed) 94.5 35 – 162 30.5
Articles Posted (Discussions articles posted) 17.9 0 – 39 5.5
Student Academic Achievement
Student academic achievement was measured by the objective test items on three midterm examinations and one
final examination. The midterm examinations were not cumulative, assessing student knowledge of a relatively
limited amount of course material. The final examination was cumulative, assessing mastery of all course
content. Each midterm examination contained 24 multiple choice items and the final examination contained 80
multiple choice items (36 items assessed knowledge of course material previously tested and 44 items assessed
knowledge of untested material, that is, subsequent to the third midterm examination). While the midterm and
final examinations included case study analyses that contributed to examination marks, due to the subjective
nature of marking, these items were not included in any metric of student achievement. Multiple choice items
were evenly distributed across four cognitive domains (i.e., factual, comprehension, application, conceptual) as
specified in the supplementary test item bank (Renaud, 2003). Correct responses for each type of multiple choice
item were summed across the midterm and final examinations to result in four measures of student academic
achievement: total number of 1) factual, 2) comprehension, 3) application, and 4) conceptual examination items
answered correctly. Table 3 describes these four measures of academic achievement for the group of
participating college students.
Table 3. Descriptive Statistics for Measures of Student Academic Achievement

Multiple Choice Examination Item Category Mean Range SD
Factual Items 22.9 14 – 29 3.5
Comprehension Items 17.0 9 – 26 3.7
Application Items 18.7 11 – 25 3.3
Conceptual Items 14.9 9 – 23 3.1
Note: Each of three midterm examinations contained 24 multiple choice items for a total of 72 items. The final
examination contained 80 multiple choice items. 72 + 80 = 152 items divided by four question-types resulted in
38 test questions in each cognitive category. Metrics of student academic achievement reflect the number
answered correctly in each multiple choice question category.
Procedures
Toward the end of the academic term, during regular class time, following completion of all required and
optional online assignments and midterm examinations but prior to the final examination, students were invited
to participate in the study. Students were told that the study was directed toward understanding issues related to
college student e-learning. The consent form included request to use student course marks and WebCT records
for purposes of the study. The course instructor did not have access to the student completed alienation rating
scale items and consent forms until after all course grades were submitted. Following completion of the final
examination, the course instructor emailed all students, using WebCT Mail, describing the study in detail and
providing direction to online information regarding study findings and implications for instructional practice.
Results
The eight student-rated alienation items were correlated (Pearson Product-Moment) with the three measures of
WebCT use (refer to Table 4) and the four measures of academic achievement (refer to Table 5). Several
182
significant correlations emerged in a direction consistent with previous research. For example, as perception of
self as a competent learner (i.e., I am a good student) increased, use of WebCT (i.e., Hits) and achievement (i.e.,
factual test items) tended to increase. Some significant correlations, however, were inconsistent with anticipated
direction. As student rating of the item, I should get along with others better than I do, increased, use of WebCT
(i.e., Hits, Articles Read, Articles Posted) tended to increase. In general, alienation items correlated more with
student use of WebCT than with measures of academic achievement.
Table 4. Correlations between Measures of Student Alienation and WebCT Use

Student Self-Rated Alienation Item WebCT Hits Articles Read Articles Posted
In this course, the marking system is fair. -.29*
The workload in this course is excessive. -.34* -.34*
I should get along with others better than I do. .40** .32* .35**
I get discouraged in school. -.31*
I feel upset at school.
I am not doing as well in college as I would like.
I am a good student. .35** .29*
When I take a test, I am afraid that I will fail.
*p < .05 **p < .01
Table 5. Correlations between Student Alienation and Academic Achievement

Multiple Choice Examination Item Category
Student Self-Rated Alienation Item Factual Comprehension Application Conceptual
In this course, the marking system is fair.
The workload in this course is excessive.
I should get along with others better than I do.
I get discouraged in school.
I feel upset at school. -.35*
I am not doing as well in college as I would like. -.31*
I am a good student. .34*
When I take a test, I am afraid that I will fail. -.31* -.35*
*p < .05
Given the highly exploratory nature of the investigation, linear and quadratic regression analyses were conducted
with measures of WebCT use as both independent and dependent variables. With WebCT use as the independent
variable, Articles Posted evidenced a curvilinear relationship with factual test items answered correctly (DF =
48, F = 3.25, P = .047) accounting for approximately 12% of the variance in that metric of student achievement.
Visual inspection of the quadratic regression line revealed that increased student postings predicted increased
student achievement until approximately 28 postings, after which achievement decreased slightly as postings
increased. WebCT Hits approached a significant curvilinear relationship with application test items (DF = 48, F
= 3.10, P = .054) accounting for approximately 11% of the variance in that metric of student achievement.
Visual inspection of the quadratic regression line revealed that the fewest and the most WebCT Hits predicted
low student achievement with maximum student achievement predicted by moderately high WebCT Hits
(approximately 400 hits, one standard deviation above the mean).
Linear and quadratic regression analyses were conducted in an attempt to apprehend the capacity of academic
achievement to predict student use of WebCT (i.e., academic achievement as the independent variable). Factual
test items evidenced a curvilinear relationship with WebCT Articles Posted (DF = 50, F = 4.63, P = .014)
accounting for approximately 16% of the variance in that metric of WebCT use. Linear regression of factual test
items was also predictive of WebCT Articles Posted (DF = 51, F = 5.52, P = .023), accounting for
approximately 10% of the variance in that metric of WebCT use. Observed scores revealed three students within
the average achievement range (i.e., 22 to 25 factual test items correct) who made excessive WebCT postings
(i.e., more than 2 SD above the mean). These three students affected both linear and quadratic regression lines.
Number of application test items answered correctly predicted both WebCT Hits (DF = 49, F = 4.00, P = .025)
and Articles Read (DF = 49, F = 3.81, P = .029) in significant curvilinear relationships accounting for, in both
cases, approximately 14% of the variance in student use of WebCT (Figures 1 and 2). High achieving and low
achieving students were lighter users of WebCT than were average achieving students.
183
Figure 1. Curvilinear regression analysis: Application test items predicting WebCT Hits
Figure 2. Curvilinear regression analysis: Application test items predicting WebCT Articles Read
Discussion
Student alienation is typically conceptualized as a uniform construct (Johnson & Johnson, 1983). Given the
pattern of correlations between student self-rated alienation items and measures of WebCT use, this may not be
accurate in hybrid e-learning environments. The pattern of significant relationships is explained by
categorization of the eight items used to measure student alienation. Two items measured Course Alienation
(The workload in this course is excessive. In this course, the marking system is unfair). One item addressed peer
relations and may be considered indicative of Peer Alienation (I should get along with others better than I do).
The remaining five items assessed global student alienation (e.g., I get discouraged in school) and are
categorized as indicators of Learning Alienation. Such a classification of alienation self-rated items reveals a
distinguishable pattern of correlations.
As Peer Alienation increased, all measures of student use of WebCT tended to increase. Apparently, as students
experienced disconnection from peers, they sought virtual connection with course content (i.e., WebCT Hits) and
184
with peers (i.e., Articles Posted and Articles Read). As Course Alienation increased, students were less likely to
utilize WebCT. Students alienated from the course may have been generally resistant to involvement with course
materials and learning activities which included WebCT. As Learning Alienation increased, student use of
WebCT tended to decrease. Students who rated themselves as discouraged in school accessed WebCT less than
students who were not estranged from learning processes. Peer Alienation and Course Alienation were not
significantly correlated with any measure of academic achievement; Learning Alienation inversely related to
achievement. In this regard, student alienation was related to academic achievement but in ways that may be
unique to hybrid e-learning environments. Learning alienation appeared most critical to student achievement,
although all categories of alienation were related to student use of WebCT.
Curvilinear relationships between student use of WebCT and academic achievement are suggested by the results
of this study. Some metrics of student use of WebCT (i.e., Articles Posted and Hits) predicted some dependent
variables (i.e., factual and application test items answered correctly). The fewest number of WebCT postings
predicted the lowest student achievement. However, the greatest number of WebCT postings did not predict the
highest student achievement. Few WebCT postings may have adversely affected achievement by minimizing
active student involvement in e-learning opportunities. At the same time, excessive virtual postings may have
reduced time available for other forms of e-learning (e.g., online practice quizzes). Some students may have
inaccurately concluded that increased postings compensate for lack of alternative learning behaviors.
Additionally, relative to the class mean, few WebCT Hits predicted low student achievement and many WebCT
Hits predicted average student achievement (i.e., application test items correct). Limited use of WebCT may
have compromised student learning by limiting course engagement; excessive use of WebCT may reflect too
narrow a range of student study strategies and learning behaviors. No metric of student use of WebCT was
predictive of comprehension and conceptual test items suggesting that e-learning may have uniquely affected
concrete forms of cognitive processing such as those involved in responding to factual and application test items.
The capacity of academic achievement to predict student use of WebCT was generally greater than the capacity
of WebCT use to predict academic achievement, although a reciprocal effect could be argued since significance
was achieved in both predictive directions. In general, students who correctly answered the fewest number of
factual test items made the fewest number of WebCT postings. As achievement increased, WebCT postings
increased until, in some cases, average achievement (approximately 22 factual test items answered correctly)
was associated with excessive posting. High achieving students (i.e., at least one standard deviation above the
mean on factual test items) made an average number of WebCT postings (i.e., approximately 18 postings).
Perhaps, highly motivated but averaging achieving students engaged in excessive postings in hopes of improving
their achievement, which proved ineffective. High achieving students may have engaged in a variety of virtual
and real learning behaviors, which was effective in mastering course content.
Application test items predicted both WebCT Hits and Articles Read in curvilinear fashion with no evidence of
linear relationships (Figures 1 and 2). Students who achieved the lowest success with application test items were
the least likely to capitalize on WebCT learning events, perhaps indicative of general lack of study effort and
course engagement. Students who achieved the highest success with application test items were less likely than
average achieving students to utilize WebCT tools and content pages, perhaps indicative of a wider range of
learning behaviors. Students who correctly answered the most and fewest application test items were also those
who read the fewest WebCT postings. Average student achievement (i.e., approximately 18 application test
items correct) was a better predictor of high WebCT use (i.e., Articles Read) than was low or high student
achievement. Low achievers, alienated from the learning process, may not have read other students’ postings in
WebCT Discussions, a behavior which may have further contributed to low achievement. High achieving
students, while conscientious about satisfying required course assignments, did not exert excessive effort reading
other students’ postings. Such limited reading of postings may have further contributed to high achievement by
allowing time for other study and learning behavior.
Limitations and Further Research

A limitation of the current investigation was sample characteristics, most notably, sample size and gender bias.
The small sample (N = 53) may have resulted in idiosyncratic data. Because approximately 90% of the sample
was female, generalization of findings to male students is not possible. Peer alienation, for example, may be
experienced differently by males and females; online strategies to compensate for such alienation may be
gender-specific.
185
The proposed categorization of student self-rated alienation items has face validity but content validity was not
established. Indeed, there is no evidence that the eight alienation items used in this study factor into the three
proposed categories (i.e., peer, course, and learning alienation). Conceptualization and corresponding
measurement of the psycho-educational construct of student alienation requires further investigation. The
assumption that alienation is a single unified construct may be accurate in traditional learning environments but
in online instructional contexts, student alienation may require a typology. The underlying causes of student
alienation may be consistent across real and virtual learning environments but the consequences in terms of
student behavior may vary as a function of real versus virtual learning environments. While heavy WebCT use
was associated with peer alienation, we cannot conclude that heavy WebCT use was detrimental to students.
Heavy WebCT use may have compensated for peer alienation and functioned as a mechanism of virtual peer
connection.
The pattern of significant predictive relationships between metrics of WebCT use and measures of student
achievement requires further investigation. Only factual and application test items were predictive of, or
predicted by, student use of WebCT. Significance was not reached with comprehension and conceptual test items
as independent or dependent variables. Factual and application test items appear to measure concrete
interpretation of course material. Comprehension and conceptual test items, conversely, require a more abstract
level of course content extrapolation. It may be that e-instruction facilitates concrete forms of student cognitive
processing at the expense of more abstract forms of thinking.
A finding of the current investigation that requires further investigation is the predictive differences in WebCT
Articles Posted and Articles Read. Posting and reading in e-learning environments were differentially associated
with student academic success. A real classroom equivalent is students contributing versus actively listening in
cooperative groups. (The analogy must include active listening. In real discussion groups, students may be
physically present but not actively listening. In virtual discussion groups, postings are opened as hyperlinks
which require student intention and action.) Student success with factual test items predicted number of Articles
Posted in WebCT Discussions. Success with application test items predicted number of times online articles
were read. What are the relative learning advantages and disadvantages of posting versus reading in e-
discussion? What cognitive and social processes are involved in virtual student expression as opposed to virtual
student reception?
Conclusion and Implications for Practice

Figure 3 presents a graphic representation of the interrelations between and predictive utility of student
alienation, academic achievement, and WebCT use. Peer alienation is associated with increased WebCT use
while learning alienation and course alienation are associated with low WebCT use. Only learning alienation is
associated with academic achievement, in an inverse relationship. The direction of causation between academic
achievement and WebCT use remains unclear, although there is evidence of reciprocal influence with
achievement controlling WebCT use more than WebCT use controls achievement. Regression analysis requires
assumption of dependent and independent variables. Given the highly exploratory nature of the current
investigation, dependent and independent variable status alternated in order to determine predictive relationships.
In most cases, significant predictive relationships between academic achievement and student use of WebCT
were curvilinear.
While decisive conclusions may be premature, the current investigation lends support to the notion that variation
in online learning behavior is related to college student achievement and alienation. Educational system
developers could easily implement some form of alert that may be psychologically useful to learners and
pedagogically useful to teachers. For example, in the future, instructors may be able to input user parameters
associated with the best psycho-educational outcomes. In the context of the current investigation, such
parameters may be 4 to 5 articles posted and 40 to 100 WebCT hits following one month of instruction, 8 to 10
articles posted and 80 to 200 WebCT hits following two months of instruction, and so on. The instructor could
be electronically alerted to specific students whose e-learning behavior fell outside of such parameters. Via email
or other means, the instructor could advise identified students, on the one extreme, of the benefits of e-learning
events and, on the one extreme, of the benefits of balanced and varied approaches to virtual and hybrid learning.
Conceivably, such an alert system could directly notify the student that his/her online behavior may not be
conducive to maximum achievement outcomes. Indeed, at some future point, online learner behaviors that are
clearly linked to less than ideal student functioning (e.g., peer alienation and excessive asynchronous
communications) may result in alerts and information links (e.g., college clubs, volunteer opportunities) that
move some students toward greater social and psychological connection.
186
Figure 3. A model of the interaction between student alienation, academic achievement, and WebCT use
References
Al-Bataineh, A., & Brooks, L. (2003). Challenges, advantages, and disadvantages of instructional technology in
the community college classroom. Community College Journal of Research and Practice, 27, 473-484.
Allan, E. J., & Madden, M. (2003). Chilly classrooms for female undergraduate students at a research university:
A question of method? Paper presented at the Annual Meeting of the American Educational Research
Association, Chicago, IL, USA, ERIC Document Reproduction Service No. ED479384.
Barnett, R. (1994). The limits of competence-knowledge, higher education and society, Buckingham, England:
Society for Research into Higher Education/The Open University Press.
Brown, M. R., Higgins, K., & Paulsen, K. (2003). Adolescent alienation: What is it and what can educators do
about it? Intervention in School & Clinic, 39, 3-7.
Cadieux, C. P. (2002). Variables associated with a sense of classroom community and academic persistence in
an urban community college online setting. Doctoral dissertation submitted to Old Dominion University,
Virginia, ERIC Document Reproduction Service No. Ed474545.
Cooper, D. E. (1995). Technology: Liberation or enslavement? In R. Fellows (Ed.), Philosophy and technology
(pp. 7- 18), Cambridge, MA: Press Syndicate of the University of Cambridge.
Curtin, J. (2002). WebCT and online tutorials: New possibilities for student interaction. Australian Journal of
Educational Technology, 18, 110-26.
Frosh, S. (1991). Identity crisis: Modernity, psychoanalysis and the self, London, England: Macmillan.
Ghaith, G. (2003). The relationship between forms of instruction, achievement and perceptions of classroom
climate. Educational Research, 45, 83-93.
Hedberg, J. G. (2003). Ensuring quality e-learning: Creating engaging tasks. Educational Media International,
40, 175-186.
187
Huffman, T. (2001). Resistance theory and the transculturation hypothesis as explanations of college attrition
and persistence among culturally traditional American Indian students. Journal of American Indian Education,
40, 1-23.
Janz, T. A., & Pyke, S. W. (2000). A scale to assess student perceptions of academic climates. Canadian Journal
of Higher Education, 30, 89-122.
Johnson, D. W., & Johnson, R. T. (1983). Social interdependence and perceived academic and personal support
within the classroom. Journal of Social Psychology, 120, 77-82.
Johnson, G. M., & Howell, A. J. (in press). Attitude toward instructional technology following required vs.
optional WebCT usage. Technology and Teacher Education.
Johnson, G. M., Howell, A. J., & Code, J. R. (in press). Online discussion and college student learning: Toward
a model of influence. Technology, Pedagogy and Education.
Kendall, M. (2001). Teaching online to campus-based students: The experience of using WebCT for the
community information module at Manchester Metropolitan University. Education for Information, 19, 325-46.
Knapp, C. E. (1998). The emperor has no clothes … and computer-based information technology harms the
quality of all life. Science, Mathematics, and Environmental Education, 101, 7-11.
Linder, J. R., & Murphy, T. H. (2001). Student perceptions of WebCT in a web-supported instructional
environment: Distance education technologies for the classroom. Journal of Applied Communications, 85, 36-47.
McArthur, D., Parker, A., & Giersch, S. (2003). Why plan for e-learning? Planning for Higher Education, 31,
20-28.
McNabb, M. (2001). In search of appropriate usage guideline. Learning and Leading with Technology, 29, 50-
53.
Mann, S. J. (2001). Alternative perspective on the student experience: Alienation and engagement. Studies in
Higher Education, 26, 7-13.
Miller, M. T., & Lu, M. Y. (2003). Serving non-traditional students in e-learning environments: Building
successful communities in the virtual campus. Educational Media International, 40, 163-69.
Muse, H. E. (2003). The Web-based community college student: An examination of factors that lead to success
and risk. Internet and Higher Education, 6, 241-261.
Newmann, F. M. (1981). Reducing student alienation in high school: Implications of theory. Harvard
Educational Review, 51, 117-129.
Parish, T. S., & Parish, J. G. (2000). Determining a profiler of disconnected students. Paper presented at the
Annual Meeting of the Mid-Western Educational Research Association, Chicago, IL, USA, ERIC Document
Reproduction Service No. ED447191.
Renaud, R. (2003). Test item file for Educational Psychology (2nd Canadian Edition), Toronto, ON: Pearson
Education Canada.
Redden, C. E. (2002, October). Social alienation of African American college students: Implications for social
support systems. Paper presented at the National Convention of the Association for Counselor Education and
Supervision, Park City, UT, USA, ERIC Document Reproduction Service No. ED470257.
Rintala, J. (1998). Computer technology in higher education: An experiment, not a solution. Quest, 50, 366-378.
Rokach, A., Bauer, N., & Oreck, T. (2003). The experience of loneliness of Canadian and Czech youth. Journal
of Adolescence, 26, 267-282.
188
Ross, E. W. (2000). Alienation, exploitation, and connected citizenship. Theory and Research in Social
Education, 28, 306-310.
Seifried, T. J. (2000). The chilly classroom climate revisited: What have we learned, are male faculty the
culprits? PAACE Journal of Lifelong Learning, 9, 25-37.
Serex, C. P. (1997). Perceptions of classroom climate by students in non-traditional majors for their gender.
Paper presented at the Annual Meeting of the Association for the Study of Higher Education, Albuquerque, NM,
USA, ERIC Document Reproduction Service No. ED416753.
Slater, M. D. (2003). Alienation, aggression, and sensation seeking as predictors of adolescent use of violent
film, computer, and website content. Journal of Communication, 53, 105-121.
Taylor, J. S. (2001, April). Through a critical lens: Native American alienation from higher education. Paper
presented at the Annual Meeting of the American Educational Research Association, Seattle, WA, USA, ERIC
Document Reproduction Service No. ED452753.
Taylor, J. S. (2000, April). Portraits in alienation: Native American students on a predominantly white campus.
Paper presented at the CIC Graduate Research Conference, Iowa City, IA, USA, ERIC Document Reproduction
Service No. ED478059.
Tell, C. (2000). Generation what? Connecting with today’s youth. Educational Leadership, 57, 8-13.
Thorpe, P. K. (2003, April). A mediation model relating teacher ratings of student achievement to student
connectedness at school. Paper presented at the Annual Meeting of the American Educational Research
Association, Chicago, IL, USA, ERIC Document Reproduction Service No. ED476417.
Trusty, J., & Dooley-Dickey, K. (1993). Alienation from school: An exploratory analysis of elementary and
middle school student’s perceptions. Journal of Research and Development in Education, 19, 1-12.
van Eijl, P., & Pilot, A. (2003). Using a virtual learning environment in collaborative learning: Criteria for
success. Educational Technology, 43, 54-56.
Valdez, G., McNabb, M., Foertsch, M., Anderson, M., Hawkes, M., & Raack, L. (1999). Computer-based
technology and learning: Evolving uses and expectations, Oakbrook, IL: North Central Regional Educational
Laboratory.
WebCT Inc. (2004) WebCT: Learning without limits, retrieved November 30, 2004, from
http://www.webct.com/
Willett, H. G. (2002). Not one or the other but both: Hybrid course delivery using WebCT. Electronic Library,
20, 413-19.
189
Le Heron, J., & Sligo, F. (2005). Acquisition of simple and complex knowledge; a knowledge gap perspective. Educational
Acquisition of simple and complex knowledge; a knowledge gap

perspective
Judy Le Heron
Senior Tutor
The Institute of Information & Mathematical Sciences
Massey University, Albany Campus
Auckland, New Zealand
Tel: +64 9 414 0800
J.L.LeHeron@massey.ac.nz
Frank Sligo
Associate Professor and Head
Department of Communication and Journalism
Massey University
PO Box 756, Wellington, New Zealand
Tel: +64 4 801 5799 ext 6395
F.Sligo@massey.ac.nz
ABSTRACT
This paper assesses university students’ acquisition of simple and complex knowledge, in exploring
whether the knowledge gap hypothesis (KGH) with its origins in community-based research into people’s
informal learning from mass media, provides insights into students’ acquisition and retention of
information. The KGH posits that attempts to equalise knowledge within a community by releasing new
information into it often either has no such effect, or even worsens knowledge inequities. The present study
set out to discover if members of a diverse student class, when presented with the same course-related
information, both simple and complex, acquired knowledge at similar rates. After formal university
education, those with less prior education acquired less knowledge from the same learning opportunities
than those with more prior education at each survey point. Second, although entry education level led to
different learning outcomes initially, those with less prior education did subsequently achieve the same
level of knowledge as those with more prior education. The paper finally reflects on whether existing KGH
claims, based on informal education through the media, apply in a formal education environment.
Keywords
Knowledge gap hypothesis, Simple and complex knowledge, Acquisition of information, Knowledge
inequalities, Student learning
Introduction
A key pedagogical issue in teaching diverse groups of students, especially in a tertiary educational setting
currently attracting large numbers of international students from many cultures, is to determine how successfully
differing subgroups access and retain course-related information. Traditionally, successful knowledge
acquisition and retention are measured by assessment results but this does not reveal the influence of specific
factors such as prior educational attainment. The potential contribution to this issue from the knowledge gap
hypothesis (KGH) (e.g., Tichenor et al., 1970; Eveland & Scheufele, 2000) is the light it has shed on unequal
learning outcomes in informal learning such as in a community context. Distribution of information into a given
population, even with the intent to equalise disparate knowledge within it, often has the opposite outcome to that
intended. That is, inequality of knowledge among respondents either may not be reduced or is even worsened by
such information, perhaps because “information-rich” (those with better possession and command of
information) more successfully access and employ new information, in contrast to the “information-poor” (who
may lack access to information or the skills to maximise its usefulness).
The present study sought to discover if members of a large, diverse class of students, when presented with the
same information, acquired and retained that information at a similar rate, or if there were significant differences
by subgroups, and if so, what might be the nature of those differences. In particular, we wished to determine any
influence of prior educational attainment on knowledge gaps, given that much KGH research has used
respondents’ education level as a key differentiator within samples (e.g., Tichenor et al., 1970). The socio-
economic status (SES) of students was not investigated primarily because entry to university is usually based on
190
prior education not SES. In addition, many KG researchers oppose the idea that SES significantly determines
ability to learn (Dervin, 1980; Viswanath & Finegan, 1996).
Knowledge gap theory has also examined acquisition and retention of simple versus complex knowledge
(Genova & Greenberg, 1981; Ettema et al., 1983). Intuitively, one might expect that students who possessed
lesser ability or experience (such as those with less formal preparation for tertiary study) might acquire simple
rather than complex knowledge, while students of greater ability or experience would be better able to access
complex as well as simple knowledge.
A general aim was to assess whether the KGH findings identified in mass media studies (e.g., Dervin, 1980,
Gaziano, 1997) also apply to the tertiary education experience. Considering the role of education as a
discriminating factor in informal learning contexts, the absence of KGH-specific research based within formal
learning environments, e.g., tertiary education, is surprising. A catalyst for this study was some researchers’
observations (Chew & Palmer, 1994; Weenig & Midden, 1997) that as prior education influences informal
learning, formal learning environments should also be explored to see if new information worsens or improves
the knowledge inequities based on pre-entry education.
Literature Review
Knowledge gap theory arose from attempts to evaluate public information campaigns (e.g., public affairs and
health education) where, despite campaign designers’ intentions and best efforts, the public did not become
equally informed. The theory evolved as researchers tried to explain these unexpected results and tested their
theories in different contexts (Viswanath, Kahn, Finnegan, Hertog, & Potter, 1993; Weenig & Midden, 1997). A
contentious aspect of KG theory is that those at the lower end of the socio-economic spectrum do not acquire the
same level of knowledge as those in upper socio-economic groups and, moreover, attempts to equalise
knowledge gaps within a community by releasing information into it, in fact may widen rather than lessen
knowledge gaps (Viswanath & Finnegan, 1996). The implication, to which many researchers have taken
exception, is that socio-economic status (SES) strongly affects ability to learn (Dervin, 1980; Viswanath &
Finnegan, 1996). Their unease is justified particularly because in previous research SES has often been inferred
on educational attainment rather than financial or status criteria. Key elements in the KGH are its assumption of
“the initial existence of a gap in knowledge between society’s ‘haves’ and ‘have-nots’’’ (Eveland & Scheufele,
2000, p.216) and mass media campaigns’ tendency to exacerbate pre-existing gaps. Using education level to
indicate socio-economic status (SES), Tichenor et al.’s (1970, p.159) landmark study named unequal uptake of
information disseminated by the media, as a ‘knowledge gap’ and observed that “increasing the flow of news on
a topic tends to greater acquisition of knowledge on that topic among the more highly educated segments of
society”. Importantly, knowledge gaps are defined as “differentials in information acquired and retained by
people through a learning process” (Gaziano, 1997, p.238). Much research has sought to isolate reasons for
knowledge gaps. Gaziano’s 1983 survey (cited in Chew & Palmer, 1994) and Viswanath and Finnegan’s (1996)
review of 25 years of research showed conflicting evidence: some studies demonstrated widening gaps, others
showed narrowing gaps, while in others the gaps remained.
Dervin (1980, p.76) observed that the KGH undermined the assumption “that the presence of information serves
as an equaliser”. Moreover, Haywood (1995, p.65) contends that “most publicly financed education systems
have to assume that we all start from the same point” despite evidence that this does not hold for the rest of the
population (Mastin, 1998). This paper acknowledges concern in the literature (Chew & Palmer, 1994; Weenig &
Midden, 1997) that if education is important in informally acquiring new information then education processes
need investigation to ascertain any presence of knowledge gaps, with particular reference to acquisition and
retention of simple versus complex knowledge in relation to prior educational attainment.
Simple and Complex Knowledge – Acquisition and Retention

At issue in formal education is the complexity of information offered for learning. Indeed, this dimension has
received some attention in informal community learning. Studies have considered the type of knowledge gained,
differentiating between simple or superficial knowledge of facts, and complex, structural or deep knowledge of
relationships among facts (Genova & Greenberg, 1981; Ettema et al., 1983). Gaziano (1984, p.562) simplifies
these as “knowledge of” and “knowledge about”. Spitzer and Denzin (1965) show that those with the most
superficial level of knowledge used fewer, mostly interpersonal, information sources. When simple knowledge is
acquired, it is not necessarily retained over time. Price and Zaller (1993) found that knowledge retention was
191
affected by the level of prior knowledge, Snyder (1990) thought it was influenced by education level, Chew and
Palmer (1994) identified interest over education level, while Griffin’s (1990) research indicated the
interdependence of interest and level of educational attainment. In a further thread, Moore (1987), Wanta and
Elliot (1995) and Gaziano (1997) established that over time knowledge gaps between more and less educated
remain constant or narrow on simple issues, but widen on complex issues. Educators Ramsden (1992) and Biggs
(1999) confirm that “surface” learning of isolated facts is often associated with misunderstanding principles and
a weak long-term ability to recall detail. “Deep” learning, which focuses on main ideas and underlying
meanings, is associated with the integration of new and prior knowledge and the ability to apply these ideas in
new situations.
Prior Educational Attainment

Level of educational attainment is dominant in the KGH literature as a differentiating factor, following Tichenor
et al.’s (1980) assumption that education provided information processing skills and ability to see the relevance
of information, thereby accounting for knowledge gaps. Nearly all findings associate greater knowledge
acquisition directly with higher educational achievement or indirectly through higher levels of prior knowledge
that education provides. However, level of education has not influenced knowledge gain in all studies (Genova
& Greenberg, 1981). Viswanath et al. (1993) noted that education determined people’s exposure and attention to
information, a prerequisite for acquiring knowledge. Wanta and Elliot (1995, p.313) contended that “Education
influences knowledge gain since it is associated with better comprehension, retention, communication
capabilities and higher levels of previously stored knowledge”. Interestingly, Fischer et al. (1996) contend that
those in the top academic classes at school are taught that knowledge is to be used, while those in the bottom
academic classes are taught to remember what is presented to them. This is supported by Mastin (1998, p.514)
who, approaching the topic slightly differently, notes, “Individuals who do not frequently interact with
information as a problem solver, or as a knowledge enhancer often do not perceive information as a valid
commodity”. Weenig and Midden (1997) investigated whether if allowed more time, those with less education
would achieve the same level of new knowledge as those with more education, but found they did not. They
ascribed the differences to a lower degree of attention among the lower educated, rather than motivational or
cognitive differences.
Studying Knowledge Gaps in Information Systems (IS) Learning

Considering prior knowledge gap research has mainly addressed informal learning in mass communication, then,
what might be found within a tertiary environment in respect of possible gaps in formal learning practice?
Because entry to tertiary education is voluntary and relatively expensive, we assume students motivated to enrol
in IS courses in this way define their interest in the subject. Further, because IS is vocationally oriented, we also
assume that students consider the domain knowledge is useful to them. Enrolment in these classes also places
students alongside others who consider the subject relevant. A further assumption is that the opportunity to
formally explore IS courses in lectures and tutorials, as well as informal discussions with staff and classmates,
should improve the likelihood of acquiring deep, structural knowledge about the IS programme, because course
content relates to or leads onto other IS courses. For these reasons, namely students’ study motivation, their
presumed view of information usefulness, and their opportunity to discuss information with others, we assume
that university education potentially provides a more equal environment for acquiring equivalent levels of
knowledge, compared to informal community learning.
Methodology
With the introduction of a new programme of IS courses at Massey University’s Auckland campus the KGH
framework appeared an appropriate way to evaluate the effectiveness of disseminating programme information.
The classes are large (between 100 and 300), and the ability to survey students at lectures enhanced the
possibility of a high response rate. A wide range of ages and educational backgrounds are represented in each
class, and although instruction is in English, only 39% of the class speak English as their first language with 31
other ‘first’ languages spoken by students representing 28 different cultures. The dominant composite cultural
group represented is Asian (41%) with European New Zealanders, the majority in the community, a minority in
this environment (28%). This allowed a variety of factors to be explored; however knowledge acquisition and
retention did not differ greatly between these two groups. In addition, the students could be surveyed at several
points throughout the academic year, permitting a longitudinal study. In the present study the student population
192
includes graduates from secondary school, persons with some tertiary education, those with a previous tertiary
degree, and “second chance” individuals admitted to university (though not having graduated from high school)
over the age of 21 years. Therefore, students represent a wide variety of educational attainment.
The knowledge domain investigated was the programme of courses offered in the new IS programme and the
inter-relationship of courses. This research was designed to inform IS students of programme developments,
with knowledge gap theory providing the framework to measure effectiveness of these information interventions,
while also exploring the applicability of knowledge gap theory in formal education. The information was
prepared after discussion by one author with IS colleagues, and was designed to meet the University’s
expectations for what students needed to know when planning their course of study, i.e., the names and numbers
of relevant courses, what the courses are about and how they build upon one another, as specified by
prerequisites.
The investigation measured acquisition and retention of simple and complex knowledge at varying levels of
complexity using surveys at three points in second year (US-junior equivalent) IS classes over a seven-month
period. Information describing IS subject areas, and individual courses covering aspects of the discipline,
appears in university enrolment packs, handbooks and the website. In addition the information was distributed to
students during the investigation, describing a discrete set of 13 IS courses, their names, numbers and how they
related to one another. This course information is not difficult to understand, and students had completed the first
year IS course and surveyed over a period during which they were likely to be enrolled in up to four 2nd year IS
courses. Potentially all students were capable of learning all this information over the two semesters when
information was given out. Therefore, theoretically any knowledge gaps could reduce to zero as less
knowledgeable subgroups caught up. We assumed, as stated earlier, that exposure to the academic content of
courses would result in improved knowledge of the programme. In addition, the surveys captured demographic
data, information about student motivation and interpersonal communication practices.
Survey Characteristics
Two courses were surveyed, one each semester. Students were surveyed at the first and last lectures in semester
one and at a lecture halfway through semester two. The first semester course, 157.225 System Development:
Analysis (Analysis) is a compulsory course for several different subject majors in the information technology
(IT) field, with a prerequisite of the IS 1st year course. The second semester course, System Development:
Design (Design), is always smaller because although it continues on from the first semester course, it is elective
and requires a pass in its only prerequisite, Analysis. These two second year courses are prerequisites for three
third year courses (US-senior equivalent). The characteristics of the surveyed courses are summarised in Table 1.
Table 1. Survey Dates and Response Rates
Student attendance at lectures declined as the year progressed. This is not uncommon. Some students enrol but
never attend e.g., 17% did not attend the first Analysis lecture. Some students rely on the lecture notes posted on
the course website and some withdraw during the semester. Therefore, because the investigation involved three
surveys with different sized populations, N=194, N=146 and N=81, the survey groups were tested to see whether
they appeared to represent the same population. The expectation, as mentioned previously, was that the class
composition should be similar in the first and second semesters. Chi square tests employed five variables
193
(gender, age, education level, English as first language and cultural background) and supported the null
hypothesis of no significant class differences over the three surveys.
The initial survey in Semester 1 revealed a low level of knowledge about the new IS programme, so extra
explanatory material was provided in a lecture and posted to the class web site. This consisted of two diagrams,
(one simplifying the prerequisite structure, one displaying semester availability) and a description of how IS
courses comprised three main categories. The second survey at the end of Semester 1 showed an improved but
still low knowledge particularly about inter-relationships among IS courses. Therefore, in Semester 2, a diagram
of IS courses’ categorisation was provided in a lecture and added to the web site.
Defining Simple and Complex Knowledge

Knowledge of course numbers and names (e.g., 157.225 Systems Development: Analysis), was considered to be
simple factual knowledge. As students majoring in IS are required to take four courses (out of five) at 2nd year
and four courses (out of eight) at 3rd year they must make choices. Therefore, they need to understand into
which three specialist areas (described below) each course fits so they can select courses to specialise in one
area, or to select from across all three areas. These three specialist areas are: an information system’s
organisational context; the developer activities required to produce an information system; and the major
components for an information system. These three specialist areas reflect basic IS concepts which determine
how an IS course relates to or leads onto other IS courses. In addition, entry to each 3rd year course requires
passes in one or more 2nd year prerequisite courses. Both the categorisation of courses and knowledge of
prerequisite requirements for 3rd year courses were considered complex relational knowledge.
When reviewing the simple knowledge content we expected that second year IS students would easily identify
the 2nd year courses but would find identification of subsequent 3rd year courses more difficult. To reflect this
differing level of difficulty, knowledge about 2nd year courses was considered as ‘very simple’ (VSK) and
knowledge about 3rd year courses as ‘simple’ (SK). When reviewing complex knowledge content we expected
that students would gain understanding of specialist IS areas from their current 2nd year courses and therefore
find that less difficult than acquiring knowledge about entry prerequisites for 3rd year courses for their following
year. Therefore, knowledge about the three specialist areas of IS courses was defined as ‘complex’ (CK) and
knowledge about prerequisites for 3rd year courses was defined as ‘very complex’ (VCK). These decisions
appear in Table 2 and the relevant survey questions are in Table 3.
Table 2. Defining Simple & Complex Knowledge
194
Table 3. Survey Questions to Determine Student Knowledge Acquisition
Averages, Standardised Scores, Moderated Scores and Gaps

Respondents provided three kinds of information: accurate knowledge, inaccurate knowledge, or no knowledge
at all about some topics. Because inaccurate information may lead to inappropriate decisions, it was considered
misleading to evaluate only accurate knowledge. Therefore, the knowledge scores used throughout this study are
moderated scores meaning that the accurate knowledge score (gross) less the inaccurate knowledge score equals
the moderated knowledge score (net). Thus, the knowledge scores and knowledge gaps discussed here are a
conservative estimate of student knowledge, since for the sake of providing a means to calculate overall
knowledge, inaccurate knowledge has been subtracted from accurate knowledge. Standardised scores (out of
100) represent student characteristics e.g., level of knowledge.
Table 4. Defining Knowledge Scores
These decisions are summarised in Table 4. The average standardised score is used to calibrate most
comparisons not only across simple and complex knowledge types but also against prior education. The average
was chosen to indicate subgroup performance. A knowledge gap between two subgroups within the class was
determined by calculating the difference between the average knowledge scores of those groups for each survey.
Focusing on change in this difference over time reveals whether the education programme equalises knowledge
between those groups. An increase in the gap could arise from the movement of one or both groups (up or down)
with respect to each other in that period.
Exploring the Knowledge Gap Hypothesis

Initially knowledge scores and knowledge gaps were assessed for the whole class. The contours of the
knowledge scores, for the class as a whole (Figure 1), suggest two trends over the surveys. First, a systematic
decline in the level of knowledge occurs as knowledge complexity increases. Second, there is a discernible shift
upward in knowledge scores as the class received IS information over time.
195
Figure 1. Comparison of Class Knowledge Scores at Each Survey
Education Level as a Discriminator

The KGH suggests that those with higher education are likely to acquire more knowledge than those with less
education (Tichenor et al., 1970). Initially, the groups with the lowest and highest prior education were
compared. However, although the proportion in each category remained the same in all three surveys, the
absolute number who responded dropped to 5 and 15 respectively by the third survey. Consequently, we decided
to combine the two lowest and the two highest categories and to test the hypothesis by comparing those who had
received secondary education only and those with any level of tertiary education. The key issue is the extent to
which absolute and relative changes in knowledge gaps are influenced by students’ prior education (Figure 2).
Figure 2. Knowledge Scores - With & Without Prior Tertiary Education
In the first survey little difference (approximately 2 points) occurred in total knowledge scores, however the gap
widened by the third survey to approximately 7 points, as those with tertiary education acquired more total
knowledge than those with secondary education. However, the knowledge scores also showed both groups not
only retained knowledge (i.e., didn’t have less knowledge than at the previous survey), but also continued to
improve their knowledge level. Interestingly, the ability to answer questions at different levels of complexity
appeared to be associated with a corresponding higher incidence of inaccurate knowledge. The implication of
this is now considered.
196
Inaccurate Knowledge
The proportion of respondents attempting to answer the knowledge questions on each survey appears in Table 5.
Almost all respondents attempted to answer the first (very simple) knowledge question although fewer attempted
the remaining three questions. A marked increase in response levels for simple, complex and very complex
knowledge questions was noticeable between the first and third surveys although answer rates for very simple
and very complex knowledge peaked in survey 2.
Table 5. Percentage of Respondents Attempting Knowledge Questions
However, this increased response rate to the knowledge questions resulted not only in knowledge gains generally
but also in a higher proportion of people recording some inaccurate knowledge. An earlier study (Gaziano, 1997)
associated a higher level of inaccurate knowledge with those with less education. In the light of this, we
subdivided respondents into those with and without prior tertiary education to examine the percentage of each
subgroup recording inaccurate knowledge. In all three surveys, a slightly smaller proportion of the less educated
group recorded errors than those in the better educated group (Figure 3). This also revealed an unanticipated high
percentage of respondents with inaccurate knowledge, so the error rate was looked at more closely.
The average error rate was calculated by dividing the total number of errors by the number of respondents
recording one or more errors. The respondents with inaccurate knowledge in each survey were 27%, 54%, and
65% respectively.
Figure 3. Percentage of Respondents with Inaccurate Knowledge
When the error rate of respondents with inaccurate knowledge is examined against prior educational level
(Figure 4), the average error rate of the secondary educated group increased over the three surveys whereas the
error rate of the tertiary educated group increased then decreased. More importantly, the average error rate range
between both groups varies little at each survey (0.3 –0 .4), suggesting that previous educational attainment does
not strongly influence current knowledge inaccuracies. Although the group with more prior education increased
its knowledge score by 12 points compared to the increase of 7 points by the group with less prior education, as
shown in Figure 2, prior educational attainment does not affect the accuracy of knowledge acquisition.
197
Figure 4. Average Error Rate of Respondents with Inaccurate Knowledge
Discussion
This investigation differs from previous knowledge gap research in four ways. First, the population studied is
specialised, being the tertiary education sector in New Zealand and students in IS. Prior education level would
place most university students at the top of the population, and they could be regarded as ‘information-rich’.
However, within this group a range of values exists.
Second, investigating knowledge gaps in formal education represents a different environment compared to
informal learning, the main earlier focus. This formal education environment posed both challenges and
opportunities for method. Most immediately, existing tenets of KG literature must be reinterpreted for university
and IS scene in New Zealand. On the other hand, the formal environment permitted a longitudinal study,
enabling more thorough investigation than usually possible in KGH studies.
Third, the traditional ‘simple’ (factual) and ‘complex’ (relational) knowledge were extended to include ‘very
simple’ and ‘very complex’ knowledge. This was because, unexpectedly, one of the factual knowledge questions
was easier to answer, and one of the relational knowledge questions was also easier (Table 2). The ‘very simple’
scores were consistently higher than the ‘simple’ scores, despite both questions being factual knowledge not
relational knowledge.
Fourth, the longitudinal approach was successfully operationalised. Of paramount importance to the
investigation was having sufficient information about the students surveyed (in addition to their knowledge
scores) to better define characteristics of each surveyed subgroup. Survey data collected enabled a common
population across the three surveys to be statistically verified and allowed subsequent analysis to proceed in
reasonable confidence that changes in scores reflected prior education and were not aberrations resulting from
exits from and new entrants to the class.
The Knowledge Ceiling

As earlier noted, the information distributed comprised a description of a 13-course IS programme along with
details of how courses inter-related. The specifics of this information meant all students could acquire all of
these facts (i.e., reach the knowledge ceiling), so knowledge gaps among subgroups theoretically could reduce to
zero as the less knowledgeable caught up. This did not occur and there are several possible explanations.
Although courses are described by course name and number in all formal university contexts, identification of
courses within departments is much more casual. For instance, while the University Calendar may list 157.225
Systems Development: Analysis (where ‘157’ indicates an IS course), staff and students refer only to ‘225’ in
normal conversation. This informal convention may have influenced the ways students answered the two simple
knowledge questions asking for the name and number of each course. As many students provided both as
requested it is hard to know if those who gave the number only, did so because they were following this informal
convention, or whether they didn’t know or remember the course name. Consequently, it is likely that those
198
surveyed knew more simple knowledge about the courses than they recorded. This did not impact on responses
to the two complex knowledge questions because those questions asked for course name or course number.
Second, in the past many students may have enrolled in IS courses without knowing what the discipline was
about, other than “IT”. The present study grew out of a concern that students understand the content and
relevance of the IS programme to enable them to make informed enrolment choices. Therefore, the material
disseminated was prepared following collegial discussion, and it could not be more simple or less complex
without being trivial.
To encourage student responses, (i.e., to avoid concerns about additional assessment) course content was not
directly addressed in the surveys. Although the knowledge domain was descriptive material about the IS
programme rather than course content, participation in courses should also improve understanding of other
courses and how they interrelate. But knowledge about the IS programme is of course not assessed in
examinations, and in today’s pressured academic environment non-examinable material is unlikely to motivate
student study. As one mature student who answered the questionnaires said, “I’m sorry I didn’t answer all the
questions but I know where the information is and I’ll look it up when I need it.”
Access to student classes for surveys must go via the University’s Human Ethics Committee, whose permission
is granted only for research deemed of sufficient merit. The introduction of the new IS programme and the
opportunity to survey a sizeable student population over seven months encouraged the use of a comprehensive
questionnaire. Therefore, a range of issues potentially relevant to tertiary education were incorporated resulting
in surveys of 20, 25 and 19 questions respectively. This questionnaire length may have influenced some
respondents to skip the knowledge questions and to answer what may have appeared as the easier, multiple-
choice questions.
IS staff often comment on a general recent decline in student essay writing ability across all cultural groups.
Further, students strongly prefer multiple-choice tests. Therefore, a reluctance to write answers may have been as
much a factor in the low answer rate for the knowledge questions, as the length of the questionnaire. So although
reaching the “knowledge ceiling” was possible for all students, the reasons outlined above may explain why no
respondents achieved this.
Finally, the survey results were echoed by student academic performance. Although the simple knowledge
scores were low and the complex knowledge scores were even lower, these results were mirrored in the mid-
semester tests of student mastery of course content where less than half the class passed each semester’s first
test. Although these surveys evaluated a different (albeit related) knowledge domain, the students’ academic
performance is consistent with the survey results and can therefore (with caution) be extrapolated to the wider
domain.
The Influence of Prior Education

Students possess a range of educational backgrounds, from only secondary education to completed prior tertiary
education. Comparing knowledge scores of the secondary educated and the tertiary educated revealed that (with
the exception of very simple knowledge at survey 1), those with tertiary education acquired more knowledge at
all three surveys. This reinforces earlier findings (Tichenor et al. 1970; Tichenor et al. 1980; Viswanath et al.
1993) where lesser-educated people gained less knowledge. However, our results differ from others (Moore,
1987; Wanta & Elliot, 1995; Gaziano, 1997), which considered retention of simple and complex knowledge over
time (although it is unclear what time frames they used). Those studies found that gaps in simple knowledge
between more and less educated remained constant or narrowed but the gaps in complex knowledge widened. In
our study, the gaps in both simple and complex knowledge narrowed over three months but widened over seven
months. Nevertheless, both groups improved their knowledge scores at each survey, across all knowledge types
(except that the less educated did not retain their level of very complex knowledge at survey 3). So, although the
secondary educated may acquire both simple and complex knowledge more slowly than those with prior tertiary
education, they do retain that knowledge over the longer term. However, a ‘lag effect’ emerged suggesting that
the secondary educated subgroup can catch up with the tertiary educated over time. This contrasts with Weenig
and Midden’s (1997) finding that more time made no difference to knowledge levels acquired by the less
educated. Nevertheless, we found that overall, the KGH (that knowledge gaps between more and less educated
widen over time) applies to these students at specific survey points.
199
Inaccurate Knowledge and Prior Education
As this investigation progressed, more respondents attempted to answer the knowledge questions, which
mirrored the increase in knowledge levels. Two patterns appeared. Most importantly, as the percentage of
respondents attempting to answer each question rose at each survey, the percentage of each educational subgroup
committing errors also rose. Second and consequently, the range of average error rates of these ‘inaccurate’
respondent groups within each survey (0.3 – 0.4), and the range within each group across all surveys (0.5 – 0.8)
were very narrow. Therefore, in this study, educational attainment level does not appear to influence either the
proportion of the subgroup that records some level of inaccuracy or their error rate, thus does not support
Gaziano (1997) who found that those with less education have a higher level of inaccurate knowledge.
Conclusion
This paper explored two key questions. Do knowledge gaps revealed in informal education by the mass media,
also occur in tertiary education? Are university education outcomes influenced by entry education level?
Investigating effects of introducing new university courses provides tentative answers. The study transferred
knowledge gap theory from its original context of informal education of the general population, to a university
context evaluating outcomes of formal education of a more highly educated group. The findings fall into four
categories.
First, some results supported the KGH, with those possessing less prior education acquiring less knowledge than
those with more prior education. The secondary educated acquired less knowledge than the tertiary educated in
the same period, i.e., the knowledge gap widened.
Second, several results did not support aspects of knowledge gap theory. Earlier findings suggest that even when
given more time to acquire knowledge, the less educated were not able to achieve the same level as the more
educated. However, our study found that over seven months, although the tertiary educated kept ahead, the
secondary educated appeared able to match the very simple, complex and total knowledge levels of the tertiary
educated in the subsequent period. That is, at survey 2, secondary educated reached the survey 1 level achieved
by tertiary educated, and at survey 3 they had reached the survey 2 level achieved by the tertiary educated. Also,
previous studies have associated lower educational levels with acquisition of a higher level of inaccurate
knowledge. In contrast, our study detected similar levels of inaccurate knowledge across educational levels.
Third, our results conditionally support theoretical claims that those with less education did not retain complex
knowledge. In our study, both the secondary-educated and tertiary-educated groups retained complex knowledge
after seven months and very complex knowledge after three months. However, although the tertiary educated
also retained very complex knowledge after seven months the secondary educated did not.
Fourth, the study considered two new aspects of potential interest to KGH research, namely the intention of
education and the categorisation of knowledge types. Whether education is formal or informal, presumably
information disseminators intend to improve knowledge levels across the whole population. What this study
showed, however, was that although levels of simple and complex knowledge generally increased over time, the
increases were not very great. Unexpectedly, the occurrences of inaccurate knowledge also increased. But, as
importantly, the numbers of people able or prepared to respond to the knowledge questions increased. An
important aspect of knowledge acquisition and retention, therefore, is not just how high the level of knowledge is
in the surveyed population, but in addition how many of the surveyed population have been able to acquire some
knowledge.
Sub-classifications of simple and complex knowledge categories were created for this study using questions
varying in difficulty. The level of student knowledge differed dramatically between very simple and simple
knowledge. This opened up a greater level of detail about what knowledge the respondents had acquired and
retained over time. The difference in the very simple and simple knowledge scores demonstrates how different
questions can produce quite different results.
An impetus for this investigation was whether tertiary education can meet expectations of governments, when
they “profess to see educational opportunity as the key factor in developing their human capital” (Haywood,
1995, p.60), and academic leaders, who declare “our priority is to reduce any barriers to students achieving an
excellent university education” (McWha, 2001). The findings suggest that such expectations are not
straightforward. Knowledge gaps observed after informal public education by the mass media have also been
200
observed after formal education at university. Those with less prior education acquire less knowledge from the
same learning opportunities than those with more prior education at specific points in time. However, although
entry education level leads to different outcomes initially, it does appear that over time those with less prior
education can catch up. This is indeed good news – education can make a difference. Assessments, like surveys,
are snapshots in time. This study highlights how the assessment process ranks students with their peers and
recognises top achievement but does not necessarily measure or value the progress that students make in their
educational journey.
Acknowledgements
We acknowledge input from IS colleagues and support from IS students who responded so willingly to three
comprehensive surveys.
References
Biggs, J. (1999). Teaching for quality learning at university, Buckingham: Open University.
Chew, F., & Palmer, S. (1994). Interest, the knowledge gap, and television programming. Journal of
Broadcasting and Electronic Media, 38 (3), 271-287.
Dervin, B. (1980). Communication gaps and inequities: Moving toward a reconceptualisation. In Dervin, B. &
Voight, M. (Eds.), Progress in Communication Sciences Vol. II, New Jersey: Ablex , 73-112.
Ettema, J., Brown, J., & Luepker, R. (1983). Knowledge gap effects in a health information campaign. Public
Opinion Quarterly, 47, 516-527.
Eveland, W., & Scheufele, D. (2000) Connecting news media use with gaps in knowledge and participation.
Political Communication, 17 (3), 215-237.
Fischer, C., Hout, M., Jankowski, M. L., Swider, A., & Voss, K. (1996). Cracking the bell curve myth, New
Jersey: Princeton University Press.
Gaziano, C. (1984). Neighbourhood newspapers, citizens’ groups and public affairs knowledge gaps. Journalism
Quarterly, 61, 556-566, 599.
Gaziano, C. (1997). Forecast 2000: Widening knowledge gaps. Journalism & Mass Communication Quarterly,
74, 237-264.
Genova, B., & Greenberg, B. (1981). Interests in news and the knowledge gap. Mass Communication Review
Yearbook (vol. 2), Sage, Beverley Hills, CA, 494-506.
Griffin, R. (1990). Energy in the eighties: Education, communication and the knowledge gap. Journalism
Quarterly, 67 (3), 554-566.
Haywood, T. (1995). Info-rich info-poor: access and exchange in the global information society, London:
Bowker Saur.
Mastin, T. (1998). Employees’ understanding of employer-sponsored retirement plans, a knowledge gap

perspective. Public Relations Review, 24 (4), 521-534.
McWha, J. A. (2001). Message from Vice-Chancellor - August 30, E-mail to staff of Massey University, New
Zealand.
Moore, D. W. (1987). Political campaigns and the knowledge gap hypothesis. Public Opinion Quarterly, 51,
186-200.
Price, V., & Zaller, J. (1993). Who gets the news? Alternative measures of news reception and their implications
for research. Public Opinion Quarterly, 57 (2), 133-164.
201
Ramsden, P. (1992). Learning to teach in higher education, London: Routledge.
Snyder, L. B. (1990). Channel effectiveness over time and knowledge and behaviour gaps. Journalism
Quarterly, 67, 875-886.
Spitzer, S. P., & Denzin, N. K. (1965). Levels of knowledge in an emergent crisis. Social Forces, 44, 234-237.
Tichenor, P., Donohue, G., & Olien, C. (1970). Mass media flow and the differential growth in knowledge.
Public Opinion Quarterly, 34, 159-170.
Tichenor, P., Donohue, G., & Olien, C. (1980). Community conflict and the press, Beverley Hills, CA: Sage.
Viswanath, K., Kahn, E., Finnegan, J., Hertog, J., & Potter, J. (1993). Motivation and the knowledge gap: Effects
of a campaign to reduce diet-related cancer risk. Communications Research, 20 (4), 546-563.
Viswanath, K., & Finnegan, J. (1996). The knowledge gap hypothesis: Twenty-five years later. Communication
Yearbook, 19, 187-227.
Wanta, W., & Elliot, W. (1995). Did the magic work - knowledge of HIV/AIDS and the knowledge gap
hypothesis. Journalism & Mass Communication Quarterly, 72 (2), 312-321.
Weenig, M., & Midden, G. (1997). Mass-media information campaigns and knowledge-gap effects. Journal of
Applied Social Psychology, 27 (11), 945-958.
202
Tsai, C.-C. (2005). Preferences toward Internet-based Learning Environments: High School Students’ Perspectives for
Science Learning. Educational Technology & Society, 8 (2), 203-213.
Preferences toward Internet-based Learning Environments: High School

Students’ Perspectives for Science Learning
Chin-Chung Tsai
Institute of Education & Center for Teacher Education
National Chiao Tung University
1001 Ta Hsueh Rd., Hsinchu 300, Taiwan
Tel: 886-3-5731671
Fax: 886-3-5738083
cctsai@mail.nctu.edu.tw
ABSTRACT
Constructivist Internet-based learning environments are advocated by contemporary educators, but few
studies investigated students’ preferences toward the environments for learning a specific school subject
such as science. Therefore, the purpose of this study was to develop a questionnaire to explore students’
preferences toward constructivist Internet-based science learning environments. The questionnaire included
eight scales: ease of use, relevance, multiple sources, student negotiation, cognitive apprenticeship,
reflective thinking, critical judgment and epistemological awareness. The questionnaire responses were
gathered from 853 Taiwan high school students. Through factor analysis, these scales revealed highly
satisfactory validity and reliability in assessing students’ perceptions for Internet-based science learning
environments. The students’ responses also showed that they strongly preferred the Internet-based learning
environments that could connect scientific knowledge with real life situations. Moreover, female students
tended to place more emphasis on the instructional guidance offered by the Internet-based environments for
science learning, as well as the presentation of scientific knowledge in authentic contexts than did male
students. Future research and the implications for Internet-based instruction derived from this study were
also discussed.
Keywords
Learning environment, Internet, science, constructivism, Taiwan
Introduction
In recent years, numerous science educators have used computer-related tools or simulations in enhancing
students’ knowledge construction and science learning (Ardac & Akaygun, 2004; Huppert, Lomask &
Lazarowitz, 2002; Lux & Davidson, 2003; Vreman-de Olde & de Jong, 2004). Some Internet-based science
learning activities and assessment systems have also been implemented (Hoffman, et al., 2003; Tsai, Lin &
Yuan, 2001, Tsai & Chou, 2002; Wallace, 2004). In addition, many researchers have perceived Internet-based
instruction as an important way of applying the ideas of constructivism (e.g., Downing, 2001; Tam, 2000;
Taylor, Casto & Walls, 2004; Yakimovics & Murphy, 1995). Or, the pedagogy of Internet-based instruction is
quite consistent with the philosophy of constructivism (Chou & Tsai, 2002). In particular, the integration of the
constructivist thoughts and Internet-based instruction has been often applied to science education (Jonassen et
al., 1999, 2003; Tsai, 2001a, 2001b). To implement constructivist Internet-based science learning environments,
the researchers have proposed many instructional principles. For example, Tsai (2001a) has suggested
“observations in authentic activities,” “contextualizing prior knowledge,” “cognitive apprenticeship,”
“collaboration” and “multiple interpretations and manifestations” as some major principles for the constructivist
Internet-based science learning environments. In other words, constructivist Internet-based science instruction
should relate the scientific knowledge to real-life situations or prior knowledge, provide sufficient apprenticeship
to guide students’ learning, encourage student negotiation and cooperation, as well as present the scientific
knowledge in a variety of aspects. Jonassen et al. (1999) have also highlighted the presentation of the knowledge
in real problem situation or project-based learning by providing rich information resources and effective
knowledge-negotiation tools. Tsai (2001b, 2004a) has further emphasized the metacognitive and epistemological
activities involved in the constructivist Internet-based science learning environments; for example, adequate
reflective thinking, critical judgment and epistemological awareness are fairly important when navigating in the
environments where a variety of (perhaps conflicting) information concurrently exists.
Although there are some attempts on the development of the constructivist-oriented Internet-based learning
environments for science students (Tsai, 2001a) and some successful or exemplary cases have been presented
(Jonassen et al., 1999, 2003; Linn, 2003; Linn et al., 2003), few studies have explored science students’
preferences toward these environments. If science educators can have more information about students’
203
perspectives or expectations about Internet-based learning environments, they can create more favorable
environments or systems for science students.
In order to explore students’ preferences toward the constructivist Internet-based learning environments, Chuang
and Tsai (2005) and Wen et al. (2004) have developed a questionnaire to survey high school students’
preferences toward the constructivist Internet-based learning environments, called Constructivist Internet-based
Learning Environment Survey (CILES). CILES included the scales such as “ease of use,” “relevance,” “student
negotiation,” and “reflective thinking,” to assess students’ preferences toward some features of the Internet-
based learning environments. For example, the scale of “relevance” investigates the extent to which students
prefer the Internet learning environments are authentic and represent real life situations, and the scale of “student
negotiation” measures the extent to which students prefer to have opportunities to share and discuss their ideas
with other students in the Internet-based learning environments. Tsai (2004b) further modified CILES by
integrating more scales for a more complete description of the features involved in the constructivist Internet-
based learning environments. For example, the “multiple sources,” “cognitive apprenticeship,” “critical
judgment” and “epistemological awareness” as provided and promoted by the Internet-based learning
environments were addressed in modified CILES. However, CILES assessed students’ expectations for Internet-
based learning environments in general, without focusing on any learning domain. To acquire more detailed
information about students’ perspectives, it is a need to develop a similar questionnaire like CILES, but with a
clear learning domain for exploration. Therefore, this study, based upon the scales as revealed by a series of
studies related to CILES (e.g., Chuang & Tsai, 2005; Tsai, 2004b; Wen et al., 2004), attempted to construct a
questionnaire to investigate students’ preferences toward the constructivist Internet-based learning environments
for science, called CILES-S.
The scales used in CILES-S concurred with the principles of the constructivist Internet-based science learning
environments reviewed previously. For example, the scales of “relevance” and “student negotiation” were
consistent with Tsai’s (2001a) principles of “observations in authentic activities,” “contextualizing prior
knowledge” and “collaboration” as well as Jonassen et al.’s (1999) ideas about problem-based learning and
knowledge negotiation. The “multiple sources” scale was coherent with Tsai’s (2001a) principle of “multiple
interpretations and manifestations.” The scales of “reflective thinking,” “critical judgment” and “epistemological
awareness” addressed the importance of metacognitive and epistemological activities engaged in the
constructivist Internet-based science learning environments highlighted by Tsai (2001b, 2004a). Therefore, the
measurement of CILES-S was based upon relevant literatures on constructivist Internet-based instruction and it
also corresponded to the features of the constructivist science teaching.
In current stage, the Internet has been increasingly used in K-12 schools for assisting learning and instruction
about science (Bruce et al., 1997; Linn, 2003; Tsai, 2001a). Particularly, high school students may gradually
become one of major groups of learners participating in the Internet-based science learning environments (Clark
& Slotta, 2000; Lumpe & Butler, 2002). Therefore, this study explored a group of high school students’
perspectives for constructivist Internet-based science learning environments.
Method
Sample
The sample included 853 high school students (438 males and 415 females) from ten high schools in Taiwan.
The sample population was stratified into three demographic areas, Northern, Central and Southern Taiwan. Four
high schools from Northern Taiwan, three schools from Central Taiwan and three schools from Southern Taiwan
were selected. The school number ratio selected roughly corresponds to the actual high school number ratio
across these three areas. These students were tenth to twelfth graders. For each school, two or three classes were
selected. These students were in 27 classes and they were in various social-economic backgrounds and science
achievement. Among these classes, ten were in urban district; nine were in suburban district, while the remaining
eight classes were in rural region. The selected students’ age ranged from 15 to 19, with an average of 17.3-year-
olds. Although this sample could not be viewed as a national sample, the surveyed students came from a variety
of high schools in Taiwan, across different demographic areas and backgrounds, and they may, to a certain
extent, be said to represent many high school students in Taiwan. In their seventh grade to ninth grade, all of
them had enrolled the courses of biology, earth sciences, physics and chemistry as required by Taiwan’s national
curricula. All of them had experiences of using the Internet, and used on-line resources and activities for
learning. For instance, most of them had searched some on-line information to complete school science tasks or
projects.
204
In addition, similar to the process used by Chuang and Tsai (2005), before responding to the CILES-S, all of the
participating students were advised to visit some local Internet-based instructional sites and systems, which had
been nominated as the best science instructional web sites by the Ministry of Education, Taiwan, or had been
perceived as creating constructivist Internet-based science learning environments for high school students. For
example, all of them had visited the virtual physics laboratory developed by National Taiwan Normal University
(http://www.phy.ntnu.edu.tw/demolab), and the web system has provided numerous simulations, discussion
groups and animations, which build on activation of students’ prior knowledge and content knowledge.
Therefore, all of the students had acquired some understandings what a constructivist Internet-based science
learning environment was.
The development of CILES-S
The CILES-S was modified from CILES (Chuang & Tsai, 2005; Wen et al., 2004) as well as the revised version
of CILES developed by Tsai (2004b). However, each item in CILES-S investigated students’ preferences toward
Internet-based environments particularly for science learning. That is, each item in CILES was modified to the
statement for assessing students’ perceptions toward the constructivist Internet-based learning environments
especially for science. Similar to the development of CILES (Chuang & Tsai, 2005), before actual administration
of CILES-S, two experts in the field of Internet-based science instruction had commented on the items of
CILES-S for face validity, and six high school students and one high school science teacher had been chosen to
clarify the wording of each statement. Except minor wording modification, no considerable change was made
derived from this evaluation process. A detailed description for each CILES-S scale, with a sample questionnaire
item, is presented below.
Ease of use scale: measuring perceptions of the extent to which students prefer that the Internet-based science
learning environments are easy-to-use, e.g., When navigating in the Internet-based science learning
environments, I prefer that they are easy to navigate.
Relevance scale: assessing perceptions of the extent to which students prefer that the Internet-based science
learning environments are authentic and represent real life situations, e.g., When navigating in the Internet-based
science learning environments, I prefer that they show how complex real-life environments are.
Multiple sources: exploring perceptions of the extent to which students prefer that the Internet-based science
learning environments contain various information sources and interpretations, e.g., When navigating in the
Internet-based science learning environments, I prefer that they can connect to rich relevant web resources.
Student negotiation scale: assessing perceptions of the extent to which students prefer to have opportunities to
explain and modify their ideas to other students in the Internet-based science learning environments, e.g., In the
Internet-based science learning environments, I prefer that I can discuss with other students how to conduct
investigations.
Cognitive apprenticeship scale: exploring perceptions of the extent to which students prefer to have
opportunities to acquire helpful and timely guidance provided by the Internet-based science learning
environments, e.g., When navigating in the Internet-based science learning environments, I prefer that they can
provide experts’ guidance to facilitate advanced learning..
Reflective thinking scale: measuring perceptions of the extent to which students prefer to have the opportunities
to promote critical self-reflective thinking in the Internet-based science learning environments, e.g., In the
Internet-based science learning environments, I prefer that I can think deeply about how I learn.
Critical judgment scale: assessing perceptions of the extent to which students prefer to have opportunities to
critically evaluate information in the Internet-based science learning environments, e.g., In the Internet-based
science learning environments, I prefer that I can evaluate the features of various information sources.
Epistemological awareness scale: assessing perceptions of the extent to which students prefer to have
opportunities to explore the value, source, merit or nature of knowledge in the Internet-based science learning
environments, e.g., When navigating in the Internet-based learning environments, I prefer that they can explore
deeply about the nature of knowledge.
205
In this study, each scale included five items, presented in a five-point Likert mode, ranging from “strongly
agree” to “strongly disagree.” Consequently, a total of 40 items were included for developing CILES-S. Two
experts in the field of Internet-based instruction commented on the items for content validity, and six high school
students were selected to clarify the wording of each item.
Students’ responses were scored as follows. For the “strongly agree” response was assigned a score of 5, while
for the “strongly disagree” response was assigned a score of 1. Consequently, students gaining higher scores in a
certain scale showed stronger preferences toward the specific feature of the constructivist Internet-based science
learning environments.
Data Collection
Paper-and-pencil copies of CILES-S were mailed to the science teachers of the twenty-seven selected classes.
The sample students responded to the questionnaire in their science classes. The science teachers collected the
completed questionnaires and then returned to the researcher for further analysis. The total number of the
students in the classes was 973; however, only 853 students responded to the items of CILES-S. The response
rate was approximately 88%, and the valid sample in this study included 853 high school students.
Results
Factor analysis
This study utilized exploratory factor analysis, principle component analysis with varimax rotation, to clarify the
structure of the preferences toward the Internet-based science learning environments. An item was retained only
when it loaded greater than 0.50 on the relevant factor and less than 0.50 on the non-relevant factor. The high
school students’ preferences were grouped into eight factors, including: ease of use, relevance, multiple sources,
student negotiation, cognitive apprenticeship, reflective thinking, critical judgment and epistemological
awareness, presented in Table 1. These factors were exactly the same as the scales originally proposed in this
study. Nevertheless, one item in the “critical judgment” scale did not be remained as a result of the factor
analysis. The eight factors with 39 questionnaire items retained in the CILES-S explained 67% of the variance.
The eigenvalues of the eight factors from principle component analysis were all larger than one. The reliability
(alpha) coefficients respectively for these scales were 0.89, 0.78, 0.89, 0.87, 0.82, 0.90, 0.82 and 0.85, and the
overall alpha coefficient for all 39 items was 0.94. These coefficients suggested that these scales of the CILES-R
had highly sufficient reliability in assessing high school students’ preferences toward the Internet-based science
learning environments.
Table 1. The scales and item factor loadings for CILES-S

Item Factor 1: Factor 2: Factor 3: Factor 4: Factor 5: Factor 6: Factor 7: Factor 8:
Ease of use Relevance Multiple Student Cognitive Reflective Critical Epistemological
sources negotiation apprenticeship thinking judgment Awareness
Factor 1: Ease of use α=0.89
1 0.638
2 0.729
3 0.771
4 0.832
5 0.839
Factor 2: Relevance α=0.78
6 0.575
7 0.732
8 0.740
9 0.792
10 0.659
Factor 3: Multiple sources α=0.89
11 0.672
12 0.695
13 0.727
14 0.766
15 0.656
206
Item Factor 1: Factor 2: Factor 3: Factor 4: Factor 5: Factor 6: Factor 7: Factor 8:
Ease of use Relevance Multiple Student Cognitive Reflective Critical Epistemological
sources negotiation apprenticeship thinking judgment Awareness
Factor 4: Student negotiation α=0.87
16 0.563
17 0.654
18 0.774
19 0.762
20 0.723
Factor 5: Cognitive apprenticeship α=0.82
21 0.722
22 0.705
23 0.715
24 0.791
25 0.717
Factor 6: Reflective thinking α=0.90
26 0.655
27 0.733
28 0.646
29 0.788
30 0.647
Factor 7: Critical judgment α=0.82
31 0.808
32 0.804
33 0.525
34 0.537
Factor 8: Epistemological awareness α=0.89
35 0.622
36 0.766
37 0.759
38 0.799
39 0.627
Total variance explained: 67.43%, alpha for entire items, 0.94.
Students’ scores on the scales
Table 2 shows the students’ average item scores and standard deviations on the eight scales of the CILES-S.
Their average item scores on each CILES-S scale were all above the value of three, the mean of 1-5 Likert scale.
These students, on average, showed high preferences for each feature of the constructivist Internet-based science
learning environments as assessed by CILES-S. The students scored highest on the relevance scale (an average
of 4.35 per item), suggesting that they preferred that the Internet-based learning environments could help them
make more meaningful associations between real life situations and the scientific knowledge. The students also
attained relatively higher scores on the “ease of use,” “multiple sources” and “cognitive apprenticeship” scales of
CILES-S (mean = 4.13, 4.15 and 4.16 respectively). These results indicated that the students also expected the
Internet-based science learning environments to be user-friendly, to contain a variety of information sources, and
to provide useful guidance for advanced learning. However, they had the lowest item scores on the “critical
judgment” scale (mean = 3.88), implying that some students might not intend to critically evaluate the
information obtained from the Internet.
Table 2. The descriptive data on the scales for CILES-S

Scale # of Items Mean per item S.D. Range
Ease of Use 5 4.13 0.73 1-5
Relevance 5 4.35 0.46 2.4-5
Multiple sources 5 4.15 0.64 1-5
Student negotiation 5 3.95 0.67 1-5
Cognitive apprenticeship 5 4.16 0.51 2.2-5
Reflective thinking 5 4.10 0.70 1-5
Critical judgment 4 3.88 0.68 1-5
Epistemological awareness 5 4.00 0.68 1-5
207
Gender differences on the scales of CILES-S
This study further investigated students’ gender differences on the scales, shown in Table 3. The results
indicated that female students gained significantly higher scores on the “relevance” and “cognitive
apprenticeship” than male students (t = -4.23, p<.001; t = -2.03, p<.05, respectively). The female students tended
to prefer the Internet-based science learning environments where they could relate scientific knowledge to real-
life problems, and they can acquire helpful guidance from the systems, on-line experts or experienced peers.
Educators need to pay more attentions to these features when creating Internet-based science learning
environments for female students.
Table 3. Gender differences on the scales of CILES-S

Male (mean, S.D) Female (mean, S.D)
Scale t
(n = 438) (n = 415)
Ease of use -0.43
4.12 (0.70) 4.14 (0.75)
Relevance 4.28 (0.46) 4.42 (0.45) -4.23***
Multiple sources 4.13 (0.65) 4.18 (0.64) -1.34
Student negotiation 3.96 (0.67) 3.95 (0.67) 0.25
Cognitive apprenticeship 4.13 (0.49) 4.20 (0.53) -2.03*
Reflective thinking 4.06 (0.69) 4.14 (0.70) -1.81
Critical judgment 3.87 (0.68) 3.90 (0.68) -0.75
Epistemological awareness 4.00 (0.66) 4.01 (0.71) -0.14
Discussion and Conclusion

By gathering 853 Taiwan high school students’ responses, this study developed a questionnaire, CILES-S, to
investigate their preferences toward the Internet-based science learning environments. CILES-S included eight
scales, which were ease of use, relevance, multiple sources, student negotiation, cognitive apprenticeship,
reflective thinking, critical judgment and epistemological awareness. These scales, by employing factor analysis,
showed adequate validity and reliability in assessing students’ perceptions for Internet-based science learning
environments.
The students attained the highest score on the “relevance” scale, suggesting that they placed high emphases on
the connection between real life situations and scientific knowledge as offered by the Internet-based science
learning environments. Educators and web content developers should carefully present scientific knowledge in
authentic tasks and relate the scientific facts to realistic situations when designing Internet-based instruction for
science students.
This study also revealed some gender differences on the CILES-S responses. Female students, when comparing
to male students, tended to prefer the Internet-based science learning environments where they could have more
opportunities to explore science in realistic contexts and obtain helpful guidance for further learning. Science
education research has revealed that female students are often disadvantaged in learning science (e.g., Kahle &
Meece, 1994; Kenway & Gough, 1998). If Internet-based instruction is perceived as a possible way of lessening
the gender gap in learning science, educators should pay particular attentions to creating the environments that
highlight these features for female students.
CILES-S was quite different from similar instruments or questionnaires developed in the past, as CILES-S
included a particular learning topic, that is, science, for exploration. Science educators can further develop more
scales to elaborate students’ perceptions of the Internet-based learning environments for other aspects of learning
science. For example, the visualization, inquiry, or problem-based learning for science proposed by Jonassen et
al. (1999, 2003), and the stimulation of cognitive conflict for science instruction recommended by Tsai (2000a,
2001c) can be expanded to new scales of CILES-S. Furthermore, educators in other learning domains, such as
208
mathematics or language, can develop a series of similar instruments like CILES-S for their domains, and to
survey students’ relevant preferences for individual learning domains. It is clear that each domain has some
unique features, and their learning in the Internet-based environments may be quite different; thus, students may
respond differently to these surveys. Based on these results, it is expected that educators as well as web content
developers can construct more appropriate Internet-based learning environments for each knowledge domain,
and then facilitate their learning and knowledge growth.
In addition, many educators have also asserted students’ learning environment preferences, to a certain extent,
represent their epistemological beliefs (Hofer & Pintrich, 1997; Tsai, 2000b). Tsai and Chuang (2005) showed
an interplay between students’ epistemological beliefs and learning preferences for Internet-based environments.
Research findings have supported that students’ epistemological beliefs may guide their strategies of knowledge
construction and then influence their learning outcomes (e.g., Hofer, 2001; Tsai, 1998a, 1998b, 2000c).
Therefore, a better understanding about students’ preferences toward certain learning environments may be
viewed as a fundamental step for exploring their learning processes and outcomes in the environments. CILES-S
provided a useful tool for this purpose.
Finally, as proposed by Chuang and Tsai (2005) and Tsai (2004b), a parallel version of CILES-S should be
developed to probe students’ perceptions toward certain existing Internet-based science learning environments.
That is, the current form of CILES-S explored students’ perspectives toward ideal Internet-based science
learning environments, whereas the parallel version should investigate their perception toward some actual
Internet-based science learning environments. Many studies have revealed that learners’ preferences or
expectations toward the classroom environments are often different from those in realty (Fraser, 1998, Tsai,
2003). By using both CILES-S forms described above, educators and web content developers can well
understand the possible gap between students’ expectations and perceptions toward some existing Internet-based
learning environments. Additional improvements of these environments can be more effective to be undertaken.
Acknowledgements
Funding of this research work was, in part, supported by National Science Council, Taiwan, under grant numbers
NSC 92-2524-S-009-003 and NSC 93-2524-S-009-003. The author also expresses his gratitude to the Editor and
three anonymous reviewers’ helpful comments for the further development of this article.
References
Ardac, D., & Akaygun, S. (2004). Effectiveness of multimedia-based instruction that emphasizes molecular
representations on students’ understanding of chemical change. Journal of Research in Science Teaching, 41,
317-337.
Bruce, B. C., Carragher, B. O., Damon, B. M., Dawson, M. J., Eurell, J. A., Gregory, C. D., Lauterbur, P. C.,
Marjanovic, M. M., Mason-Fossum, B., Morris, H. D., Potter, C. D., & Thakker, U. (1997). ChickScope: An
interactive MRI classroom curriculum innovation for K-12. Computers & Education, 29, 73-87.
Clark, D. B., & Slotta, J. D. (2000). Evaluating media-enhancement and source authority on the internet: the
Knowledge Integration Environment. International Journal of Science Education, 22, 859-871.
Chou, C., & Tsai, C.-C. (2002). Developing Web-based curricula: Issues and challenges. Journal of Curriculum
Studies, 34, 623-636.
Chuang, S.-C., & Tsai, C.-C. (2005). Preferences toward the constructivist internet-based learning environments
among high school students in Taiwan. Computers in Human Behavior, 21, 255-272.
Downing, K. (2001). Information technology, education and health care: constructivism in the 21st century.
Educational Studies, 27, 229-235.
Fraser, B. J. (1998). Classroom environment instrument: development, validity and applications. Learning
Environment Research, 1, 7-33.
209
Hofer, B. J. (2001) Personal epistemology research: Implications for learning and teaching. Educational
Psychology Review, 13, 353-382.
Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: beliefs about knowledge
and knowing and their relation to learning. Review of Educational Research, 67, 88-140.
Hoffman, J. L., Wu, H. K., Krajcik, J. S., & Soloway, E. (2003). The nature of middle school learners’ science
content understandings with the use of on-line resources. Journal of Research in Science Teaching, 40, 323-346.
Huppert, J., Lomask, S. M., & Lazarowitz, R. (2002). Computer simulations in the high school: students’
cognitive stages, science process skills and academic achievement in microbiology. International Journal of
Science Education, 24, 803-821.
Jonassen, D. H., Peck, K. L., & Wilson, B. G. (1999). Learning with technology: A constructivist perspective,
Upper Saddle River, NJ: Prentice Hall.
Jonassen, D. H., Howland, J., Moore, J., & Marra, R. M. (2003). Learning to solve problems with technology: A
constructivist perspective, Upper Saddle River, NJ: Prentice Hall.
Kahle, J. B., & Meece, J. (1994). Research on gender issues in the classroom. In D. L. Gabel (Ed.), Handbook of
Research on Science Teaching and Learning (pp. 542-557), New York: Macmillan.
Kenway, J., & Gough, A. (1998) Gender and science education in schools: A review ‘with attitude’. Studies in
Science Education, 31, 1-30.
Linn, M. C. (2003). Technology and science education: starting points, research programs, and trends.
International Journal of Science Education, 25, 727-758.
Linn, M. C., Clark, D., & Slotta, J. D. (2003). WISE design for knowledge integration. Science Education, 87,
517-538.
Lumpe, A. T., & Butler, K. (2002). The information seeking strategies of high school science students. Research
in Science Education, 32, 549-556.
Lux, J. R., & Davidson, B. D. (2003). Guidelines for the development of computer-based instruction modules for
science and engineering. Educational Technology & Society, 6, 125-133.
Tam, M. (2000). Constructivism, instructional design, and technology: implications for transforming distance
learning. Educational Technology & Society, 3, 50-60.
Taylor, L. M., Casto, D. J., & Walls, R. T. (2004). Tools, times, and strategies for integrating technology across
the curriculum. Journal of Constructivist Psychology, 17, 121-136.
Tsai, C.-C. (1998a) An analysis of scientific epistemological beliefs and learning orientations of Taiwanese
eighth graders. Science Education, 82, 473-489.
Tsai, C.-C. (1998b) An analysis of Taiwanese eighth graders’ science achievement, scientific epistemological
beliefs and cognitive structure outcomes after learning basic atomic theory. International Journal of Science
Education, 20, 413-425.
Tsai, C.-C. (2000a). Enhancing science instruction: The use of “conflict maps”. International Journal of Science
Education, 22, 285-302.
Tsai, C.-C. (2000b). Relationships between student scientific epistemological beliefs and perceptions of
constructivist learning environments. Educational Research, 42, 193-205.
Tsai, C.-C. (2000c). The effects of STS-oriented instruction on female tenth graders’ cognitive structure
outcomes and the role of student scientific epistemological beliefs. International Journal of Science Education,
22, 1099-1115.
210
Tsai, C.-C. (2001a). The interpretation construction design model for teaching science and its applications to
Internet-based instruction in Taiwan. International Journal of Educational Development, 21, 401-415.
Tsai, C.-C. (2001b). A review and discussion of epistemological commitments, metacognition, and critical
thinking with suggestions on their enhancement in Internet-assisted chemistry classrooms. Journal of Chemical
Education, 78, 970-974.
Tsai, C.-C. (2001c). Collaboratively developing instructional activities of conceptual change through the
Internet: Science teachers’ perspectives. British Journal of Educational Technology, 32, 619-622.
Tsai, C.-C. (2003). Taiwanese science students’ and teachers’ perceptions of the laboratory learning
environments: Exploring epistemological gaps. International Journal of Science Education, 25, 847-860.
Tsai, C.-C. (2004a). Beyond cognitive and metacognitive tools: the use of the Internet as an “epistemological”
tool for instruction. British Journal of Educational Technology, 35, 525-536.
Tsai, C.-C. (2004b). The preferences toward constructivist Internet-based learning environments among
university students in Taiwan. Submitted to journal publication.
Tsai, C.-C., & Chuang, S.-C. (2005). The correlation between epistemological beliefs and preferences toward
Internet-based learning environments. British Journal of Educational Technology, 36, 97-100.
Tsai, C.-C., Lin, S. S. J., & Yuan, S.-M. (2001). Students’ use of web-based concept map testing and strategies
for learning. Journal of Computer Assisted Learning, 17, 72-84.
Tsai, C.-C., & Chou, C. (2002). Diagnosing students’ alternative conceptions in science. Journal of Computer
Assisted Learning, 18, 157-165.
Vreman-de Olde, C., & de Jong, T. (2004). Student-generated assignments about electrical circuits in a computer
simulation. International Journal of Science Education, 26, 859-873.
Wallance, R. M. (2004). A framework for understanding teaching with the Internet. American Educational
Research Journal, 41, 447-488.
Wen, L. M. C., Tsai, C.-C., Lin, H.-M., & Chuang, S.-C. (2004). Cognitive-metacognitive and content-technical
aspects of constructivist Internet-based learning environments: A LISREL analysis. Computers & Education, 43,
237-248.
Yakimovicz, A. D., & Murphy, K. L. (1995). Constructivism and collaboration on the Internet: case-study of a
graduate class experience. Computers & Education, 24, 203-209.
211
Appendix: The questionnaire items in CILES-S
Ease of use scale:
When navigating the Internet-based science learning environments, I prefer that they…
1. have an interesting screen design.
2. are easy to navigate.
3. are fun to use.
4. are easy to use.
5. take only a short time to learn how to use
Relevance scale:
When navigating the Internet-based science learning environments, I prefer that they…
1. show how complex real-life environments are.
2. present data in meaningful ways.
3. present information that is relevant to me.
4. present realistic tasks.
5. have a wide range of information
Multiple sources scale:

When navigating the Internet-based science learning environments, I prefer that they can…
1. provide a variety of relevant web links.
2. discuss a learning topic though various perspectives.
3. present a learning topic by different methods.
4. offer various information sources to explore a learning topic.
5. connect to rich relevant web resources.
Student Negotiation scale:

In the Internet-based science learning environment, I prefer that…
1. I can get the chance to talk to other students.
2. I can discuss with other students how to conduct investigations.
3. I can ask other students to explain their ideas.
4. other students can ask me to explain my ideas.
5. other students can discuss their ideas with me.
Cognitive apprenticeship scale:

1. offer timely guidance.
2. provide useful feedback to guide learning.
3. inspire valuable questions to provoke thinking.
4. provide experts’ guidance to facilitate advanced learning.
5. design interactive content to assist learning.
Reflective thinking scale:

1. I can think deeply about how I learn.
2. I can think deeply about my own ideas.
3. I can think deeply about new ideas.
4. I can think deeply how to become a better learner.
5. I can think deeply about my own understanding.
Critical judgment scale:

1. I can critically evaluate web content.
2. I can critically judge the value of different perspectives.
3. I can examine a variety of information and then make judgment.
4. I can evaluate the features of various information sources
Epistemological Awareness scale:

1. display the source of knowledge.
212
2. explore deeply about the nature of knowledge.
3. evaluate the merits of knowledge.
4. present the process of knowledge development.
5. display the hidden value of knowledge.
213
Yang, J.-C., Ko, H. W., & Chung, I. L. (2005). Web-based Interactive Writing Environment: Development and Evaluation.
Web-based Interactive Writing Environment: Development and Evaluation

Jie Chi Yang
Graduate Institute of Network Learning Technology
National Central University
No. 300, Jungda Rd., Jhongli City, Taoyuan, Taiwan 320, R.O.C.
Tel: +886 3 4227151 ext.35414
Fax: +886 3 4275336
yang@cl.ncu.edu.tw
Hwa Wei Ko
Graduate Institute of Learning & Instruction
National Central University
No. 300, Jungda Rd., Jhongli City, Taoyuan, Taiwan 320, R.O.C.
Tel: +886 3 4227151 ext.33851
Fax: +886 3 4273371
hwawei@cc.ncu.edu.tw
I Ling Chung
Department of Electronic Engineering
Ching Yun University
No. 299, Chien-Hsin Rd., Jhongli City, Taoyuan, Taiwan 320, R.O.C.
Tel: +886 3 4581196 ext.5127
Fax: +886 3 4588924
ling@cyu.edu.tw
ABSTRACT
This study reports the development and evaluation of a web-based interactive writing environment designed
for elementary school students. The environment includes three writing themes, “story pass on”, “story
chameleon” and “thousand ideas”, to encourage reading comprehension, creativity and problem-solving
skills of students. Three assessment mechanisms, expert assessment, self-assessment and peer assessment,
are also designed to provide constructive comments to foster students to review and criticize other writers’
essay, to enable students to review their own essay to find strengths and weaknesses in writing, and to
encourage students to improve their writing skills. The writing environment comprises four functional
modules – writing, assessment, tool and system management. The system was integrated with multilayer
educational services platforms, which are designed to support the establishment of online social learning
communities for K-12 students and teachers. The system logs and assessment results have been analyzed
through the system usage over two years. The results reveal that students can improve their writing skills by
participating in the writing environment, submitting many essays, interacting with other students online and
reviewing other essays. The comparison result of early and late student writing also demonstrates the
improvement of writing. Analysis of the assessment mechanism reveals that expert assessment and peer
assessment do not significantly differ. It appears that the assessment criteria proposed in this study fit the
needs of both the expert and elementary school students. It is convinced this writing environment
effectively helps students to write with designed social interaction, creative writing themes, and reflective
assessment criteria.
Keywords
Online writing, Web-based learning environment, Assessment criteria, Peer assessment, Self-assessment
Introduction
Writing represents literacy, and its teaching in the classroom is thus essential to improve students’ literacy
(Almog & Hertz-Lazarowitz, 1999). In a conventional classroom writing environment, students work on an
assigned topic within a limited period of time and accept the teacher’s guidance and interpretation on how to
write well. After receiving submissions from students, the teacher reviews works. Students generally receive
feedback only from the teacher. Writing itself is more a teacher-oriented job than a student-oriented task.
Integrating information and communication technologies into a computer-based writing environment can
enhance interactions among students and the teacher over the conventional writing environment. Studies have
214
shown that the writing with a computer rather than using pen and paper can reduce students’ errors (Grejda &
Hannafin, 1992) and increase the writing quality (Breese et al., 1996; Lam & Pennington, 1995). Studies dealing
with automated essay scoring (AES) demonstrate that employing natural language processing or artificial
intelligence techniques can provide students with appropriate feedback to automatically advise them about ways
of improving their writing skills (Burstein et al., 2004; Burstein et al., 2003; Shermis & Burstein, 2003). The
AES systems provide tools for analyzing the grammar, usage, mechanics, and discourse structure of student
essays. Most AES systems are used for large-scale assessments testing writing ability.
A web-based writing environment can improve students’ writing skills over the conventional writing
environment. For example, students can easily review and learn from each other’s work. The anonymity of the
Internet may help motivate students to review other students’ work. Additionally, various Internet features, such
as interactive discussions, enable students to interact with each other and with the teacher. Teachers can
constructively criticize students’ work. Lin (1997) summarized the advantages for a web-based writing
environment as: enabling students to inspect and learn from each other; enabling students to give and receive
feedback; enabling students to publish their work, and providing a good editing environment for students, and
providing a learning environment.
The objective of this study is to develop and evaluate a web-based writing environment encourage and improve
elementary school students’ writing. The writing environment incorporates various writing themes and
assessment mechanisms to help students to develop good writing skills. The overall system design principles
include: to enhance interaction, to provide opportunity to observe, to set up standards for writing reviewing, and
to attract students to write by providing them challenging themes. The proposed system was evaluated by
examining the system log data over two years, with assessment results and a get-together activity.
Literature review
Generally, the writing process was understood on a linear order model until 1970 (Daiute, 1985). However, this
model over-simplifies the writing process, by emphasizing the result rather than the cognitive process of writing.
Investigating the writing process from the perspective of cognitive psychology, Flower & Hayes (1980, 1981)
proposed the prominent writing process model. In this model, composing is seen as having three major
components – the composing processor, the writer’s long-term memory, and the task environment. The
composing processor includes three operational sub-processes, planning, translating and reviewing, which are
controlled by a monitor. The planning process includes setting goals, and generating and organizing ideas. The
translating process translates the writer’s ideas into words. Reviewing, including evaluating and editing the
content, is the most important part of the composing processor. Reviewing occurs, for example, when a writer is
unhappy with his work, considers alternative ideas, or does not plan his work successfully. These three sub-
processes do not occur in any particular order. A writer might be writing, moving his ideas and discourse
forward, then immediately backtrack, rereading and digesting what he had written. These sub-processes are
recursive, with one often interrupting the other, represented a shift in the understanding of the writing process
(Gillespie, 1999). The writer’s long-term memory and the task environment provide resources and stimuli for the
composing processor. Ideas in the planning process are turned into words by the translating process and then
reviewed in the reviewing process.
In contrast to the role of cognitive processes in student writing skills, some studies have investigated the
influences of social or motivational process on writing (Hidi & Mclaren, 1991; Hidi et al., 2002; Renninger et
al., 2002). Bruning & Horn (2000) suggested that four conditions are necessary to enhance the motivation to
write. These four conditions include nurturing functional beliefs regarding writing, fostering student engagement
through establishing authentic writing goals and contexts, providing a supportive context for writing, and
creating an emotional environment conducive to writing. Hidi et al. (2002) added another aspect, namely,
ensuring that children have sufficient knowledge of the contents of the topics they were asked to write about, and
examined the design procedures to improve student emotional and cognitive experiences during argument
writing.
During the writing process, the most important of all is to cultivate the reviewing skill. That is, to be able to
correct one’s own spelling, and to substitute words and phrases or even rewrites his entire work. Writing is also a
social activity, in which social learning helps individual learning, cognitive structure and interaction (Bereiter &
Scardamalia, 1987). To help cultivate students to review, observation and peer assessment in social learning
(Falchikov, 1995; Topping, 1998; Rada, 1998; Lin et al., 2001; Hertz-Lazarowitz & Bar-Natan, 2002) is adopted
to design the assessment criteria. Through interactive assessment of themselves and their peers, students will
215
have the chance to read, draw upon, correct and challenge one another’s work, enabling them to gradually reflect
upon reviewing criteria.
In writing, studies have shown that the narrative mode of writing is easier for students to understand than the
exposition mode, even though the two modes are equally complex (Graesser et al., 1980). A study has also
shown that children agreed narrative writing mode is easier than exposition writing, because children most often
read narrative texts (Tsai, 1996). For instruction purpose, in this study the authors adopt both genres.
Assessments of children’s writing have divided the criteria either of the narrative or of the exposition modes into
three main categories: mechanical (including grammatical), organization and information. Under each category,
items are listed to define the category. Items will be elaborated later in this paper.
System design principles

An expert, a well-known writer of children’s literature in Taiwan, helped design and implement the writing
environment. The expert proposed topics in three writing themes. During two-week periods, students wrote
essays on writing themes of their choice. The expert checked the website frequently, and assessed and
commented on all submitted writings.
Writing themes design
Three writing themes were designed to encourage students to improve their reading comprehension, creativity
and problem-solving skills. The first writing theme, “story pass on”, is designed to foster students’ reading
comprehension and logical thinking. Figure 1 illustrates the operational model of “story pass on.” The expert
starts a story, which students take turns to continue by adding new paragraphs. Therefore, a student must
adequately comprehend the previous paragraphs to add a new paragraph, in order to form a well-organized story.
To terminate each story on a specific date, a complete story comprises four paragraphs, meaning that a story is
completed in four turns. Hence, a new story begins every two months for “story pass on.”
Figure 1. The operational model of “story pass on”
The second writing theme, “story chameleon”, is designed to foster students’ creativity. Students are expected to
rewrite a well-known story, preserving its original essence but giving it a new flavor. Students are advised to
adapt some factors in the story such as characters, times, places and plots. The third writing theme, “thousand
216
ideas”, is designed to train students to write expository text that describes how to solve a problem. Students are
asked to write about problems in daily life and their solutions. Students are expected to propose solutions to
problems and express their thoughts expositionally. Table 1 summarizes the three writing themes and their
related characteristics.
Table 1. The three writing themes

Writing theme Mode Duration Fostering ability
Story pass on Narrative Two weeks for a paragraph, Reading comprehension &
two months for a complete story logical thinking
Story chameleon Narrative Two weeks Creativity
Thousand ideas Exposition Two weeks Problem solving
Assessment mechanisms design
This study designs three assessment mechanisms, expert assessment, self-assessment and peer assessment. The
expert assesses and constructively comments on students’ writing to improve their writing skills. The expert also
chooses good writings, showing students how to write well. The self-assessment enables students to find their
strengths and weaknesses in writing. The self-assessment process reflects the writing process model reviewing
process. The peer assessment teaches students to review writing and give critical comments to others. During the
peer assessment process, students learn to appreciate others’ writings and extend their views to write.
The criteria for writing assessment include three categories: (1) mechanical (including grammatical), (2)
organization (transition and continuation), and (3) information. In each category, three to four items elaborated
the area are listed and explained. For example, in mechanical, there are: (1-1) Spelling and grammar are correct;
(1-2) Words and phrases are not repeated; (1-3) Words and phrases used are vivid. Examples are provided
following each item. Each item is evaluated on a five-point scale score which represented by words instead of
score for easy understanding. The word descriptions are “Great! Perfect!”, “Good! But might better than this.”,
“OK! Average.”, “Not so good! Need more efforts on writing.”, “Sorry! Unacceptable.”, from higher score 5 to
lower score 1. Reviewers could also comment on each reviewed work.
The assessment criteria are explained by the writing genre (narrative or exposition). Based on the assessment
categories described earlier, assessment criteria are adopted and tested to fit elementary school level. Tables 2
and 3 list assessment criteria of the two writing modes, respectively. Moreover, an additional assessment item is
added to each writing theme to evaluate its attribute as shown in Table 4.
Table 2. Assessment criteria for “story pass on” and “story chameleon” (Narrative mode)
Category Criteria Description
Mechanical Elegant words Using appropriate words and phrases
Organization Clear paragraph Making a clear paragraph
Organization Coherence Making coherent relation from the beginning to the end of the writing
Organization Title consistent Making sure that the content of the writing is consistent with the title
Information New and original New and original ideas on writing
Table 3. Assessment criteria for “thousand ideas” (Exposition mode)

Category Criteria Description
Mechanical Correct grammar Correct use of words and phrases with correct grammar
Organization Vivid writing The style of writing is smooth and clear, the content of writing is
vivid
Organization Abundant sentence Using various words and phrases with abundant sentence
Organization Coherence Making coherent relation among paragraphs
Organization Strong beginning Providing persuasiveness sentences at the beginning
Information Reasonable elaboration Making reasonable illustrations and descriptions
Information Abundant arguments Using various examples, opinions and thoughts to express the theme
Information Original view New and original ideas
Information Simplicity Simplifying sentences to express important thoughts and making it
easy to understand
217
Table 4. Additional assessment criteria for each writing theme
Writing theme Criteria Description
Story pass on Smooth connection The plot of the story is smoothly connected to the previous paragraph
Story chameleon Originality The rewritten story is new with original ideas
Thousand ideas Problem solving The proposed solution can solve the problem
The writing environment

Figure 2 illustrates the system architecture of the web-based writing environment. The writing environment
comprises four functional modules, writing module, assessment module, tool module and system management
module. The server was implemented on the Sun Solaris operating system with an Apache Web Server using the
MySQL database to store log data such as individual student information as well as student submitted essays and
scores in the three assessments. The system was developed using the PHP and Java Script programming
languages. The system was integrated with the EduXs platforms, which are multilayer educational services
platforms designed to support the establishment of online social learning communities for K-12 students and
teachers (Chan et al., 2001; Chang et al., 2003). The system was integrated user accounts for log on to the
system with the EduXs platforms using a Cookie mechanism so that users can use a single account to log on to
different systems via the platforms. Through this integration, the online virtual learning community of the
system is linked to the real world social learning community since virtual communities are mapped to real social
communities on the EduXs platforms.
The target users of the writing environment are elementary school students. The system identifies users when
they log on to the system. Only the target users can submit their work using the system. Other users can login
and browse the web pages of the system and interact with others in the discussion boards, but cannot submit
work.
Figure 2. System architecture of the web-based writing environment
218
Writing module
The writing module provides an online writing environment for students on the three writing themes. For “story
pass on”, a complete story is composed in four paragraphs. For each writing topic during a two-month period, all
paragraphs are displayed as a four-level tree structure in paragraph order. Figure 3 presents a screenshot of the
tree structure with the corresponding English translation. Students can compose paragraph by following their
own paragraphs or connecting to others’ paragraphs, and can compose several paragraphs connected to a same
previous paragraph. By clicking any paragraph, a student can read it and decide whether to connect a new
paragraph to it. When a new paragraph is composed and submitted, it is immediately appended to the appropriate
position on the tree structure according to its content. Students can easily understand the relationships among all
paragraphs through the tree structure. For “story chameleon” and “thousand ideas”, all works must be submitted
during a half-month period. Therefore, the two writing themes have only a single level for the tree structure in
the order in which work is submitted.
Figure 3. A screenshot of the tree structure (left) with the English translation (right) for “story pass on”
Assessment module
The assessment module provides user interfaces for the three assessment mechanisms. When a work is
submitted, links to the three assessment interfaces are displayed. In the expert assessment interface, the expert
assesses work by giving constructive comments and scores. The good writings are labeled ‘good’, which serve as
samples for other students to read and reflect. Self-assessment and peer assessment use the same interface. The
system determines the form of assessment from the student’s account. A web page summarizes the results of the
three assessments with the assessment criteria. Students can look at the scores and comments by the expert and
peer(s), and compare these results with those from self-assessment, enabling them to further reflect on their
work. To make assessments reliable and valid, students cannot view the expert and peer assessments before they
review a work. And the reviewer’s name is not displayed.
Tool module
The tool module provides tools to facilitate and motivate students’ writing. The system provides four tools.
¾ Thesaurus: Most published thesauruses are designed for adults, with words and phrases explained in ways
which children might not understand. Therefore, children cannot clearly distinguish words and phrases using
adult thesauruses. In this study, a thesaurus was built with common words and phrases for elementary school
students, including a total of 4,346 common words (Chinese characters) and 8,331 common phrases. The
common words are made based on the announcement of “common words for elementary school students” by
the Taiwanese Ministry of Education, while the common phrases were found from studies in which students
make their phrases from the common words, and the common phrases are determined by the frequency of
use (Ko & Yin, 1990; Ko et al., 1990). The thesaurus also provides synonyms and antonyms for common
phrases. Students can use the thesaurus to search for phrases from a common word, and for synonyms and
219
antonyms from a common phrase. Thus, the thesaurus teaches students new words and phrases to enrich
their writing.
¾ Personal data management interface: This tool enables users to manage their personal data, which include
individual bulletin boards, discussion boards, writing records and registered data. The individual bulletin
board displays system messages and self- and peer assessment results. Once a student submits his work, he
can request a manager of an individual discussion board to lead a discussion topic related his essay topic.
The system provides an interface for students to manage their own discussion boards enabling them to focus
on the discussions, motivating students on writing topics. Each individual writing record displays a
particular student’s essays, enabling anyone to review his previous work. Individual registered data, such as
nickname, status, name of school, grade, telephone and address can be edited.
¾ Searching interface for good writing: The system provides a searching interface for students looking for
good works. Students can search on various items including writing theme, topic, duration, writer, school
and grade. The content of writings of the hit results are shown.
¾ Ranking boards: A ranking board can help motivate learners use a system. The system contains six ranking
boards to encourage students to participate in the writing community. The boards display high-achieving
schools’ and students’ names and achievements. The diligence ranking board ranks students by the number
of works. The school ranking board ranks schools by the number of works. The good writing ranking board
ranks students by the number of works assessed by the expert as good. The peer assessment ranking board
ranks works by peer assessment. The reviewer ranking board ranks students by the number of works
reviewed. Finally, the ‘discussion board’ ranking board ranks students by the number of messages they post
on the discussion board.
System management module
The system management module provides system resources management interfaces and developer and expert
settings. This module manages student accounts, bulletin boards, discussion boards and ranking boards. The
expert also uses this module to publish and maintain the three writing themes, for example, by publishing new
writing topics, deleting inappropriate writings and replying to questions in the discussion boards.
Comment from the expert
The expert individualized comments for students. The comments can be divided into the following categories.
1. Indirect encouragement – “Wow, it takes a tremendous trouble to show the idea. Please keep on this
interesting topic.”
2. Bringing up concrete comments –
¾ “Presented the reasonable cause and effect, it is very persuasive.”
¾ “It never occurred to me that you would connect the conclusion with the original topic in such a
sharp way. GOOD JOB!”
3. Suggestions for improvement –
¾ “There are lots of ways to present an abstract idea, try to make it more readable.”
¾ “Writing is not like keeping a diary, make ways for readers is one of the responsibilities of a
writer.”
4. Indirect clarification of moral values –
¾ “It is a very special ending. If you were the person, would you choose to wander in the dream for it
is a beautiful dream to experience the reality… to accomplish yourself?”
¾ “You do not have to sacrifice others, you can create the chance of win-win situation.”
¾ “It is a very positive and active attitude, using the viewpoint to look yourself, your life, and the
world will be just wonderful. You will feel happy too.”
The expert employed a reader’s tone rather than a teacher’s tone to make comments. For example, “Please
present your thought by paragraphs instead of a single block, it is tiring to read without paragraphs. Remember
to segment your writing when submitting next time, is that ok for you?” Sometimes the expert would use a more
serious tone to indicate a lack of progress here, for example, “The above two paragraphs are very interesting.
Compared with them, this paragraph seems rather dull.” However, the students expect feedback from the expert.
After submitting a work, a student often left messages for the expert, like, “Read and give comments, I have sent
my writing for story pass on.”
220
Evaluation
Since this writing environment is not compulsory, students could come and go as they wish. It is not easy to
carry out a formal evaluation of its function. Nevertheless, the authors could observe the students behaviors on
Internet to see: their frequency of submitting writings, their frequency of assessing other’s writings, the content
of discussion board. All these behaviors could illuminate students’ interest in our design. In particular, the
authors could also examine if the design principles were caught on by the students.
This section describes three analyses on the system usage, the system log analysis, assessment result analysis
and description of a get-together activity. The system log analysis describes the system usage from the system
log file in various perspectives, in particular including a comparison of early and late student writing. The
assessment result analysis describes results of analyzing the three assessment mechanisms. These two analyses
were performed in two years from November 1, 2002 to October 31, 2004. The get-together activity was a face-
to-face activity for schoolteachers and students involved in the writing environment. Students’ experience and
feedback on the writing environment were also analyzed.
System log analysis
Registered users
In total, 3,695 users were registered on the system, of which 2,510 were elementary school students from 257
different schools. The students were invited by schoolteachers or parents, or found the site themselves while
surfing the Internet. The distribution of elementary students for each grade was: grade 1: 68, grade 2: 75, grade
3: 220, grade 4: 766, grade 5: 714, and grade 6: 667 (The number shows the distribution of current users. The
system had operated for two years and some original elementary school students became junior high school
students. Their writings are not included in the following results). Thus, more higher-grade students than lower-
grade students registered for the writing environment. Most of the users were found to be fourth to sixth graders.
Submitted students and writings
Table 5 shows the numbers of submitted students and writings with writing themes and grades. More students
submitted work to “story chameleon” than to the other two themes; while more works were submitted to “story
pass on” to the other two themes. The submission ratio can be calculated from the numbers of submitted writings
to students. The submission ratio was found to be highest in “story pass on”, even though this theme did not
have the most submitting students. The reason could be attributed to the following. A complete story in “story
pass on” has four paragraphs. Students continue to write their paragraphs to make a complete story because they
are interested in the story’s progress. This finding demonstrates that “story pass on” effectively encourages
students to write. Additionally, although “story chameleon” has the most submitting students, it has the lowest
submission ratio, perhaps due to the task complexity for requiring elementary school students to think creatively
within an old framework. It is not an easy job for them.
In terms of grade, first and second graders show low submitted student numbers and submission ratio, while
third graders show low submitted students but high submission ratio. By contrast, fourth graders, though forming
the largest number of submitted students, have a low submission ratio; fifth and sixth graders are comparatively
high in both the number of submitted students and submissions, as well as the submission ratio. Sixth graders
were found to have the highest submission ratio. Further analysis of the system log data shows that some fourth
graders were assigned by teacher to complete their homework using the system, leading to the large number of
submitted fourth-grade students. However, most fourth-grade students only submitted once and did not carry on
writing. By contrast, fifth and sixth graders were found to be interested in writing and used the system
voluntarily. These findings indicate that those who write are those who are motivated by interests in this writing
environment. Those who are required by assignments will not stay long.
The ratio between writing themes and graders show that first, second and third graders like to use “story
chameleon”, while fourth, fifth and sixth graders prefer “story pass on”. This result implies that lower graders
are interested in rewriting stories, while higher graders prefer to pass on stories and are willing to keep on
writing.
221
Table 5. Submitted students and writings*
Story pass on Story chameleon Thousand ideas Total
Grade Student Writing Ratio Student Writing Ratio Student Writing Ratio Student Writing Ratio
1 12 15 1.25 6 10 1.67 10 14 1.40 24 39 1.63
2 4 4 1.00 4 5 1.25 1 1 1.00 8 10 1.25
3 20 27 1.35 22 58 2.64 24 52 2.17 43 137 3.19
4 105 179 1.70 111 163 1.47 82 135 1.65 210 477 2.27
5 79 217 2.75 99 198 2.00 87 194 2.23 176 609 3.46
6 71 209 2.94 68 120 1.76 81 151 1.86 133 480 3.61
Total 280 651 2.33 297 554 1.87 272 547 2.01 567 1,752 3.09
*
Since the same student might submit to different writing themes, the sum of numbers of students in three writing
themes does not equal to the total number of the three individual numbers. Additionally, a student may not have
made a submission in over a year during which, the student has moved up a grade placing his submission into a
different grade. Therefore, the sum of students in each grade does not equal to the total number of the six
individual numbers.
Good writings
Table 6 shows the numbers of submitted writings and “good writings” as chosen by the expert with writing
themes and grades. The result shows that over 70% of the works were considered to be good. As for the writing
themes, works in “story chameleon” were less likely to be considered good than those in the other writing
themes, perhaps because of the creativity required in rewriting making it very tough for most students. Writings
in “thousand ideas” were mostly considered to be good, indicating that students have good solutions for daily
life problems. Although the writing mode for “thousand ideas” is expositive, which is more difficult than
narrative writing for elementary school students, the finding demonstrates that writing about daily life problems
and solutions can prepare students for writing exposition.
As for the grades, generally the higher the student’s grade, the better the writing. Quality and student’s grade
showed a positive correspondence. The ratio of good writings and of submission from each grade was also found
to be positive. The result indicates a positive correlation between students’ submission ratios and their
performance in writing using the system.
Table 6. Submitted and good writings

Story pass on Story chameleon Thousand ideas Total
Grade Writing Good Ratio Writing Good Ratio Writing Good Ratio Writing Good Ratio
1 15 7 46.67% 7 3 42.86% 17 8 47.06% 39 18 46.15%
2 4 2 50.00% 5 2 40.00% 1 0 0.00% 10 4 40.00%
3 27 18 66.67% 58 33 56.90% 52 38 73.08% 137 89 64.96%
4 179 106 59.22% 163 84 51.53% 135 103 76.30% 477 293 61.43%
5 217 159 73.27% 198 124 62.63% 194 175 90.21% 609 458 75.21%
6 209 167 79.90% 123 90 73.17% 148 126 85.14% 480 383 79.79%
Total 651 459 70.51% 554 336 60.65% 547 450 82.27% 1,752 1,245 71.06%
Comparison of early and late student writing
To further examine whether students’ writing skills improve, the authors compared early and late student
writing. The data used for analysis were the scores evaluated by the expert to the essay of each student. Before
the analysis, the data was pre-processed as follows. Items for students who had only submitted one essay were
222
deleted, because they could not be used for comparison. Items for students who had submitted essays only within
two weeks (two months for “story pass on”) were also deleted. That is, the first and the last essays are submitted
within two weeks (two months for “story pass on”) were not used for the analysis because the duration for one
topic in the writing environment is two-week long (two months for “story pass on”). Scores for early writing
were calculated based on the average of scores of early student writing within two weeks (two months for “story
pass on”), and scores for late writing were calculated based on the average of scores of late student writing
within two weeks (two months for “story pass on”). Following the pre-processing, a T test analysis (2 tailed) was
performed to compare the scores for early and late student writing. Table 7 summarizes the comparison of the
early and late student writing for the three writing themes. The comparison results show significant differences
between early and late student writing for “story pass on” (t=-2.936, df=43, p=0.005<0.01), and for “thousand
ideas” (t=-2.642, df=41, p=0.012<0.05). These results demonstrate that students have improved their writing
skills by using the writing environment. However, no significant difference was found for the result for “story
chameleon” (t=-1.305, df=37, p=0.200>0.05). The reason for this lack of any difference may be the fact that
rewriting is a creative task which is very tough for most students, as mentioned earlier.
Table 7. Comparison of the early and late student writing for the three writing themes
Story pass on Story chameleon Thousand ideas
Early writing Late writing Early writing Late writing Early writing Late writing
N 44 44 38 38 42 42
Mean 3.43 3.60 3.46 3.56 3.52 3.71
SD 0.28 0.30 0.36 0.38 0.38 0.29
Ranking boards
Table 8 shows the analytical results the ranking boards. The table lists the students’ accounts (in abbreviation)
with their corresponding ranking numbers in different boards. Each ranking board measures a different
characteristic of the student or school. The diligence ranking board ranks students by number of works
submitted. The good writing ranking board ranks students by the number of good works submitted. The peer
assessment ranking board ranks students by favorable reviews obtained for their works. The reviewer ranking
board ranks students by number of reviewing comments made. The ‘discussion board’ ranking board ranks
students by contributions to online discussions. The results of Table 8 clearly show that the rankings of names on
different boards were similar, indicating that more writing, reading, reviewing, discussing and interacting lead to
good writing. In other words, writing is improved through involvement in the writing environment, making
many submissions, interacting with other students online and reviewing other works.
Table 8. Analytical results of the ranking boards*

Ranking Diligence Good writing Peer assessment Reviewer Discussion board
board
Ranking
1 FK (76) FK (64) FK (62) S4 (78) JO (102)

2 OU (49) OU (43) OU (47) RI (59) FK (76)
3 A0 (40) SA (37) SA (45) OU (50) CL (63)
SA (40)
4 PE (35), S4 (35) A0 (34) JO (30) JO (48) OU (48)
5 JO (31), ZO (31) PE (31), S4 (31) A2 (29) SA (42) A2 (47)
6 LK (30) ZO (29) DY (27) S8 (40) PE (42)
7 A2 (29), AN (29) JO (27) LK (26) S7 (35) ST (30)
8 DY (26) A2 (25) A0 (22), GI (22) LK (34), N5 (34) ZO (26)
9 CL (23) AN (23), CL PE (21) DY (31), LI (31), N4 (22)
(23), DY (23) F1( 31)
10 KR (21), RI (21), LK (20), RI (20), S4 (20) ST (29) YG (20)
ST (21) ST (20)
*
In the column, the students’ accounts (in abbreviation) are listed on the left, and the numbers in the parentheses
show the number of writings and the frequency of participating.
223
Submitted time
Figure 4 plots the number of writings against time submitted, and shows two peaks in the graph. The first peak
locates at 1 pm to 5 pm, when the students are at school, and the second peak locates at 8 pm to 10 pm, when
they are at home. Clearly, the students use the system for writing both at school and at home.
Figure 4. The number of writings and submitted time
Discussion board
From the discussion board content, the responses of students using the system show they got the essence of our
design. The content can be divided into four categories (illustrated by quotes from students).
1. Writing topics
Typical school writing is considered to be formal, conservative and rather dull, causing difficulty in
arousing students’ interests. Students found topics given by the expert to be much interesting, unique and
challenging than typical school topics. Examples include: “… Maybe that’s because the topics here are more
creative! Usually the topics from the school are rather dull!”; “I cannot agree more. The topics from the
school are pretty much the same and I am fed up with it. I can even guess the topic. I prefer the topics here
because I can write anything I want. It’s a place for your imagination to run wild.”
2. Writing appreciation
Students here can learn something by appreciating others’ works. Examples include “The way she describe
it is impressive, …” “I think you did a great job here,” and “You are fabulous!”
3. Writing community
Students can enhance their social interaction, improve their ways of thinking and discuss their future goals
through discussion and interaction by themselves on the discussion board. For example, Student A wrote
“The topic for story pass on was difficult. I am good at writing the story on people…well, I think I would
give it a try. I am really not good at this, though.” Student B then wrote “It takes me like forever to think of
this topic…after all I still want to pass it on.” However, student C wrote “I feel quite easy as long as I think
of it as a person. Everything is somewhat related to human beings after all! Don’t give up!”
4. Reflection
Students occasionally doubt their writing skills, and seek assistance on the discussion board. They ask
questions such as “What is the level of my writing skills?” “What topic is more suitable for me?”
224
Assessment result analysis
Result of self-assessment
Table 9 shows the numbers of submitted writings, self-assessment and peer assessment. The result shows that the
ratio of self-assessment to submitted writings is lower than 40% as a whole. In writing themes, “story pass on”
has the highest ratio and “story chameleon” has the lowest ratio in self-assessment. This finding could be due to
the lowest submission ratio for higher graders in “story chameleon”, as described earlier. Self-assessment is
dominated by the higher graders. Fourth graders submitted more self-assessment than fifth graders as they form
the highest number of submitted students.
Table 9. Submitted writings, self-assessment and peer assessment

a. by writing themes
Writing theme Submitted writing Self-assessment Ratio Peer assessment Ratio
Story pass on 651 287 44.09% 466 71.58%
Story chameleon 554 174 31.41% 372 67.15%
Thousand ideas 547 193 35.28% 317 57.95%
b. by grades
Grade Submitted writing Self-assessment Ratio Peer assessment Ratio
1 39 7 17.95% 15 38.46%
2 10 2 20.00% 4 40.00%
3 137 46 33.58% 80 58.39%
4 477 176 36.90% 380 79.66%
5 609 205 33.66% 370 60.76%
6 480 218 45.42% 306 63.75%
Total 1,752 654 37.33% 1,155 65.92%
Result of peer assessment
The ratio of peer assessment to submitted writings is over 60% as a whole which is higher than that for self-
assessment. This finding shows that students are interested in assessing their peers and often reviewed and
assessed other students’ works. In writing themes, the ratio is the highest in “story pass on” and lowest in
“thousand ideas”, probably because most of the works in “thousand ideas” are short, but assessment criteria are
relatively long. It might make assessment judgment difficult. Moreover, higher grade students generally assessed
more writings. Fourth graders have the highest ratio because they have the highest number of submitted students.
The reliability of assessment mechanisms
Table 10 summarizes the analytical results of the three assessment mechanisms for the three writing themes.
Comparisons were done by one-way ANOVA among different groups. The results show significant differences
among expert, self- and peer assessments in the three writing themes as “story pass on”: F(2,1128)=63.75, p<.001,
“story chameleon”: F(2,781)=41.67, p<.001, and “thousand ideas”: F(2,810)=47.35, p<.001.
Table 10. Analytical result of the three assessment mechanisms

Writing theme Expert assessment Self-assessment Peer assessment
N 446 270 415
Story pass on Mean 3.49 4.09 3.39
SD 0.38 0.88 1.11
N 314 156 314
Story chameleon Mean 3.48 4.17 3.33
225
Writing theme Expert assessment Self-assessment Peer assessment
SD 0.41 0.90 1.30
N 374 172 267

Thousand ideas Mean 3.42 4.15 3.52
SD 0.37 0.86 1.22
To further understand the relationships among the three assessment mechanisms, Turkey’s post hoc test was
carried out. Significant differences arose between expert assessment and self-assessment and between self-
assessment and peer assessment (all p<.001), whereas no significant difference was found between expert
assessment and peer assessment in the three writing themes (all p>.05).
Self-assessment is different from the other two assessments. As the table clearly shows, average scores are
higher for self-assessment than for either expert or peer assessment, perhaps because students are generally
unable to identify their errors in writing. Therefore, students tend to give themselves high scores.
No statistically significant difference was found between expert assessment and peer assessment and it is
consistent among all three writing themes. This finding demonstrates that the assessment criteria proposed in this
study fits the expert and elementary school students. It indicates a certain amount of consensus and objectivity of
the assessment criteria posted by the authors.
Relationship between expert and peer assessments
To further analyze how expert and peer assessments are related, the number of works assessed by the expert and
peers was calculated. Average scores could range from 1 to 5, according to the five-point scale used in this
study. The result from expert assessments for students’ writings was divided into three groups. The group with
average scores in the range 1.00–2.33 is the low-scored group. The group with scores averaging of 2.34–3.67 is
the middle-scored group. The group averaging 3.68–5.00 is the high-scored group. First, the number of works in
each group from expert assessment was calculated, followed by the number of works assessed by peers. The
result is shown in Table 11.
The ratio of peer assessment to expert assessment in each group is highest in high-scored group, and lowest in
the low-scored group. This finding seems to indicate that most students prefer to assess the writings in high-
scored group, while those in low-scored group were less assessed by students. The results in the three writing
themes are consistent.
Above finding indicates that most students, like experts, can discriminate between good (high-scored) and poor
(low-scored) writing. The better the writings (high-scored group), the higher the ratio of students assess. This
result also indicates that students read each work, think about its content and have an implicit evaluation of it.
When the evaluation is good, then he determines whether to grade it. It clearly shows that the assessment criteria
provided in this study can assist students to appreciate good from poor writings and to learn the reviewing
process in writing.
Table 11. Relationship between expert and peer assessments

High-scored group Middle-scored group Low-scored group
Writing Expert Peer Expert Peer Expert Peer
theme assessment assessment Ratio assessment assessment Ratio assessment assessment Ratio
Story pass 79.02 54.43 34.81
on 143 113 % 305 166 % 181 63 %
Story 88.46 49.53 43.15
chameleon 104 92 % 214 106 % 197 85 %
Thousand 53.16 51.54 45.78
ideas 158 84 % 227 117 % 83 38 %
71.36 52.14 40.35
Total 405 289 % 746 389 % 461 186 %
226
Analysis on a get-together activity
Some of the online writers might be curious about the other writers and the expert for its virtual writing
environment. Though they probably have read many other students’ works, they never have the chance to meet.
They would especially like to meet the expert. Therefore, all the writers and the expert, as well as schoolteachers
were invited to attend the get-together activity. Other reasons for holding this activity were to maintain the
momentum of writing motivation, and to promote the online writing environment to schoolteachers. Students
were honored by recognizing them as most diligent writer, best writer, most popular writer, best reviewer, and
most active contributor in the discussion board. Additionally, to make this activity more meaningful, top students
had their works published in a book entitled “Challenge Writing Environment” (Ko et al., 2004).
Invited students shared their personal experiences and feedbacks with each other. Those are valuable sources to
acknowledge our goals have been accomplished, and can be divided into three categories.
1. Interesting and creative topics

¾ LI (sixth grader) – “It’s a place you can run your imagination wild. The writing topics given by the
expert are very lively and the short opening in the passage which makes me easy to pass on. I feel like
I’m playing rather than doing something boring in this writing environment.”
¾ WA (third grader) – “My favorite is story chameleon because I can use my full imagination and
creativity, and finish in one go; Story pass on is more difficult because you have to be coherent with the
topic and you have to continue four times…”
2. Interaction in the writing community
¾ LA (sixth grader) – “I want to thank every writer for helping me in this interactive writing environment.
It unconsciously improves my writing skills though it is not evident as it shows. However, through
interaction with the other students, I can feel my own improvement.”
3. Viewpoints toward writing
¾ OU (sixth grader) – “It was a difficult task for me to write an essay, especially for the school writing
class before I found out this website. While my mom found this website, I felt inspired. The more I
write, the better I feel. As long as I sit in front of the computer, I can’t stop writing. It is just wonderful.”
¾ WO (fourth grader) – “I feel writing is much easier than it was before. From the movies, extracurricular
reading, the content on the Chinese textbook and my fantasy, I can create many plots and scenes. I can
use material from the monkey in Monkey Goes West of the Chinese classics, woodpecker doctor from
the Chinese textbook and the magic power I have longed for from Harry Potter.”
¾ LI (fifth grader) – “I used to get headache when the teacher assigned the writing as our homework. But
now, I won’t feel that and I become interested in that.”
¾ LA (sixth grader) – “Because of the writing environment, I feel great improvement in my writing and
creation which make me interested in writing a lot more.”
Conclusion
This study reported the development and evaluation of a web-based writing environment, which is constructed to
be lively, interactive and reflective. The environment is targeted at elementary school students and designed with
three versatile writing themes, “story pass on”, “story chameleon”, and “thousand ideas”, to encourage students’
reading comprehension, creativity and problem solving skills through writing. Three assessment mechanisms,
expert assessment, self-assessment and peer assessment, were included in the writing environment. A well-
known children literature writer acts as an expert to interact frequently with students. Helped by the expert and
the three assessment mechanisms, students were encouraged to write, review works and comment critically to
other students’ works, and examine their works to find their strengths and weaknesses in writing.
Since most of the students voluntarily participated in this writing environment, it is difficult to carry out a formal
evaluation of students’ writing achievement. The authors could only adopt log files to examine the frequency of
all kinds of participation. The analytical results of the evaluation, based on analyzing data over two years,
demonstrate that involvement in the writing environment, making more submissions, interacting with other
students online and reviewing more of other students’ works all benefit writing. These results were confirmed by
the comparison result of early and late student writing. The study also found that no significant difference
between expert and peer assessments. It reveals that the assessment criteria proposed in this study are concrete
enough for elementary students to hold the essence of them. When students were guided with the criteria to
review other students’ writing, it is believed that they would learn to appreciate and to critique other’s writings.
In return, they would reflect upon their own writings.
227
Moreover, the writing themes and topics and social community are the important features in the system design,
which are the attractions that the students participating in this writing environment. The writing themes are lively
and challenging as participated students described. The topics allowed them to ponder, to reason, and to create.
As for the community, Bereiter and Scardamalia (1987) had stated, writing is fundamentally a social activity.
The authors designed the interactive functions to let students know that they could communicate with readers
including the expert, and believe that this factor explains why the elementary school students were willing to
write more than once, and to stay with this environment relatively long. The system log data reveals that students
continue to use the system to compose their writings on a regular time basis. The longest duration for a student
to use the system is almost two years, and the top ten students have used the system over fifteen months.
Moreover, students’ experience and feedback on the writing environment indicated that they were highly
interested the interaction in the writing community provided by the study. In particular, as a result of the writing
environment, students who previously thought writing was difficult came to feel writing it was much easier than
before.
Based on the outcome of this study, the authors suggest that many-sided topics should be chosen, and social
interactive factors should be included, when teaching writing. Furthermore, the writing assessment criteria
developed by the authors should be considered as the guidelines for writing reviewing. It helps students learn to
read and write reflectively and critically.
For future development, the authors plan to analyze in detail students’ writing behavior, and to enhance the
present system and integrate it into a classroom-based writing environment to benefit more students who are
struggling with writings.
Acknowledgements
The authors would like to thank Mr. M.H. Yang for his efforts in the system development. This study was
partially supported by a grant from Taiwanese Ministry of Education under Grant No. 92-H-FA07-1-4.
References
Almog, T., & Hertz-Lazarowitz, R. (1999). Teachers as peer learners: Professional development in an advanced
computer learning environment. In A. M. O’Donnel & A. King (Eds.), Cognitive perspectives on peer learning,
Hillsdale, NJ: Erlbaum Press, Ch 12, 285-313.
Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition, Hillsdale, NJ: Lawrence
Erlbaum Associates.
Breese, C., Jackson, A., & Prince, T. (1996). Promise in impermanence: Children writing with unlimited access
to word processors. Early Child Development and Care, 118, 67-91.
Bruning, R., & Horn, C. (2000). Developing motivation to write. Educational Psychologist, 35 (1), 25-37.
Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The criterion online writing
service. AI Magazine, 25 (3), 27-36.
Burstein, J., Marcu, D., & Knight, K. (2003). Finding the WRITE stuff: Automatic identification of discourse
structure in student essays. IEEE Intelligent Systems, 18 (1), 32-39.
Chan, T. W., Hue, C. W., Chou, C. Y., & Tzeng, O. J. L. (2001). Four spaces of network learning models.
Chang, L. J., Yang, J. C., Deng, Y. C., & Chan, T. W. (2003). EduXs: multilayer educational services platforms.
Daiute, C. (1985). Issues in using computers to socialize the writing process. Educational Communication and
Technology, 33 (1), 41-50.
228
Falchikov, N. (1995). Peer feedback marking: Developing peer assessment. Innovations in Education and
Training International, 32 (2), 175-187.
Flower, L., & Hayes, J. R. (1980). The dynamics of composing: Making plans and juggling constraints. In L. W.
Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing, Hillsdale, NJ: Lawrence Erlbaum, 31-50.
Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College Composition and
Communication, 32 (4), 365-387.
Gillespie, M. K. (1999). Using research on writing. Focus on Basics, 3, Issue D, retrieved April 2, 2005, from
http://www.ncsall.net/?id=339.
Graesser, A. C., Hauft-Smith, K., Cohen, A. D., & Pyles, L. D. (1980). Advanced outlines, familiarity, text
genre, and retention of prose. Journal of Experimental Education, 48, 209-220.
Grejda, G. F., & Hannafin, M. J. (1992). Effects of word processing on sixth graders’ holistic writing and
revisions. Journal of Educational Research, 85 (3), 144-149.
Hertz-Lazarowitz, R., & Bar-Natan, I. (2002). Writing development of Arab and Jewish students using
cooperative learning (CL) and computer mediating communication (CMC). Computers & Education, 39 (1), 19-
36.
Hidi, S., Berndorff, D., & Ainley, M. (2002). Children’s argument writing, interest and self-efficacy: An
intervention study. Learning and Instruction, 12 (4), 429-446.
Hidi, S., & McLaren, J. (1991). Motivational factors and writing: The role of topic interestingness. European
Journal of Psychology of Education, 6 (2), 187-197.
Ko, H. W., Chen, Y. W., & Wu, M. H. (Eds.). (2004). Challenge Writing Environment, Jhongli: National Central
University (In Chinese).
Ko, H. W., Lee, H., & Lee, C. F. (1990). A study of common words and words difficulty for elementary school
students: A case on sixth graders. Unpublished technical report (In Chinese).
Ko, H. W., & Yin, M. C. (1990). A study of common words and words difficulty for elementary school students:
A case on first to third graders. Unpublished technical report (In Chinese).
Lam, F. S., & Pennington, M. C. (1995). The computer vs. the pen: A comparative study of word processing in a
Hong Kong secondary classroom. Computer Assisted Language Learning, 8 (1), 75-92.
Lin, C. S. (1997). A study of computer-networked writing instruction in elementary school. Unpublished

doctoral dissertation (In Chinese).
Lin, S. S. J., Liu, E. Z. F., & Yuan, S. M. (2001). Web-based peer assessment: feedback for students with various
thinking-styles. Journal of Computer Assisted Learning, 17 (4), 420-432.
Rada, R. (1998). Efficiency and effectiveness in computer-supported peer-peer learning. Computers &
Education, 30 (3-4), 137-146.
Renninger, K. A., Ewen, L., & Lasher, A. K. (2002). Individual interest as context in expository text and
mathematical word problems. Learning and Instruction, 12 (4), 467-491.
Shermis, M. D., & Burstein, J. (Eds.). (2003). Automated essay scoring: A cross-disciplinary perspective,
Hillsdale, NJ: Lawrence Erlbaum.
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational
Research, 68 (3), 249-276.
Tsai, M. J. (1996). The effect of strategy teaching on text structure analysis for children to improve reading
comprehension and writing skill. Unpublished doctoral dissertation (In Chinese).
229
Verhaart, M. (2005). Software review: Time Engineers Software. Educational Technology & Society, 8 (2), 230-236.
Time Engineers Software

(Software review)
Reviewer:
Michael Verhaart
Faculty of Business and Computing
Eastern Institute of Technology
Hawke’s Bay
New Zealand
Tel: +64 6 974 8000
mverhaart@eit.ac.nz
Product details:
Product Name: Time Engineers
Product Category: Educational
Developer/Publisher: Ray Shingler and Valparaiso Unviersity
Website http://www.timeengineers.com/
Target group Middle and high school students grade 5th through 9th (13-17 year old)
Product Price: US$20 personal, US$500 class, US$1,200 District.
Contact: Ray Shingler
Snapshot review: (Max 5)

Ease of use 4
Ease of navigation 4
Documentation 4
Price/value ratio 3
Pedagogical foundation 4
Instructional value 3
Brief product overview

Time Engineers as supplied contained a spiral bound
book with a teacher manual, lesson plans, two CD-
ROMs (one with the software and the second with
PDF files of the documentation) and a set of
transparencies.
Minimum system requirements:

¾ Windows 95
¾ 166 MHz Pentium processor
¾ 24 MB RAM
¾ 2 x CD-Rom drive
¾ Sound card (will work without but some
instructions for tasks only given as audio files)
¾ Screen resolution: All images rendered for 640 x
480, (works fine at higher – tested using 1024 x
768)
230
¾ Installation
o Three options, run from CD, CD-Rom Play with Support files for classroom and home use installed
on hard drive (6.8MB) and Full installation takes 620 MB.
¾ De-installation
o This is achieved via an uninstall option and was very straightforward. No obvious residual files
were left on the hard disk.
¾ Costs (http://www.timeengineers.com/purchase_e-commerce.html ). As at 28 March, 2005 the following
options were available (I have rounded them up). Purchasing can be made on-line.
US$20 Home Edition single CD

US$60 Single User Teacher edition Guide + CDs
US$200 5 user Lab Pack Guide + CDs
US$300 10 user lab pack Guide + CDs
US$500 25 user lab pack Guide + CDs
US$600 75 user lab pack 2 x Guides + CDs
US$1,200 School District Unlimited License 2 x Guides + CDs
Using the system

The system can either be installed completely on a Hard Drive or be run from the CD-ROM. A brief splash
screen introducing Time engineers leads to an introductory screen shown in Figure 1.
Figure 1. Introduction Screen
The Navigate option describes the techniques used to move around time engineers, while the Preview gives a
quick video sequence to introduce Time Engineers.
231
Selecting Start, takes you to a desk in a library
(Figure 2). Clicking on the open books describes
various engineering occupations (Civil, Electrical
and Mechanical). The center pad gives a list of
courses a student should consider if they wish to
pursue a career in engineering. Selecting the arrow
begins the journey with the first task to find the time
pod. Successive scenes follow by clicking on the
navigation arrows (Figure 3). Occasionally multiple
direction arrows are displayed. If the wrong path is
taken, you take a detour, but are always redirected
onto the correct path.
Figure 2. Library Desk with career information
Figure 3. Navigation from library to binary lock
An arrow with "Interactive" indicates that the next scene will either be a movie or require some user interaction.
Apart from using the menu options displayed at the top of the screen, there does not appear to be any way to go
back one step.
Eventually you are confronted with a panel where

57 needs to be converted into a binary number
(Figure 4). After three incorrect tries a student can
use the override button. 57 is the only number to
convert. Once achieved, a further navigation
sequence takes you to the next task which is to
provide power to the time pod. This task involves
turning on switches using Boolean algebra (AND,
OR, NOT). Once the system is powered up, the next
navigation sequence takes you to controls for the
time pod.
Figure 4. Binary conversion task
232
A time period is selected (left of radar screen), then
by clicking the buttons in the sequence directed, the
time pod is powered up.
Once all the buttons are pressed, the audio
command "Enter time pod" is heard and the green
button [Enter] flashes. It seems like you have to
click the button to proceed (which I did many
times!), but actually the sequence is automatic.
To get to Egypt the first time took approximately 20
minutes.
Figure 5. Time pod control
An animated sequence follows where the time pod is activated and passes through a time vortex, arriving in
Egypt. Once the pod is locked a selection panel allows you to choose from two tasks. One of the tasks is to
decide on the amount of water required to irrigate a field by adjusting the water flow. This is illustrated in Figure
6.
Figure 6. Weirs and water flow
Clicking [Irrigate] allows you to see how well you have done. In Figure 6 I have over irrigated and the fields
are flooded (I did manage a 97.74% yield but won't give the answer away in this review!). Once the task is
complete, the second Egyptian task is to add a layer to a pyramid (Figure 7).
Figure 7. Building a layer in a pyramid
233
Here the goal is to estimate the ramp angle and workers required to build a layer of a pyramid. Unfortunately a
problem with this task is that the final target was not clear.
There are three "Temporal Jumps" Giza Egypt 2560 BC, London England 1940 and Angers, France 1238.
Tasks in London England 1940 are illustrated in Figure 8, and involve techniques for Electrical Engineering.
Figure 8. England 1940
A submarine task was displayed (Figure 8, left). Without reading the instructions it was not immediately clear
what to do. Clicking the throttle button which at the time seemed the logical thing to do started a fairly long
automatic simulation with audio indicating what was going on, but little in the way of graphics. The simulation
could improve by checking to see whether any changes had occurred, and possibly providing more interactive
scenes. Basically what was required is to set three variables; power, engines and motor HP, and a simulation
voyage of the submarine going from the US to England and how well the U-Boats were avoided occurs. After
setting the switches and crossing the Atlantic I felt I didn't really understand what happened.
The second task was to manipulate some settings on 5 radars. An audio simulation similar to that of the
submarine, of planes being shot down with some ending video sequences followed. Even though the simulation
indicated I had successfully saved London - in spite of 3 radars being destroyed - I did not feel I had been
successful, yet was directed away from the radar station, and back to the time machine!
The third temporal jump was to Angers, France in 1238, (Figure 9), and involve techniques for Mechanical
Engineering.
Figure 9. Drawbridge task
Probably the most visually appealing task was the Drawbridge. The task required setting weights and
counterbalances to open and close the drawbridge. Unlike other simulations, different animations were displayed
234
dependant on the settings. It was good to see for example the drawbridge structure break when the forces were
too great! In this case, the task had to be completed before the system allowed you to proceed.
The final task is a catapult. It was good to find in this task that the distance varies each time it is performed.
Observations
As shown in Figure 9, students are able to resume from where they left off using either the lessons selection
(shown) or the Temporal jumps.
The tasks can be completed without understanding the actual mathematics or physics involved, and in most cases
can be completed using iterative guesswork. The level of mathematics and physics described in the information
is at a senior high school level.
Figure 10, shows some of the formulae involved, Q = the weir water flow rate formula, and Preq = pulling force
required, and Wspring = work done by a spring.
Figure 10. Sample formulae
Main features and strong points

¾ The main feature is that it provides an introduction to Engineering.
¾ Educational Edition comes complete with a hardcopy and softcopy (CD) Guide, that includes lesson plans
and overhead transparencies.
¾ Quality 3D animated sequences (actually short video clips) play throughout giving a realistic transition
between scenes.
¾ The measurements and units were appropriate to the time period, giving a historical perspective to the tasks
¾ The tasks could be completed without understanding the underlying mathematics or physics, giving it appeal
to a younger audience.
¾ Information included some sophisticated formulae to give some indication as to what needs to be learnt in
the specific areas targeted by the tasks.
¾ Contact with the author was straightforward and replies to any queries were received promptly.
Criticism and suggestions

From a teaching perspective, few of the problems vary, so if one student solves the task the answer can given to
others. It would have been nice to have had a little more variation in some tasks, such as the binary task.
There was some inconsistency in the navigation. For example, in the pyramid task, clicking on the time-pod
button takes you forward , whether or not you have solved the task, whereas the drawbridge needs to be solved
before you can proceed to the time-pod.
Scroll boxes need to be set to the top. This is a common problem with applications written in Director. Also the
[Esc] button quits the entire presentation. A screen "Are you sure you want to exit?" would be useful.
My personal opinion is that the price for class set of 25 (US$500) is a little too high, given the financial
constraints placed on many educational institutions. Otherwise the personal and unrestricted copies seem to be
reasonably priced.
235
From a user perspective the World War 2 tasks could be improved, with audio the main simulation interface.
Also some indication as to the goal of the task in the pyramid layer building would be useful. The drawbridge
simulations were excellent.
Conclusion
The system achieves its objective of introducing engineering and its application to students. The user interface
has clear, quality graphics and navigation is intuitive. Images are of a high quality, as should be evident from the
screen shots displayed in this review, and a large amount of time has obviously been spent making them clear
and look realistic.
Most of the tasks can be completed by guesswork but there is supporting information that discusses that the
mathematics and physics behind the simulation. A little more "randomness" would enhance the reusability of the
tasks.
Overall, it introduces students to the various branches of engineering in an interesting and enjoyable way.
236
Lin, T. (2005). Book review: Understanding Designers Designing for Understanding (Ingrid Mulder). Educational
Understanding Designers Designing for Understanding

(Book Review)
Reviewer:
Taiyu Lin
Graduate Researcher
Advanced Learning Technologies Research Centre
Massey University, Palmerston North
New Zealand
t.lin@massey.ac.nz
Textbook Details:
Understanding Designers Designing for Understanding
Ingrid Mulder
2004, Telematic Instituut, P. O. Box 589, 7500 AN Enshchede, The Netherlands
ISBN: 90-75176-39-2
196p.
Understanding Designers Designing for Understanding is a book that provides great insight into collaborative
learning. The tango metaphor weaves through the theoretical and practical works and provides an integrating
thread for the entire book. This book sets out to examine collaborative learning in ad hoc design teams which use
video-based technologies as the communication channel. The pursuit of such goal is carried out in three main
phases: an exploratory study, a development phase of a supporting tool, and a quasi-experimental study. The
three phases will be introduced in a bit more detail in the following paragraphs.
An initial hypothesized framework on collaborative learning, which consists of Conceptual Learning, Expression
of Affect, Feedback and Questioning, was proposed by the author as a tool to explain collaboration to learn, and
formation of shared understanding in learning groups. The initial framework, which emphasized on the
Conceptual Learning and Expression of Affect, resulted in suboptimal question-answer and reflective behaviour
during the exploratory study.
The suboptimal behaviour of question-answer led to reflection on and then revision of the initial framework. The
revised framework brings Questioning and Feedback to the central position of the framework based on the
assumption that having more question-answer behaviours would yield more reflective activities which in turn
based on the assumption that reflection could lead to better result of collaborative learning. A question-answer
supporting tool called Q-tool is thereby created.
Using the Q-tool, a quasi-experiment was carried out to test whether a technology-supported (using Q-tool)
environment could boost the questioning behaviour and higher reflective thinking. Even though the final result
showed no significant difference of share-understanding between groups using Q-tool and groups not using it,
Understanding Designers Designing for Understanding offers insight into learning and understanding in video-
mediated teams and provides an starting point that could eventually lead to better understanding of the process of
collaborative learning and development of shared understanding.
237
International Forum of Educational Technology & Society
http://www.ifets.info

10 1 1 126

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

10 1 1 126

Transféré par

Droits d'auteur :

Formats disponibles

Journal of

Educational Technology &

Subscription Prices and Ordering Information

Abstracting and Indexing

Formal discussion summaries

Implementing an Innovation Cluster in Educational Settings In Order to Develop 7-15

Full length articles

Using discussion webs to Develop an Academic Community of Learners 16-39

Effects of the Cognitive Level of Thought on Learning Complex Material 40-53

Towards A Web-Based Handbook of Generic, Process-Oriented Learning Designs 66-82

Authentic Teaching as the Context for Language Learning 83-93

ICT-Pedagogy Integration in Teacher Training: Application Cases Worldwide 94-101

Teachers’ Perceptions on the Roles on Educational Technology 102-106

Hypermedia as a Cognitive Tool: Student Teachers’ Experiences in Learning by Doing 107-117

Web-based Cognitive Apprenticeship Model for Improving Pre-service Teachers’ 136-149

Instructional Uses of Instant Messaging (IM) During Classroom Lectures 150-160

Understanding Teacher Mindsets: IT and Change in Hong Kong Schools 161-169

The Effectiveness of Computer-Assisted Instruction in Teaching Introductory Statistics 170-178

Student Alienation, Academic Achievement, and WebCT Use 179-189

Acquisition of simple and complex knowledge; a knowledge gap perspective 190-202

Preferences toward Internet-based Learning Environments: High School Students’ 203-213

Web-based Interactive Writing Environment: Development and Evaluation 214-229

Time Engineers Software 230-236

Understanding Designers Designing for Understanding 237-237

Transitional Developments in Online Courses and Programs: Theory and

Moderator & Sumamrizer:

Not-For-Profit Online Courses

2. U.K. eUniversity: A Contemporary E-Learning Case Study

4. University of Phoenix Innovations: rEsource, E-CaD, and Virtual Organizations

Student Academic Expectations

Faculty Academic Expectations

5. Online Instructor Concerns

Suggested Questions for Discussion and Debate

Implementing an Innovation Cluster in Educational Settings In Order to

Moderator & Sumamrizer:

Art Schneiderheinze, Ph.D.

1. the productive aspects of collaborative professional development for innovative educators,

The CBLE unit

Design Problem Solving Model

Questions for Online Forum:

Innovation and Technology in Education

Change in Educational Settings

Brown, A. (1995). The advancement of learning. Educational Researcher, 23 (8), 4-12.

Salomon, G. (1993). No distribution without individuals’ cognition: A dynamic interactional view. In G.

Vygotsky, L. (1986). Thought and language, Cambridge: The MIT Press.

Wertsch, J. V. (1998). Mind as action, NY: Oxford University Press.

Using discussion webs to Develop an Academic Community of Learners

Renee Hayes, Ph.D.

Mary Jane Pluta

Table 1. Research questions and coding topics

Table 2. Web discussion functional failures and successes

1. Intersubjectivity: Shared focus on academic themes

3. Problematicity: Defining and addressing professional/academic inquires in individual web postings

4. Dialogicity: Academic dialogic relations among web postings of different authors

5. Motivation: Ownership of the web discussion

Method and Procedures

The Interactive Discussion Web (WebTalk)

Web Talk Pedagogical Design: Theoretical Underpinnings

Data Coding and Analysis

Findings and their Discussion

The Web as a Medium for Preservice Teachers to Discuss Educational Issues:

Figure 2. Web as a Medium for Students' Classroom Discussion