Evaluation strategies

This study is about :An Evaluation of Two Courses for Graduate-Level Professors
Designed to Improve Academic Internet Use in Research and Teaching

Study by: Anise A. G. D'O. FERREIRA <anise@mailmac.macbbs.com.br>

Catholic University of São Paulo
Brazil

Abstract
This paper discusses insights gained from the author's experience in planning and teaching two
courses aimed at improving academic Internet use in research and teaching by college teachers,
professionals, and master's degree candidates. The courses involved different educational
strategies. The first one had prescriptive goals and tasks; the second had a content-based, goal-
oriented activity, in which certain abilities, subgoals, and knowledge were implicitly required.
The evaluation of both courses was based on students' effective engagement in course activities,
measured by class attendance, dropout rate, students' reports, interaction, and performance
observations in both online and in-class activities. The results highlighted three important
influences on learning: institutional infrastructure and policies for faculty and graduate student
improvement, students' previous experience (e.g., proficiency in English reading or computer
use), and motivational engagement, which can be ascribed to a participatory and constructive
learning environment at the cost of more time-consuming interaction among participants and
with the instructor. This paper raises questions about institutional policies for faculty training,
theoretical and methodological issues in motivational aspects of the learning environment,
implications for future online courses, and their evaluation.

First course
The first course had strategies that were based upon behavioral goals and tasks related to
projected learner needs. The strategies had activities organized in units of knowledge. These
units involved lectures and practice exercises. There was some flexibility in the practice sessions,
since the content target of tasks had to be related to the participants' main area of interest and the
pace of performance had to be based on their competence in computer use. The instructor was
available to give in-class guidance and feedback during practice sessions. Pedagogically, this
teaching plan reflects, in part, some of the behaviorist principles of educational methodology,
such as previous specification of goals and tasks in sequential steps, and the belief that the
educator can control and predict required learning

At the end of the course, the participants would be expected to:

• know the types of Internet connection and access;

• use Internet communication and information service clients;
 • access a remote computer using telnet, file transfer protocol (FTP), gopher,
and World Wide Web (WWW);
• communicate using e-mail;
• subscribe to and participate in academic e-conferences;
• send and receive files through e-mail and ftp;
• search and analyze relevant academic material in databases and available
information services on the Internet; and
• draft an educational or research project using the Internet as a tool.

In order to attain the objectives, the participants were expected to:

• attend lectures and demonstrations;

• accomplish assignments: four blocks of exercises, working in pairs in the
laboratory, which had to be turned in by e-mail or in printed form, directly to
the instructor, every other week;
• answer a questionnaire; and
• turn in a final individual paper related to the last block of exercises.

Participants

Enrolled in the course were 30 male and female adults, consisting of faculty members (part-time
college professors), invited coordinators, invited librarians, and administrative employees. This
group was heterogeneous in academic level, English language proficiency, professional interests,
and familiarity with both microcomputers and the Internet. The department of technological
development wanted this opportunity to reach the greatest possible number of people.

Internet access and the available infrastructure

The lectures were offered in a classroom in which a microcomputer (personal computer; PC)
directly accessed the Internet through a 64 Kb line connecting to the Internet service provider
(ISP).This microcomputer was a Pentium PC (133 MHz, 32 MB RAM, 1 MB Video RAM)
equipped with CD-ROM drive and multimedia resources, running Win95 and the necessary
Internet clients. A datashow was used to project the Microsoft PowerPoint slides and Internet
applications during explanations of their features and academic usage. Since the connection was
neither fast nor stable, the instructor sometimes had to present the material off-line.

The institution provided Internet access on campus via direct connection, since it was not able to
provide an off campus dial-up service. An Internet laboratory, with 40 identically configured PCs
(without multimedia resources), was available for students who wanted to practice in the
afternoons. However, they preferred to use the equipment available in the teachers' room, which
was equally slow and nonstable but offered more privacy.

The invited coordinators, librarians, and administrative employees had microcomputers in their
offices with direct or dial-up connections to the Internet. Since those connections were not stable,
and some machines had maintenance problems, some preferred to use their personal equipment
and connections at home.
Procedure

The participants were given printed material that contained:

• course objectives;
• activity descriptions and accomplishment expectations; and
• copies of two Web pages (printed with URL [uniform resource locator]
addresses) built specially to provide initial references for navigation in WWW,
tutorial addresses, and software.

The course was divided into sessions and assignments:

• Eight weekly, one-and-a-half hour lecture classes conducted by the instructor

for the entire group.
o The content of the lecture sessions, based on introductory material
(PowerPoint slides) freely distributed by The included:
 Networking, the Internet, and the Internet in Brazil
 Types of Internet access and accounts; software and hardware
required
 Internet clients: WWW, e-mail, ftp, telnet, chat
 Information and communication tools and services
 Characteristics of Internet addresses
 Client server model
 Strategies of using information and communication services
o Demonstrations, tips, and explanations were provided during the
lecture classes, through online access whenever possible.
• Weekly two-hour (minimum) practice sessions for individual or pair work,
tutored by the instructor
o The practice sessions were guided by assignments, four blocks of
exercises with answer forms. Here are some examples of exercise
items:
 week 1 exercise: "Visit the information services listed in the Web
page "x" (the one printed) and select an item of information
among those which most interested you. Justify."
 week 3 exercise: "Select two academic lists and subscribe to
them; then, write down a) the address of the respective list
servers and the list addresses; b) their institutions which
maintain these lists; c) the weekly volume of information; and d)
some characteristics of the communication in these lists."
o Also, the participants had to answer two questionnaires: one after the
second week, and the other after the sixth week. Here are examples of
questions:
 from the first questionnaire: "What is the configuration of the
computer you are using in terms of hardware and software; are
there communication software and hardware?"
 from the second questionnaire: "What do you do to find an e-
mail address?" "What do you do to subscribe to a list and to
participate in a discussion group?"
 o The individual assignment was related to the fourth block of exercises
which focused on the analysis of academic information sites. The
participants had to evaluate those sites that they liked most and
propose a draft of the structure and/or organization of their ideal
information server in terms of their own academic interest.

Results and discussion

By the third week, those who had no familiarity whatsoever with computers began dropping out.
They said they had no available time to practice. The dropout rate reached 66.6 percent. Only 10
(six graduate students, two coordinators, one administrative employee, and a librarian), from the
initial 30, attended the lectures consistently. During the eight weeks, the presence of the
instructor in practice sessions was requested three times by each pair of the more assiduous
students. E-mail was also used to communicate with those participants. Nobody turned in all the
assignments. All of them complained about lack of time and blamed their unproductive days on
the nonstable infrastructure conditions. Moreover, some reported that they tried at least to attend
the lecture sessions because they felt obligated by their superiors. On the other hand, there was
some positive feedback. One of the coordinators reported having been very satisfied with being
able to find documents he needed. The librarian and administrative employee started an informal
interaction with instructor in order to solve specific problems with their professional activities on
the Internet.

While it is possible to imagine many reasons why the participants didn't satisfactorily complete
their assignments or finish the course, the obvious reasons were institutional infrastructure
problems and participants' time constraints, since some were part-time professors and had their
own businesses. Of course, one could say that a better infrastructure and paid release time would
help. In fact, two hours a week dedicated to practice falls short of learning needs. At least 10
hours a week would be necessary . The instructor noted that those who persisted in continuing
the course, despite the difficulties, exhibited common characteristics: a) engagement in activities
in which the Internet fulfilled immediate needs and b) English reading proficiency. Among those
who continued were the coordinators, senior researchers, and professionals, who experienced
immediate benefits. For the librarian and employee, the course provided insights into the use of
Internet tools. On closer inspection, it turns out that the course may have started from the false
premise that participants were equally as motivated as this latter group.

Pedagogically, one can say that the teaching strategies, based on prescribed and structured
exercises, were not adequate for all the learners. Even though the exercises were related to the
solution of a genuine problem, they were not embedded in a meaningful or relevant project for
the majority of the participants.

The importance of English reading proficiency makes sense, since most sources of relevant
information at that time were in English. Many still are, such as indexes, abstracts, publishers,
electronic lists, newsletters, and general information websites. These results demonstrated that
infrastructure and time can be powerful factors, but motivational aspects embedded in strategies
may also be powerful.
2. Second course
The second course had practically the same general objectives as the first one, focusing more on
a critical analysis of academic performance through the Internet. However, task goals and
specific contents were not previously established by the professor as in the first course. The
design took into account the remaining unsolved problems of infrastructure and students' lack of
time. In order to minimize these effects, course duration was increased, criteria for selection of
participants were established, and course content and strategies were modified; the design
became more flexible, according to students' difficulties and motivation.

This course proposed content-based, goal-oriented activity, in which many abilities and subgoals
and much knowledge were implicitly required. The final objective was to build an academic
information resource website (individually and in group) in which specific contents were
negotiated with the instructor and among the students, based on their real work and research
issues. The students, along with the instructor, evaluated and selected a free service for Web page
storage, such as Geocities, Xoom, and Starmedia, where they stored their assignments and
structured their website.

In summary, the position assumed by the instructor here gave importance to creating
opportunities for the students to perform and accomplish tasks in a genuine situation, to perform
collaborative tasks, and to reach goals according to group and individual needs and desires.

Participants

Nine graduate students, one journalist, one business administrator, one social communication
college teacher, one marketing college teacher, two teachers of English as foreign language, and
three pedagogical advisors applied for this course. The enrollment requirements were a) to have
their own Internet account and connection at home, b) to be a proficient English language reader,
c) to have basic computer skills in the operating system, word processors, browser, and e-mail
clients, and d) to have 10 extra hours a week available to practice. The students who judged that
they could not fulfill all the requirements (either because their level of English reading was poor,
or because their computer skills were inadequate) were accepted upon agreeing to work
additional hours. A student who had more experience with Internet than his classmates did
offered to help the others with technical problems. Although this student was experienced in the
use of Internet tools, he was not confident using them as academic tools.

Internet access and the available infrastructure

Aside from their own Internet access and accounts, students could use the undergraduate
laboratory in the afternoons. The undergraduate campus was situated slightly less than one
kilometer from the graduate department. The instructor could use a shared dial-up access
available in one room in the graduate department, or use her own Internet dial-up access and
account at home. The undergraduate laboratory was reserved for classes one morning every other
week. It had 40 PCs (133 MHz, 32 MB RAM, HD 2 GB, 1 MB Video RAM) connected in a
Novell local network with direct access to the Internet through a 64 MB line to the ISP. The other
sessions took place in a regular classroom on the graduate campus, with desks and a white board,
where lectures and discussions were carried on.

Procedure

The course was divided in classroom and remote activities. Sixty accredited hours were given:
15 weekly four-hour sessions with students and the instructor in a regular classroom (desks and
white board) or with students and instructors at the laboratory every other week from the sixth
session on, where some online activities took place in pairs or groups. The remote activity -- that
is, homework and online work -- was not computed as instruction hours for administrative
purposes.

The general content involved the following topics in theoretical discussions and practical
activities:

• the infrastructure and general purpose of an academic network

• client server model, Internet software and hardware, protocols
• communication and information Internet services and addresses
• search tools and engines in the WWW
• publishing tools for the WWW
• types of academic information distributed by different Internet services and
tools
• search of bibliographical data and specific academic services based on the
WWW
• the Internet as a tool and object of study
• user-interface studies and research
• evaluation of information quality on the Internet
• cyberculture and virtual communities studies
• teaching and learning studies in the Internet context
• introduction to the planning of Web-based courses
• pedagogical issues regarding Web-based classes

Classroom and laboratory activities involved:

• short instructor lectures

• student presentations
• discussion of themes and questions related to academic readings proposed
by the instructor and students
• discussion of issues raised from online activities
• planning and organization of the Web project content
• planning of instructional activities and feedback mechanisms
• writing short papers about classroom discussions and questions
• practicing and discussing the online use of Internet clients, tools, and
services.

Remote activities involved:

 • asynchronous communication through e-mail and list by the group and
instructor
• gathering of online material
• reading and analysis of the material gathered
• material preparation for Web storage purposes
• Web storage and organization of online information

The first two sessions were dedicated to planning and organizing the work routine, based on
discussions of students' research and work themes. The themes for readings were derived from
instructor-student dialogue. Readings and discussions, alternating with short lectures, involved
the selected topics mentioned above. The complexity of practice activities was negotiated with
the students during the course. In order to help initiate practice, two Web pages were provided --
one containing academic resource information addresses, the other containing guidelines for
bibliography searches.

In order to become more technically familiar with the tools, each student agreed to store
homework in individual Web pages. Then, they had to edit all the individual material
(handwritten or in HTML [hypertext markup language] format) and organize it in a collective
website according to data and criteria they had read and discussed.

Student evaluation was based on in-class and online participation and production, in both
individual and collective publication of assignments negotiated with the instructor (some in
HTML, others distributed through e-mail).

Results and discussion

Face-to-face attendance and dropout rate

There were no dropouts, although one of the students missed classes, was always 30 minutes
late, and couldn't accomplish any of the work established by the instructor and group. He
complained about his defective microcomputer and lack of time either to have it fixed or to work
on campus. The average rate of attendance was 91 percent.

Online interaction

By the fifth session, the site for the Web page storage was chosen. Since the site had a forum
service, some of the messages exchanged by the group remained at the site and could be revisited
anytime. However, not all the messages exchanged were stored at the site.

Two hundred and forty messages were sent to the group list. Thirty-one percent were sent by the
instructor, and 29.5 percent were sent by the student who was more experienced in the Internet.
He also forwarded some peer messages to the list. Three percent came from another graduate
professor who collaborated in finding course material once in a while. Thirty-nine percent were
sent by the other students, who could be divided into two groups of three: a) average-level
interaction, ranging from 0.7 to 14 percent, and b) minimum interaction with less than 0.7
percent. It is important to notice that some messages were not sent directly to the list by the
students, but were sent to classmates. As the course progressed, online interaction decreased: 41
percent of the total messages sent during the course were exchanged in the first month, 28
percent in the second, 10 percent in the third, and 5 percent in the last.

The content of these communications was mostly organizational (instructor's feedback,

comments and suggestions to all students, and students' messages to all proposing and assuming
tasks) and technical (students asking for help, instructor and students giving help). There was
student-student communication, too. At the beginning of the course, the numbers showed an
active list. As soon as the course evolved, the list activity decreased. This has a simple
explanation. The individual tasks decreased and collective work was required in order to
accomplish the collective site; the greater part of these messages were sent directly to the
classmates with whom the student was sharing the activity.

The level of remote, online interaction was less than that expected by the instructor, since the
group had agreed that part of the theoretical discussions and work would be distributed to all in
the class list. The students took more time to solve technical difficulties, and they didn't have
time to deal electronically with the material gathered in the interval between classes. So, they
printed the material, read it at home, and discussed it in the classroom.

Performance observations and reports

The assignments exhibited in class and in the website formed the students' portfolio for
evaluation. These materials showed that practically all of the students (89%) were able to:

• look for Web page services and evaluate them in order to select useful
academic information,
• look for academic material related to their area of interest,
• communicate and share interests with other researchers or students through
e-mail and participation in discussion lists, and
• edit HTML and publish their reviews and other academic findings on an
individual site using basic graphic resources practiced during the course and
organize them in one collective site, according to certain concepts they read
and discussed in class.

Priority was given to content organization and quality, instead of graphic embellishment. The
collective site was remarkably superior to the individual pages both in quantity and quality
(URL: http://members.xoom.com/tgcarmona/default.html).

In terms of specific skills for using Internet software, tools, and services, they were able to get an
Internet account, install communication and Internet software, communicate more frequently
through e-mail, navigate the WWW, use search engines, download and upload material through
an ftp client, and edit HTML. Of course, there were different levels of performance and
achievements. Although most had already used e-mail and the WWW before the course, some
took much more time to achieve results than others and still had to follow personal written notes
and instructions.
Performance expectations

Judging by the commitment to course requirements, the instructor expected a higher level of
accomplishment. However, the students emphasized that their ability to gather relevant academic
material through Internet information and communication services increased (in some cases,
from zero). According to discussions, assignments, and reports collected during sessions, all the
students, except one, had insufficient previous experience with Internet clients or other software,
and six were slow English readers. All students needed more time to complete activities, some as
much as 15 hours. One group misjudged the time they had available for the courses, and in some
cases, had as few as six hours a week to work. The most experienced student said he needed
seven hours a week to complete his work. Even those who had more experience with word
processors or graphic editors demonstrated difficulty in transferring these abilities to other tools
as Internet clients. For example, the student who was a proficient English reader and had very
good computer skills with text and graphic editors said that six to eight hours a week were
necessary in order to achieve her goals. Although the students also complained about the slow
Internet access they had in the laboratory sessions, having some classes in the Internet laboratory
proved to be essential for those who had poorer computer skills.

The quality and quantity of discussions of course content were negatively affected by the effort
spent resolving technical problems. Some students reported that they wanted to explore and
discuss subjects of interest in greater depth and quantity. Technical difficulties forced the
instructor to be less demanding concerning the focus of the course. Changes were made to
accommodate the majority of student needs. However, the students became aware of and put into
practice important concepts of academic Internet use, such as criteria for establishing quality of
information, usability, and peculiarities of the media in communication and education. At the end
of the course, they reported that their technical level as well their comprehension of Internet
technology use improved (some had serious prejudices about using the Internet in academic
work). They also reported that the benefits they got from the Internet for their research and
educational processes increased substantially during the course and that they were satisfied with
their progress. Some reported that the Internet helped them to find theoretical and
methodological trends in their research areas.

These better results in the second course can be ascribed to modifications in course methodology
motivated by the necessity of managing student lack of time, access difficulties on campus, and
maintenance of student interest in Internet academic usage.

Refrence:

http://www.isoc.org/inet99/proceedings/2f/2f_4.htm