Vous êtes sur la page 1sur 6

2016 20th International Conference on System Theory, Control and Computing (ICSTCC), October 13-15, Sinaia, Romania

A Playful Approach in Course Structuring for an


Effective Student Evaluation
Sabin SIMIONESCU, Marius MARIAN
Dept. of Computers and Information Technology
University of Craiova
Craiova, Romania
simionescu.sabin@ucv.ro, marius.marian@cs.ucv.ro

Abstract This paper presents our proposal for a change of


paradigm in what concerns student education and evaluation. It
is based on efficient use of technology to improve the practical
skills of students. The main difference between mainstream
higher education and this proposal consists in the combination of
how information is disseminated plus the way knowledge
absorption is validated by the educators.
Keywordscomputer education; playful learning; pedagogical
approaches; automatic evaluation; gamification; game with a
purpose.

I. INTRODUCTION
The economic partners of academia in Romania are
frequently complaining about the low-quality practical skills of
graduates when facing real-life problems. The apparatus for
solving engineering problems in industry is one of the most
troubling aspects of integrating young specialists in practice.
Without exercise the practical skills cannot be built in and
assimilated. Digital natives are somewhat reluctant in using the
traditional tools of the XXth century engineer.
II.

MOTIVATION

A. Inertia
First, we need to ascertain the fact that the most frequent
pedagogical approach is based on delivering a topic and
requiring students to memorize the materials. This approach is
effective whenever immediate access to materials is difficult.
While this is not an issue in subjects like mathematics or
classic literature, where new discoveries/works are easily
integrated into the curricula, in engineering the quantity of new
stuff to be integrated is simply too high when dealing with the
at least 5 or 6 years of pre-planning required by current laws.
B. Presentation form
The typical forms of presentations have evolved in the
educational system from blackboard and chalk to whiteboards
and slides. However, the information that flows from teachers
to students has mostly remained the same. Still, 3D simulations
are only now peeking into the classrooms, and interactive
options during lectures are still rare. Feedback from students to
teachers is another significant problem some teachers simply
put out the information, and students are expected to assimilate
it. Courses do not evolve dynamically based on what the
students have actually understood from the previous

978-1-5090-2720-0/16/$31.00 2016 IEEE

presentations, there is just a simple 10-14 week sequence of


talks that are considered mandatory for students.
C. Integration of theory and practice
Lack of investments in the education system and the poor
update rate in the salary policy (between public and private
sectors) has triggered a gap between what is taught as theory
and what is actually repeated in a practical experiment. The
rate at which practical knowledge evolves is significantly faster
than the rate at which new theory is developed. Thus, courses
have to make small adjustments in the theoretical background,
while making significant changes in the practical experience
gaining processes.
D. Summer practice
Private companies require productivity from their
employees. There is little room and understanding for a
graduate to adjust to the needs of the business. Nevertheless,
companies are reluctant in accepting large numbers of students
in summer camps and as interns during their studies. This
increases the pressure on universities to deliver already trained
students and graduates, capable of brining-in value to the
company from day 1.
E. Evaluation efficiency
One question received from the industry is what does a
grade represent? Besides being a threshold for a diploma. This
aspect of actual mapping grade into skills and competencies is
urgent to address.
III. STATE OF THE ART
Gamification has been used efficiently as a motivation
factor in education. Another aspect that increases learning is
gratification. Systems like stackoverflow.com and quora.com
are making plenty use of it.
Gratification on the technical questions-and-answers
(Q&A) website stackoverflow.com is mostly based on the ego
and willingness to share their expertise of developers that want
to be acknowledged among their peers. To do this, the site uses
a mix of methods to makes developers feel appreciated. These
include placing the chosen good answer at the top of the thread,
rewarding good answers with points, allowing comments to be
added to almost all (non-locked, very few) answers. All this is
done in order to increase transparency and overall, to offer the
best solution to the questioner, and to anyone else reading the

824

thread. Also, they mark questions as duplicates, to discourage


people to simply ask a question and then wait for a personal
reply, when they could have instead simply searched the
websites database.
The value of these community-powered knowledge sources
for developers is huge. The first author of this paper can
confirm this while being mostly a self-taught programmer, who
learned Visual Basic in secondary school, and then followed
with PHP, Adobe (at that time Macromedia) Flash, and Action
Script using sites like stackoverflow.com, codeproject.com and
planetsourcecode.com.
Quora.com, this time a Q&A generalist site, includes the
possibility to offer a bounty (accumulating points) for a good
answer to a question. This increases the chance for a better
answer being provided. Expert or experienced users will be
motivated to spend the extra time needed to write an even
better answer simply to get a larger amount of points. This
increases the overall value of the site, since having a larger
library of improved answers will automatically impact its
visibility on search engines, and thus bringing more traffic on.
In turn, this new traffic will potentially add new questions,
making the site even better.
Stackexchange.com is another example of a centralized
community of around 160 development and engineering sites,
such as stackoverflow.com and askubuntu.com to mention only
a few. Users that possess a minimum of points on any of the
sites in the community, whenever they register into another site
of the community, will start with a minimum reserve of 100
points on that new site. This is provided to enable good
contributors to have the same community rights as on the site
they began with. Also, there is a requirement for a minimum

limit of points for actions such as commenting on some


questions, to up-vote and down-vote a question, and many
other on-site activities.
IV. THE PLAYFUL APPROACH
We start with the nave assumption that things can only get
better and that students are learning best while playing. They
enjoy the ludic spirit, and games typically trigger both
competition and personal improvement. Continuous contact
with a technology-based educational platform that rewards
involvement and work ethics is appealing to the younger
generation. The generation of digital natives is accustomed to
use electronic resources, available now and at all times.
From a technological point of view, our plan is to
implement this approach into the Moodle Learning
Management System (LMS) as a plug-in or a set of plug-ins.
The LMS will act as a concentrator for the student community
and the academic staff.
The topics that will be addressed in a first step, concern
computer programming and web development. Content from
these two topics will be created in form of short text
explanations, videos presentations and tutorials, questions and
answers, quizzes, automated multiple choice questions, jigsaw
matching
tests,
short
snippets
of
code
to
discuss/interpret/compile, error identification in code (logical,
syntax, semantics, at compile- and/or run-time), automated
compiling of code uploads.
Student community will be trained to adopt a new learning
paradigm on several layers. First, the teaching materials will be
provided on a weekly basis for students to process individually.

Figure 1 Weekly timeline of the Lecture Q&A practical applications workflow

825

During the week #N, the students have to complete a sequence


of evaluations designed to assess the knowledge accumulated
from the teaching material provided in the same week. Students
are encouraged (via motivation and gratification means) to
extract from the grey or shadow areas of the textbook questions
and points of potential ambiguity, and channel these findings
towards the LMS and the experts (i.e. teachers, assistants and
mentors). This feed-back is useful to continuously improve the
content and add supplementary materials necessary to keep the
courses up-to-date with the latest advances.

the skill is kept recent in their mind. The system monitors


which skills have not been recently used, and will recommend
tasks that include both new skills scheduled for the current
week, plus the not-so-recently-used ones.

This weekly daisy-chain approach also ensures that


students have already acquired the necessary theoretical
knowledge to be able to solve the practical challenges proposed
during lab and seminary activities. At the end of the application
activities the students will be assessed again (automated/semiautomated).

The grading and evaluation system works in the following


manner. Each student once enrolled into a class will have a
grading system in which knowledge depreciation and regular
exercise constitute the main pillars. Knowledge depreciation is
a downgrading element for the student grade. We acknowledge
the fact that the course length is the students only chance to
accumulate a solid skill set for his future career. While students
are keen to procrastinate, we believe that any serious learning
environment should also strive to make the student aware and
responsible concerning her time resource. Students will
gradually and continuously gain experience points while they
are exercising concepts and knowledge specific to the topic of
the course. Every day matters and every effort is gratified. The
reverse is that the student will lose experience points if they do
not exercise or omit for some period of time to practice the
concepts. While at first this may seem a useless and punishing
approach, we strongly believe that this approach will get
students disciplined and responsible about their time resources
and career options.

Skill evaluation will be done continuously: for example,


iterative structures are generic computing concepts that are
used in specific ways in many programming languages.
Practicing them means using them. Whether students use these
in a web application, an Internet of Things gadget, or in
embedded programming while building a bloodstream device,

The presentation of this concept is obviously related to the


field of gamification. Students play, and by playing they learn
in a collaborative environment. Based on the individual
experience points and their knowledge gain, a multitude of
competition-like schemes can be devised and added to the
LMS. Also a few roles can be derived for the participants to

In week #N+1, the teacher will discuss and detail during the
usual class, the topics already provided via the LMS in the
week #N.
Additionally, the best questions asked by the students will
be gratified by providing answers in the class. These will keep
the students motivated to remain involved and attend lectures.

Figure 2 Minimal database structure example

826

this approach. There will be roles based on experience (novice,


intermediate, expert) and on administrative and learning
(mentoree, mentor, teacher, teaching assistant, content creator,
administrator). While role is a generic concept, behind it we
have levels of knowledge. The partition of a specific topic into
levels can use a linear cumulative approach while distributing
knowledge acquisition throughout time and ensuring constant
regular exercise.
To further stimulate competitiveness, we intend to instill
some form of elitism among students. For them, there will be
an additional procedure. Elite students are allowed to inspect
all the previous materials posted into the LMS (lectures,
briefings, results from previous years, etc.), without having to
wait for the weekly material. Many times, students in some
groups have stood out as having significantly above-average
knowledge on the taught subject. To better accommodate their
learning needs, they are allowed to access everything from the
first week. As they most likely will cover the entire course in
far less time than an entire semester, the teachers can then give
them dedicated tasks that will enhance their learning. For
example, in an HTML5 teaching class, a student that already
knows HTML5 from other sources (or self-taught) is allowed
to fast-forward through the course, and then she will be
provided with materials on CSS and Responsive Web Design.
Continuing the example, provided the students already know
these as well, they will be encouraged to pursue practical tasks,
like building a website using their current knowledge on web
design. In order to gain access to the elite group, a student
must pass all weekly evaluations with a desired rate (e.g. at
least 85%) and also pass a special test quiz with time
constraints.
The generic grading system is inspired from the free
language-learning system Duolingo1. Student accounts are
instantiated with zero number of experience points, zero
percentage of topic knowledge, and all topic sub-modules are
marked as not yet started. As student advances through the
curricula of the course and once she passes a sub-module, a
decrement counter will be started with a maximum value for
that sub-module. The purpose of this counter is to keep track
of knowledge depreciation, and to encourage students to take
measures to revisit regularly this topic. The indubitable power
of regular exercising is meant to yield knowledge imprinting in
students.
To the former point, in our community a benign role is that
of the mentors. These are usually senior students that act as
both experts and also guides for the novices. Integrating Q&A
between freshmen and senior students, and also establishing
mentoring relationships is beneficial for keeping both of them
focused on the curricula and their career building. For each
question raised by a student, the grading schema works in this
manner: first, indolence is penalized students that repeat a
former question available and already answered in the LMS
repository will lose experience points. The system has to
encourage them to look for answers by themselves before
asking out for help. Reversely, questions judged by an expert
(either a mentor or a teacher/teaching assistant) as pertinent and
new may be awarded some experience points. Both senior
1

students and freshmen are rewarded experience points if they


correctly answer a new question. The hierarchy of
experience points conjugated with the topic knowledge
percentage yield to higher roles for students (e.g. mentors,
experts). The LMS will let the teaching staff know when a
participant has accumulated enough experience points to be
granted access to a superior role. Teaching staff will usually act
as moderators/judges in the LMS (obviously some of them may
even try it by themselves as a personal experience lifelong
learning).
V. THE EFFICIENCY OF EVALUATION
Concepts like awareness, memory depreciation, practical
skill building, learning by doing, and continuous evaluation are
paramount.
Students need to be made aware of the fact that they will forget
if they will not exercise. Memory depreciation is a real enemy.
A gaming approach introduces experience points that will
make the student acknowledge her level at a certain moment in
time.
Continuous evaluation is meant to keep the student alive in
the topic. Lack of exercise and implicitly of evaluation will
lead to losing points. Keeping constant the level of training will
only get a student to a bounding upper limit. To overpass that,
students need to give more and be proactive by starting solving
real life complicated tasks.
Similar to Test Driven Design, this evaluation method can
be considered as Evaluation Driven Learning [3]. It is
common knowledge in the teaching community that when
students are given a set of subjects for the exam, they will
concentrate on learning (memorizing) exactly that given set,
effectively ignoring the rest of the curricula. The evaluation
method described in this paper builds on top of existing student
behavior, using the principle of the beaten path. This method
effectively forces every student to confirm every knowledge
piece included in the materials pack, at lesson level. By moving
the testing from competencies list, with random selection,
down to such a small granulation, the education provider can
have a significantly better overview of what each and every
student has assimilated from the knowledge presented in the
materials.
The metadata provided by using this evaluation method
on-screen can later be used to create statistics that allow
teachers to evaluate the materials themselves, and figure out
where to make improvements.
For example, when >50% of students in one group failed
one question, while the same student group passed most of the
other questions from the first attempt, that question is to be
singled out for restructuring. Then follows a thorough analysis
of the lecture materials covering that question + any
prerequisites considered necessary for a newbie to answer that
specific question. This introduces a feedback loop into courses
development: whenever a disproportionate failure of a student
group happens, the knowledge unit evaluated in that question is
to be put up for review before next student group reaches that
chapter/course.

Available at http://www.duolingo.com.

827

Based on results from these weekly evaluations, student


profiles can be built, allowing the e-learning system to provide
every student with a personalized learning plan. The courses
that include this feature will provide different materials to
students, based primarily on how the student learns better (i.e.
one student might learn C# by being shown a comparison to
C++, while another student learns better by converting pseudocode into actual C# code).
Important note: this evaluation method is designed to
ensure that students go through the course materials, not to
provide grades. There is a method of obtaining grades (or parts
of grades) from answers provided by students in these tests, but
it is indirect. Instead of measuring whether a student knows
something or not, an evaluator can look at how fast a student
can answer a question correctly. Example: Can emails can be
sent via port 52 on the default postfix configuration? A
student that knows how SMTP works, will know the default
ports are 25, 465 and 587, and answer no from memory. That
should take about 5 seconds more than reading the question. A
student that has not yet absorbed this knowledge item will have
to search for it, either in the course materials or online,
procedure which adds to the time required to answer the
question.
By linking correct answers (and ignoring the time factor) to
minimum passing grades, students are encouraged to prefer
answering correctly than just guessing an answer. Then, the
faster a student answers, the higher the grade. There will be
students that have previous experience with the subject taught,
and they will effectively set the threshold as to what retrieve
from memory. For the initial memory-vs.-searching threshold,
teaching assistants can be evaluated.
We consider the student grade to be the result of the
following factors: time spent on subject, IQ and previous
experience on subject and related fields. An average student
can absorb the same amount of knowledge as a smart student, it
Pass

Q.1.
1
Q.1.
2
Q.1.
3

50%

Pass

Q.3

This grading method is not designed to be used alone. A


good practice would be to allocate 25% of the grade to this
theoretical evaluation, and 75% to practical implementation of
what the course teaches. In a situation where the course subject
allows it, automatic evaluation of practical implementations
(i.e. file uploads + server compile + server code running @ 10
tests per file with pre-calculated input-output pairs) can provide
automatic grading with unlimited scalability. The practical
implementation evaluation can follow a similar workflow,
allowing the student to figure out what he/she does not yet
master, and where to allocate more time to improve
proficiency.
VI. FUTURE WORK
Over time, the Q&A section will accumulate a significant
amount of knowledge, properly structured on the subjects of
the courses. This knowledge base can be used in many ways,
one of the most interesting being a training data set for an
Artificial Intelligence program [2]. This can further be
developed into a solution where students ask questions, the AI
looks through the knowledge base and provides the answer in
natural language. With multilingual processing, the answer can
be found in a document in one language, and rendered to the
student in another language.
Another feature planned for implementation is student
grade distribution analysis. We have chosen the model of the
Gauss Standard Deviation as the target. The courses where
evaluation driven learning is used is modeling the curricula, the
grades are expected to start from 5 up to 10 (because of the
recursive materials presentation system, which shows the
student the information as many times as it takes, until it
Pass

Q.n

Q.1
redo

Fail

Pass

Q.2

Fail

Q.1

only takes more time. The goal of the LMS is to ensure all
students have achieved a certain minimum, and for those that
show potential on a certain area, provide extra materials and
challenges to motivate them to investigate and learn further.

Test
complete

Q.2.1

Fail
Show iframe with
materials needed to
gain knowledge item
in Q.1. Enforce
reading/watching.

Queue Q1
after Q.n

* You should
consider
[re]reading
the
following
chapters:

Q.1.
n

Figure 3 Evaluation Driven Learning

828

becomes knowledge). In standard courses, grades follow the


distribution from 1 to 10. We will also make a comparison
between the grades from previous years and the grades
obtained by students using the LMS.

partnerships and an overall better educational experience for all


parties involved.
By switching from potentially-monologue lectures to
materials distribution in week N followed by lecture + Q&A in
week N+1, students are provided with a system where they can
ask as many questions are they feel the need for, without the
peer pressure of we are waiting for you so we can leave the
classroom. This is expected to uplift the skills and grades of
introvert students, without affecting the grades of extrovert
students.
By offering a clear mapping between course materials and
skills, the students will be able to know what their strengths
and weaknesses are, and map out an academic path, followed
by a career trajectory.

Figure 4 Gauss curve shaped grades distribution model [4]

VII. CONCLUSIONS
The paper presents a proposed e-learning system highly
inspired from the free language-learning system Duolingo of
which the authors are fans and users. [1]
We believe that the collaborative approach towards
learning (involving mentors, mentorees, and teaching staff)
described in the paper will help improve the students focus on
skill building and professional improvement. By reducing the
age gap between mentor and mentoree (vs. teacher student)
we believe that strong bonds can be constructed between
people with similar interests. This can in time lead to long term

The paper presents a new paradigm in both knowledge


distribution and evaluation, taking advantage of technological
advances in recent years. The LMS implementation presented
is designed for digital natives, which are today reaching the age
to apply for university studies.
REFERENCES
[1]
[2]
[3]
[4]

L. von Ahn, "Games with a purpose," in Computer, vol. 39, no. 6, pp.
92-94, June 2006. doi: 10.1109/MC.2006.196
J. Hendler et al., "AI's 10 to Watch," in IEEE Intelligent Systems, vol.
23, no. 3, pp. 9-19, May-June 2008. doi: 10.1109/MIS.2008.40
Ram, Ashwin, and David B. Leake. Goal-driven Learning. Cambridge,
MA: MIT, 1995. Print ISBN 9780262181655
https://upload.wikimedia.org/wikipedia/commons/8/8c/Standard_deviati
on_diagram.svg

829