Evaluation Framework

A project partially funded in the framework of

the SOCRATES Programme – MINERVA Action of the
Directorate General for Education and Culture of the European Commission
 Evaluation Framework

TA B L E O F C O N T E N T S

INTRODUCTION.......................................................................................... 1

1. THEORETICAL EVALUATION FRAMEWORK .............................................. 2

1.1 INTRODUCTION .....................................................................................2
1.2 MAIN EVALUATION RISKS IN PROJECT IMPLEMENTATION........................3
1.3 COUNTERMEASURES...............................................................................3

2. EVALUATION PLAN ................................................................................ 5

2.1 THE PURPOSES OF EVALUATION .............................................................6
2.2 THE SCOPE OF EVALUATION ..................................................................7
2.3 EVALUATION CRITERIA ............................................................................7
2.4 METHODS AND TECHNIQUES.................................................................8
2.5 EVALUATION AND PROJECT MANAGEMENT ...........................................8
2.6 UTILISATION ...........................................................................................9

3. OPERATIONAL PLAN ............................................................................ 10

3.1 FORMATIVE EVALUATION ACTIVITIES.....................................................10
3.2 SUMMATIVE EVALUATION ACTIVITIES ....................................................10

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M II

 Evaluation Framework

INTRODUCTION

The SPOT+ project is motivated by the fact that, despite the large research production in
the field of the use of ICT in higher education institutions, very few studies focus on the
students’ point of view and experiences.
SPOT + thus aims at exploring, conceptualising and developing university students'
perspectives with regard to a targeted use of ICT for educational purposes.

In particular, four issues are at the centre of our attention, representing potential
motivations on the students' side and influencing the value of higher education to their
eyes:
ICT as a means to develop a number of “transversal skills”, such as social,
communication and organisational skills, which are fundamental to live and work
in the Information Society. The critical use of ICT integrated in traditional
classroom-based lectures would allow students to mature their key competencies
and transversal skills, increasing their employability and savoir-être-related skills.
ICT as a tool to develop a collaborative approach to learning and to stimulate a
more autonomous, learner-centred, democratic way of learning. By increasing the
degree of familiarity with technological devices allowing remote communication
and distance learning, students are introduced to a complex scenario, in which
they are part of a learning community and experience self-managed learning
processes.
ICT as a tool to lower barriers of access to higher education for those segments of
student population that are less favoured in social, economic, organisational,
physical or geographical terms.
ICT as a tool to add a global component to the study experience of most
European students, which prepares, follows and complements the physical mobility
of European students across national borders.

Considering the main aims of the project, the continuous monitoring and improvement
of methodologies and procedures in terms of analysis and communication among
partners will play a strategic role in the development of activities and will contribute to a
constant increasing of project performance in terms of quality, which will benefit not only
partners but also project’s beneficiaries.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 1

 Evaluation Framework

1. T H E O R E T I C A L E V A L U AT I O N
FRAMEWORK

1.1 INTRODUCTION

In this section both the general characteristics of the evaluation approach proposed and
its adaptation to the SPOT+ structure are described, articulating a set of evaluation
activities at project level.

The SPOT+ evaluation approach stems from a consolidated experience developed

through the management and evaluation of European projects. Specifically, relevant
references have been taken from three main documents, namely the “Evaluation
Guidelines - Handbook for Learning Technology Innovation”, produced by the Tavistock
Institute, London, within the ARTICULATE Evaluation Project in 1995, the “Evaluation and
Monitoring System”, produced by the Tavistock Institute of Human Relations – EDRU,
London, and SCIENTER, Bologna, within the TRENDS project in 1997, and the
“Evaluation Framework”, produced by SCIENTER within the MODEL Project in 2000.

Accordingly, the evaluation approach proposed focuses on:

Existing evaluation theory and practice
Project design and development process
Context of project re-implementation.

Concerning the first point, a constructivist evaluation approach is adopted, focussed on

an active involvement of project actors/stakeholders within the development of the
project. This implies the capability of the project to:
a) address different actors having different needs and interests with ad hoc tools;
b) adapt to actors’/stakeholders’ changing needs and requirements throughout project
duration.

Concerning the project design and development process, particular attention should be
devoted to the communication system, allowing interaction among partners and
stakeholders but bearing also serious risks of hindrance to the healthy development of
the project (if communication systems and rules are not defined, commonly agreed and
constantly monitored).

The context, that is the technological, social, organisational, economic and pedagogical
setting deeply influences the project development. In relation to SPOT+, this is true for
the development of the planned outcomes. Since in SPOT+ field research and
implementation phases directly involve the addressed target groups (students and
university staff), it appears extremely relevant to take into account “context-related”
variables affecting concepts like “innovation in teaching/learning”, and “perception of
ICT potential for university studies”.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 2

 Evaluation Framework

1.2 M A I N E V A L U AT I O N RISKS IN PROJECT

I M P L E M E N TAT I O N

The following main evaluation problems might arise:

Bad scheduling
Mismatching of results vs. set objectives
Methodological problems, also related to data collection (elaboration of the
questionnaire for baseline survey) and processing
Information management problems related to a misuse of the project
communication services and/or to the bad functioning of the learning
management system used for the delivery of training modules.

1.3 COUNTERMEASURES

Bad scheduling.
Constant monitoring of project activities will be carried out. Although in the
SOCRATES Programme it is not required such a detailed definition of the schedule
of deliverables as for example in the IST programme, partners have agreed to
implement a planning of the scheduled activities and to respect the intended
deadlines. In case of timing underestimation, “recovery measures” will be put in
place, such as: creation of dedicated internal task-forces, re-prioritasation of due
activities, allocation of additional resources (within the limits of the project budget),
so to respect the deadlines.
Mismatching of results vs. set objectives.
Project monitoring will allow SPOT+ partners to realise in due time possible
mismatched between the obtained results and the intended objectives. The
formative approach adopted for evaluation activities allows for an ongoing
reflection on the outcomes delivered, as well as for a continuous learning process
within the partnership. Thus, continuous verification of the work being developed is
meant to assure coherence among the different phases of SPOT+, and
consistency between these and the planned outcomes.
Methodological problems, also related to data collection and processing.
Research phases (desk and field) represent the backbone of SPOT+ activities. The
definition of ad hoc methodological guidelines is essential in order to coherently
carry out research. Thus, it is going to be of extreme importance that partners
actively and collaboratively contribute to the elaboration of a shared research
approach, negotiating among the different academic traditions and approaches.
In these regards, co-ordination meetings represent a valid occasion to propose
and refine methodological lines.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 3

 Evaluation Framework

Following the phase of methodology definition, the field research phase is going
to pose challenging questions in terms of reaching the addressed target groups (as
specified in the project proposal, approximately 1.500 European students) and
getting from them the needed information and involvement. The active role of the
partner networks (namely the Coimbra Group and ESIB) is going to be
fundamental. They will particularly care for the proper distribution of survey tools
and for their collection, as well as for the further involvement of students in the
planned discussion fora and training activities.
As regards data processing and interpretation, the four partner universities will
assure a scientifically correct treatment of data.
Information management problems related to a misuse of the project
communication services and/or to the bad functioning of the learning
management system used for the delivery of training modules.
The communication services – both internal to the partnership, and external
involving students and university staff – will be organised and monitored by the
responsible partner (FIM). All the SPOT+ partners will contribute with discussion
animation and support. The sharing of tasks and responsibilities is meant to
assure continuous monitoring as well as support – whenever needed.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 4

 Evaluation Framework

2. E V A L U AT I O N P L A N

The Evaluation Plan is designed to provide tools to assess the development of the
project.

The development of an evaluation system that can accommodate the scope and
complexity of the SPOT+ overall activities is crucial for the success of the project. Within
this context, the evaluation system will focus on the following objectives:
to contribute to the continuous and regular monitoring of the project progress,
with linkages and interactions with the project and quality management
procedures;
to promote reflexive learning in the project in order to provide an input towards
the sustainability of the results of SPOT+.

The constructivist approach adopted implies, in addition to the already mentioned

involvement of stakeholders, the contextualisation of evaluation procedures into the core
of project activities and related domains. In fact, evaluation design and implementation
does not take place in a vacuum: effective assessment can only occur if there is a clear
understanding of the domain in which the project and its related systems operate. This
means for the project SPOT+: activities of data gathering and analysis are meant to be
continuously verified in a way which leads to a gradual improvement of the research
process.

The project aspects considered for evaluation will focus on:

PM performance: being evaluation responsive to the lifecycle of the project
development process, the evaluation approach proposed places great emphasis
on linkages between evaluation itself and activities that are traditionally associated
with Project Management. In addition, synergies will be developed between the
Evaluation and the Quality Plans and related activities, in order to ensure
continuous monitoring of processes and outputs.
Communication patterns: the SPOT + project implies the need for an efficient
communication system among partners. A mailing list has already been set up,
and constitutes the main communication mean for the first months of the project
duration. Later on, after the establishment of the SPOT + website, electronic for a
will give the opportunity of a continuous exchange of opinions and thoughts about
the defined relevant themes among the project partners.
Quality of the outcomes obtained: the quality of outcomes, both in terms of reports
and services (on-line support tool, thematic electronic fora) will undergo a process
of internal and external evaluation. The actors involved in the monitoring of quality
will be project partners as well as the Project Steering Committee. A crucial role
will be played by the users of the products and services developed by the project
(EU Commission, LEONARDO project promoters, other interested parties).
Dissemination activities: dissemination plays a key role within the development of
this project. The evaluation of dissemination will mainly concern the assessment of

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 5

 Evaluation Framework

the searching/browsing facility and of the online support tool service. The quality
of the Thematic reports will also be evaluated, mainly with the support of the
Project Steering Committee, and of all the partners. An electronic forum on project
monitoring will be created, so to get a feedback on quality and users’ satisfaction.
Being the survey activities conducted twice during the project, the evaluation and
assessment of the activities and results of the first support will support partners in
the improvement/refinement of strategies.
Sustainability: the sustainability of the project in terms of visibility and of
satisfaction of users’ needs plays a key role for the success of the project. The
partnership relies on competent staff carrying out the work (all partners have a
long and relevant experience both in the filed of lifelong learning and in ODL and
ICT use in learning systems)

The Evaluation Plan is composed of six “building blocks”, explained in the following
Sections.

2.1 T H E P U R P O S E S O F E V A L U AT I O N

The methodologies, solutions and tools of the evaluation plan globally aim at defining a
threefold set of purposes: operational, summative, and learning purposes.

Operational purposes refer to how the project is being developed, implying a clear
reference to the project management style, to the quality of partners’ participation, to the
quality and efficiency of the communication system and of the information management
system, to the respect of deadlines, etc. The evaluation activities assure that the project
management and the other partners continuously monitor the quality of the complex
process being enacted.

Summative purposes refer to the traditional approach to evaluation, i.e. to judging and
assessing the match between the expected results, the invested resources, and the goals
achieved. This dimension of evaluation interests both internal actors (i.e. the project
partners) and external stakeholders, whose attention mainly focuses on the quality and
usability of the SPOT+ outcomes.

Learning purposes refer to the overall assessment of the “lesson” that can be drawn
from the project in terms of information and communication management and of setting
up students-centred services at European level.
This dimension of evaluation is relevant to a variety of actors:
project partners, for whom it is a conclusive step;
stakeholders (European Commission, SOCRATES Programme project promoters,
working in the field of ICT used for learning purposes, other interested parties), for
whom it represents an overview of the sustainability and potential of dissemination
of the project’s contents;
users of the different services (students, university staff).

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 6

 Evaluation Framework

2.2 T H E S C O P E O F E V A L U AT I O N

The clear identification of the scope or unit of analysis, of the evaluation is of extreme
relevance in order to compose a coherent evaluation plan.

For the purposes of the SPOT+ evaluation plan, evaluation activities will deal with two
different but integrated evaluation objects:
the “summative” evaluation context, concerning the evaluation activities and tools
aiming, for instance, at assessing the added value of SPOT+ outcomes vis-à-vis
the expectations and needs of the addressed target groups (students and university
staff);
the “formative” evaluation context, concerning the “informal” elements which
characterise the project process and lifecycle (e.g. the identification of
developmental opportunities, the description of the relationships among the
stakeholders, etc.).

2.3 E V A L U AT I O N C R I T E R I A

The process of selecting appropriate evaluation domains is crucial. The SPOT+

evaluation will refer to the following criteria:
project management: control on the management efficiency and verification of the
intermediate and final results against the initial objectives and timing of the
project;
organisational effectiveness of the project partnership;
effectiveness of the information exchange
content quality of the services provided by the project;
economic efficiency of the activities carried out, in terms of human and financial
resources;
potential for dissemination and sustainability of project activities and results.

The criteria continuously monitored will concern: project management, organisational

effectiveness and information exchange effectiveness.

The assessment of the first survey and the feedback given by stakeholders and users will
constitute the basis for corrective/refining measures.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 7

 Evaluation Framework

Finally, an assessment of the potential for dissemination and sustainability of project

activities and results will be provided by month 24. The reactions of users and
stakeholders will help to determine the extent to which the project can be seen as
successful and in which parts it can be improved.

2.4 METHODS AND TECHNIQUES

Project partners will provide their feedback on the nature of the project work carried out;
in these regards, evaluation attention will focus on the internal functioning of
collaborative work, communication, effectiveness of the processes taking place during
the project lifecycle. Focus groups and face-to-face exchange of comments and
information will periodically take place during project co-ordination meetings, during
which critical processes and aspects will be highlighted, discussed and solved jointly.

As regards field research and the implementation of project outcomes, involved target
groups will be asked to provide their feedback, structured according to assessment tools
which will be prepared as the planned services will be activated.

Evaluation actions will cover all the activities related to the dissemination of project
results, with particular reference to the web site and to the online services. Evaluation will
pertain mainly to the functioning of the web site. With reference to the online support
tool service, evaluation will focus on the quality of the information provided, as well as to
the capability of quick and punctual feedback to the requests made to the service. Users
of the service will play a key role in assessing the service quality.

2.5 E V A L U AT I O N A N D P R O J E C T M A N A G E M E N T

Integration between monitoring/evaluation activities, quality principles and project

management at the project level will help facilitate the management of contingencies
that may occur as the project develops. Monitoring and evaluation will feed into Project
Management providing it with instruments for project self-review (periodic self-
assessment by individual partners). Moreover, the foreseen meetings of the Steering
Committee (occurring during project co-ordination meetings) will include specific
sessions aimed at reviewing project activities and products and at highlighting strengths
and weaknesses of the Consortium work.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 8

 Evaluation Framework

2.6 U T I L I S AT I O N

Evaluation activities also focus on how users and stakeholders participate in the process
of systems design, implementation and utilisation. It is crucial that users and
stakeholders get involved and feel engaged in the various phases. Lack of participation
should be pointed out immediately, so as to enact strategies to engage uninterested
users. For instance, focus groups (both formal and on-line) can be organised to foster
participation.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 9

 Evaluation Framework

3. O P E R AT I O N A L P L A N

3.1 F O R M AT I V E E V A L U AT I O N A C T I V I T I E S

Mid-term review of progress and processes

Activation of electronic forum on project monitoring
Implication of user groups in evaluation of project processes and outcomes

3.2 S U M M AT I V E E V A L U AT I O N A C T I V I T I E S

Evaluation of project results vs. objectives

Evaluation of the surveys methodology
Evaluation of the on-line support service performance
Evaluation of the impact on target users (at Project and at Programme level)
Evaluation of dissemination impact.

Financial Agreement n. 90310-CP-1-2001-1-MINERVA-M 10