See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/242278708

HOW TO EVALUATE AN EUROPEAN PROJECT?

Article

CITATIONS READS

0 193

1 author:

Eila Lindfors
University of Turku
12 PUBLICATIONS 10 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Safe School - Turvallinen koulu View project

From Gender-segregated Craft Subject to Multi-material Craft subject View project

All content following this page was uploaded by Eila Lindfors on 14 June 2015.

The user has requested enhancement of the downloaded file.

 FISTE - A Future Way For In-Service Teacher Training Across Europe

ICT IN EDUCATION: REFLECTIONS AND PERSPECTIVES – Bucharest, June 14-16, 2007

HOW TO EVALUATE AN EUROPEAN PROJECT?

E. LINDFORS*,1
1
Department of Teacher Education in Hämeenlinna, Faculty of Education, University of
Tampere, P.O. Box 513, 13111 Hämeenlinna, Finland

Abstract:
The aim of this paper is to present the evaluation of the FISTE-project (Future in-service
teacher training across Europe) in the frame of Comenius 2.1 program funded by European
Commission as a sample case. The partners of the project are universities from Finland,
Romania, Latvia, Iceland and an in-service teacher training center from Spain. The FISTE-
project started in 2004 and aimed at offering European teachers skills and knowledge on
how to use ICT in teaching in a pedagogical and innovative way. The project tried from the
very beginning to find innovative and efficient ideas for in-service teacher training by
organizing national and European courses for promoting the pedagogical use of ICT.
The topics of evaluation concerned the objectives of the project, the management of the
project and the impact of the project. The evaluation is done in the project on three levels:
institutional, national and European. This article presents the evaluation of FISTE-project
as a sample.
The experiences gained in FISTE show the huge and detailed work concerning the
evaluation. FISTE-project highlights a meaning of a concept of evaluation, use of the
evaluation data in the project and longitudinal work like valorization and co-operation after
the financing period of the project. This will guarantee the necessary possible experience
and expertise for future projects in EU.

Keywords: evaluation; project work

1. General remarks on project evaluation

In education as well as in other sectors of EU co-operation and shared work between the
member states will increase. There will be even more European people in the future. New
applicant countries will join and old member states will continue and increase the co-work
with each other. All this means new projects with new partners in the future. In that work
the evaluation of the shared projects will have an important role. There are lessons to be
learnt and new innovational ideas to be applied.
The evaluation of the project is neither easy, nor a simple thing to do. It is a diverse
phenomena consisting of different stages, levels, sectors and time lines during the project
lifetime. The project is understood here as a group of people from different institutions
joined in collaborative and joint work in order to reach objectives officially defined and set

*
Corresponding author: e-mail: eila.lindfors@uta.fi

22
 FISTE - A Future Way For In-Service Teacher Training Across Europe

ICT IN EDUCATION: REFLECTIONS AND PERSPECTIVES – Bucharest, June 14-16, 2007

together. The evaluation made in the frame of FISTE –project: A Future In-Service teacher
Training across Europe, Comenius 2.1, was understood on the basis of a definition made by
Stufflebeam & Webster already in 1980. They saw evaluation as “the systematic
determination of merit, worth, and significance of something or someone and define the
making of evaluation as an informed act of ascertaining or fixing the value or worth of a
given project or product. [1].
The definition gives important guide lines for applying evaluation in real situations
where there are many levels of work in order to reach the objectives set for the project.
Systematic means that there has to be a carefully made plan how to proceed from stage to
stage. Determination insists that there have to be defined criteria to be able to decide upon
the merit, worth and significance of the target of the particular evaluation. The meaning of
the evaluation differs from one situation to another. It can be just fixing the value or merit
of a thing or it can be a catalyst for a change in way of working or developing objectives.
On the European project level the evaluation varies and has many objectives and
targets: the objectives set for the project, the outputs produced in the project, the impact of
the work of a project as well as the way of doing evaluation itself within the project. A
Survival Kit for European Project Management [2] gives very strict and concrete advice for
evaluation. ‘In the context of European project work evaluation is a process which supports
the project and acts as a check on whether the targets have been met. It allows the results to
be improved based upon judgments made about the value and quality of the project and it
simplifies decision making and can assist with fundamental changes in the project, should
these be necessary.’ Above all evaluation is seen as a process that must not be left to the
final stages of the project.
An evaluation strategy agreed together within the partnership is a basis for evaluation
criteria and methods of doing it. Evaluation is a methodologically diverse area using both
qualitative and quantitative methods such as case studies, survey research, statistical
analysis, etc. Evaluation data can be qualitative and quantitative or mixed. The structure of
evaluation can be very complicated according to project objectives and targets, duration of
a project and partnership. At least the following issues should be considered when defining
and organizing the evaluation of a project:
1. To know where and on what to start
2. To follow the work done
3. To be able to learn and develop the national and European expertise
4. To be able to react and continue or alter the way and content of working
5. To be able to make final conclusions on the basis of ex post evaluations
6. To be able to present the work done for an external evaluator
7. To assure the funding resources the quality of the project
8. To be ready for future challenges and co-operation
In the following, the case of FISTE-project is presented as an example of the evaluation
fulfilled in one project. Dillon and Åhlberg have underlined in their recent journal article
2006 [3] that quality in on-line learning and in-service education is associated among other
things with work in real and authentic matters which are personally meaningful and
professionally relevant and connect theory and practice by constructing integrating theories

23
 FISTE - A Future Way For In-Service Teacher Training Across Europe

ICT IN EDUCATION: REFLECTIONS AND PERSPECTIVES – Bucharest, June 14-16, 2007

of participants own. This can be seen as criteria for evaluating the substance of the project.
The evaluation of management presents the efforts and weaknesses that the project work
has faced. The impact of the project can be analyzed on the basis of the two mentioned
categories by taking a selection of data into the interpretation as an internal evaluation.
External evaluation is an outside analysis of the project work and results and helps the
project partners to get a different view on the work done.

2. A Case: The evaluation in the FISTE-project

The FISTE-project started in 2004. The project was aimed at developing in-service
teacher education for today and especially for future on an European level. In essence, the
project was aimed at finding new ways of how to teach in-service teachers in in-service
teacher training and how the teachers themselves could learn and upgrade their knowledge
and teaching methods by using ICT. The project tried from the very beginning to find
innovative and efficient ideas by organizing national and European courses for promoting
the pedagogical use of ICT [4]. The partners of the project were universities from Finland,
Romania, Latvia, Iceland and an in-service teacher training center from Spain. This means
seven partners from five European countries, the old members of EU and the newcomers
worked together and gained experience from different levels and stages of the project [5].
Table 1 presents details how the partnership made the evaluation. There was organised
an evaluation group as a sub-group of the project. Basically the evaluation was the
responsibility of the group which designed, planned and fulfilled the evaluation. The group
made detailed plans how to evaluate and assess the whole project and decided what tools to
use. The project work was divided into monitoring and evaluation levels as the Creation
stage, the Testing stage and the Dissemination stage. This helped to plan the timing
followed the project work and in evaluation.
Table 1 shows the huge evaluation work made in the project. Because of the importance
of the evaluation, it was kept in mind from the beginning of the project and specific
mmonitoring and evaluation dimensions which were agreed in the application formula [6]
of the project were defined. These were:
1. The organization of the project in order to achieve the objectives on national and
European levels.
2. The starting level and development of in-service teachers during the project.
3. The experiences and expertise developing in all participating institutions.
4. The advantages and disadvantages of developing the face-web-method (blended
learning) from the future perspective.
The evaluation is presented here in Table 1. There are three different perspectives: I. the
evaluation of gaining the objectives, II; The evaluation of the project management and
project work in order to reach the set objectives and III; The evaluation of impacts of the
project as a whole.

24
 FISTE - A Future Way For In-Service Teacher Training Across Europe

ICT IN EDUCATION: REFLECTIONS AND PERSPECTIVES – Bucharest, June 14-16, 2007

Table 1. The evaluation made in the FISTE-project. Detailed table.

Times in the project
The topic of Level of
The target of evaluation The method 1st 2nd 3rd total
evaluation evaluation
year year year
on-line course, national On-line questionnaire - 1 1 2
on-line course, European On-line questionnaire - - 1 1
Qualitative and quantitative
Database 1 1 2
evaluation
I. Teaching materials Qualitative evaluation 1 1
The objectives of Assessment tools Qualitative evaluation 1 1 1 3
the project: Project’s Web-site Qualitative evaluation 1 1 2
Dissemination seminar Questionnaires: Fixed and open
1) 1 1
Institutional ended questions
Evaluation
dimensions / Research results and Classification of the publishing
National 1 1
agreed in the articles journal
project /
application European Additional materials Qualitative evaluation 1 1
Qualitative and quantitative
School network 1 1
evaluation
2)
Students’ final products
Outputs Qualitative expert evaluation 1 - 1
national
3) Dissemination Best practices guide Qualitative expert evaluation 1 1
Students’ final products
Qualitative expert evaluation - - 1 1
European
Implementation feedback Qualitative expert evaluation 1 1
Questionnaires: Fixed and open 2 3
Project meeting 2 7
ended questions
Institutional
II. Project work Fixed and open ended questions 1 1 1 3
The Project meeting
management of Qualitative summary 2 2 3 7
summary
the project
Project work summary Qualitative summary 1 1 1 3
European
Project management -
Qualitative summary - - 1 1
Final

III. Quantitative data and summaries from every year: the use of
Internal Logs and stats
The impact of technologies and participating
evaluation
the project
Evaluation dimensions Qualitative collections of specified data
External The final evaluation on the - - 1 1
Expert evaluation
evaluation basis of internal report

2.1 The objectives of the project

One of the tasks of the evaluation is to show and present how the project itself and the
partners themselves have managed in achieving the specific objectives agreed together in
the application of the project. In Fiste project these objectives were [6]:
1. To develop methods for integrating face to face and web-based learning (ODL) in meaningful
ways according to the everyday work of in-service teacher;
2. To apply the methods for teaching in various learning environments in the work of joined
partners;

25
 FISTE - A Future Way For In-Service Teacher Training Across Europe

ICT IN EDUCATION: REFLECTIONS AND PERSPECTIVES – Bucharest, June 14-16, 2007

3. To improve teacher education possibilities to use new types of technology for in-service
teacher education;
4. To improve in-service teachers’ use and understanding of ICT to support their own work in
meaningful ways;
5. To develop European cooperation and awareness;
6. To improve the research base of knowledge of how to integrate and best combine face to face
learning and web-based learning in European in-service teacher education;
7. To disseminate the results of the European in-service teacher education project (FISTE) on
local, national and European level.
Table 1 presents the variety of evaluation made in the project in order to be able to
make judgements. The level of evaluation is institutional, national and European. The
targets of evaluation are different outputs made and experiences gained in the project. The
data is qualitative and quantitative. The evaluation is made mainly during the second and
third year.

a. The management of the project

The evaluation of the management of the project started in the first meeting. Since that
this part of evaluation has been regularly made (table 1). The summaries were made to be
able to get the European view along with the institutional one. The project itself had the
responsibility to work in order to reach the set objectives. The management evaluation
made it possible to change some parts to be able to reach the objectives better.

b. The impact of the project

The impact of the project is based on all the evaluation which was made in the project.
It is like a collection of selected evaluation items as an internal part. The external part is
made by an outside evaluator who comes in FISTE-project from other than participating
countries. The external evaluation is a formative evaluation which helps the partners to
make the final evaluation report of the project. One of the main targets of the evaluation
report is to clarify to partners in the project as well as to EU as financing administrative the
work made and the results gained in the project.

3. Considerations on the evaluation of a project

One of basic issues to be discussed in the beginning of any project is the meaning of the
concept of evaluation. In order to get high level and meaningful evaluation results partners
should share common understanding of the content of the concept of evaluation. This is
absolutely the most important thing to be agreed in the situation where partners of the
project have different tradition and background in their professional area. The second issue
is the commitment to the evaluation process within the project lifetime. Without shared
understanding of evaluation as a concept and commitment to evaluation processes it is
impossible to get high level results in evaluation or in a project itself.

26
 FISTE - A Future Way For In-Service Teacher Training Across Europe

ICT IN EDUCATION: REFLECTIONS AND PERSPECTIVES – Bucharest, June 14-16, 2007

A good question is how much the evaluation results cause changes in the project
objectives or way of working. FISTE-project was approved in time when there were very
strict orders concerning the financing part of how to proceed. In the last steps of the project
it can be argued that there should be more flexibility to be able to alter or develop some
parts of the project without a need to get permission from the financing officers. The
evaluation shows that FISTE-partners have really been committed to the project work.
Table 1 shows the huge work made in the context of evaluation in the FISTE-project. A
lot of data has already been analyzed. At the same time there is material waiting to be
analyzed. This gives material for the further co-operation and valorization of the project.
This means that the evaluation made till the end of the project will develop further. On the
other hand, evaluation is so wide an area, especially scientifically, that the best results will
come after the project is already closed.

Acknowledgements The article is made in the frame of FISTE –project: A Future In-Service teacher
Training across Europe No. 118766-CP-1-2004-1-RO-COMENIUS-C21. The FISTE team in each
participating institution has given their input into the evaluation process within the project. Without
this work and The EU funding the evaluation of the project or this article about the project evaluation
would not have been possible to write.

References
[1] Stufflebeam, D. L., & Webster, W. J. (1980). An analysis of alternative approaches to evaluation.
Educational Evaluation and Policy Analysis. 2(3), 5-19.
[2] H. Bienzle. A Survival Kit for European Project Management. (2004) in
http://www.eacea.ec.europa.eu/.../docbots/Documents/TCP/Survival%20Kit/survival%20kit-EN-
updated%20version%202004.doc (Red on 23rd of September 2007).
[3] P. Dillon & M. Åhlberg. Integrativism as a theoretical and organisational framework for e-
learning and practitioner research. Technology, Pedagogy and Education, 15 (1), 7-30 (2006)
[4] T. Hämäläinen and E. Lindfors. ECSUT -course: innovative ideas, practical solutions. An article
in this volume (2007).
[5] E. Lindfors. Partner experiences in an European project. An article in this volume (2007).
[6] A Future In-Service teacher Training across Europe called FISTE. No. 118766-CP-1-2004-1-RO-
COMENIUS-C21. The application formula (2004)

27

View publication stats