Quality in Higher Education

ISSN: 1353-8322 (Print) 1470-1081 (Online) Journal homepage: http://www.tandfonline.com/loi/cqhe20

Among Demons and Angels: Attitudes towards

system evaluation for quality improvement in
advanced vocational education
Mats Lindell & Magnus Svensson
To cite this article: Mats Lindell & Magnus Svensson (2002) Among Demons and Angels:
Attitudes towards system evaluation for quality improvement in advanced vocational
education, Quality in Higher Education, 8:2, 127-136, DOI: 10.1080/1353832022000004322
To link to this article: http://dx.doi.org/10.1080/1353832022000004322

Published online: 18 Aug 2010.

Submit your article to this journal

Article views: 30

View related articles

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=cqhe20
Download by: [University of New Mexico]

Date: 16 December 2016, At: 08:39

Quality in Higher Education, Vol. 8, No. 2, 2002

Among Demons and Angels:

attitudes towards system evaluation
for quality improvement in
advanced vocational education
MATS LINDELL & MAGNUS SVENSSON
Lulea Technological University, SE-971 87 Lulea, Sweden

This paper investigates attitudes among educational organisations towards the use of
system evaluation for quality improvement in Swedish higher education. The subject of this
investigation is the evaluation of the pilot project on advanced vocational education (AVE), which
was conducted between 1997 and 1999 by a research team at the Lulea Technological University.
The investigation was carried out in the form of a telephone inquiry with a strategic selection of the
education organisers. The investigation shows that 65% of the respondents regard the work of the
research team, built on the system-evaluation model, as an instrument to inspire con dence and
contribute to renewal.
ABSTRACT

Introduction
Evaluations have become an increasingly important instrument in carrying out reforms of
higher education in Sweden. There are several reasons for this development. One is a
desire to acquire knowledge of the objectives set up for certain educational activities and
to check whether these have been attained (Franke-Wikberg & Lundgren, 1979). A second
motive is the desire to know what primary and secondary effects have resulted from the
educational activities in question. A third motive is that representatives of the political
establishment want more information on the economic resources used to implement the
activities (Stevrin, 1991; Franke-Wikberg, 1992).
In recent years, a parallel development has become discernible in view of educational
organisations increasing interest in the evaluation models developed in quality management. There are two main motives behind this development. First, evaluation for improving quality is perceived by the educational organisers to be particularly important in
maintaining a competitive edge over the growing market of private education and training
companies in Sweden (Lindell, 2001). Second, the models illustrate the importance of
taking advantage of various stakeholders needs and expectations in the best way possible.
The needs to be considered are often complex and varied since they are derived from
stakeholders with different motives, such as, educational and policy planners at the state
level, employer and employee bodies, local companies and students. Evaluation and
quality assurance, in the sense that the expectations are met, are very important for
providing education that is sought after and ts the demands of working life. This interest
can be exempli ed by the establishing of quality awards such as the European Quality
ISSN 1353 8322 print; 1470 1081 online/02/020127 10
DOI: 10.1080/135383202200000432 2

2002 Taylor & Francis Ltd

128

M. Lindell & M. Svensson

Award and Malcolm Baldrige National Quality Award. In both Sweden and abroad
(Muller & Funnell, 1992), this interest has also been carried over to organisations in the
educational eld. In Sweden, for instance, a special award for schools, the Swedish Schools
Award, was established in the mid 1990s.
With the increased importance of evaluation in higher education, research and debate
has focused on the choice and design of models of evaluation. The most widely used
models in Sweden for evaluating educational reforms are often referred to as `engineersmodels. In these, evaluators attempt to be independent and impartial, avoiding, as far as
possible, contacts and communication with respondents. Nor is the evaluator allowed to
make suggestions for improvements or support during the process of the evaluation:
comments are communicated separately (Vedung, 1995).
However, this attitude is changing in Sweden with the development of various alternatives to the `engineers-models. According to several studies, this has taken place because
of increased political awareness of the demands of pressure groups during the evaluation
(Karlsson, 1995). System evaluation is one of a number of models that build on this new
evaluation concept.
One major problem is the lack of research into how interactive system-evaluation models
are perceived by those evaluated. Only a few of the national evaluation projects for higher
education reforms conducted in Sweden during the 1990s have used systems evaluation
(Kim, 1997). This investigation starts from a national evaluation project on Advanced
Vocational Education (AVE) that uses the concept of systems evaluation. It studies how the
evaluation process was perceived by those evaluated. The evaluation was performed
between 1997 and 1999 by a research team from Lulea Technological University (SOU,
1999).
This investigation aims at answering two questions. The rst concerns the extent to
which the work of the research team on the AVE-project is perceived as a support for
development of the educational organisations. The second aims at answering whether the
work of the research team on the AVE evaluation is viewed with con dence or mistrust
among the educational organisations.

Principles of the System Evaluation Model

The systems-oriented perspective implies that an attempt is made to describe and characterise the totality constituted by the subject of the evaluation (Stevrin, 1991). Instead of
dividing the subject of this analysis into its constituent parts, an attempt is made to
understand the activities and its various parts starting from the totality and its properties.
The systems analysis is focused on illuminating the relations and interaction among the
various parts of the whole. In systems evaluation, the main interest is focused on the
structure present in the object being evaluated and the intrinsic processes. Thus, the
systems-oriented approach consists of two partial analyses: structural analysis and process
analysis.
In the case of structural analysis, the evaluator starts from the degree of agreement
between and within the various system levels. For the activity, as a whole, to work
satisfactorily, there should be a suitable degree of agreement among the various parts of
the activity and between the activity and society in general.
In its turn the process analysis aims at providing a deeper understanding of how the
system works and how it has developed over time. In this connection, the evaluators also
try to form an opinion of such things as the patterns and mechanisms that characterise the

Among Demons and Angels

129

FIG. 1. The various channels for collecting data of AVE activities .

system. Other criteria are adaptability, the ability to identify changes in requirements, and
mechanisms for feedback of information (Stevrin, 1991).

The Work by the Research Team on the AVE-Project

The Swedish government launched AVE in 1996, and it was intended as a new form of
post-secondary education in which one-third of the course time is devoted to advanced
application of theoretical knowledge at a workplace. In January 2002, AVE became a
permanent part of continuing vocational training (Proposition 2000/01). The primary focus
of training is to provide basic vocational education for the unemployed. It was intended
that courses should be based on active workplace-centred learning and problem-solving in
an overall educational context. The length of courses ranges from 40 to 120 points, where
each point is equivalent to 1 week of full-time study. Courses are structured around close
co-operation between enterprises and various course providers (upper-secondary school,
municipal adult education, higher education, and commercial training companies). There
are currently no restrictions on the sectors in which AVE is provided: it is open to people
coming directly from upper-secondary school and to those who are already in employment
and who wish to develop their skills within a de ned area.
The task of evaluating the pilot project with AVE during the period 1997 1999 has been
illuminated with trial activities from a total of 10 different perspectives, and has encompassed some 12,000 students, distributed over about 200 different course programmes.
These programmes were in turn divided into 12 different elds, each representing a sector
of the Swedish labour market, such as health care, tourism, manufacturing and nance.
One part of the evaluation was mainly intended to support the activities of those being
evaluated. This part was based on the use of self-assessment based on the theories of total
quality management (TQM).

Generic Picture of the Evaluation Process

The various channels for collecting data for the research team can be summarised in a
model (Figure 1). Six channels for data collection were used in the evaluation.

The Evaluation Scheme

The basic work of designing the evaluation model and the speci c technical problems
associated with the evaluation of AVE can be viewed both at a national and at a local level.
The two levels are combined with the six channels for data collection. At national level,
self-assessment, survey studies of the students and studies of national statistics are

130

M. Lindell & M. Svensson

performed. At local level, the research team performs visitations and face-to-face interviews and telephone-interviews.
National Level: the overall design and objectives of the trial activities
On the national level, the research team based their work on reports and surveys of the
Swedish labour market with a view to catering for future requirements. These included the
need for a higher vocational educational level and a more versatile workforce. One
problem in this form of post-secondary education is the uncertainty about how and what
students should learn in a rapidly changing labour market, which is characterised by
international competition.
In view of this uncertainty, the research team found it important not to use an objective
model but instead use a model that would support the improvement work during the trial
period rather than checking the extent to which the goals have been met after conclusion
of the project. Furthermore, the research team found it valuable to use a model that permits
the subject of evaluation to change its established goals in the course of the trial activities
but is still able to judge whether or not the change implies an improvement. The principal
means of supporting improvement work among the AVE organisations, and at the same
time taking care of data collection for the evaluation work, was self-assessment in the
respective educational organisations. All AVE educational organisations were, therefore,
offered a tool for self-assessment to support improvement work. Self-assessment also
constituted a means of collecting data required for the evaluation. A second information
channel was the collection of national statistics for the educational system, which were
then compared with AVE.
Self-assessment in AVE Organisations
A special tool for the self-assessment, based upon TQM and called `Laroverket [1], was
constructed at the Division of Quality Technology & Statistics at Lulea Technological
University (Svensson & Klefsjo, 2000). Here TQM is de ned as `a management system
consisting of core values, methodologies and tools, which support each other in order to
increase customer satisfaction with less amount of resources (Hellsten & Klefsjo, 2000).
Self-assessment, developed and supported by various quality awards during the last
decade, has become established as a tool for quality improvements. In this project,
self-assessment has been introduced as a process consisting of the following four phases:
planning, description of the organisation, analysis of the description, and creation of a plan
for improvement (Svensson & Klefsjo, 2000, p. 238). These four phases are closely related
to the four phases of the improvement cycle `plan do study act described by Deming
(1994).
`Laroverket was developed as a basis for the self-assessment work and is built on the
core values by Bergman and Klefsjo (1994). The structure of `Laroverket is inspired both
by the criteria for the Swedish Quality Award and a tool named `Springboard (Hellsten,
1997a,b). `Laroverket has ve evaluation dimensions and from these a number of typequestions are formulated intended to stimulate the systematic work with quality improvement. The dimensions and the questions are:
1.
2.
3.
4.

Approach: how are we working?

Deployment: to what extent do we do it?
Results: what results does the deployment of the approach lead to?
Re ection: how do we re ect upon our approaches and deployments?

Among Demons and Angels

131

5. Improvement: how do we work to improve our approaches and deployments?

In `Laroverket, business is structured according to four areas, namely, customer co-operation, leadership, employee commitment, and process-management. In each one of the
four areas, approximately 10 questions are asked. In each one of these questions, the three
rst type-questions (approach, deployment, and results) are used as sub-questions. The
two last questions in each area are related to the two last type-questions (re ection and
improvement) and refer to the area as a whole. Formulation of the questions in `Laroverket is, in some cases, adapted to the school system and the AVE evaluation.
Three-day courses about self-assessment are available to support AVE organisations. In
spring 1998, 10 education sessions, including the rst 2 days, were held at eight places
around Sweden. In autumn 1998, the third day was held at six different places. A total of
98 education programmes have participated in the AVE-project. The main reason for
organisations to use `Laroverket was the possibility of getting a base for continuous
improvements. Another reason was the potential for the researcher group to collect data
about how AVE organisations were working. Therefore, organisations were asked to send
their descriptions to the researcher group and 75 education programmes have responded.
The remainder had to answer a special questionnaire instead (Svensson & Klefsjo, 2000).
Support for some parts of the fundamental ideas in tools like `Laroverket can be found
for instance in Garvin (1988), who points out the advantages of using more than a single
de nition of quality in an organisation. Harvey and Green (1993) discuss the opportunity
and the bene ts of using several different de nitions of quality when working with quality
in educational organisations. This reasoning may support use of tools like `Laroverket,
while these tools help organisations working with different aspects of quality development. For instance, the area `customer co-operation in `Laroverket refers to user-based
approaches and the area `process-management at least partly refers to the manufacturingbased approaches.
Survey Studies of the Students
In parallel with the self-assessment instrument, three directed survey studies were conducted with the students. Each of these studies had different sets of questions depending on
the phase of the educational process the students were in.
The rst survey, of 9800 students, was distributed at the start of the course. Its purpose
was to survey, by means of background questions, the characteristics of people who apply
for the course.
The second survey, of 4500 students, was distributed at the mid-point of the course and
aimed at establishing whether the course met the requirements for on-the-job learning and
integration of theory and practice. A further aim of the study was to survey the students
personal impressions, experience and impressions of the education.
The third survey, of 1100 students, was carried out 6 months after the students had
graduated. This survey, carried out with the aid of the organisation Statistics Sweden,
explored the extent to which graduates had received offers of employment.
Local Level: the respective educational organisations
On the local level, the evaluation team went through the dif culties and questions solved
by the individual educational organisations. It was in connection with this level that the
discussion regarding altered grounds for evaluation arose.

132

M. Lindell & M. Svensson

According to Reitberger and Uhlin (1996), older cognitive models of evaluating trial
activities are no longer valid when the trend of development is towards new types of
educational organisations. New technology, new sets of values and new geopolitical
conditions have created not only new educational forms but also new leadership ideals
and new forms of communication. Thus, it is not only the physical, organisational and
social infrastructures that have changed but also the mental ones. Examples that support
Reitberger and Uhlins hypotheses in the AVE case are the ability of the evaluator to
evaluate the part of the education known as on-the-job learning, which accounts for
one-third of the total course time.
On the basis of the above discussion, the research team visited 25 education providers.
During these visits, the school management, a number of teachers and students were
interviewed, together with several administrators and representatives of working life, all
of whom must, according to the ordinance, be associated with all levels of education.
Telephone-interviews were conducted with several representatives of organisations that
have arranged places for the advanced application of theoretical knowledge at a workplace, and of educational co-ordinators.
In summary, the evaluation study of AVE is an example of the concept of system
evaluation in two respects. First, it emphasises multiple channels for input of data, on both
national and local level. Second, it offers educational organisations active participation in
the evaluation process.
Attitudes towards System Evaluation: model for analysis
The Lundin and Soderholm model (1995) has been used to investigate the attitudes
towards the use of system evaluation in the AVE-project. According to this, the encounter
between an evaluator and the activity to be evaluated depends in part on the expectations
the two parties have and in part on how the party being evaluated experiences the
evaluation.
Expectations may either be positive, in the sense that the work of evaluation will lead to
development potential being demonstrated, or negative, in the sense that an evaluation is
seen as a threat to existing activities. In the former case the evaluation will be viewed with
con dence and in the latter with mistrust. Evaluations may also be experienced as
`backward-pointing, where evaluators appear only to look for shortcomings, or as `forward-pointing, where there is a sense of renewal.
By a combination of the con dence mistrust and `forward/backward-pointing dimensions it is possible to construct a four- eld matrix that describes the four different types of
evaluators (Figure 2).
An evaluator who is regarded primarily as an instrument of authority can be denoted
type I. In this role evaluators are viewed with mistrust as they intervene in the life of those
being evaluated with the approval of those in power. This is done, with little interest in the
activity, to record discrepancies that it will later be possible to correct. In addition, the
corrections made will be mainly of a negative nature. Type I evaluators often encounter
various defence mechanisms. Type I lies close to what is perhaps the traditional evaluators
role, that is to say, evaluation as a form of supervision for someone elses purposes and
with a clear power relation in the background.
Type II evaluators represent the face of benign power. They are met with con dence as
they are seen as positive, with a well-developed feeling for how the results of the
evaluation will be handled. Those being evaluated have, in this case, much less fear of
sanctions and punishment than the type I role gave rise to but this is nonetheless a

Among Demons and Angels

133

FIG. 2. Evaluator roles (after Lundin & Soderholm, 1995).

supervising function. To summarise, type I and II, respectively, can be seen as comparable
to the earlier mentioned roles within the `engineers-model.
The characteristic feature of type III evaluators is that their methods do not aim
primarily at looking for and reporting faults and inconsistencies but rather they are
concerned with the future welfare of the subject. Type III is thus the true activity developer
and assistant, with whose aid those being evaluated can hope to develop their job in a
forward-pointing direction. Another important characteristic of type III is the lack of clear,
strong connections with the assigner of evaluators and thus, at least in theory, there is no
possibility of sanctions. A metaphor often used for the type III evaluator role is an `angel.
Finally, type IV evaluators are the representative of an `evil power, who tries, with
various temptations, to induce those being evaluated to enter into certain contracts for the
future that are not perceived as legitimate. It is true that type IV is future-oriented but, in
contrast to type III, it represents an undesirable and threatening future against which it is
important to protect oneself. In evaluation contexts, the type IV evaluator is referred to as
a `demon.
Description of Methods, Selection and Implementation
Telephone-interviews were used to investigate the opinion of those being evaluated within
the AVE-project. The interviews consisted of six questions, divided into two separate parts.
The selection of respondents was made on the basis of strategic considerations. Of the total
population of 123 educational organisations, those organisations and within them those
individuals who, conceptually, were most familiar with the work of the evaluation team,
insofar as that they had participated in part of the `Laroverket education, administered
student inquiries and participated in the programme of visits of the evaluation team. The
selection frame encompassed a total of 22 educational organisers. Before telephone-interviews were performed, the selected educational organisers had received inquiry forms.
The interviews consisted of six questions, and were divided into two separate parts. The
rst part of the telephone-interviews consisted of four questions with xed alternative
answers arranged on a Likert scale (Hellevik, 1984). They were created on the basis of
Lundin & Soderholms analysis model and the concept pairs `con dence/mistrust and
`forward/backward-pointing. Each pair of concepts was operationalised into two contradictory assertions. Each respondents attitude was placed in the four- eld matrix (Figure
2). First, the given alternative responses were graded on the Likert scale with numerical

134

M. Lindell & M. Svensson

FIG. 3. Overall picture of the respondents attitude to the work of the evaluation team; each cross represents
one organisation.

values from 1 to 5. The numerical values assigned to the respondents responses were
compiled for each of the assertions. Also, the mean values for the four concepts were
calculated and entered into the box-plot, which were combined with the four- eld matrix
(Figure 3).
The second part of the inquiry consisted of two questions with open alternative answers.
The questions served to reduce any ambiguity and contradictory answers given in the rst
part. In addition, background data on the respective educational organisations that can be
assumed to bias the answers has been used, that is line of business, ownership form,
geographic location and economic nancing.
Demons, Angels or What?
Of the educational organisations consulted, all 22 respondents answered the rst part of
the telephone inquiry. However, two respondents chose not to answer the rst part of the
inquiry with xed alternative responses with Likert scales, which gave a response frequency of 91%. The investigation shows that 65% of the respondents regard the work of
the research team, built on the system-evaluation model, as an instrument to inspire
con dence and contribute to renewal.
The overall picture of the results of the study (Figure 3) shows that 10 of the respondents
(50%) denoted the work of the research team as forward-pointing. Five respondents (25%)
considered it backward-pointing. The remainder were `undecided. Sixteen (80%) had
con dence in the evaluation and three (15%) mistrusted it. All 10 who indicated forwardpointing evaluation also exhibited con dence in the process. Fifty percent of the sample
denoted the research team as type III (angels). No one identi ed the team as type IV
(demons), although two respondents were on the borderline in the matrix.
Looking more closely at the organisations that express the evaluation of the research
team with con dence and as forward-pointing in the matrix, the data do not give any
evidence that the organisations have something in common that could explain the concentration. However, geographically, there is a tendency for organisations near Lulea Technological University to have greater con dence.

Among Demons and Angels

135

On the other hand, looking at the organisations that express mistrust in the matrix, two
of the three are private companies. If number 16 (in Figure 3), who lies on the boundary
between con dence and mistrust, is also included in this group the result would be that
three of the four who lack con dence in the evaluation project are private companies. Also,
three of the ve who are of the opinion that the evaluation project has not been
forward-pointing have chosen not to work with `Laroverket.
Although the study indicates a positive view of the work of the evaluation team, there
are three conceivable sources of error. First, the small and selective choice of respondents:
given that the selection constitutes only 16% of the population, the results should be
approached with caution. Second, the selected group that was chosen had participated in
the evaluation teams programme of visits. Thus, they have good knowledge of the method
of working but, in addition, one cannot ignore the likelihood that personal contacts and
conversations have toned down earlier critical opinions. Third, the analysis of the interview transcriptions that was done afterwards showed that some respondents had clearly
been bent on giving their judgement of `Laroverket and its use, in spite of the study being
concerned with their views on the work of the evaluation team as a whole.
However, the study shows that a majority of the respondents perceive the work of the
evaluation team as both forward-pointing and con dence-inspiring. Most respondents
perceive the evaluators roles as of type III. Thus, there is support for the use of models of
the systems-evaluation type based on an interactive and learning attitude.
Acknowledgements
We would like to thank Research Engineer Gorgen Edenhagen, Lulea Technological
University, for his help with the telephone-interviews. We would also like to thank our
supervisors, Professor Jan Johansson at the Division of Industrial Work Environment and
Professor Bengt Klefsjo at the Division of Quality Technology & Statistics. Also, we would
like to thank the referee for all the suggestions that have improved this article.
Note
[1] The name `Laroverket is not translated into English, because of dif culties in achieving the two
interpretations in Swedish. It means a secondary grammar school, but can also mean a book, or
something from which you can learn.

References
BERGMAN , B. & KLEFSJO , B., 1994, Quality Satisfaction: from customer needs to customer satisfaction (New York,
McGraw-Hill).
DEMING, W.E., 1994, The New Economics for Industry, Government and Education (Boston, Massachusetts,
Center for Advanced Engineering Study).
FRANKE-WIKBERG, S., 1992, Utvarderingens Mangfald (Stockholm, Universitets-Och Hogskoleambetet).
FRANKE-WIKBERG, S. & LUNDGREN, U.P., 1979, Att Utvardera Utbildning. Del 1 (Stockholm, Wahlstrom &
Widstrand).
GARVIN, D.A., 1988, Managing Quality (New York, The Free Press).
HARVEY, L. & GREEN , D., 1993, `De ning quality, Assessment & Evaluation in Higher Education, 18, pp. 9 35.
HELLEVIK , O., 1984, Forskningsmetoder I Sociologi Och Statsvetenskap (Stockholm, Natur Och Kultur).
HELLSTEN , U., 1997a, `The Springboard a TQM-based tool for self-assessment , Licentiate thesis, Division
Of Quality Technology & Statistics, Lulea Technological University.
HELLSTEN , U., 1997b, `The Springboard: a strategy for continuous improvement of small and medium-sized
companies, Total Quality Management, 8(2/3), pp. 183 186.
HELLSTEN , U. & KLEFSJO, B., 2000, `TQM as a management system consisting of values, techniques and tools,
TQM Magazine, 12(4), pp. 238 244.

136

M. Lindell & M. Svensson

K ARLSSON , O., 1995, Att Utvardera Mot Vad? Om Kriterieproblemet Vid Intressentutvardering (Stockholm, Hls
Forlag).
K IM, L., 1997, `Utvardering Av Hogre Utbildning Pa Nationell Niva Systemutvardering Eller Kvalitetskontroll? , paper presented at the 1st Conference on Quality in Higher Education (Uppsala, National Agency
for Higher Education).
LINDELL , M., 2001, `Trends and development in vet systems: exibility, transferability, and mobility issues,
International Journal of Vocational Education and Training, 9(2), pp. 33 46.
nglar Och Demoner. Om Utvarderarroller , in ROMBACH, B.
LUNDIN , R.A. & SO DERHOLM, A., 1995, `Bland A
& SAHLIN-ANDERSSON , K. (Eds.) Fran Sanningssokande Till Styrmedel. Moderna Utvarderingar I Offentlig
Sektor (Stockholm, Nerenius & Santerus Forlag).
MULLER , D. & FUNNELL , P., 1992, `An exploration of the concept of quality in vocational education and
training, Educational & Training Technology International, 5, pp. 257 261.
PROPOSITION, 2000/01, `Kvali cerad Yrkesutbildning [`Advanced vocational education], Swedish Government Proposition Series, No. 63, Stockholm.
., 1996, `Vaxtkraft Mal 4 Ett Mangdimensionellt Och Dynamiskt Larsystem,
REITBERGER, G. & UHLIN , A
personal correspondence.
SOU, 1999, Evaluation of Advanced Vocational Education, Swedish Of cial Report Series (Stockholm, Ministry
of Education and Science).
STEVRIN, P., 1991, Utvardering for Forandring (Lund, Studentlitteratur).
SVENSSON , M. & KLEFSJO, B., 2000, `Experiences from creating AV quality culture for continuous improvements in Swedish school sector by using self-assessments , Total Quality Management, 11, pp. 800 807.
VEDUNG , E., 1995, `Utvardering Och De Sex Anvandningarna, in ROMBACH, B. & SAHLIN- ANDERSSON , K.
(Eds.) Fran Sanningssokande Till Styrmedel. Moderna Utvarderingar I Offentlig Sektor (Stockholm, Nerenius
& Santerus Forlag).