Vous êtes sur la page 1sur 22

http://evi.sagepub.

com
Evaluation
DOI: 10.1177/1356389003094002
2003; 9; 383 Evaluation
Mara Bustelo
Evaluation of Gender Mainstreaming: Ideas from a Meta-Evaluation Study
http://evi.sagepub.com/cgi/content/abstract/9/4/383
The online version of this article can be found at:
Published by:
http://www.sagepublications.com
On behalf of:
The Tavistock Institute
can be found at: Evaluation Additional services and information for
http://evi.sagepub.com/cgi/alerts Email Alerts:
http://evi.sagepub.com/subscriptions Subscriptions:
http://www.sagepub.com/journalsReprints.nav Reprints:
http://www.sagepub.com/journalsPermissions.nav Permissions:
http://evi.sagepub.com/cgi/content/refs/9/4/383
SAGE Journals Online and HighWire Press platforms):
(this article cites 3 articles hosted on the Citations
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
Evaluation of Gender Mainstreaming
Ideas from a Meta-evaluation Study
1
MAR A BUS TEL O
Complutense University at Madrid, Spain
The new policy strategy of gender mainstreaming poses particular
challenges for the evaluation of public gender-equality policies. To elaborate
on this issue, a rst step is to analyse the evaluation of those public gender-
equality policies that have formally adopted the gender mainstreaming
strategy. In this article, results are summarized from a meta-evaluation of 11
evaluation processes of gender-equality plans implemented between 1995
and 1999, both at regional and national levels in Spain. This meta-evaluation
focused on analysing the evaluation processes rather than the outcomes of
those evaluations. First, the different types of evaluation carried out in the
11 studies are discussed. Second, some contextual factors are identied as
inuencing elements in the evaluations. Third, some conclusions and lessons
learnt are presented, using the framework provided by the meta-evaluation
criteria previously established. Finally, taking into account those lessons, a
discussion of the evaluation of gender mainstreaming is presented,
elaborating on the ways in which gender mainstreaming strategies and
gender perspective can be evaluated and which should be used to conduct
useful evaluations.
KEYWORDS: gender-equality policies; gender mainstreaming; meta-evaluation;
programme and public-policy evaluation; public administration
Introduction
Public policies aiming to promote gender equality have evolved substantially in
the last decades. Ever since 1975, when the United Nations established Womens
International Year, and most of the western nations started to acknowledge
gender inequality then known as womens discrimination as a public issue
that deserved public intervention, the strategies and political instruments of
those policies have been changing. Focus on sex discrimination (discrimination
based on biological differences) and especially womens discrimination has
evolved to focus on gender (based on the cultural and social consequences of
those biological differences). In addition to specic actions for women positive
actions gender mainstreaming emerged as a new and necessary strategy for
combating gender inequality in the long term. Gender mainstreaming means
Evaluation
Copyright 2003
SAGE Publications (London,
Thousand Oaks and New Delhi)
[13563890 (200310)9:4; 383403; 040429]
Vol 9(4): 383403
383
02 Bustelo (ds) 19/11/03 11:34 am Page 383
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
that, in addition to specic policies addressing gender discrimination which are
still necessary to deal with actual gender discrimination there is a need to look
for a gender perspective in all public policies.
2
This strategy implies a much more
complex and widespread political action, and a revolutionary change in the
process of policy making. Thus, evaluation of gender-equality policies must take
into account this new strategy and the challenges it poses about the way in which
those policies should be evaluated. How should gender-equality public policies
be evaluated if gender mainstreaming is the assumed strategy? How can gender
mainstreaming itself be evaluated?
The rst step for elaborating on that issue is to analyse the way public gender-
equality policies are currently evaluated. If these policies formally assume
gender mainstreaming as a strategy in their formulation, and, supposedly, in
their implementation, how do they deal with it in the evaluation phase? A good
way to analyse and assess evaluations is by means of meta-evaluations. Meta-
evaluation refers to the evaluation of evaluations and, in addition to assessing
evaluation quality, it may contribute to the description, analysis and assessment
of evaluation studies or processes.
Alternatively, the evaluation phase has been the least studied in the area of
policy analysis and public policy studies. However, governments and policy
makers must evaluate in order to obtain explicit feedback to adjust their actions.
There are other information sources, such as polls, statistics, observatories,
investigations and so on, but evaluation is an especially useful tool for facilitating
decision making for the improvement, accountability and enlightenment of
public policies and for organizational learning. Although the practice of public
policy and programme evaluation has developed rapidly in recent years, this is
not the case for the study of their evaluation.
3
Thus, there is a strong need for
studying and analysing public policies with specic regard to their evaluation.
This is not only a way to better understand those policies, but also to improve
evaluation processes.
This article proceeds from a meta-evaluation in which 11 evaluation experi-
ences of public gender-equality policies in Spain, at the national and regional
levels, were analysed, compared and evaluated. In that work, we did not conceive
meta-evaluation as a procedure for evaluation quality control, but as a means to
study and analyse public gender-equality policies, in the specic aspect of their
evaluation. Therefore, instead of evaluation quality, we focused on its suitability
to the evaluative and political context and its contribution to the performance of
the evaluation. In addition, we proposed to analyse whether or not those evalu-
ations took into account the new strategy of gender mainstreaming, and shed
some light on how gender-equality evaluation may be sensitive to that strategy.
The Case Study: The Evaluation of Public Gender-equality
Policies in Spain
The starting point in Spain of the so-called state feminism or Femocracies (e.g.
Meehan and Sevenhuijsen, 1991; Stetson and Mazur, 1995; Gardiner, 1997) can
be set at the end of 1983, when the Instituto de la Mujer (Womens Institute) was
Evaluation 9(4)
384
02 Bustelo (ds) 19/11/03 11:34 am Page 384
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
created at the national level (Valiente, 1995, 1996, 1997). From that time on, what
has been called state feminism extended to the regional governments (Comu-
nidades Autnomas), which created their own womens agencies between 1988
and 1994, (Bustelo, 1999a; Granados, 1999).
4
State feminism has been the
driving force behind gender-equality policies in Spain.
Similar to the Instituto de la Mujer at the national level, regional gender-
equality agencies present themselves not as the direct executors of gender-
equality policies, but as the promoter and co-ordinator of such policies. Their
aim is to persuade other government units to assume and develop objectives and
activities to promote gender equality in their respective areas. For this purpose,
the main policy instrument is the elaboration and approval of gender-equality
plans, which refer to gender mainstreaming as an important strategy. These plans
consist of a structured set of initiatives in the various areas affecting women and
gender inequality, and almost all the government departments are involved in
their implementation. These plans are approved by the national or autonomous
(regional) cabinets for a certain period of time (usually from three to ve years).
The elaboration and implementation of these gender-equality plans have
entailed, in some cases, parallel evaluation processes. That is, evaluation of some
of the plans has begun, although in a heterogeneous and irregular way. Similarly,
there have been several generations of gender-equality plans in Spain (at the
national and the regional levels), and generations of evaluations of those plans
can be distinguished. The rst generation consisted of four very limited evalu-
ation experiences that took place in the early 1990s. The second-generation
evaluations are the object of this investigation and consisted of 11 evaluation
experiences at the national and regional levels. These evaluation processes took
place from 1995 to 1999.
As mentioned, the rst-generation evaluation attempts were very feeble and
limited to checking the implementation of the actions proposed in the plans. This
weakness was to be expected, not only because of the lack of previous experi-
ences and of an evaluation culture and competence, but also because the tech-
nical quality of those rst plans was poorer than that of the subsequent ones. The
second-generation evaluation experiences were also limited. However, all these
experiences are of value by themselves, just because they exist and, in some way,
were milestones for the evaluation of gender-equality policies in Spain. Also, due
to these experiences, an evaluation culture has been created and the evaluation
of gender-equality plans has become widespread. In almost all the third-gener-
ation gender-equality plans, some kind of evaluation is included within their
design, which was not the case with the rst two generations of plans.
Methodology
Eleven evaluation processes, among the regional and national gender-equality
plans, were selected for the meta-evaluation study (one evaluation of the national
plan and 10 of regional plans). All second-generation evaluations, which actually
took place between 1995 and 1999, were selected. We decided to select all the
existing evaluation processes of the overall plans, excluding evaluations of
Bustelo: Evaluation of Gender Mainstreaming
385
02 Bustelo (ds) 19/11/03 11:34 am Page 385
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
specic programmes within the plans. Although, in some cases, it was not clear
whether these experiences were real evaluations, they were included in the study,
providing that the womens agency regarded them as such and referred to them
as evaluations of the respective plans.
As explained above, the evaluation processes themselves, and not their results,
were the focus of the study. From the meta-evaluation perspective, the purpose
of the study was to evaluate those evaluation processes. In order to dene the
assessment criteria of the evaluation processes, an evaluation framework was
established, based especially on the works of:
Patton (1980, 1997): importance of useful evaluations and utilization, stake-
holder participation, exibility, context and communication;
Weiss (1987, 1998): political nature of evaluation, importance of context and
actors;
Scriven (1995): concept and centrality of systematic judgment in evaluation;
and
Chelimsky (1998): implications of the political nature of the evaluation,
including the need for credibility and timeliness.
This evaluation framework was also reinforced by the authors previous experi-
ence as an evaluation consultant.
Thus, the (meta) evaluation criteria were established around three main
aspects:
1. The overall design of the evaluation processes, including their responsive-
ness to context, clarity of purposes and objectives, existence of institutional
structures and resources for the evaluation, and the utilization of different
types of evaluations.
2. Key evaluation elements, such as the stakeholders involved in the evaluation
processes, the time when these processes took place, the evaluation criteria
established and the methodology and techniques used for gathering and
analysing information.
3. The evaluation utilization, taking into account the adequacy and usefulness
of the information produced, its communication and dissemination
processes, and its utilization and impact both on gender-equality policies
and on womens agencies.
The techniques for gathering and analysing information employed in the meta-
evaluation included document collection and analysis, on-site visits to womens
agencies, 17 in-depth interviews with the heads of womens agencies, and tele-
phone and on-site interviews with the technicians in charge of the evaluation in
the womens agencies.
Types of Evaluation
In order to present a summary of the meta-evaluation results, rst, the types of
evaluations are described. Four basic classications are used to describe the types
of evaluation performed: formative versus summative evaluation; design, process
Evaluation 9(4)
386
02 Bustelo (ds) 19/11/03 11:34 am Page 386
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
or results evaluation; ex ante versus ex post facto evaluation; and internal versus
external evaluation (see Table 1 for comparisons).
Evaluation role All 11 experiences analysed are summative. A summative
evaluation occurs when the role is one of recapitulation of the plan or programme
evaluated. A formative evaluation is used as an instrument for changing the plan
during the implementation process. There are three regional evaluations that could
be considered partially formative. In the case of the evaluation of the rst plan in
Aragn, to some extent, it was formative in nature because it was implemented
after a political change in the regional government, and its purpose was mainly to
establish priorities for the last year of the evaluated plan. In the other two cases
(La Rioja and Pas Vasco), annual reports were made, so that the evaluation could
be used for correcting certain aspects during plan implementation. However,
Bustelo: Evaluation of Gender Mainstreaming
387
Table 1. Types of Evaluation in the Second-generation Evaluations of Gender-equality
Plans in Spain
Region Role Content Timing Agent
Instituto Mujer Summative Process and Ex post facto Mixed:
II (199395) results internal design,
(National) part of external
implementation
Andaluca Summative Design, process Ex post facto External
II (199597) and results
Aragn Formative/ Process and Ex post facto Internal
I (199496) Summative results
Canarias Summative Process and Ex post facto External
I (199596) results
Castilla-La Mancha Summative Design, process Ex post facto External
II (199599) and results
Castilla y Len Summative Results Ex post facto External
I (199496)
Catalua Summative Design, process Ex post facto Mixed:
II (199496) and results external technical
assistance in design,
internal
implementation
Galicia Summative Design, process Ex post facto External
II (199597) and results
Madrid Summative Process and Ex post facto External
II (199395) results
La Rioja Formative/ Process and Ex post facto Internal
II (199699) Summative results
Pas Vasco Formative/ Design, process Ex post facto Mixed:
II (19962000) Summative and results external technical
assistance in design,
internal
implementation
02 Bustelo (ds) 19/11/03 11:34 am Page 387
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
collecting information at the beginning and continuously does not necessarily imply
a formative nature. In fact, the annual reports of the Basque Womens Institute pre-
sented in Parliament are more summative in nature because they emphasize the
accountability function and the recapitulation of what had been accomplished each
year. In La Rioja, the collection of information from the beginning was more due
to the need to keep records during the process in order to evaluate at the end; so
it appears that the intention was more summative than formative.
Evaluation content Three types of evaluation are distinguished: design evalu-
ation, process evaluation and results evaluation. A design evaluation refers to the
assessment of the plan as a whole, including evaluation of the plans rationale,
objectives, internal coherence, strategies for implementation, etc. Only in two of
the evaluation experiences (Andaluca and Castilla-La Mancha) is a design evalu-
ation explicitly considered. From these two experiences, the Andalusian is the
more complete, considering more design aspects, including analysis of the
(in)equality concept for which the plan was designed. Another three experiences
(Catalua, Galicia and Pas Vasco) contain some design evaluation elements.
These experiences analysed the types of activities or actions implemented, target
population and budget, and some interesting classications were made. This kind
of analysis allows assessing the plan as a whole and provides interesting infor-
mation about the relative importance of each kind of action or activities. This is
also, to some extent, a way to evaluate the design of the plans.
Regarding process evaluation, all the experiences, excepting one, formally
included this, but this kind of evaluation was often wrongly conceived. In most
cases, the so-called process or implementation evaluation consisted exclusively
of assessing the accomplishment level, a term referring to a general inspection
of the actual implementation of the planned actions. The author suggests that
these kinds of evaluations could be considered more as a product or output evalu-
ation than a process evaluation. Indeed, a process evaluation should assess the
implementation process, timetable adjustment, whether the target population is
being reached, difculties that arise during that implementation process (infor-
mation, time or location problems) and co-ordination processes with other
departments and agencies required for that implementation. This last aspect is
especially important for the evaluation of gender-equality plans and the gender
mainstreaming strategy, because the plans involve many stakeholders and need
a lot of co-ordination for their implementation. However, the analysis of the
experiences found that these aspects were not all evaluated. The more complete
process evaluations were carried out in Andaluca, Castilla-La Mancha, La Rioja
and Pas Vasco, although they still have some limitations.
The results evaluation can be divided into product or output evaluation and
effect or outcome evaluation. Some product evaluation was carried out in
almost all the experiences analysed by means of the aforementioned accomplish-
ment level: by the study of what had been generated by the plan, its products
or outputs. However, in some experiences, this was limited to simply verifying
which planned actions had actually been implemented. Rarely was the quality of
the products assessed.
Evaluation 9(4)
388
02 Bustelo (ds) 19/11/03 11:34 am Page 388
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
In three of the experiences analysed (the second national plan and those of
Madrid and Castilla y Len), an effects assessment was aimed for, although they
were not really achieved. The experiences were very good studies of womens
social status but they did not assess plan effects. It is well known how difcult it is
to assess the so-called net effects of a programme or plan (i.e. the effects attribu-
table to the plan and not to other factors: effects that would have never occurred
in the absence of the programme or plan). In the three above-mentioned cases,
several variables related somehow to the plan objectives (e.g. increases of women
at university) were measured at the beginning and at the end of the plans. These
measures were compared, indicating their evolution during that period of time. But
there is no variable or factor connecting that evolution to the existence of the plan;
that is, if there were any changes, they might have been caused by many other
factors and not only by the plan implementation. The only experience that had
planned an actual effect evaluation was the evaluation of the second Andalusian
plan (it had an evaluation plan which included four different surveys, combining
different variables, moments and populations, and a womens social-status study
carried out in 1988). However, due to various factors, that evaluation was not
completed in the way it was planned at the beginning of the evaluation period.
Evaluation timing All the second-generation evaluation experiences were ex
post facto, launched when the plan had already begun. There were no ex ante
evaluations, including a previous evaluation design or pilot study of some actions
in order to subsequently extend them. Moreover, one of the main difculties of
this generation of evaluations is not their ex post facto nature itself, but the fact
that the evaluation design and implementation usually started very late (when a
large part of the plan had already been implemented). This circumstance weakens
the evaluation tool, rstly, because it is not designed parallel to the plan, and sec-
ondly, because the opportunity of collecting valuable information from the begin-
ning of the implementation process is lost. The only two evaluation experiences
that were planned from the beginning were those in the Pas Vasco and La Rioja.
The others started shortly before the last year of their plan periods.
Evaluation agent Evaluations may be internal or external, depending on
whether or not they are conducted by someone who is related to the nancing,
planning or execution of the evaluation plan. There were three mixed experi-
ences, although each somewhat different. In the case of the second national plan,
the evaluation design was internal, and an external team participated in the infor-
mation collection and analysis of only one part of the evaluation study. However,
in the Basque and the Catalonian cases, an external team was employed for the
evaluation design, although part of the evaluation implementation was internal.
Contextual Factors that Inuence Gender-equality Policy
Evaluation
Some contextual factors were identied as elements that inuence the evaluation
of gender-equality policies, either positively or negatively. We called them
Bustelo: Evaluation of Gender Mainstreaming
389
02 Bustelo (ds) 19/11/03 11:34 am Page 389
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
contextual because they do not directly involve the evaluation processes, but are
institutional factors concerning gender-equality policies or specic characteristics
of those policies. By means of the meta-evaluation, 10 contextual factors were
identied, ve with positive inuence on the evaluation, and another ve with
negative inuence.
Factors that Inuence Gender-equality Policy Evaluation Positively
The existence of structured plans The fact that plans are the main policy instru-
ment facilitates evaluation. All public actions are included in a structured plan,
including objectives and actions, a time dimension and resources assigned (at
least, formally). Moreover, the existence of a plan increases government aware-
ness of what is being done and what needs to be done. Evaluation is a good way
to keep governments and policy makers informed.
The increase of formal commitments concerning gender-equality plans and their
evaluation Gender-equality plans in Spain are approved by the respective
Cabinet (either at the national or regional level). This may include some sort of
commitment to plan evaluation.
International support for gender-equality policies The 4th World Conference
on Women in Beijing in 1995 and the 4th Action Programme on Equal
Opportunities between Men and Women (19962000) of the European Union
were especially important. Moreover, these international forces have led to some
formal agreement on the need for gender-equality policies and the strategies for
implementing them specic policies and gender mainstreaming. Theoretically,
gender-equality policies should be revised, monitored and evaluated.
The existence of consolidated gender-equality organisms Currently in Spain,
although heterogeneously, there is a relatively powerful womens machinery or
state feminism at the national and regional levels.
Acknowledgement of programme and policy evaluation in recent years
Although this acknowledgement is mainly symbolic it is declared that public
programmes and policies must be evaluated, although usually they are not
evaluation is currently fashionable. This is a big step and could be a facilitating
factor for real evaluation.
Factors which Inuence Gender-equality Policy Evaluation Negatively
The generality of gender-equality plans Although, across generations of plans,
they have become more specic, they are still quite general because of their
nature multi-dimensional and with many stakeholders involved. Evaluating
general and ambiguous plans is much more difcult than evaluating specic ones.
In the authors opinion, this generality is partly an indicator of a lack of clarity
about what should be done and how; i.e. the programme theory and the
implementation theory (Weiss, 1998) of the gender-equality plans are not
Evaluation 9(4)
390
02 Bustelo (ds) 19/11/03 11:34 am Page 390
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
explicitly clear.
5
We still need not only to keep on investigating gender relations
but also to translate theory and evidence into practice, implementing sound
policies and plans.
The integral and horizontal nature of gender-equality plans Gender-equality
plans are integral in nature: they are intended to solve a problem (gender inequal-
ity) from all the possible points of view, containing actions involving education,
health, employment, culture, social and political participation, and so on.
Therefore, they are complex and multi-dimensional in nature, making their
evaluation more difcult. In addition, discrimination at the base of gender
relationships is, itself, very complex and profoundly rooted in social structures.
Combating that discrimination is also complex, because it implies a different
outlook and a different way of analysing and dealing with reality. Moreover, plans
are conceived as horizontal, proposing actions through regular public policies
and with gender mainstreaming as the general strategy to follow. For that
purpose, the dynamic and co-ordinated engagement of multiple actors and stake-
holders is needed.
The policy of persuasion Because of gender mainstreaming and the horizontal
nature of plans, femocrats must persuade their colleagues in the government to
implement and become responsible for gender-equality actions, which are a part
of those plans. It is very interesting to analyse the strategies that femocrats
employ for this purpose but, in general, we think that the personal factor is a key.
This is a weakness, and quite difcult to evaluate. Femocrats have no power to
sanction their colleagues if they decide to pay no real attention to gender issues,
so the strategies they use for persuading them tend to be informal and personal,
for example, appealing to statistics, being patient, repeating the same things
over and over, appealing to justice, keeping a low prole or no direct con-
frontation, allowing colleagues to be political protagonists of gender actions,
appealing to colleagues personal experience, using a sense of humour and so
on (Bustelo, 2001: 41215).
Lack of experience, culture of evaluation and consolidation of the evaluation
function within Spanish public administrations
6
For these second-generation
experiences, there were practically no previous references to gender-equality
plan evaluation. Thus, evaluation was something new and, in a sense, nobody
knew exactly how to do it. Also, because evaluation is not yet a part of the regular
and daily tasks in public administrations, it was rarely conceived at the appropri-
ate time, leaving it until almost the end of the plan.
Lack of competent and prepared teams for the evaluation of gender-equality
policies Related to the former factor is the fact that, at that time in Spain, there
was little evaluation specialization at the academic or the professional levels.
Moreover, it is quite difcult to nd evaluators who, in addition, are knowledge-
able and sensitive about gender issues.
Bustelo: Evaluation of Gender Mainstreaming
391
02 Bustelo (ds) 19/11/03 11:34 am Page 391
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
Results and Lessons Learnt
Some broad results are now summarized, following the meta-evaluation
criteria established and provided earlier in this article. Firstly, for each of the
three main issues (overall evaluation design, key evaluation elements and
evaluation utilization), some general conclusions are drawn from the evidence
found about the state of evaluation practice in this area. Secondly, and follow-
ing the format of the conclusions, some lessons learnt are commented upon.
For each lesson, some evidence is also presented. These lessons learnt from
the analysed evaluation experiences may also contribute to: the general
discussion about which aspects of gender mainstreaming can be evaluated and
how; and guide the improvement and optimization of the evaluation function
of gender policies.
Some Results and Conclusions about the State of the Evaluation
Practice in the Field
For the overall design of the evaluation processes, we analysed evaluation
responsiveness to context, clarity of evaluation purposes and objectives, exist-
ence of institutional structures and resources for the evaluation, and utilization
of different types of evaluations (discussed above).
Regarding the rst two issues (responsiveness to context and clarity of evalu-
ation purposes), we found evidence that may be summarized in the following
conclusions.
1. There was a lack of clarity in the evaluation purposes and this led to con-
fusion of gender-equality policies and plan evaluation, and the investi-
gation of the status of women.
2. There was also a lack of a global outlook of public action taken to promote
gender equality. That is, we found a generalized trend to evaluate specic
actions and plans instead of policies.
3. We also noted the lack of acknowledgement of the political nature of
evaluation, considering plan evaluation to be essentially technical.
Regarding the institutional structures and the resources invested in evaluation,
we found the following.
4. It was very important to have adequate institutional structures for the
evaluation processes. Given the horizontal nature of the evaluated plans,
these structures have positive inuence on the complex but necessary
gathering of information.
5. Generally speaking, the resources invested in these second-generation
evaluations were very poor.
The key evaluation elements that we took into account were the stakeholders
involved, timing, the evaluation criteria established and the methodology and
techniques used. The following was found regarding these key elements.
6. Evaluation processes were exclusively run by technicians, either internal
or external (evaluation was generally conceived as an essentially
Evaluation 9(4)
392
02 Bustelo (ds) 19/11/03 11:34 am Page 392
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
technical endeavour). This led to the result that the involvement of other
stakeholders seldom occurred.
7. Most of the experiences analysed were planned almost at the end of the
implementation period of the plan. This led to serious difculties in infor-
mation collection.
8. There was no rigorous and previous establishment of evaluation criteria.
As a result, the evaluation experiences were organized around specic
information-gathering techniques instead of around evaluation criteria.
Finally, regarding evaluation utilization, we took into account the adequacy
and usefulness of the produced information, communication and dissemination
processes, and the evaluation impact both on gender-equality policies and on
womens agencies.
9. We found an important decit in almost all the experiences in what we call
practical elaboration: the establishment of judgements and recommen-
dations.
10. Generally speaking, communication and dissemination of the evaluation
processes and results were also very poor.
11. Except for the case of the Basque Womens Institute, evaluation was
perceived as a secondary function in the national and regional womens
agencies.
Lessons Learnt
1. There is a need to distinguish between gender-equality policy evaluation and
investigation of gender inequality or the status of women.
Analysing the evaluation experiences, the conclusion was reached that it is neces-
sary to distinguish between research on the status of women and evaluation of
gender-equality plans and policies. We have already observed that these two
concepts were confused in some experiences, such as those carried out in Castilla
y Len, Madrid and at the national level. Essentially, these investigations
produced indexes for measuring gender inequality. But, clearly, they are not
evaluations of gender-equality plans because there was no direct or formal
linkage between the variables studied and the plan being evaluated.
There are many reasons for distinguishing research from evaluation (Bustelo,
1999b), but here two are highlighted. The rst is related to the object. In this
case, the object of research is gender inequality (how the status of women differs
from that of men) and its evolution over time. The object of evaluation, however,
is the way in which the authorities deal with the gender inequality problem and
how public actions inuence the public issue of gender inequality: the plans and
policies themselves, not the problem they are meant to address. Secondly, there
are differences in the purposes, with improvement and accountability being the
main purposes of evaluation, and knowledge construction the purpose of
research.
Obviously, research on gender inequality and the status of women contributes
to better policy formulation. But this does not preclude the need for evaluating
Bustelo: Evaluation of Gender Mainstreaming
393
02 Bustelo (ds) 19/11/03 11:34 am Page 393
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
those policies. Both activities investigation and evaluation are crucial for
womens agencies. Moreover, the so-called equality observatories (whose aim
is to collect information periodically about gender inequality) are very useful for
evaluating gender-equality policies, in addition to their utility for policy formu-
lation. But the very fact of their existence does not imply evaluation.
2. Evaluations should enhance a global view of the public action taken to
promote gender equality. In addition to programme evaluation, a broader policy
evaluation perspective is required.
None of the experiences analysed is an evaluation of the global gender-equality
policy, but rather, they evaluate specific plans approved by their respective
governments. The aim of the plans is to promote, co-ordinate and gather all
the government actions regarding gender equality so that, in a sense, the best
way to evaluate gender-equality policies is to evaluate gender-equality plans.
Plan evaluation facilitates observation of the development and results of that
plan but does not question the role of the plan or the plans themselves. Issues
concerning the adequacy of the plan as the main policy tool, the actual effec-
tiveness of the gender mainstreaming strategy and its pertinence to the role
and functions of womens agencies were not addressed in the evaluations
studied.
3. There is a need to acknowledge the political nature of evaluation.
In most cases, the evaluation function was considered an exclusively technical
matter, leaving key elements in the evaluation processes unattended. This
circumstance has, in part, led to useless or residual evaluation experiences.
This acknowledgement of the political nature of evaluation has been increas-
ing in the last few years. Weiss was one of the rst authors who pointed out the
relationship between politics and evaluation:
1. policies and programmes are products of political decisions;
2. evaluation is intended to feed the decision-making process; and
3. by its own nature, evaluation implicitly establishes programme problems,
needs, etc., and this implies a political stance (Weiss, 1987).
But lately, this political nature has become a key issue not only for evaluation
practice but also for evaluation theory. Chelimsky (1998) considered it necessary
to have a view of politics as central to evaluation. Chelimsky (1998: 39) states
still, even today, we see politics as merely the context of an evaluation, as
something that intrudes upon good practice, rather than the engine driving it.
This acknowledgement entails the need to extend the way we think about politics
and policy making, the need for credibility, timeliness and exibility of the evalu-
ation processes.
Evaluation 9(4)
394
02 Bustelo (ds) 19/11/03 11:34 am Page 394
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
4. The importance of adequate institutional and co-ordination structures for
evaluation.
One of the main problems, especially mentioned by the femocrats in charge of
the evaluation processes, was the difculty of gathering information from other
government units that participate actively in the implementation of the evalu-
ated plan. As a result of the horizontal nature of gender-equality plans, much of
the information required was not held in the womens agency concerned (which
was the responsible unit for the evaluation) but in various other government
units. This task of collecting information was not an easy one.
After the analysis, we concluded that it is extremely important to design and
have adequate institutional and co-ordination structures for the evaluation,
which allow the collection of valid and reliable information. We think this is even
more important than the specic techniques used for data collection and analysis.
Having a valid and reliable instrument for gathering information is a necessary
but not a sufcient condition to obtain the relevant and reliable information for
evaluating the plan.
In addition to having appropriate information systems, there is a need for
adequate co-ordination structures. This is especially important, for at least the
following two reasons.
Due to the horizontal nature of gender-equality plans, their actions are imple-
mented by different agencies, so that the information is going to be spread
out; thus, it is necessary to design systems that allow the centralizing of that
information in a continuous, stable and homogeneous manner.
For an evaluation, not only is the information that is gathered ad hoc very
important, but also the information produced by and for plan implemen-
tation and management. Availability of valid information of that kind
implies less effort in the evaluation and a better opportunity for under-
standing the evaluated programme.
We analysed the different co-ordination structures of the 11 evaluation experi-
ences and we concluded that the involvement of the following three types of
structures might drive evaluation.
1. Structures linked to the plans and to womens agencies (it is benecial for
these general structures to have responsibilities, at least formally, in the
evaluation).
2. Structures linked to the legislative power (the legislative control over the
executive power is a very good stimulus for evaluation).
3. Formal structures for womens participation (we found almost no partici-
pation of this type in the processes studied).
Thus, the existence of an explicit evaluation commitment within the plan or the
commitment to report to the parliament, the existence of ad hoc government
committees for monitoring and evaluating the plan, the existence of a partici-
pation mechanism in the monitoring and evaluating processes, were all key and
facilitating elements for evaluation.
Bustelo: Evaluation of Gender Mainstreaming
395
02 Bustelo (ds) 19/11/03 11:34 am Page 395
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
5. There is a need for greater resource investment in evaluation.
It was almost impossible to measure the real investment of Spanish womens
agencies in the evaluation of second-generation gender-equality plans. It was
easy to calculate the direct evaluation expenses in the external experiences, but
it was rather difcult to objectively estimate the indirect expenses, both in
external and internal evaluations (cost of time spent by internal professionals on
the evaluation processes).
Nevertheless, we estimated investment in evaluation by means of the direct
expenses employed. For this purpose, we compared the direct evaluation
expenses of the external experiences using two different gures: plan expenses
and womens agency expenses during the plan period. However, in some cases,
we could not even do this, because there was no specic estimation of the general
plan expenses. Logically, when those estimates were available, the general plan
expenses were much higher than the womens agencies expenses, because other
government units, and not only the respective womens agency, contributed to
the plan. Although with limitations, after the analysis, we were able to conclude
that investment in evaluation was extremely low. For example, the most expen-
sive external experience (which was three times more expensive than the next
one, in absolute terms), the evaluation of the second Andalusian plan, represents
0.97 percent of the agencys expenses (Instituto Andaluz de la Mujer) and only
0.07 percent of the general plan expenses.
7
Obviously, there is a need for
increased resources.
6. There is a need for promoting greater stakeholder participation in the evalu-
ation processes.
Active stakeholder participation is considered to be a key element for obtaining
context-responsive and useful evaluations (Patton, 1997; Monnier, 1995; Weiss,
1998). The use of stakeholders opinions as a source of information does not
imply that those stakeholders participate in the evaluation process. That is why
we mention active participation, that is, in one or more parts of the evaluation
process: study design, denition of and decisions about evaluation criteria, tech-
nique selection, information gathering, analysis and judgement, establishment of
recommendations, report writing, and communication of results. Curiously
enough, because gender-equality plans are conceived as horizontal, proposing
actions by means of regular public policies, dynamic and co-ordinated engage-
ment of multiple actors and stakeholders is needed, and this should have been
taken into account in the evaluation.
However, although participation of the womens movement and other actors
in public gender-equality policies is considered an important strong point for
those policies, generally speaking, stakeholder participation in the evaluations
analysed was almost nonexistent. This was partially due to the technical charac-
ter of the evaluation processes, with evaluation considered exclusively an affair
of professionals or specialists. In some cases, there was symbolic participation by
the womens movement, because they were invited to the presentation of the
Evaluation 9(4)
396
02 Bustelo (ds) 19/11/03 11:34 am Page 396
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
evaluation results. Formal representative structures (e.g. Consejos de la Mujer)
had practically no opportunity to participate in the evaluation processes.
However, a lack of participation in the evaluation phase was to be expected;
Spanish public gender-equality policies have always been half-hearted about the
participation of the Spanish womens movement in the other phases of policy-
making processes (formulation and implementation).
7. The importance of timeliness.
There are two time-related issues, which are key elements for evaluation
processes, and gender-equality policies are no exception. The rst issue is the
moment in which the evaluation is planned and started. Evaluation is a rather
more powerful tool if it is planned from the start and in a continuous way. As
already pointed out, most of the evaluations analysed were planned almost at
the end of the implementation period of the plan, and this led to serious dif-
culties in data collection.
Secondly, there is a need for timeliness in the evaluation processes. Evaluation
criteria arise in a specic context, situation and moment. If much time is required
to answer evaluation questions, they might become irrelevant or the beneciaries
might not be the same. Evaluation responsiveness is clearly linked to the evalu-
ation timetable. The time factor has deeply determined the experiences analysed,
and delays in some evaluation processes have decreased the chances of it being
useful.
8. The importance of evaluation criteria as evaluation organizers.
One of the rst and crucial steps in the evaluation process is the determination
of what is going to be evaluated. As already mentioned, the evaluation criteria
determine and guide the evaluation study, being the main orientation for the
gathering and analysis of the necessary information. That is, evaluation criteria
determine which variables to study and, also, the logic base for systematically
judging the information previously gathered and analysed.
In the experiences analysed, there seems to have been no previous and
profound reection about what to evaluate. Mostly, the evaluation criteria were
established and dened by means of the specic techniques used, giving priority
to this term (what we have called the dictatorship of methodology and tech-
niques). A good example is the evaluation of the second national plan, a mixture
of three different studies organized by the techniques applied.
8
9. The importance of judgement and recommendations as essential parts of
practical evaluation.
In addition to information collection and interpretation, evaluation involves two
more levels of analysis: systematic judging (guided by the evaluation criteria
previously set) and establishing recommendations (Patton, 1980). Judging and
establishing recommendations is what we call practical elaboration because, in
Bustelo: Evaluation of Gender Mainstreaming
397
02 Bustelo (ds) 19/11/03 11:34 am Page 397
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
order to be useful and used, information must be elaborated or translated into
specic and relevant guides of action. It is clear that evaluation should be useful,
practical and action-oriented for the improvement, accountability and enlighten-
ment functions. Through the meta-evaluation study, the author concluded that
these last analysis phases judging and establishing recommendations were
almost nonexistent in the experiences analysed.
In some cases, there was even a lack of a global analysis. If there were a
process evaluation and a results evaluation, there was no overall or nal
analysis for putting everything together. In a few cases, there were some nal
conclusions although when they existed, they were very short and not global
but none of the 11 reports contained any recommendations. In sum, it seemed
as though the evaluation processes were not really nished, missing opportunities
to be really useful.
10. A need to consolidate communication and dissemination throughout the
evaluation process.
Communication processes are extremely important to aid the use of evaluation.
A rst step is to have a good product that reects the evaluation process well:
complete and appropriate evaluation reports. Following that, there is a need for
dissemination of those reports, i.e. of the evaluation results.
The communication processes that were part of the experiences studied were
quite decient, and evaluation results could have been disseminated much more
effectively. A related issue is that of the accessibility of the evaluation reports.
When gathering information for the meta-evaluation, in some regions, we had
considerable problems obtaining the evaluation reports and were unable to get
the whole evaluation report. In general, the accessibility and the public nature
of evaluation reports are still not clear and there are no accessibility criteria. This
is an issue requiring earnest discussion.
11. The important role which womens agencies should play in policy evaluation.
As mentioned above, in Spain, womens agencies have been the main leaders
and promoters of gender-equality policies. Although the key component of these
policies is gender mainstreaming (i.e. they are intended to involve other govern-
ment actors), the agencies are still the main actors formulating, promoting, co-
ordinating and monitoring the policies. Taking this into account, policy
evaluation should be a crucial issue among agency functions. The evaluation
function is precisely the one which might help to dene the specic role of
womens agencies in gender-equality policies that are conceived with a main-
streaming strategy. Moreover, the fact that the nal aim of these policies is to
achieve a gender perspective in other public policies could place at risk the
existence of specic womens agencies. Therefore, playing a leading evaluation
role could aid their denition. In addition, the evaluation function is an especially
important one for these policies because of their innovative nature.
However, and with the exception of the Basque womans institute, the
Evaluation 9(4)
398
02 Bustelo (ds) 19/11/03 11:34 am Page 398
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
evaluation function is still a residual one. In fact, the Spanish agency at the
national level (Instituto de la Mujer) has missed a good opportunity to play a
leading evaluation role among the regional agencies. That could have been a
sound chance to recover a leading position, which was lost with the second- and
third-generation plans. In the third generation of regional plans, inuence is
much more international (the 4th World Conference on Women in Beijing in
1995 and the 4th Action Programme on Equal Opportunities between Men and
Women [19962000] of the European Union) than national, which was obvious
in the rst regional plans (Bustelo, 1999a: 377).
Conclusions: Ideas about how Gender Mainstreaming Can
Be Evaluated
Following the discussion of the lessons learnt by the meta-evaluation, the criteria
that should be used for evaluating gender-equality policies will now be described
in more detail. Generally speaking, and in the long term, one should try to assess
whether there is an evolution of womens social status and gender relationships.
However, as already seen, it is very difcult to assess the net effects of such a
broad and general policy, i.e. the quantity of that change which is due to the
policy and not to other factors. In the assessment of public actions, it is neces-
sary not only to evaluate the effects of the action, but also the way in which that
public action is conceived and implemented to deal with a particular problem.
Specically, intermediate aspects (design, implementation, quality of outputs)
should be evaluated before attempting to assess the nal effects. Currently, the
evaluation of those intermediate aspects is sometimes more useful than that of
the nal effects for policy, especially for improving purposes.
Moreover, when the aim is so abstract (such as to modify gender relationships
and the status of women and, hence, of men), it is necessary to establish inter-
mediate aims and objectives that normally can be dened by means of the
strategies designed to deal with the problem. And here, one should take into
account the strategy of gender mainstreaming. If the main strategy of gender-
equality policies were gender mainstreaming, one would probably have to seek
gender perspective as the searched effect in other public policies (that is, whether
public policies not the gender-equality policy are formulated, executed and
evaluated with gender perspective), in addition to evaluating the gender policy
itself. In this sense, there have been some interesting experiences, such as the
Gender Impact Assessment tool, developed in the Netherlands, Belgium and
New Zealand (Verloo and Roggeband, 1996; Meier, 1998; Verloo, 1998).
Although this instrument was conceived more as a tool for an ex ante rather than
a ex post facto evaluation of the effects of a specic gender-equality policy, in
our opinion, it might be a useful source to evaluate gender mainstreaming efforts.
However, this is still not sufcient. It is crucial to recognize the importance of
evaluating the strategies employed to full the proposed aims, and not only of
measuring the achievement of those aims. In this sense, the chief aim to achieve
(and, thus, the effects to pursue) is gender perspective in all public policies, with
gender mainstreaming being the strategy to deliver this. Therefore, in addition
Bustelo: Evaluation of Gender Mainstreaming
399
02 Bustelo (ds) 19/11/03 11:34 am Page 399
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
to their results, it is necessary to evaluate the design of those gender main-
streaming strategies and the process of their implementation.
However, by means of the meta-evaluation, the author concludes that one of
the factors identied as crucial by the femocrats was what can be called the co-
ordination factor. This factor was considered essential for:
1. persuading others to plan and implement gender actions and to assume a
gender perspective; and
2. obtaining valid and reliable information about those actions and the plan
management or evaluation.
None of these aspects was addressed by any of the evaluations studied. In the
authors opinion, this co-ordination factor is directly related to the mainstream-
ing strategy. How do femocrats persuade other government units to assume a
truly active role in gender-equality plans? Which strategies and instruments work
better, and in what circumstances? Which are the best co-ordination structures
for the plan and its evaluation and at what level political or technical?
9
Why
are some government units more sensitive about gender issues than others? What
is needed to awaken that sensitivity, or which factors inuence it? To what extent
is the personal factor counted on? If this dimension is not evaluated, we will
probably fail to understand much about that horizontal action and, hence, about
the gender mainstreaming strategy. We must be aware of the fact that evaluation
of gender mainstreaming strategies is complex, because the rst step is to dene
and typify the gender mainstreaming strategy in different contexts, moments and
policies. In relation to this, the Group of Specialists on Mainstreaming in the
Council of Europe elaborated an excellent document dening the term, promot-
ing ideas and demonstrating good practices to deal with it. If they are imple-
mented, evaluating those practices might be especially important due to their
innovative nature. It is not sufcient to implement them; they must be evaluated
in order to determine their effectiveness.
Moreover, any form of gender mainstreaming poses particular challenges for
evaluation practice; and as we have seen, current evaluation practice does not
address those challenges completely. In order to facilitate progress in this area,
there is a strong need for new tools and methodological approaches, which also
take seriously into account the lessons learnt from the analysis of the practice of
evaluation.
A new evaluation perspective implies that both the gender mainstreaming
strategy itself should be evaluated, in addition to the evaluation of its effects.
Alternatively, rather than exclusively emphasizing the evaluation of the
implementation of proposed actions, evaluation of gender-equality policies should
stress the assessment of their global design as policy instruments, their implemen-
tation processes, government co-ordination and organizational performance.
We have presented above a case for meta-evaluation in gender-equality policy
evaluation. However, meta-evaluation has potential value in other policy areas.
Important conclusions and guidance to improve the evaluation function in a
specic eld can be drawn from the systematic analysis and assessment of evalu-
ation practices, their strengths and weaknesses and their nal utility.
Evaluation 9(4)
400
02 Bustelo (ds) 19/11/03 11:34 am Page 400
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
Notes
1. Earlier versions of this article were presented at the 4th Conference of the European
Evaluation Society in Lausanne (October 2000) and at the European Consortium for
Political Research (ECPR) 2001 General Conference, University of Kent in the UK
(September 2001). This nal version of the article was made possible in part by a Del
Amo-Complutense Fellowship in the University of California at San Diego during the
summer 2002.
2. Gender mainstreaming is dened as the (re)organization, improvement, develop-
ment and evaluation of policy processes, so that a gender-equality perspective is incor-
porated in all policies at all levels and at all stages, by the actors normally involved in
policymaking (Council of Europe, 1998). This concept is based on the idea that no
public policy is gender neutral, so that the different effects any public action may have
on women and men must be taken into account.
3. The author is not able to nd any published reference of a similar study. The term
meta-evaluation has been sometimes used to name synthesis of evaluations, that is,
studies that focus on the evaluation results, instead of on the evaluation processes.
Some other references have appeared after this meta-evaluation was conducted.
4. The Spanish Constitution is from 1978 and the 17 regional governments and parlia-
ments were created between that year and 1983. Thus, the creation of these womens
agencies was, in a sense, a parallel process to the creation of the political and admin-
istrative regional organization.
5. Programme theory refers to the mechanisms that mediate between the distribution
and delivery of a programme and its results (the set of beliefs underlying action or the
set of hypotheses on which the programme is built), and implementation theory deals
with what is required for translating the objectives into programme operations.
Whereas the focus of programme theory is on peoples responses to programme activi-
ties (the operational mechanism is not the programme activities per se but the
responses those activities generate), implementation theory emphasizes the services
or activities that are implemented and the way they are offered (Weiss, 1998: 57).
6. There is still a low evaluation culture in Spain. Possible causes for this are, at least, the
following (Ballart, 1993: 2212): evaluation emerged in Spain in a time of economic
crisis and resource scarcity; the parliamentary system at the national and the regional
level is based on strong party discipline which tends to limit its controlling function;
the National Audit Ofce, and the similar ofces at the regional level, limit their role
to controlling exclusively public nances; the number of professionals specialized in
evaluation is very scarce; and, nally, information systems within the public adminis-
tration, although they have improved a lot in recent years, are still inadequate for evalu-
ation purposes.
7. In Andaluca the cost of the external evaluation study was 55,980,000 pesetas, the
general plan expenses were 80,401,000,000 pesetas, and the expenses of the Instituto
Andaluz de la Mujer (IAM) during the three-year plan period (1995, 1996 and 1997)
were 5,792,819,000 pesetas. In addition to that, we estimate that the indirect expenses
of this evaluation experience were not very high because the external team was quite
autonomous.
8. The evaluation of the II National Plan consisted of three separate studies: quantita-
tive evaluation, qualitative evaluation and implementation evaluation.
9. There is no agreement among femocrats about whether the co-ordination structures
should be political or technical or a mixture of both. Being political for example, as
in Andaluca, the committee for monitoring the plan must be made up of viceconsejeros
Bustelo: Evaluation of Gender Mainstreaming
401
02 Bustelo (ds) 19/11/03 11:34 am Page 401
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
(the equivalent to a vice minister) is the only way to have real horizontal authority,
but being technical is the only way to be operative. Some of the femocrats think that it
is very difcult to be effective if there are no motivated technicians who are sensitive to
gender issues.
References
Ballart, X. (1993) Evaluacin de polticas. Marco conceptual y organizacin institucional,
Revista de Estudios Polticos 80(3): 199224.
Bustelo, M. (1999a) Polticas Pblicas de Igualdad de Gnero en Espaa: evolucin y
evaluacin in M. Ortega, C. Snchez and C. Valiente (eds) Gnero y ciudadana. Revi-
siones desde el mbito privado, pp. 36789. Madrid: Ediciones de la Universidad
Autnoma de Madrid.
Bustelo, M. (1999b) Diferencias entre evaluacin e investigacin: una distincin nece-
saria para la identidad de la evaluacin de programas, Revista Espaola de Desarrollo
y Cooperacin 4(Spring-Summer): 1529.
Bustelo, M. (2001) La evaluacin de las polticas pblicas de igualdad de gnero de los
gobiernos central y autonmicos en Espaa: 19951999, Doctoral dissertation. Madrid:
Universidad Complutense de Madrid.
Chelimsky, E. (1998) The Role of Experience in Formulating Theories of Evaluation
Practice, American Journal of Evaluation 19(1): 3555.
Council of Europe (1998) Gender Mainstreaming. Conceptual Framework, Methodology
and Presentation of Good Practice. Final Report of Activities of the Group of Specialists
on Mainstreaming (EG-S-MS). Strasbourg: Council of Europe Publishing.
Gardiner, F., ed. (1997) Sex Equality Policy in Western Europe. London: Routledge.
Granados, E. (1999) El Instituto Andaluz de la Mujer: La creacin de un rgano
autonmico para la igualdad entre los gneros, in M. Ortega, C. Snchez and C.
Valiente (eds) Gnero y ciudadana. Revisiones desde el mbito privado, pp. 391405.
Madrid: Ediciones de la Universidad Autnoma de Madrid.
Meehan, E. and S. Sevenhuijsen, eds (1991) Equality Politics and Gender. Newbury Park,
CA: Sage.
Meier, P. (1998) Policy Innovation in Terms of Mainstreaming? The Development of
Gender Impact Assessment in Flanders, Belgium, paper presented in the ECPR 26th
Joint Sessions of Workshops, Warwick, UK.
Monnier, E. (1995) Evaluacin de la accin de los poderes pblicos. Madrid: Instituto de
Estudios Fiscales, Ministerio de Economa y Hacienda.
Patton, M. Q. (1980) Practical Evaluation. Newbury Park, CA: Sage.
Patton, M. Q. (1997) Utilization-Focused Evaluation, 3rd edn. Thousand Oaks, CA: Sage.
Scriven, M. (1995) The Logic of Evaluation and Evaluation Practice, in D. M. Fournier
(ed.) Reasoning in Evaluation: Inferential Links and Leaps. San Francisco, CA: Jossey-
Bass.
Stetson, D. M. and A. Mazur, eds (1995) Comparative State Feminism. Thousand Oaks,
CA: Sage.
Valiente, C. (1995) The Power of Persuasion. The Instituto de la Mujer in Spain, in D.
M. Stetson and A. Mazur (eds) Comparative State Feminism, pp. 22136. Thousand
Oaks, CA: Sage.
Valiente, C. (1996) El feminismo institucional en Espaa: El Instituto de la Mujer,
19831994, Revista Internacional de Sociologa 13(1): 163204.
Evaluation 9(4)
402
02 Bustelo (ds) 19/11/03 11:34 am Page 402
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from
Valiente, C. (1997) State Feminism and Gender Equality Policies. The Case of Spain
(198395), in F. Gardiner (ed.) Sex Equality Policy in Western Europe, pp.12741.
London: Routledge.
Verloo, M. (1998) Policy Design and Frame Alignment: The Case of Mainstreaming,
Paper presented in the ECPR 26th Joint Sessions of Workshops, Warwick, UK.
Verloo, M. and C. Roggeband (1996) Gender Impact Assessment: The Development of
a New Instrument in the Netherlands, Impact Assessment 14(1): 320.
Weiss, C. H. (1987) Where Politics and Evaluation Research Meet, in D. J. Palumbo (ed.)
The Politics of Program Evaluation, pp. 4769. Newbury Park, CA: Sage.
Weiss, C. H. (1998) Evaluation. Methods for Studying Programs and Policies, 2nd edn.
Upper Saddle River, NJ: Prentice-Hall.
MARA BUSTELO is associate professor of political science and public
administration and Head of the Evaluation Unit at the Centro de Gestin
(Complutense University). She has worked for 15 years as an evaluation
consultant and her main interests are public policies evaluation, evaluation
training, and the evaluation of gender equality and drug prevention policies.
Please address correspondence to: Departamento de Ciencia Poltica y de la
Administracin II, Facultad de Ciencias Polticas y Sociologa, Universidad
Complutense, Campus de Somosaguas s/n. 28023 Madrid, Spain.
[email: mbustelo@cps.ucm.es]
Bustelo: Evaluation of Gender Mainstreaming
403
02 Bustelo (ds) 19/11/03 11:34 am Page 403
2003 SAGE Publications. All rights reserved. Not for commercial use or unauthorized distribution.
at INDIANA UNIV on July 9, 2008 http://evi.sagepub.com Downloaded from