Program SPACES FOR ENGAGEMENT: USING KNOWLEDGE TO IMPROVE PUBLIC DECISIONS GDNet / CIPPEC Study on monitoring and evaluation

n of the research impact in the public policy of Policy Research Institutes (PRIs) in the region Martin Lardone and Marcos Roggero I. INTRODUCTION This paper aims to investigate one of the key dimensions in the relationship between science and policy, between research and decision-making, i.e. the existence of mechanisms to monitor and evaluate the influence that PRIs exert on public policy, and the use that PRIs make of, and the way in which they adapt, such instruments and mechanisms. The problem of identifying the link of effective impact between the research generated by PRIs and public policy appears to be particularly complex, despite which, evaluation of the effectiveness of impact is a key input for the work of PRIs and for perfecting their strategies and agendas. Follow-up and evaluation are fundamental if an organisations actions are to be constantly perfected. Evaluating whether public policy incorporates the knowledge and evidence generated by PRIs, as well as monitoring how public policy incorporates them, are crucial information inputs in improving the efficacy of future impact actions. Our intention is to look into the methods that PRIs use to make this kind of evaluation. This line of research means that particularly interesting phenomena can be addressed, such as: the different kinds of impact that organisations may have on decision-makers; the way in which different organisations, including PRIs, relate to and connect with the State and public decision-makers; and how the type of impact varies according to the stage of the public policy process on which there is influence, to mention just a few. In short, work is done on variables which help shape the relationship between PRIs and the State. In short, the general aim of our study is to analyse the current state of PRI capacities to monitor and evaluate (M&E) their actions of impact on public policy, and also to identify the M&E impact mechanisms currently available. More specifically, we hope to identify those factors which facilitate or obstruct the capacity of PRIs to monitor and evaluate their influence on public policy. To do so, we pose a series of questions that we hope to answer in this paper: How do PRIs monitor and evaluate the impact of the research they carry out on public policy? How do they measure the influence of the knowledge and evidence they produce in shaping and implementing public policy? Which relevant methodologies does this type of organisation have at its disposal to
1

follow up and evaluate impact on public policy? What are the current capacities of the PRIs involved in the field? These are the most important questions to be discussed in this study. The study is organised into five parts. After the introduction, the second part presents the analytical framework formulated to address the phenomenon of M&E impact capacities. It defines what is understood by PRI in our region, establishes the internal or external factors that motivate or discourage M&E, and warns about the complexity involved in measuring influence. This part ends by specifying the mechanisms currently available to PRIs in the region to M&E impact. Having posed the framework of analysis and the mechanisms available, the third part concentrates exclusively on an analysis of the PRIs included in the study sample. The fourth part offers the main conclusions of the paper and, lastly, the fifth gives a series of recommendations for PRIs in the region to incorporate impact M&E mechanisms. II. ANALYTICAL FRAMEWORK AND IMPACT M&E MECHANISMS AVAILABLE 2.1 Policy Research Institutes (PRIs) PRIs in the political-administrative reality of the region As shown above, PRIs (or think tanks) have experienced exponential growth and development over recent years in the region, thanks to the main political and economic transformations (Braun et al., 2006; Camou, 2005). So our goal is to make clear what exactly we are referring to when we study one of the parts of the impact relationship, i.e. PRIs, since it is the monitoring and evaluation (M&E) capacities of PRIs that we consider to be a dependent variable in this paper. First of all, it is important to bear in mind that PRIs are part of a type of organisation that includes different variants, and that they have frequently been analysed under the common denominator of think tanks. In this regard, the literature on studies of think tanks stresses the polysemy of the concept and the heterogeneous nature of the organisations to be surveyed. There are even some who make a distinction between think tanks and PRIs in the belief that they are in fact different types of organisation (Dunn, 1996). For our paper we have taken up the idea put forward by Acua (2009), who in turn bases himself on the work of other authors (Abelson 2006; Ua, Lupica & Strazza 2009). He maintains that PRIs/think tanks (which he believes to be synonymous) tend to be thought of as those non-profit research institutes, civil society and private organisations that produce information and knowledge for the basic purpose of influencing some aspect of the public policy process (Acua, 2009: 5). In this regard, the proposal rejects the narrow visions of the
2

limits to PRIs/think tanks, which ignore other possible types, such as academic centres, private research centres, political foundations and advocacy groups. They subscribe to an Anglo-American tradition on think tanks that has little to do with the dynamic that characterises them in other parts of the world (Acua, 2009)1. In short, Acua (2009) rejects the excessively classificatory botanical approach to defining PRIs, and proposes a conceptualisation which stresses the functions and properties of these organisations as an analytical starting point. The proposal is to consider as PRIs/think tanks, 1) collective, 2) formally institutionalised, 3) non-profit actors, 4) whose dominant, formalised or de facto organisational function is to influence public policy, 5) an influence exerted through the production and transmission of knowledge, 6) a transmission whose focus can include various stakeholders who might weigh directly or indirectly on the shaping and implementation of policy (whether they be government, para-government, social players or citizens in general)2 (Acua, 2009: 8). Now, in our region the relative importance of think tanks within the policy drafting process has grown very rapidly (Braun et al., 2006), although they are still far from the level that these organisations play in countries in Europe and the United States. However, in recent decades, the appearance of these organisations has modified the form of linkage in our region between specialised knowledge and decision-making, and the general link between different fields of public policy production (Camou, 2005). The structural transformations that occurred towards the end of the seventies underpinned this growth, while the demands imposed by these changes made them more complex and difficult to manage. This context has revealed weak growth in State capacities on key issues, especially policy design and implementation. Changes in the traditional functions of the State, and consequently its new form of relating with society, broadened the gap between the demands of certain civil society groups and the (policy) decisions adopted. This void partially revealed an amalgam of organisations geared to investigating public problems and actively intervening in the agenda of decision-makers at different levels of government. This situation represented, in turn, the possibility of impacting on the direction and quality of the policies adopted (Braun et al, 2000). In this context, think tanks have progressively become a pillar for many policy formulation processes due to the weak political

For further information on this criticism of the broad vision see the paper by Garc (2006)

This definition also resembles that of other authors, such as Thompson (1994) who stresses the

organisations, or think tanks, seeking to influence policy, the processes that give birth to them and, more recently, their growing role in implementation actions. Its key characteristic is the production of quality research, with scientific precision, but without detaching them from their potential influence on the orientations of the policies of the decision-makers (Thompson, 1994)

and administrative capacity of Latin American states, and to their low capacity of response to citizens needs (Zuvanic & Iacoviello, 2005). Some think tanks work on the basis of specific interests while others have developed in a broader framework of problems. The origin of many of these centres is related with the interest of private sector stakeholders in promoting certain issues, highlighting the lack of response by the State to important social demands, and to the multiplication and growing importance of academic centres and universities. Different political sectors have developed spaces in which to study public policy and problems and which serve to support their proposals and projects. The platitude about this broad range is simply an attempt to bridge this gap between the demands of numerous civil society groups and government decision-making (Braun et al, 2002). These bridges have often remedied the backlog in administrative processes and the inefficacy of state structures. Nevertheless, despite this heterogeneity, these organisations can generally be said to have increased their role in public policy design, just as many of them actively intervene in the implementation of innovative management processes within public administration, proposing and technically assisting different types of reforms in areas as diverse as the administration of public resources, social security, the organisation of human resources, the provision of health services, the provision of public security systems, and justice, as well as the economic plans of different governments (Ua et al, 2004).

2.2. Organisational analysis of PRIs The interest here is to discuss some of the elements that we put forward as an organisational approach. We are particularly interested in knowing which features of PRIs (conceived as organisations) and of their environment can be seen as possibilities or restrictions on their capacities for monitoring and evaluating impact on public policy. Likewise, we shall dedicate a special section to the mechanisms PRIs use (or could use) to monitor and evaluate their impact. PRIs and measuring influence A particular feature of PRIs is that they deal not only with problems of research but also with public and social problems, aiming to produce solutions and policy proposals. So, within PRIs, research dynamics often coexist with dynamics of practicality, social utility, practical application and impact, and the knowledge and evidence produced tend to impinge on government and public policy. It is not merely a question of making progress in the academic understanding of an issue or in scientific knowledge, but also of generating concrete and plausible solutions for societys public problems.
4

PRIs analyse policies, articulating traditional research and a methodological toolkit with practical arguments and recommendations, and data with value judgements, offering the set of inputs for impact on public policy, whether by defining or redefining public problems, generating solutions, analysing implementation or evaluating policies. This should not be seen as renouncing independence and autonomy, or sacrificing the objectivity of the studies in pursuit of affinity with the government and guaranteed entry to government decision-making areas. On the contrary, reliability, neutrality, independence and quality, make for the reputation of the PRIs and are a decisive factor in their capacity of influence (Braun et al; 2006). Now, this goal, or the practical orientation of the knowledge and evidence produced by PRIs is what has led to much debate on the impact on public policy of the products and actions of PRIs, and where their impact is most effective. In itself, impact or influence are very complex processes for social sciences to trace or reconstruct, which has led to heated debate on the matter, focused in large measure on the forms of measuring that phenomenon (Stone, 1996; Krastev, 2000; Abelson, 2002)3. One of the principal drawbacks of this point of view is how to attribute a certain government measure or decision to the influence of a PRI4. Now, to consider influence in this way, it is necessary to suppose that there is an agenda for PRI A and an agenda for government X or Y, to analyse to what extent the policies adopted by X and Y resemble the agenda proposed by A. An exercise of this nature implies adopting a series of assumptions that are not empirically sustainable or provable (Lardone, n/d)5. In this regard, Abelson (2002) maintains that the main methodological problem in measuring influence is that both policy actors and academics have differing perceptions of what makes influence and how it can be measured. Furthermore, while the policy-making community becomes more complex and populated, tracing the origin and history of a certain policy becomes more difficult and actually rather pointless. This author opts to analyse at which stage of the policy cycle think tanks are mostly involved, and thus explain how they exert influence and the resources they use.

The concept of influence is central to political science, in that it refers to the nucleus of the power relationship although it is difficult to specify and measure. One of the best-known definitions of power is that by Dahl (1957) in which A influences B inasmuch as it causes B to do something it would not otherwise do. In this model, a PRI exerts influence on public policy when it causes a government to adopt a measure it would not have taken if it had not been for the action of the PRI. Douglas (1979), in contrast, defines influence as a causal relationship between the preferences of an actor concerning an outcome and the outcome itself. Douglas criticises Dahls definition because it fails to consider the rule of anticipated reactions. 4 There is much discussion on the problem of attribution as regards analysis of the phenomenon of influence. See, for example, Carden (2003) Bachrach P. & M. Baratz (1962), Lukes (1974). 5 As Braun and his colleagues (n/d) point out, this idea is linked to the fact that PPIs are like hidden participants in the policy process, while decision-making by political parties, legislatures and executives in formal settings are a more transparent process.
3

In this aspect, the paper by Braun et al. (n/d) is particularly useful. The authors believe that there is rarely correspondence between a book or a study and a particular climate of policy change or public opinion. Numerous forces intervene and can alter the impact of the research, as well as any cause-effect relationship there may be between PRIs and government decision-making. However, they also point out, that many PRIs do not restrict their goals to driving policy change, but that they also aim to improve the quality of the dominant paradigms in public debate. As a result, the authors identify other impact mechanisms based on the work of Court and Pollard (2005: 6): Influence on the agenda-setting process and on public debate mechanisms. Influence on policy formulation, through the presentation of evidence and proposals based on research results. Influence on policy implementation through the presentation of evidence that is critical for improvement in the program. Influence on the policy monitoring and evaluation process.

So, according to Abelson (2002), it is important to develop a study that will reveal the mechanisms of influence, and of the functioning of influence on the basis of case analysis, since although useful indicators to evaluate the visibility of think tanks exist, they provide no insight into how this production of applied knowledge impacts on a policy or on a decision-maker. These difficulties have at times been addressed by solutions which are of a more conceptual nature. Such is the case of the exercise of separating the concept of policy influence from that of policy impact. At the impossibility of measuring and evaluating the latter, policy influence would appear to be less profound or of less scope, focusing more on intermediate influences (Adamo; 2002) of evidence and knowledge, in which impact is measured not only by different stakeholders involved in policy but also by interpretations and meanings6. On this point, and bearing in mind the complexity of the impact and/or influence process, the aim of our paper is to identify the various mechanisms and instruments that exist for impact M&E, and to analyse how and why they have (or have not) been incorporated by a series of selected PRIs in the region. Internal and external factors in the organisation that encourage or obstruct M&E

This distinction is seen to be analogous to that made between the evaluations of outcomes and impacts, as shall be seen at a later stage.
6

Monitoring and evaluation is doubtless the most absent phase in any programmatic action (be they projects, programs or plans) of organisations, regardless of whether they are public, private or of civil society, and PRIs do not appear to be an exception to the trend. This absence is probably due to the fact that the M&E phase is not essential for the formulation and implementation of programmatic action. Devising a project (formulating it) and putting it into practice (implementing it) are unavoidable moments, albeit minimal ones, for any action that prides itself on having been programmed. However, monitoring how it is being done or evaluating the outcomes and impacts are moments that can be omitted or abandoned, even when the programmatic action continues over time. This leads us to wonder, for example, about the origin of motivation or the demand to do M&E, if doing M&E is an initiative inherent to PRIs or if, on the contrary, it is a requirement of the counterparts. In this paper, the framework of analysis is formulated on the basis of different contributions. The dimensions of analysis given below are constructed on the basis of the contributions thrown up by the approaches of Ducot (2006) and Braun et al. (2006) in analysing the impact capacities of PRIs in Latin America, of the World Banks Operations Evaluation Department in strengthening the development of evaluation capacities, and of SADCI methodology that focuses on institutional capacities. Likewise, the set of dimensions considered can be divided into two big families: one has to do with the environment, external demand for knowledge and evidence for PRIs and, therefore, exogenous variables; the other, on the supply side, with the characteristics of PRIs internal capacities for impact M&E and, therefore, with endogenous variables. Each of the dimensions is discussed in brief below. A) Demand/exogenous variables/environment These are the events which have to do with the PRI environment, the context in which they exist. From the institutional framework, passing through the demand for research, to the web of organisations that also participate in the projects, they contribute to exogenous factors, external pressures on PRIs which could be associated to the capacity to M&E impact, or even influence it. I. Rules of the game (institutional framework in which PRIs operate)

This part refers to the social, political and economic contextualisation of the countries where the PRIs studied reside. Attention will be paid to the particularities of the institutional contexts in which PRIs operate, which have to do with questions of regulations, openness to political particpation, degree of government capacity, or those that arise out of the cases. Institutional stability,
7

professionalisation of the public sector and the permanence of functionaries in their posts for an excessive length of time are some of the factors that we imagine may affect the possibility to M&E the impact of PRIs on policy. II. Requirement to M&E research impact

This is an external demand that especially involves the counterparts of the PRIs, who insist that the latter reveal the outcomes or impacts of their projects in such a way as to account for the funds used. The nature of such a requirement is that it comes from outside the organisation and is an external constraint that PRIs need to deal with. As we shall see below, on occasions the use of M&E mechanisms is motivated by requirements made by the organisations funding the projects. These organisations include donors, enterprises, the State, international philanthropic foundations, international cooperation agencies and multilateral bodies (Ducot; 2006). Some of these organisations, especially the latter two, have made great efforts to construct approaches, analytical frameworks or models for project formulation and implementation, placing special emphasis on their M&E (see Logical Framework of the World Bank, RAPID method and the ODIs Outcome Mapping Model, Strategic Evaluation of IDRC-Supported Research, among others). These tools are supplied to (sometimes demanded by) PRIs for the preparation of projects, at least those PRIs that wish to receive funding from the multilateral agency or body. III. Web of organisations involved in PRI projects

Complementary organisations are another factor to bear in mind, i.e. knowing whether one of the other organisations that also participate in PRI projects is carrying out impact M&E or not, how they do it and whether they exert some kind of pressure on PRIs to do the same. This dimension has little to do with the socialisation and incorporation of impact M&E experiences in similar organisations. B) Supply/endogenous variables/internal-organisational capacities Unlike the previous dimensions related to exogenous variables and the environment of the PRIs, several questions arise concerning internal capacities. These are endogenous variables, which characterise PRIs and largely define their capacity for impact M&E. What is of interest here is a certain insight into what extent traditional institutional capacities may be associated, in the case of PRIs, to their impact M&E capacities for their public policy projects. In addition, it gives rise to questions concerning the specific resources of PRIs to influence policy, i.e. research and knowledge, and their form of managing it (looking both inwards and outwards).
8

I.

Internal organisation and distribution of functions (structure) / Organisational government

This dimension focuses on the organisational structure of PRIs, particularly how they are organised, how many areas they have, and to what extent these characteristics can be associated to M&E impact capacities. In this regard, it is of interest to ask if PRIs have a specific area responsible for M&E impact. But this dimension also focuses on other questions. In this case, the requirement comes from the organisation itself and consists of an initiative by the PRI. Even here we can speak of certain accountability, but between internal units within the organisation. For instance, it may be that the source is the director or the board of directors of the PRIs, which require M&E of the projects under way in the organisation they direct. The same thing could occur in the area responsible for project funding. Horizontal relations of accountability can also be found, such as those that might emerge in the projects competing for funding, but also as a result of institutional recognition within and outside the organisation. As an example, it is worth mentioning the annual planning and internal management processes, especially the annual work plans mentioned as one of the determinant internal capacities of impact in policy. In this respect, Ducot (2006) says that strategic plans function as an instance in which the goals to be reached by each area of the organisation are set, and should each contain specific objectives posed in terms of expected outcomes, in such a way as to serve as performance measuring instruments. The results from the areas are then linked to the PRI impacts. The directors of the organisation responsible for their coordination should supervise and evaluate the projects to ensure that they comply with the work plan, meet the stipulated objectives, and have the desired external impact while complying with the annual plans. II. Financial and physical capacities

These include the factors which relate to the size of the budget, the period of time covered by the funding, the origin of the funds (e.g. from the public sector or from international financing) and the amount. It is also of interest to know whether the PRIs have their own office, and their material needs to allow them to function optimally. The aim here is to establish whether there is a relationship between budgetary and material shortcomings, on the one hand, and absence or presence of impact M&E, on the other. III. Personnel policies and systems of rewards

This dimension concerns itself with the number of persons who work in the PRIs studied, either permanently or on specific projects. Although it
9

complements the former dimension in determining questions of size of the PRI, it also looks at the existence of incentive systems for the staff to M&E impact. IV. Individual qualifications and management capacities of those who work in agencies with responsibility in the area/ Human resources

What is the human resources profile? Which areas are they from? What is their origin/professional background? These are just some of the questions included in this dimension. To be specific, it looks at the possibility of attributing M&E impact practices to a certain profession or human resources profile, especially to a particular type of background of the leading members. On the other hand, this dimension looks at the methodologies used to M&E impact. It also investigates the methodology used to formulate and implement projects in general and whether M&E was required or chosen, or any other reason. It also looks at indicators for impact M&E and whether they are constructed or defined. In short, it is interested in the technical capacity of the human resources to do impact M&E and the use of specific tools to make it effective. V. M&E and research management

This section looks in greater depth at impact M&E as a feedback mechanism of the PRI, as a form of self-evaluation to improve performance and the the PRIs impact. Unlike the accountability dynamic in the relationship of PRIs with their counterparts (in the demand for research dimension), this dimension looks more closely at the information system, or feedback mechanisms, used by PRIs to perfect both their present and future interventions. On the other hand, it investigates certain suppositions such as whether it is more feasible to carry out impact M&E in projects dealing specific issues, i.e. whether impact M&E varies according to the topics (subjects) on which the projects act. In that same line of reasoning, it is interesting to establish whether impact M&E is carried out regardless of the stages of the policy process or, on the contrary, if it is carried out more frequently in certain stages in particular. VI. Institutional communication

The interest here involves looking into the dissemination of he PRIs impact and to what extent it is fitting in terms of the relationship with its counterparts, especially decision-makers. It may be that decision-makers prefer discretion in connection with the discussions and all type of interactions experienced with PRIs. It is also likely that many PRIs prefer to highlight their independence from decision-makers, in their search for greater objectivity in research or of greater future possibilities when there is a different party in government. In any
10

case, it is important to look in greater depth at the way in which PRIs manage their institutional communication in relation to impact.

2.3 PRIs and the evaluation of their actions What is to be measured: the actions of impact by PRIs. Before discussing the relevent mechanisms for impact M&E of PRIs, an explanation is required of the interventions by PRIs with which they seek to influence policy, and the impact to be evaluated. What we mean here is that the relevance of M&E mechanisms or their construction will always be significantly conditioned by the type of intervention to be measured and evaluated. Therefore, an analysis and characterisation of the actions or interventions, of the activities and products undertaken by PRIs, will be of great use in clarifying the search for M&E mechanisms with real possibilities of being assimilated. According to the paper by Adamo (2002), which evaluates the projects financed by IDRC that sought to influence public policy, among the types of influence on policy mention should be made of the generation of research and analysis of relevant policy; the increase in research capacities, organisations, civil society and policymakers; and the successful dissemination of research products. The list continues with types of influence with greatest scope in public policy, such as the use of research results as inputs for policy; the strengthening of dialogue in connection with policy; changes in attitudes and approaches of policy/decision-makers and other stakeholders; researchers acting as government advisors or occupying important government positions and playing an active role in policy design at the different levels of government; contribution to the development of policy alternatives and proposals. Likewise, Adamo (2002) provides the types of influence disaggregated into activities and products, as shown in the following diagram.

Activities/approaches to influence policy

Produce policy relevant research and/or analysis Participation of government agencies, policy/decision-makers at various levels, extension agents etc. in project Capacity building: Workshops, seminars, conferences, roundtables Training (e.g. short courses) Small / competitive grants Peer Review Mentoring Networking Enablement

11

Learning by doing Dissemination of information/knowledge: Publications, reports, working papers, newsletters, policy briefs Internet websites, databases Outreach / networking / awareness raising with government officials and other key stakeholders Workshops, seminars, meetings, policy roundtables Facilitate/strengthen policy dialogue: Working groups, task forces Policy roundtables, workshops Consultations / networking

Source: Adamo; 2002 Similarly, Ua, Cogliandro & Labaqui (2004), speak of activities and products when referring to interventions. In their study, the activities thought to be plausible for a PRI in Argentina are: research and specialised studies; technical advice and assistance; training; organisation of seminars and conferences; public policy management and implementation; monitoring and evaluation of public policy; promotion of topics of public interest; and consultancy work. As regards products, the authors mention books, working documents, leaflets, newsletters or bulletins, magazines, management indicators and press articles. This difference between activities and products is particularly useful because it allows differentiation between product, outcome and impact indicators. The different proposals and concrete indicators we are reviewing may be classified in this way. For example, reports, analyses, studies, publications, newsletters, are products; indicators of media coverage, television exposure and media exposure in general, are principally outcome indicators; and indicators which have more to do with impact on policies and that testify to policy modifications, innovations, incorporation of evidence produced, etc., are impact indicators. Writing on the general impact of civil society organisations, Leiras (2007) associates the actions of impact to different stages in the public policy process (understood as agenda-setting, decision-making, implementation and monitoring and evaluation). So at the public agenda-setting stage, the author links the production of relevant and reliable information with the timely dissemination of information. At the decision-making stage, he places the facilitation of intragovernmental negotiations, innovation and dissemination of examples. In implementation, the main interventions include the distribution of goods and services, reinforcing public policy impact and removal of obstacles to access to public services. Lastly, in monitoring and evaluation, once again the production of relevant and reliable information appears. Lastly, Lindquist (2001), working on the creation of a strategic approach for the research evaluation financed by IDRC, offers different types of influence of research in public policy.
12

Types of influence policy

Expanding Policy Capacities Improving the knowledge/data of certain actors Supporting recipients to develop innovative ideas Improving capabilities to communicate ideas Developing new talent for research and analysi Broadening Policy Horizons Providing opportunities for networking/learning within the jurisdiction or with colleagues elsewhere Introducing new concepts to frame debates, putting ideas on the agenda, or stimulating public debate Educating researchers and others who take up new positions with broader understanding of issues Stimulating quiet dialogue among decision-makers Affecting Policy Regimes Modification of existing programs or policies Fundamental re-design of programs or policies

Source: Lindquist; 2001 According to the literature reviewed, the different types of impact can increase in complexity even further if we consider, for instance, the levels of government over which influence is sought, or the stages in the policy process and the topics/subjects on which there is impact. This is something that needs to be reconsidered in the interview guide. Types of evaluation Monitoring and evaluation in the Latin American region have become particularly important as a result of the structural reforms implemented by multilateral credit bodies over the last two decades of the 20th century, specifically, with the predominance of results-based management in public organisations. Bureaucratic management systems have been severely criticised and the control of procedures has given way to the preeminence of outcomes as a value criterion in programmatic actions. Control has thus been made subject to outcomes, in such a way that the existence of outcome and impact evaluation indicators has become a necessary requirement. Although there is considerable literature on M&E, there is little precision about the different types. The typology that we will call upon here is that which establishes the five types of evaluation based on the natural sequence of any programmatic action, starting from resources and activities to reach products, and continue to outcomes and impacts on reality. By resource evaluation, we
13

refer to those methodologies that principally concern themselves with evaluating strategy for bringing together resources and inputs; and, by activity evaluation, to those that focus on the quality of the process followed to obtain the products. In product evaluation what matters is the value attached to the products obtained by the PRI. Outcome and impact evaluations look at the direct and immediate effects and at the indirect and mediate effects of the products, respectively. Such distinctions between the different types of evaluation are particularly useful, due to the fact that they allow us to define the depth or scope of the impact to be measured with the M&E mechanisms (currently or potentially) employed by the PRIs. In this paper, we believe that on investigating the impact of the evidence and knowledge produced by PRIs in public policy, allusions are made to the outcomes and impacts that the products of these organisations have on public policy. In other words, there is more interest in the effects of PRIs on the environment than in the production function used to generate the products.7 As already analysed in other points in this section, there is a generalised consensus within the literature on the methodological difficulties concerning the concept of impact. As inferred from the preceding paragraph, impact M&E in policy is more consistent with outcome evaluations: impact is rightly associated with the effects on the reality studied. However, there are certain obstacles which obstruct calculations of the impact of evidence and knowledge on public policy; in other words, they prevent linear determination of the impact of X on Y (Saumier; 2003). Among the typical obstacles to establishing the causal attribution of a phenomenon, the most significant in the literature is that which has to do with the circular nature of the public policy process, since it is frequently a question of indirect effects, through different actors, or of intermediate influences (Adamo; 2002). One way of solving this problem, according to the literature, is to speak of logical or reasonable linkage to the projects and actions undertaken by PRIs, rather than a linear causality. Some of the methodologies we will mention prefer to say as little as possible about impacts, understood as the most indirect medium- and long-term effects, whose control and attribution is possible with large budgets, referring only to products and outcomes (understood as the direct and immediate effects of the actions of PRIs). Another way would seem to be greater association between measuring impact and qualitative indicators. Since they frequently have to do with policy modifications, which are in all probability preceded by changes in attitudes and practices, impact evaulation, i.e., of the impact at its greatest scope and depth, is

This was the criterion that guided the selection of available mechanisms of impact M&E, as shall be seen in the following section.
7

14

possible by means of more qualitative measuring instruments, which look at each case in greater relative depth. M&E mechanisms available Mention is made below of the mechanisms identified in the literature reviewed. The vast majority of them are cooperation agencies, multilateral bodies and international foundations which deliberately strive to provide project formulation and implementation tools to facilitate the M&E of interventions. The purpose of this list is to produce a chart of mechanisms reflecting the depth, or rather, the scope of impact (product, outcome, influence) of PRIs in public policy8. Organising the available M&E mechanisms in this way makes it easier for the reader to see at a glance how deeply impact M&E is being carried out in any given PRI (if it is indeed being carried out) or which methodology should be chosen depending on the scope of the impact to be monitored and evaluated. It is worth mentioning an example that reveals the usefulness of organising the mechanisms in this way. It is not at all uncommon for PRI representatives to find the question of impact M&E solved with indicators on media follow-up. Recording the number of times that their research papers are quoted in newspapers, the occasions on which researchers were invited to television programs, the number of articles published in the press, etc., are very valuable indicators, more so when only a minority have them, but they give no idea of maximum impact, and are of no use in evaluating impact on public policy. As shown in the chart below, indicators such as these correspond more closely to outcome evaluation (they record the immediate effect), and to presence on the public agenda, but they do not reflect impact on public policy. A similar situation occurs with the evaluation of papers, articles, websites or newsletters, but in these cases, the evaluations are products; they have even less scope. On the contrary, case studies, the RAPID method or outcome mapping, to mention just a few, strive to evaluate impact. Different types of M&E mechanisms are mentioned below. Some refer to the evaluation of products, others of outcomes and others impact. The mechanisms referring to resource evaluation and activity evaluation have been discarded, due to the proximity (mentioned above) of the outcomes and impact. For a brief description of the mechanisms see Annex 1.

Available M&E mechanisms

The strategy used by Ingie Hovland in her research paper Making a difference: M&E of policy research (2007) will be followed in organising the M&E mechanisms of policy research.

15

Mechanism/ Methodology Evaluating academic articles and research reports (quality of science criteria) Evaluating policy and briefing papers Evaluating websites Evaluating networks After Action Reviews Media exposure Impact Logs New Areas for Citation Analysis User Surveys Framework for a Strategic Evaluation of IDRCSupported Research Temporal Logic Model Most Significant Change Innovation Histories Episode Studies RAPID Outcome Assessment Outcome Mapping

Products X

Outcomes

Impacts

X X X X X X X X X

X X X X X X

2.4. Methodological framework Selection of cases for analysis The idea of the empirical analysis study is to take a sample of PRIs selected on the basis of certain analytical elements and to make a case-by-case comparison of the way in which they incorporate impact M&E practices, and the incentives and the determining factors to do so, as explained above. Due to constraints of time and resources we are unable to work on a sufficiently representative sample in statistical terms, so we shall take the most widely recognised PRIs in each country, identified in the database prepared at the time by CIPPEC9, and when necessary we shall offer the opinions of the experts consulted in each country. The idea is to take a qualitatively representative sample, so within the sample there should be sufficient variation as to the different relevant characteristics of the PRIs (funding sources, thematic orientations, size, age, nature or institutional leanings). Of course, our interest is to include PRIs that have not

www.researchandpolicy.org

16

incorporated the practices being surveyed so as to be able to identify the factors that stand in the way of their incorporation. In the original design of the sample, and bearing in mind the regional distribution of cases, one was chosen from each of the small and medium countries of South America, i.e. Bolivia, Ecuador, Paraguay, Peru, and Uruguay and Venezuela. Two cases were taken from each of the largest countries in the region: Argentina, Brazil, Colombia, Chile and Mexico. For Central America and the Caribbean, four cases were selected from the different countries, in accordance with the previously established criteria. This gives a total of 20 cases from 15 countries. So, to any variation obtained according to the criteria, we added the variability of national contexts. However, we encountered several logistical problems when conducting three interviews (one of the cases from Brazil, and the cases from Paraguay and the Dominican Republic). However, we believe that a final sample of 17 cases from 13 countries ensures the levels of representation that we originally envisaged. Survey methodology. The survey methodology was split into two stages. First of all, a study was made of the documentation produced by each PRI and available on their websites. This stage is a sort of approximation, in that the aim is to define the profile and type of activities developed by the PRI. Secondly, the central instrument used in the research consisted in conducting semi-structured telephone interviews with the persons in charge of each selected PRI (one interview per PRI). They were carried out with a semistructured interview protocol that made it possible to define the information to be collected for later comparative analysis (see Annex 3). In these interviews, attempts were made to identify whether documentation is produced by the PRIs in evaluating their actions which could be subjected to documental analysis for this investigation. Dimensions of the questionnaire The dimensions of the questionnaire are the same as those developed in the analysis framework. The semi-structured interviews were geared to collecting data. Throughout the research process it appeared that some questions (see Annex 3) could be placed in more than one dimension, and this occurred with greater frequency in the questions at the intersection between the environment of the organisation and the units responsible for addressing it (e.g., institutional communication). As a reminder, the dimensions are the following:
Demand/ exogenous variables/ 17 Supply/ endogenous variables/

environment

organisational capacities

a) Rules of the game a) Internal organisation and distribution b) Demand for impact M&E of the of functions (structure) / Organisational government. research c) Web of organisations involved in PRI b) Financial and physical capacities. c) Personnel policies and rewards projects systems. d) Individual qualifications and management capacities. e) M&E and research management. f) Institutional communication and other offerors.

III. CASE ANALYSIS There follows a comparative analysis of the cases selected in accordance with the dimensions identified, and on the basis of which the questionnaire and interviews with members of the chosen PRIs were conducted. A first observation is that standardised M&E practices are fairly infrequent among the PRIs analysed. Although many of them monitor how much impact is generated by their public policy actions, there are not many who use defined and standardised methodologies for such evaluation.

3.1. Different M&E methods used One of the most frequent forms of evaluating and monitoring impact is by studying appearance or media presence, the frequency with which research papers or the contributions of the PRI appear in the media. However, in these cases it is generally assumed that this method does not provide clear proof of impact, but only of the presence of the PRI in public debate, which does not speak of impact in itself but of fairly indirect forms. For example, the Getulio Vargas Foundation (FGV) from Brazil is said to receive a weekly press release from the media. Likewise, each of the units that make up the Foundation receives a report of how much information appeared in the media recently, which is weighted as low, medium and high. The calculation made is how much the FGV should pay to the media for the space taken up with their appearance. The coverage includes the whole country, and newspapers from all places and of all sizes. This press release is received by all units and comparisons are made between the results obtained, i.e. the comparisons are internal. But low exposure to the media is not necessarily associated to low quality of the production, and the units with low exposure are
18

not punished with a lower budget. Now, it is suggested that greater efforts should be made to appear in the media. One of the most important points to consider, therefore, is how to disseminate the research, i.e. what to do to publicise the outcomes. In other cases, such as the Fundacin Expansiva from Chile, follow-up of media presence is complemented with other M&E mechanisms. A study is made of visits to the Foundations web page to get an idea of the target audience. Secondly, it follows up presence in the media. Thirdly, there is monitoring of impact through the presence of its members in reform programs of agencies, policies, or of the State. An example of this is the presence of Fundacin Expansiva in an institutionalised consortium within the Interior Ministry, in which around eleven think tanks come together to address a Presidency reform program. At the same time, it is clear that there is no M&E system at a more micro level, or at the level of impact in policy formulation and implementation. In other cases in which media follow-up and monitoring is combined with others, such as CEDICE of Venezuela, surveys are made among specific audiences to see to what extent the contributions of this PRI are known and used. In general, these surveys are made after publication of any particular report, or a specific newsletter. Fusades from El Salvador follows up the impact of studies, reports, proposals, events, publications and other documents resulting from research and analysis from the foundations thought centre. Use is made of media, formal meetings and legislative and judicial observatories (www.observatoriolegislativo.org.sv and www.observatoriojudicial.org.sv), which are permanent tools that allow them to evaluate and monitor impact. One of the cases in which more standardised M&E methodologies are applied concerns the Consorcio de Investigacin Econmica y Social (CIES), which monitors and evaluates impact through Outcome Mapping in some of their projects. The CIES also produces other kinds of basic indicator which have to do with activities, frequency of meetings, appearance in the press, i.e. process indicators. But progress is being made in relating impact experiences, and in documenting the histories of impact. However, CIES maintains that the solution lies not only in having a special formal methodology. It stresses the need to adopt realistic methodologies which take into account the political dynamic of decision-makers. It emphasises that researchers must necessarily involve themselves in political processes which, contrary to existing prejudices, does not entail assimilation of the PRI by the governing party. In order to get round these criticisms or risks in the relationship between knowledge and policy, the CIES promotes and strengthens its links through persons rather than institutions.
19

In other cases we found that certain follow-up or impact analysis is done, but in a more informal (or casuistic) and rather unsystematic way. This is the case of Fundacin IDEA, which claims that as far as they are concerned indicators consist in seeing their recommendations reflected in documents from public offices, while recognising that it is not necessarily a systematic and structured survey. Specifically, they speak of two forms: either we see it reflected in the document itself, or they should hire us again to implement the recommended improvements. Although those responsible for this PRI recognise that it is homemade follow-up, they defend the fact that it is a question of stronger indicators than that of media follow-up, since if a recommendation appears in a public document, it is more than tangible proof that impact has been made. Other PRIs, such as Fedesarrollo (Colombia), Estado de la Nacin (Costa Rica), and CEES (Guatemala), recognise that they do follow up their impact, but without standardised and formally structured methodologies as organisational practice. The practices in these cases range from meetings of the teams of researchers to analyse the results of their papers, internal publications to PRIs disseminating the work done, and mentions in the press. In the particular case of Estado de la Nacin, an attempt is made to follow systematically the process of creating laws in the parliamentary process, to see to what extent the process includes the contributions of the reports produced by them. Different methods of M&E used: the most frequent form of M&E impact is based on appearance or media presence, on the frequency with which the research papers or contributions of the PRI appear in the media; the follow-up of presence in the media is complemented with other M&E mechanisms, such as surveys of visits to the web pages of the PRI, presence of its members in reform programs of agencies (of policies or State reform), surveys of specific audiences to see to what extent the contributions of the PRI are known or used, observatories, among others; of the more formal or standarised M&E impact methodologies, Outcome Mapping is practically the only one used, mostly thanks to Canadian funding; emphasis is placed on the importance of adopting methodologies that consider the politicial dynamic of decision-makers, and that involve researchers in political processes; to a large extent M&E of impact is made in a more informal (or casuistic) and rather unsystematic way, such as seeing their recommendations reflected in documents from public offices, or having public offices hire them again. Also through meetings of researchers to analyse the results of their papers, mentions in the press, following the process of creating

20

laws in Congress to see to what extent the contributions of the reports produced by the PRI are included, etc. 3.2. Demand/exogenous variables/environment As stated above, it is a question of events that have to do with the environment of the PRIs, with the context in which they are situated. The question is to what extent variables such as the institutional framework, the demand for research, and the web of organisations that also participate in the projects, are exogenous factors that can be associated with, or even influence, M&E of impact capacity. 3.2.1. Rules of the game (institutional framework in which PRIs operate) This part covers the social, political and economic contextualisation of the countries in which the PRIs studied reside. In general, we found no very clear links between this factor and M&E capacities, except in some very specific cases in particular institutional contexts, as we shall now discuss. Otherwise, there is little evidence that these characteristics particularly affect the M&E of impact mechanisms. In the case of CEBEM (Bolivia), political instability, especially the high turnover of high-ranking public officials (ministers), is a factor which is seen to contribute negatively to the M&E of impact. According to the sources consulted, this means that the M&E of projects is not really very feasible. Ministers leave and with them go the public officials that work with them, making the continuity of projects difficult. We can thus speculate that political instability, absence of career civil servants or professionalisation of the public administration, and other factors associated with the permanence of public officials in their positions, may be some of the exogenous variables that come into play in the absence of M&E of impact. Another case in point is that of CEDICE from Venezuela. Here there are two problems. On the one hand, this PRI gives priority to its strategy of influencing lawmakers, but the majority of initiatives in legislative policies comes from the Executive. They claim that in consequence it is very difficult for them to evaluate the true impact they exert. On the other hand, they maintain that Venezuela suffers from low freedom of the press and high Government influence over the media, with which it becomes complicated to follow up the media in order to evaluate the presence of the PRI in the public arena. Another case is that of CIEPLAN from Chile, particularly during the years of the transition and the first Concertacin government. This PRIs impact strategy was simply to provide government officials, and in fact many of its researchers went on to occupy ministerial positions (Puryear, 1994). Therefore, the need to devise mechanisms to measure impact did not apply, since it was understood
21

that this loss of men from CIEPLAN to the government was a case of direct impact. The program of Estado de la Nacin (Costa Rica) speaks of the complexity of the political system and the number of players participating in the public policy process as an additional difficulty when it comes to doing impact M&E. They maintain that impact is a long-term process based on the construction of prestige and networks of trust with the various public policy players, and that therefore, this characteristic of the relationship of the PRI members with other external actors means that impact is very difficult to measure. Institutional framework in which PRIs operate: no very clear links were found between contextual factors and M&E capacities, save in some very specific cases which reveal particular institutional contexts; it can be speculated that factors associated with the permanence of public officials in their positions, such as political instability, the absence of career civil servants or professionalisation of the public administration, could be impacting negatively in the probability of impact M&E; likewise, in contexts of great controversy concerning freedom of the press and Government influence over the media, it becomes more complicated to follow up the media to evaluate the presence of the PRI in the public arena and, therefore, impact M&E10; in contexts of transition and democratic openness the impact of PRIs may be more direct, in that it incorporates researchers into the public sector. 3.2.2. Demand to M&E research impact This is an external demand that refers especially to the counterparts of PRIs, which demand that the latter reveal the outcomes or impacts of their projects so as to account for the funds used. It is not unusual for the use of M&E mechanisms to be motivated by demands from the organisations funding projects. However, although no very clear correlation can be observed between external funding and demands for M&E, it is important to look at certain questions on this point. First of all, it is clear that the origin of funding determines the carrying out of M&E. The most obvious cases are those in which a large part of project funding comes from international organisations, or external donors such as cooperation agencies or international foundations. There seems to be less obligation to evaluate impact when PRIs are sustained by contributions from local companies, or when the largest part of their funding comes from consultancy projects with the public sector.

10

Or at least it requires an in-depth study to identify correctly the impact patterns.

22

In the case of CIES from Peru, its members recognise that their M&E practices were originally requested by their external donors, particularly in the case of Canadian cooperation. They also recognise that if the PRI manages to prove its impact capacity, its chances of achieving this type of funding obviously improve. However, they claim that it is important to institutionalise M&E in accordance with the PRIs strategies and plans so that it is not merely a formality or requirement for donors to comply with. Another very clear case of impact of international financing is that of CEBEM from Bolivia, in which the incorporation of M&E methodologies occurs as a result of these exogenous factors. The demand made by international cooperation and the provision of methodologies for it appear to be effective here, especially when CEBEM reiterates the complete absence of a culture of M&E, both in public and in non-state organisations, and when practically the only finance comes from abroad. Something similar occurs in the case of the CEDICE from Venezuela. However, some other considerations apply here. In the case of organisations that work for projects, the main incentive for PRI members is obviously to obtain funding for their projects. In this regard, the obligation to do impact M&E only applies inasmuch as it is a specific funding requirement. FGV from Brazil claims that a very important, albeit informal, criterion to evaluate the performance of the different units is not so much its impact but its capacity to obtain funding. Each unit must strive to develop its own strategies to obtain funding both through the organisation of training programs and through participation in international competitions. In Expansiva from Chile, obtaining funding is also a priority incentive for their researchers, and this is linked in some way to the strategic plan which establishes impact as a priority objective. The plan establishes the target publics on which to impact (policymakers, members of congress, political parties, entrepreneurs, academic world). However, in order to obtain project funding, Expansiva researchers maintain that production indicators (number of papers, working documents, articles in the press) are very important, more so than those of impact, and that they only measure impact when explicitly requested to do so by the source of funding. A good question here is: to what extent does the initial conditioning that leads to M&E practices being carried out allow these practices to be institutionalised beyond the duration of the funding? We found few details to reveal that such practices would be able to outlive the projects and their funding. An interesting case is that of Estado de la Nacin from Costa Rica, which began to function as a UNDP project, meaning that it had to incorporate certain evaluation
23

guidelines for its programs. This laid the bases to allow them, without UNDP support, to develop certain mechanisms to evaluate the impact of the program. Finally, a fairly common observation among the PRIs that work with projects with external funding is that, inasmuch as M&E is done only for those projects that demand it as a condition for financing, it produces certain disarticulation in the global capacity of the PRI to carry out M&E and leads to a loss of vision of the whole, since the analysis is restricted to certain specific projects. Demand to M&E research impact: it is clear that the origin of funding determines the realisation of M&E; when the funding comes from international organisations, or from external donors such as cooperation agencies, or international foundations, it is more likely that impact M&E is done with more formal or standardised methodologies (as those listed in annex 1); but this M&E is applied as part of a specific requirement for funding; concentrating on M&E alone in a specific project can cause certain disarticulation in the global capacity of the PRI to carry out M&E and leads to a loss of vision of the whole; in the PRIs whose funding comes mostly from consultancy projects with the public sector, it is less likely that they will do impact M&E. 3.2.3. Web of organisations involved in PRI projects The question at this point is to know to what extent the incorporation of M&E practices responds to an effect of dissemination or transference based on imitating or taking practices followed by other organisations with which the PRIs have contact, or with which they interact in networks or joint projects. Certainly there is very little evidence regarding this type of dissemination of practices, which can partly be seen to be a consequence of the fact that there is generally very low application of formal and standardised M&E mechanisms. The evidence we found is of organisations in which the fact that they began operating with international funding coincides with the fact that they also involve unions or associations with other previously existing organisations. Such is the case of the CIES from Peru and the Estado de la Nacin from Costa Rica, which each bring together several organisations. Besides, in these cases, the influence of external funding is always present, although complemented by relations with other local and extra regional PRIs. We also found some evidence in this regard in the case of PRIs that develop network projects, or which interact as counterparts of other PRIs from more developed countries, or with university centres. This is the case of Fusades from El Salvador, which works on joint projects with research partners such as the
24

universities of Chicago, Harvard, Salamanca, among others, which allows this PRI to incorporate the know-how and working methodologies of these centres. Web of organisations involved in PRI projects: it can be surmised that when PRIs participate with other similar ones in network projects, they interact as counterparts of other PRIs from countries where impact M&E is carried out, or they are involved in institutional communication strategies in the style of consortia. The socialisation of practices of M&E is thus more likely, even more so when these practices begin with international funding. 3.3. Supply/endogenous variables/organisational capacities On this point, as we have said, questions arise on the internal capacities of PRIs. An idea needs to be formed as to what extent traditional institutional capacities can be associated in the case of PRIs to M&E capacities of the impact of its projects on public policy. 3.3.1. Internal organisation and distribution of functions (structure) / Organisational government. This dimension focuses on the organisational structure of PRIs, specifically how they are organised, how many areas they possess, and whether there is a specific area in PRIs responsible for impact M&E. In reference to this last question, we found that the PRIs analysed do not generally have specific areas responsible for M&E, but this is solved via other channels. In the case of the PRIs that only carry out media monitoring, this is done by the areas responsible for dissemination and institutional communication. In other cases it is an outsourced function or one contracted from a specialised company in the field. In the case of those that use some other mechanisms associated with project funding, this function comes under the control of the project leader. Another question sought to detect whether M&E comes from a certain internal demand from the PRI associated with the need to plan activities and accountability, or the need of the directors to control the activities of the organisation, or to another type of similar need. Here we find several examples that mention the fact that monitoring is linked to questions on the optimisation of the PRIs resources, improvement in their internal management, and greater effectiveness both in production and in their impact strategies. In Fundacin Expansiva, they claim that it is a form of optimising resources and of being more efficient in the use of peoples time. It is thus attributed to a
25

structural need of the organisation and to a form of planning of work, and to a methodology designed to boost its institutional prestige. Another question is to what extent the M&E is effectively related to the PRIs strategic planning, if both practices actually exist, and knowing if the M&E incorporates the feedback into its planning, or whether these practices can meet the guidelines set out in planning. Although there are practices in this respect, we have found or identified no clear mechanisms to link planning with impact M&E, at least not in a structured systematic way. In the case of the FGV, the Foundations activities are indeed planned. An annual meeting is held of all the unit directors at the Foundation, and at the same time annual planning meetings are held with all the members of each unit. The idea is that this planning should serve to see how each unit becomes involved and contributes to the mid-term objectives of the FGV. However, they underline the fact that it is a fairly new process, which is just being implemented, on which we have no clear evidence of connection with M&E practices. What the M&E done by FGV does influence is that it identifies the units that achieve greater media exposure, and therefore allows for a correction in the work of those with least exposure. However, these are not modifications in the agenda, or in the FGVs strategic plan, but in its capacity for dissemination. In general, almost all the PRIs analysed carry out some form of more or less strategic planning of their activities. In some cases, the feedback for defining this planning, as regards the perception of impact and the strategic definition of directions and objectives, comes from the various directors, in fairly formal processes. Such is the case of IIERAL from Argentina, Fedesarrollo from Colombia and Fusades from El Salvador, among others. Although we identified no standardised and formalised mechanisms using M&E evidence in the PRIs planning, it is worth mentioning that one of the forms of incorporating M&E mechanisms is the fact that impact objectives should be explicitly mentioned in the strategic plans. Expansiva is a good case in point since, as mentioned above, the strategic plan sets impact as a priority objective and defines the target publics on which to impact (policy makers, members of congress, political parties, entrepreneurs, academic world). Internal organisation and distribution of functions (structure): the PRIs analysed do not generally have specific areas responsible for M&E, but this is solved via other channels: - PRIs that only carry out media monitoring do so through the areas responsible for dissemination and institutional communication;

26

this function can be outsourced or contracted from a specialised company in the field; - in cases in which M&E is associated with project funding, this function comes under the control of the project leader; in only one case is impact M&E linked to questions on the optimisation of PRI resources; although almost all PRIs do some kind of strategic planning, no clear mechanisms were found to link planning with impact M&E, at least not in a structured systematic way; one of the channels for incorporating M&E mechanisms is that in the strategic plans impact objectives should be clearly explained. 3.3.2. Financial and physical capacities. A part of our investigation was geared to identifying the organisational capacities of the PRIs, in order to see, at a later stage, to what extent these capacities allow them to conduct M&E processes effectively. Our conclusion is that the PRIs analysed generally have very limited human and financial resources to assign to this type of activities, and that they prefer to assign them to generating other kinds of projects, usually linked to obtaining funding. M&E does indeed become a priority whenever those responsible for the PRIs realise that it would allow them to obtain greater funding. In this regard, the organisational structures are fairly heterogeneous among the PRIs analysed. By way of illustration we shall mention a few. FGV is one of the largest PRIs in terms of human resources, since it has a high teaching component among its functions. It is a structure of highly qualified teachers and researchers (with Ph.Ds and Masters degrees) with great practical experience (principally management experience). In turn, the resources for consultancy are generally the same as those allocated to research. It is calculated that between full-time and potential contract workers the FGV has around one thousand professionals. The Fundacin Expansiva, in contast, has a much smaller and flexible structure, with around ten researchers, and a few management and directorship positions. The PRI is managed on a project basis, which means that the researchers and resources vary according to the projects obtained. The budget largely depends on the number of projects. The permanent researchers have been included on the university degree course (UDP), and so many of their incentives respond to those of the institutions degree course (publications, specialisation, etc.). Except for certain large PRIs with numerous personnel and large budgets, the vast majority of the PRIs analysed are small or medium-sized and highly flexible (functioning principally on a project basis). In this regard, although the
27

lack of financial resources is a clear constraint to developing M&E capacities, what we found is that there is no direct relationship between higher budgets and doing M&E. Even the largest PRIs analysed, such as the FGV, Fedesarrollo, or the CLAEH, among others, do not have very developed or standardised M&E mechanisms. In fact, the existence of these mechanisms seems to be more closely linked, as we have said, to the source of funding, and to the existence of specific resources for that end. A clear example of this is Fedesarrollo of Colombia, which is a large structure, but does not have many international cooperation funds. It is financed principally by the private sector, or by projects with the Government, so it is not as compelled to conduct M&E practices. In general, the development of effective M&E capacities requires permanent structures dedicated to that end, with permanent and specific funding. For IDEA, the lack of development of more formal M&E mechanisms appears to be linked to the short lifespan of the organisation, and therefore also with the lack of human and financial resources to allow professionals to be assigned exclusively to this type of work. They say they are interested in establishing more concrete indicators of their impact, but so far have developed a more direct positioning strategy, given the limited work and development time. They recognise that the incorporation of M&E mechanisms would allow them to boost their reputation, thus generating a virtuous circle to obtain greater resources and projects, but they have not yet had the time to do any. CEGA of Colombia, on the other hand, does not carry out this kind of practices, largely because it concentrates all its efforts on obtaining funding. At the end of 2007, the CEGA-Uniandes alliance was created and is currently trying to consolidate itself institutionally, leaving M&E in a secondary plane. However, the PRIs that (at least partially) function by capturing funds for specific projects, such as IIERAL in Argentina, also insist on the fact that when there is a need to seek sources of funding for projects, the inclusion of M&E practices makes the project more expensive and therefore makes it difficult to obtain funding. Financial and physical capacities: the PRIs analysed generally have very limited human and financial resources to assign to impact M&E, and they prefer to assign them to generating other kinds of projects, usually linked to obtaining funding; M&E does indeed become a priority whenever those responsible for the PRIs identify that this practice allows them to obtain greater funding;
28

although the lack of financial resources is a clear constraint to developing M&E capacities, there seems to be no direct relationship between higher budgets and doing M&E; the development of effective M&E capacities is more closely linked, to dedicated permanent structures, and to permanent and specific funding; some maintain that when there is a need to seek sources of funding for projects, the inclusion of (own inititive) M&E practices makes the project more expensive and therefore makes it difficult to obtain funding. 3.3.3. Personnel policies and systems of rewards. Save exceptions, most PRIs analysed are small structures (between 15 and 20 permanent staff) and very flexible ones, since they function by projects. This means that one of the main incentives of PRI members is to obtain funding for their projects. The CORDES, from Ecuador, for instance, is a small PRI of some 17 persons (mostly economists). They have no funding from the public sector, and 60% comes from European foundations. The remainder, about 30%, comes from locally sold products. A similar structure is that of the CEBEM, with a staff of 15 persons. However, as with Expansiva and the CEDICE, the CEBEM has a fairly flexible structure in terms of the projects in which it participates, and the networks in which it is involved with other organisations in the region. IDEA is also a small structure, organised into three different areas (economic development and innovation, education, and evaluation of social programs). In all, this involves 15 persons, of whom some 12 are researchers. Although, as we have said, there is no very formal M&E activity, there is a personnel evaluation system to monitor compliance with the PRIs planning goals. In this regard, we found no clear link between incentive and awards mechanisms for personnel performance and conducting M&E practices. In most cases in which the main incentive of researchers is to obtain funding for their projects, the relationship between impact M&E and incentive structures of human resources is mediated by the question of funding. In other words, PRI members are concerned to show the real or potential impact of their projects in the hope that this will increase their chances of obtaining funding for them. Personnel policies and systems of rewards: since they function by projects they have very small and flexible structures, and the incentive system among PRI members is directly linked to obtaining funding for their projects;
29

the personnel evaluation systems identified seek to monitor compliance with the PRIs planning goals; no clear link was found between mechanisms of incentives and awards for personnel performance and carrying out M&E practices; the relationship between impact M&E and incentive structures for the human resources is mediated by the question of funding (PRI members are concerned to show the real or potential impact of their projects in the hope that this will increase their chances of obtaining funding). 3.3.4. Individual qualifications and management capacities. This dimension is concerned with the human resources profile, i.e. the areas they come from, and their professional background. In short, what is sought is the possibility to attribute impact M&E practices to some profession or human resources profile, and very especially according to the type of background of the most important members. It is precisely here that we found no clear correlation. Evidently, the professional profiles of the staff of the PRIs analysed are closely linked with the subject areas on which these PRIs work. However, there is no evidence to suggest that certain professional areas are more likely to adopt M&E practices. Individual qualifications and management capacities: there is no evidence to suggest that certain professional areas are more likely to adopt M&E practices. 3.3.5. M&E and research management. Here we looked into different, albeit interlinked, questions,: on the one hand, at impact M&E as a PRI feedback mechanism, as a form of self-evaluation to improve the performance and the impact of the PRI; on the other hand, certain suppositions are investigated, such as whether impact M&E varies according to the topics (subjects) on which the projects act, or whether it varies according to the stages of the policy process, or even whether it is done more frequently in any of the particular stages. Regarding the first point, as explained in 3.3.1, we were unable to identify clear and standardised M&E mechanisms within the feedback and planning process. As for the other suppositions, interesting responses were given, especially related to the feasibility of impact M&E. As regards topics, it was seen that when it is a question of local governments the chances of impact M&E are greater than when dealing with the national
30

government due to the fact that the former engage in the provision of goods and services that are easily measurable. As for the policy process, it is worth citing the response of the CIDE from Mexico, which maintained that when hiring is done for public policy purposes, impact M&E is easier at the formulation or design stage. However, when the PRI takes responsibility for evaluating impact, (e.g., public policy was already implemented when a large number of actors came on the scene), the causal attribution, determining the research impact on policy changes, becomes more complicated. In other words, the source continues, it is easier to do M&E at the ex ante stage of implementation, rather than ex post. This brings us to an important element that we found in M&E management, clearly mentioned in our conceptual framework, which deals with the nature of the influence and/or impact process. Although not mentioned in a generalised way by all PRIs, it is very clearly sustained by some in particular, such as the Programa Estado de la Nacin of Costa Rica and the CIDE of Mexico. It speaks of the impossibility of reliably evaluating the impact process, given its complex and multicausal nature. It generally deals with complex problems with many dimensions and different interwoven causal relations, as well as a complexity and multiplicity of players, all of which makes it very difficult, if not impossible, to trace clearly and unambiguously the impact or influence process. They maintain that impact in public policy has a great deal of negotiation, of lobbying, of sitting down to talk with the decision-makers to see how to make impact. The problem is that many of these things are not tangibly reflected, and in order to build an indicator one has to know where to obtain information. The risk is to end up forcing the indicator, and the errors that can be measured on the basis of those indicators are very dangerous, especially when incorrectly attributing impacts. M&E and research management: impact M&E is not considered to be a PRI feedback mechanism or a form of self-evaluation to improve performance; when dealing with local governments, the possibilities for impact M&E are greater, since they mainly deal with the provision of goods and services that are tangible and, therefore, easier to measure; it is easier to do M&E at the ex ante stages of public policy implementation (especially design), rather than ex post; questions have been raised about the possibility of impact M&E, given the multicausal nature of the impact phenomenon.

31

3.3.6. Institutional communication and other offerors. Here we were interested in investigating the dissemination of impact by the PRI, and in what measure it contributes to the objectives of the PRI. On this point, it is possible to find clear comparative evidence regarding the importance that PRIs give to the dissemination and communication of their activities and of their impact capacity. This is indeed a key dimension of the work of PRIs, given that communicating their impact serves precisely for the construction of their prestige, which at the same time, supposedly, contributes to an increase in their presence in public debate and their capacity for influence. In short, it would appear to be a virtuous circle that no PRI wants to waste. This explains to a certain extent that one of the most commonly used M&E mechanisms, as we pointed out at the beginnning of this section, is monitoring of the media, and measuring the media appearance of PRIs. In this respect, all the PRIs interviewed carry out some kind of more or less structured and professional media follow-up, but it is still an activity which is always present. Institutional communication: PRIs value the dissemination of their impact very highly; there seems to be a virtuous circle in which communication of impact contributes to the prestige of the PRI, which in turn helps increase its presence in public debate and its capacity for influence; this would explain why media follow-up is the most generalised M&E mechanism in PRIs. IV. CONCLUSIONS

By way of conclusion, we shall go back over the most important findings identified in the study. It is worth recalling that the study was of an exploratory nature, so the conclusions mentioned are not premises with any statistical significance, but they are the first appraisals that we dare make, and deserve to be looked at very closely. We can begin by saying that in the PRIs analysed the most commonly used form of impact M&E is media exposure, which in many cases is complemented with other M&E mechanisms (surveys of the visits to PRI web pages, the placing of members on reform programs, surveys of specific audiences, etc.). We can also say that there is interest in the issue of impact, in that it is a topic which is firmly installed on the agenda of the PRIs consulted. However, there is a very limited use of the most formal or standardised impact M&E mechanisms, specifically recommended for the matter. Outcome Mapping was especially identified, exclusively so in the PRIs that received Canadian funding, specifically from the IDRC. Now, the PRIs that do impact M&E do it more informally, casuistically and not very systematically.
32

In relation to the institutional framework in which PRIs operate and the extent to which it can affect the M&E of impact capacities, there are no very clear associations, except in some very specific cases. In these cases it was found that factors associated with the permanence of public officials in their positions (absence of a career civil service or of professionalisation of the public administration) and contexts of important controversy concerning freedom of the press and the influence of the Government on the media could actually reduce the probability of monitoring and evaluating impact. Following on with the exogenous factors, where there are clear associations is in the demand to M&E research impact. International funding can be said to be an important factor in doing impact M&E, especially when we speak of more formal or standarised methodologies. This mainly occurs when impact M&E is a specific requirement for funding. However, in those PRIs whose funding is mostly from the public sector, the chance of their doing impact M&E is lower. A topic to continue to observe and study more closely in the future is in what measure impact M&E practices promoted by the sources of international financing manage to outlive the financed projects. PRIs appear to be incorporating impact M&E as a requirement or demand by those who provide funding, so it is worth asking to what extent those practices have been institutionalised. Lastly, in relation to the contextual factors, it is claimed that the participation of PRIs in network projects, their interaction with PRIs from countries where impact M&E is carried out or their involvement in institutional communication strategies, facilitate the socialisation and incorporation of M&E practices. Of the endogenous factors, especially organisational structure, PRIs can be said not to have specific areas for M&E impact, but rather they do it through the areas entrusted with dissemination and institutional communication (with media follow-up), or through the project leader (when M&E is a demand for funding), or they outsource it. Another topic to emphasise is that it was observed that impact M&E is not seen by PRIs to be a self-evaluation mechanism, which feeds information back in order to improve performance and effectiveness. Neither were clear mechanisms found to link the planning of PRIs with impact M&E. However, the incorporation of impact as an explicit objective in strategic plans presents an opportunity to incorporate M&E mechanisms, due to the fact that many PRIs construct indicators for M&E and for attaining their strategic objectives, although they do not have impact as an objective.

33

As regards financial and physical capacities, PRIs have very limited human and financial resources and their exclusive assignation to impact M&E is long overdue, since it is thought to be a non-priority objective. In that respect, impact M&E appears as a second-order need, totally displaced by other more urgent needs which always arise in PRIs, such as financing. So impact M&E becomes a priority only if the incorporation of the practice allows more funding to be obtained. The M&E of impact capacities does not appear to be linked to a larger or a smaller budget, or to its continuity and permanence, because it would mean permanent structures dedicated to that end. This explains the system of incentives in PRI members, which is fundamentally structured around the need to obtain funding for their projects, since they function on a project basis. The personnel evaluation systems identified have to do with determining compliance with the PRIs planning objectives, which in turn makes clearly describing impact objectives in the strategic plans (as mentioned above) very important. Dissemination, another of the internal capacities analysed, appears to be highly valued by the PRIs, to such an extent that one can speak of a virtuous circle between communicating impact, the prestige of the PRI, its presence in public debate and its capacity to influence. On the one hand this would explain why media follow-up is the most generalised M&E mechanism in PRIs, but on the other it would appear to be another window of opportunity for promoting the use of more formalised or standardised M&E of impact mechanisms. Given the characteristics and the findings mentioned, an alternative could be for the existence of internationally agreed mechanisms and indicators to allow the sources of international financing to generate competition among PRIs to demonstrate effectiveness in the impact of their projects. In relation to the role of policy topics in the feasibility of impact M&E, it was claimed that when dealing with tangible topics, such as public works, the goods and services provided by the municipalities, M&E is easier. As regards stages, impact M&E before implementation appears to be easier than afterwards, due to the complexity of actors that appear on the scene with implementation. Finally, we would like to refer to this last comment on the constraints on feasibility of impact M&E in public policy. Many PRIs were sceptical on the subject of impact M&E, arguing that it is a complex, dynamic and diffuse process which makes the causal attribution to actions included in specific projects impossible. There is no doubt that determining the factors that affect
34

public policy is one of the most challenging methodological decisions in policy analysis. However, in accordance with these limitations, many of the mechanisms available speak of a logical or reasonable link to the projects and actions undertaken by PRIs, highlighting that that does not necessarily mean a linear causality. In fact, some methodologies prefer to speak as little as possible of impacts, considering them to be the most indirect medium and long-term effects, referring only to products and outcomes (the latter being seen as the direct and immediate effects of the actions). In conclusion, it is worth returning to one of the comments made in the study, which stressed the importance of adopting methodologies which, without giving up the characteristics of objectivity and precision, are able to contemplate the political dynamic of decision-makers, i.e., decisions that are also feasible. Therefore, the involvement of researchers in policy processes and, if possible, in much of the public policy process (before and after implementation), appears to be the most satisfactory alternative for impact M&E in policy. RECOMMENDATIONS

V.

A fundamental recommendation is that the counterparts (international foundations, international organisations, etc.) promote and demand impact M&E. It is precisely as counterparts that PRIs must be accountable, and one way of being accountable is by revealing the outcomes and impacts generated. These possibilities of effectiveness mean that the promotion of impact M&E based on demand is essential. From our review of the literature, organisations were found to be promoting this type of M&E and even constructing related methodologies (especially the IDRC and ODI), with satisfactory results. A further recommendation consists in either extending the period of project funding, or organising it in a sequence of at least two parts, in which the second occupies itself with the M&E of outcomes and impacts, in our case of impact. This recommendation is particularly useful in those PRIs that lack regular public financing and which depend on international financing and specific competitions for funds. If this is coupled with factors such as political instability and the constant change of public officials, continuity in funding by international sources becomes very important. Another recommendation consists in explaining the objectives of impact from the moment the specific projects are formulatied. Although they do not use, nor do they have, knowledge of specific mechanisms for impact M&E, the majority of PRIs know and have experience of project formulation methodologies.
35

Although these methodologies do not concern themselves with impact M&E in particular, they do look at M&E in general (for instance, the logical framework), whether it be of activities, products or outcomes, and require the construction of indicators in that regard. Therefore, getting the PRIs to clearly describe objectives of impact when formulating their projects would mean that specific indicators of impact M&E are automatically included. A similar recommendation is that PRIs outline their impact objectives in their strategic plans. In the study it was seen that the majority of PRIs have strategic plans and that some of them have indicators to measure progress in achieving their strategic objectives. It was also observed that in the personnel evaluation systems, the contribution to achieving the strategic objectives is one of the main criteria. Therefore, in the same vein as with project formulation, specifying the objectives of impact in planning would entail important modifications in favour of impact M&E. Another interesting recommendation has to do with the promotion of network projects, interactions between PRIs or the formation of consortia, wherever there is a PRI with experience in impact M&E. The socialisation of practices is one of the most effective channels for incorporating this type of mechanisms. Moreover, using this channel for promotion is both more efficient and more effective because these spaces bring together several PRIs. In the particular case of consortia, their principal purpose involves institutional relations and an improvement in communication between PRIs and decisionmakers. Therefore, the level of awareness of impact and its M&E may be more significant. An additional recommendation could be the creation of a specific area entrusted with media follow-up and media exposure of the PRI. Given the great value that PRIs attach to dissemination, and their lack of resources to undertake the construction of a new and exclusive structure for impact M&E, an alternative to introducing these practices is to begin with the areas of communications and dissemination. There are also more economical alternatives, such as those mentioned in the analysis, where the service is contracted directly from companies dedicated exclusively to the matter. Another form, involving less re-engineering, is to strengthen the institutional area of PRIs, a natural space for responsibility in these functions. As could be seen, this area does not yet enjoy adequate institutional weight as observed by those who reflect on the phenomenon of think tanks. Mainly in those PRIs where a majority of members are from the academic world, and those with few researchers with political experience, the importance of impact may be seen as a desirable objective but one that is not so easy to measure. Echoing the words of
36

one of the interviewees, it could be said that there are PRIs which give greater importance to the Think part over the Tank part, and vice versa. Obviously, a balanced combination between both extremes is the wisest recommendation. Therefore, one recommendation is that those PRI members with the required profiles to interact in the policy process should be incorporated, or at least use should be made of them. Also, policy research should involve itself in the largest number of stages of the public policy process, in such a way as to guarantee impact of the projects. Another recommendation consists in disseminating M&E methodologies more broadly, promoting discussion of them in academic spaces. The aim here is to begin to discuss the constraints but also the potentialities of the current methodologies for impact M&E, to improve, adapt or reject them in consequence, according to criteria of objectivity, precision and especially feasibility. We thus conclude our study, hoping that we have been able to contribute to a strengthening in the link between knowledge and public policy. We also hope to have encouraged discussion or closer scrutiny of the different lines identified here.

37

ANNEX N 1: Mechanisms available for impact M&E11 There used to be more mechanisms available for the M&E of projects of policy research, but as they are mechanisms to M&E processes, i.e. to evaluate resources (or strategy) and activities (quality), and cannot therefore be associated with impact M&E, they have not been developed. The mechanisms to M&E resources or strategy were the following: Organisational Assessment: A Framework for Improving Performance; Enhancing Organisational Performance: A Toolbox for Self-assessment; Institutional Assessment: A Framework for Strengthening Organisational Capacity for IDRC's Research Partners; Logical Framework; Social network analysis; Impact Pathways; and Modular Matrices. The mechanisms to M&E activities or quality: Fit for Purpose Review; Lighter Touch Quality Audit: Expert visits; Horizontal Evaluation: Visits from colleagues for mutual learning; Appreciative Inquiry. The mechanisms available for impact M&E found in reviewing the literature are discussed in brief below. 1. Evaluating academic articles and research reports (quality of science criteria): This consists in evaluating academic articles and research reports, and the evaluation criteria therefore depend on the academic quality of the articles and the precision of the research reports. The questions evaluated have to do with the contribution and robustness of the hypotheses, the sources of the data and the gathering techniques used, the information processing techniques and the analysis, as well as the journals where the papers are published. In short, whether they have been produced in accordance with the evaluation criteria usually considered in academic and scientific institutions. 2. Evaluating policy and briefing papers: The M&E criteria of these mechanisms are somewhat different from those used for academic articles and research reports; the logic of argumentation and construction used in the documents has more to do with policy analysis. In that regard, Hovland (2007), warns that both policy and briefing papers are usually written to shed light on a certain area of policy, with the difference that the latter are short (between 1 and 6 pages). Therefore, the evaluation criteria relate to the clarity they bring to the public problem, the possible solutions and the recommendations to follow (Young & Quinn; 2002, cited in Hovland; 2007). 3. Evaluating websites:

11

Most of the mechanisms mentioned are in the research paper by Hovland (2007) Making a difference: M&E of policy research, of the ODI.

38

Hovland (2007) thus refers to the paper by the NGO Training and Research Centre (INTRAC) in Oxford, which has developed a guide on how to evaluate websites dedicated to communicating research to an international audience. Among the value criteria used are architecture (the structure and logic with which the pages interconnect, i.e. the navigability of the website), the technology (which determines accessibility to and availability of the site according to the different software and hardware capacities), the style (appearance of the site), the content (in terms of quality, authority, relevance, but also in terms of interaction with the users), strategy (to what extent the indicated objectives concerning the target audience were found) and administration (the human and financial resources the sites have at their disposal) (Hovland; 2007). 4. Evaluating networks: In this mechanism Hovland (2007) refers to the paper by the International Institute for Sustainable Development (IISD), specifically the guide on how to evaluate knowledge networks. The guide takes elements from other methodologies of formulation and of M&E and establishes three phases. One is planning, which is used at the start of the activities of the network to record the work plan, the beneficiaries of the work and the indicators of change desired for the network projects and for the network considered as a whole. The second is monitoring-progress journals, consisting of a record and follow-up of the activities and developments of the members of the network. Lastly, the evaluation, constructed largely on the basis of progress journals, affords the network an appraisal as to whether its programs and projects are being directed along the correct channel, whether the desired outcomes are being achieved and whether some type of adjustment in the network is necessary. 5. The After Action Reviews: This mechanism consists in the members of an institute gathering to discuss and evaluate a certain event after the fact. It is a very practical technique that requires no specific standardised patterns for its execution, and enormously facilitates improvement in activities and organisational change (Hovland; 2007). 6. Media exposure: There are several indicators dedicated to M&E media exposure. Some of them are coverage in the written press, television exposure, the sum of testimonies made to commissions in the Senate and the Chamber of Deputies, the number of publications produced (Abelson; 2006). There are also addresses, opinion columns, interviews and consultations (Camou; 2006). These indicators are linked with what is usually known as media follow-up and which is distributed among PRIs as a press release. 7. Impact Logs:

39

Returning to the list by Hovland (2007), impact logs are a RAPID (ODI) technique used to gather direct answers generated by the research products. It is not a systematic channel for collecting users perceptions, but is relatively informal feedback in which the users and persons that have come into contact with the products offer comments or anecdotes that are of use to the organisations in improving future projects and decisions. 8. New Areas for Citation Analysis: This mechanism combines the analysis of academic quotes with new analysis measures, which may be used to study and trace the impact of policy research products in different areas (Hovland; 2007). They include international standards, government policy documents, operation handbooks of public organisations or professional associations, training manuals and text books, press articles, websites (including mentions of the products, links to PRIs, or links for access to the products). 9. User surveys: They range from programmed questionnaires for users to the holding of focus groups. 10. Framework for a Strategic Evaluation of IDRC-Supported Research: Starting with the difficulties in measuring linearly the influence of policy research projects, this approach looks at the range of actors involved in the project, the nature of the relationship between those actors and a good sense of how the policy network has evolved over time (Lindquist; 2001). It maintains that an evaluation of the influence of policy has to concern itself with carefully identifying intermediate influences, such as expanding the policy capacities of the players, extending the horizons of those who make up a policy network and implementing policies. Developed by the IDRC, it suggests that any approach for strategic evaluation must concern itself with a description of the nature and evolution of the policy network involved, the objectives and expectations of the projects, and the products and outcomes of the projects (including unforeseen effects) (Lindquist; 2001). 11. Temporal Logic Model: Also suggested by the Evaluation Unit of the IDRC, the temporal logic model appears as a methodology to improve on the logical framework. The main criticism is that the latter is built on strong premises that presuppose social reality as a closed system, contrary to the temporal logic model, which is designed to act on a changing reality, more in line with a soft, open system. But the reasons why this methodology is included in this list have to do with the fact that this conception of an open system, allows different logics to be incorporated in all the formulation and implementation process of the policy research projects, in our case, the logic of the policymakers. By incorporating

40

this logic from the outset, impact is practically assured and M&E of it is highly probable. 12. Most Significant Change: This mechanism consists in gathering experiences of significant change in the eyes of the members of the organisation, and the consequent selection of the most important. Members are consulted as to the most significant changes that occurred as a result of their projects. Subsequently, these experiences were filtered upwards to the upper hierarchy by means of a transparent process, to reveal the most significant experience. Besides impact M&E of policy research projects, this technique permits internal comparison between the projects of a single PRI. 13. Innovation Histories: Similar to the above, these histories have to be of projects that have generated some kind of innovation. They are accompanied by a great deal of discussion, collection of documents and evidence, and participation by stakeholders, to decide upon the innovation (Hovland; 2007). They may include the realisation of case studies. 14. Episode Studies: Developed by ODI, this mechanism emerged in response to the methodological difficulty of attributing policy changes to a particular research project. It draws attention to the impact produced by the project in question, but by going backwards along the causal chain, it seeks to identify other factors, in addition to the project, that may have influenced in the change recorded. What it aims to identify is what influenced the policy change and the relative role of the research project. Unlike a linear regression, an historical narrative is built here on the basis of policy decisions and practices, making use of a review of the literature, interviews with key players, recordings of the experiences of the players, discussions in workshops (Hovland; 2007), etc. 15. RAPID Outcome Assesment: RAPID Outcome Assessment (ROA) was developed as part of the Process and Partnership for Pro-poor Policy Change (PPPPC), a project developed by the CGIAR-affiliated International Livestock Research Institute (ILRI) and ODI.s Research and Policy in Development Group (RAPID). According to Hovland (2007), the ROA takes elements of outcome mapping and was designed as a learning tool to evaluate the contribution of actions and investigations in a project on policy change and its environment. It is based on a visual methodology, which permits mapping of the changes in policy or its environment. This is done mainly on the basis of comparisons between the situation before and after the execution of the project, of the policy environment, of the players involved and of their behaviours.

41

16. Outcome Mapping: A technique developed by the International Development Research Centre (IDRC), outcome mapping promotes constant self-evaluation from the planning stage of the project, contrary to the perception that M&E must be carried out after implementation. Instead of concentrating on evaluating changes in the state of affairs (static outcome), it looks at changes in behaviour, relations, actions or activities of the persons, groups or organisations with whom the policy research projects work directly. It looks at M&E of the improvements that PRIs can generate in the direct influence of their projects. It includes three phases: intentional design, outcome and performance monitoring, and evaluation planning.

42

Annex N2: The cases analysed. Bolivia 1. Centro Boliviano http://www.cebem.org/ de Estudios Multidisciplinarios (CEBEM)

The Bolivian Centre for Multidisciplinary Studies (CEBEM) is a non-profit association dedicated to multidisciplinary research and teaching at postgraduate level in the field of social sciences, as a private and autonomous centre. Its main areas of action are research, reflection and discussion of important problems in Bolivian society and in Latin America on the basis of multidisciplinary perspectives and approaches; the dissemination of its intellectual production through publications, seminar, workshops and national and international knowledge networks; and, the organisation of professional refresher courses. The CEBEM focuses its efforts on studying the scope and impact of State policy, which, transforms the States relations with society, alters the methods in which the economy traditionally functions, and impacts in the composition of the popular sector and the practices of the social, indigenous and trades union movements. The work of the CEBEM began in the 1980s, with the institutional support of SAREC of Sweden. The idea was to create an institutional space where research could be carried out within the field of social sciences but independently of teaching and government institutions. The CEBEM is one of a series of institutions that appeared during the dictatorships of the 1970s in Latin America, a period in which the majority of state universities, and the social sciences especially, had no space for development, and the international cooperation dedicated to scientific research supported the formation of independent centres where social scientists could pursue their work. Peru 2. CIES Consorcio de Investigacin Econmica y Social http://cies.org.pe/ The Consortium of Economic and Social Research (CIES) is an association of 44 prestigious institutions in Peru dedicated to research and teaching in economic and social sciences. From the start of its activities in 1989, the Consorcio has received the support of the Canadian Technical Cooperation Fund, specifically of the Canadian International Development Agency for (CIDA) and of the International Development Research Centre (IDRC). This allowed a series of investigations to be carried out, which subsequently became a source of knowledge for the structuring and planning of public policy by different State agents. In 1999, after a joint effort by a broader group of
43

institutions, the CIE incorporated a social agenda within its objectives and became the CIES, in which funds for research are allocated by means of an annual competition. In this new stage, the Consorcio continues to respect its initial purpose: to strengthen the community of Peruvian researchers with the aim of producing and disseminating useful knowledge for the design and execution of public policy. The purpose of the CIES is to strengthen the Peruvian academic community in the production and dissemination of useful knowledge for analysts and decision agents in the public sector, civil society, the media, private enterprise and international cooperation. Over the years, the Consorcio has developed various mechanisms to ensure that the outcomes of research feed into public policy processes in the country. The Consorcio publishes the monthly virtual newsletter Anlisis de Polticas and the quarterly research journal Economa y Sociedad. CIES is basically engaged in promoting applied research. As a second-tier institution, it carries out research through a system of competitions in which independent juries rate not only the academic quality but also the relevance and impact of the public policy projects. The CIES favours research in networks, user participation and an interdisciplinary approach. Ecuador 3. CORDES Corporacin http://www.cordes.org de Estudios para el Desarrollo

The Corporation of Studies for Development, CORDES, is a private non-profit institution founded in 1984, on the initiative of former president Osvaldo Hurtado, together with a group of scholars and professionals with experience in the public and private sectors. CORDES sees as its mission to generate and promote solutions for the development of Ecuador and to perfect the democratic process by means of studies, consultancies, training and projects. Hence its principal objectives are: to investigate Ecuadors economic, political and social reality; to study the political institutions and their influence on democratic stability and on development; to promote the dissemination of research in seminars, conferences and publications. CORDES is active in training, research and consultancy work. It offers training programs for the public and private sectors in macroeconomics, economic policy, econometrics, financial markets, economic theory and international economics. It publishes three periodicals: Carta Econmica (monthly magazine), Discusin (newsletter on issues of social market economics and the rule of law), Notas Tcnicas (free electronic newsletter on technical matters of national interest).
44

CORDES offers a consultancy service to national and foreign bodies engaged in specific areas of work. The institution has permanent researchers and associated consultants. CORDES has carried out consultancy work for organisations such as WB, IDB, CAF, ECLAC, FAO, CAN and the Banco Central del Ecuador, among others. CORDES receives permanent financial support from the Konrad Adenauer Foundation. CORDES has been very influential in certain important events in Ecuadorian political life, such as the drafting of the countrys new Political Constitution in 1997 and 1998, when Osvaldo Hurtado, President of CORDES, chaired the Constituent Assembly. Throughout the calendar year, CORDES organises technical conferences and meetings which broach topics of national life. These events bring together persons from the public and private sectors including politicians, entrepreneurs, trades unionists, journalists, scholars and professionals. Venezuela 4. CEDICE Centro de Divulgacin del Conocimiento Econmico para la Libertad http://www.cedice.org.ve/ CEDICE is a non-profit civil association, whose central objective is publishing, education and training in the principles that underpin the free action of individual initiative, as well as promoting the organisations production of knowledge, research and analysis of the conditions that enable a free and responsible society to exist. It was set up in 1984 by a group of Venezuelan entrepreneurs and intellectuals concerned to disseminate liberal thought, and its two great component elements are freedom and democracy. The initial objective was to publicise "world political and economic reflection that has as its philosophy the free interplay of individual freedom and the fundamental beliefs that underpin it." With a view to achieving this objective it set out to disseminate the publications that underpin the notions of freedom, collective action and the democratic system.

Uruguay 5. Centro Latinoamericano http://www.claeh.org.uy de Economa Humana (CLAEH)

The CLAEH was born in 1947, when a group of young people set up Equipos del Bien Comn (Teams for the Common Good) and began to carry out research on social reality, under the influence of the ideas of Louis-Joseph Lebret, a Breton priest. In 1957, the members of the Equipos del Bien Comn, in
45

coordination with other groups that had been created in Brazil, Colombia, Venezuela and Argentina at the initiative of Lebret himself, founded the Centro Latinoamericano de Economa Humana (Latin American Centre for Human Economics). They also launched the publication of a journal Cuadernos Latinoamericanos de Economa Humana, which published papers on the social sciences. In 1973, after the coup dtat, and on the initiative of Juan Pablo Terra, the CLAEH showed renewed impetus in its research and activities, promoting pluralism, free thought, freedom and democratic ideas. The CLAEH became an academic centre distinguished by pluralism and interdisciplinarity. Its work took on a professional nature and its programs began to incorporate researchers, taking staff levels to more than twenty persons. In 1979, the Promotion Department was set up to research the issue of mediation and intervention in society. The CLAEH thus became a complex institution engaged in academic, social intervention, research and training activities. At the beginning of the nineties, the CLAEH became an autonomous institution and, on the basis of its basic commitments, prepared to tackle the challenges of a new period marked by the democratisation of the country and the challenges of globalisation. It investigated rising poverty levels among the population of young children, introduced a model of local development into the country, developed a model of social intervention, investigated the most characteristic features of Uruguayan politics and its institutions, studied the history of Uruguay, and looked at issues of international insertion. It set up its own publishing arm and publishes Cuadernos del CLAEH, the oldest social science review in Uruguay. Argentina 6. Centro de Estudios de Estado y Sociedad (CEDES) www.cedes.org The State and Society Study Centre was founded on 1 July 1975, when Argentina was in the midst of a particularly serious institutional situation, as reflected in the crisis that involved several of the most important academic and higher education institutions. Therefore, a group of social science professionals, decided to set up the Centre with the purpose of constructing an independent and pluralistic space to safeguard and give continuity to the research work and thinking previously being developed in other scenarios. The Institution was given the form of a civil organisation, independent of the State, political parties and any enterprise or social organisation. From its birth CEDES has studied the social, political and economic problems of Argentina and Latin America. In its early days, work in those subject areas was largely possible thanks to the significant support obtained from various foreign institutions, including Sarec (Sweden), the Ford Foundation, IDRC (Canada), the Andrew Mellon Foundation, and the Inter-American Foundation.

46

The academic research and activities carried out by members of the Centre have been and are guided by a basic concern over the quality and scientific precision of their production, as well as a concern over political democracy and pluralism, economic growth, social equity and cultural diversity. It should be stressed that the CEDES has received the Certificate of Qualification from the National Agency for Scientific and Technological Promotion, which certifies that the centre "has presented scientific-technological research programs that it either executes directly or supports by means of promotion activities that meet the requirements of quality and originality to be considered as such" (ANPCyT Resolution N 54/08). During 2007, the CEDES won two important institutional recognitions: - as Special Consultative Body of the United Nations Economic and Social Council (ECOSOC). - as a Unit Associated to the National Council of Scientific and Technical Research (CONICET), Resolution N 2486. From the point of view of its internal organisation, the CEDES functions as a cooperative of researchers, each one of which makes the commitment to contribute individual monthly overheads, the amounts of which have varied historically but have always been set according to a scale that reflects the academic categories mentioned above. These contributions are designed to cover a series of common expenses of the institution. 7. IERAL Fundacin Mediterrnea http://ieral.win-servers.com/data/ Fundacin Mediterrnea is a non-profit civil association created in the city of Crdoba, Argentine Republic, on 6 July 1977, on the initiative of 34 enterprises in the province of Crdoba, convened by Pedro Astori, to: promote research into national economic problems; contribute to a better knowledge and solution of Latin American economic problems; create an apolitical forum in which to discuss the most important national and Latin American problems, where scholars can contribute their intelligence to devise economic solutions with the sole condition imposed by its absolute respect for the freedom and dignity of the human person. To carry out these objectives, the Fundacin Mediterrnea created the Institute of Economic Studies on Argentine and Latin American Reality, which has a team of professional economists engaged full-time in research work. The Foundations group of promoters recommended concentrating efforts to help design a sectoral and geographically integrated economy, which would effectively make use of the countrys human and natural resources and would allow Argentina to firmly project itself to its Latin American neighbours, in an attitude of solidarity, so as to participate together with more equitable outcomes in the world economic system. From its creation to 1982 it undertook
47

broad research work that was disseminated among all the members of the Association of Political Economics and to all business sectors. Much of the influence of this PRI was provided by Domingo Cavallo12, one of the directors of IERAL, and his positions in the national government. This made it possible for the studies made at the national level to be completed with studies at a regional level. The work method applied consisted in setting an annual research program and promoting discussion between researchers and businessmen on working drafts, the former contributing their professional training and the latter their experience as operators in the real economy. Periodically ordinary meetings are held between entrepreneurs and researchers, in addition to plenary sessions with members from all over the country. In order to comply with the objective of analysing national and Latin American economic problems and offering alternative solutions, the IERAL has several research areas (made up of teams led by one or several researchers). Brazil 8. Fundacin Getulio Vargas http://www.fgv.br/fgvportal/ The mission of the FGV is to push forward the frontiers of knowledge in the field of social sciences, and to produce and transmit data and information rather than just conserving and systematising it, with a view to contributing to the socio-economic development of the country, to an improvement in national ethical standards in order to attain responsible and shared governance, and the countrys insertion into the international arena. The Vargas de Getulio Foundation was set up on 20 December 1944. Its initial aim was to prepare or train personnel for work in the countrys public and private administration. When Brazil began to lay the bases for the growth that would be consolidated in the following decades, in anticipation of the arrival of a new era for the country, the FGV decided to broaden the focus of its functions and of the restrictive field of administration, moving into the broadest of the social sciences. The institution went beyond the frontiers of education and entered the areas of research and information. In Brazil, the Getulio Vargas Foundation inaugurated the graduate and postgraduate degree course in public and private administration, as well as a postgraduate degree course in economics and psychology, among others. The FGV also put in place the bases for a solid economy, along with the creation and analysis of national accountancy indicators and of economic indices. Initiatives like these helped the professional in search of training and even the common

12

In January 1991, Domingo Cavallo became the Argentine Minister of the Economy and Public Works and Services, taking with him as collaborators the leading researchers in the Institute.

48

citizen to better understand the economic and social functioning of Brazil. The FGV invests in and fosters the development of internationally acknowledged research, and the subjects include the macro- and micro-economy, finance, law, health, social assistance, poverty and unemployment, pollution and sustainable development. In addition to its important technical and academic training programs, the FGV undertakes work on request for the public sector and international bodies, such as the World Bank and the IDB. Mexico 9. Centro de Investigacin http://www.cide.edu/ y Docencia Econmicas CIDE

Created in 1974, the CIDE is a centre of research and higher education specialising in social sciences, based on international standards of quality and funded with public resources. The central aim of the CIDE is to contribute to the development of the country by generating accurate and relevant knowledge, as well as training a new generation of leaders capable of acting with creativity and responsibility in an open and competitive world. The production and dissemination of knowledge are the focus of the life of the Centre. It has committed to achieving a better and more solid understanding of the great economic, political and social issues of our time. The CIDE sees its role as being global in the search for intellectual progress. The CIDEs full-time staff of research professors has PhDs. from the worlds best universities, and the student body is of around 400 pupils. The CIDE engages in three fundamental activities: scientific research; meritocratic training of future leaders at graduate, Masters and Ph.D. level; and dissemination of socially useful knowledge. These activities are carried out in six academic divisions: Public Administration, Economics, International Studies, Legal Studies, Political Studies, History. The research area is looking to reach an optimum combination between work geared to advancing scientific knowledge, including competing and speaking in international scenarios, with studies that are able to nourish decision-making in the public, private and social sectors. The CIDE has pledged that its research products should be relevant and, therefore, useful tools for the design of solutions to the principal problems facing the country. In brief, CIDEs commitment is focused on accurate and relevant research, as well as on its teaching programs of international quality designed to make a strategic contribution to social mobility. 10. Fundacin IDEA http://www.fundacionidea.org.mx/
49

IDEAs mission is to design and promote innovative public policy to ensure equality of opportunity for Mexicans, through economic growth and a reduction in poverty levels; and to be a reliable source of independent analysis for government officials and the public in general. IDEAs vision is of a Mexico free of poverty and with a vigorous economy, in which public officials make decisions with a clear understanding of the different options within their grasp and in which citizens have the information and mechanisms to demand that their leaders be held accountable for their decisions and performance. The challenge is to be an independent, specialised public policy organisation which will contribute to public debate, identify best practices in the world, and help towards designing and implementing effective public policy. On this basis, they carry out quality research and analysis of the public policy in force and offer creative and politically feasible proposals to solve Mexicos public problems. IDEA sees equality and defeating poverty as its main focus of work. IDEA designs and provides information on policies to generate the conditions that will allow Mexicans to defeat poverty and inequality. It identifies barriers to sustained economic development, analyses them in depth, and uses best practices internationally to recommend the best policies to eliminate them. IDEA seeks to drive the necessary changes by presenting its arguments directly to those responsible for public policy, to opinion leaders, and to the public in general. They believe in the capacity of open debate to influence public decisions, and they plan to raise the efficacy of political discussion in Mexico by publishing analyses and proposals accessible to the public in general. Colombia 11. FEDESARROLLO Fundacin para la Educacin Superior y el Desarrollo http://www.fedesarrollo.org.co The Foundation for Higher Education and Development (Fedesarrollo), was created in 1970, at a time when there was in Colombia no reliable economic information or technical studies made systematically to underpin the shaping of coherent development policies designed to guide decision-making in the private sector. Its creation was linked to two important facts: firstly, a real interest by private enterprise in supporting the foundation of an independent research centre that would contribute to the formation of leaders with a profound knowledge of the economic reality of the country and that, in turn, would permit the establishment of a closer bond between universities, government leaders and private enterprise; and secondly, the existence of highly qualified professionals willing to embark on this undertaking.
50

Its independence has been possible thanks to the consolidation of critical and objective opinion on the countrys economic situation, based on a precise technical analysis regardless of party positions and schools of economic thought. The high academic level of the researchers, their flexibility in addressing new subject areas and lines of analysis, the use of reliable sources of information, and the permanent updating of economic science have been fundamental elements in guaranteeing accuracy and quality in research. Fedesarrollo has fostered debate on the most relevant economic issues of the moment and has attempted to influence the design of economic policy through the outcomes of its research, the periodic analysis of the economic and social situation, and the dissemination of its opinions in press articles. Similarly, Fedesarrollo has made great efforts to introduce research into social themes as a priority area for work. In reference to its short-term macroeconomic analysis, it has made numerous sectoral studies that have contributed to an understanding of its dynamic and its prospects. With the support of international bodies and with the creation of International Research Networks, Fedesarrollo has done very valuable research into comparative economics and has published studies on the Andean economies. Knowledge of these experiences has been useful in identifying policies with harmful effects and to suggest those that could be applied in the case of Colombia. Fedesarrollo does no work of a private or confidential nature. All its research is in the public domain and the director of the institution has as one of his principal responsibilities to ensure its maximum dissemination. Press articles, the publication of books on different topics, especially text books, have contributed significantly to the formation of students, to knowledge of the Colombian economy from a perspective independent of the government and the trades unions, and to identifying problems and needs in fields such as education, health, justice and the environment. 12. CEGA Corporacin http://www.cega.org/ de Estudios Ganaderos y Agrcolas

A private and independent foundation created in 1982 for social and economic research in Latin America. Cega was a centre of stockbreeding and agricultural studies which was an important presence up to 10 years ago, when for different reasons its staff of researchers and resources began to shrink. At the end of 2007, the CegaUniandes alliance was created and it is attempting to revive Cega little by little. To date, Cega has no budget to allow it to do research and it must compete with other research centres on proposals.
51

Cega conducts an entrepreneurial opinion poll among farmers every quarter (a sort of analysis of the general situation, but with qualitative questions) and has as partners the Farmers Society of Colombia, the Finance Fund of the Farming Sector, the Pro-Farming Chamber of the National Association of Industrialists and the periodical Portafolio (a newspaper dedicated entirely to economic issues) which belongs to El Tiempo (the leading newspaper in Colombia). Outcomes are published in Portafolio and in other regional newspapers, and are presented in trades union assemblies and in certain journals. Chile 13. CIEPLAN Corporacin de Estudios para Latinoamrica www.cieplan.org CIEPLAN was born in 1976 (out of the CEPLAN of the Universidad Catlica), with its own projection and personality. It is an academic institution based on values and strong intellectual precision. Its periodical Coleccin Estudios CIEPLAN has become one of the most widely read reviews in Latin America. With the support of large international cooperation agencies, such as the Ford Foundation, CIEPLAN was a key research centre during Pinochets dictatorship in Chile, providing a space for critical analysis and democratic reflection on Chiles economic and social policies. The role of CIEPLAN was fundamental in guaranteeing the transition process to democracy in Chile, and the drawing up of an agenda of economic and social reforms that underpinned the governments of the Concertacin, and which became the space in which many of its leaders and officials were formed. CIEPLAN currently works with a vast network of Latin American institutions in the fields of macro-economics, social policies, democratic governance and innovation. Some of the most important projects under way are the Project for Modernisation, Reform and Perfecting of the Chilean Electoral System (this project aims to enrich the debate over the Chilean electoral matrix, observing it from a perspective that includes reform and modernisation of the electoral regime, the party system and the electoral system itself); the Project for a New Economic and Social Agenda for Latin America (a project designed to identify specific reform agendas to advance in achieving the Millennium Development Goals (MDGs); the Social Cohesion in Latin America Project (which aims to contribute to the knowledge and debate on public policy to overcome the causes of social fragmentation and to foster social cohesion in Latin America). 14. Expansiva UDP http://www.expansivaudp.cl/acerca_de/?tema=organizacion# The Expansiva UDP Public Policy Institute was born from a strategic cooperation agreement signed in July 2008 by the Corporacin Expansiva and
52

the Universidad Diego Portales, with the aim of linking the disciplinary and academic knowledge available with the research, design, execution and evaluation of public policy. Taking its inspiration from the shared principles and values of both the institutions that led to its creation, such as meritocracy, pluralism and the critical independence of its members, the Institute began operations on 1 October 2008 within the Universidad Diego Portales. The Expansiva UDP Public Policy Institute brings together all contributions in favour of debate and reflection on public policy made since 2001 by Expansiva in numerous investigations, seminars, round tables, books and columns in the media. In turn, the Institute incorporates into its work all the academic knowledge, experience, critical spirit and interdisciplinary work for which the Universidad Diego Portales is known. According to Expansiva PRI, Chile needs better decision-making which requires institutional mechanisms that use knowledge from the scientific disciplines and which take the necessary precautions against the influence of private interests. The Expansiva UDP Public Policy Institute takes up this challenge and channels the knowledge available in the University towards the decision-making processes on public topics. The transformations that have occurred in Chile over the last decades have led to greater demands for quality in public policy: persons expect tangible results in their lives, they want to be part of decisionmaking processes and give priority to new topics, such as the environment or culture, in their evaluation of progress in society. With a view to achieving its purpose, the Institute has defined three strategic areas of focus: research, dialogue and training. Research is used to support the decisions and the policies that society has to adopt. In consequence, its work focuses on key topics for the progress of Chile, which are addressed with a high degree of academic excellence, encouraging the creation of knowledge in collaboration with other institutions, universities and researchers. The Institutes mission is to improve public policy through research and constructive dialogue, with independence and a critical spirit. Costa Rica 15. Programa Estado de la Nacin http://www.estadonacion.or.cr/ The State of the Nation is a program of research and training on sustainable human development created in 1994, with the aim of endowing society with easily accessible instruments to study its evolution, develop instances of accountability, strengthen mechanisms of participation and negotiation, and contribute to the creation of national consensus, which is vital in times of profound reform.
53

The State of the Nation Report on Sustainable Human Development is an independent initiative, promoted in Costa Rica by the state universities represented in the National Council of Rectors (CONARE13) and the Ombudsman, whose mission is to improve access by citizens to a wide range of timely and reliable information on matters of public interest. This project is based on the conviction that, in order to address social, economic and environmental challenges, a democracy requires informed citizens, with aspirations and constructive capacity. A citizen with information is a citizen with democratic power, but also one with greater responsibility towards his community and country. From its first edition, the Report has retained its basic structure, which is modified and enhanced year after year, with the aim of identifying and following important trends in analysing national performance. This structure considers: a permanent set of chapters on national aspirations for sustainable human development in economic, social, environmental and political matters; a statistical compendium containing over 248 variables, plus levels of disaggregation according to different criteria (sex, zone, region, age groups, production or institutional sectors, etc.), from 32 official sources of information. As of its eleventh edition (2005) the Informe incorporated a section on Debates for development in which there is discussion on themes related to public policy alternatives for the promotion of human development under a comparative but purposeful perspective with other countries. The execution of what is now the Programa Estado de la Nacin is based on three orientations: academic precision, social legitimacy and broad dissemination. Since its creation in 1994, the Estado de la Nacin Project, today called Programa Estado de la Nacin, has made great efforts to drive and deepen the study and discussion on sustainable human development in Costa Rica. The annual preparation of the State of the Nation Report has been the basis to achieving that purpose. With the combination of research processes and consultations with representatives from diverse social sectors, networks of relationships have been established to promote a sense of belonging and legitimacy in the analyses, as well as links to facilitate dissemination and access to information obtained through them. Since 1994, the annual publication of the State of the Nation Report has managed to generate different currents of public opinion on the countrys sustainable human development and to place topics that had been relegated or

13

The Consejo Nacional de Rectores (CONARE) is the body entrusted with coordinating State University Higher Education. The Higher Education Planning Office (OPES) is the technical and advisory arm of the CONARE. Both bodies were created under the Coordination Agreement for State University Higher Education in Costa Rica, signed on 4 December 1974. The CONARE is made up of rectors from the University of Costa Rica, the Technological Institute of Costa Rica, the National University and the State University for Distance Learning.

54

absent on the national agenda. A case in point is that of highway and transport infrastructure, (a topic analysed in reports 1, 3, 7, 9, 10, 11 and 12), quality of employment (4 and 8) or special chapters in the third, fourth, sixth and seventh reports which made progress in analysing the gaps in sustainable human development at the subnational level, in rural zones and in gender. The report has also become a reference point for those who concern themselves with the development and management of public policy (academics and university students, political parties, public institutions, civil society organisations, media) who value it as a useful input for their work. Guatemala 16. CEES Centro de Estudios Econmicos y Sociales http://www.cees.org.gt The Centre for Economic and Social Studies was created on 18 November 1959 to study and disseminate ideas on freedom. Since then it has worked on the promotion and safeguarding of respect for the fundamental rights of the person. The CEES is a private, cultural, academic, apolitical, non-religious and non-profit body which believes that ideally the inhabitants of Guatemala should live and defend the fundamentals of freedom and the rule of law, in which the respect for individual freedom to produce, consume, exchange and serve, without coercion or privileges, is a priority. So, the efforts of CEES are directed to achieving an ever-freer world and a Guatemala which is increasingly inspired by those values and integrated into that world. El Salvador 17. FUSADES Fundacin Salvadorea para El Desarrollo Econmico y Social www.fusades.org The Salvadorian Foundation for Economic and Social Development is a nonprofit private development organisation, created in 1983 by a group of independent and professional entrepreneurs, with the vision of improving the economic and social conditions of Salvadorians. The foundations work strategy consists in promoting public policy proposals in the economic, social, environmental and institutional fields, involving Salvadorian society. The fundamental objective of Fusades is to develop all kinds of activities designed to foster security and the economic, social, intellectual and physical well-being of the inhabitants of El Salvador, under the guidelines of a system of economic and individual freedoms. Among the main activities carried out by the foundation to achieve its goals are the following: a) Systematic and periodic study of the economic and social situation. b) Formulating and promoting the execution of specific programs to help solve the problems of development.
55

c) Promoting national and foreign capital investment in all kinds of production activities. d) Creating awareness in the private sector as to national problems and their possible solutions. e) Carrying out activities to incorporate different population groups into the production effort for their economic and social betterment. f) Fostering development of the free enterprise system. In 1985, the foundation presented the first economic diagnosis, which led to the drawing up of the first five-year strategy for economic and social development. In the book, Fusades indicated that the economic model based on protectionism which existed at that time had run out of steam. Urgent change was needed to open the country to a free market economy model. Based on this diagnosis, the foundation began to draw up five-year strategies which included public policy proposals to secure the economic, social and institutional development of the country. This public policy has been put to society in general, so that it can be reconsidered and set in motion in a period of five years.

56

ANNEX N 3: Data collection instrument: semi-structured interview A) Demand/exogenous variables/environment I. Rules of the game (institutional framework in which PRIs operate) This section is a small social, political and economic contextualisation of the countries where the PRIs studied reside. Attention will be placed on the particularities of the institutional contexts in which PRIs operate, which have to do with legal questions, opening up to political participation, degree of government capacity, or with those that emerge from the cases. II. Demand to M&E research impact In the PRI to which you belong, is there M&E of the impact that projects have on public policy? Why? Which factors do you believe explain that situation (carrying out impact M&E, or otherwise)? What or who encourages the PRI to carry out this type of M&E? Was it under your own initiative, a requirement (or recommendation) of your donors or pressure from other organisations involved in the projects? If it was your own initiative, from which unit or area did it come? If it was a requirement (or recommendation) of your donors, which organisation was responsible? If they were other organisations, which ones were they? Can you say that the impact M&E is a condition of funding? How often? III. Web of organisations involved in PRI projects (This dimension is closely linked with B) VI) Complementary organisations Do any of the other organisations that participate in your projects carry out impact M&E? How do they do it? Do they not demand (directly or indirectly) that you do it too? Do you know of other PRIs in the region that effectively carry out impact M&E? Which ones? Why do you think they do it? How would you describe them in brief (main characteristics)? Does your PRI have any kind of relationship with them? Has it exchanged experiences or knowledge of impact M&E with them?

B) Supply/endogenous variables/internal-organisational capacities

57

I.

Internal organisation and distribution of functions (structure) / Organisational government

Purposes and goals of the PRI: strategic planning Which are the main goals of your PRI? Is impact one of them? Does your PRI carry out annual strategic planning of your projects and activities? Is the plan embodied in any sort of document? Are objectives and goals set in the plan? Purposes and goals of the projects: contribution to the PRIs strategic plan What is the degree of congruence with the plan required by the projects? Is the contribution made by the projects to the purposes and goals of the annual plan evaluated, especially those relating to impact? Are the goals of the projects carried out by your PRI explained in detail? Is each and every one of the projects put down on paper? Are impact goals included in them? Why? Organisational structure, vertical and horizontal relations How is your PRI organised (how is it structured)? How many areas has your PRI? Is there a specific area responsible for impact M&E? Which one? What other responsibilities/functions does it have? Why was it created? To whom does that area answer? What role does the academic council play regarding impact M&E? What importance has impact M&E in the renewal of projects or in an increase in funding? Is competition generated between the different projects? II. Financial and physical capacities / Funding

What was the approximate annual budget last year? What are the PRIs sources of funding? In what proportions? Could you establish any relationship between budgetary and material shortcomings, on the one hand, and absence or presence of impact M&E, on the other? Is there any particular shortcoming you could relate to impact M&E? III. Personnel policies and rewards systems (remuneration)

How many people work permanently in the PRI? On which specific projects? Is impact M&E promoted or encouraged by the PRIs directors? Why? How? IV. Individual qualifications and management capacities of those who work in the agencies with responsibility in the area/ Human resources
58

Human resources profile Which areas do your human resources come from? Where did they train? What is their origin/professional background? Are you able to associate impact M&E practices with any particular profession or profile among your human resources? Methodologies of impact M&E Do you use any special methodology in formulating and implementing projects? Which one? Why? Was it requested, chosen, what motivated it? Do you use the same methodology for all projects or do they vary depending on the topics, donors, counterparts, etc.? Do you construct or define M&E indicators when formulating projects? Do you include impact indicators among them? Why? Which ones are they? Which is the source of these indicators: did you construct them yourselves, or were they recommended by counterparts, by peers, etc.? V. Research management and M&E

Impact M&E as feedback from the PRI To what extent do you consider the results and impacts of the finished projects when deciding to undertake new projects or to improve their impact? Does the PRI have procedures or a system to supply information and make it available (information infrastructure)? Which ones are they? Themes for research Which are the main lines of research of the PRI? How long have you been working on them? Could you say that it is more feasible to carry out impact M&E in projects that deal with some topic in particular? Which ones? Why do you think it happens? Stages in the public policy process At which moment of the public policy process do most of your projects intervene? Is impact M&E carried out indistinctly of the stages of the policy process, or is it carried out with greater frequency in any stage in particular? Which one? Why? VI. Institutional communication and other offerors

Institutional communication
59

When you carried out M&E of effective impact, did you make the results of that M&E public? Why? Do you inform your donors, or other organisations involved, of the results of impact M&E, even though they were not as you, your donors or the other organisations had expected? Has carrying out impact M&E brought you benefits in your relationship with donor organisations? And with the beneficiaries?

60

ANNEX N4: List of interviewees 1) CORDES Corporacin de Estudios para el Desarrollo 2) Consorcio de Investigacin Econmica y Social CIES 3) Centro Boliviano de Estudios Multidisciplinarios 4) Expansiva de la UDP 5) Fundacin IDEA 6) CEGA Corporacin de Estudios Ganaderos y Agrcolas 7) Fundao Getulio Vargas 8) Programa Estado de la Nacin 9) FUSADES Fundacin Salvadorea para El Desarrollo Econmico y Social 10) Centro de Investigacin y Docencia Econmicas CIDE 11) Fundacin para la Educacin Superior y el Desarrollo 12) Centro Latinoamericano de Economa Humana (CLAEH) 13) Centro de Estudios de Estado y Sociedad (CEDES) 14) IIERAL Fundacin Mediterrnea 15) CEDICE 16) CEES Centro de Estudios Econmicos y Sociales 17) CIEPLAN Corporacin de Estudios para Latinoamrica

61

BIBLIOGRAPHY AND DOCUMENTS CONSULTED Abelson, D. (2002) Do Think Tanks Matter? Assessing the Impact of Public Policy Institutes. McGill-Queens University Press, Quebec, Canada Abelson, D. (2006) A Capitol Idea: Think Tanks and US Foreign Policy, McGill-Queens University Press. Acua, Carlos (2009), Espacios para el Compromiso: Usando el Conocimiento para Mejorar las Polticas Pblicas en Favor de los Pobres. Anlisis comparativo de cuatro estudios de caso sobre Institutos de Investigacin de Polticas (o think tanks) en Mxico, Brasil, Ecuador y Uruguay, Informe Final, GDN-CIPPEC Adamo (2002) Strategic Evaluation of Policy Influence: What Evaluation Reports Tell Us About Public Policy Influence by IDRC-Supported Research. Prepared for the Evaluation Unit, IDRC, Abra Adamo 30 April 2002.pp 16). Bachrach P. et M. Baratz (1962), Two faces of power, American Political Science Review, 56 (4), p. 947-952 Banco Mundial (2002) El desarrollo de la capacidad de evaluacin: una prioridad cada vez ms importante, Departamento de Evaluacin de Operaciones del Banco Mundial, Autumn 2002, Number 29. Banco Mundial (2004) Seguimiento y Evaluacin. Instrumentos, mtodos y enfoques, Banco Internacional de Reconstruccin y Fomento/BM. Braun, M., Chudnovsky, M., Ducot, N. & V. Weyrauch (2006) Lejos de Thinktanklandia: los institutos de investigacin de polticas en los pases en desarrollo. In A. Garc & G. Ua (comps.), Think tanks y polticas pblicas en Latinoamrica. Dinmicas globales y realidades regionales. Prometeo, Buenos Aires. Braun, M., Chudnovsky, M., Ducot, N. & V. Weyrauch (n/d) A Comparative Study of Think Tanks in Developing Countries, mimeo CIPPEC, Buenos Aires. Brunner, J. J. (1996) Investigacin social y decisiones polticas: El mercado del conocimiento. In Nueva Sociedad, 146, Caracas, Venezuela. Carden, Fred, Stephanie Neilson with Terry Smutylo, Denise Deby, Sarah Earl, Bryon Gillespie (2002) IDRC-supported research in the public policy process: A strategic evaluation of the influence of research on public policy, Evaluation Unit, IDRC: Ottawa. Carden, F. (2003) Issues in Assessing the Policy Influence of Research. Draft Article for the International Social Science Journal. International Development Research Centre (IDRC), Ottawa, Canada. Carden, Fred (2004) Issues in assessing the policy influence of research, UNESCO. Camou, A. (2006) El Saber detrs del Trono. Intelectuales-expertos, tanques de pensamiento y polticas econmicas en la Argentina democrtica (1985 2001). In A. Garc & G. Ua (comps.), Think tanks y
62

polticas pblicas en Latinoamrica. Dinmicas globales y realidades regionales. Prometeo, Buenos Aires. Camou, Antonio (2005), Un toque de atencin o una tribuna de doctrina? Expertos econmicos y polticas econmicas en la Argentina a travs de la prensa especializada (1985-2001), presentation to the VII Congreso Nacional de Ciencia Poltica, Sociedad Argentina de Anlisis Poltico (SAAP) - Universidad Catlica de Crdoba Dunn, William (1996), A Look Inside Think Tanks, CIPE, Economic Reform Today, Number 3 Garc, Adolfo (2006) Una interface estrecha e inestable. Think Tnaks y partidos polticos en Uruguay. In A. Garc & G. Ua (comps.), Think tanks y polticas pblicas en Latinoamrica. Dinmicas globales y realidades regionales. Prometeo, Buenos Aires. Hanney, S., Miguel A Gonzalez-Block, Martin J Buxton and Maurice Kogan (2003) The utilisation of health research in policy-making: concepts, examples and methods of assessment, Health Research Policy and Systems 1:2. IDRC (2003) Evaluation Seminar on the Influence of Research on Policy in Ukraine, Ottawa, Canada, March 26, 2003. IDRC (2003) Annual Report on Evaluation Findings 2003, Evaluation UnitInternational Development Research Centre. IDRC (2004) Annual Report on Evaluation Findings 2004, Evaluation UnitJones, H. and Hearn, S. (2009) Outcome Mapping: a realistic alternative for planning, monitoring and evaluation, Background Note, ODI: UK. Jones, N., Harry Jones, Liesbet Steer and Ajoy Datta (2009) Improving impact evaluation production and use, Working Paper 300, ODI: UK. Kuruvilla, Shyama (2005) CSO Participation in Health, Research and Policy: A review of models, mechanisms and measures, Working Paper 251, Overseas Development Institute: UK. Lardone, Martin y Luciano Donadi (2007), Construyendo consenso. La modernizacin del Estado y los mecanismos de articulacin entre BMDs y Think Tanks en Argentina, in A. Garc & G. Ua (comps.), Think tanks y polticas pblicas en Latinoamrica. Dinmicas globales y realidades regionales, Editorial Prometeo, Buenos Aires Lardone, Martin (s/d), Prter ou influencer. Banques Multilatrales Dveloppement et Relations Intergouvernementales en Argentine et le Chili, Thse Doctorale, Institut dEtudes Politiques de Paris Lindquist (2001) Discerning Policy Influence: Framework for a Strategic Evaluation of IDRC-Supported Research, School of Public Administration University of Victoria, September 1st, 2001. Lusthaus, Charles, Gary Anderson, and Elaine Murphy (1995) Institutional Assessment. A Framework for Strengthening Organizational Capacity for IDRC's Research Partners, IDRC: Ottawa. Lusthaus Charles, Marie-Hlne Adrien, Gary Anderson y Fred Carden (2001) Mejorando el desempeo de las organizaciones. Mtodo de autoevaluacin, IDRC: Ottawa.
63

Lukes, Steven (1974), Power. A radical view, Macmillan, London. Mackay, Keith (1999) Fortalecimiento para la Capacidad de Evaluacin. Gua para el diagnstico y marco para la accin, Document N 6 in the series of working documents on strengthening evaluation capacity, January 1999, Operations Evaluation Department of the World Bank. Operations Evaluations Department (2004) Evaluation Capacity Development: OED Self-Evaluation, BM. Pollard, A. and Court, J. (2005) How Civil Society Organisations Use Evidence to Influence Policy Processes: A literature review. Overseas Development Institute, Working Paper 249, London, UK. Puryear, Jeffrey (1994), Thinking Politics. Intellectuals and Democracy in Chile, 1973-1988, The Johns Hopkins University Press, Baltimore. Roggero, Marcos (2009) Polticas Pblicas y Largo Plazo. El programa social mexicano Progresa-Oportunidades, EDUCC: Crdoba. Saumier, Andr (2003) The impact of research on public policy IDRCs Programs in Vietnam, IDRC: Ottawa. Start, D. y Ingie Hovland (2004) Herramientas para el Impacto en las Polticas Pblicas: Manual para Investigadores, Instituto de Desarrollo en Pases Extranjeros 2004: UK. Stone, D. (1996) Capturing the Political Imagination: Think Tanks and the Policy Process. Frank Cass & Co., London, UK. Thompson, A. (1994) "TTs" en la Argentina. Conocimiento, instituciones y poltica. Centro de Estudios de Estado y Sociedad (CEDES), Buenos Aires, Argentina. Tuplin, Tracy (203) The Influence of Research on Policy: The case of MIMAP Senegal, IDRC: Ottawa. World Bank Independent Evaluation Group and the Thematic Group for Poverty Analysis, Monitoring and Impact Evaluation Evaluation Capacity Development (2009) Institutionalizing impact evaluation within the framework of a monitoring and evaluation system, Independent Evaluation Group - The International Bank for Reconstruction and Development/The World Bank. Zaltsman, Ariel (2006) Experiencia con la institucionalizacin de sistemas de monitoreo y evaluacin en cinco pases latinoamericanos: Argentina, Chile, Colombia, Costa Rica y Uruguay, Desarrollo de Capacidades de Evaluacin (DCE) Series of working documents N 16, Independent Evaluation Group, WB. Zamora Serrano, Evelin (2006) Gua para la Evaluacin del Impacto de Internet en las Organizaciones, Fundacin Acceso: Costa Rica. Zuvanic, Laura and Mercedes Iacovello (2005), Institucionalizacin y Burocracia en Amrica Latina. 10th International Congress of the CLAD on the Reform of the State and Public Administration. Santiago, Chile.

64