Workshop: Theory of Change & Assessing Social Impact Friday, 10 May 2013 Speaker: Lance Potter, Director of Evaluation,

New Profit Summary: This workshop walked through elements of a Theory of Change model, the process to break down the details, and the application and value of his approach. Theory of Change is not impact measurement rather it is a tool for planning, clarifying goals, and maybe help identify the measurement methods to use. This tool helps define what to achieve, how to achieve it, and how to measure. Walking through the process for developing a theory of change is helpful for project and programme planning. It helps flush out goals into tangible details; it can be used to forecast behaviours at micro and macro levels; and it is a roadmap that includes specifying and testing assumptions/ hypotheses in the programmes approach. Where a log frame ties the inputs/ resources, the program output, and outcomes, it does not require laying out pre-conditions required for success and baseline useful for measurement as theory of change may. Many programmes will often deal with multiple theories of change. It begins with setting a target what you intend to change, along with the importance of defining indicators that will suggest if the progress is for the better or worse. Lance drew a distinction between outcome and impact, saying that while impact does not have a strict definition, one can think of it as something that is long-term and sustained. Lance provided an example of an overview process, saying that a theory can be built around the whole ecosystem with multiple players. The bottom tier (a, b, c) helped define countries to spell out their measurements (tie to these goals and guidelines). This may lead to deal breakers, and end the partnerships that don't make it pass the baseline condition. In another example, he spoke of common problems. The first is that several terms are too poor to be acted on and measured: "all children", "quality opportunity", and "success" are such examples. While terms such as these are sufficient for a vision statement, they are not enough for a theory of change. Second, the indicators a project proposes may point to grant-making, but not the focus of the impact they are looking for (around children). To illustrate his point about a good example, he talked about the importance of knowing how to draw a diagram with the use of nouns in boxes and verbs alongside arrows. Second, he talked of the need to start with simple first steps that are critical for instance, the action enrolment leads to a series of potential paths. Third, it is important to explain assumptions, with references to interventions required along the path. Fourth, some assumptions that need to hold throughout the programme are referenced closer to the end goal. Fifth, not every box is measured, but it should identify what is success for each box. Lastly, one should look for biases.

The workshop also contained a group exercise, the highlights of which were the following: 1. Mapping process enables prediction of other behaviours. It can also clarify boundaries of what one can take credit for and can take responsibility for. That can help one clarify the long term outcome. 2. Identifying lagging indicators of the boxes (by # , %) 3. To achieve a long term goal, much of the activities may be focusing on a condition (or improving a leading indicator). For example, one group focused on developing and filling middle management to build an organization's ability to scale. 4. For changing normative beliefs, the challenge is the tightness of the relationship between mindset change and the organization's indicators. It is hard to measure because of how people change their mind. It may be helpful to simplify, segment and target the stakeholders they work with without touching the public mindset (for example, measuring industry activities/ market size and shifts). 5. Next: Consider the conditions needed to start each step. What does it take? At what scale? (Define and build the preconditions) The workshop drew several questions from the participants. In response to a question about the point at which an organisation plugs in its stakeholder/ landscape analysis, Lance said one has to go through the theory of change map first, then identify the other players. It can help identify indicators where one wants to monitor the other players. When asked how much an organisation would change its theory/ plan as the programme plays out, he said that many a time, this exercise happens while a programme has started. Teams bring in data, compare the plan with the field as they talk, plan against what they can/cannot influence, and plan against what they do/don't believe is possible. One of the panellists asked him whether the purpose of the theory was to see the macro environment, to which he replied that from a planning perspective, there are only so many ways one can do it. The intervention lines may give rise to more questions and more iterations. In response to a question on how assumptions can be identified, Lance said one exercise is to do a pre-mortem, where one can envision failure down the line and predict factors that failed and caused the programmes failure. Those factors then become the factors that will be assumed to be true for success. For example, programmes often make cultural assumptions, or are not able to identify them, which lead to failure. One participant asked Lance what some limitations or blind spots are in designing a theory or model. He replied that unintended effects are common, and one hopes that this process helps mitigate the likelihood of that. He said it is sometimes challenging to believe the theory enough to abide by it closely and work by it fully. He also said that a well-designed theory of change is a good tool for an evaluator to use as a base for evaluation.