WORKING PAPER TOWARDS THE EVALUATION

APPROACH OF TCBL
December 2015

Joe Cullen, Kerstin Junge, Giorgia Iacopini

Co#funded)by)
Horizon)2020

D 8.2: TCBL Website

646133 - TCBL
Textile & Clothing Business Labs

TABLE OF CONTENTS

TABLE OF CONTENTS ............................................................................................................ 2)

1.)

INTRODUCTION ................................................................................................................ 3)

2.)

ABOUT TCBL .................................................................................................................... 3)

2.1)

BACKGROUND AND CONTEXT: THE EUROPEAN TEXTILES AND CLOTHING SECTOR ............. 3)

2.2 KEY AIMS AND ACTIVITIES OF TCBL ..................................................................................... 5)

2.2)
3.)

KEY FEATURES OF TCBL .............................................................................................. 7)

THE APPROACH TO EVALUATING TCBL ..................................................................... 8)

3.2)

THE PURPOSES OF THE TCBL EVALUATION ................................................................... 9)

3.2 REFINING THE EVALUATION CONCEPT................................................................................ 13)

3.2.1 An Ecosystems perspective on Evaluation ............................................................. 13)
3.2.1 Evaluating Complexity............................................................................................. 19)
3.2.1 Large Scale Change ............................................................................................... 23)
3.3 SUMMARY AND IMPLICATIONS ............................................................................................ 26)
4. THE EVALUATION DESIGN ............................................................................................... 28)
4.1 EVALUATION AUDIENCE AND QUESTIONS ............................................................................ 28)
4.1.1 Evaluation Audiences and their information needs ................................................. 28)
4.1.2 Evaluation Questions .............................................................................................. 31)
4.2 EVALUATION METHODOLOGY ............................................................................................. 37)
4.3 EVALUATION METHODS ...................................................................................................... 41)
5. TOWARDS A BASELINE TCBL THEORY OF CHANGE................................................... 43)
6. NEXT STEPS: IMPLEMENTING THE EVALUATION ........................................................ 49)
BIBLIOGRAPHY...................................................................................................................... 50)

1. INTRODUCTION
This working paper presents the core ideas for the evaluation of TCBL as a key milestone
towards producing D6.3 - Evaluation Approach - in Month 9. The Paper is based on and
integrates the following information sources:

information on TCBL from the DOA, the TCBL website (in particular the working group
discussions),
the results of two kick-off meetings,
a literature review carried out by the evaluation team on key relevant concepts
the first round of project partner interviews held in November 2015.

In doing so, it elaborates on the early thinking on the evaluation design presented to partners
in the Prato and Brussels meetings.
The document has three main purposes:

To bring together the results from the evaluation scoping work carried out thus far;
To develop a conceptual framework for the evaluation design for TCBL as a concrete
step towards the production of D 6.3;
To share these ideas with consortium partners and invite feedback that can be used to
a) further advance the evaluation approach, and b) to use some of the content of this
paper as a basis for a collaborative exercise on evaluation during the January 2016
partner meeting.

This document is therefore structured as follows:

Chapter 2 sets the scene for the evaluation design. It presents the background and
context to the TCBL project as well as a high level overview of its key aims and
activities. From this, it extracts the key features of TCBL as relevant for the evaluation
task.
Chapter 3 then takes this information to present an outline of the evaluation approach,
focusing on the conceptual foundations of the evaluation as well as evaluation
purposes.
Chapter 4 outlines the audiences for the evaluation as well as the evaluation
questions, and the core data collection methods that will be used.
Chapter 5 presents a baseline theory of change for TCBL, both in diagram and
narrative format. This will be further elaborated with partners during the January 2016
coordination meeting.
The concluding Chapter 6 outlines the activities that will be carried out to produce the
first evaluation deliverable and the partner input required.

2. ABOUT TCBL
2.1 BACKGROUND AND CONTEXT: THE EUROPEAN TEXTILES
AND CLOTHING SECTOR
In line with many established sectors, the Textile & Clothing industry (T&C) reflects a long
declining trend in employment, export and production capacity, driven by globalisation and
3

646133 - TCBL
Textile & Clothing Business Labs
recession. However, it still plays a crucial role for the EUs economy and social well-being. In
2010, the 127,000 active T&C companies in the EU had an overall turnover of 172 billion
Euros, employed over 1.9 million workers and represented investments of around 5 billion
1
Euros. A key characteristics of the sector is that is shows a negative trade balance, with 34
billion Euros of textile and clothing products exported and 84 billion Euros imported from Third
markets in 2010. But if the landscape is broadened to include the entire fashion industry, and
high-end products and the non-clothing applications of textiles (where Europe is a worldwide
leader), employment goes up to over 5 million persons, which is equivalent to 3.7% of the total
non-financial business economy (in 2009), according to Eurostat.
Over the past two decades, the T&C industry has developed three main strategies to meet the
global competitive pressure:

A retention strategy for price-sensitive mass-market customers, relying on a cost-oriented

approach, which used relocation to low-income countries, including the New Member
States, as its major instrument;

A product/service-oriented approach, diversifying the spectrum towards high-quality and

specialty textile products;

A productivity-oriented approach, based on automation and IT-based supply chain management, helping improve flexibility and create global sourcing systems.
In recent years, the T&C sector has increasingly looked towards applying research to
stimulate innovation, including the setting up of a Technology Platform (http://www.textileplatform.eu/) to meet the requirements of FP7. The T&C sector is also highlighted in the EC
Communication For a New Industrial Renaissance of January 2014, which calls for a
movement towards innovative, high-added value products; taking full benefit from market
liberalisation, notably via Free Trade Agreements; addressing delocalisation by placing more
emphasis on creative and high-end products manufactured in the EU; protecting European
fashion and high-end companies against counterfeiting; reducing the gap between longer
financial cycles and production cycles; retaining traditional skills and know-how; improving
consumer trust in online shopping.

The above elements are reflected in an Action Plan for the fashion and high-end industries
presented by Commissioner Tajani on 3 December 2013 in London. Although the Action Plan
is intended to enhance the competitiveness of the T&C sector, the TCBL approach argues that
the Plan will only be implemented effectively if it is supported by business innovation in order
to effectively reverse the system of incentives that has supported industrys restructuring processes over the past two decades. This innovation particularly needs to focus on three areas:
taking advantage of the emerging opportunities from the new Making Economy (e.g. personal robotics, home production, etc.); redirecting the capacities of old artisans and family workers or fasonists and re-connecting their knowledge with e.g. new, young, creative people;
taking full advantage of the benefits of Future Internet technologies for the T&C global supply
chain (diffused ecommerce networks, IoT tracking systems, virtual warehouses, customer engagement, etc.) in the light of a new customer-driven approach based on market intelligence.

The European Apparel and Textile Confederation

646133 - TCBL
Textile & Clothing Business Labs

2.2 KEY AIMS AND ACTIVITIES OF TCBL

Against this background, the Textiles and Clothing Business Labs (TCBL) project, funded by
the European Unions Horizon 2020 programme, aims to transform the European textiles and
clothing industry from a reliance on conventional (price driven) manufacturing to one that is
more socially and environmentally sustainable (e.g. by using more agile / disruptive and
customer driven approaches to production). It aims to do this by encouraging real world
experimentation across locations in the eight countries of the TCBL partnership and beyond
- via the vehicle of a network of business labs and a number of business pilots (systems).
Business labs are spaces in which actors involved in TCBL can draw on existing and
emerging models to freely experiment with new ways of designing, making, producing within
specific locations in the countries covered by the TCBL partnership (eight EU member states).
They are set up to explore future business scenarios based on emergent societal and
technological trends and driven by new market trends. TCBL includes three types of labs:

Design Labs (immaterial value, emotion-oriented) explore tools and methods for
designing textiles and clothes, working with professionals, fashion students, or
anyone, even working from home;
Making Labs (material value, labour-oriented) experiment with production methods
and machinery old and new, from re-discovering traditional tailoring to 3D printing and
laser cutting;
Place Labs (spatial value, community and socially-oriented) investigate the local and
social dimensions of clothes making, with new modes of organisation of work such as
on-demand or home DIY production, community lab spaces, and networks of artisan
shops.

The essential purpose of these labs is to produce and transfer knowledge and innovation into
Business Systems, motivating potential pilots to emerge. There will be 15 start-up labs, run by
existing project partners as an extension of their existing activities. Examples of labs include:
exploring new ways for design (co-design, open design, etc); innovative equipment for cutting
and sewing (make lab); the re-valorisation of artisan knowledge on a local level (place lab).
This will be scaled up to 45 labs in the network via open calls starting in year 3 of the project.
These laboratories interact with a substantial number of sector enterprises of various
dimensions pilot businesses who compose innovation elements coming from different
Business Labs to identify transition scenarios that can accompany their shift from current ways
of working towards more innovative and competitive business models. These business
systems (pilots) are essentially established systems and processes that embody existing and
concrete supply and value chains which will provide the infrastructure and processes for
innovation transfer of business model elements from the Business Labs. These systems will
work in two types of activities within TCBL:

Pilot laboratories (small existing structures) will pilot some small scale activities and
productions, with an emphasis on developing social innovations that have an impact
on the local community.
Pilot factories involve bigger units (factories) that want to experiment with existing
innovations or new processes (e.g. reorganisation of working positions, multi-tasking).

The pilots are part of the innovation ecosystem of TCBL services but are external to the
TCBL partnership with their engagement entirely driven by the value proposition of TCBL and
methods of engagement (DOA, p. 44). The aim is to set up 160 pilots, 100 recruited at a later
stage through an open call.
5

646133 - TCBL
Textile & Clothing Business Labs
This process is supported by interactive knowledge and learning services together with an
open repertoire of business services supporting specific moments of the new business models
(training, logistics, etc).
Knowledge spaces capture knowledge about the textile and clothing system particularly
(innovative) practices of production and including information on what works that can be used
to support networking and learning. They are an online, interactive business model repository
that hosts and links to embedded and emergent materials and manufacturing knowledge for
T&C, as well as market, technology, economic and social trend observations and policy
watching (DOA p. 38). The knowledge spaces are to trigger and support learning of its
stakeholders about business models, resources and opportunities.
The Business Process Support Services support the satisfaction of market needs by
supporting business labs and business systems in accessing, assimilating and adapting knew
knowledge and innovative business processes to enable new ways of working in textiles and
clothing to be developed and implemented. These business process services will utilise and
valorise the evolving knowledge created through the Knowledge Spaces, as well as the
additional services brought in by the Associate Service SMEs, for example providers of ICT
services to the TCBL ecosystem (estimated to be around 60 at projects end and 100 after 5
years).
Figure 1 below shows how these core components of the TCBL project link together:
Figure 1: TCBL conceptual scheme (DOA p. 10)

The interactive and creative processes in TCBL aim to gradually build an integrated business
ecosystem covering the entire value chain. The initial configuration in project year 2 will
consist of 15 Business Labs and 60 Pilot Businesses, while by the projects end this will grow
to 90 Labs, 240 Pilots, 35 service enterprises, and 15 start-ups. This exponential path of
ecosystem growth will be sustained by yearly Calls for Expression of Interest in the TCBL
Associates Programme, which will select new members to receive specific support and

646133 - TCBL
Textile & Clothing Business Labs
assistance from project partners. The box below shows the key impacts that TCBL is expected
to achieve:
Box 1: TCBL anticipated outcomes and impacts

5 per cent increase in manufacturing capacity within five years after the end of the
project
Reduction in the environmental footprint by 20 per cent compared to products
produced in the traditional value chains through less stock, less waste and less
transportation
Creation of a novel supply network involving at least 100 organisations and
individuals at the end of the project and 1000 organisations and individuals within 5 years
after the end of the project.
Creation of new embedded services supporting the customer driven supply chain

TCBL is thus expected to have a widespread impact on the T&C industry in Europe, shifting
consumer goals, expectations, and even engagement in the processes of designing and
making clothes. This in turn will have both social and environmental impacts, as well as
influencing attitudes towards responsible fashion and significantly improving the prosperity of
Europes diffused system of production.

2.2 KEY FEATURES OF TCBL

From the description of the TCBL project above we can extract a number of key
characteristics that are embedded in the project design and that have relevance for choosing a
suitable evaluation approach. These are:

Emergence. The project is designed to be implemented in four phases: set-up (where

aspects of the projects architecture are being made more specific); internal pilot
cycle (where the idea of labs will be tested by actors internal to the TCBL consortium;
engagement and impact (where the ecosystem is widened to include external actors
as locations for labs and pilot activities); scaling up and out. Before each phase starts,
transition times have been built into the project, allowing for the shaping / design of
this phase and the taking on board learning from the preceding phase. These
transition phases are the focus of project activities and partner commitments (DOA, p.
34). This emergence will shape the TCBL ecosystem itself in unpredictable ways, not
only because we cannot foresee what kind of organisations will join (and what
innovative transition paths they will bring) but also because through their work they will
shape the business services and knowledge spaces themselves the content
contributed, and potentially the services offered to pilots.
TCBL has therefore been designed as an open system. The ecosystem [metaphor]
used in the DOA to describe the TCBL construct, together with the emergence
concept described above, evokes the image of a living and breathing organism in
which the different components (consortium partners, labs, associate pilots,
stakeholders, technologies,) grow and interact, both with each other and
(potentially) the wider social and ecological environment, to change the textiles and
clothing industry.

646133 - TCBL
Textile & Clothing Business Labs

TCBL is thus an innovative, if not experimental, project : it creates an ecosystem

that, through networking, aggregates shared knowledge, innovation, creativity and
business innovations; this ecosystem combines innovation trends so that they may
have an aggregate systemic impact on the T&C sector; uses technology as a tool for
this change (rather than imposing it as a driver of change); enhancing existing
services to textiles and clothing businesses. To paraphrase one partner interviewed:
this is a new approach to pave the way for a different way of doing business, which
has not really been done before. To paraphrase another interviewee: TCBL takes a
whole industrial sector and messes the whole thing up, sees how the pieces fall and
then re-aligns them.
Diversity of experience. TCBL can draw on significant experience and expertise
through the composition of the partnership which is not only diverse (combining
academics, leading industry actors and technology experts) but has, in parts, worked
together before on projects with related topics. This diversity may produce innovation
itself, as well as offer a valuable backdrop to the implementation of the project (in the
absence of another template or recipe for carrying out this kind of at scale innovation
successfully).

Within the framework of activities described in the DOA and the broad impacts to be achieved,
TCBL as an intervention into the textiles and clothing system is thus characterised by a
significant degree of uncertainty of interim outcomes and causal pathways of how these and
final impacts will be achieved. It is a project / intervention that is organic shaping itself and
being shaped by the environment with which it interacts. This makes TCBL a complex
3
intervention , and the approach to the evaluation needs to be able to account for these key
features. This is discussed further in the next section.

3. THE APPROACH TO EVALUATING TCBL

Evaluation, broadly speaking, is a process of making a judgement on an intervention. It goes
beyond monitoring (which usually involves the collection of information to track the progress of
an intervention against targets and can also provide data on spend and outputs) in that it
assesses process, outcomes and impacts of an intervention in relation to its intended aims.
Evaluations tend to be done for any or all of the following purposes: accountability (to make
sure money is spent appropriately); learning (about what works); or knowledge generation
about effectiveness for the future.
Depending on the kind of intervention that is to be evaluated, the purpose of the evaluation
and the kind of question(s) to be answered, an evaluation will typically be based on one of the
following strategies:

A before and after design, where the only purpose is accountability and evaluation
questions seek to measure the extent to which predicted outcomes have been
achieved. This only works in simple interventions where cause and effect

From Consortium interviews, November 2015.

Rogers, P Using Programme Theory to Evaluate Complicated and Complex Aspects of

Interventions Evaluation, Vol 14 No 1, p 31

646133 - TCBL
Textile & Clothing Business Labs

relationships are easily detectable (e.g. the effect of speed cameras) as essentially
what is involved is a comparison of monitoring information from before the
intervention to the state after the introduction of an intervention to understand the
scale of change achieved.
Experimental designs specifically test causal relationships and answer questions
about whether the intervention has achieved the anticipated results. To this end, the
evaluator compares two groups of people or situations: one that received a
treatment (for example a new drug), another that did not (the control group or
situation). Differences in outcomes between the two groups/situations are attributed
to the intervention. Using this approach requires: a highly specified intervention that is
implemented identically in different situations or locations; a high level of existing
knowledge about likely cause and effects; ideally, randomisation of the treatment and
control groups; and ideally a relatively short implementation period.
Theory based evaluations focus on exploring whether an intervention has worked
and on explaining why and under what conditions change has been achieved in
different contexts. They are particularly strong for evaluations of very diverse and
long term interventions that display a mix of activities, target groups, delivery
mechanisms and settings (i.e. interventions that are very complex). That is, they tend
to be used in situations where the above approaches would not deliver reliable results
because impact pathways are not straightforward (e.g. an intervention contains
multiple strands of activities that may or may not interact), a number of external (or
intervening) variables may influence the results of an intervention or where there is
little prior knowledge about causality. In these instances, theory based evaluations
work by articulating the implicit and explicit change theory underpinning an
intervention (often represented in a theory of change or intervention logic model) and
interrogating this theory throughout the evaluation process. The focus is on improving
knowledge about causal pathways that lead towards anticipated outcomes and
impacts whilst also capturing unintended consequences of the intervention. Theorybased approaches also allow inferences to be made about the possible long term
impacts of an intervention.

It is clear from the short description above and the characterization of TCBL in the preceding
section, that the TCBL evaluation approach needs to follow a theory based design as the other
two designs are not suitable for the way the project has been conceptualised. In this context,
the following section discusses what are the key purposes of the TCBL evaluation.

3.2 THE PURPOSES OF THE TCBL EVALUATION

According to Task 6.3 - Process and Impact Evaluation - of the DOA, the remit of the TCBL
evaluation is as follows:
Box 2: TCBL evaluation task according to the DOA

This activity carries out ex ante, process and ex post evaluation of the innovation dynamics
amongst the different stakeholders of the TCBL ecosystem and their different goals,
objectives, and criteria of success. In the broader framework of the objectives of the project
(), it carries out the benchmark and process evaluation required to make a reliable
estimate of the longer term impacts of the project (p. 49)

646133 - TCBL
Textile & Clothing Business Labs
This will include: the ongoing collection and assessment of business model innovations, the
analysis of the manifest and hidden interdependencies between these innovations and the
broader sectorial, socio-economic and policy related aspects, and creation of a set of strategic
and operational guidelines for the various stakeholders involved (pp 18-19 DOA).
When asked what they see as the purpose of the TCBL evaluation, partners gave a range of
responses, speaking to all three evaluation purposes outlined above:

Accountability: understanding whether project goals are being achieved and to this
end monitor what partners are doing by tracking progress along KPIs on tangible
project outcomes and suggesting improvements / alignments to project delivery (and
avoid deviations).
Learning: from activities undertaken to influence and improve project delivery. This
includes helping the project partners understand how others outside the immediate
partnership see TCBL.
Knowledge generation: to create an evidence base about what works (business
models, what kind of companies join the ecosystem, how they work together) and to
generate and share best practices.

The Box 3 below presents the range of answers partners gave when asked about the purpose
of evaluating TCBL.
Box 3: Partner thoughts on the purpose of evaluating TCBL

Accountability is the TCBL grant spent well?

We need to understand what has been achieved, so we understand whether we achieve
goals, defining indicators to measure our performance very important.
Developing key performance indicators on tangible project outcomes and tracking progress
along those.
Find out whether what we think we are doing we are actually doing and do it efficiently.
Of course we need to know if we are going to reach our goals.
For example, it will be important to see if we have reached the certain number of factories that
we have wanted to reach.
At KPI level, evaluation of the different instruments that are aimed at achieving these
objectives. Degree to which all of these internally run efficiently and contribute to the overall
objectives of the project is something that deserves quite some evaluation. Look at KPIs and
make them more tangible. What do they mean? E.g. carbon footprint: what does it mean, how
do you get there.
Main purpose is to monitor what were doing and give us guidelines on solving some of the
issues that are not performing as well.
To provide us with information about how much we have deviated from some of our original
tasks in order to assess how we are doing and if necessary to go back and re-plan our path, in
line with what is really possible when confronted with reality. So this is the formative element
of the evaluation which has a very important role in monitoring of the project.
An embedded facility to support progress and alignment of activities
A mirror to be used as early warning for deviations, recurrent mistakes etc.
A management activity needs to inform the project Steering Committee about effectiveness
and potential issues.
10

646133 - TCBL
Textile & Clothing Business Labs

Learning what works and how can TCBL delivery be improved?

To feed back into and influence ongoing work.
Learning and reflecting. About questioning and understanding. Both in terms of working
together and what we are achieving. Its not much about measuring. Because if you start an
experiment, you dont really know how to measure. It is a creative process, it is not
measurable in the same way as other things, there are different outcomes than more standard
things. For example, failure in an experimentation is not necessarily a bad outcome.
To orient and guide us as a partnership and guide the activities within the project and then to
also help partners to implement the right activities and the expected results.
To help us understand if we have a common image, vision of what we are doing.
Together with this I think we need an evaluation of how we are perceived outside [externally]
because we might have the possibility of a situation of us having a good understanding
internally that all is going well but in some distance outside actors are thinking of us differently.
When I say outside I mean pilots in TCBL and those who are not involved but who are part of
our audience in terms of dissemination and external comms. So, industry associations; big
industry players, policy-makers at regional and national and EU level. This will be important so
that we can correctly handle external comms to be more convincing about the worth of what
we are doing and so we can organise accordingly.
Needs to provide evidence-based advice on the different choices that are available, and which
choices are likely to work best.

Knowledge generation contributing to the evidence based of effectiveness for the

future
To demonstrate that outcomes and impacts have been caused by the intervention of the
project.
For example, it will be important to see if we have reached the certain number of factories that
we have wanted to reach. But it is more than that. So, it is not only about companies joining
the platform, but also about them working together. So it will also be interesting to
understand/evaluate how we can see what kind of companies, entrepreneurs are ready to join
the TCBL project.
To identify and share best practices.
To enable the results of the project particularly the generation of new business models to
be appropriated especially by smaller and newer businesses in the T&C sector. In essence
evaluation is financial advice. TCBL expects small and new businesses to adopt new
business models. This will have an economic impact which could possibly be negative if the
business model doesnt work. Its crucial that evaluation data provides evidence about which
business models will work for whom in what circumstances.

All of this suggests that both a formative (focusing on processes) and a summative evaluation
(focusing on results) are needed in TCBL. These each have two purposes.
The formative evaluation has both a design and a developmental purpose:

11

646133 - TCBL
Textile & Clothing Business Labs

The design purpose is to help clarify the intended aims and outcomes of the project
plan. This corresponds to the ex-ante phase in the project and evaluation life cycle
and ex-ante evaluation mode. However, as TCBL is an emergent project, the project
life cycle does not have a conventional linear phasing path, but follows a series of
transition paths. Ex-ante evaluation is not therefore confined to the beginning (or
planning) stage of the project but follows these transition paths. These transition paths
mark stages in innovation that the Business Pilots in TCBL are expected to go through
as they move towards new forms of business models which, in turn, will be transferred
into the wider T&C sector through established and evolving Business Systems. The
pilots provide the evidence base for the transferability of new knowledge and practices
into real Business Systems. There are three transition points in these transition paths:
T1 - existing business models and standardised formulations, i.e. from set-up to pilot;
T2 - the integration of one or more elements from the Business Laboratories pilots into
real businesses; T3 from individual to scaled up results.
The developmental purpose is about following the project process and supporting
the different stakeholders involved in assessing how the initiative is doing and whether
it is on track. This focuses not only on collecting ongoing data as the project evolves
for example collection and assessment of business model innovations but also
assessing these data and feeding the results back to help the Business Labs, pilot
labs and pilot factories to work more effectively (i.e. evaluation in operational mode).

The formative element therefore speaks to both the accountability and the learning purposes
required of the TCBL evaluation. It addresses the requirement in the DOA for a process
component to evaluating TCBL. Indeed, this is particularly important because of the
ecosystems design of TCBL. Ecosystems ecology tries to understand how the system as a
whole (rather than individual parts) operates: This means that, rather than worrying mainly
about particular species, we try to focus on major functional aspects of the system. These
functional aspects include such things as the amount of energy that is produced by
photosynthesis, how energy or materials flow along the many steps in a food chain, or what
controls the rate of decomposition of materials or the rate at which nutrients are recycled in the
4
system. The analogy to the TCBL ecosystem is that the task of the process evaluation is to
investigate what the major functional aspects of the TCBL ecosystem are and how they work.
The summative evaluation has a learning and knowledge generation purpose:

The knowledge purpose is about providing evidence on the outcomes and impacts
of TCBL, including the likely long-term impacts on key targets like reducing the
industrys environmental footprint and increasing manufacturing capacity. This
includes in particular exploring the granularity of the TCBL ecosystem the
interactions between ecosystem actors, their context and TCBL activities and whether
and in what ways these lead to change. The summative evaluation also needs to
produce meaningful indicators that enable us to assess the extent to which the project
has achieved its intended objectives and outcomes. It needs to demonstrate what is
likely to have happened to the T&C sector if the TCBL ecosystem had not been

The Concept of the Ecosystem:

http://www.globalchange.umich.edu/globalchange1/current/lectures/kling/ecosystem/ecosyste
m.html

12

646133 - TCBL
Textile & Clothing Business Labs

established the counter-factual perspective. There is therefore also an

accountability dimension to this purpose.
The learning purpose of the TCBL evaluation is about both assessing the
transferability of the results of TCBL to similar initiatives in the future and contributing
to supporting the replication and sustainability of TCBL innovations.

Again, these purposes support the use of a theory based evaluation design. The question now
is how to shape this design so it matches with the key concepts embedded in TCBL. This is
the aim of the next section.

3.2 REFINING THE EVALUATION CONCEPT

This section presents the results of a review of three key concepts that shape the TCBL
approach and which therefore need, in turn, to be reflected in the evaluation approach and
design. The three key concepts are: ecosystems; complexity and large scale change. The
discussion below is based on a narrative review of the literature in these three fields.

3.2.1 AN ECOSYSTEMS PERSPECTIVE ON EVALUATION

There is not much literature on designing evaluations from an ecosystems perspective. The
literature is dominated, perhaps unsurprisingly, by work carried out in the field of natural
resources management focusing in particular on two areas: first, methodologies developed
to assess the ecological status of a specific territory or habitat (for example carrying out an
environmental audit to identify the extent to which an areas species are threatened);
second, methodologies developed to assess environmental impact. However, work in this
mainstream field of environmental ecosystems has stimulated work in other fields, borrowing
and adapting concepts and methods from natural resource management, for example, to
develop a body of knowledge in fields of interest for TCBL notably in business ecosystems.
This branch of the literature presents similar preoccupations to the work in environmental
ecosystems, utilising concepts and methods drawn from systems theory and complexity
theory.
E NVIRONMENTAL ECOSYSTEMS AND RELEVANCE FOR TCBL
One important strand in this literature is the work that has developed around the Convention
on Biological Diversity. This has focused on promoting standardisation in ecosystems
management, and has involved a range of stakeholders including government agencies,
NGOs and scientists, working within the umbrella of the United Nations Environment
Programme (UNEP) to explore the need for an international convention on biological diversity.
The Convention identifies 12 principles that are recommended to be applied in ecosystems
management, represented in the box below.
5

Box 4: Principles for ecosystems management :

https://www.cbd.int/ecosystem/principles.shtml

13

646133 - TCBL
Textile & Clothing Business Labs

1. The objectives of management and land, water and living resources are a matter of
societal choices this focuses on the need for all stakeholders voices to be
represented in any intervention that affects the ecosystem. Ecosystems need to be
managed for their intrinsic values and in a fair and equitable way.
2. Management should be decentralised to the lowest possible level this emphasises
the need to apply participatory approaches in ecosystem management that balance
local interests with the wider public interest, and the use of local knowledge in
designing and implementing interventions.
3. Ecosystem managers should consider the effects of their activities on adjacent and
other systems this highlights the need to consider the displacement and multiplier
effects of interventions.
4. Understand and manage the ecosystem in an economic context this highlights the
need to understand how ecosystem interventions have significant, and often complex,
economic impacts. For example, introducing alternative methods of land use can
trigger market distortions that ultimately can have a negative effect on local living
standards.
5. Conservation of ecosystem structure and functioning should be a priority target of the
ecosystem approach this argues that it is not enough to focus simply on protecting
species. Ecosystem functioning and resilience depends on the dynamic relationships
between, within and among species and their environment.
6. Ecosystems must be managed within the limits of their functioning this emphasises
the need for caution in designing and implementing interventions. You need to know
the limits of the ecosystem and the constraints to its development.
7. The ecosystem approach should be taken at the appropriate spatial and temporal
scales this essentially means defining the boundaries.
8. Recognising the variability of temporal scales means that ecosystem management
should focus on the long-term perspective this highlights the tendency for humans to
prioritise short-term gains. Instead, the perspective should focus on the long-term.
9. Management must recognise that change is inevitable this emphasises the need to
anticipate how changes to the ecosystem will inevitably create systemic and often
unpredictable changes throughout the system. Management processes therefore
need to adopt systems and processes to both predict possible evolutionary
trajectories and to track these trajectories.
10. The ecosystem approach needs to balance conservation and diversity there is a
need to shift to more flexible approaches that recognise the need for preservation, but
can also incorporate human intervention.
11. The ecosystem approach should consider all forms of relevant information effective
ecosystem management strategies need to draw on information and knowledge from
a wide range of sources, combining scientific with indigenous and local knowledge.
12. The ecosystem approach should involve relevant sectors of society and scientific
disciplines this reinforces principle 11: the knowledge and expertise of all
stakeholders are equally valuable.

The literature on evaluation of ecosystems though limited reflects the principles set out in
the Convention. The James Hutton Institute has developed an evaluation framework for
assessing what has been termed the ecosystem approach (Waylen et al, 2014). The
framework highlights the need to understand how trade-offs work in ecosystems. Because
ecosystems are complex systems, in which a wide range of actors work in a continually
changing environment, the key to evaluation lies in capturing how these different actors
14

646133 - TCBL
Textile & Clothing Business Labs
negotiate and manage transactions. Echoing the principles of the Convention on Biological
Diversity, the Hutton Framework sets out a number of principles for ecosystem evaluation:

Decentralisation and empowerment of stakeholders is important to the ethos of the

Ecosystem Approach.
Iterative monitoring and evaluation is necessary to allow adaptive management and
learning, so the project should ideally plan for monitoring, and monitor a range of
social, economic and cultural issues as well as ecological issues
The complexities of natural systems present several challenges for evaluation: since
natural systems are very complex, and sometimes poorly understood, it is often hard
to track or understand changes resulting from management actions.
Ecological processes sometimes take decades to reflect the effects of changes in
management practices.
Single indicators or measures are unlikely to be sufficient as it becomes harder to
define what success is, let alone how to measure it.
The Ecosystem Approach has social goals that are not directly about the
environment: not only does it attempt to conserve resources, it is explicit that there
should be equitable use and benefits from those natural resources.
These social goals may be about project processes e.g. raising awareness or
empowering vulnerable groups, or may be about outcomes e.g. ensuring stakeholders
will receive economic benefit. Thus it is necessary for evaluators to study process as
well as outcomes.
Different goals may even be in tension: for example some environmental goals could
be achieved efficiently without meeting goals for stakeholder involvement or social
benefit. Thus, any single project will have more than one goal or objective to consider
when evaluating the overall success of the project
Evaluation will never result in a simple objective binary distinction of success or
failure. This is because different social and ecological goals may or may not all be
achieved, for a variety of reasons, and because there will be multiple points of view as
to whether success has been achieved.
However, tracking multiple dimensions of project design, context, progress and
outcomes can help to produce a balanced and fair understanding of what a project
has achieved and why.

These evaluation principles mirror to a large extent the broader principles of ecosystem
management embodied in the Convention: the centralisation of stakeholders and local
knowledge in evaluation; the need to capture complexity; an evolutionary, formative focus for
evaluation; the need to apply multiple indicators, reflecting the different perspectives and
constructions of different stakeholders.
An example of the implementation of this kind of approach in practice is an evaluation of a
tropical fishery initiative, the Mombasa coral reef and seagrass ecosystem, undertaken by
Daw et. al. (2014). This focused on surfacing, mapping and exploring how the complexity of
interactions between a wide range of different actors involved in the initiative were expressed
in the form of different trade-offs between economic, cultural, ecological and societal
considerations. Using an evaluation approach that combined ecological simulation with
participatory evaluation, the evaluation showed that food production, employment, and wellbeing of marginalized stakeholders were differentially influenced by management decisions
leading to trade-offs. Some of these trade-offs were suggested to be taboo trade-offs
between morally incommensurable values, such as between profits and the well-being of
marginalized women. These were not previously recognized as management issues.
15

646133 - TCBL
Textile & Clothing Business Labs
Stakeholders explored and deliberated over trade-offs supported by an interactive toy model
representing key system trade-offs, alongside qualitative narrative scenarios of the future. The
concept of taboo trade-offs suggests that psychological bias and social sensitivity may exclude
key issues from decision making, which can result in policies that are difficult to implement.

B USINESS E COSYSTEMS
A central strand in the literature on business ecosystems and their evaluation focuses on
exploring the relationship between business entities and business models and how these
inter-relate to form business networks. This is particularly relevant for TCBL, which seeks to
promote new kinds of networks, and hence new kinds of value chains within the Textile and
Clothing sector, by stimulating the production, diffusion and replication of business innovations
within the Business Labs and pilots. In this context a business, or service, ecosystem is
defined as a value co-creation configuration of people, technology, shared information, and
value propositions connecting internal and external service systems (Allee, 2013). Allee
describes relationships between business entities by three types of value transactions: goods,
services, and revenue; knowledge; and intangible value. Business ecosystem evaluation has
concentrated on identifying how these relationships and transactions work and what added
value they contribute. In value network research and evaluation, a business model is typically
used to describe the roles and relationships of a company, its customers, partners, and
suppliers, including the flow of goods, information, and money among these parties and the
financial benefits for those involved. Business networks describe the infrastructure that
underpins an ecosystem connecting these different business models.
Work on exploring how these networks operate has drawn extensively on game theory and
gaming models (Baron, 2012). An important part of modelling and analysis of a business
ecosystem is to capture the dynamic interactions among ecosystem business entities.
On example is the BEAM approach (Tian et al, 2008). BEAM developed a comprehensive
framework to integrate business ecosystem modeling and analysis capabilities using game
theory as the main entity behavior model, and value network modelling as a systematic
modelling method. A case study of a retail B2B service platform was used to demonstrate how
actual system performance can differ from expected behaviours, which do not take into
account the effects of interactions among business entities. Figure 2 below shows the
components used in the BEAM approach to analyse the value chains in a business ecosystem
in order to assess impact.
Figure 2: Ecosystem attributes: the BEAM model (Tian et al, 2008)

Class

Concept

Properties

Resource

Such elements as monetary, human capacity,

machine, software, and power that can be
consumed in the execution of business
activiteis or invested to realize a service

Owner

Activity

A task that uses resources

Resource consumption

Decision

The selection of a course of action among

variations, such as pricing and capabitliy
allocation

Objective

Unit cost

Decision variable set

Constraint set

16

646133 - TCBL
Textile & Clothing Business Labs
Related decision variable
Metric

Role

Performance indicator of a business object

(activity, business entity or service ecosystem)

Business object

A set of connected activities and decisions in a

service ecosystem

Activity list

Value

Decision list
Metrics list

Business
entity

A general term used for enterprises, business

units and regulators

Goals (e.g. payoff funciton,

risk attitudes)
Demographic properties
(e.g. capacity, size, and
location)

Business
model

The roles and relationships of a company, its

customers, partners, and suppliers as well as
the flow of goods, information, and money
among these parties and the financial benefits
for those involved.

Parntership
Decision-making structure
Decision making
mechanism

As the Figure shows, the key attributes that need to be evaluated in a business ecosystem,
according to this model, are resources (e.g. human capacity, money, ICTs), activities that use
these resources, decisions (that directly affect business activities), roles (the set of connected
activities and decisions within the service ecosystem), metrics (used to measure the
performance of the role, a decision list that lists all the decisions the role can make, and an
activity list that lists all the business activities played by the role, where activities, metrics, and
decisions are each elements of the ecosystem model). Data collected for these entities are
then input into a simulation model that calculates the expected outcomes and impacts of the
ecosystem.
The BEAM example shows how business ecosystems because of the complex interactions
that take place between divergent actors within an environment that is constantly changing
depart from predictive models of organisational behaviour that focus primarily on rational
strategies that are rooted in economic decisions.
Rong and Shi (2014), in their analysis of business ecosystems that have evolved around the
hi-tech sector in China, convincingly show how pointless it is to isolate individual
organisational entities from their wider eco-sphere. Taking as one example the (unregulated,
and therefore illegal) growth of the electric vehicle sector in China, Rong and Shi demonstrate
how supply chains cannot be separated from a much wider range of business, environmental,
societal and regulatory systems each of which combine to produce the different value
propositions of the ecosystem.
The challenge for evaluation in this field is to capture the dynamism of these rapidly evolving
networks and value chains whilst at the same time imposing some degree of boundary and
structure on the evolving ecosystem as a process in order to demonstrate value added,
effectiveness and impact. Li et al (2013) provide one evaluation model that attempts to do this
within the field of health ecosystems. This model builds on a framework initially developed by
Guo et al (2012) which suggests five attributes of the business ecosystem that need to shape
evaluation: logical community structure; favourable non-physical environment; efficient system
productivity; sustainability; co-ordination of management. Li et al expand this framework to
17

646133 - TCBL
Textile & Clothing Business Labs
cover eleven ecosystem attributes: energy, vitality, resilience, service to maintain business
ecosystem, transmission on innovation, investment reduction, harm to neighbouring systems,
effect on human health, power performance, inclusiveness and robustness. The model was
tested in a real world health system using ecological and complex adaptive systems theory.
Table 1 below shows how these attributes are translated into evaluation criteria and indicators
and outcomes.
Table 1: Ecosystem Evaluation Model (Li et al, 2013)

Evaluation criteria

Indicators

Outcomes

Ecological attributes

Ability to promote
performance by strategic
arrangement and resource
integration

Emergence

Completion level from price to

brand

Adaptability

Synergetic evolution
Self organisation

Efficient and innovative

management
No. of supportive enterprises

Basic enterprises

No. of complementary
enterprises
Structural attributes

Financial contribution of
suppliers to enterprises

Expanded enterprises

Technical accumulation

Supporting institutions

Investment prospects

Functional attributes

Promotion of products by
independent institutions

Complementary
organisations

Industrial policy and

regulatory support

Correlative environment

Rate of return on equity

Productivity

Technical capital growth rate

Enterprise survival rate

Vitality

New enterprise success rate

New enterprise growth rate

Operational attributes

Differentiated enterprises
growth rate

Creativity

Transparency and
confirmation of value
platform strategy

Strategic clarity

Commercial opportunities of
value platform strategy

Platform compatibility

Trust level and co-operative

distribution

Stability

18

646133 - TCBL
Textile & Clothing Business Labs

Lifecycle attributes

IMPLICATIONS FOR

Conflict resolution

Conflict co-ordination

National economy and

technical level of ecosystem

Life stage staged

development

TCBL EVALUATION DESIGN

The above presentation of key issues for evaluating from an ecosystems perspective identified
a number of recurring themes that reflect the distinctive attributes of ecosystems and which
need to shape the TCBL evaluation. It seems clear that the TCBL evaluation approach should
embody:

A participatory evaluation methodology that incorporates an element of action

research, based on engaging with key stakeholders in particular real world
business entities who will access and adapt business innovations emerging from the
Labs and pilots. In this context, the evaluation role becomes one of co-creation of
knowledge and action.
This also means that the evaluation needs to combine a broad range of data and
information that reflects the spectrum of involved stakeholders using scientific data
as well as local knowledge. This in turn requires the construction of multiple sets of
evaluation indicators. Whilst these need to be in line with the evaluation questions,
these could follow the model developed by Li et al. (see Annex 1).
It follows that, since stakeholders and their different constructions of and positions on
TCBL are centralised within the evaluation, it needs to devote attention to ensuring
that all stakeholder voices are represented particular those who are less powerful.
This will be a real challenge, since there is a clear tension in TCBL between the
interests of small entities (and individuals), communities and major players like the
large T&C retailers. Sensemaking actions will therefore be crucial in the evaluation.
The evaluation should prioritise formative (process and action-driven) evaluation over
summative evaluation.
Summative evaluation needs to consider the long-term view. The evaluation needs to
incorporate methods to calculate the expected impacts of TCBL in the long term, with
a time horizon of 2020.
Process evaluation needs to focus in particular on the following: how entities within
the ecosystem access and engage with business innovations; how they make
decisions on the utility of these innovations; how they trade-off different aspects of
innovation utility to arrive at a calculation of value for their business; how different
entities make trade-offs between themselves within the ecosystem; how the extracted
value of business innovation is then applied in practice; whether this extracted value
is short term or long term.

3.2.1 EVALUATING COMPLEXITY

Working with the ecosystems idea in the TCBL evaluation means also working from a
complexity perspective (and we have already characterised TCBL as a complex intervention).
A COMPLEXITY PERSPECTIVE ON THE TCBL EVALUATION
Complexity theory is an interdisciplinary field of research that has become recognised as a
new field of inquiry focusing on understanding and better describing dynamics and processes
19

646133 - TCBL
Textile & Clothing Business Labs
of change found in a range of physical and biological phenomena. The notion of complexity
and complexity theory gradually absorbed elements of general systems theory, cybernetics,
chaos theory and information theory (ODI: 2008).
Following initial applications to the physical sciences, increasing attention is now being paid to
how the key ideas and concepts of complexity science (e.g. notions of emergence,
nonlinearity, dynamic systems, interactions and interrelations) can help researchers and
practitioners understand (and influence) social, economic and political phenomena. The
underpinning assumption is that much change in contemporary situations is not linear (Rogers
2008). Within this shift, complex systems thinking was applied in management and to the
study of social phenomena by the social science disciplines, to development and to policy
analysis. Complexity theory has also increasingly been discussed and applied within
evaluation theory and practice over the past decade (Ling: 2012) in order to evaluate what are
defined as complex interventions. In particular, the interest in complexity centres on a
growing recognition of the disjunction between the non-linearity and unpredictability of change
processes and the protocols and procedures that govern development interventions that
assume otherwise. The box below summarises 10 key concepts of complexity theory.
Box 5: 10 principles of complexity theory

Complexity and systems. These first three concepts relate to the features of systems which
can be described as complex:
1. Systems characterised by interconnected and interdependent elements and dimensions are
a key starting point for understanding complexity science.
2. Feedback processes crucially shape how change happens within a complex system.
3. Emergence describes how the behaviour of systems emerges often unpredictably from
the interaction of the parts, such that the whole is different to the sum of the parts. In complex
interventions, the notion of emergence means not that certain patterns emerge as our
understanding of them improves (knowledge which can then be used to predict similar
interventions in the future), but that the specific outcomes, and the means to achieve them,
emerge during implementation of an intervention.
Complexity and change. The next four concepts relate to phenomena through which
complexity manifests itself:
4. Within complex systems, relationships between dimensions are frequently nonlinear, i.e.,
when change happens, it is frequently disproportionate and unpredictable.
5. Sensitivity to initial conditions highlights how small differences in the initial state of a system
can lead to massive differences later (butterfly effect).
6. Phase space is a way to describe complex systems because it does not seek to establish
known relationships between selected variables, but instead attempts to shed light on the
overall shape of the system by looking at the patterns apparent when looking across all of the
key dimensions to build a picture of the dimensions of a system, and how they change over
time. This enables understanding of how systems move and evolve over time.
7. Chaos and edge of chaos describe the order underlying the seemingly random behaviours
exhibited by certain complex systems.
Complexity and agency. The final three concepts relate to the notion of adaptive agents, and
how their behaviours are manifested in complex systems:

20

646133 - TCBL
Textile & Clothing Business Labs

8. Adaptive agents react to the system and to each other, leading to a number of phenomena
(contextual sensitivity).
9. Self-organisation characterises a particular form of emergent property that can occur in
systems of adaptive agents.
10. Co-evolution describes how, within a system of adaptive agents, co-evolution occurs, such
that the overall system and the agents within it evolve together, or co-evolve, over time.

Evaluating from a complexity perspective also means thinking through what a complex
intervention is. In line with the principles above, complex interventions are not predictable,
involve emergent and responsive interventions and causal processes which cannot be
completely controlled or predicted in advance. The interactions between determinants of the
sub-problems can lead to non-linear causal relations between potential causes and outcomes.
Also, context-sensitivity can make a problem complex. As a consequence, outcomes are
unpredictable. To solve complex problems, formulae and standardised solutions that proved
effective in the past provide little guidance. Instead, complex problems are solved through
safe-fail experiments that allow learning by doing or by making sense of events post facto.
A particularly useful report on evaluating complexity, by Preskill and Gopal (2014) seems to
encompass most aspects of complexity concepts. They outline some key characteristics of
complex systems and, from them, infer propositions for evaluating complexity. In coming up
with the following characteristics of complex interventions, they have borrowed from a variety
of complexity scientists and theorists, as well as evaluation practitioners.
Table 2: Features of complex interventions

A complex system is always changing, often in unpredictable ways; it is never static

A complex intervention has multiple components, interacting with each other; everything is
connected; events in one part of the system affect all other parts
Interventions are open systems and change through learning as stakeholders come to understand
them.
Context matters
Each situation is unique; best principles are more likely to be seen than best practices
Relationships between entities are equally if not more important than the entities themselves
Cause and effect is not a linear, predicable, or one-directional process; it is much more iterative
Patterns emerge from several semi-independent and diverse agents who are free to act in
autonomous ways

Complexity theory thus represents a challenge to the linear assumptions about how change
happens that underpins many funded programmes. These tend to be based on a predictive
logic, i.e. if a particular set of actions are taken these will result in a particular set of outcomes.
This works very well where there are known solutions to problems, and stable and replicable
contexts, but is often problematic in real-world social contexts which are constantly changing
and in which multiple actions are combining with each other time and again to produce a
highly unpredictable environment, such as the case of TCBL.

21

646133 - TCBL
Textile & Clothing Business Labs

IMPLICATIONS FOR

TCBL EVALUATION DESIGN

The text above has the following implications for the design of the TCBL evaluation:
Designing an evaluation approach that is iterative and able to capture data in real-time.
Because complex interventions are not fixed, their evaluation requires real-time data
collection, to support ongoing learning and reflexivity. In this sense, outcomes and impact can
only be assessed against the decisions made during the course of the programme, not against
the programming logic at the start of the programme. This means that process and content
(including any changes in direction and the rationale for them) is important and must be
documented as we go along. The iterative nature of the process enables to: understand what
has changed in the system, and what specifically has changed as a result of an action taken;
adjust actions/goals etc to maximise impacts; set new baselines and indicators of success
against those baselines. Overall, complexity calls for a more adaptive mode, similar to the
intentions of developmental evaluation (Patton: 2006, cited on:
http://betterevaluation.org/plan/approach/developmental_evaluation), where real-time
evaluative practice stays close to unfolding intervention so that it is part of it, or longitudinal
study of impact evaluation (CDI conference paper, 2011).
Inclusion and sense making. In conditions of complexity, multiple competing explanations of
why and how change happens exist, including tensions between the large number and
diversity of stakeholders involved. This has two key implications for the TCBL evaluation. The
first is that evaluation will need to include all TCBL stakeholders at all levels of the project to
enable the multiple perspectives to emerge. Second, the evaluation will need to build in sensemaking opportunities. Much of the literature reviewed specifically used the term in
cooperation with partners rather than participation with partners, in ways that allow multiple
perspectives to inform the findings while being mindful of power relations among participants.
In general, the two points discussed above (the iterative nature of an evaluation design that
seeks to evaluate a complex intervention and the importance of sense-making in the
evaluation process) tend to stress the usefulness of adopting principles of Action Research
methodologies when assessing impact in complex and highly dynamic environments.
Strengthening the formative role for the evaluation. Perhaps similarly to the points above,
a common message from the literature is that when evaluating a complex intervention, there is
value in, or indeed a necessity for, shifting towards an evaluative practice that emphasises
reflective practice and organisational learning in the form of a formative evaluation. Attached
to this, is the notion that there is often a blurring or an overlap between the impact evaluation
and the creation of evidence by the project itself as it learns and adapts (formative). Due to the
emergent nature of complex interventions, it is not uncommon for the balance between the two
to shift towards a more formative role because it takes on, as noted above, more of the
characteristics of a real-time evaluation (Ling: 2012).
The importance of context. Context in evaluation typically involves understanding five
specific dimensions: demographic characteristics of the setting and the people in it, material
and economic features, institutional and organisational climate, interpersonal dimensions or
typical means of interaction and norms for the relationships in the setting, and political
dynamics of the setting, including issues and interests (Preskill and Gopal: 2014). Because
complex initiatives tend to involve multiple actors and organisations, are implemented over
multiple years, and naturally adapt in response to changing conditions (challenges and
opportunities, as well as negative and positive stimuli), evaluations need to capture
information on how the initiative and its context are co-evolving. In other words, the
evaluation should not only study the context and its influence, but also measure the ways in

22

646133 - TCBL
Textile & Clothing Business Labs
which the initiative affects the context. Evaluation findings should be interpreted and grounded
in context to support claims about an initiatives progress and impact.
Theory of change and counterfactual. Understanding the projects Theory of Change
remains an important feature of evaluations of complex programmes (Rogers: 2012; Ling:
2012; Burns 2014). In emergent projects designed to accommodate the uncertainties
associated with complexity, the Theory of Change will include attention to the importance of
learning and adaptation. It will also identify the key dependencies upon systems and
subsystems which lie outside the formal structures of the intervention. Burns (2014) also
points out that a theory of change should constantly be tested in action through an iterative
process, as learning takes place. Whilst in complex interventions it is often much harder to
identify the counterfactual, nevertheless it is crucial to pose the core question in an evaluation
which is did it make a difference?. Answering this question must in turn involve asking
compared with what?. However, the counterfactual for a complex intervention is not a single
outcome but a counterfactual space of more or less likely alternative states. This might be
produced by scenarios, modelling, simulation, or even expert judgement.

3.2.1 LARGE SCALE CHANGE

Finally, we suggest it is useful to think of TCBL as a large scale change effort, and to consider
what this means for evaluation design.
Large scale change (LSC) can be defined as the emergent process of mobilising a large
collection of individuals, groups and organisations towards a vision of a fundamentally new
future state, by means of:

High leverage key themes

A shift in power and a more distributed leadership
Massive and active engagement of stakeholders
Mutually reinforcing changes in multiple systems and processes.

Done properly, this leads to such deep changes in attitudes, beliefs and behaviours that
6
sustainability becomes largely inherent.
7

According to the NHS Institute for Innovation and Improvement , there are three ways of
thinking about large scale change:
1. Kurt Lewins distinction between incremental and transformational (ie large
scale) change. Transformational change may be experienced as a series of small
(incremental) steps, but has a vision behind it where, by the end of the process,
everything would be different.
2. The complex systems lens. Large scale change requires integrated changes in
structures, processes and patterns (of behaviour and outcome).
3. Three dimensions of LSC. According to NHS Institute, large scale change is one that is:

NHS Institute for Innovation and Improvement (2013) Leading Large scale change: a
practical guide, p. 30 www.nhsiq.nhs.uk/download.ashx?mid=8526&nid=8530

NHS Institute for Innovation and Improvement (2013) Leading Large scale change: a
practical guide, www.nhsiq.nhs.uk/download.ashx?mid=8526&nid=8530, pp18-21

23

646133 - TCBL
Textile & Clothing Business Labs
a. Widely spread across geographical boundaries, multiple organisations, or multiple
distinctive groupings.
b. Deeply challenging to current mental models and ways of thinking (it feels
uncomfortable and evokes some push back from others because it is so different
from the usual).
c.

Broadly impacting on what people do in their lives or time at work and requiring
co-ordinated change in multiple systems.

Large scale change normally involves a combination of technological and social systems
challenges and is approached from an open systems view, especially when pursued through
an ecology or social movement lens.
Figure 3: Dimensions of large scale change

Source: NHS Institute for Innovation and Improvement (2013) Leading Large Scale Change, p.
22
There are 10 key principles that underpin Large Scale Change. These are represented in the
box below.
Box 6: 10 Principles of Large Scale Change

1. Movement towards a new vision that is better and fundamentally different from
the status quo. LSC is fuelled by the passion that comes from the fundamental belief
that there is something very different and better that is worth striving for.
2. Identification and communication of key themes that people can relate to and
that will make a big difference. The vision is out there and in the future. Typically,
if it is truly LSC it will seem so distant and so in contrast with the current reality that it
may feel overwhelming or impossible. In order for people to get engaged they have to
see what they can do, now or soon, that would be clear and meaningful step along the
journey.

NHS Institute for Innovation and Improvement (2013) Leading Large scale change: a
practical guide, p. 26-29, www.nhsiq.nhs.uk/download.ashx?mid=8526&nid=8530

24

646133 - TCBL
Textile & Clothing Business Labs

3. Multiples of things (lots and lots). LSC is complex with many different
stakeholders, agendas (both hidden and open), points of view, needs and wants,
details, systems that need to change and so on. Attempts to isolate or work around
some groups, or to ring fence some parts of the system to be left alone while others
must change, typically result in something less than LSC.
4. Framing the issues in ways that engage and mobilise the imagination, energy
and will of a large number of diverse stakeholders in order to create a shift in
the balance of power and distribute leadership. Because there are lots of lots, a
small band of leaders cannot possibly make the LSC happen. Tight, centralised
planning and control actually works against LSC. Instead, multiple leaders from across
the system and at all levels, drawn to the vision must engage and commit their will
and energy to the effort of achieving LSC. As more distributed leadership emerges
and is enabled across the system, cross boundary and partnership working increases
and change happens at a massive scale and pace. But the key lies in gaining the
commitment of others to act, not merely their compliance in doing what you tell them
to do. Experience shows that change based on compliance without commitment is
difficult to sustain over time.
5. Mutually reinforcing change across multiple processes / subsystems. If the
vision is sufficiently clear and the collection of key themes comprehensive enough,
what may seem at first like a chaotic lack of control actually comes together in the
form of changes that connect with and build upon, one another.
6. Continually refreshing the story and attracting new, active supporters. While
LSC efforts often start small, with just a few people who are switched on by the vision,
the life blood of LSC is the continual stream of new supporters who become attracted
to the vision when they see it progressing. Without a steady stream of new supporters
becoming committed to the change, LSC efforts can plateau or run out of energy.
7. Emergent planning and design, based on monitoring progress and adapting as
you go. Because of the complexity and uncertainly involved, LSC outcomes are
impossible to predict at a detailed level. Flexibility, adaptability and engagement of
others are the keys. It is OK to have detailed plans and milestones () just dont
spend too much time on them before you start actually doing something and dont be
surprised if every detail does not work out as planned.
8.

Many people contribute to the leadership of change, beyond organisational

boundaries. As LSC efforts spread and become increasingly complex, more and
more leaders need to be recruited for the change effort. Leadership of the LSC effort
is not dependent on a small number of key individuals in a hierarchy. This is
distributed leadership; a variety of different sources of leadership expertise move into
play and are spread around the system. Leaders pool their effort and expertise so that
the collective result is significantly greater than the outcomes of individual leadership
actions.

9. Transforming mindsets, leading to inherently sustainable change. According to

the literature, when LSC is done well, sustainability is the natural by product. If people
have really become engaged and believe that that vision is more desirable than the
status quo (and you have addressed multiple structures, processes and patterns) they
will be committed to and will fight to keep the new way, showing little interest in going
back to the old. However, if powerful leaders have simply demanded compliance and

25

646133 - TCBL
Textile & Clothing Business Labs
not brought others along in the thinking, those on the frontline will put things back to
the old ways. The way they still believe was right.
10. Maintaining and refreshing the leaders energy over the long haul. The case
study literature on LSC makes it clear that large scale change can take some time to
unfold completely. Too many leaders simply run out of steam.

It is easy to see that these resonate quite well with the TCBL project design, in particular
themes 1, 3, 4, 6, 7 and 8. Large scale change thus offers a practical lens for the evaluation
that is compatible with ecosystems and complexity thinking: the ten principles articulated
above appear particularly useful to shape the formative and process evaluations of TCBL by
offering some concrete markers against which to analyse TCBL activities and where
necessary suggest improvements.

3.3 SUMMARY AND IMPLICATIONS

The table below summarises the key elements that each of the three concepts brings to the
TCBL evaluation task. It shows that the concepts complement each other and highlights the
specific dimensions that each of the three perspectives has to offer. The next step is to work
these requirements into the details of the evaluation design: the questions, process and data
collection methods. This is done in the next chapter.

26