Building the Learning Organisation

The Critical Role of Evaluation

Keith T Linard
Senior Lecturer, Department of Civil engineering
University College (University of New South Wales)
Keithlinard#@#yahoo.co.uk
(Remove hashes to email)
ABSTRACT:
Why is the average life of major corporations less than half that of the humans who staff
them? Why do so many managers so often make decisions which, in retrospect, appear so
blindingly stupid?
My basic premise is that managers do not make stupid decisions because they like doing so.
Rather, where pathological performance exists one usually finds aspects of the system
which encourage, rewarded or reinforced this pattern of behaviour. The very organisational
architecture trains staff to act in ways detrimental to the whole. These systemic structures
are themselves the product of the individual and collective mental models nurtured within
the corporate melieu.
Redressing this situation requires revisiting not only the organisational architecture, but the
deep learning cycle which nurtures and forms the organisational culture. The discipline of
system dynamics provides a rigorous basis and a veritable Toolbox of techniques for
understanding the why of organisational behaviour and for planning intervention strategies.
KEYWORDS: Evaluation; performance indicators; performance monitoring; system
dynamics; learning organisation; Powersim.
Keith Linard, as Chief Finance Officer, Australian Department of Finance, was responsible
for the Financial Management Improvement Section during the 1984-88 'reform' of the
federal bureaucracy. Keith currently runs the system dynamics program at the Australian
Defence Force Academy and co-directs the postgraduate program in project management.
Keith is also a Director of Synergy International Consulting Pty Ltd.
_________________________________

The real voyage of discovery consists not in

seeking new landscapes but in having new eyes.
Marcel Proust

INTRODUCTION
New eyes for a voyage of discovery . . .
Fewer than 40% of the Fortune 500 companies that existed in 1950 still exist today. Surely, I
ask myself, these companies could have afforded the best advisers, the best management
systems and the best managers. Why didnt the entrepreneurs, the managers or the financial
experts see the emerging problems well before bankruptcy stared them in the face.
Over the past six months our TV news has carried revelation after amazing revelations from
the NSW Royal Commission into police corruption. I ask myself how those in authority,
outside the corruption web, could have been so blind for so long ... the managers ... the
politicians of all parties ... the press.
I think over my experience both in the Federal Department of Finance responsible for public
sector reform and subsequently as a management consultant. Time and again I encountered
stupid management decisions and asked myself why the managers and their corporate or
political masters could be so blind for so long.
The blinding power of culture . . .
A key reason for this failure to see is that
many organisational practices (the way
we do things around here) derive from
the organisational culture. Decision
makers and other experts are so
immersed in the cultural ambience that
they are not aware of the blinkers it
imposes. The culture, the shared mental
models which develop over years, is
shaped by, and helps shape, the
management architecture, the accepted
practices and the expectations.
The first step to enduring management
reform is an understanding the culture,
Figure 1: The blinding power of culture which demands seeing the organisation
with a new set of eyes.
A second factor in such management breakdowns is the pervasive focus on events . . . the
stuff of news reports, parliamentary questions and, all too often, performance indicators.
This leads to reactive solutions which generally do not address the underlying symptoms, and
indeed which often entrench the problem into the culture (refer Reward the incompetent).
Seeing events with a new set of eyes we may discern patterns of behaviour produced by
systemic structures, which in turn are the product of pervasive mental models. A secure field
of employment is that of business writers who, with 20-20 hindsight, uncover the underlying
patterns which led to disaster, discernible years and sometimes decades before. The
somewhat belated investigative reporting of the failed Bond Corporation is a case in point.
The blinding power of reductionism
Closely related to the events virus is the reductionist syndrome which often terminally
impairs our corporate vision. There is no doubt that the industrial revolution, and with it our
advances in the sciences, was made possible by lines of inquiry that broke systems into their
component parts. Similarly, incredible advances in business management have been made
through concepts such as organisation breakdown structures, work breakdown structures and
charts or accounts. But these are not the whole story.
A new set of eyes will reveal the blindingly obvious . . . that a system is always more than the
sum of its components . . . and that many pervasive business ideas are, for example
benchmarking, are anti-systemic and hence fundamentally flawed.
The blinding power of belief in unidirectional causality
The final blinding factor, especially prevalent in Western psyche since the time of Aristotle
in the 3rd century BCE,, is the belief in unidirectional causality. An example of this is the so
called logic framework which underpins the evaluation paradigms of the NSW and
Commonwealth Governments.
A new set of eyes will uncover the impact of feedback . . . not simply a dotted line which we
sometimes put on our sketches then ignore, but a powerful reactive force that overwhelms our
initial assumptions.

SEEING THE CULTURE OF ORGANISATIONS

The 1980s Financial Management Improvement Program
The success of the reforms in the federal bureaucracy in the 1980s was due largely to the
fact that the Financial Management Improvement Program (FMIP) focussed explicitly on the
inter-relationships between organisation culture (expressed through service-wide standards
and practices), the organisation architecture (corporate management style, program
management framework, organisation design, management information systems and
evaluation framework) and the legal and regulatory environment.
When the FMIP was established in 1984 our first step was to develop a sustainable
framework (we reckoned on a decade) to nurture transformational change through ongoing
learning.
To this end members of the team visited a wide range of managers and staff in central and
state offices of all federal departments. We asked individuals and groups: What factors,
regulations or standard practices stop you doing your job as well as you would like, and what
would you change if you had the power?

Legal, regulatory & Reflecting on the diverse

budgetary environment responses the FMI team
evolved a model of organis-
ational learning which then
shaped subsequent reform
agenda. The key elements

Perceived standards ACTIONS were:

& practices Budgetary and legal

environment
Management systems and
Management systems & organisational
organisational architecture architecture
Service-wide standards &
Figure 2: Interrelated elements impacting on the public practices
service culture and on results

Of feedback loops and pathological structures

A basic premise of the FMI team's approach was that managers do not make stupid decisions
because they like doing stupid things. Rather, where pathological behaviour was seen to
exist, we looked for those aspects of the "System" which encouraged, rewarded or reinforced
this pattern of behaviour.
Let me give a simple example. Pre-1984, both within and between Departments, a "success
to the incompetent archetype" was endemic. Suppose that a mid-year budget review in
Department X foreshadowed a budget blow-out in (due to poor management) in the NSW
Branch. The Regional Director would undoubtedly raise innumerable reason justifying the
situation. The press and lobby groups would demand more resources for the services
concerned. The Department would typically respond by reallocating funds from State
Branches which were coming in on target or with a modest surplus. Clearly, they were over-
endowed if they had no problems! Thus, the incompetent managers were rewarded and the
competent were penalised.
Such "fixes" worked in the short term, indeed they often served to spur good managers to do
better with less. Over time, however, well managed areas would build in pathological
responses to protect "their" budget. This is depicted in Figure 3.

Failure Success Interpreting the diagram

s
of A of B
o s arrowhead S means
s s that change in the
Incentive for Allocation to A Incentive for preceding factor will
A to manage instead of B B to manage cause a change in the
o s o Same direction
o DE
s LA
Y O means that change
Resources Resources in the preceding factor
to A to B
will cause a change in
the Opposite direction.
Figure 3: Reward the incompetent archetype
In this scenario, Manager
A blows budget while manager B comes in under budget. The Executive decides to
reallocate funds from B to A.
Immediate result
B's surplus bails out A, and the organisation, on average, is on target.
Short term impact of executive action
A has no incentive to improve, and likelihood of future failures increase
B is challenged to do better and may do so.
Longer term impact
B realises it is "a mugs game" and performance drops
Such causal loop diagrams are valuable in a number of respects. First, they serve as a
language for presenting and clarifying ideas. Thus, in the above case, two analysts could
come up with seemingly opposite conclusions. The decision to help A will motivate B to
do better, or do worse. Both analyses could be right. They could be tracing different causal
relationships, whose relative influence varies over time.
Events and patterns of behaviour
More importantly, causal loop diagrams provide a framework to discover leverage points to
intervene in the system. What are the pathological feedbacks to be severed; what are
beneficial paths to be reinforced. This is illustrated in figures 4 and 5.

Regulations Figure 4 depicts typical historical

& Rules S development of government procurement
systems. Inevitably scandals occur . . . :
an increase in scandals reduces
Parliamentary parliamentary confidence
confidence S
O Reliance on reduction in parliamentary confidence
rules & regs. leads to increases in rules and
O Scandals in
O
regulations
purchasing
in the short term, increases in rules
O reduce scandals.
Job status BUT IN THE LONGER TERM
Staff Creative mgt.
competencies responsesD further rules increase reliance on rules
EL
S Learning
AY
increased reliance on rules reduces job
status
Retention of S
competent staff reduced job status leads to reduction in
competent staff and to increase in
Figure 4: Shifting the burden archetype scandals
By 1986 the dead hand of a centuries of laws and regulations (the genesis of many of the
Commonwealths procurement practices could be traced to the 17th century) was estimated to
be costing between $600 million and $1000 million per year.

Regulations
& Rules
Figure 5 depicts the conceptual framework
S
of the approach by FMIP task force on
procurement. In essence, recognising the
Parliamentary perverse impact of over reliance on
confidence S regulation, the task force set out to
O Reliance on strengthen management of procurement
rules & regs. by:
O Scandals in
O purchasing devolving responsibility from central
agencies to department heads
O
Job status
a focus on building staff competencies
Staff Creative mgt. dramatic reduction in rules, with a
competencies responses
DE
LA
focus instead on outcome objectives
S S Learning S Y (probity, economy etc)
Retention of S increase in job status
competent staff

Figure 5: Addressing the fundamentals

This reversal of 3 centuries of ever
accumulating rules and regulations was one of the more significant, but less remarked upon
reforms of the period.
System structures and the roller coaster syndrome
The causal loop diagram is invaluable in positing causality relationships. A variety of tools,
from statistical techniques to simple graphing of historical data, can then assist our seeing
patterns over time and thereby help refine our analysis. But how can we be confident that we
are not seeing a figment of our imagination.
Over the past decade a variety of powerful graphically oriented computer system dynamics
modelling tools have become available. These allow us to map more rigorously the structural
relationships and to compare the simulated outcomes with both our expectations and with
historical data. These tools are particularly useful in evaluation, in testing the implications of
presumed causality, and helping to understand how performance indicators feedback to
modify behaviour. In short, they help us learn how changes in individual parts of a structure
feedback to affect the structure as a whole.
Figure 6 illustrates a simple inventory structure, where the ordering rule is to order whatever
is sold plus gap between target and actual inventory (the performance indicator). This
structure, which mirrors the structures for capital investment strategies, product marketing,
staff numbers management and many others, is inherently unstable and readily generates a
roller-coaster or boom-and-bust cycle through inappropriate use of performance
indicators.

Figure 6: Dynamic 'roller-coaster' behaviour generated by inappropriate business rules

relating to a performance indicator
As discussed in my paper to the 1995 National Evaluation Conference, Dancing Towards
Disaster - Performance Indicators in the Public Sector, the Australian public (and private)
sector approach to performance indicators has fundamental flaws.
Misperceptions of feedback - the critical issue
An implicit and fundamental assumption behind the Australian public sector approach to
selecting performance indicators is that the feedback they give to the decision maker (either
directly or via resultant pressure or direction from others) will cause the decision maker to
make appropriate adjustments to the inputs or processes. This presumes the decision makers
can judge with reasonable accuracy the consequences of their decisions.
However, there is abundant research in the field of system dynamics1, as well as in the fields
of experimental economics and psychology which suggest that managers have great difficulty
managing dynamically complex tasks. Sterman argues persuasively from his work at MIT
that there is systematic misperception of feedback especially when there are delays in the
system. Mosekilde, Larsen and Sterman2 present the results of 48 simulations of the Beer
Game3 (a simulation of a simple factory-warehouse-retail system) run with 192 graduate
students from MIT and senior executives of major US firms. The results show that their
decision-making on the basis of fairly straight forward performance indicators, but in the face
of delays, resulted on average in costs more than 10 times the optimum!
Simulations run by the author at the Australian Defence Force Academy, involving 27
undergraduate and graduate students, showed a similar pattern. In both the MIT simulations
and those at ADFA, highly educated managers and students failed to comprehend the
significance of feedback in the face of delay induced dynamics.
In more recent experiments, where graduate students had full information, training, incentives
and opportunities for gaining experience, Diehl and Sterman still found poor managerial
performance in the face of variations in feedback strength and delay.4 Often they found the
subjects were outperformed by a simple no-control rule. Diehl and Sterman argue that the
mental constructs and heuristics that managers bring to bear on complex tasks are
fundamentally dynamically deficient:
Subjects were unable to account well for delays and feedback effects because (1) peoples
mental representations of complex tasks are highly simplified, tending to exclude side effects,
feedback processes, delays, and other elements of dynamic complexity; and (2) even when
these elements are known, peoples ability to infer correctly the behaviour of even simple
feedback systems is poor.

1
Sterman, J. Deterministic Chaos in Models of Human Behaviour. System Dynamics Review, 1988, 4, 148-178.

Sterman, J. Misperceptions of Feedback in Dynamic Decision Making. Organisational Behaviour and Human Decision
Processes, 1989, 43(3), 301-335.

Sterman, J. Modelling Managerial Behaviour: Misperceptions of Feedback in a Dynamic Decision Making Experiment.
Management Science, 1989, 35(3), 321-339.

Paich, M, & J Sterman. Boom, Bust and Failures to Learn in Experimental Markets. Management Science, 1993, 39(12),
1439-1458.

Smith, V, G Suchanek and A Williams. Bubbles, Crashes and Endogenous Expectations in Experimental Spot Asset
Markets, Econometrica, 1988, 56(5), 1119-1152.

Funke, J, Solving Complex Problems: Exploration and Control of Complex Systems, in R Sternberg and P Frensch (eds.),
Complex Problem Solving: Principles and Mechanisms. Erlbaum Assoc., New Jersey, 1991.
2
Mosekilde, E, E Larsen and J Sterman. Coping With Complexity: Deterministic Chaos in Human Decision making
Behaviour, in J Casti and A Karlqvist (eds.), Beyond Belief: Randomness, Prediction and Exploration in Science. CRC
Press, Boston, 1990.
3
The Beer Game is described in detail in Senge, P, The Fifth Discipline - The Art and Practice of the Learning
Organization. Doubleday, New York, 1990
4
Diehl, E and J Sterman. Effects of Feedback Complexity on Dynamic Decision Making. MIT Sloan School of
Management, Research Report D-4401-1. March 1994.
The first deficiency can certainly be addressed through training. The second, however, ... is
a fundamental bound on human rationality - our cognitive capabilities do not include the
ability to solve systems of high-order non-linear differential equations intuitively.
The evaluators toolkit
Where does this leave us? In essence, this research suggests that, in situations involving
feedback or delay, managers require the assistance of tools to assist in understanding the
consequences of alternative decision choices in the face of feedback and delay.
Daniel Kim5 identifies 10 categories of systems thinking tools build on one another to
produce ever deeper insights into dynamic behaviour. Kims selection is reproduced in
Figure 7. Full discussion of these is a full workshop in its own right, so I will pass over them
to the more difficult problem, how to move towards a learning organisation.

Figure 7: A Palette of Systems Thinking Tools

5
Kim, D., Systems Archetypes 1 - Diagnosing Systemic Issues. Pegasus Communications, Cambridge MA, 1992.
 TOWARDS A LEARNING ORGANISATION
Of all the buzz words in management over the last few decades learning organisation is the
most difficult to pin down. When you listen to those from MIT who first coined the term,
Peter Senge, Fred Kofman and Daniel Kim, or their colleagues such as Margaret Wheatley or
Sandra Seagal you feel a sense of excitement and adventure. But when you start to analyse
their words in detail it is like trying to grasp a slippery bar of soap.
Fundamentally, the learning organisation is one where the structures foster an on-going
conscious attention to and examination of the fundamental contextual assumptions and
governing values of the organisation, where there is honest exploration of these values and
where these feed back into the organisational architecture.

Awareness and
Sensibilities
Attitudes
and beliefs DOMAIN OF
ENDURING
CHANGE
Skills and
Capabilities
Guiding ideas
DEEP LEARNING CYCLE
DOMAIN
OF ACTION
Innovations in Theories, Tools
infrastructure and Methods

ORGANISATIONAL
ARCHITECTURE

RESULTS

Figure 8: Framework for a learning organisation

(Adapted from Senge, P., Building Communities of Commitment, Vol4/10, Dec 1993. p.3.)

Figure 8, Senges framework for a learning organisation, suggests a radical reorientation of

focus for organisation design. The traditional focus of organisational architecture - the hard
quantitative aspects incorporating the corporate planning framework, the organisational and
program structure and standard operating procedures - is seen as the more ephemeral. A new
department head or CEO can dictate changes to these elements overnight.
The heart of the learning organisation is the deep learning cycle where the enduring values
are learned and embedded. Police Commissioners (presumably with a distaste for corruption)
came and went in the NSW Police Force, but the corrupting culture remained. Department
Heads came and went in the Federal Department of Finance, structures changes, but the
powerful ethos of protecting the public purse remained. It is in this Domain of Enduring
Change where the organisations attitude and belief structures, good or bad, are constructed.
It is here that corporate mental models are nurtured, for good or bad; where humanitarian (or
utilitarian or sexist or racist) corporate cultures are fostered.
Words . . . words . . . words . . .
So what? Is this just a rerun of the annual corporate planning love-fest, before we return to
the real jungle? Or has it any practical value?
I suggest we are talking of no less than corporate survival! Reflect, for one moment, on the
percentage of fortune 500 companies that survive to old age. They didnt fail just because a
competitor came on the market. There are innumerable examples where the influx of
competitors has spurred somnolent companies into a new lease of life. They didnt fail
because of a single bad decision, or because of the fraudulent action of an employee in
Singapore or New York. They failed because of a pattern of events over time, which were the
product of systemic structures, which in turn were the outgrowth of collective culture - the
mental models nurtured within the corporations over years or decades.
How we (individually or collectively) think, act and value are all associated with our
particular view of reality, our weltaungshuung. This acts as a filter through which we
interpret the events and patterns of behaviour, and also as masks which hides the real us
from others.
Whether we, as managers, view an event or pattern of events as a problem depends on our
frame of reference. Problems as such do not exist out there, independent of the people
who view them. How we articulate that problem can determine whether our solutions set up
a vicious cycle of tinkering with symptoms (vide reward the incompetent) or whether they
address the root cause of the corporate disease. How we articulate problems can shape the
future direction of our reality.

Problem
Articulation
LIFE Problem Solution
DELAY

Figure 9: Problems do not exist out there . . . they are events / patterns interpreted and
articulated through managers mental models
For example, is a dramatic cut in Australian tariffs a problem? It depends whether I am an
importer, exporter or service organisation. It depends whether my production centre is in
Melbourne or Manilla. It also depends on whether I have the organisational culture to rise to
new and exciting challenges of competition on world markets.

Double loop learning

A step towards developing a learning organisation is to establish a framework which
explicitly addresses the contextual assumptions and governing values of the organisation.
This is depicted in Figure 10.
The top portion of the diagram reflects the traditional performance evaluation framework. In
the short term it addresses efficiency (process or formative evaluation - are we achieving the
specified outputs in an economic fashion?) and in the longer term, effectiveness (summative
evaluation - have we achieved the desired outcomes and are these outcomes still valid given
the efflux of time and changed circumstances?)
 Measurable System

Resources State
Capabilities
Intervention Input Process Output

Design of Environment
Corrective
Action Observation of Results
Performance
Evaluation
Examination of Contextual
Assumptions and Governing Values

Figure 10: Towards a learning organisation - double loop performance evaluation

The lower loop, however, addresses a more fundamental question. What are the foundational
ethical human values that are driving us? How far back do we go in seeking these
fundamental values? In the graduate classrooms of the MIT Sloan School of Management,
in MITs learning organisation workshops for CEOs of Fortune 100 companies, in
executive management conferences run by MITs Systems Thinking project issues as basic as
the meaning of life and death are being explored. Workshops are replete with terms such as
love, stewardship, service, economy of cooperation.
Fred Kofman, the architect of the Leading Learning Communities program, a joint project
between the Organisational Learning centre at MIT and the American IS giant, EDS, states:
In order to create a new way of doing business, it is not enough to teach behavioural
techniques. It is necessary to explore the depths of the soul. Its about looking at our fears,
our anxieties, our unexamined desires, our attachments, our self importance, the control of
our ego, the little miseries that we create for each other every day out of mindless behaviour.
From that examination we move on to be able to touch all of that with love and compassion,
thereby dissolving whatever is stuck there.6 He goes on the to talk of the need for the
individual manager within the business entity to see her/him self as part of an interdependent
web, where the whole is greater than any one of the threads.
Top corporate America, Ford, General Motors, Du Pont, EDS etc, are confronting their
masks. Are Australian managers brave enough to remove their masks and to confront their
understandings of the meaning of life? For that is a prerequisite to building true learning
organisations. If not, they should forget this buzz word, and hope their ultimate meaning is
discovered before their organisation joins the inevitable march to oblivion.
___________________________________

6
De Rosia, M. Fred Kofman and the Leading Learning Communities Program. Living City, NY, May 1995. p.13-15.