Management System

Auditing
Assessment of auditing methods

David Hoyle 2012

Objective

David Hoyle 2012

To assess whether current auditing

methods will hold back or stimulate
developments to ISO 9001

Audit methods
Clause based
Department based
Contract based
Process based
Behaviour based

David Hoyle 2012

Determine
Select a what
you
are going
to
method
that
AlwaysAT
know
LOOK
and
fulfils
your
whatyou
youfind
areit
when
objectives
and
attempting
to
what
you
are
matches
the
establish
going to LOOK
information
and
relative
to
your
FOR
resources
objective
available. (NAO)

Approach
Examine the method
Identify assumptions
Evaluate the method

David Hoyle 2012

Clause based approach

ISO 9001

Select clause of the standard

Choose department
?

Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
David Hoyle 2012

Assess conformity with

requirements by

Audit
Report

Asking searching questions of each

person
Revealing evidence of conformity with
clauses of standard

Produce report of nonconformities

against clauses

Assumptions

David Hoyle 2012

Departments are exclusively responsible for meeting

certain clauses
Evidence of conformity in one Dept is indicative of
conformity in other Depts
Conformity with clauses is evidence of capability
Correcting non-conformity will improve system
effectiveness
People operate from the same causality as nature
and machines

Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.

David Hoyle 2012

Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?

Score on a
scale of 0-5

Clause based approach

ISO 9001

Select clause of the standard

Choose department
?

Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
David Hoyle 2012

Assess conformity
with requirements by

Audit
Report

Asking searching questions of each

person
Revealing evidence of conformity with
clauses of standard

Produce report of nonconformities

against clauses

Department based approach

Choose department
ISO 9001

Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
David Hoyle 2012

Select clauses of the standard

that apply
Assess conformity
?
with requirements by

Asking searching questions in each area

Following a trail through each
department
Revealing evidence of conformity with
clauses of standard

Audit
Report

Produce report of nonconformities

against clauses

Assumptions
Departments

Conformityare
with
clauses responsible
is evidence
exclusively

forof
meeting
certain clauses
capability

Evidence
of conformity
in one Dept
Correcting
non-conformity
willis

Correcting non-conformity will

indicative
conformity
in other Depts
improveofsystem
effectiveness
People operate from the same
causality as nature and machines

David Hoyle 2012

Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.

David Hoyle 2012

Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?

Score on a
scale of 0-5

Department based approach

Choose department
ISO 9001

Select clause of the standard

?

Objective
To determine
the extent to
which the QMS
conforms to
requirements of
ISO 9001
David Hoyle 2012

Assess conformity
with requirements by

Asking searching questions in each area

Following a trail through each
department
Revealing evidence of conformity with
clauses of standard

Audit
Report

Produce report of nonconformities

against clauses

Contract/
Order

Contract based approach

Select random sample of contracts/orders and

critical characteristics
Obtain or produce flow chart of activities
from contract/order to fulfilment

?
Objective

David Hoyle 2012

Asking searching questions at each stage

Following a trail through each process
from input to output
Revealing evidence of conformity with
customer requirements and with clauses
of standard

ISO
9001

To determine
whether the organization
has the ability to
consistently provide
product that meets
customer requirements

Assess conformity with

contracts and clauses by

Audit
Report

Produce report showing the degree

of conformity with requirements

Assumptions

David Hoyle 2012

Conformity with clauses is evidence of

capability
Conformity with ISO 9001 will enable
consistent provision of conforming
products
Correcting non-conformity will improve
system effectiveness
People operate from the same causality as
nature and machines

Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.

David Hoyle 2012

Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?

Score on a
scale of 0-5

Contract/
Order

Contract based approach

Select random sample of contracts/orders and

critical characteristics
Obtain or produce flow chart of activities
from contract/order to fulfilment

?
Objective

David Hoyle 2012

Asking searching questions at each stage

Following a trail through each process
from input to output
Revealing evidence of conformity with
customer requirements and with clauses
of standard

ISO
9001

To determine
whether the organization
has the ability to
consistently provide
product that meets
customer requirements

Assess conformity with

contracts and clauses by

Audit
Report

Produce report showing the degree

of conformity with requirements

So whats the problem?

People dont operate from the same
causality as nature and machines
We think we can design organizations
in the same way we design machines
A management system is not a set of
documents

David Hoyle 2012

The circles of influence that

create
management
systems
Outputs are the results of processes
Satisfied
Dissatisfied
Stakeholders
Stakeholders

Demands

Business
environment

Produce

Influence
ineffectively
System of effectively
Organization
managedprocesses
processes
managed

Undesirable
Outputs
Desired
Outputs
Delivers

David Hoyle 2012

Organization
Organization
open
isisaan
closed
system
system

Reality check

David Hoyle 2012

Conformity is only part of the picture

People make choices
People are influenced by images, events and
their interaction with others
Many invisible risks are hidden in behaviour
The impact of behaviour is what effects
outcomes and capability

Alternative Approaches
Process management audit
Behaviour assessment

David Hoyle 2012

Mission,
Objectives
Measures

Process based approach

Identify organizations mission, objectives and
success measures
Identify processes and sub-processes that
achieve these objectives

Assess process effectiveness by

Asking searching questions at each
stage
Revealing how processes are being
managed
Revealing evidence of capability
against objectives and measures

Objective
To determine whether
the organizations
processes are being
managed effectively
David Hoyle 2012

ISO
9001

Audit
Report

Produce report showing the

effectiveness of the system of
processes

Demands
Inputs

A Process Audit

Plan
Production

Produce
Product

Deliver
Product

Resources

S
R
Q

Support
Product

T
Satisfied
Outputs

How do you

Demands

How do you
How do you
Process
Process
Improved
Improved
Improved
know
you
are
How do you
know
know
you
areit happen?
make
What are
you trying
objectives & efficiency
monitoring
performance
effectiveness
doing
the
right
you are doing it right?
to do?
Measures
thing?
in the best
way?
&
What will success
Process review
look like?

David Hoyle 2012

Process
improvement

Assumptions
The system as revealed through
systematic enquiry is the system that is
producing the results
The information presented by the
organization is legitimate and has not been
fabricated

David Hoyle 2012

Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.

David Hoyle 2012

Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?

Score on a
scale of 0-5

Mission,
Objectives
Measures

Process based approach

Identify organizations mission, objectives and
success measures
Identify processes and sub-processes that
achieve these objectives

Assess process effectiveness by

Asking searching questions at each
stage
Revealing how processes are being
managed
Revealing evidence of capability
against objectives and measures

Objective
To determine whether
the organizations
processes are being
managed effectively
David Hoyle 2012

ISO
9001

Audit
Report

Produce report showing the

effectiveness of the system of
processes

Behaviour based approach

Goals,
Objectives
Drivers

ISO
9001

Identify the system objectives and

performance drivers
Identify a range of behaviours as outcomes
Maturity
Grid
that reflect levels of maturities and cross
ref clauses and apply weighting
Identify people who experience what is
happening

Objective
To measure the
inherent level of risk
to optimum business
outcomes and
governance issues
caused by the impact
of everyday patterns
of behaviour
David Hoyle 2012

Mainframe

Audit
Report

Invite on-line 360 participation

based on the part they play
confidential
Run analysis engine to compute results

Produce report showing scored risks to

the achievement of objectives and
drivers

Fulfilment process results (part)

#

Outcome

Av Score

We know the objectives that have to be achieved and how our

performance will be measured

39.5

We know the activities that need to be carried out to achieve the

objectives

31.7

The necessary physical and human resources are provided when

needed

28.2

Provisions made to minimize risk are successful

28.5

Work commences on time

35.7

Work is executed in accordance with policies and plans

30.5

Work flows without unplanned interruption

41.6

When things go wrong we put them right

39.3

No work is released until found in conformity with all requirements

43.4

10

Outputs delivered on time

26.0

David Hoyle 2012

Maturity Grid for outcome 8

Outcome Which statement matches your experience the closest?
When
things go
wrong we
put them
right.

When things
go wrong we
put them
right as best
we can and
get on with
the job

We normally
correct
mistakes as
they occur
following our
procedures
but we
wouldnt
record these

Level

0.00

1.00

Clauses
linked

8.5.2 Corrective action

David Hoyle 2012

4.2.4 Control of records

Records
show our
procedures
have been
applied to
correct
mistakes
and
prevent
them
happening
2.00
again.

We usually
review all
mistakes
formally in
line with our
procedure
and take
effective
action to
prevent
them
3.00
recurring

Records of
all mistakes
show that
it is rare
for the
same
mistake to
happen
twice
5.00

Graphical Representation
By stakeholder

By clause

By business process

5
2

20

1 Process owner
2 Process workers
3 Process supplier
4 Process customer
David Hoyle 2012

20
40

40

8
20

40

1 Mission management
2 Resource management
3 Demand creation
4 Demand fulfilment

4 Quality management system

5 Management responsibility
6 Resource management
7 Product realization
8 Measurement, analysis &
improvement

Assumptions

David Hoyle 2012

The system as revealed through systematic

enquiry is the system that is producing the
results
Patterns of behaviour are lead indicators
of risk
Correlation between clauses and business
drivers relative to performance
The statements dont produce wildly
different interpretations

Evaluation
1.
2.
3.
4.
5.
6.
7.
8.
9.

David Hoyle 2012

Conformity?
Inherent risks?
Performance?
Capability?
Efficiency?
Effectiveness?
Improvements?
Confidence?
Use of resource?

Score on a
scale of 0-5

Behaviour based approach

Goals,
Objectives
Drivers

ISO
9001

Identify the system objectives and

performance drivers
Identify a range of behaviours as outcomes
Maturity
Grid
that reflect levels of maturities and cross
ref clauses and apply weighting
Identify people who experience what is
happening

Objective
To measure the
inherent level of risk
to optimum business
outcomes and
governance issues
caused by the impact
of everyday patterns
of behaviour
David Hoyle 2012

Mainframe

Audit
Report

Invite on-line 360 participation

based on the part they play
confidential
Run analysis engine to compute results

Produce report showing scored risks to

the achievement of objectives and
drivers

Confidence level
Clause based

Not for 3rd Party Audits

Department based Not for 3rd Party Audits

Contract based
Process based
Behaviour based
David Hoyle 2012

Acceptable for 2nd Party

Audits
Acceptable for 3rd Party Audits
Acceptable when combined
with other audit methods

Conclusion

Which methods will hold back or

stimulate developments to ISO 9001?
Clause based

Hold back

Department based Hold back

Contract based

Hold back

Process based

Stimulate

Behaviour based

Stimulate

David Hoyle 2012

For Further Details:

Behaviour Assessment contact
ian.rosam@the-hpo.com
Process Audit contact
hoyle@transition-support.com
Thank you and have a safe journey home
David Hoyle 2012