Risk Analysis in Calibration

Risk Analysis in Calibration 10/30/2016
Risk Analysis in Calibration
Presented by:
Dilip A. Shah
E = mc3 Solutions
(330) 328-4400
emc3solu@aol.com
Abstract
The 2015 revision of ISO 9001 does not
mention calibration. Does that mean that
the organization does not have to calibrate
their equipment anymore? Instead of
making a global requirement for calibration,
the new standard requires an organization
to make a risk based decision.This
presentation will provide a framework for
managing risk in the organization’s
equipment calibration program.
Risk Analysis in Calibration - E = mc3
10/30/2016 Solutions
E = mc3 Solutions - Reproduction Prohibited 1

Agenda
LEARNING OBJECTIVES:
 Learning Objectives:
 Managing Calibration Suppliers
 Managing Metrological Traceability and
Measurement Uncertainties
 Managing Probability of False Accept
(PFA)
 Managing Calibration Intervals

ISO 9001:2015
 ISO 9001:2015 incorporates what the
International Standard has termed “Risk-
based Thinking” in its requirements for
the establishment, implementation,
maintenance and continual improvement
of the quality management system.
 ISO 31000 provides guidelines on formal
risk management which can be
appropriate in certain organizations

How do you show risk-based thinking

during a certification audit?
 Analyze and prioritize the risks and
opportunities in your organization:
◦ What is acceptable?
◦ What is unacceptable?
 Then plan actions to address the risks.
Ask yourself:
◦ How can I avoid or eliminate the risk?
◦ How can I mitigate the risk?

How do you show risk-based thinking

during a certification audit?
Then …
 Implement the plan – take action
 Check the effectiveness of the actions –
does it work?
 Learn from experience – continual
improvement
However, this list presupposes that one has
identified risks and opportunities. So if one
hasn’t yet, how does one approach risk
identification in calibration?

What clauses of ISO9001:2015

apply to calibration?
7.1.5 Monitoring and measuring resources
7.1.5.1 General
The organisation shall determine and provide the resources
needed to ensure valid and reliable result when monitoring or
measuring is used to verify the conformity of products and
services to requirements.
The organization shall ensure that the resources provided:
a) are suitable for the specific of monitoring and measurement
activities being undertaken;
b) are maintained to ensure their continuing fitness for their
purpose.
The organization shall retain appropriate documented
information as evidence for purpose of the monitoring and
measurement resources.

9 Performance evaluation
9.1 Monitoring, measurement, analysis and evaluation
9.1.1 General
The organization shall determine;
a) what needs to be monitored and measured;
b) the methods for monitoring, measurement, analysis and
evaluation needed to ensure valid results;
c) when the monitoring and measuring shall be performed;
d) when the results from monitoring and measurement shall be
analysed and evaluated.
The organization shall evaluate the performance and the
effectiveness of the quality management system.
The organization retain appropriate documented information as
evidence of the results.


9.1.3 Analysis and evaluation
The organization shall analyse and evaluate appropriate data and
information arising from monitoring and measurement.
The results of analysis shall be used to evaluate;
a) conformity of products and services;
b) the degree of customer satisfaction;
c) the performance and effectiveness of the quality management
system;
d) if planning has been implemented effectively;
e) the effectiveness of actions taken to address risks and
opportunities;
f) the performance of external providers;
g) the need for improvements to the quality management system.
NOTE Methods to analyse data can include statistical
techniques.

9.3.2 Management review inputs
c) Information on the performance and effectiveness of the
quality management system, including trends in:
1) Customer satisfaction and feedback from relevant interested
parties;
2) The extent to which quality objectives have been met;
3) Process performance and conformity of products and
services;
4) Nonconformities and corrective actions;
5) Monitoring and measurement results;
6) Audit results;
7) The performance of external providers;
e) the effectiveness of actions taken to address risks and
opportunities (see 6.1 Planning);


10 Improvement
10.1 General
The organization shall determine and select opportunities for
improvement and implement any necessary action to meet customer
requirements and enhance customer satisfaction.
These shall include:
a) Improving products and service to meet requirements as well as to
address
future needs and expectations;
b) Correcting, preventing or reducing undesired effects;
c) Improving the performance and effectiveness of the quality
management
system.
NOTE Examples of improvement can include correction, corrective action,
continual improvement, breakthrough change, innovation and re-
organization.

10.2 Nonconformity and corrective action
10.2.1 when a nonconformity occurs, including any arising from complaints, the
organization shall:
a) react to the nonconformity and, as applicable:
1) take action to control and correct it;
2) deal with the consequences;
b) evaluate the need for action to eliminate the cause(s) of the nonconformity, in order
that it does not recur or occur elsewhere, by:
1) Reviewing and analysing the nonconformity;
2) Determining the causes of the nonconformity;
3) Determining if similar nonconformities exist, or could potentially occur;
c)Implement any action needed;
d) Review the effectiveness of any corrective action taken;
e) Update risks and opportunities determined during planning, if necessary;
f) Make changes to the quality management system, if necessary.
Corrective actions shall be appropriate to the effects Risk
of the nonconformities
Analysis in Calibration - E = mc3
encountered. 10/30/2016 Solutions


10.2.2 The organization shall retain
documented information as evidence of:
a) the nature of the nonconformities and
any subsequent actions taken;
b) the results of any corrective action.

Choosing the right calibration

standard for compliance.


http://www.bipm.org/en/publications/guides/

http://www.bipm.org/en/publications/guides/





What is Metrological Traceability?

 Traceability: Property of a measurement result whereby the
result can be related to a reference through a documented
unbroken chain of calibrations, each contributing to the
measurement uncertainty.
◦ NOTE 1 For this definition, a ‘reference’ can be a definition of a
measurement unit through its practical realization, or a measurement
procedure including the measurement unit for a non-ordinal quantity, or
a measurement standard.
◦ NOTE 2 Metrological traceability requires an established calibration
hierarchy.
◦ NOTE 3 Specification of the reference must include the time at which
this reference was used in establishing the calibration hierarchy, along
with any other relevant metrological information about the reference,
such as when the first calibration in the calibration hierarchy was
performed.
◦ NOTE 4 For measurements with more than one input quantity in the
measurement model, each of the input quantity values should itself be
metrologically traceable.

Metrological Traceability
BIPM
NATIONAL
METROLOGY
INSTITUTES
(NIST)
REFERENCE METROLOGY
LABORATORIES
WORKING METROLOGY
LABORATORIES
GENERAL CALIBRATION
PROCESS MEASUREMENTS


Test Accuracy Ratio (10:1)

BIPM
(100,000)
NATIONAL
METROLOGY
INSTITUTES (NIST)
(10,000)
REFERENCE METROLOGY
LABORATORIES (1000)
WORKING METROLOGY LABORATORIES

(100)
GENERAL CALIBRATION (10)
PROCESS MEASUREMENT (1)

Test Accuracy Ratio (4:1)

Needless to say:
TARs should be RIP ASAP!! BIPM
(1024)
NATIONAL
METROLOGY
INSTITUTES (NIST)
(256)
REFERENCE METROLOGY
LABORATORIES (64)

(16)
GENERAL CALIBRATION (4)
PROCESS MEASUREMENT (1)


Measurement Uncertainty is
Cumulative (for Metrological Traceability)
BIPM
(.005)
NATIONAL
METROLOGY
INSTITUTES (NIST)
(.011)
REFERENCE METROLOGY
LABORATORIES (.02)

(.05)
GENERAL CALIBRATION (.07)
PROCESS MEASUREMENT (.1)

Traceable Calibrations of
Instruments
 ISO/IEC 17025 Accredited
laboratories provide traceable
calibrations.
 HOW?
 Request Measurement Uncertainty
data with your accredited
calibration report.
 Accredited laboratories should have
ILAC Mutual Recognition
Arrangements (MRA).

Metrological Traceability:
Purchasing
When requesting traceable calibration service,
equivalent language such as mentioned below should
ALWAYS BE in the calibration purchase requisition to
protect the customer’s interest:
ISO 17025 Accredited Calibration with actual AS FOUND, AS
LEFT and ACTUAL MEASUREMENT UNCERTAINTY data is
required on all calibration certificates. Blanket statements of
compliance and statements of Measurement Uncertainty shall
not be accepted.
Calibration interval should not be specified on the PO unless

required for regulatory reasons (the supplier may not specify
calibration interval if not specified in the purchase order or
contract). The end user defines the calibration interval.
Accredited Laboratory
Risk Analysis inshall be- E accredited
Calibration = mc3 by ILAC Mutual
10/30/2016
Recognition Solutions
Arrangement (MRA) accrediting body.
5.4 Calibration intervals

Calibration intervals and/or process controls shall be
established for measuring and test equipment that are included in the calibration system to monitor
and maintain equipment performance to the stated application requirements.
The procedures for establishing and adjusting calibration intervals and process controls shall be
documented and utilized.
The calibration and other measurement process data shall be regularly monitored to evaluate the
achievement of the performance requirements.
The calibration intervals shall be adjusted to maintain the measurement performance
requirements.
5.4.1 Measuring and test equipment calibration intervals
Measuring and test equipment within the scope of the calibration system shall be calibrated at
periodic intervals established and maintained to assure acceptable measurement uncertainty,
traceability, and reliability.
The calibration interval process shall be free from conflicts of interest and systematically
applied.
The process shall be based on data resulting from the calibration of measuring and test equipment.
The process shall also include stated measurement reliability or measurement uncertainty
requirements and a method of verifying that the requirements are being attained.
Calibration intervals shall be reviewed regularly and adjusted when necessary to assure
continuous compliance of the specified measuring and test equipment performance requirements.
The calibration system shall include mandatory recall of measuring and test equipment to assure
timely recalibrations and preclude use of an item beyond its calibration due date.
All exemptions from periodic calibration shall be documented and authorized.
The recall system may provide for the temporary extension of the calibration due date for limited
periods of time under specified conditions, such as completion of a test or calibration that is in
progress, that do not unreasonably impair the satisfaction of the customer’s requirements.
NOTE: NCSLI RP-1-1996 should be used as a basis (Note: there is a new version of this
document released in 2010.) for the analytical process in the determination of calibration
intervals. Risk Analysis in Calibration - E = mc3

Available from
www.ncsli.org
New 2010
revision.
Previous
revision was
1996

Available from http://www.ilac.org/documents/ILAC_G24_2007.pdf


“Typical” Performance Specification:

At 10 Clucks: +/- 2 mClucks (aka +/- 0.002 Clucks) *
(on 0 – 20 Clucks scale and 0.01 Cluck resolution)
The Fine Print:
* Achieved if the equipment is used with left hand1 only, while standing on right foot2
only and the right eye3 closed, while maintaining a 23 0F +/- 0.033 0F environment4
using “hypertronic-wormhole TM“ temperature control5.
1 If used with right hand, the results will vary (and we won’t tell you by how much – we
used left handed technician)
2 If standing on left foot, you may get more tired and we won’t guarantee performance.
3 If right or both eyes are closed, you are on your own – Good Luck!
4 Do not attempt to use at -5 0C as we do not sanction its use on Celsius
scale. Do not even think about the Rankine scale (We really do not know what
Rankine is)!!!
5 Achieved once, never repeated. Fine tuning hammer used at times.We call it single
measurement bliss.
(Happiest place to work –looking for warm breathing bodies – will not train).
Happy Days Instrument Company (where Precision =Accuracy=Resolution or whatever)
Enjoy our fine, master crafted Instrument – knocking out one at a time!
Other Risk Management Tool:

FMEA


Purchasing
Check the website of accrediting bodies for the
accredited laboratories’ individual scope of
accreditation.
The first or last page (ISO 17025 certificate) is
“useless” after determining that the laboratory is
accredited.
It is the scope of accreditation that determines the
laboratory’s capability.
Ensure that the calibration laboratory’s process
measurement uncertainty is much less than the
your equipment’s specification
Scope of Accreditation Example


Purchasing
Check the website of accrediting bodies for the
accredited laboratories’ individual scope of
accreditation.
http://www.a2la.org
http://www.l-a-b.com/
http://anab.org/accredited-organizations/
http://www.iasonline.org/
http://www.nist.gov/nvlap/
http://www.pjlabs.com/search-accredited-labs

The International Picture

ILAC
EA APLAC
IAAC
ARAC
AFRAC
SADCA
EA European Cooperation for Accreditation

APLAC Asia Pacific Laboratory Accreditation Cooperation
ILAC International Laboratory Accreditation Cooperation
IAAC Inter-American Accreditation Cooperation
SADCA Southern African Development Community Accreditation
AFRAC African Regional Accreditation Cooperation
ARAC Arab Accreditation Cooperation
Unaffiliated Bodies Risk Analysis

Peer evaluated ABs who are not in Calibration
geographically - E = mc3
located in one of the
established regions

Definitions
 Measurement uncertainty,
Uncertainty of measurement,
uncertainty: Non-negative parameter
characterizing the dispersion of the
quantity values being attributed to a
measurand, based on the information used.

Measurement Uncertainty
References
 NIST 1297:1994 (free PDF download
from NIST web site)
 ISO Guide to Uncertainty of

Measurement (GUM)
◦ ISO Guide 98-3: 2008 - Replaces GUM
1995
 ANSI/NCSL Z540-2: 1997

(Americanized version of GUM)

Measurement Uncertainty
Graphically Expressed
Nominal Instrument
(True) Value Measurement
Uncertainty
-U +U
-Tolerance Measured +Tolerance

Value

Why is it important to understand

Measurement Uncertainty?
 Because it is a good practice.
Without knowing the uncertainty of
your measurement, you are not
measuring what you think you are
measuring.


Types of Errors
Type I - Type II Error
Calibration
In Tolerance Out Of Tolerance
(GOOD) (BAD)
(1-α) Calibration
β Type II Error
Called In Lab's Confidence
(Probability of
Tolerance - (Probability of
False Accept -
ACCEPT Correct Accept -
PFA)
PCA)
Decision
Made
(1- β) End User's
Called Out α Type I Error Confidence
of Tolerance (Probability of (Probability of
- REJECT False Reject - PFR) Correct Reject -
PCR)

Types of Errors
Type I - Type II Error
Calibration
In Tolerance Out Of Tolerance
(GOOD) (BAD)
(1-α) Calibration
β Type II Error
Called In Lab's Confidence
(Probability of
FalseSTOP
Tolerance - (Probability of
Accept -
ACCEPT Correct Accept -
PFA)
PCA)
Decision
Made
(1- β) End User's
Called Out α Type I Error Confidence
STOP of
of Tolerance (Probability (Probability of
- REJECT False Reject - PFR) Correct Reject -
PCR)


Risk Examples

Risk Examples


Risk Examples

Risk Examples


Risk Examples

ANSI/NCSLI Z540.3 Section 5.3(b)

requirements


Instrument Measurement Uncertainty

Guard Banding

Instrument Measurement Uncertainty

Guard Banding


Applying Risk Data

 Calibration reports
 Test reports
 Test Uncertainty Ratios (T.U.R.)
 Uncertainty data is cumulative
(hierarchical)

QUESTIONS?
FEEDBACK


Risk Analysis in Calibration

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Risk Analysis in Calibration

Transféré par

Droits d'auteur :

Formats disponibles

Risk Analysis in Calibration 10/30/2016

Risk Analysis in Calibration

E = mc3 Solutions - Reproduction Prohibited 1

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 2

How do you show risk-based thinking

Risk Analysis in Calibration - E = mc3

How do you show risk-based thinking

E = mc3 Solutions - Reproduction Prohibited 3

What clauses of ISO9001:2015

What clauses of ISO9001:2015

E = mc3 Solutions - Reproduction Prohibited 4

What clauses of ISO9001:2015

What clauses of ISO9001:2015

E = mc3 Solutions - Reproduction Prohibited 5

What clauses of ISO9001:2015

What clauses of ISO9001:2015

E = mc3 Solutions - Reproduction Prohibited 6

What clauses of ISO9001:2015

Risk Analysis in Calibration - E = mc3

Choosing the right calibration

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 7

Risk Analysis in Calibration - E = mc3

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 8

Risk Analysis in Calibration - E = mc3

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 9

What is Metrological Traceability?

Risk Analysis in Calibration - E = mc3

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 10

Test Accuracy Ratio (10:1)

WORKING METROLOGY LABORATORIES

GENERAL CALIBRATION (10)

PROCESS MEASUREMENT (1)

Risk Analysis in Calibration - E = mc3

Test Accuracy Ratio (4:1)

WORKING METROLOGY LABORATORIES

GENERAL CALIBRATION (4)

PROCESS MEASUREMENT (1)

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 11

WORKING METROLOGY LABORATORIES

GENERAL CALIBRATION (.07)

PROCESS MEASUREMENT (.1)

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 12

Calibration interval should not be specified on the PO unless

5.4 Calibration intervals

E = mc3 Solutions - Reproduction Prohibited 13

Risk Analysis in Calibration - E = mc3

Available from http://www.ilac.org/documents/ILAC_G24_2007.pdf

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 14

“Typical” Performance Specification:

Other Risk Management Tool:

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 15

Scope of Accreditation Example

Risk Analysis in Calibration - E = mc3

E = mc3 Solutions - Reproduction Prohibited 16

Risk Analysis in Calibration - E = mc3