Fluke Calibration

Web Seminar Series

Measurement Decision Risk and

Decision Rules in the new ISO/IEC
17025

February 27, 2019

©2019 Fluke Corporation 1

ISO/IEC 17025:2017

• Your Presenter:
• Jeff Gust
• Chief Corporate Metrologist

• Your Host:
• Bruce Fuller

©2019 Fluke Corporation 2

Abstract

This webinar introduces the requirements of ISO/IEC

17025:2017 regarding making statements of conformity to
specifications, provides an overview of what decision rules
are and how much risk is associated with each type and
recommends some easy to implement decision rules that
optimize the risk for the customer.

©2019 Fluke Corporation 3

When I ask for my instrument to be
calibrated…
• What do I expect to get?
• A sticker?
• Low cost (preferably free)?
• Repair?
• An adequate procedure is used?
• (Accredited) certificate of calibration?
• What does calibration Mean?
• Adjustment to nominal?
• Measurement data?
• Uncertainty?
• Measurement Traceability?
• In or out of tolerance information?
• We need to use common definitions for this important term...

©2019 Fluke Corporation 4

Language of Metrology

• Let’s all try to use the same language, the

International Vocabulary of Metrology (VIM)
• The VIM is available for free download at
http://www.bipm.org/en/publications/guides/
• For ease of understanding, some of the terms used
in this presentation are not always exact definitions
but are consistent with the VIM definitions

©2019 Fluke Corporation 5

Calibration Defined
• Calibration - operation that, under specified conditions, in a
first step, establishes a relation between the quantity
values with measurement uncertainties provided by
measurement standards and corresponding indications
with associated measurement uncertainties and, in a
second step, uses this information to establish a relation for
obtaining a measurement result from an indication
• Jeff’s Translation – calibration is a measured value with an
associated uncertainty of measurement
• Calibration is not adjustment!
• Calibration is not “In Tolerance” or “Out of Tolerance”
• If you are performing calibration, you need to know about
uncertainty

©2019 Fluke Corporation 6

Statements of conformance

• Verification - provision of objective evidence that a given

item fulfills specified requirements
• Review the calibration data, compare to a specified
requirement (i.e. manufacturer specification) to determine
In Tolerance or Out of Tolerance condition
• If you need the measurements and uncertainty, ask for
calibration
• If you need a statement of conformance (in/out of
tolerance info), ask for calibration and verification
• Maybe best call this “Calibration Service”

©2019 Fluke Corporation 7

ISO/IEC 17025 Today on Conformance

• (reporting the results) 5.10.4.2 When statements of

compliance are made, the uncertainty of measurement shall
be taken into account

• Changed from using the term compliance to conformance

• The term compliance is used for legal requirements,
conformance is used for statement of suitability

©2019 Fluke Corporation 8

ISO/IEC 17025:2017

• (Review of requests, tenders and contracts) 7.1.3 When a

customer requests a statement of conformity to a
specification or standard for the test or calibration (e.g.
pass/fail, in-tolerance/out-of-tolerance) the specification or
standard, and the decision rule shall be clearly defined.
Unless inherent in the requested specification or standard,
the decision rule selected shall be communicated to, and
agreed with, the customer

©2019 Fluke Corporation 9

DIS ISO/IEC 17025

• (Reporting of results) 7.8.6.1 When a statement of

conformity to a specification or standard is provided, the
laboratory shall document the decision rule employed,
taking into account the level of risk (such as false accept
and false reject and statistical assumptions) associated with
the decision rule employed and apply the decision rule.

©2019 Fluke Corporation 10

DIS ISO/IEC 17025
• 7.8.6.2 The laboratory shall report on the statement of
conformity such that the statement clearly identifies:
a) to which results the statement applies; and
b) which specifications, standard or parts thereof are met or
not met;
c) the decision rule applied (unless it is inherent in the
requested specification or standard).
• NOTE For further information see ISO/IEC Guide 98-4.
• Statements of conformity are in addition to reporting
measurement data and uncertainty of measurement

©2019 Fluke Corporation 11

What’s a decision rule?

• (Terms and Definitions) 3.7 – Decision rule: a rule that

describes how measurement uncertainty is accounted for
when stating conformity with a specified requirement

©2019 Fluke Corporation 12

Why are decision rules important?

• When performing a calibration and subsequently

making a statement of conformance with an
identified metrological specification (such as in or
out of tolerance to manufacturer’s specifications)
there are two possible outcomes
• You are right
• You are wrong
• Each measurement has an associated uncertainty,
and it is the uncertainty that can affect your chance
of being right or wrong

©2019 Fluke Corporation 13

Measurement Decision Risk

• If your uncertainty is large compared to the specified

requirement, how confident are you in your declaration of In
Tolerance or Out of Tolerance?

©2019 Fluke Corporation 14

Measurement Decisions

• When an instrument is declared in tolerance but the

instrument is actually out of tolerance, it is referred to as a
false accept
• When an instrument is declared out of tolerance, but the
instrument is actually in tolerance, it is referred to as a false
reject
• False rejects cost money by requiring unnecessary
adjustment, repair or replacement
• False accepts cost everybody money by putting out of
tolerance test equipment in the field
• Insert horror story here….

©2019 Fluke Corporation 15

Measurement Decisions Involve Risk

• The chance of making a false accept or false reject can be

understood as a probability, which is usually expressed as a
percentage
• When a customer requests a calibration they
• Usually don’t worry about false reject risk
• May specify a maximum acceptable false accept risk
• How much false accept risk is too much?
• 1%?
• 5%?
• 10%?
• 33%?

©2019 Fluke Corporation 16

Controlling risk

• Measurement decision risk can be controlled through the

application of Decision rules
• Two types of Decision Rules
• Rules that control risk by specifying a minimum uncertainty
(3:1 or 4:1 rule)
• Rules that controls (consumer) risk by testing to a value that
is smaller than the specification (guardbanding)

©2019 Fluke Corporation 17

N:1 rules

• Evaluate the specification you are calibrating to

• Evaluate the uncertainty associated with the measurement
(calibration)
• Determine the Test Uncertainty Ratio

4:1 TUR ≈ 0.9 % FA, 1.6 % FR*

3:1 TUR ≈ 1.1% FA, 2.3% FR
2:1 TUR ≈ 1.5% FA, 4.3% FR
1:1 TUR ≈ 2.2% FA, 13.4% FR

*Evaluated using joint probability, 95% EOPR

©2019 Fluke Corporation 18

Guardbanding

• Decreases False accept risk by developing a “test limit” that

is less than the specification limit

False
Accept
Excess risk

Risk
Acceptable risk

Test
Limit

Specification
Limit

©2019 Fluke Corporation 19

Guardbanding

STD
Indeterminate
Region

Indeterminate
Region
UUT

-SL -TL Nom Meas +TL +SL

©2019 Fluke Corporation 20

Guardband Selection Methods

• For a 2:1 TUR

• Root difference of Squares
• ≈ 0.6 % FA, 8.2 % FR
TL = SL − U
2 2
95%

• ISO 14253-1 (U95)

• ≈ 0.1 % FA, 35 % FR TL = SL −U 95%
• Dobbert method TL = SL −U 95%×M
• ≈ 2 % FA, 10 % FR
• Zero Guardband TL = SL
• ≈ 1.5 % FA, 4.3 % FR
• All evaluated with joint probability model
• ISO 14253-1 evaluated by conditional probability 2.5% FA

©2019 Fluke Corporation 21

Jeff’s philosophies on decision rules
• People don’t pay us a lot of money to receive an indeterminate
result
• Although the true value can exist anywhere within the uncertainty
bars, the best estimate of the true value is the measurement itself
• If a measured value is within the tolerance limits, it is highly likely
that it is in tolerance
• If the measurement is between the guardband and the tolerance limit, better
to use conditional pass over indeterminate
• If we can have sufficiently small risk without using a guardband,
this is the best approach
• If we have to guardband, use a method:
• That optimizes false accept and false reject probabilities
• Is easy to apply

©2019 Fluke Corporation 22

Conclusions

• The customer and the calibration lab need to communicate

in order to understand what is provided by the calibration
service
• Clarify if a statement of conformance is needed or not
• Agree on a decision rule and communicate the level of risk
associated with it
• Report the information on the certificate of calibration
• Lots of NCSLI conference papers available on the subject
• 173 (Metrology Practices) committee has lots of experts
that can help
• NCSLI Workshop and Symposium, Cleveland Aug 24-29
2019, T-2 and T-18, T-27

©2019 Fluke Corporation 23

For Further Reading

• (BIPM) JCGM 106:2012 Evaluation of measurement data –

The role of measurement uncertainty in conformity
assessment
• Deaver, Dave “How to Maintain Your Confidence (in a
World of Declining Test Uncertainty Ratios)
• Deaver, Dave “Guardbanding with Confidence”
• Deaver, Dave “Guardbanding and the World of ISO Guide
25 Is There Only One Way?”
• Dobbert, Michael “A Guard Band Strategy for Managing
False Accept Risk”
• NCSLI RP-18 Estimation and Evaluation of Measurement
Decision Risk

©2019 Fluke Corporation 24

Thank You for Attending!

Jeff Gust
Chief Corporate Metrologist
Fluke Corporation
6920 Seaway Blvd
M/S 275G
Everett, WA 98203
Direct: +1 425 446 5471
Email: jeff.gust@flukecal.com
Web: www.fluke.com www.flukecal.com
Twitter: @Jeff_Gust

©2019 Fluke Corporation 25