Establishing the Integrity

of Data:
Measurement Systems Analysis

prepared by

Jeffrey T. Luftig, Ph.D.

W. Edwards Deming Professor of Management
Lockheed Martin Engineering Management Program
University of Colorado at Boulder
 Topics
 Measurement Scales and Types of
Data
 Establishing the
 Reliability and Validity of
Instrumentation, or
 Precision and Accuracy of
Instrumentation
Measurement Scales and Data
Measurement Scales and Data
 Measurement as a Process
 As in any process, regardless of the nature
of data collected or generated,
measurement systems must demonstrate
Stability through time, or control
Minimal variation as a proportion of
specifications, or capability
Minimal variation as a proportion of
process variation
 Measurement as a Process

Standard

Product or MEASUREMENT
Procedure Equipment Measurement
Process to be PROCESS
Measured

Operator
Ambient
Environmental
Characteristics
 Definition of Terms

 Reference Value
The theoretically or agreed upon correct
value of the characteristic being measured,
traceable to some standard

 Resolution
The smallest increment, or unit of measure,
available from a measurement process
Generally at least 1/10th of the specification
range
 Definition of Terms
 Precision
The degree of agreement (or variability)
between individual measurements or test
results from measuring the same
specimen(s)

 Accuracy (Bias)
The difference between the average of the
measurement error distribution and the
reference value of the specimen measured
Precision Precision vs. Accuracy

Accuracy
 Definition of Terms
 Repeatability
The variation in repeated measurements of the
same items with a single measurement system
Within appraiser/system variation
 Reproducibility
The variation in the average measurements by
different appraisers or systems measuring the
same items
Between appraiser/system variation
 Measurement Error
Distribution of repeated
Precision
measures on a single
- Repeatability
specimen or part
- Reproducibility

Accuracy
(Bias)

Reference Value
 Terms
 Linearity
The degree to which bias changes with
changes in the magnitude of the
characteristic measured

 Stability
The dependability, or consistency of the
measurement process over time
 Measurement
Systems Capability
 The variability resulting from measurement
error must not exceed a significant
proportion of the intended specifications
said to be capable
 In addition, it is not desirable for
measurement error to exceed a significant
proportion of the total process variability
 Capability is not the same as acceptability,
acceptability must be determined on a case
by case basis
Measurement Systems Capability
LSL Measurement USL
Error Distribution

 E   R pt   R pd
2 2

5.15E

(USL - LSL)
5 .1 5 E
% R & R  1 0 0
U SL  L SL
Measurement System Studies
 Potential Studies
Assess potential of a measurement
system to be capable over the long term
10 parts measured 2–3 times by one or
more appraisers
A “quick and dirty” study to find out if you
are in the ballpark
Assesses repeatability and reproducibility
Often called an R&R study
Measurement System Studies
 Potential Studies
Error Through Time
Tests of Between-Subjects Effects

Dependent Variable: ADAS ADA within Period

Type III Sum

Source of Squares df Mean Square F Sig.
Time Period 54.021 3 18.007 .876 .469

Specimen 325.873 7 46.553

Residual 431.604 21 20.553

Total 811.498 31

Bias Through Time

Tests of Between-Subjects Effects

Dependent Variable: MEASURE Gram Weight

Type III Sum

Source of Squares df Mean Square F Sig.
Time Period 21.844 3 7.281 .097 .961
Specimen 925.719 7 132.246
Residual 1571.406 21 74.829
Total 2518.969 31
 Measurement System Studies
Potential Studies (continued)

M e a n G r a m W e ig h t V a lu e s R e p e a te d T h r o u g h T im e
Gram Weight
4 6 4 .0
a,b
Tukey HSD

Time of Day Parts Subset

4 6 3 .5
Were Measured N 1
3 Early Afternoon 8 461.63
4 6 3 .0
1 Early Morning 8 463.13
4 Late Afternoon 8 463.25

4 6 2 .5 2 Late Morning 8 463.88

E s tim a te d M e a n s

Sig. .953

Means for groups in homogeneous subsets are displayed.

4 6 2 .0
Based on Type III Sum of Squares
The error term is Mean Square(Error) = 74.829.
4 6 1 .5
a. Uses Harmonic Mean Sample Size = 8.000.
E a r ly M o r n in g L a te M o r n in g E a r ly A f te r n o o n L a te A f te r n o o n

b. Alpha = .05.
T im e o f D a y P a r ts W e r e M e a s u r e d
 Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
 This result of the
previous analysis
allows us to
calculate the
average variance
of the repeated
measures, which
when we take its
 2 = 66.39
square root gives  = 8.15
us the estimate of
the standard
deviation due to
measurement
error:
 Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
 Using the estimate
of measurement
error, we can
calculate the
Precision-Tolerance
ratio, which in the
case of short-term
studies, should be
 2 = 66.39
less than 10%.
 = 8.15
Assuming the
engineering P/T = Precision-Tolerance Ratio =
tolerance for this = 6() / USL-LSL
process is 470 (USL) = 6(8.15) / 470 – 450
– 450(LSL) = 20: = 2.44
= 244% > 10% Requirement (S-T)
 Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
 Likewise, we can
estimate the
Accuracy (amount
of Bias) in the scale
by calculating the
average of the
differences between
the Means of the
Repeated Measures
 = -3.09
and the True Values Estimate Bias at 3.09 Grams; as
for the associated compared to the Precision estimate,
specimens: this is arguably an inconsequential
value.
Measurement System Studies
 Short-term Studies
25 parts measured 5-8 times by one or
more appraisers
A more thorough short-term assessment
 Long-term Studies
8-10 parts measured 25+ times by one or
more appraisers
Assesses through time stability
Measurement System Studies
 Long-term Studies
Measurement System Studies
 Long-term Studies
 Measurement Systems
Requirements
 Summary
 Regardless of the type of data gathered by an
instrument, and the assessment methodology
employed, the instrument or device utilized to
obtain criterion data must meet three
requirements before the experiment should
proceed:
 The instrument must be precise or reliable;
 The instrument must be accurate or valid; and
 The instrument should be / must be operating in a
state of statistical control.
 Sources and References
 The material used in the PowerPoint presentations associated with this course was drawn
from a number of sources. Specifically, much of the content included was adopted or adapted
from the following previously-published material:
 Luftig, J. A Quality Improvement Strategy for Critical Product and Process
Characteristics. Luftig & Associates, Inc. Farmington Hills, MI, 1991
 Spooner-Jordan, V. Understanding Variation. Luftig & Warren International, Southfield, MI
1996
 Luftig, J. and Petrovich, M. Quality with Confidence in Manufacturing. SPSS, Inc. Chicago,
IL 1997
 Littlejohn, R., Ouellette, S., & Petrovich, M. Black Belt Business Improvement Specialist
Training, Luftig & Warren International, 2000
 Ouellette, S. Six Sigma Champion Training, ROI Alliance, LLC & Luftig & Warren,
International, Southfield, MI 2005
 Luftig, J. An Overview of Total Quality Management, Luftig & Warren, International, 1992
 Luftig, J. Dr. Deming’s Theory of Profound Knowledge as a Foundation for Strategic
Planning and Policy Deployment, Luftig & Warren, International, 1997
 Luftig, J. and Jordan, V. Design of Experiments in Quality Engineering, McGraw-Hill/Irwin
Publishing Company, 1998
25