Six Sigma in Measurement Systems:

Evaluating the Hidden Factory

OK
Inputs Operation Inspect First Time
Correct
NOT
Rework OK
Hidden Factory
Scrap

Time, cost, people

Bill Rodebaugh
Director, Six Sigma
GRACE

slide 1
 Objectives
The Hidden Factory Concept
What is a Hidden Factory?
What is a Measurement Systems Role in the Hidden
Factory?
Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization

slide 2
The Hidden Factory -- Process/Production
OK
Inputs Operation Inspect First Time
Correct
NOT
Rework OK
Hidden Factory
Scrap

Time, cost, people

What Comprises the Hidden Factory in a Process/Production Area?

Reprocessed and Scrap materials -- First time out of spec, not reworkable
Over-processed materials -- Run higher than target with higher
than needed utilities or reagents
Over-analyzed materials -- High Capability, but multiple in-process
samples are run, improper SPC leading to over-control

slide 3
The Hidden Factory -- Measurement Systems
OK
Sample Lab Work Inspect Production
Inputs
NOT
Re-test OK
Hidden Factory
Waste
Time, cost, people

What Comprises the Hidden Factory in a Laboratory Setting?

Incapable Measurement Systems -- purchased, but are unusable
due to high repeatability variation and poor discrimination
Repetitive Analysis -- Test that runs with repeats to improve known
variation or to unsuccessfully deal with overwhelming sampling issues
Laboratory Noise Issues -- Lab Tech to Lab Tech Variation, Shift to
Shift Variation, Machine to Machine Variation, Lab to Lab Variation

slide 4
 The Hidden Factory Linkage
Production Environments generally rely upon in-
process sampling for adjustment
As Processes attain Six Sigma performance they begin
to rely less on sampling and more upon leveraging the
few influential X variables
The few influential X variables are determined largely
through multi-vari studies and Design of
Experimentation (DOE)
Good multi-vari and DOE results are based upon
acceptable measurement analysis

slide 5
 Objectives
The Hidden Factory Concept
What is a Hidden Factory?
What is a Measurement Systems Role in the Hidden
Factory?
Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization

slide 6
 Possible Sources of Process Variation
Observed Process Variation

Actual Process Variation Measurement Variation

Long-term Short-term Variation Variation due Variation due

Process Variation Process Variation w/i sample to gage to operators

Repeatability Calibration Stability Linearity

2 Observed Pr ocess 2 Actua l Pr ocess 2 Measurement System

2 Measurement System 2 Re peatability 2 Re producibility
We will look at repeatability and reproducibility as primary
contributors to measurement error

slide 7
 How Does Measurement Error Appear?

Actual process variation - 15

LSL USL
No measurement error

Frequency
10

30 40 50 60 70 80 90 100 110
Process

15

Observed process LSL USL

variation - 10

With measurement error

Frequency

30 40 50 60 70 80 90 100 110
Observ ed

slide 8
 Measurement System Terminology
Discrimination - Smallest detectable increment between two measured values
Accuracy related terms
True value - Theoretically correct value
Bias - Difference between the average value of all measurements of a sample and the
true value for that sample
Precision related terms
Repeatability - Variability inherent in the measurement system under constant
conditions
Reproducibility - Variability among measurements made under different conditions
(e.g. different operators, measuring devices, etc.)
Stability - distribution of measurements that remains constant and predictable over time for
both the mean and standard deviation
Linearity - A measure of any change in accuracy or precision over the range of instrument
capability

slide 9
 Measurement Capability Index - P/T
Precision to Tolerance Ratio

. * MS
515
P/T Usually
Usually expressed
as
expressed
percent
as percent
Tolerance
Addresses what percent of the tolerance is taken up by
measurement error
Includes both repeatability and reproducibility
Operator x Unit x Trial experiment
Best case: 10% Acceptable: 30%
Note: 5.15 standard deviations accounts for 99% of Measurement System (MS) variation.
The use of 5.15 is an industry standard.

slide 10
 Measurement Capability Index - % GR&R

MS
%R & R x 100 Usually
Observed Pr ocess Variation Usually expressed
expressed
as percent
as percent

Addresses what percent of the Observed Process Variation is

taken up by measurement error
%R&R is the best estimate of the effect of measurement
systems on the validity of process improvement studies (DOE)
Includes both repeatability and reproducibility
As a target, look for %R&R < 30%

slide 11
 Objectives
The Hidden Factory Concept
What is a Hidden Factory?
What is a Measurement Systems Role in the Hidden
Factory?
Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization

slide 12
 Case Study Background
Internal Raw Material, A1, is necessary for Final Product production
Expensive Raw Material to produce produced at 4 locations Worldwide
Cost savings can be derived directly from improved product quality, CpKs
Internal specifications indirectly linked to financial targets for production costs are used to
calculate CpKs
If CTQ1 of A1 is too low, then more A1 material is added to achieve overall quality higher
quality means less quantity is needed this is the project objective
High Impact Six Sigma project was chartered to improve an important quality variable,
CTQ1
The measurement of CTQ1 was originally not questioned, but the team decided to study
the effectiveness of this measurement
The %GR&R, P/T ratio, and Bias were studied
Each of the Worldwide locations were involved in the study
Initial project improvements have somewhat equalized performance across sites. Small
level improvements are masked by the measurement effectiveness of CTQ1

slide 13
 CTQ1 MSA Study Design (Crossed)

Site 1 Lab Site 2 Lab Site 3 Lab Site 4 Lab

Site 1 Sample 1 Site 1 Sample 2 Site 2 Sample 1..

Op 1 Op 2 Op 3 6 analyses/site/sample
2 samples taken from each site
T1 T2 2*4 Samples should be representative
Each site analyzes other sites sample.
Each plant does 48 analyses
6*8*4=196 analyses
slide 14
 Gage name: Z-14 MSA

CTQ1 MSA Study Results (Minitab Output)

Date of study: JULY 2002
Surface Area Reported by: All Labs
Tolerance: 110
Misc:

Components of Variation Response By Sample

120 890
%Contribution
100
%Study Var
80 840
Percent

%Tolerance
60
40 790

20
740
0
Gage R&R Repeat Reprod Part-to-Part Sample 1 2 3 4 5 6 7 8

R Chart by Operator Response By Operator

100 CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3 890
Sample Range

840
50 UCL=52.45

790
R=16.05
0 LCL=0 740
0 Oper CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3

Xbar Chart by Operator Operator*Sample Interaction

900 CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3 900 Operator
CB1
CB2
Sample Mean

850 UCL=851.5 Average 850 CB3

Mean=821.3 LC1
800 LC2
LCL=791.1 800
LC3
V1
750
V2
0 Sample 1 2 3 4 5 6 7 8
V3
W1
W2

slide 15
 CTQ1 MSA Study Results (Minitab Session)
Source DF SS MS F P

Sample 7 14221 2031.62 5.0079 0.00010

Operator 11 53474 4861.27 11.9829 0.00000
Operator*Sample 77 31238 405.68 1.4907 0.03177
Repeatability 96 26125 272.14
Total 191 125058
%Contribution
Source VarComp (of VarComp) Sample, Operator,
& Interaction are
Total Gage R&R 617.39 90.11 Significant
Repeatability 272.14 39.72
Reproducibility 345.25 50.39
Operator 278.47 40.65
Operator*Sample 66.77 9.75
Part-To-Part 67.75 9.89
slide 16
 CTQ1 MSA Study Results
Mean
P/T Equal Variances Differences
Site %GRR R-bar
Ratio within Groups
(Tukey Comp.)
94.3
All 116 16.05 No (0.004) Only 1,2 No Diff.
(78.6 100)*
38.9
Site 1 29 7.22 Yes (0.739) All Pairs No Diff.
(30.0 47.6)
91.0
Site 2 96 17.92 Yes (0.735) Only 1,2 Diff.
(70.7 100)
80.0
Site 3 79 20.37 Yes (0.158) All Pairs No Diff.
(60.8 94.8)
98.0
Site 4 120 18.67 Yes (0.346) Only 2,3 No Diff.
(64.8 100)
*Conf Int not calculated with Minitab, Based upon R&R Std Dev
slide 17
CTQ1 MSA Study
DotplotsResults (Minitab Output)
of C16 by C17
(group means are indicated by lines)
Dotplot of All Samples over All Sites
890

840
C16

790

740

C17

WO SA
VF SA
CB SA

LC SA

Site 1 Site 2 Site 3 Site 4

slide 18
 CTQ1 MSA Study Results (Minitab Session)
Analysis of Variance for Site
Source DF SS MS F P
Site 3 37514 12505 26.86 0.000
Error 188 87518 466
Total 191 125032
Individual 95% CIs For Mean
Based on Pooled StDev
Level N Mean StDev -+---------+---------+---------+-----
Site 1 48 824.57 15.38 (---*---)
Site 2 48 819.42 22.11 (---*---)
Site 3 48 800.98 20.75 (---*---)
Site 4 48 840.13 26.58 (---*---)
-+---------+---------+---------+-----
Pooled StDev = 21.58 795 810 825 840

Site and Operator are closely related

slide 19
Per
40 790

CTQ1 MSA Study Results (Minitab Output)

20
740
0
Gage R&R Repeat Reprod Part-to-Part Sample 1 2
X-bar R ofbyAll
R Chart Samples for All Sites
Operator
100 CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3 890
Discrimination
Sample Range

Index840is 0,
50 UCL=52.45
however
790
can
R=16.05 probably see
0 LCL=0
differences
740
of 5
0 Oper CB1 CB2 C

Xbar Chart by Operator O

900 CB1 CB2 CB3 LC1 LC2 LC3 V1 V2 V3 W1 W2 W3 900
Sample Mean

850 UCL=851.5 Most850of the

Average
Mean=821.3 samples are
800
LCL=791.1
seen as
800
noise
750
0 Sample 1

slide 20
 50

Pe
CTQ10 MSA Study Results (Minitab Output)
Gage R&R Repeat Reprod Part-to-Part

X-bar R ofRAll Samples

Chart for Site 4
by WO OP
70 W1 W2 W3
60 UCL=60.99
Sample Range

50
40
30
20 R=18.67
10
0 LCL=0
0

Xbar Chart by WO OP
900 W1 W2 W3

UCL=875.2
Sample Mean

850
Mean=840.1

800 LCL=805.0

0
Mean differences are seen in X-bar area
Most of the samples are seen as noise
slide 21
Sampl CTQ1 MSA Study ResultsR=17.92
Process Linkage
Site 2 Example
0 LCL=0 760
0 LC OP LC1

Xbar Chart by LC OP LC OP*Sa

860 LC1 LC2 LC3 850
850 UCL=853.1
840
MSA Study
Sample Mean

840
830

Average
830
820
810
Mean=819.4 820
Results with
810
800 Mean = 819.4
800
790
LCL=785.7 790
780
0 I and MR Chart for TSA (t) Sample 1 2 3

1000
1
11 2002 Historical
Individual Value

1 1 1
1 1
900 55 UCL=899.2

4
2
6 22
22
6 6
662 62
2 22
Process
Mean=832.5
800
6 6 6
2
22 4 2
LCL=765.8
Results with
5
Mean = 832.5
1
1 1
700
Subgroup 0 100 200 300 400

150

1 1
Selected
1
11
1 1 Samples
1
are Representative
g Range

11
100 1 1
1
UCL=81.95
slide 22
 CTQ1 MSA Study Results Process Linkage
50 810
Perc
I and MR Chart for TSA (t)
0 Site 2 Example 760
Gage R&R Repeat Reprod Part-to-Part Sample 1 2 3 4 5 6 7 8
1000
1
R Chart by LC
1 1OP By LC OP
Individual Value

1 1 1
100 1 LC1 1LC2 LC3
900 55860 UCL=899.2
22 6 6
6 662 62
MSA Study Results
Sample Range

2 22
4 2 22
UCL=58.54 Mean=832.5
222 4
LCL=765.8with Range = 17.92,
800
50 2
6 6 6 810
5 1
Calc for Subgroup
1 1
R=17.92
700
0 LCL=0 760
Subgroup 0 100 200 300 400
0 LC OP LC1 LC2 LC3

Xbar Chart by LC
1 OP LC OP*Sample Interaction
150
860 LC1 1
1
LC2 LC3 850
2002 Historical LC O
1 1 11 1 UCL=853.1
Process L
Range

850 11 840
100 1 1 L
1
MovingMean

840
830 UCL=81.95

Average
Results with
830 L
820 Mean=819.4 820
50
Sample

810
Range = 25.08
810
800 R=25.08
2 800
790 2 2 2 LCL=785.7
0 22 222 LCL=0
780
0
2 2
Sample
790
1 2 3
Calc
4 5
for
6
pt7 to8 pt

When comparing the MSA with process operation, a large

percentage of pt-to-pt variation is MS error (70%) --- a
back check of proper test sample selection
slide 23
 CTQ1 MSA Study Results Process Linkage
Site 2 Example

Key issue for Process Improvement Efforts is When will we see

change?
Initial Improvements to A1 process were made
Control Plan Improvements to A1 process were initiated
Site 2 Baseline Values were higher than other sites
Small step changes in mean and reduction in variation will achieve goal
How can Site 2 see small, real change with a Measurement System with
70+% GR&R?

Use Power and Sample Size Calculator with and without impact
of MS variation. Lack of clarity in process improvement work,
results in missed opportunity for improvement and continued
use of non-optimal parameters

slide 24
 CTQ1 MSA Study Results Process Linkage
Site 2 Example
2-Sample t Test 2-Sample t Test
Alpha = 0.05 Sigma = 22.23 Alpha = 0.05 Sigma = 6.67
Sample Target Actual Sample Target Actual
Difference Size Power Power Difference Size Power Power
2 2117 0.9000 0.9000 2 192 0.9000 0.9011
4 530 0.9000 0.9002 4 49 0.9000 0.9036
6 236 0.9000 0.9002 6 22 0.9000 0.9015
8 133 0.9000 0.9001 8 13 0.9000 0.9074
10 86 0.9000 0.9020 10 9 0.9000 0.9188
12 60 0.9000 0.9023 12 7 0.9000 0.9361
14 44 0.9000 0.9007 14 5 0.9000 0.9156
16 34 0.9000 0.9018 16 4 0.9000 0.9091
18 27 0.9000 0.9017 18 4 0.9000 0.9555
20 22 0.9000 0.9016 20 3 0.9000 0.9095

Simulated Reduction of Pt to Pt variation by 70% decreases

time to observe savings by over 9X.
slide 25
 CTQ1 MSA Study Results Process Linkage
Site 2 Example
Benefits of An Improved MS
Realized Savings for a Process Improvement Effort
For A1, an increase of 1 number of CTQ1 is approximately $1 per ton
Change of 10 numbers, 1000 Tons produced in 1 month (832 842)
$1 * 10 * 1000 = $10,000
More trust in all laboratory numbers for CTQ1
Ability to make process changes earlier with R-bar at 6.67
Previously, it would be pointless to make any process changes within the 22 point range. Would you really see
the change?
As the Six Sigma team pushes the CTQ1 value higher, DOEs and other tools will have greater
benefit

slide 26
 Objectives
The Hidden Factory Concept
What is a Hidden Factory?
What is a Measurement Systems Role in the Hidden
Factory?
Review Key Measurement System metrics including
%GR&R and P/T ratio
Case Study at W. R. GRACE
Measurement Study Set-up and Minitab Analysis
Linkage to Process
Benefits of an Improved Measurement System
How to Improve Measurement Systems in an
Organization

slide 27
 Measurement Improvement in the Organization
Initial efforts for MS improvement are driven on a BB/GB project basis
Six Sigma Black Belts and Green Belts Perform MSAs during Project Work
Lab Managers and Technicians are Part of Six Sigma Teams
Measurement Systems are Improved as Six Sigma Projects are Completed
Intermediate efforts have general Operations training for lab personnel, mostly laboratory management
Lab efficiency and machine set-up projects are started
The %GR&R concept has not reached the technician level
Current efforts enhance technician level knowledge and dramatically increase the number of MS projects
MS Task Force initiated (3 BBs lead effort)
Develop Six Sigma Analytical GB training
All MS projects are chartered and reviewed; All students have a project
Division-wide database of all MS results is implemented

slide 28
 Measurement Improvement in the Organization
Develop common methodology for Analytical GB training

slide 29
 Final Thoughts

The Hidden Factory is explored throughout all Six Sigma programs

One area of the Hidden Factory in Production Environments is
Measurement Systems
Simply utilizing Operations Black Belts and Green Belts to improve
Measurement Systems on a project by project basis is not the long term
answer
The GRACE Six Sigma organization is driving Measurement System
Improvement through:
Tailored training to Analytical Resources
Similar Six Sigma review and project protocol
Communication to the entire organization regarding Measurement System
performance
As in the case study, attaching business/cost implications to poorly performing
measurement systems

slide 30