Comparing Two Measurement Devices

0011 0010 1010 1101 0001 0100 1011

Brian Novatny 2003

Background
0011 0010 1010 1101 0001 0100 1011

Each measurement made by an instrument or measuring device consists of the true, unknown level of the characteristic or item measured plus an error of measurement. In practice it is important to know whether or not the variance in errors of measurement of an instrument, or the imprecision of measurement, is suitably small as compared to the variance of the characteristic or product measured, or the product variability.

Background
0011 0010 1010 1101 0001 0100 1011

For efficiency of the measuring process, the variance in errors of measurement should be several or many times smaller than the variability of the characteristic measured or product variance. High measurement error causes the power of most statistical tests to decrease unless compensated for by larger sample sizes
Power = 1 - Beta Beta is the Type II error

Measurements are indicated by

0011 0010 1010 1101 0001 0100 1011

Device 1 = B1 + Xi + Ei1 where B1 = bias for device 1 Xi = true value Ei1 = random errors for device 1

Device 2 = B2 + Xi + Ei2

Regression Approach
0011 0010 1010 1101 0001 0100 1011

Y = b0 +b1*X Device 1 = Intercept + Slope * Device 2 Intercept value should be zero, if not, it indicates bias of the two devices Slope term should be around 1 indicating Similarity Mean Square Error estimate Precision R-Square term estimates some measure of strength

Problems with Regression

0011 0010 1010 1101 0001 0100 1011

The X variable (independent variable) should be measured without error

this is never the case, but errors in the X variable should be small, and it wont be when comparing devices they will be on equal footing

Asymmetry - specifically designate one device to predict the other

prefer symmetric approach where is doesnt matter which variable is the input and which is the output

Inverse regression can pose several problems when trying to resolve the asymmetry problem

Problems with Regression

0011 0010 1010 1101 0001 0100 1011

What should the R-square value be?

No objective justification

How to handle the case when there are multiple measuring devices?
Pairwise comparisons multiple testing error problem

Some Solutions
0011 0010 1010 1101 0001 0100 1011

Grubbs model

uses sums and differences

Pittman and Morgan along with Maloney and Rastogi

provide proofs and refinements to Grubbs model

Blackwood and Bradley

multivariate test on bias and precision

Tan and Iglewicz modify the standard regression approach based on Mandels work of Errors in Variables

Grubbs Model
0011 0010 1010 1101 0001 0100 1011

Involves calculating the Sums and Differences of the two devices The differences will be used to estimate bias
standard paired t-test with n-1 degrees of freedom

Performing a correlation analysis analysis on the sums and differences is used to estimate precision of the two devices
follow students t distribution with n-2 degrees of freedom

Provides two independent tests

one for bias of the two measurement devices one for precision equivalency

Simultaneous Test for Precision and Bias

0011 0010 1010 1101 0001 0100 1011

Multivariate approach instead of independent tests Differences are regressed on the Sums
Difference = Intercept + Slope * Sums

Model F-test uses the UNCORRECTED SUMS OF SQUARES to get the correct number for the df

instead of the familiar corrected sums of squares the overall Type I error (Alpha) rate is exact

Simultaneous Test for Precision and Bias

0011 0010 1010 1101 0001 0100 1011

If the Model F indicates significance, then tests for the Bias and Precision are just the individual F-tests

the overall test is generally more powerful it can reject the equivalence assumption of the two devices even though each individual test does not

Simultaneous Test for Precision and Bias

0011 0010 1010 1101 0001 0100 1011

The Precision test is exact The Bias test is exact only when the precision between the two devices is equal

use the paired difference t-test otherwise more powerful Uses UMVU (uniform minimum variance unbiased) estimate of the variance

Advantage of Simultaneous Test

0011 0010 1010 1101 0001 0100 1011

Type I Error Rate is exact Overall test could reject even though individual test do not
power of test

Statistical modeling
usual array of diagnostics
residuals

Regression approach
0011 0010 1010 1101 0001 0100 1011

Regression can still be used, but an adjustment to the model must be made In Simple Linear Regression, the results (Beta Hat) are achieved by minimizing the sum of squared residuals in the direction of the dependent variable The correction is to achieve Beta hat by minimizing the sum of squared residuals in the direction of -Lambda/Beta hat

Regression approach
0011 0010 1010 1101 0001 0100 1011

Lambda is a calculated value to adjust the slope calculations

called the Precision Ratio
ratio of the machines repeatability machine 1 / machine 2 or inverse

Lambda is determined by performing the standard Gauge RxR studies or taking repeated values

Regression approach
0011 0010 1010 1101 0001 0100 1011

The approach still uses regression and Lambda value helps solve the asymmetry problem
as lambda approaches infinity, implies X values approach 0, so Beta hat is Sxy/Sxx, which is where X is the independent variable as lambda approaches 0, implies Y values approach 0, so Beta hat is Syy/Sxy, which is where Y is the independent variable

Regression approach
0011 0010 1010 1101 0001 0100 1011

Approach handles precision but not bias Uses Polar coordinates for confidence intervals
Slope = TAN (Theta) Intercept = Tau/COS (Theta)

Example
0011 0010 1010 1101 0001 0100 1011

Device1 5.00 5.17 5.17 5.00 8.50 5.67 8.00 9.00 8.50 11.17 9.00

Device2 4.73 4.83 4.63 4.37 7.03 4.50 7.03 7.93 7.50 9.57 7.70

Sum 9.73 10.00 9.80 9.37 15.53 10.17 15.03 16.93 16.00 20.73 16.70

Diff 0.27 0.33 0.53 0.63 1.47 1.17 0.97 1.07 1.00 1.60 1.30

Standard Regression Analysis

0011 0010 1010 1101 0001 0100 1011

R-Square = 98.24% SQRT MSE = 0.31 Device 2 = -0.28 + 1.2 * Device 1 95% Confidence Intervals
Intercept (-1.07,0.52) Slope (1.07,1.31)

Conclusion
No Bias, but not similar

Simultaneous Test for Bias and Precision

0011 0010 1010 1101 0001 0100 1011

Regress Diff on Sums (Uncorrected SS) Model F = 73.22 ==> p-value = 0.0000027 Therefore, devices are different relative to their bias and precision Individual Precision Test
F = 17.5 ==> p-value 0.0024

Individual Bias Test (not exact)

F = 128.93 ==> p-value = 0.0000012

Paired t-test has a T value = 6.98 and pvalue less then 0.00001

Correct Conclusion
0011 0010 1010 1101 0001 0100 1011

The two devices measure differently Strong Bias

Strong lack of precision (repeatability)

Potential Problems
0011 0010 1010 1101 0001 0100 1011

Methods do not account for a difference in Gain, or slope of devices Devices might measure equally well or poor at the low and high ends of the scale, but the relationship is not constant
collect data at one end of the data range power of the test could be compromised

Multiple Measuring Devices

0011 0010 1010 1101 0001 0100 1011

Grubbs and others propose technique for three measuring devices

comparisons when one device is a Standard with three devices, get a more powerful test

Multivariate methods lead to fuller choice of sub-hypothesis and can be used regardless of the number of measurement devices

Multiple Measuring Devices

0011 0010 1010 1101 0001 0100 1011

One method involves performing a multivariate regression on q-1 measurement devices

independent variable = mean of each part dependent variable = deviations from that mean

Independent variable is averaged each part across all the measurement devices Dependent variable is calculated by the differences of each value from that mean

Multiple Measuring Devices

0011 0010 1010 1101 0001 0100 1011

Generally have to fit a Full model and a Reduced model (intercepts only) Then compare the two models
usually through some matrix manipulation

Technique can be performed by most software packages that can perform MANOVA techniques

Authors Opinion
0011 0010 1010 1101 0001 0100 1011

As the title says, this is just my opinion and not based on any concrete proof
such as simulation studies

My preferred method of analysis would be the Multivariate approach using Blackwood and Bradleys Regression with the Uncorrected Sums of Squares
this procedure seems to have a more powerful test in finding differences eliminates the possibility of getting a negative variance, which Grubbs method could get

Authors Opinion
0011 0010 1010 1101 0001 0100 1011

With the Multivariate approach, there is a natural extension to testing more than two measuring devices

Of course, there is no reason to try both the multivariate approach and Grubbs approach since they are easily computed using standard data analysis techniques

Final Comments
0011 0010 1010 1101 0001 0100 1011

These methods are not to replace Gage RxR studies, but to evaluate two devices against each other

Each device should be tested for bias and repeatability and linearity as desired
corrective action should be taken as needed

Final Comments
0011 0010 1010 1101 0001 0100 1011

The test for Bias is only a test for agreement between the two devices, not a bias against a standard
both devices could be grossly off from the standard (but in the same direction and amount)

If there is a claim that one device is superior to another (better precision), these methods could prove the validity of the claim and provide the precision estimates

References for Two Device Comparison

0011 0010 1010 1101 0001 0100 1011

Grubbs, F.E. (1973). Errors of Measurement, Precision, Accuracy and the Statistical Comparison of Measuring Instruments , Technometrics Vol. 15 pp. 53-66 Bradley, E.L. and Blackwood, L.G (1991). An Omnibus Test for Comparing Two Measuring Devices, Journal of Quality Technology, Vol. 23 pp. 12-16 Tan, C.Y. and Iglewicz, B. (1999). MeasurementMethods Comparisons and Linear Statistical Relationship, Technometrics, Vol. 41 pp. 192-201

References for Multiple Device Comparison

0011 0010 1010 1101 0001 0100 1011

Christensen, R. and Blackwood, L.G (1993). Tests for Precision and Accuracy of Multiple Measuring Devices, Technometrics, Vol. 35 pp. 411-420

Bedrick, E.J. (2001). An Efficient Scores Test for Comparing Several Measuring Devices, Journal of Quality Technology, Vol. 33 pp. 96-102

0011 0010 1010 1101 0001 0100 1011

brian.novatny@us.michelin.com