Materi Sesi I General

Fundamentals of
uncertainty estimation
on measurement,
calibration and testing
Alexandre ALLARD
LNE, France
Laboratoire national de mtrologie et dessais
Outline
Introduction
The concept of measurement uncertainty
Reference documents
Basic statistical concepts
Step 1 : Analysis of the measurement process
GUM methodology
Supplement 1 to the GUM (Monte Carlo)
Case studies
Linear example
Mass calibration examples
Conclusion
Scope of the GUM/GUM-S1
Bayesian methods
General conclusion
20/05/2015
Workshop measurement uncertainty Jakarta
The concept of uncertainty

Uncertainty arises with any kind of measurements
Measurement result (VIM 2012) : set of quantity values being

attributed to a measurand together with any other available
relevant information
Note 2 : A measurement result is generally expressed as a single
measured quantity value and a measurement uncertainty.
20/05/2015
Why Must an Uncertainty in a Measurement

Result Be Indicated ?
GUM Introduction :
When reporting the result of a measurement of a physical
quantity, it is obligatory that some quantitative indication of
the quality of the result be given so that those who use it
can assess evaluate its reliability. Without such an
indication, measurement results cannot be compared, either
among themselves or with reference values given in a
specification or standard []
20/05/2015
Why Must an Uncertainty in a Measurement

Result Be Indicated ?
GUM 3.3.1 :
The uncertainty of the result of a measurement reflects the
lack of exact knowledge of the value of the measurand.
The result of a measurement after correction for
recognized systematic effects is still only an estimate of
the value of the measurand because of the uncertainty
arising from random effects and from imperfect correction
of the result for systematic effects.
20/05/2015
Usage : Comparison between Two Measurement

Results
Values
Laboratory B
Laboratory A
How to compare results from laboratories A and B ?

20/05/2015
Usage : Comparison between Two Measurement

Results
Values
Laboratory B
Laboratory A
The statement of a measurement uncertainty enables

to compare results from laboratories A and B ?
20/05/2015
Comparison between a Measurement Result and

a Specification
Manufacturing processes are never perfect and lead to a
dispersion of product specificities
Define a tolerance area in which the product is considered to be in
compliance with its specifications
Max. Tolerance
LImit
Min. Tolerance
Limit
Specification Area
20/05/2015

a Specification
Measurement Result
and Uncertainty
Max. Tolerance
Limit
Min. Tolerance
Limit
Specification Area
20/05/2015
Comparison between a Measurement Result and a

Specification
Neither conformity or non-conformity can be assessed
Measurement Result
and Uncertainty
Max. Tolerance
Limit
Min. Tolerance
Limit
Specification Area
20/05/2015
10

a Specification
Guard band
Min. Tolerance
Limit
Guard band
Acceptation Area
Max. Tolerance
Limit
Specification Area
The client benefits from the doubt

20/05/2015
11

a Specification
Guard band
Guard band
Acceptation Area
Min. Tolerance
Limit
Specification Area
Max. Tolerance
Limit
The producer benefits from the doubt

20/05/2015
12

a Specification
shared risk
shared risk
Acceptation Area
Min. Tolerance
Limit
Max. Tolerance
Limit
Specification Area
Client and producer share the risk (GUM-Conformity)

20/05/2015
13
Documentation - JCGM
Joint Committee for Guides in Metrology (JCGM)
WG1
GUM +
Supplements
20/05/2015
WG2
International
Vocabulary of
Metrology (VIM)
14
Documentation - VIM
VIM (JCGM 200:2012) : International Vocabulary of
Metrology Basic and general concepts and associated
terms, 2012.
Bilingual English/French document
Harmonization of the vocabulary usd in metrology

in order to have everyone concerned with
metrology all over the world use the same terms
for the same concepts.
20/05/2015
15
Documentation GUM + Supplements

GUM (JCGM 100:2008) : Guide to the expression of
uncertainty in measurement, 2008.
1st version published in 1994.
Minor modifications until the last version in 2008
GUM-S1 (JCGM 101:2008) : Supplement 1 to the Guide to

the expression of uncertainty in measurement
Propagation of distributions using a Monte-Carlo method,
2008.
Alternative method when the conditions for a valid application of the GUM
are not fulfilled
Requires a software
20/05/2015
16
Documentation GUM + Supplements
GUM-S2 (JCGM 102:2012) : Supplement 2 to the Guide to

the expression of uncertainty in measurement Extension
to any number of output quantities, 2012.
Deals with multiple measurands : common applications in chemistry,

electricity, dimensional measurements,
Extension of both GUM and GUM-S1 methodologies
GUM-Conformity (JCGM 106:2012) : The role of

measurement uncertainty in conformity assessment ,
2012.
20/05/2015
17
Terminology
Measurand (VIM 2012, 2.3) : quantity intended to be
measured
The measurand has a unique and unknowable true value
Uncertainty (VIM 2012, 2.26) : non-negative parameter
characterizing the dispersion of the quantity values being
attributed to a measurand, based on the information used
Standard uncertainty (VIM 2012, 2.30) : Measurement uncertainty
expressed as a standard deviation . Denoted as u or uc.

Expanded uncertainty (VIM 2012, 2.31) : product of a combined
standard measurement uncertainty and a factor larger than the
number one . Denoted as U.
Coverage interval (VIM 2012, 2.36) : interval containing the

set of true quantity values of a measurand with a stated
probability, based on the information available
20/05/2015
18
Terminology : Measurement error

Measurement error (VIM 2012, 2.16) : measured
quantity value minus a reference quantity value
The result of a measurement (after correction) can

unknowably be very close to the value of the measurand
(and hence have a negligible error) even though it may
have a large uncertainty. Thus the uncertainty of the
result of a measurement should not be confused with
the remaining unknown error.
20/05/2015
19

ISO 3534-1 : Statistics Vocabulary and symbols
General statistical terms and terms used in probability
Mean : Parameter that characterizes the location of a
series of measurements
Consider n repeated measurements
,,
+ +
Standard deviation : Parameter that characterizes the

dispersion of a series of measurements
=
20/05/2015
1
20
Variance : Parameter that characterizes the dispersion of

a series of measurements. Equals the squared standard
deviation
=
Interesting mathematical properties
Central in the Law of Propagation of Uncertainty (LPU GUM)
20/05/2015
21

Quantile (p-quantile) qp : value of x equal to the infimum of
all x such that
Useful for the determination of a coverage interval associated with a

Monte Carlo simulation
q0.50 is the median : half possible values of X are below the median and
the other half are over the median.
2.5%
q0.025
20/05/2015
22
Basic statistical concepts Estimation

Statistics are used when we dont know something : in
general, a sample is used to estimate properties related
to a larger population.
Surveys, conformity assessment, Risk analysis,
In metrology, we dont know the value of the measurand :

measurements are performed in order to obtain a sample.
The value of the measurand is then estimated using the
sample (the measured values).
Estimation (ISO 3534-1, 1.36) : procedure that obtains a
statistical representation of a population from a random
sample drawn from this population
20/05/2015
23
Basic statistical concepts Estimation
Population
,,
Data
Mean
Standard
deviation
20/05/2015
Sample
,,
1
1
24
Basic statistical concepts Probability

distribution
CONTINUOUS RANDOM VARIABLES
g ( x ) is the probability density function for X = x
probability de nsity function - continous v ariable
g(x)
-2
x
+
g ( x ) dx = 1
{x ; g(x)} the probability distribution of the random variable X

20/05/2015
25
Uncertainty of Measurement Evaluation : 4 steps
Step 3: Propagation of Uncertainty (LPU, MCM)
Step 2:
Quantification
of the sources
of uncertainty
of the xi
Standard
Uncertainty
or
distribution
Step 1: Analysis of the Measurement Process

Step 4:
Uncertain
Input
Quantities :
xi
Mathematical
Model
of the Measurement
Process
Measurand
Y = f(Xi)
Final
Expression of
the Result
yU
f(X1,..,Xn)
Step 3: Sensitivity Analysis, Prioritization
Feedback
20/05/2015
26
Step 1 : Analysis of the measurement

process
Detailed definition of the measurand Y
List of potential uncertainty sources
Selection of the most important uncertainty sources
, ! = 1, ,
Creation of the measurement function # :
$=#
20/05/2015
,,
27
Step 1 : The Measurand
quantity intended to be measured (VIM 2.3)
[] The specification of the measurand requires

knowledge of the kind of quantity, the description of the
phenomenon, the body or the substance carrying the
quantity []
The definition of the measurand may require indications

about quantities such as temperature, pressure, etc.
measurement conditions
reference temperature 20 C
the principle of measurement in physics
etc.
20/05/2015
28
Step 1 : Examples of Measurand Definitions
1)1) Distance
Distance between
centre and
upper side
entre the
le centre
dethe
la face
of the gauge and
joint plan,
20Csur
and at a
suprieure
de the
la cale
et leatplan
lequel
elle est adhre, 20 C et
vertical position.
en position verticale.
2)2) Distance
Distance between
centres
of the sides of
entre the
lestwo
deux
centres
des
facesatde
la cale,
20 is
C,
lahorizontal
the gauge,
20C.
The gauge
at a
cale
tant en position horizontale.
position.
3) Distance entre deux plans
3) parallles,
Distance between
plans,
at 20C.
20 two
C,parallel
la cale
tant
en
horizontale.
Theposition
gauge is at
a horizontal position.
20/05/2015
29
Step 1 : Measurement error

The errors are due to:
The instrumentation (standards, instruments, environment ...)
The measured object (measurand)
True value
(unknown)
systematic
random
error
Measurement error (VIM 2012, 2.16) : measured quantity value minus a
reference quantity value
Systematic measurement error (VIM 2012, 2.17) :
component of
measurement error that in replicate measurements remains constant or
varies in a predictable manner
Random mesurement error (VIM 2012, 2.19) : component of measurement
error that in replicate measurements varies in an unpredictable manner
20/05/2015
30
Step 1 : How can errors be reduced ?
Random errors are usually reduced by increasing the number of

independent observations (replicate measurements) and by taking
into account the mean of the values
Systematic errors are reduced by applying corrections
These two rules are the basis of the metrologists endeavour to give
the best possible measurand evaluation
20/05/2015
31
Step 1 : The Measurement Process
Do not focus your attention solely on the instrument but observe

the process that leads to the measurement result instead
The uncertainty characterizes the result, not the instrument
The following impact the result :
Operators
Instruments, calibrators
The measurement method and operating mode
The measurement conditions (temperature, pressure, etc.)
The measured object or material
20/05/2015
32
Step 1 : Why Must the Measurement Process be

Analyzed ?
Identification of the factors which influence the measurement result

(causes of error) to provide a list as exhaustive as possible.
So as to reduce their influence by:

Using corrections,
Repeating the measurements and calculating the arithmetical
mean out of the series of observations
To determine the measurement model associated to the

measurement process
20/05/2015
33
Step 1 : Full analysis of the measurement

process
The objective of a measurement is to determ ine the
measurand value []
As a consequence, a measurement begins with
an appropriate definition of the measurand
an appropriate definition of the measurement method
generic description of a logical organization of operations used in a

measurement
and an appropriate definition of the measurement procedure
detailed description of a measurement in according to one or more

measurement principles and to a given measurement method,
based on a measurement model and including any calculation
destined to obtain a measurement result
Measurement principle : Phenomenon serving as a basis of a measurement
Measurement model : Mathematical relation among all quantities known to be
involved in a measurement
20/05/2015
34
Step 1 : Measurement Modelling
In many cases, a measurand Y is not measured directly but is

determined from N other quantities X1, X2, XN through a functional
relationship f :
$=#
,,
The input quantities X1, XN represent all the usable data for the
calculation of the result.
The input quantities X1, XN which the output quantity depends on

can also be taken as measurands as well as depend on other quantities,
including, in the case of systematic effects, the corrections.
The f function does not merely express a physical law, but the
measurement process. More specifically, it must contain all the
quantities that contribute significantly to the final resulting uncertainty.
20/05/2015
35
Step 1 : Mathematical Model of an Atmospheric

Pressure Measurement with a Barometer
0
= % 1 + ( ) 20 cos /
1 + 1 ) 20
corrective term
barometer verticality
corrective term
gravity acceleration
dilatation
density of mercury at
measurement temperature
Reading of the rule
20/05/2015
36
Summary Step 1 : Analysis of the measurement

process
Define the measurand Y
Identify the input quantities Xi,,XN
Obtain the measurement function f :
$=#
,,
Main step for the evaluation of measurement uncertainty
20/05/2015
37
Outline
Introduction
Reference documents
GUM methodology
Case studies
Linear example
Conclusion
Bayesian methods
General conclusion
20/05/2015
38
Measurement uncertainty evaluation
Measurement Process
analysis
Reproducibility Tests
no
yes
Analytical Method
Reference
documents
Does the model

exist ?
Estimation of standard
uncertainties
Analytical
method
assumptions?
yes
ISO 5725
ISO GUM
ISO GUM
non
supplment 1
Variance Analysis
Law of Propagation
Monte Carlo Method
Result and
Uncertainty
"Validation" and update
20/05/2015
39

Propagation of Uncertainty (GUM)
x 1, u(x 1)
x 2, u(x 2)
Y = f(X1, X2, X3)
y, u(y)
y U (k=2)
x 3, u(x 3)
Propagation of Distributions (GUM-S1)
Y = f(X1, X2, X3)
y~ ;u ( y~)
[ylow;yhigh]
20/05/2015
40
Step 2:
Quantification
of the sources
of uncertainty
of the xi
Standard
Uncertainty u(xi)
or
distribution

Step 4:
Uncertain
Input
Quantities :
xi
Mathematical
Model
of the Measurement
Process
Measurand
Y = f(Xi)
Final
Expression of
the Result
yU
f(X1,..,Xn)
Step 3: Sensitivity Analysis, Hierarchisation
Feedback
20/05/2015
41
GUM Step 2 : Quantification of the standard

uncertainties
2
Experimental Data
Type A
No Experimental Data
A Priori Information
Type B
Presentation of the Statistical Tools Used by GUM : each

quantity is considered as a random variable
20/05/2015
42
GUM Step 2 Random variable

RANDOM VARIABLE [ ISO 3534-1,1.2]
o
A function which can be applied to any value of all potential results

and to which a distribution of probability is attached
note 1 : a r.v which can only be applied to isolated values is called

discrete. A r.v. which can be applied to all values within a finite or
infinite interval is called continuous .
note 2 : the probability of an event A is noted as Pr(A) or P(A)
Examples :
o
The result of a dice throw (discrete : possible values {1;2;3;4;5;6} )
The size of an individual chosen at random from a given population

(continuous)
A measurement result (continuous)
20/05/2015
43
GUM Step 2 : Statistical parameters

Mean or Expectation
o
Location parameter of a probability distribution/a sample
Linear operator
=4
Variance
Scale parameter : quantifies the spread of a distribution or a sample
Non linear operator
o
o
54
+ 2
<
5 56 789
If independent random variables

=
20/05/2015
44
GUM Step 2 : Type A

n repeated measurements under repeatability conditions (same
operator, apparatus, short time interval, independent
measurements)
Measure
Temperature of the
laboratory (C)
22.3
22.7
23.5
23.1
23.3
22.8
23.0
22.9
22.4
10
23.2
20/05/2015
Mean value
====
= = 22.92 7
Standard deviation
2 = =
1
1
= =
= 0.38 7
45
GUM Step 2 : Type B

Investigation of the available
knowledge for the input
quantity Xi
Transcription of that knowledge
in terms of a probability
distribution
Determination of the standard
uncertainty as the standard
deviation of the chosen
probability distribution
20/05/2015
Available
knowledge ?
Probability
distribution
Standard
deviation u(Xi)
46
GUM Step 2 : Type B (available information)

Where can you find available information ?
Available
knowledge ?
Calibration certificate (with an associated expanded uncertainty U)
Verification certificate
Resolution of a displaying device
Maximum Permissible Errors (MPE)
Bounds for an oscillating phenomenon
Expert knowledge
Physical limits
20/05/2015
47
GUM Step 2 : Type B (choice of the probability

distribution)
Common probability distributions in metrology :
Available information
Probability
distribution
Best estimate xi with an

associated expanded uncertainty
U (calibration certificate)
Gaussian
Resolution q of a displaying
device
Rectangular
Maximum Permissible Error (

MPE)
Rectangular
Oscillating phenomenon
between two values a and b
Arcsine (U-shaped)
Probability
distribution
Standard
deviation u(Xi)
20/05/2015
Standard deviation
D
=
E
12
F 4
3
G5
8
48
GUM Step 2 : Type B
o Interval of values
o A distribution of probability
-a
or
-a
20/05/2015
u(xi)
(GUM)
or S (Stats)
...
49
GUM Step 2 : Covariance/Correlation

The covariance may be either positive or negative :
Covariance between the outside temperature and the inside temperature ?
Positive : the two quantities vary in the same direction
Covariance between the outside temperature and the power consumption ?
Negative : the two quantities vary in opposite directions
The influence of the covariance on measurement uncertainty may be

positive or negative, depending on :
The sign of the covariance
The measurement model
The correlation coefficient r may be easier to interpret :
1 H 1
789
=H
.2
.2
It quantifies the intensity of the relationship between the variations of the

two involved quantities
20/05/2015
50
GUM Step 2 : intensity of the

Covariance/Correlation
35
Correlation between Tout and

Tins
35
25
25
15
15
5
15
20
25
Old building, single glazing

windows, bad thermal insulation :
r = 0.97
20/05/2015
Correlation between Tout and

Tins
15
20
25
New building, double glazing

windows, good thermal insulation : r =
0.55
51
GUM Step 2 : Calculation of the covariance

n Simultaneous observations of the two quantities X and Z.
1
J J
I89 , J =
Measure
Tin
Tout
1
1
22.3
23.2
22.7
15.0
23.5
23.6
23.1
24.6
23.3
25.3
22.8
15.8
23.0
19.2
22.9
20.6
22.4
20.3
10
23.2
22.2
Mean
22.92
20.98
Std dev
0.38
3.52
20/05/2015
I89 = , =KLM = 0.493

H
= 0.41
52
GUM Step 2 : Covariance arising from the

measurement model
Consider L independent random variables (r.v):
Q1, Q2, Q3,.., QL
X1 r.v (F) function of

previous r.v:
X2 r.v (G) function of

previous r.v:
X1= 2Q1-5Q3+Q4-3Q8
X2 =Q1+6Q2+Q8
u ( x 1 , x 2 ) = u ( F (Q 1 , Q 2 ,..., Q L ), G (Q 1 , Q 2 ,..., Q L )) =
i =1
F G
u 2 (Q i
Q i Q i
u ( x 1 , x 2 ) = u (2Q 1 - 5Q 3 + Q 4 - 3Q 8 , Q 1 + 6Q 2 + Q 8 ) = (2) (1) u 2 (Q1 ) + ( 3) (1) u 2 (Q8 )
Example : Two temperature measurements T1 and T2 with the same

sensor/thermometer :
o
The trueness error is the same for the two temperatures
T1 and T2 have a common uncertainty component : the trueness of the sensor

20/05/2015
53
Step 2:
Quantification
of the sources
of uncertainty
of the xi
Standard
Uncertainty u(xi)
or
distribution

Step 4:
Uncertain
Input
Quantities :
xi
Mathematical
Model
of the Measurement
Process
Measurand
Y = f(Xi)
Final
Expression of
the Result
yU
f(X1,..,Xn)
Feedback
20/05/2015
54
GUM Step 3 : Law of Propagation of

Uncertainty (LPU)
$=#
,,
,,
Uncertainty --> deviations from the expectation values

o Y = Y - E(Y)
o Xi = Xi - E(Xi)
o E(Y) and E(Xi) being constant V(Y )=V(Y) and V(Xi )=
V(Xi)
Hypothesis 1 : all deviations are small enough for order
1 of the Taylor development to be applied exclusively:
Error Propagation Formula
20/05/2015
55
GUM Step 3 : Law of Propagation of

Uncertainty (LPU)
Hypothesis 2 : all partial derivatives are constant around the point
applying the formula of the variance of a sum, the result is :
Variance Propagation Formula

or Law of the Propagation of Uncertainty (GUM)
20/05/2015
56
GUM Step 3 : Law of the Propagation of

Uncertainty (LPU)
2
N 1 N
f
f f
2
(
)
u c2 (y ) =
u
x
+
2
u xi, x
i
i =1 j = i +1 x i x j
i =1 x i
N
2 2
c
i u (x i ) + 2
i =1
ci
i =1 j = i + 1
N 1
u c2 (y ) = c i2u 2 ( x i ) + 2
i =1
u 2 (y ) =
c j u xi , x j
N 1 N
u c2 (y ) =
c c u (x ) u (x )r (x , x )
N
i =1 j = i +1
N 1
i =1
i =1 j = i +1
2 2
c
i u ( xi ) + 2
ci c j u (xi , x j )
Revised GUM
20/05/2015
57
GUM : Uncertainty budget/Sensitivity analysis

In the case of independent input quantities
2 $ =
I 2
Sensitivity coefficient associated to the input quantity Xi.

O#
I =
O
Sensitivity = Contribution of the input quantity Xi to the overall
measurement uncertainty
I 2
=
2 $
Identification of the most influential input quantities in order to :
Reduce the uncertainty associated to the output quantity by the reduction of
the uncertainty associated to the most influential input quantities
Fix the non influential input quantities to their best estimate
20/05/2015
58

Step 4:
Step 2:
Quantification
of the sources
of uncertainty
of the xi
Standard
Uncertainty u(xi)
or
distribution
Uncertain
Input
Quantities :
xi
Mathematical
Model
of the Measurement
Process
Measurand
Final
Expression of
the Result
yU
Y = f(Xi)
f(X1,..,Xn)
Feedback
20/05/2015
59
GUM Step 4 : Final expression of the result
y U unit (value of k )
Expanded uncertainty
Coverage interval
20/05/2015
Coverage factor
P Q; P + Q
60
GUM Step 4 : Definitions
Standard uncertainty (GUM 2.3.1)
Expanded uncertainty (GUM 2.3.5)
uncertainty of the result of a
Quantity defining an
measurement
around
expressed
as
standard deviation
about
measurement
the
interval
result
that
may
of
a
be
expected to encompass a large

fraction of the distribution of values
u(y)
that could reasonably be attributed

to the measurand
k : coverage factor
U = k u(y)
20/05/2015
61
GUM Step 4 : Expanded uncertainty
U = k uc(y)
Expanded
uncertainty
Coverage Factor
Combined standard
uncertainty of y
(Step 3 Result)
(GUM 2.3.6) numerical factor used as a multiplier

of the combined standard uncertainty in order to obtain
an expanded uncertainty
(VIM 3 2.38) number larger than 1 by which a combined standard
measurement uncertainty is multiplied to obtain an expanded
20/05/2015
62
GUM Step 4 : Determining the coverage factor

Model y = f(x1, x2, , xN)
Does Y have a gaussian distribution or approximative
gaussian ? (Valid assumptions of Central Limit
Theorem (CLT))
Yes
k =2
So that p =
p 95%
compute
No
(1 )%
eff
(dof)
k = Student Quantile eff
(1 )%
k=2
So that p =
p=?
Monte Carlo
Simulations
See GUM
20/05/2015
63
GUM Step 4 : Determining the coverage factor

The Central Limit Theorem (CLT)
The Central Limit Theorem: any sum of a large number of independent
random variables, identically distributed, is a normal asymptotic variable
Corollary : the only necessary condition for its application is that regardless
of the distributions of the different variables, no variable or group is to be
predominant compared to the others, within the sum.
In simpler terms:
A phenomenon may be represented by a normal distribution if it results
from a great number of additional, small and independent effects
20/05/2015
64
GUM Step 4 : Final expression of the result
y U unit (value of k )
In order to have a physical significance, a measurement result must be
rounded according to the uncertainty related to the result.
o U is rounded up to at most 2 significant digits
o The measurement result y is rounded accordingly
For instance, there is no need to announce a length measurement by the
micron if the uncertainty related to the measurement with a sliding caliper
corresponds to the tenth of a millimeter
The rounding must not be carried out in several stages but only once, when
reporting the final result
20/05/2015
65
GUM Step 4 : Summary
U = k u(y)
u(y) obtained at Stage 3
Very often k = 2
U rounded to 2 significant digits
y U
unit
(k=2)
y rounded to the same position as U

Note the unit
20/05/2015
66
Outline
Introduction
Reference documents
GUM methodology
Case studies
Linear example
Conclusion
Bayesian methods
General conclusion
20/05/2015
67
Different methods for the evaluation of

Measurement Process
analysis
no
yes
Does the model

exist ?
Analytical
method
uncertainties
hypotheses
?
Analytical Method
Reference
documents
yes
ISO 5725
ISO GUM
ISO GUM
non
supplment 1
Variance Analysis
Law of Propagation
Monte Carlo Method
Result and
Uncertainty
20/05/2015
68
GUM-S1 : The Monte Carlo method

Name derived from the Monaco roulette, which is
considered to be a mechanism designed specifically to
display numbers randomly.
The objective of the Monte Carlo method is to artificially
reproduce a random phenomenon
More specifically, it consists of simulating a factitious
sample of realisations of that phenomenon based on
random variable assumptions. The statistical analysis
made on a large sample will be close to reality
20/05/2015
69
GUM-S1 : GUM uncertainty framework

Application of the Law of Propagation of Uncertainty
f approximated by the 1st order of the serial development of Taylor
uc2
N 1 N
(y ) = f u 2 (x i ) + 2 f f u x i , x j
i =1 x i
i =1 j =i +1 x i x j
N
low deviations are considered

f is linear (linearisable)
values of the derivatives available
y, u(y)
output quantity Y is Gaussian
The Central Limit Theorem is to be applied
independence of xi s
The variance associated to Y is greater than any ci (xi)
component for an xi whose distribution is not gaussian
20/05/2015
yU
70

x 1, u(x 1)
x 2, u(x 2)
Y = f(X1, X2, X3)
y, u(y)
y U (k=2)
x 3, u(x 3)
Y = f(X1, X2, X3)
y~ ;u ( y~)
[ylow;yhigh]
20/05/2015
71
GUM-S1 : Different steps of the Monte Carlo

method
Writing of the
mathematical model
Step 1
Y=f(x1,x2,x3)
Choice of distributions
for input variables
Step 2
Step nb.1
Step nb.2
Step 3
Step nb. M
X1
X2
x11
x12
x1M
x21
x22
x2M
X3
Calculation of M values
resulting from Y
Empirical distribution of
the measurand
Step 3
y1
x31

x32 f y 2

x3M
y M
Random generation of M
input variable samples
20/05/2015
Step 4
~
y ; u( ~
y)
[ylow;yhigh]
Extraction of the:
Mean
standard deviation
interval at p%
72
GUM-S1 : Choice of the distributions

x 1, u(x 1)
x 2, u(x 2)
Y = f(X1, X2, X3)
y, u(y)
y U
x 3, u(x 3)
Y = f (X1, X2, X3)
20/05/2015
y~ ;u ( y~)
[ylow;yhigh]
73
Reminder : GUM Type B
o Interval of values
o A distribution of probability
-a
or
-a
20/05/2015
u(xi)
(GUM)
or S (Stats)
...
74
GUM-S1 Step 2 : Probability distributions

Probability distributions commonly used in metrology:
o
Normal or Gaussian: calibration certificates
Rectangular or Uniform: device resolution
Triangular
Student: repeated number of measurements
U-shaped: oscillating phenomenon
Other probability distributions :

o
Beta, gamma, weibull, Poisson, exponential, log-normal,
20/05/2015
75
GUM-S1 Step 2 : Associating a Distribution to

an Input Quantity
Maximum entropy principle, advocated in Supplement 1 to
the GUM
Choice of expert, difficulty to quantify prior information
available.
Statistical goodness-of-fit tests between the experimental
and theoretical distributions:
Tests: Anderson-Darling, Kolmogorov-Smirnov, Chi-squared test,
Use of statistical software
20/05/2015
76
GUM-S1 Step 2 : Maximum entropy principle
Objective: Selection of a probability density function that is consistent with

our knowledge and introduces no unwarranted information1
Formula for a density probability g:
h(g ) = g (t ) log(g (t ))dt
Maximum Entropy: used to determine a prior probability distribution The

principle consists of finding the distribution g which maximizes the quantity
h(g) (representing misreadings of the phenomenon) given the only available
information
Important property in Bayesian methods
20/05/2015
77
GUM-S1 Step 2 : Assignment of probability

distributions
Available information on
quantity X
Distribution to be assigned
to X according to
maximum entropy
Lower limit: a
Higher limit: b
R (a,b)
Rectangular
Best estimate x with

associated uncertainty
u(x)
N (x, u(x))
X varies sinusoidally
between limits a and b
(a<b)
U-shaped(a,b)
Best estimation of a
positive quantity
graph
Gaussian
Arcsine Derivative
1
e( )
Exponential
Normal distribution corresponds to maximum entropy taken from all the distributions which
have an identical mean x and a identical standard deviation u(x)
20/05/2015
78
GUM-S1 : Assignment of probability

distributions (Type A)
Using the Student distribution : Approximation to the Gaussian
distribution for n 30
Example: SO2 Concentration Readings (in ppb)
112.9
114.1
112.9
113.4
113.4
113.6
113.2
115.4
113.1
113.6
113.5
114.4
112.7
113.6
113.5
Applying the Student distribution with

n 1 degrees of freedom
Mean = 113.55
Scale (experimental std
deviation) = 0.68
Number of df = 14
Student Standard Deviation :
20/05/2015
79
GUM-S1 Step 3 : Generating random numbers

The pseudo random number generator must be tested
prior to any calculation:
o
Resistence to adequation tests
Period
independence of generated values
Sampling of the distribution function

o
Generation of a random number in the interval [0 ; 1]
Calculation of value x1 verifying G(x1) =
(G-1()= x1)
G( x ) =
g (t )dt
20/05/2015
80
GUM-S1 Step 3 : Generating random numbers

Example : Sampling from a Gaussian distribution N(0,1)
o Generation of a random number in the interval [0 ; 1]
o Calculation of value x1 verifying G(x1) =
(G-1()= x1)
R = 0.71
0.5534
20/05/2015
81
GUM-S1 Step 3
Random trials of M input

variable samples
Step N1
Step N2
Step 3
Step NM
X1
X2
x11
x12
x1M
x21
x22
x2M
X3
Calculation of M values
resulting from the empirical
measurand distribution Y
Step 4
y1
x31

x32 f y 2

x3M
y M
Number of M Monte Carlo simulations?

GUM-S1 : 106
20/05/2015
82
GUM-S1 Step 3 : Choice of Simulation Software

Many software have pseudo random number generation
algorithms : Excel, Crystal Ball, Matlab, R, C++,
Decision criteria :
o
Software validation,
Ergonomy, ease of use,
The duration of the calculations (from a few seconds to several

hours) also depends on :
The complexity of the model,
The number M of draws.
Costs: purchase, maintenance, training or computation development
20/05/2015
83
GUM-S1 Step 3 : Experience and validation

Use of Matlab : Mersenne-Twister algorithm
o
recognized as a good pseudo random number generator
Period T = 219937-1
Quoted in ISO 28640:2010: Pseudo Random Number Generating

Methods
Validation of another tool (ex : CRYSTAL BALL V.5.2.2 )

o
Identification and properties of pseudo random number generator

used
Adequation test of simulated distributions (rectangle, normal,etc.)
Comparison and statistical validation with Matlab (or any tool

validated beforehand)
20/05/2015
84
GUM-S1 Step 4
Most frequently k = 2
x 1, u(x 1)
x 2, u(x 2)
Y = f(X1, X2, X3)
y, u(y)
y U
x 3, u(x 3)
Interval at p%
Coverage interval
Y = f(X1, X2, X3)
20/05/2015
[ylow;yhigh]
85
GUM-S1 Step 4 : Distribution of the measurand

The value attributed to the measurand is obtained by calculating
the empirical mean of the output distribution:
V
1
TU =
F
The standard uncertainty of the measurand is obtained from the

experimental standard deviation of the output distribution:
1
2 TU =
F1
T TU
The expanded uncertainty corresponds to the interval at p%

which contains proportion p chosen among the possible
values of the measurand:
TWKX ; TY ZY
20/05/2015
86
GUM-S1 Step 4 : Choice of the coverage

interval
Quantile function (= Cumulative distribution function)
[\ <
= T such that = P $ T
p = 0.95
1
(
2
0.975
0.025
[\ <
(
2
Symmetrical distribution
(
<
\
[
; [\ < 1
2
20/05/2015
[\ <
1.96
(
2
avec (
(
2
1.96
87
GUM-S1 Step 4 : Choice of the coverage

interval
Asymmetrical distribution
o
o
Different candidates coverage intervals

GUM-S1 recommends to choose the shortest interval among all
possible intervals :
Determine ( 0; 1
such that [\ <
[\ <
is minimum
Determine ylow and yhigh such that %a TWKX = %a TY ZY

Example of Triangular distribution between 1 and 9 with Mode 3
Iy = 8.13-1.74 = 6.36
20/05/2015
Iy = 7.48-1.14 = 6.34
Iy = 7.68-1.47 = 6.21
88
GUM-S1 Step 3 : Uncertainty budget /

sensitivity analysis
Not straightforward : it requires extra calculations
Proposal of the GUM-S1 : Perfoms the Monte Carlo
simulation holding all quantities but one fixed at their best
estimate
o
N additional Monte Carlo simulations for N input quantities in the

measurement model
Does not evaluate interaction effects
Correlation/ Regression coefficients :
No additional Monte Carlo simulations

Does not evaluate interaction effects
Variance based sensitivity indices (ex: Sobol indices)

o
Requires additional Monte Carlo simulations
Gives an evaluation of significant interaction effects
A.Saltelli, Sensitivity analysis, Wiley, 2000

A.Allard & N.Fischer, Recommended tools for sensitivity analysis associated to the evaluation of
measurement uncertainty, World Sc. Publ. Comp., 2012, pp 1-12
20/05/2015
89
Different methods for the evaluation of

Measurement Process
analysis
no
yes
Does the model

exist ?
Analytical
method
uncertainties
hypotheses
?
Analytical Method
Reference
documents
yes
ISO 5725
ISO GUM
ISO GUM
non
supplment 1
Variance Analysis
Law of Propagation
Monte Carlo Method
Result and
Uncertainty
20/05/2015
90
Inter-Laboratory Comparison (ILC)

Standard ISO 5725-2 : Accuracy (trueness and precision)
of measurement results and methods
o
Alternative uncertainty estimation method when one does not know

how to write the model
estimation of the trueness and the dispersion of a measurement

method through interlaboratory tests
Trueness (VIM 2012,2.14) : closeness of agreement

between the average of an infinite number of replicate
measured quantity values and a reference quantity value.
Precision (VIM 2012, 2.15) : closeness of agreement
between indications or measured quantity values obtained
by replicate measurements on the same or similar objects
under specified conditions.
20/05/2015
91
Inter-Laboratory Comparison (ILC)

Step 1
no
y=f(x1,x2,...xn)
yes
Step 2
quantify
Standardized
Method
+
Interlaboratory
tests
ISO 5725 : sr et sR
no
Estimate
One intermediate
uncertainty
yes
u(xi)
participation ?
no
Proof
Normalized
Method
yes
Step 3
uncertainty
propagation
uc(y)
uc(y) = sR
Step 4
Expanded uncertainty
U = kuc(y)
20/05/2015
92
ILC : Expression of the precision
Minimum
5725-2
Maximum
Reproducibility
conditions (R)
repeatability (r)
conditions
Intermediate Precision
5725-3
same method
identical individuals
same laboratory
same operator
same equipment
short time interval
20/05/2015
same method
identical individuals
different laboratories
different equipment
93
ILC : Calculation for one level (same ni )

Results
raw
Lab 1
y11
Mean
T
y1n
Lab i
Descriptive
Statistics
s1
yi1
Tb
yin
Lab p
Standard
deviation
si
yi1
ypn
Grubbs
sp
Homogeneity Tests (filter p p*)
Cochran
Hypothesis : ni = n constant
Tc =
Tb
20/05/2015
1
1
repeatability
standard deviation
Tb Tc
reproducibility
standard deviation
94
Outline
Introduction
Reference documents
GUM methodology
Case studies
Linear example
Conclusion
Bayesian methods
General conclusion
20/05/2015
95
Example 1 : Linear model Step 1

Determination of the thickness of a sample
Step 1 : analysis of the measurement process

Measurand : Thickness e of the sample expressed in mm, using
a sliding caliper, at (20 2) C
Uncertainty sources :
Trueness of the sliding caliper

Quantification of the sliding caliper
Repeatability
Mathematical model :
t = tu +
20/05/2015
+ t
96
Step 2 : Quantification of the uncertainty sources

Type B :
Information available for the trueness : a calibration certificate

with an expanded uncertainty U = 0.1 mm (k=2)
Information available for the quantification : The sliding caliper

has a quantification step of q = 0.1 mm
Type A :
Repeatability : The operator performs n = 10 measurements
20/05/2015
97

GUM
GUM-S1
Trueness
o
2 tu =
Trueness
v
w
= 0.05 mm
o Gaussian distribution
o Mean : 0, std deviation :
v
.
2 tu = w =
= 0.05 mm
Quantification
o
Rectangular distribution
Quantification
o Rectangular distribution
x
= 0.029 mm
o Lower bound : = 0.05 mm
3.1 3.0
Repeatability
3.1 2.9
n = 10 repetitions
3.2 3.2
t = 3.05 mm
3.0 3.1
2 t =
= 0.11 mm
20/05/2015
2.9 3.0
o Upper bound : = 0.05 mm
Repeatability
o Student distribution
o t = 3.05 mm
o I5t = = 0.11 mm
98
Example 1 : Linear model (Step 3 and 4)

GUM
GUM-S1
2 t = 2 tu + 2
+ 2 t
2 t =
+ 2 t
2 t =
2 tu + 2
M = 106 Monte Carlo iterations
0.05 + 0.029 + 0.11

2 t = 0.123 mm
D = 2 2 t = 0.25 mm
Coverage interval :
T D; T + D = 2.80; 3.30
20/05/2015
t = 3.05
2 t = 0.14
Coverage interval :
tWKX ; tY ZY = 2.78; 3.32
99
Example 1 : Uncertainty budget

GUM
GUM-S1
Input quantity
Coontribution
to the variance
(%)
Input quantity
Coontribution
to the variance
(%)
Trueness eT
16%
Trueness eT
14%
Repeatability t
79%
Repeatability t
81%
Quantification q
5%
Quantification q
5%
20/05/2015
100
Example 1 : Conclusion
Good agreement between GUM and GUM-S1

Linear measurement model
The repeatability is the most influential input quantity
Importance of the number of repeated measurements performed
20/05/2015
101
Example 2 : Mass calibration (GUM-S1)

Mass calibration
o
Step 1 : analysis of the measurement process

Measurand : Mass difference m between the conventional mass mW,c
to calibrate and the nominal mass mnom, expressed in mg.
Uncertainty sources :
Conventional mass mR,c(mg) of the reference weight,

Conventional mass mR,c(mg) of a small weight added to R to
balance it with W,
Mass density a(kg/m3) of air,

Mass density W(kg/m3) of the weight W to calibrate.
Mass density R(kg/m3) of the reference weight R.
20/05/2015
102
Example 2 : Mass calibration Step 2

GUM-S1
GUM
o
o
a
o
o
W
o
o
r, = 100 000 mg
2 r, = 0.05 mg
mR,c
o
mR,c
R
o
o
r, = 1.234 mg
2 r, = 0.02 mg
0 = 1.20 kg/m
2 0 = 0.058 kg/m
0 = 8000 kg/m
2 0 = 577 kg/m
0r = 8000 kg/m
2 0r = 29 kg/m
20/05/2015
mR,c
o Mean : r, = 100 000 mg, Standard
deviation : 2 r, = 0.05 mg
mR,c
o Mean :r, = 1.234 mg, Standard
deviation : 2 r, = 0.02 mg
a
o Lower bound : 1.10 kg/m, Upper
bound : 1.30 kg/m
W
o Lower bound : 7000 kg/m, Upper
bound : 9000 kg/m
R
o Lower bound : 7950 kg/m, Upper
bound : 8050 kg/m
103
Example 2 : Mass calibration Steps 3 and 4

GUM-S1
GUM
2 = 7r, 2 r, + 7r, 2 /r,
M = 106 Monte Carlo iterations
+7 2 0 +7 2 0 +7r 2 0r
2 =
2 r, + 2 /r,
2 = 0.05 + 0.02
2 = 5.4 10< kg
= 1.234 10< kg
= 7.5 10< kg
2
D = 2 2 = 11 10< kg
Coverage interval :
T D; T + D = 1.124 10< ; 1.343 10<
20/05/2015
Coverage interval :
WKX ; Y ZY =
1.083 10< ; 1.383 10<
104
Example 2 : Mass calibration (uncertainty

budget)
GUM-S1
GUM
Input quantity
Coontribution
to the variance
(%)
Input quantity
Coontribution
to the variance
(%)
Convent. Mass of
the reference
weight mR,c
86%
Convent. Mass of
the reference
weight mR,c
43%
Convent. Mass of
the added weight
mR,c
14%
Convent. Mass of
the added weight
mR,c
8%
Interaction
between the mass
densities a and
W
Not available
Interaction
between the mass
densities a and
W
49%
20/05/2015
105
Outline
Introduction
Reference documents
GUM methodology
Examples
Linear example
Conclusion
Bayesian methods
General conclusion
20/05/2015
106
Conclusion : Scope of the GUM/GUM-S1
GUM
Measurement model
GUM-S1
Explicit mathematical expression
Linearity
Linear or
approximately linear
No assumptions
Distribution associated
to the measurand
Often Gaussian
Any distribution
Software
implementation
Excel sheet
Software with a validated

random generation
number algorithm
Sensitivity analysis
Directly availabe
(partial derivatives)
Requires additional
calculations or simulations
Indirect measurements
20/05/2015
Not addressed
107
Conclusion : Bayesian methods

The Bayesian methods offer a framework that combines
both experimental data X and prior knowledge of a
measurand .
Prior probability
distribution
Bayes theorem
| =
Likelihood function
;
|
Posterior probability
distribution
The combination of prior information with information carried

out by experimental data is summarized in the posterior
probability distribution
o Useful for inverse and regression problems (see JRP NEW04
Novel mathematical and statistical approaches to uncertainty
evaluation , http://www.ptb.de/emrp/new04-home.html )
20/05/2015
108
General conclusion (1/4)

The evaluation of measurement uncertainty is a
requirement : a measurement result without any associated
has no meaning and is useless.
Harmonization of the methodology to evaluate uncertainty :
o The Guide to the expression of Uncertainty in Measurement
(GUM),
o The propagation of distributions using a Monte Carlo method
(GUM Supplement 1)
Harmonization of the vocabulary used in metrology :

o The International Vocabulary of Metrology (VIM)
20/05/2015
109
Before any uncertainty quantification, a deep analysis of

the measurement process is required.
GUM steps :
Step 1 : Analysis the measurement process : $ = #
,,
Step 2 : Quantification the standard uncertainty u(Xi) associated with the

input quantities
Step 3 : Propagation of uncertainty using the Law of Propagation of

Uncertainty.
Step 4 : Reporting of the measurement result
The GUM has been the reference method since 1994

(more than 20 years) !
20/05/2015
110
However, in some applications, GUM methodology may

not be suitable :
Non linear measurement models
Models having one predominant non-Gaussian input quantity
Computational codes
GUM-S1 steps :
Step 1 : Analysis the measurement process : $ = #
,,
Step 2 : Attribution of a probability density function to each of the input

quantities
Step 3 : Propagation of distributions using Monte Carlo simulation
Step 4 : Reporting of the measurement result
20/05/2015
111

In most cases, both methods are consistent because in
metrology, we often deal with :
Small deviations from the best estimates (the linear approximation of the
GUM is then often adequate).
Many different uncertainty sources that contribute signifiicantly to the

measurand (the measurand is then often Gaussian).
Despite its wide use in the world, the GUM needs revisions
Harmonization with its supplements and the VIM
Examples in a separate document
A revised GUM has been reviewed by NMIs in early 2015
and comments are under investigation by JCGM
But the methodology still relies on the LPU
20/05/2015
112
Questions ?
Thank you !
Merci !
terima kasih !
20/05/2015
113

Materi Sesi I General

Transféré par

Informations du document

Description originale:

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Materi Sesi I General

Transféré par

Droits d'auteur :

Formats disponibles

Fundamentals of

Laboratoire national de mtrologie et dessais

The concept of measurement uncertainty

Basic statistical concepts

Step 1 : Analysis of the measurement process

Mass calibration examples

Scope of the GUM/GUM-S1

Workshop measurement uncertainty Jakarta

The concept of uncertainty

Measurement result (VIM 2012) : set of quantity values being

Workshop measurement uncertainty Jakarta

Why Must an Uncertainty in a Measurement

Workshop measurement uncertainty Jakarta

Why Must an Uncertainty in a Measurement

Workshop measurement uncertainty Jakarta

Usage : Comparison between Two Measurement

How to compare results from laboratories A and B ?

Workshop measurement uncertainty Jakarta

Usage : Comparison between Two Measurement

The statement of a measurement uncertainty enables

Workshop measurement uncertainty Jakarta

Comparison between a Measurement Result and

Workshop measurement uncertainty Jakarta

Comparison between a Measurement Result and

Workshop measurement uncertainty Jakarta

Comparison between a Measurement Result and a

Workshop measurement uncertainty Jakarta

Comparison between a Measurement Result and

The client benefits from the doubt

Workshop measurement uncertainty Jakarta

Comparison between a Measurement Result and

The producer benefits from the doubt

Workshop measurement uncertainty Jakarta

Comparison between a Measurement Result and

Client and producer share the risk (GUM-Conformity)

Workshop measurement uncertainty Jakarta

Joint Committee for Guides in Metrology (JCGM)

Workshop measurement uncertainty Jakarta

Bilingual English/French document

Harmonization of the vocabulary usd in metrology

Workshop measurement uncertainty Jakarta

Documentation GUM + Supplements

1st version published in 1994.

Minor modifications until the last version in 2008

GUM-S1 (JCGM 101:2008) : Supplement 1 to the Guide to

Workshop measurement uncertainty Jakarta

Documentation GUM + Supplements

GUM-S2 (JCGM 102:2012) : Supplement 2 to the Guide to

Deals with multiple measurands : common applications in chemistry,

Extension of both GUM and GUM-S1 methodologies

GUM-Conformity (JCGM 106:2012) : The role of

Workshop measurement uncertainty Jakarta

expressed as a standard deviation . Denoted as u or uc.

Coverage interval (VIM 2012, 2.36) : interval containing the

Workshop measurement uncertainty Jakarta

Terminology : Measurement error

The result of a measurement (after correction) can

Workshop measurement uncertainty Jakarta

Basic statistical concepts

Consider n repeated measurements

Standard deviation : Parameter that characterizes the

Basic statistical concepts