RELIABILITY

Concept of Correlation:
• Correlation Coefficients
Identifying and Evaluating Error
• Fallibility of Scores
• Sources of Error

LAO, Jem
CORRELATION COEFFICIENTS
Pearson’s coefficient
Is based upon calculating the average cost-
product of test scores given as z scores
and the correlation coefficients then have
numerical values between -1 and +1.
‘r’ was developed to identify both
magnitude and direction of a relationship,
and to represent the shared variance
between two variables
100% - maximum, identical
0% - minimum, no relationship LAO, Jem
PEARSON R

© Math Bits
© STUDY
BLUE

© Math is Fun
LAO, Jem
 CORRELATION COEFFICIENTS

Figure 1. Graphs showing a correlation of -1, 0 and +1

© Correlation Coefficient: Simple Definition, Formula, Easy Steps. Retrieved from

http://www.statisticshowto.com/probability-and-statistics/correlation-coefficient-formula/

LAO, Jem
CORRELATION COEFFICIENTS
+1 or -1 Perfect correlation (ideal)

+1 Perfect positive correlation

0 No correlation between variables
Independent and unrelated
variables
In statistics, orthogonal
-1 Perfect negative correlation
LAO, Jem
CORRELATION COEFFICIENTS

*The significance of relationship

is shown by:
Level/Strength of association
Coefficient of Determination, r
2 or R 2

- Is ratio of the explained variation to

the total variation.
0 < r 2 < 1 - denotes the strength of the
linear association between x and y.
LAO, Jem
 GUIDELINES TO INTERPRETING
PEARSON'S CORRELATION
COEFFICIENT
Coefficient, r

Strength of Association Positive Negative

Small .1 to .3 -0.1 to -0.3

Medium .3 to .5 -0.3 to -0.5

Large .5 to 1.0 -0.5 to -1.0

© Pearson Product-Moment Correlation. Laerd Statistics

CORRELATION COEFFICIENTS
Effect Size (Cohen 1988)
Research: where r is used,
.2 - small
.5 - moderate
.8 - large
Organizational psychology (test measures):
where r is used,
.10 – weak
.3 – moderate
.5 – quite large
.8 – particularly large
0 – no correlation between variables and that there is no
degree of association.
LAO, Jem
CORRELATION COEFFICIENTS
Values can be distorted by factors
including the following:
1. Restricting the range of a
variable.
2. Using non-linear variables
3. Variability or variance in the
variables
4. Errors in the measurement of
some variables LAO, Jem
CORRELATION COEFFICIENTS
Sample Size Variables Based on Target Population
 Population Size
 Margin of Error (Confidence Interval)
 Confidence Level
 Standard of Deviation
Necessary Sample Size = (Z-score)2 * StdDev*(1-StdDev) /
(margin of error) 2
Sample: a 95% confidence level, .5 standard deviation, and a
margin of error (confidence interval) of +/ - 5%.
((1.96)2 x .5(.5)) / (.05)2
(3.8416 x .25) / .0025
.9604 / .0025
384.16 =385 respondents are needed
60 – small, 200 or more – good
Kline (2000) – minimum sample size of 100
LAO, Jem
Index Description Effect Size Comments

Between Groups

Cohen’s d
SUMMARY
D = M1 –M2/s
M1-M2 Is the difference
Small = 0.2
Medium = 0.5
Can be used at planning
stage to find the sample
between the group Large = 0.8 size required for sufficient
means (M) Very = Large =1.3 power for your study
S id the standard
deviation of either group.

Measures of Association

Pearson’s r Correlation Range, -1 to +1 Small = +-0.2 Measures the degree of

Medium = +- 0.5 linear relationship
Large = +- 0.8 between two quantitative
variables

R squared coefficient of Range, 0 to +1; usually Small = 0.04 Proportion of variance in

determination expressed as percent Medium =0.25 one variable explained by
Large = 0.64 the other.

Adapted from Ferguson et. al

Based on Soper.
IDENTIFYING AND
EVALUATING
ERROR
Fallibility of Scores
Sources of Error
LAO, Jem
 FALLIBILITY OF SCORES
Classical Test Theory (CTT)
Relates the true score and observed
scores on any measure.
Items contained within it are thought to
consist of random sample taken from
an infinite universe of all relevant
ones.
Every measurement is fallible in
practice and includes some amount of
error (Nunnally, 1978).
LAO, Jem
 CLASSICAL TEST THEORY (CTT)

The relationship between these components is represented

mathematically as
X=T+E
In which,
X = a person’s test score (raw score)
T = the person’s stable characteristics or knowledge (true score)
E = chance events (error scores)
We can also use the equation to determine the proportion or
percent of a test score reflecting each factor :
T/X = proportion of test score reflecting person’s true stable
knowledge or characteristics
E/X = proportion of test score reflecting random error
RELIABILITY TEST = T/X > E/X
TWO SOURCES OF ERROR
VARIANCE
SYSTEMATIC
Having a predictable effect on scores
by introducing a consistently
measureable bias every time a test is
used.
POTENTIAL CAUSES:
1.Age or Cultural biases
2.Factors in Test Design
3.Response Styles Adopted by People

LAO, Jem
TWO SOURCES OF ERROR
VARIANCE
RANDOM (random measurement error)
• effect is unpredictable
• will make scores bigger or make them
smaller, resulting to reduced reliability.
• are made up of factors which differ every
time a measure is used:
• Non-standardized administration; Environmental
factors; Inadequate scoring (Thompson, 1994).
• Longer-term developmental and historical factors
LAO, Jem
EVALUATION OF SOURCES
OF ERROR VARIANCE
(JENSEN, 1980)
Candidate-related sources
are the state of mind of test-
takers. This can include
aspects which can vary widely:
feelings, anxiety, motivation,
general well-being, fatigue, and
other preoccupations
LAO, Jem
EVALUATION OF SOURCES OF ERROR
VARIANCE (JENSEN, 1980)

Test - related sources

1. Sampling Error (test length and
representativeness)
2. Poor Item Construction
3. Physical construction, legibility,
clarity of instructions, and
response procedure design
LAO, Jem
EVALUATION OF SOURCES OF ERROR
VARIANCE (JENSEN, 1980)

Procedural Sources
involve ergonomic factors related to test
administration .
1. Whether there are any interruptions, comfort of
the test-takers at the time, noise level, and any
copying or cheating
2. People’s understanding of what is needed,
keeping to conditions such as time limits, and
completing answer sheets correctly.
3. Accurate use of scoring keys, and clerical errors
in managing data .
LAO, Jem
EVALUATION OF SOURCES
OF ERROR VARIANCE
(JENSEN, 1980)
Environmental Sources
Includes the ambient temperature and
lighting;
Any disruptive noise;
Facilities available;
Comfort of seats provided
 REFERENCES

Coaley, K. (2016). An introduction to psychological

assessment and psychometrics. (2 nd ed). Singapore:
Sage
Publications Inc.
Friedenberg, L. (1995). Psychological testing: Design
analysis and use. Boston: Allyn and Bacon
Jensen, A.R. (1980) Bias in mental testing. New York: Free
Press
Nunnally, J. O. (1978). Psychometrics theory. New York:
McGraw-Hill.
Sullivan, G., & Feinn, R. (2012).J Grad Med Educ., 4(3): 279–
282. doi: 10.4300/JGME-D-12-00156.1
Thompson, B (1994). Guidelines for authors. Educational and
Psychological Measurements Yearbook, 54, 837-847.
 REFERENCES FROM WEBSITES

Statistics How To. Correlation coefficient: Simple definition ,

formula, easy steps. Retrieved April 14, from
http://www.statisticshowto.com/probability -and-
statistics/correlation -coefficient-formula/
Kline. (2005). Classical test theory, assumptions , equations,
limitations, and item analyses. Retrieved April 14, from
https://us.sagepub.com/sites/default/files/upm -
binaries/4869_Kline_Chapter_5_Classical_Test_Theory.p
df
Khan Academy. Creating and interpreting scatterplots.Retrieved
April 14, from
https://www.khanacademy.org/math/probability/scatterplo ts-
a1/creating -interpreting-scatterplots/e/interpreting -scatter-plots
Laerd Statistics. Pearson Product -Moment Correlation. Retrieved April
15, from https://statistics.laerd.com/statistical -guides/pearson-
correlation -coefficient-statistical-guide.php
Math Bits. Correlation Coefficient . Retrieved April 15 from,
https://mathbits.com/MathBits/TISection/Statistics2/correlation.
htm