CHAPTER OVERVIEW

• The Measurement Process

• Levels of Measurement
• Reliability and Validity: Why They Are
Very, Very Important
• A Conceptual Definition of Reliability
• Validity
• The Relationship Between Reliability
and Validity
• A Closing (Very Important) Thought
THE MEASUREMENT PROCESS
• Two definitions
– Stevens—”assignment of numerals to
objects or events according to rules.”
– “…the assignment of values to
outcomes.”
• Chapter foci
– Levels of measurement
– Reliability and validity
 LEVELS OF MEASUREMENT
Level of
Measureme For example Quality of Level
nt
Ratio Rachael is 5’ 10” and Gregory is Absolute zero
Interval 5’ 5”
Rachael is 5” taller than Gregory An inch is an inch is an
inch
Ordinal Rachael is taller than Gregory Greater than
Nominal Rachael is tall and Gregory is Different from
short
• Variables are measured at one of these four
levels
   • Qualities of one level are characteristic of the
  
  next level up
• The more precise (higher) the level of
measurement, the more accurate is the
measurement process
 NOMINAL SCALE

Qualities Example What You What You

Can Say Can’t Say
 
Assignment of Gender— Each An observation
labels (male or observation represents
female) belongs in “more” or
Preference— its own “less” than
(like or category another
dislike) observation
Voting record—
(for or
against)
ORDINAL SCALE

Qualities Example What You What You

Can Say Can’t Say
Assignment of Rank in college One The amount
values along Order of observation that one
some finishing a race is ranked variable is
underlying above or more or less
dimension below than another
another.
INTERVAL SCALE

Qualities Example What You What You

Can Say Can’t Say

Equal Number of One score The amount of

distances words spelled differs from difference is an
between correctly another on exact
points Intelligence test some representation
scores measure of differences
Temperature that has on the variable
equally being studied
appearing
intervals
RATIO SCALE

Qualities Example What You What You

Can Say Can’t Say

Meaningful Age One value Not much!

and non- Weight is twice as
arbitrary zero Time much as
another or
no quantity
of that
variable can
exist
WHAT IS ALL THE FUSS?
• Measurement should be as precise as possible
• In psychology, most variables are probably
measured at the nominal or ordinal level
• But—how a variable is measured can
determine the level of precision
RELIABILITY AND VALIDITY
• Reliability—tool is consistent
• Validity—tool measures “what-it-
should”
• Good assessment tools →
– Rejection of Null hypotheses
OR
– Acceptance of Research hypotheses
A CONCEPTUAL DEFINITION OF
RELIABILITY

Method Error
Observed Score = True Score + Error Score
Trait Error
A CONCEPTUAL DEFINITION OF
RELIABILITY
Method Error
Observed Score = True Score + Error Score
Trait Error

• Observed score
– Score actually observed
– Consists of two components
• True Score
• Error Score
A CONCEPTUAL DEFINITION OF
RELIABILITY
Method Error
Observed Score = True Score + Error Score
Trait Error

• True score
– Perfect reflection of true value for individual
– Theoretical score
A CONCEPTUAL DEFINITION OF
RELIABILITY
Method Error
Observed Score = True Score + Error Score
Trait Error

• Error score
– Difference between observed and true score
A CONCEPTUAL DEFINITION OF
RELIABILITY
Method Error
Observed Score = True Score + Error Score
Trait Error

• Method error is due to characteristics of the

test or testing situation
• Trait error is due to individual characteristics
• Reliability of the observed score becomes
higher if error is reduced!!
INCREASING RELIABILITY
→ Decreasing Error

• Increase sample size

• Eliminate unclear questions
• Standardize testing conditions
• Use both easy and difficult questions
• Minimize the effects of external events
• Standardize instructions
• Maintain consistent scoring procedures
HOW RELIABILITY IS MEASURED
• Reliability is measured using a
– Correlation coefficient
– r test1•test2
• Reliability coefficients
– Indicate how scores on one test
change relative to scores on a second
test
– Can range from -1.0 to +1.0 (perfect
reliability)
TYPES OF RELIABILITY
Type of What It Is How You Do It What the
Reliability Reliability
Coefficient Looks
Like
Test-Retest A measure of Administer the same rtest1•test1
stability test/measure at two
different times to the same
group of participants
Parallel A measure of Administer two different rform1•form2
Forms equivalence forms of the same test to
the same group of
participants
Inter-Rater A measure of Have two raters rate Percentage of
agreement behaviors and then agreements
determine the amount of
agreement between them
Internal A measure of Correlate performance on Cronbach’s alpha
Consistency how consistently each item with overall
each item performance across
measures the participants
same underlying
construct
VALIDITY
• A valid test does what it was
designed to do
• A valid test measures what it was
designed to measure
A CONCEPTUAL DEFINITION OF
VALIDITY
• Validity refers to the test’s results,
not to the test itself
• Validity ranges from low to high, it
is not “either/or”
• Validity must be interpreted within
the testing context
 TYPES OF VALIDITY
Type of Validity What Is It? How Do You Establish It?
Content A measure of how Ask an expert if the items
well the items assess what you want them to
represent the entire
Criterion universe of items

Concurrent A measure of how Select a criterion and correlate

well a test estimates scores on the test with scores
a criterion on the criterion in the present
Predictive A measure of how Select a criterion and correlate
well a test predicts a scores on the test with scores
criterion on the criterion in the future
Construct A measure of how Assess the underlying construct
well a test assesses on which the test is based and
some underlying correlate these scores with the
construct test scores
HOW TO ESTABLISH CONSTRUCT
VALIDITY OF A NEW TEST
• Correlate new test with an
established test
• Show that people with and without
certain traits score differently
• Determine whether tasks required on
test are consistent with theory
guiding test development
MULTITRAIT-MULITMETHOD
MATRIX Trait 1
Impulsivity
Trait 2
Activity Level
Method 1 Method 2 Method 1 Method 2
Paper and Activity Level Paper and Activity Level
Pencil Monitor Pencil Monitor
Method 1
Paper and Pencil Moderate Low
Trait 1
Impulsivit Method 2
y Activity Level Moderate
Monitor
Method 1
Paper and Pencil
Trait 2
Activity Method 2
Level Activity Level Low
Monitor

• Convergent validity—different methods yield similar

results
• Discriminant validity—different methods yield different
results
THE RELATIONSHIP BETWEEN
RELIABILITY AND VALIDITY
• A valid test must be reliable
But
• A reliable test need not be valid