Some (Simplified) Steps for Creating a

Personality Questionnaire

Generate an item pool

Administer the items to a sample of people
Assess the uni-dimensionality of the item pool
Assess the reliability of the measure
Assess the validity of the measure

Generate an Item Pool

Sample from the universe of possible items
Rational/theoretical approach
Observation (clinical, narrative, interviews, descriptions
of others)

Administration
Administer the questionnaire to a large sample of
individuals.
Can use full item set
Can randomly sample items from full set

Dimensionality
We want to ensure that there is only one major latent
variable that the items have in common.
The statistical tools that we use tend to assume unidimensionality.
Multi-dimensional constructs are treated separately.
Principal components analysis is sometimes used to
determine whether one major variable underlies the item
responses.

Choose items to factor

Choose Varimax under rotation

options
Select Scree plot option in Extraction options

Total Variance Explained

Component
1
2
3
4
5

Total
2.589
.926
.653
.522
.310

Initial Eigenvalues
% of Variance Cumulative %
51.777
51.777
18.524
70.301
13.066
83.366
10.437
93.803
6.197
100.000

Extraction Method: Principal Component Analysis.

Extraction Sums of Squared Loadings

Total
% of Variance Cumulative %
2.589
51.777
51.777

Reliability
Reliability: the extent to which measurements are free of
random errors.
Random error: nonsystematic mistakes in measurement
misreading a questionnaire item
observer looks away when coding behavior
nonsystematic misinterpretations of a behavior

Reliability
What are the implications of random measurement
errors for the quality of our measurements?

Reliability
O = T+ E + S
O = a measured score (e.g., performance on an
exam)
T = true score (e.g., the value we want)
E = random error
S = systematic error
O = T+ E
(well ignore S for now, but well return to it later)

Reliability
O=T+E
The error becomes a part of what were measuring
Once weve taken a measurement, we have an equation
with two unknowns. We cant separate the relative
contribution of T and E.
10 = T + E

Reliability: Do random errors accumulate?

Question: If we sum or average multiple observations,
will random errors accumulate?

Reliability: Do random errors accumulate?

Answer: No. If E is truly random, we are just as likely to
overestimate T as we are to underestimate T.

Reliability: Do random errors accumulate?

Obs. 1
Obs. 2
Obs. 3
Obs. 4
Obs. 5
Obs. 6
Obs. 7
Average

O=
10
9
10
11
8
10
12
10

T
10
10
10
10
10
10
10
10

E
0
-1
0
+1
-2
0
+2
0

Note: The average of the seven Os is equal to T

Reliability: Implications
These demonstrations suggest that one important way to
help eliminate the influence of random errors of
measurement is to use multiple measurements.
operationally define latent variables via multiple
indicators
use more than one observer when quantifying
behaviors

Reliability: Estimating reliability

Question: How can we estimate the reliability of our
measurements?
Answer: Two common ways:
(a) test-retest reliability
(b) internal consistency reliability

Reliability: Estimating reliability

Test-retest reliability: Reliability assessed by measuring
something at least twice at different time points.
The logic is as follows: If the errors of measurement are
truly random, then the same errors are unlikely to be
made more than once. Thus, to the degree that two
measurements of the same thing agree, it is unlikely that
those measurements contain a large proportion of
random error.

Reliability: Estimating reliability

You didnt know it at the time, but when we conducted
the subliminal recorded experiment, we assessed the
test-retest reliability of a 6-item measure of self-esteem.
The test-retest correlation was approximately .92,
suggesting that there was very little random error
present in our measurements.
The test-retest correlation is an estimate of the
proportion of true score variance present in a set of
measurements.
About 84% of the variation is true score variation; the
remaining 16% is error.

Person A
Person B
Person C
Person D
Person E
Person F
Person G

Less error
(off by 1 point)

More error
(off by 2 points)

Time
1
1
7
2
6
3
5
4

Time
1
1
7
2
6
3
5
4

Time
2
2
6
3
5
4
4
5

Time
2
3
5
4
4
5
3
6

Reliability: Estimating reliability

Internal consistency: Reliability assessed by measuring
something at least twice within the same broad slice of
time.
Split-half: based on an arbitrary split (e.g, comparing
odd and even, first half and second half)
Cronbachs alpha (): based on the average of all
possible split-halves

Alpha reliability
Click on scale in
the analyze
menu
Choose reliability
analysis

Reliability: Final notes

An important implication: As you increase the number of indicators,

the amount of random error in the averaged measurement
decreases.
An important note: Common indices of reliability range from 0 to 1;
higher numbers indicate better reliability (i.e., less random error).