PSYC2010

Psychological Research Methodology II

Lecture 7 – ANOVA: Effect size, models, and RM

Nick, n.bland@uq.edu.au
 Assignment (20%)
Scenario/dataset now available on Blackboard
- Due Oct. 2nd, 11:55pm

Independent groups ANOVA Lecture 5 / Tutorial 4

- Omnibus F

Follow-up Tests Lecture 6 / Tutorial 5

- Bonferroni correction
- Families of linear contrasts
- Orthogonality ω2 - today
2
 Assignment (20%)
Assignment marked out of 60
- Must be completed to pass the course

Submit electronically via Blackboard/turnitin

- Late submissions lose 3 marks per day
- Equivalent to 1% of course grade
- Guidelines for extensions in ECP, section 5.3

Avoid collusion by completing assignment individually

3
 Today’s Lecture

Relationship between t and F

Effect size in ANOVA
Statistical models for ANOVA
Assumptions of ANOVA
Repeated measures (RM) ANOVA

4
 Key Questions

How is F related to t?
How do interpret eta and omega squared?
What are the advantages and disadvantages of
repeated measures?
How does the partitioning of variance differ between
RM and IG ANOVA?
What are the structural models underlying RM and IG
ANOVA?

5
 F and t2

When k = 2, can perform either a t-test or an F-test

In these cases, the Fobt is equal to the tobt2

For follow-up tests, dfcontrast = 1 (which is why both

methods were shown in tutorials to be equivalent)

6
 α = .05, df = 1,18
F and t2 α = .05, df = 18

2 F = 4.41 = t2
F crit. = t crit.
t2 = 2.101×2.101
7
 Effect sizes

An attempt to assess the importance of an IV in

explaining variance observed in the DV

A low p-value only tells us that the observed effect is

beyond what we would expect by chance…
- But is the effect worth talking about?
- How much variability is actually explained?

8
 Effect sizes
Psychological intervention
(CBT, ACT, control) and
For independent groups ANOVA anxiety management

Source SS df MS F
Treatment 94.533 2 47.267 10.662
Error 53.200 12 4.433
Total 147.733 14

Eta (η) squared – Descriptive (and biased)

2 SSTREATMENT 94.533
η = 2
η = =.6399
SSTOTAL 147.733

64% of the total variability was accounted for by treatment

9
 Effect sizes
Psychological intervention
(CBT, ACT, control) and
For independent groups ANOVA anxiety management

Source SS df MS F
Treatment 94.533 2 47.267 10.662
Error 53.200 12 4.433
Total 147.733 14

Omega (ω) squared – Inferential (less biased)

2SSTREATMENT − (k −1)MS ERROR 94.533− 2 × 4.433
ω = = = .5630
SSTOTAL + MS ERROR 147.733+ 4.433

56% of the total variability was accounted for by treatment

10
 Effect sizes

ω2 generally preferred, since it attempts to estimate

the population effect size (i.e., is inferential)
- More conservative than η2 (64% versus 54%)

Can use Cohen’s (1977) conventions

- Small (.01), medium (.06), and large (.15)

11
 Statistical models

For independent groups ANOVA, each score is

structured as the sum of 3 components:

score for
person i in
X ij = µ + τ j + ε ij
condition j
mu tau epsilon
population/ treatment error for
grand mean effect for person i in
condition j condition j
12
Statistical models
Conan, Xij = 1.69 + .06 + .18 = 1.93m

ε ij

τj Average adult male = 1.75m

µ Average adult = 1.69m

13
 Statistical models
X ij = µ + τ j + ε ij
When we perform the IG ANOVA, we estimate all of
these parameters:

score for X ij = X . + ( X j − X .) + ( X ij − X j )
person i in
condition j
grand mean treatment error for
effect for person i in
condition j condition j
14
 Statistical models

For example, datapoint X72 ‘James’

DV = positive mood
•  Xij
Diet 1: restrict fat Diet 2: restrict Diet 3: no
•  X72 carbs restrictions
•  i = 7
•  j = 2 7 10 12
9 13 11
•  7th person in 8 9 15
2nd condition 12 11 7
8 5 14
7 9 10
•  Xij = 8 4 8 12
10 10 12
•  X̄j = 9 9 8 13
6 7 14
•  X̄. = 9.67
Xj 8 9 12

X . = 9.67 15
 Statistical models

For example, datapoint X72 ‘James’

•  Xij
•  X72 X ij = X . + ( X j − X .) + ( X ij − X j )
•  i = 7
•  j = 2
•  7th person in X 72 = 9.67 + (9 − 9.67) + (8 − 9)
2nd condition

•  Xij = 8 X 72 = 8
•  X̄j = 9
•  X̄. = 9.67 X ij = µ + τ j + ε ij

16
 Assumptions

Three assumptions for independent groups ANOVA:

- Independence of observations
- Normality of the residuals (i.e., ε)
- Put simply, treatment populations are normal
- Homogeneity of variance across treatment populations
- Also called homoscedasticity

Condition 1 Condition 2 Condition 3 17

 Assumptions

Robust to violation of assumptions:

- Normality of the residuals (i.e., ε)

- Okay if only approximate, or even just symmetric

- Homogeneity of variance across treatment populations

- Okay if only approximate, provided roughly equal n
- Largest no more than “four times” the smallest

18
 Data type?

Quantitative (measurement) Qualitative (categorical)

Question about One variable: Two variables:

Hypothesis testing: differences
relationship Chi-squared Chi-squared
goodness of test of
fit independence
Single sample
compared to Comparison between groups
population

If pop. If only Two Groups Multiple Groups Form of relationship Degree of relationship
variance pop. mean
known: known:
z-test t-test

linear Pearson’s
regression correlation &
point biserial
power correlation

Dependent Independent Dependent

Independent
Groups: Groups: Groups:
Groups:
Matched Independent Repeated- Multiple Comparisons Spearman’s
One-way
samples groups measures rho
ANOVA
t-test t-test ANOVA

A priori
Wilcoxon’s Wilcoxon’s (planned): Post hoc: = parametric
Signed Ranks Rank Sum Friedman Kruskal-Wallis Bonferroni t Tukey test tests
test test & Linear Scheffe Test = non-parametric
Contrasts tests
 RM ANOVA

Repeated measures (within-subjects) ANOVA

- Each participant exposed to all conditions
- Can capture consistencies within participants

- Reduces unexplained (error) variance bigger F

20
 RM vs IG ANOVA
For IG ANOVA:
SStotal
The total variability in the data

SStreatment SSerror
The variability attributable to The residual (unexplained) variance
the conditions Includes individual differences

21
 RM vs IG ANOVA
For RM ANOVA:
SStotal
The total variability in the data

SStreatment SSwithin
The variability attributable to The residual (unexplained) variance
the conditions Includes individual differences

SSerror SSparticipants
The residual Individual differences
(unexplained) variance
22
 RM ANOVA

Let’s say a researcher is interested in the effect of

room temperature on problem-solving ability:
- IV: room temperature (three levels)
- 15°C, 25°C, and 35°C
- DV: problem solving
- number of problems solved
- 5 participants
- Each participant exposed to all temperatures

23
 RM ANOVA
Some participants are
naturally better problem-
Raw scores: solvers, irrespective of
temperature!
Temperature (°C)
Participant 15 25 35
1 0 4 1 1.667

participant
2 1 3 2 2.000

averages
3 3 6 2 3.667
4 1 3 0 1.333
5 0 4 0 1.333
1 4 1

means across conditions

24
 RM ANOVA
variability within conditions is either due to
Raw scores: experimental errors or individual differences
Temperature (°C)
Participant 15 25 35
1 0 4 1
2 1 3 2
consistencies across
conditions (i.e., within
3 3 6 2 participants) therefore
reflects individual
4 1 3 0 differences
5 0 4 0

variability between conditions is either due to

experimental errors or treatment differences
25
 Partitioning Variance
SSTOTAL is unchanged SSTOTAL = ∑ ( X − X .) 2
7

6
Problems Solved

2
Grand
Mean
1

0
15 25 35

Temperature (Celsius)
26
 Partitioning Variance
SSTREATMENT is unchanged SSTREATMENT = ∑ nk ( X k − X .) 2
7

6
Problems Solved

2
Grand
Mean
1

0
15 25 35

Temperature (Celsius)
27
 Partitioning Variance
SSPARTICIPANTS is new SS PARTICIPANTS = ∑ ni ( X i − X .) 2
7

6
Problems Solved

4
P3
3

2
Grand
P1 Mean
1

0
15 25 35

Temperature (Celsius)
28
 Partitioning Variance

For IG ANOVA:

SSERROR = SSTOTAL − SSTREATMENT

For RM ANOVA:

SSERROR = SSTOTAL − SSTREATMENT − SSPARTICIPANTS

29
 Statistical models

For repeated measures ANOVA, each score is

structured as the sum of 4 components:
score for
person i in X ij = µ + τ j + π i + εij
condition j

mu tau pi epsilon
population/ treatment participant error for
grand mean effect for effect for person i in
condition j person i condition j
30
 RM ANOVA

Raw scores: SSTOTAL = ∑ ( X − X .) 2

Temperature (°C)
Participant 15 25 35
1 0 4 1
2 1 3 2
3 3 6 2
4 1 3 0
5 0 4 0
Σ 5 20 5 30 Grand mean
nk 5 5 5 15 2.000

31
 RM ANOVA

SSTOTAL = ∑ ( X − X .) 2
(Xij – Grand Mean)2
Participant 15 25 35
1 4 4 1
2 1 1 0
3 1 16 0
4 1 1 4
5 4 4 4
Σ 11 26 9 46 SSTOTAL
46.000

32
 RM ANOVA

Raw scores: SSTREATMENT = ∑ nk ( X k − X .) 2

Temperature (°C)
Participant 15 25 35
1 0 4 1
2 1 3 2
3 3 6 2
4 1 3 0
5 0 4 0
Group Mean 1 4 1 Grand mean
2.000

33
 RM ANOVA

SSTREATMENT = ∑ nk ( X k − X .) 2
(Group Mean – Grand Mean)2
Participant 15 25 35
1 1 4 1
2 1 4 1
3 1 4 1
4 1 4 1
5 1 4 1
Σ 5 20 5 30 SSTREATMENT
30.000

34
 RM ANOVA

Raw scores: SS PARTICIPANTS = ∑ ni ( X i − X .) 2

Temperature (°C)
Participant 15 25 35 Participant Mean
1 0 4 1 1.667
2 1 3 2 2.000
3 3 6 2 3.667
4 1 3 0 1.333
5 0 4 0 1.333
Grand mean
2.000

35
 RM ANOVA

SS PARTICIPANTS = ∑ ni ( X i − X .) 2
(Participant Mean – Grand Mean)2
Participant 15 25 35 Σ
1 0.111 0.111 0.111 0.333
2 0 0 0 0
3 2.778 2.778 2.778 8.333
4 0.444 0.444 0.444 1.333
5 0.444 0.444 0.444 1.333
SSPARTICIPANTS
11.333

36
 RM ANOVA

Remember,

SSERROR = SSTOTAL − SSTREATMENT − SSPARTICIPANTS

SSERROR = 46.000 − 30.000 − 11.333

SSERROR = 4.667

37
 RM ANOVA
Source SS df MS F
Participants n–1
Treatment k–1
Error (n – 1)(k – 1)
Total N–1

Source SS df MS F
Participants 11.333 4
Treatment 30.000 2 15.000 24.714
Error 4.667 8 0.583
Total 46.000 14

38
 RM ANOVA

Fobt = 24.714

dftreatment = 2
dferror = 8

Fcrit = 4.46

39
 RM ANOVA
Fobt = 24.714 > Fcrit = 4.46, so reject null

A repeated measures ANOVA revealed

that room temperature (15°C, 25°C, and
35°C) significantly impacted problem
solving ability, F(2, 8) = 24.42, p < .05.

Follow-up tests would then be needed to

know which conditions differed
significantly…
40
 RM vs IG ANOVA
What if these same scores had come from an
independent groups design?
RM ANOVA:
Source SS df MS F
dftreatment = 2
Participants 11.333 4
Treatment 30.000 2 15.000 24.714 dferror = 8
Error 4.667 8 0.583
Total 46.000 14 Fcrit = 4.46
IG ANOVA:
Source SS df MS F dftreatment = 2
Treatment 30.000 2 15.000 11.250
Error 16.000 12 1.333
dferror = 12
Total 46.000 14
Fcrit = 3.89
41
 RM vs IG ANOVA
IDENTICAL

RM has bigger Fcrit but much bigger Fobt

RM ANOVA:
Source SS df MS F
dftreatment = 2
Participants 11.333 4
Treatment 30.000 2 15.000 24.714 dferror = 8
Error 4.667 8 0.583
Total 46.000 14 Fcrit = 4.46
IG ANOVA:
Source SS df MS F dftreatment = 2
Treatment 30.000 2 15.000 11.250
Error 16.000 12 1.333
dferror = 12
Total 46.000 14
Fcrit = 3.89
42
 RM vs IG ANOVA

RM generally more powerful

- Smaller error term (less unexplained variance)
- However, fewer df error results in larger Fcrit

RM vulnerable to carryover effects

- e.g., problem solving may improve dramatically
- previous conditions (e.g., drug)

43
 Assumptions Less important
than for IG
ANOVA

Three assumptions for repeated measures ANOVA:

- Normality
- Normality of differences is more important
- Homogeneity of variances
- Again, what matters most are the differences
- Homogeneity of covariances
- Difference scores have equal variances
- Also known as sphericity

44
 Next week…

Factorial ANOVA
- Two or more IVs (i.e., no longer ‘one-way’)

Non-parametric alternatives for t-tests

- Wilcoxon’s rank-sum test (independent)
- Wilcoxon’s signed-rank test (paired)

45