SW388R6 Data Analysis and Computers I Slide 1

One-way Analysis of Variance and Post Hoc Tests

Key Points about Statistical Test Sample Homework Problem Solving the Problem with SPSS Logic for One-way Analysis of Variance with Post Hoc Tests Power Analysis

SW388R6 Data Analysis and Computers I Slide 2

One-way Analysis of Variance: Purpose

Purpose: test whether or not the populations represented by the samples have a different mean Examples:

Social work students have higher GPAs than nursing students and education majors Social work students volunteer for more hours per week than education majors, liberal arts majors, and communications majors UT social work students score higher on licensing exams than graduates of Texas State University, Baylor University, and University of Houston

SW388R6 Data Analysis and Computers I Slide 3

One-way Analysis of Variance: Hypotheses - 1

Hypotheses:

Null: mean of population 1 = mean of population 2 = mean of population 3 = mean of population 4 . Versus Research: at least one population mean is different from the others

Decision:

Reject null hypothesis if pSPSS alpha ( relationship)

SW388R6 Data Analysis and Computers I Slide 4

One-way Analysis of Variance: Hypotheses - 2

A significant finding tells us that at least one mean is different from the other, but it does not tell us which one or ones differ. To answer that question, we might try to use t-tests to compare each of the pairs of means to identify which are significantly different. However, the multiple tests would increase the probability of making a Type I error increases above the alpha that we set, i.e. greater than the risk of drawing a wrong conclusion. One strategy for controlling the error rate is the Bonferroni inequality by which we divide alpha by the number of tests we need and only report those that meet this reduced level of significance.

SW388R6 Data Analysis and Computers I Slide 5

One-way Analysis of Variance: Hypotheses - 3

As an alternative, statisticians have developed post hoc tests which control the error rate for the number of comparisons that are made. Thus, for a one-way analysis of variance, we only report findings for combinations of differences if the main test (often referred to as the omnibus test) shows statistical significance. We will use the Tukey HSD (Honestly Significant Difference) test to identify statistically significant differences between pairs of groups.

SW388R6 Data Analysis and Computers I Slide 6

One-way Analysis of Variance: Assumptions and Requirements

Variable is interval level (ordinal with caution) Variable is normally distributed Acceptable degree of skewness and kurtosis or Using the Central Limit Theorem (10+ in each group) The variance of the groups is not different

SW388R6 Data Analysis and Computers I Slide 7

One-way Analysis of Variance: Effect Size - 1

For analysis of variance, the effect size statistic, there are multiple measures of effect size: eta squared (), partial Eta squared (p), omega squared (), and the Intraclass correlation (I) We are particularly interested in eta squared which is the percent of the total variance in the dependent variable accounted for by the variance between categories (groups) defined by the independent variable, i.e. the ratio of the between-groups sum of squares to the total sum of squares.

SW388R6 Data Analysis and Computers I Slide 8

One-way Analysis of Variance: Effect Size - 2

The effect size statistic used in power analysis is f, which is computed as the square root of 1 + Cohens interpretation of f: small: f = .10 to .25 medium: f = .25 to .40 large: f = .40 and higher

SW388R6 Data Analysis and Computers I Slide 9

One-way Analysis of Variance: APA Style - 1

An One-way Analysis of Variance is presented as: F(2,97) = 5.819, p = .004 , = .35
Between groups degrees of freedom Within groups degrees of freedom Value of statistic Significance of statistic Effect size if available

Example: The results indicated a significance difference at the .004 level in patient improvement under three different therapeutic personality type as established by the Handerson Therapeutic Personality Measurement. F (2,97) = 5.819, p = .004.

SW388R6 Data Analysis and Computers I Slide 10

One-way Analysis of Variance: APA Style - 2

Reporting results of a post hoc test: A Tukey HSD post hoc was conducted finding a significance difference in improvement in borderline patients between those seeing a directive therapist and a relational therapist, p = .003. The greatest improvement was found among patients in the directive group (M = .53, SD = .0072) which was significantly greater than the improvement found in the relational group(M = .48, SD = .014). The nondirective sample fell in between these two with a mean of .51( M = .51, SD = .009) and was not statistical significant from either the relational or the directive group means.

SW388R6 Data Analysis and Computers I Slide 11

Homework problems: One-way Analysis of Variance

This problem uses the data set GSS2000R.Sav to compare the average score on the variable "highest year of school completed" [educ] for groups of survey respondents defined by the variable "subjective class identification" [class]. Using a one-way analysis of variance and a post hoc test with an alpha of .05, is the following statement true, true with caution, false, or an incorrect application of a statistic? Survey respondents who said they belonged in the working class completed fewer years of school (M = 12.58, SD = 2.50) than survey respondents who said they belonged in the middle class (M = 13.83, SD = 3.14). o o o o True True with caution False Incorrect application of a statistic
This is the general framework for the problems in the homework assignment on Oneway Analysis of Variance and Post Hoc Tests. The description is similar to findings one might state in a research article.

SW388R6 Data Analysis and Computers I Slide 12

Homework problems: One-way Analysis of Variance - Data set, variables, and sample
This problem uses the data set GSS2000R.Sav to compare the average score on the variable "highest year of school completed" [educ] for groups of survey respondents defined by the variable "subjective class identification" [class]. Using a one-way analysis of variance and a post hoc test with an alpha of .05, is the following statement true, true with caution, false, or an incorrect application of a statistic? Survey respondentsThe first paragraphbelonged in the working class who said they identifies: completed fewer years of school (M = 12.58, SD = 2.50) than survey The data set to use, e.g. GSS2000R.Sav The belonged in be middle in the respondents who saidthey groups that willthecompared class (M = 13.83, SD = 3.14). o o o o
analysis The variable compared in the one-way Anova The alpha level to use for the hypothesis test

True True with caution False Incorrect application of a statistic

SW388R6 Data Analysis and Computers I Slide 13

Homework problems: One-way Analysis of Variance Specific Relationship Tested

This problem uses the data set GSS2000R.Sav to compare the average score on the variable "highest year of school completed" [educ] for groups of survey respondents defined by the variable "subjective class identification" [class]. Using a one-way analysis of variance and a post hoc test with an alpha of .05, is the following statement true, true with caution, false, or an incorrect application of a statistic? Survey respondents who said they belonged in the working class completed fewer years of school (M = 12.58, SD = 2.50) than survey respondents who said they belonged in the middle class (M = 13.83, SD = 3.14).
The second paragraph specifies:

o o o o

The specific groups compared in True the post hoc test True with caution The sample means and standard deviation for the groups being False compared Incorrect application of a statistic

SW388R6 Data Analysis and Computers I Slide 14

Homework problems: One-way Analysis of Variance Choosing an answer

The answer to a problem will be The answer to a problem True with caution if the analysis will be True if the Anova supports the finding in the and Post Hoc Test This problem uses the data set GSS2000R.Savstatement, but the average the problem to compare the supports the finding in dependent variable is problem statement. score on the variable "highest year of school completed"ordinal level. [educ] for

groups of survey respondents defined by the variable "subjective class identification" [class]. Using aaone-way analysis of variance and a post The answer to problem will be False the hoc test with an alpha ofif.05,analysis following statement true, true with is the meets the requirements but does not caution, false, or support the finding in the an incorrect application of a statistic? problem statement. Survey respondents who said they belonged in the working class completed fewer years of school (M = 12.58, SD = 2.50) than survey respondents who said they belonged in the middle class (M = 13.83, SD = 3.14). o o o o
The answer to a problem will Incorrect application of a statistic if the analysis violates the level of measurement requirement, i.e. the dependent variable is nominal/dichotomous level the assumption of normality is violated and the central limit theorem cannot be applied

True True with caution False Incorrect application of a statistic

SW388R6 Data Analysis and Computers I Slide 15

Solving the problem with SPSS: Level of measurement - 1

In a one-way analysis of variance, the level of measurement for the independent can be any level that defines groups (dichotomous, nominal, ordinal, or grouped interval) and the dependent variable is required to be interval level. "Subjective class identification" [class] is ordinal satisfying the requirement for the independent variable.

SW388R6 Data Analysis and Computers I Slide 16

Solving the problem with SPSS: Level of measurement - 2

The dependent variable "highest year of school completed" [educ] is interval level satisfying the requirement for the dependent variable.

SW388R6 Data Analysis and Computers I Slide 17

Solving the problem with SPSS: Evaluating normality - 1

The one-way analysis of variance compares group means and requires that the dependent variable be normally distributed. We will generate descriptive statistics to evaluate normality.

Select the Descriptive Statistics > Descriptives command from the Analysis menu.

SW388R6 Data Analysis and Computers I Slide 18

Solving the problem with SPSS: Evaluating normality - 2

First, move the variable we will use in the t-test, educ, to the Variable(s) list box.

Second, click on the Options button to select the statistics we want.

SW388R6 Data Analysis and Computers I Slide 19

Solving the problem with SPSS: Evaluating normality - 3

First, in addition to the statistics, SPSS has checked by default, mark the Kurtosis and Skewness check boxes on the Distribution panel.

Second, click on the Continue button to close the dialog box.

SW388R6 Data Analysis and Computers I Slide 20

Solving the problem with SPSS: Evaluating normality - 4

Click on the OK button to obtain the output.

SW388R6 Data Analysis and Computers I Slide 21

Solving the problem with SPSS: Evaluating normality - 5

"Highest year of school completed" [educ] did not satisfy the criteria for a normal distribution. The skewness of the distribution (-.137) was between -1.0 and +1.0, but the kurtosis of the distribution (1.246) fell outside the range from -1.0 to +1.0. Having failed the normality requirement using this criteria, we will see if we can apply the central limit theorem by examining the size of each group in the independent variable.

SW388R6 Data Analysis and Computers I Slide 22

Solving the problem with SPSS: One-way analysis of variance - 1

The number of cases in each group is part of the output for the one-way Anova, so we will go ahead and compute that test to continue addressing the issue of normality.

Select Compare Means > One-Way ANOVA from the Analyze menu.

SW388R6 Data Analysis and Computers I Slide 23

Solving the problem with SPSS: One-way analysis of variance - 2

First, move the dependent variable educ to the Test Variable(s) list box.

Second, move the independent variable class to the Grouping Variable text box.

Third, click on the Post Hoc button to select the tests to compare pairs of means.

SW388R6 Data Analysis and Computers I Slide 24

Solving the problem with SPSS: One-way analysis of variance - 3

First, we mark the checkboxes for the Tukey HSD test and the Games-Howell test. We will choose which one of these we interpret based on the results of the test of homogeneity of variance.

Second, if the level of significance were different than the default of .05, we would change it here.

Third, click on the Continue button to close the dialog box.

SW388R6 Data Analysis and Computers I Slide 25

Solving the problem with SPSS: One-way analysis of variance - 4

Next, we click on the options test to specify additional output.

SW388R6 Data Analysis and Computers I Slide 26

Solving the problem with SPSS: One-way analysis of variance - 5

First, we mark the check boxes for Descriptive statistics and for the Homogeneity of variance test.

Second, click on the Continue button to close the dialog box.

SW388R6 Data Analysis and Computers I Slide 27

Solving the problem with SPSS: One-way analysis of variance - 6

Having provided the specifications for the analysis, we click on the OK button to request the output.

SW388R6 Data Analysis and Computers I Slide 28

Solving the problem with SPSS: Evaluating normality with the central limit theorem - 7

Since each class group had more than 10 members, we are able to satisfy the normality assumption with the Central Limit Theorem.

If we are unable to establish normality either by the distribution or by the central limit theorem, a one-way Anova would not be an appropriate statistic.

SW388R6 Data Analysis and Computers I Slide 29

Solving the problem with SPSS: Evaluating equality of group variances

The probability associated with Levene's Test for Equality of Variances (p=0.038) is less than or equal to alpha of .05. The assumption of equal variances on the dependent variable across groups defined by the independent variable is not satisfied. However, the analysis of variance is robust to violations of this assumption, and we can base the interpretation of post hoc differences on Games and Howell's modification of Tukey's HSD. The Games-Howell post hoc test adjusts for violations of the homogeneity of variance assumption.

SW388R6 Data Analysis and Computers I Slide 30

Solving the problem with SPSS: Answering the question - 1

The finding we are trying to verify is: Survey respondents who said they belonged in the working class completed fewer years of school (M = 12.58, SD = 2.50) than survey respondents who said they belonged in the middle class (M = 13.83, SD = 3.14).
We first check the significance of the F-test to make certain there are overall differences among groups.

The probability of the Anova test statistic was less than or equal to the alpha level of significance of .05 (F(3, 264) = 4.97, p < .01). The null hypothesis that the mean "highest year of school completed" is the same for all groups defined by the variable "subjective class identification" [class] is rejected.

SW388R6 Data Analysis and Computers I Slide 31

Solving the problem with SPSS: Answering the question - 2

The finding we are trying to verify is: Survey respondents who said they belonged in the working class completed fewer years of school (M = 12.58, SD = 2.50) than survey respondents who said they belonged in the middle class (M = 13.83, SD = 3.14).

Based on the Games-Howell post hoc test, the difference in the mean for survey respondents who said they belonged in the working class (12.58) and the mean for survey respondents who said they belonged in the middle class (13.83) was 1.25, a statistically significant difference at the 0.05 level of significance.

SW388R6 Data Analysis and Computers I Slide 32

Solving the problem with SPSS: Answering the question - 3

The means and standard deviations cited in the problem statement are correct. The statement that "survey respondents who said they belonged in the working class completed fewer years of school (M = 12.58, SD = 2.50) than survey respondents who said they belonged in the middle class (M = 13.83, SD = 3.14)" is correct.

The answer to the question is True.

SW388R6 Data Analysis and Computers I Slide 33

Logic for one-way analysis of variance: Level of measurement

Measurement level of independent variable? Any level defining groups

Measurement level of dependent variable? Interval/ordinal Nominal/ Dichotomous Inappropriate application of a statistic

Strictly speaking, the test requires an interval level variable. We will allow ordinal level variables with a caution.

SW388R6 Data Analysis and Computers I Slide 34

Logic for one-way analysis of variance: Assumption of normality

If each group contains 10 or more cases, we apply the Central Limit Theorem.

Skewness and Kurtosis between -1.0 and +1.0? Yes

No

Number of cases in each group is at least 10?

No Inappropriate application of a statistic

Yes

SW388R6 Data Analysis and Computers I Slide 35

Logic for One-way ANOVA: Assumption of equality of variances

Yes

Probability for Levene test of homogeneity of population variances less than or equal to alpha?

No

Interpret Games-Howell Post Hoc Test

Interpret Tukey HSD Post Hoc Test

SW388R6 Data Analysis and Computers I Slide 36

Logic for one-way analysis of variance: Decision about null hypothesis

Yes

Probability for F-test less than or equal to alpha?

No False

SW388R6 Data Analysis and Computers I Slide 37

Logic for one-way analysis of variance: Decision about post hoc test

Yes

Probability of post hoc test for the specified comparison alpha? No False

Direction of relationship, means, and standard deviations correctly stated? Yes Add caution for ordinal variable. No False

True

SW388R6 Data Analysis and Computers I Slide 38

Power Analysis: One-way ANOVA Problem that was False

This problem uses the data set GSS2000R.Sav to compare the average score on the variable "occupational prestige score" [prestg80] for groups of survey respondents defined by the variable "respondent's degree of religious fundamentalism" [fund]. Using a one-way analysis of variance and a post hoc test with an alpha of .01, is the following statement true, true with caution, false, or an incorrect application of a statistic? Survey respondents who said they were religiously fundamentalist had less prestigious occupations (M = 42.64, SD = 13.40) than survey respondents who said they were religiously moderate (M = 43.92, SD = 13.62). 1 2 3 4 True True with caution We can conduct a post-hoc power analysis to determine if the number of False available cases was needed to find a statistically Incorrect application of a statistic significant difference.
The answer to this problem was false because the probability for the t-test was . 476 (one-tailed), greater than the alpha of 0.01.

SW388R6 Data Analysis and Computers I Slide 39

Power Analysis: Statistical Results for False One-way ANOVA - 1

The answer to the problem was false because the one-tailed significance was p = .29 (.583 2), greater than the alpha of .05.

SW388R6 Data Analysis and Computers I Slide 40

Power Analysis: Statistical Results for False One-way ANOVA - 2

To calculate the effect size, and corresponding power, for this problem, we need a pooled estimate of the standard deviation for the two groups. SamplePower will calculate that for us, we will enter the sample sizes, means, and standard deviations for the two groups in SamplePower.

SW388R6 Data Analysis and Computers I Slide 41

Access to SPSSs SamplePower Program

The UT license for SPSS does not include SamplePower, the SPSS program for power analysis. However, the program is available on the UT timesharing server. Information about access this program is available at this site.

SW388R6 Data Analysis and Computers I Slide 42

Power Analysis for One-way ANOVA - 1

In the SamplePower program on the ITS Timesharing Systems, select the New command from the File menu.

SW388R6 Data Analysis and Computers I Slide 43

Power Analysis for One-way ANOVA - 2

First, select the Means tab to access the tests for means.

Second, since we want to enter the means for our two groups, select the option button for t-test for 2 (independent) groups with common variance (Enter means)

Third, click on the Ok button to enter the specific values for our problem.

SW388R6 Data Analysis and Computers I Slide 44

Power Analysis for One-way ANOVA - 3

Position the mouse cursor over the entry for number of levels (2). The cursor will change from an arrow to a crosshair. Click on the number 2 to open the dialog box for entering means and standard deviation.

SW388R6 Data Analysis and Computers I Slide 45

Power Analysis for One-way ANOVA - 4

Click the up arrow button on the spinner to change the Number of categories in this factor to 3.

SW388R6 Data Analysis and Computers I Slide 46

Power Analysis for One-way ANOVA - 5

First, click on the tab Enter SD to enter the Between Groups variance.

Second, enter the Between Groups variance (147.28) from the ANOVA table.

SW388R6 Data Analysis and Computers I Slide 47

Power Analysis for One-way ANOVA - 6

First, click on the tab Enter Means to enter means for the three groups.

Second, type the means for the three groups from the ANOVA output: 42.63, 43.92, and 45.49.

SW388R6 Data Analysis and Computers I Slide 48

Power Analysis for One-way ANOVA - 7

To compute the effect size statistic, f, for ANOVA problems, click on the Compute f button. Note: the effect size f is not the same as the F-ratio statistic in the ANOVA table.

SW388R6 Data Analysis and Computers I Slide 49

Power Analysis for One-way ANOVA - 8

Note: this is the effect size f based on the entries thus far. Since we still have more information to enter, the actual f will change.

When you click on the yellow Compute f button, it changes to a turquoise Register f button. When you click on the Register f button, the effect size f will be copied over to the main window and the dialog box will be closed.

SW388R6 Data Analysis and Computers I Slide 50

Power Analysis for One-way ANOVA 9

First, enter the Within Groups variance (197.89) from the ANOVA table.

Second, click on the Compute button to update the calculations for effect size and power.

SW388R6 Data Analysis and Computers I Slide 51

Power Analysis for One-way ANOVA 10

Next, click on the 0.05 for Alpha= to change it to 0.01, as stated in the problem.

Note: when you clicked on the Compute button, SamplePower updated its calculations and removed the Compute button.

SW388R6 Data Analysis and Computers I Slide 52

Power Analysis for One-way ANOVA 11

First, mark the option button for 0.01 in the Alpha panel.

Second, click on the Ok button to close the dialog box.

SW388R6 Data Analysis and Computers I Slide 53

Power Analysis for One-way ANOVA - 12

The effect size for our problem is f = 0.08, below the standard of 0.10 which Cohen labels as small. The power to reject the null hypothesis is 0.07. We should expect to correctly reject the null hypothesis in 1 of 14 tries.

Finally, we click on the up arrow button on the N of cases per cell spinner to change it to 80. SamplePower will only compute power for balanced designs where the number of cases is the same in each group. The closest we can get to the total number of 241 cases in our problem is 80 in each group.

SW388R6 Data Analysis and Computers I Slide 54

Power Analysis for One-way ANOVA - 13

To find the group sizes needed, select Find N for power of 80% from the Tools menu.

SW388R6 Data Analysis and Computers I Slide 55

Power Analysis for One-way ANOVA - 14

SamplePower indicates that we would have needed a total sample of 2016 to detect this very small effect size in the population.

Since unequal cells sizes (e.g. 88, 97, and 78 in this problem) reduce power, we should treat the estimate of 2016 cases as conservative. SAS has provisions for unequal cell sizes. The SAS estimate for the number of cases needed for power of 0.80 was 2410 cases.