PhD Rankings

Kat Bechkoff Doina Chichernea

Overview
Purpose Lit Review Survey Model Survey Method Findings Conclusion Further Research

Purpose
1. Is it important to rank PhD programs? 2. Are there appropriate PhD rankings in place? 3. Is it important to rank the college or the department? 4. What factors should be included in a ranking system?

Lit Review
Two types of rankings
Objective (Publication-Based)
The Conroy-Dusansky and Dusansky-Vernon Scott and Mitias Rankings

Subjective (Survey-Based)
US News and World Report National Research Council

Lit Review

Publication-Based Rankings
Dominant recent approach Look at outputs that are directly measurable
Publications Number of pages published Citations

Counts might be adjusted for perceived quality or importance or they might be adjusted to per capita counts

Lit Review

Publication-Based Rankings
Pros Familiar to faculty and deans Methods in place to insure accuracy of data. Academic journals considered are fairest measure of quality research Cons Publication counts provide little evidence on importance of research Not easily subjected to statistical testing Problems:
Which journals are considered important? How does one compare a page in one journal with a page in another? Over what time period are the counts to be made?

Lit Review

Survey-Based Method
US News and World Report Rankings Mail surveys to dept. heads and directors of graduate studies in those universities which had graduated at least 5 PhDs in the past five years Recipients are asked to assess each department on an integer scale of 1 to 5 as to: reputation in terms of scholarship, curriculum, faculty and graduate programs Scale: 5 = distinguished, 4 = strong, 3 = good, 2 = adequate, and 1 = marginal Each dept. was ranked based on its average response.

Lit Review

Survey-Based Method Pros Easy side by side comparisons Influence on new generations of grad students Cons Vulnerable to measurement errors Surveys may not properly reflect research output (takes time for perceptions to change)

Lit Review

Publication vs. Survey Based Ranking

Ranking systems based upon publications or alternately based on surveys of reputation will present consistent findings
Correlation coefficients within the groups is in excess of .9

Correlations btw publication & survey based rankings are high in statistical sense, yet slightly lower than within groups differing underlying information
Correlation coefficients within groups were typically in the . 6-.8 range

(Vernon Dusansky 1998)

Lit Review

MBA Rankings vs Research Rankings

Refer to different constituencies:

Students and business practitioners Academics

It is critical that accepted measures of performance capture the goals of multiple constituencies The risk when they do not is perverse learning - Universities will strive to meet only the measured performance objectives and disregard the unmeasured ones. (Meyer & Gupta 1994)

Lit Review
Conclusion:

MBA Rankings vs Research Rankings

Schools having strong MBA programs based on popular press rankings (US News and World Report; Business Week) may not be especially strong on the research dimension Faculty research rankings are significantly different from academic program rankings
(Trieschmann et al 2000)

Focusing only on one of these measures (popular press MBA rankings) threatens a schools long-term viability

Model
Are PhD Rankings Important? College vs. Department Current Rankings?

Factors

FacultyRelated Factors

Research Output Factors

Marketing Factors

Selectivity Factors

Program Factors

AlumniRelated Factors

StudentRelated Factors

Placement Factors

Survey Methods
Web-based Questionnaire 7pt Likert Scale
Very important Very unimportant

2 Distributions
PhD Coordinators (21 respondents) U.C. CoB Faculty (26 respondents)

Findings

1. Are PhD Rankings Important?

Overall: Important = 72.36% Neutral = 12.76% Unimportant = 14.88% Conclusion: PhD rankings ARE considered to be important!
Strongly Agree
Faculty 30.8% Coord. 47.6%

Strongly Disagree

2. Are There Appropriate PhD Rankings in Place?

Overall: Agree = 36.17% Neutral = 25.53% Disagree = 38.3% Conclusion: There are not appropriate PhD rankings in place.
Strongly Agree
Faculty 34.6% Coord. 47.6%

Strongly Disagree

3. Ranking the College vs. Department

Department College Agree Neutral 74.46% 10.64% 74.46% 10.64% 14.89% Conclusion: College and department rankings are equally important.

Disagree 14.89%
Strongly Agree
Faculty 42.3% Coord. 47.6%

Strongly Disagree

Dept.

Coord. Faculty 33.3% 30.8%

College

4. Factors
Selectivity Factors Program Factors Marketing Factors Research Output Factors

StudentRelated Factors

FacultyRelated Factors

AlumiRelated Factors

Placement Factors

We asked the respondents to rate each factor in terms of its importance, since our purpose was to determine which factors should be included in a PhD ranking.

Factors Faculty vs. Coordinators

Wilcoxon Signed Rank test Results indicated a significant difference btw the response averages of the two groups!
Tes t Statisticsb Faculty Co o rdinato rs -3.569 a .000

Z Asymp. Sig. (2-tailed)

a. Based o n negative ranks. b. Wilco xo n Signed Ranks Test

Factors Faculty vs. Coordinators

Why was there a difference?
Des criptive Statis tics N Co o rdinato rs Faculty 40 40 Mean 2.4425 2.6927 Std. Deviatio n .61925 .84218 Minimum 1.19 1.15 Maximum 4.29 4.46

Faculty members responses were more varied than that of the coordinators. Overall, coordinators considered the ranking factors more important than faculty did.

Objective Factors Vs. Subjective Factors

Objective Categories
Selectivity Factors Placement Factors Research Output Factors

Subjective Categories
Program Factors Marketing Factors

AlumiRelated Factors FacultyRelated Factors

StudentRelated Factors

Objective Factors Vs. Subjective Factors

Conclusion: Objective more important than subjective!
Objective 2.733648542 0.625635351 27 0 37 2.413533667 0.01043291 1.687093597 0.020865821 2.026192447 Subjective 2.263502455 0.19205813 13

Mean Variance Observations Hypothesized Mean Difference df t Stat P(T<=t) one-tail t Critical one-tail P(T<=t) two-tail t Critical two-tail

Overall Factor Importance

Most Important 1. Faculty support 2. Quality of job placement 3. # of faculty publications (weighted by journal importance) Least Important 1. Undergrad student to faculty ratio 2. Total # pages of faculty works published 3. Work experience of accepted students

Conclusion
Phd Rankings are important There are not appropriate PhD rankings in place College and department rankings are equally important Objective factors are weighed heavier in terms of importance than subjective factors.

Further Research
Perhaps a more exhaustive set of factors could be tested
Received Suggestions: Internatl exposure, Student true superstars, hands-on research, one-to-one mentoring, # of quality PhD seminars, student/professor match-up.

Increase sample size Collect data on student opinions