Académique Documents
Professionnel Documents
Culture Documents
(50%)
(25%)
(25%)
(7.5%)
(12.5%)
(2.5%)
(2.5%)
(32.5%)
(10%)
(22.5%)
3. Student Debt
(25%)
Average Federal Student Loan Debt Load
(10%)
Student Loan Default Rates
(12.5%)
Predicted vs. Actual Average Federal Student Loan Debt Load
(2.5%)
4. Four-year Graduation Rate
Actual Four-year Graduation Rate
Predicted vs. Actual Four-year Graduation Rate
(7.5%)
5. Academic Success
Student Nationally Competitive Awards
Alumni Receiving PhDs
(10%)
(5%)
(2.5%)
(7.5%)
(2.5%)
School Selection
The 650 institutions of higher education included in this list award undergraduate degrees or
certificates requiring 4 or more years of study, according to the U.S. Department of Education
and are classified by The Carnegie Foundation as Doctorate-granting Universities, Masters
Colleges and Universities, or Baccalaureate Colleges.2 We have accounted for any changes in the
names of institutions that have occurred over the past year.
In one sense, students are consumers of the education colleges and universities offer, with the
core dimension of the learning experience coming from classes taught by instructors. Asking
students what they think about their courses is akin to what some agencies like Consumers
Report or J.D. Powers and Associates do when they provide information on various goods or
services. As Otto, Sanford and Ross note, students who post ratings on the website can be
viewed as experts due to their significant experience with the professor(s) they are evaluating.3
Considering the popularity of RateMyProfessors.com (RMP), with students themselves using the
ratings to develop expectations about faculty members and set their schedules, we agree with
these scholars when they argue that online ratings should be taken seriously, albeit in a defined
and limited manner.
To be sure, the use of this instrument is not without criticism. Some would argue that only an
unrepresentative sample of students complete the forms. In some cases, the results for a given
instructor might be biased because only students who are extremely unhappy (or, conversely,
extremely happy) with a course or instructor complete the evaluation, while in other instances
perhaps an instructor urges students liking the course to complete the evaluation, biasing results
in the opposite direction.4
It is possible that the concern regarding biases has some validity as it applies to individual
instructors. But when the evaluations of dozens or even hundreds of instructors are added
together, most examples of bias are washed outor any systematic bias that remains is likely
relatively similar from campus to campus. What is important, for our purposes, is the average
course evaluation for a large number of classes and instructors, and the aggregation of data
should largely eliminate major inter-school biases in the data. In fact, on an institutional level,
there is some evidence that higher RMP scores are correlated with fewer evaluations; that is, the
lower the number of RMP evaluations per enrollment, the greater the schools composite RMP
score.
The other main objection to the RMP measure is that instructors can buy high rankings by
making their course easy and giving high grades. Again, to some extent the huge variations in
individual instructor rigor and easiness are reduced when the evaluations of all instructors are
aggregated nearly every school has some easy and some difficult professors. Nonetheless, we
take this criticism seriously and controlled for the effect that easiness has on evaluations of
instructor or course quality. Perhaps the simplest method for controlling for the bias that easiness
(or, in this case, perceived easiness) plays in instructor evaluations is to adjust the RMP Overall
Quality ratings by the Easiness ratings students report. Furthermore, there are additional
theoretical reasons (besides simply controlling for bias) to take course difficulty into account.
For example, arguably courses which students perceive to be the most difficult and challenging
are precisely those courses in which more students learning gains are the greatest, regardless of
the quality students otherwise perceive. For these reasons, we gave special consideration to the
difficulty factor in the measurement of this factor, as discussed below.
Scholarly Assessments of RateMyProfessors.com
There have been a number of studies assessing the validity and/or reliability of the RMP ratings
data. The general approach is to relate the results on this website to the more established student
3
evaluations of teaching (SET) that are routinely performed by most institutions of higher
education. Many colleges think their own SET data provide useful information in assessing the
effectiveness of faculty and instruction (after all, these evaluations are a consideration for
promotion and teaching at many educational institutions).5,6 Therefore, if these institutional
evaluations are mirrored by the RMP results, it enhances the likelihood that RMP data are a valid
measure vis--vis official SET data.
The research to date cautiously supports the view that RMP is relatively similar to the SET used
by universities themselves. One study states, The results of this study offer preliminary support
for the validity of the evaluations on RateMyProfessors.com.7 Coladarci and Kornfield,
surveying instructors at the University of Maine, note that ...these RMP/SET correlations should
give pause to those who are inclined to dismiss RMP indices as meaningless, although they also
expressed some concerns that the correlation between the two types of instruments were far from
1.00.8 Otto, Sanford and Ross conclude that their analysis of ratings on RMP reveals what would
be expected if the online ratings of professors were in fact valid measures of student learning.9
More recently, Bleske-Rechek and Michels, in an analysis of students across majors at a single
state university, contradict the popular notion that students who use RMP only post highly
negative or highly positive ratings.10 Bleske-Rechek and Michels also conclude that the evidence
does not support the common assumption that students who post on RMP are not typical of the
whole student body.
To be sure, the research is not all enthusiastically supportive of RMP. Felton, Koper, Mitchell,
and Stinson suggest that the positive correlation between RMP quality ratings and ease of course
assessments make this a questionable instrument.11 Bleske-Rechek and Michels confirm the
existence of a positive relationship between student evaluations of quality and easiness at the
instructor level, Bleske-Rechek and Michels warn that it is misguided to jump to the conclusion
that the association between easiness and quality is necessarily a product of just bias and
suggest that the RMP data may only be reflecting that quality instruction facilitates learning.12
However, regardless of the precise causes of positive relationship between student assessments
of quality and easiness, we have adjusted the RMP score for course easiness to correct for this
potential bias.
In spite of some drawbacks of student evaluations of teaching, they apparently have value for the
86% of schools that, historically, have some sort of internal evaluation system. RMP ratings give
similar results to these systems. Moreover, they are a measure of consumer preferences, which is
what is critically important in rational consumer choice. When combined with the significant
advantages of being uniform across different schools, not being subject to easy manipulation by
schools, and being publicly available, RMP is a preferred data source for information on student
evaluations of teachingit is the largest known single uniform data set for student perceptions of
the quality of their instruction.
Calculating the Schools Scores
We took the average overall quality rating for all instructors at each school based on the quality
ratings of individual professors listed on the RMP website, reported on a scale of 1 to 5. We also
derived an estimate for student perception of course rigor from the reported RMP easiness
4
variable. The RMP easiness variable, like the overall quality variable, is based on a scale from 1
to 5, with 5 being the easiest. To establish a measure of course rigor, we invert the scale of the
rating by subtracting the easiness score from 6 to yield a course rigor variable, also on a scale
also of 1 to 5.
We computed the overall RMP score by summing the quality weighting with the derived rigor
rating, such that the weighting for the quality rating was three times higher than the weighting
for the derived rigor rating. This composite score was then adjusted using Bayesian methods that
consider the number of votes submitted.13 This composite score accounts for 7.5 percent of the
final score for each school in the Forbes ranking.
amount of Pell grant received, percentage using financial aid, and the average amount of federal
loans students borrow. We also included measures of institutional selectivity: 25th percentile
SAT scores, percent of applicants admitted and the matriculation rate. We also included various
measures of per student spending, the student-faculty ratio, tuition and enrollment. Finally, we
also controlled for type of institution, degree of campus urbanization and geographic location.
In constructing the model, we first transformed the retention rate variable with the logistic
transformation (also known as the log of the odds ratio) to account for the bounded nature of the
retention rate data. We then regressed this variable against the aforementioned independent
variables using the least squares method. Due to the nature of the logistic transformation, we do
not encourage interpretation of coefficient estimates on retention rates and therefore suppress
them in this methodology.14
A school received a better score for this metric by having an actual retention rate that exceeded
that predicted by the regression model, controlling for difference between the actual retention
rate and the perfect retention rate of 100 percent. Conversely, a school received a lower score
if the actual retention rate fell below the models predicted rate. In the overall Forbes rankings,
this retention model accounts for 2.5 percent.
which is updated frequently, is one of the largest online salary surveys in the world. Persons
complete the PayScale Salary Survey in exchange for a free salary report that anonymously
compares them to other people with similar jobs in similar locations. In addition to individual
surveys, PayScale receives data from employers administered on behalf of trade associations.
Why this measure?
For many (if not most) college students, the bottom line of higher education is whether it helps
them get a good job after graduation. Other things being equal, students will choose a school that
provides them the opportunity to earn the highest possible salary upon graduation. Historically, it
has been difficult, if not impossible, to obtain institutional level data on the salaries of college
graduates. The salary data published by Payscale is among the most comprehensive salary data
publicly available and these data are increasingly used by higher education policy analysts
seeking to quantify the economic returns to higher education and compare how those returns
vary by school and major of study.15 While some may criticize the Payscale data as unreliable
due to problems associated with self-selection bias, we view these data as reasonably reliable,
particularly in light of the fact that there is not better institutional level data available at the same
scope and because these data are used by credible higher education policy analysts in other
research.
Calculating the Schools Scores
We used salary data for 1-4 years experience and data for 10-19 years experience. Specifically,
we used salary data for 1-4 years experience as an estimate for beginning salaries and the growth
in salary from 1-4 years experience to 10-19 years experience. Taking the growth rate between 14 year and 10-19 year is an indicator of value-added skills that were learned during school, both
technical and soft skills. In other words, we believe that the acceleration of growth in salary is
just as important starting salary throughout a career. There were 44 schools that did not have
Payscale data available so their scores were reweighted accordingly. The composite salary score
is weighted at 10 percent of the overall rankings.
Forbes 100 Largest U.S. Charities and the individuals listed on Forbes 30 under 30 and
Worlds Most Powerful Women. Additionally, we gathered information on those who have
been elected to the National Academy of Sciences or have been awarded a Nobel Prize (20042013), Pulitzer Prize (2004-2013), Guggenheim Fellowship (2005-2014), or MacArthur
Fellowship (2004-2013). Our list of American Leaders contains members of the Big Five
orchestras (located in New York, Boston, Chicago, Philadelphia, and Cleveland) and winners of
an Academy (2005-2014), Emmy (2004-2013), Tony (2004-2013), or Grammy Award (General
Field, 2004-2013).
We have also included biographical information for nearly 1,900 federal officials, including the
governors of all 50 states and the four American territories (American Samoa, Guam, Northern
Mariana Islands, and Puerto Rico), all one hundred United States Senators and 441 United States
Representatives (including delegates from the American territories and the District of Columbia),
all 849 Article III judges appointed by the President and confirmed by the Senate for lifetime
appointments (a number which includes nine Supreme Court Justices, 177 Court of Appeals
judges, 9 Court of International Trade Judges, and 654 District Court judges), and 506 members
of the executive branch (including the President and Vice President of the United States,
members of the Cabinet, senior members of the bureaucracy within each Cabinet department,
and the heads of major non-cabinet agencies). We have also included the President and Board of
Directors of the 12 Federal Reserve banks.
Data on the educational background for each of these leaders was obtained through various
sources. We removed any duplicates (for instance, an individual serving as CEO at one of the
leading corporations who also served on the board of one of the leading charities) to avoid
double counting. After obtaining our complete sample, totaling approximately 30,000 names, we
removed the duplicates, as well as the individuals for whom no relevant collegiate information
could be found. This process produced our total sample size of 17,600 usable individuals. After
computing the absolute number of leaders who received degrees from each school, we adjusted
the absolute number by the average undergraduate enrollment at each institution in 1980, 1985,
1990, 1995, 2000, and 2005 in such a way as to both account for the absolute number of notable
alumni a particular college produced as well as that number of alumni as a proportion of the
historic enrollment. This composite score accounts for 22.5 percent of the overall Forbes/CCAP
ranking for each institution.
Two private institutions in our sample (Hillsdale College and Grove City College) did not have
any students who took out federal student loans in 2012. The debt data are weighted at 10
percent of the overall Forbes ranking.
Predicted vs. Actual Average Federal Student Loan Debt Load (2.5%)
We have also incorporated a model which predicts the average amount of federal student loan
debt each undergraduate will accrue so we can evaluate institutions on how affordable they are,
given certain characteristics of the schools and their respective student bodies. For this measure,
we used only results of the statistical model as a component of the rankings.
We used a number of student demographic factors in our model including proportional
enrollment by ethnicity and the percentage of undergraduates who are non-resident aliens.
Additionally, we used student financial aid variables, such as the average amount of grant aid
received, percentage receiving Pell grants, and institutional support spending per student. We
also included measures of institutional selectivity such as 25th percentile SAT scores, percent of
applicants admitted and the matriculation rate. We also controlled for endowment funds per
student, tuition and enrollment. Finally, we controlled for type of institution, degree of campus
urbanization and geographic location.
In constructing the model, we first transformed the average debt load with the simple logarithmic
transformation. We then regressed this new variable against the aforementioned independent
variables using the least squares method. Similar to the retention rate and graduation rate models,
we do not encourage interpretation of coefficient estimates and therefore suppress them in this
methodology. The results from our statistical model account for 2.5 percent of the overall
ranking.
10
We rely upon a statistical model to predict what a schools four-year graduation rate is expected
to be based on a number of input criteria which measure the academic quality of incoming
students. In order to capture the quality of students, we use 25th percentile composite SAT
scores, acceptance rates, full-time enrollment rates (how many admitted students actually
matriculate), percentage of students receiving Pell Grants, percentage of students enrolled in
STEM majors,18 a dummy variable for public or private institutional control, and regional
dummy variables. We first transformed the four-year graduation rate data with the logistic
transformation (occasionally referred to as the log of the odds ratio) to account for the particular
bounded nature of that variable. We next regressed this transformed variable against the list of
regressors mentioned above using the least squares method. Due to the nature of the logistic
transformation, and the history of even respected academics misinterpreting the coefficient
estimates, we do not encourage interpretation of coefficient estimates on graduation rates and
therefore suppress them in this methodology.
A school received a better score by having an actual graduation rate that exceeded the one
predicted by the regression model, controlling for difference between the actual graduation rate
and the perfect graduation rate of 100 percent. Conversely, a school received a lower score if
the actual graduation rate fell below the models predicted rate.
A Note on the Data Sources
The primary source for the data used was the U.S. Department of Educations IPEDS database.
The actual graduation rate, according to IPEDS, is computed by dividing the total number of
students completing a bachelor degree or equivalent within 4-years (100% of normal time) by
the revised bachelor sub-cohort minus any allowable exclusions. The IPEDS database is also
the source of the data for the variables used in the statistical model, with a few exceptions. There
were several schools (notably Hillsdale College) where the data came from other sources. In
cases where current data were unavailable at any of these sources, we developed estimates based
on the most recent publicly available data.
The graduation rate component accounts for 7.5 percent, apportioned between the actual
graduation rate (5 percent) and the graduation performance of a school relative to its predicted
graduation rate (2.5 percent).
adjusted numbers for student award recipients account for 7.5 percent of the final score for each
school in the overall ranking.
13
simply because they have low tuition. In the spirit of maintaining a list that indicates high
quality, and not just low tuition, this adjustment is necessary).
14
Endnotes
1
The compilation of these rankings was done at the Center for College Affordability and Productivity, although in
cooperation and active consultation with the staff of Forbes. At CCAP, Director Richard Vedder, with the assistance
of Joshua Distel, oversaw the project team which included: Ryan Brune, Amanda Denhart, Virginia Ewen, Zakary
Frank, Daniel Garrett, Devon Halliday, Robert Hammer, Joseph Hartge, David Holman, Samuel Kissinger, Matthew
LeBar, Trevor Marburger, Timothy McKenna, Jacob Newton, Brady OBrien, Maxwell Pristic, Jonathan Robe,
James Roy, Michael Stock, Justin Strehle, William Vosler, and Jasper Wirtshafter.
2
For further information on this system of classification, see The Carnegie Foundation for the Advancement of
Teaching, The Carnegie Classification of Institutions of Higher Education, available at:
http://classifications.carnegiefoundation.org/, accessed July 21, 2014.
James Otto, Douglas A. Sanford Jr., and Douglas Ross. Does ratemyprofessor.com really rate my professor?
Assessment & Evaluation in Higher Education 33, no. 4 (August 2008): 355-368.
3
We should note, however, that if we assume that the distribution of student ratings on ratemyprofessor.com is
symmetric about the mean and that the students who submit those ratings are only either disgruntled or excessively
happy about their professors (that is, the censoring decision is also symmetric about the mean), then given a
sufficiently large sample, the expected mean using the truncated sample would, in fact, be equivalent to the true
mean using a sample of all students. For a formal mathematical proof, see:
http://www.centerforcollegeaffordability.org/uploads/RMP_Proof.pdf
James Otto, Douglas Sanford and William Wagner. Analysis of Online Student Ratings of University Faculty.
Journal of College Teaching and Learning 2, no. 6 (June 2005): 25-30.
5
Yining Chen and Leon B. Hoshower. Student Evaluation of Teaching Effectiveness: An Assessment of Student
Perception and Motivation, Assessment & Evaluation in Higher Education, 28, no. 1 (2003).
6
Michael E. Sonntag, Jonathan F. Bassett, and Timothy Snyder. An Empirical Test of the Validity of Student
Evaluations of Teaching Made on RateMyProfessors.com, Assessment & Evaluation in Higher Education (July
2008). See also, Scott Jaschik. Validation for RateMyProfessors.com? Inside Higher Ed, April 25, 2008, available
at http://www.insidehighered.com/news/2008/04/25/rmp, accessed July 1, 2013.
7
Theodore Coladarci and Irv Kornfield, RateMyProfessors.com Versus Formal In-class Evaluations of Teaching,
Practical Assessment, Research & Evaluation (May 2007).
8
April Bleske-Rechek and Kelsey Michels. RateMyProfessors.com: Testing Assumptions about Student Use and
Misuse, Practical Assessment, Research and Evaluation 15, no. 5 (May 2010).
10
James Felton, Peter T. Koper, John Mitchell, and Michael Stinson. Attractiveness, Easiness, and Other Issues:
Student Evaluations of Professors on RateMyProfessors.com, the abstract page of which is available on
http://ssrn.com/abstract=918283, accessed on August 5, 2010.
11
12
For further discussion of Bayesian approaches to rankings see James O. Berger and John Deely. A Bayesian
Approach to Ranking and Selection of Related Means With Alternatives to Analysis-of-Variance Methodology
Journal of the American Statistical Association 83, no. 402 (June 1988): 364-373.
13
15
14
One prominent example of a misinterpretation of a logistic regression coefficient is discussed in the following
letter to the editor: Andrew Gelman, Letter to the editors regarding some papers of Dr. Satoshi Kanazawa Journal
of Theoretical Biology 245, no. 3 (April 7, 2007): 597-599
For a two examples of this use of PayScale salary data, see: Philip R. P. Coelho and Tung Lui, The Returns to
College Education, working paper, Department of Economics, Ball State University, August 12, 2012 (available at:
http://econfac.iweb.bsu.edu/research/workingpapers/bsuecwp201202coelho.pdf) and Mark Schneider and Jorge Klor
de Alva, Who Wins? Who Pays? The Economic Returns and Costs of a Bachelors Degree, NEXUS and
American Institutes for Research, May 2011(available at:
http://www.air.org/files/WhoWins_bookmarked_050411.pdf).
15
16
Federal Loans are by far the most common form of undergraduate loans. In the 2011-12 academic year, more than
90% of all undergraduate loans (in terms of total dollars) originated from federal loan programs according to the
College Board. See: Sandy Baum and Kathleen Payea, Trends in Student Aid: 2012, Washington, DC: The
College Board, 2012, p. 17.
The underlying data we used can be obtained from the U.S. Department of Educations Office of Student
Financial Assistance Programs website at: http://www2.ed.gov/offices/OSFAP/defaultmanagement/cdr.html.
17
18
This control variable was added to address the (valid) criticism that, other things equal, students enrolled in STEM
majors tend to take more time to graduate because of the higher rigor associated with their fields of study. The
addition of this control variable allows us to take into account that schools with higher percentages of students
enrolled in STEM majors will likely have lower four-year graduation rates. Thus, these schools should not be
penalized by the model because of the higher percentage of STEM students.
For more information about the Survey of Earned Doctorates, see the National Science Foundations website at:
http://www.nsf.gov/statistics/srvydoctorates/. We should note that the NSFs dataset on earned doctorates included
institutions which do not report data IPEDS or participate in Title IV financial aid programs.
19
20
As of 2012, the National Science Foundation reports that the median time to completion of a doctoral degree from
the time of conferral of the bachelors degree was 9 years. See National Science Foundation, Science and
Engineering Doctorates: 2012, Table 31, available at: http://www.nsf.gov/statistics/sed/2012/data_table.cfm.
21
For private schools, in-state tuition and fees is the same as out-of-state tuition.
22
Berea College in Kentucky also offers their students full scholarship and grant coverage for the cost of tuition, but
according to the IPEDS data, students are required to pay an $876 fee to the school for the year.
23
This statement from The Cooper Unions Board of Trustees explains in further detail its change in tuition policy:
http://www.cooper.edu/about/trustees/board-trustees-statement-future-plans-cooper-union.
16