Malcolm Gladwell on College Rankings: Penn State vs. Yale

Gladwell takes on the annual college rankings published by U.S. News & World Report. His vehicle is an article in The New Yorker: The Order of Things: What College Rankings Really Tell Us. If you were suspicious about these rankings before, Gladwell will easily convince you that the process is poorly posed, and the results are generally irrelevant. But his real message is that they are not worthless. Rather, they are worse than worthless—they are positively harmful.

Before addressing college rankings, Gladwell presents several examples of the folly encountered in trying to arrive at a ranking of a population based on multiple attributes (a comprehensive ranking) when the population of interest varies dramatically in characteristics (a heterogeneous population).

“A ranking can be heterogeneous, in other words, as long as it doesn’t try to be too comprehensive. And it can be comprehensive as long as it doesn’t try to measure things that are heterogeneous. But it is an act of real audacity when a ranking system tries to be comprehensive and heterogeneous—which is the first thing to keep in mind in any consideration of U.S. News & World Report’s annual ‘Best Colleges Guide’.”

The criteria currently being used to rank schools are as follows:

22.5%   Undergraduate academic reputation
20.0%   Graduation and freshman retention rates
20.0%   Faculty resources
15.0%   Student selectivity
10.0%   Financial resources
  7.5%   Graduation rate performance
  5.0%   Alumni giving

The schools are then given a ranking based on a 100 point scale where Harvard ends up with a score of 100. Gladwell points out that this system arrives at a result where Penn State University, a low-cost public school, and Yeshiva University, a private Jewish University with two campuses in Manhattan—one for men and one for women—differ by one point in ranking. What can that possibly mean? How can these two schools be compared at all, let alone be deemed essentially equivalent in some sense.

The major contribution to the score is “academic reputation.” Gladwell cites a little experiment that was once performed. A questionnaire was sent out asking lawyers to rank a list of ten law schools in order of quality. When the results were tallied, Penn State’s law school was deemed to be in the middle of the list. That result was consistent with the type of ranking Penn State receives in the undergraduate rankings. The only problem with this finding is that at the time Penn State did not have a law school. The people completed the survey, not based on any knowledge, but based on a perception of what kind of law school Penn State was likely to have. And what source of information do you have if you wish to obtain such a perception? You go to the “Best Colleges Guide.” In other word: “rankings drive reputation.”

“....when U.S. News asks a university president to perform the impossible task of assessing the relative merits of dozens of institutions he knows nothing about, he relies on the only source of detailed information at his disposal that assesses the relative merits of dozens of institutions he knows nothing about: U.S. News. A school like Penn State, then, can do little to improve its position (47). To go higher than forty-seventh, it needs a better reputation score, and to get a better reputation score it needs to be higher than forty-seventh. The U.S. News ratings are a self-fulfilling prophecy.”

If one looks at the list of criteria employed, most of the ranking credit can be attributed to the wealth of the school. And wealth buys reputation. It is not clear why wealth should have much of anything to do with the quality of undergraduate education. In fact, many would argue that wealth is inconsistent with quality undergraduate education. If it was the case that a wealthy school could afford to charge less for tuition then one could make a cost-effectiveness argument. But, in fact, the wealthier the school the higher the tuition charge is a more appropriate correlation.

Gladwell believes the more important characteristic is what he refers to as “efficacy.”

“One common statistic used to evaluate colleges....is called ‘graduation rate performance’ (7.5%), which compares a school’s actual graduation rate with its predicted graduation rate given the socioeconomic status and the test scores of its incoming freshmen class. It is a measure of the school’s efficacy: it quantifies the impact of a school’s culture and teachers and institutional support mechanisms.”

Note that “student selectivity” at 15% is ranked twice as high as efficacy in ranking schools. Gladwell finishes with a comparison between Yale and Penn State. Yale scores highest on student selectivity. It chooses to educate people who would do well no matter what school they went to. That is their choice to make, but, given that, how do you interpret a graduation rate in the high nineties at Yale? He then assesses Penn State.

“Penn State sees its educational function as serving a wide range of students. That gives it an opportunity to excel at efficacy—and it does so brilliantly. Penn State’s freshmen have an expected graduation rate of seventy-three percent and an actual graduation rate of eighty-five percent, for a score of plus twelve: no other school in the U.S. News top fifty comes close.”

As a citizen who wants to get the best undergraduate education for the education dollar, wouldn’t you rank Penn State quite high? Gladwell would.

“Rankings are not benign. They enshrine very particular ideologies, and, at a time when American higher education is facing a crisis of accessibility and affordability, we have adopted a de-facto standard of college quality that is uninterested in both of these factors. And why? Because a group of magazine analysts....decided twenty years ago to value selectivity over efficacy....and to pretend that they can compare a large, diverse, low-cost land-grant university in rural Pennsylvania with a small, expensive, private Jewish university on two campuses in Manhattan.”

Take that, bad guys!