Weapons of Math Destruction: The Higher Education Arms Race

In 1983, the magazine, U.S. News & World Report, took upon itself the chore of ranking the nation’s colleges and universities for excellence.  That effort grew and evolved algorithms for producing a ranking that has become an important factor in evaluating a school’s performance.  The producers of this report bragged in 2008 about the extent of their influence.

“When U.S. News started the college and university rankings 25 years ago, no one imagined that these lists would become what some consider to be the 800-pound gorilla of American higher education, important enough to be the subject of doctoral dissertations, academic papers and conferences, endless debate, and constant media coverage.  What began with little fanfare has spawned imitation college rankings in at least 21 countries, including Canada, China, Britain, Germany, Poland, Russia, Spain, and Taiwan.”

One might conclude from this that the ranking enterprise has been a roaring success.  Perhaps it has been, but only for the finances of those who produce it.  Instead, it serves as an example of what Cathy O’Neil refers to as a weapon of math destruction (WMD).  She includes this ranking exercise as one of numerous WMDs that afflict us in her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.

U.S. News began their ranking activities by running a beauty contest basing its ranking on responses to a survey by university presidents.  Stanford won the first—a school one might expect to win a beauty contest.  But this was deemed to be too simpleminded—and could easily be copied by a competitor.

The U.S. News team had to arrive at a more sophisticated approach.  They knew which schools were expected to selected as the best, so whatever approach they chose had to favor the high-profile schools.  The natural thing to do then would be to assume all good schools would have the characteristics of the presumed best.  Since the actual characteristics of a great school are difficult to quantify, they had to use whatever was quantifiable.

“They couldn’t measure learning, happiness, confidence, friendships, or other aspects of a student’s four-year experience.”

“Instead they picked proxies that seemed to correlate with success.  They looked at SAT scores, student-teacher ratios, and acceptance rates.  They analyzed the percentage of incoming freshmen who made it to sophomore year and the percentage of those who graduated.  They calculated the percentage of living alumni who contributed money to their alma mater, surmising that if they gave a college money there was a good chance they appreciated the education there.  Three quarters of the ranking would be produced by an algorithm—an opinion formalized in code—that incorporated these proxies.  In the other quarter, they would factor in the subjective views of college officials throughout the country.”

A typical characteristic of a WMD is that there is no mechanism for feedback.  O’Neil uses the example of sports teams who can study the results of numerous interactions and formulate new strategies for obtaining statistically better results in those situations.  They can apply those strategies and observe any change in results.  These changes can then be used modify their procedures and try again.  This iteration is the process by which a good algorithm is produced.  But if you are ranking schools somewhat arbitrarily, there is no mechanism for feedback; the results are either credible or not.  If not credible, then the project likely dies.  But U.S. News already knew how to make their results credible.

“U.S. News’s first data-driven ranking came out in 1988, and the results seemed sensible.  However, as the ranking grew into a national standard, a vicious feedback loop materialized.  The trouble was that the rankings were self-reinforcing.  If a college fared badly in U.S. News, its reputation would suffer, and conditions would deteriorate.  Top students would avoid it, as would top professors.  Alumni would howl and cut back on contributions.  The ranking would tumble further.  The ranking, in short, was destiny.”

“Now the vast reputational ecosystem of colleges and universities was over shadowed by a single column of numbers.”

Since this ranking system became a national standard, whether it was worthy of that status or not, schools had no choice but to play by the rules the U.S. News journalists had established.  They had fifteen categories in which they could try to improve their grades—or cheat in order to inflate their scores.

“Some administrators have gone to desperate lengths to drive up their rank.  Baylor University paid the fee for admitted students to retake the SAT, hoping another try would boost their scores—and Baylor’s ranking.  Elite small schools, including Bucknell University in Pennsylvania and California’s Clarement McKenna, sent false data to U.S. News, inflating the SAT scores of their incoming freshmen.  And Iona College, in New York, acknowledged in 2011 that its employees had fudged numbers about nearly everything: test scores, acceptance and graduation rates, freshman retention, student-faculty ratio, and alumni giving”

Most schools invested money instead of cheating.  If they provided improved leisure activities, fancy student centers, better dorms, and better sports teams, they thought these would encourage more students to apply, so they could then turn more students away and improve their acceptance ratio (make it smaller).  If they invested in higher paid professors perhaps they could attract better students with higher SAT scores and up that grade.  If you wished to improve your ranking you had to spend more money—and spend it faster than the other schools were increasing their spending.

Incredibly, the U.S. News team never considered cost of education as a ranking criterion.  They claim they do this ranking for the benefit of the students so they can make better choices concerning schools, but did they think cost was of no interest to students?  Were there no students who were interested in a good education at a modest cost?  Of what benefit is it to have a market for higher education in which cost plays no role?

“By leaving cost out of the formula, it was as if U.S. News had handed college presidents a gilded checkbook.  They had a commandment to maximize performance in fifteen areas, and keeping costs low wasn’t one of them.  In fact, if they raised prices, they’d have more resources for addressing the areas where they were being measured.”

“Tuition has skyrocketed ever since.  Between 1985 and 2013, the cost of higher education rose by more than 500 percent, nearly four times the rate of inflation.”

The existence of the U.S. News rankings has required both schools and students to game the ranking system as best they can.

“All of this activity takes place within a vast ecosystem surrounding the U.S. News rankings, whose model functions as the de facto law of the land.  If the editors rejigger the weightings on the model, paying less attention to SAT scores, for example, or more to graduation rates, the entire ecosystem of education must adapt.  This extends from universities to consultancies, high school guidance departments, and, yes, the students.”

Colleges can no longer think in terms of students as independent entities, rather, they are members of an ensemble of students.  It is the ensemble against which schools are scored.  This makes acceptance decisions much more complicated.  Fortunately there are those who are prepared to provide the algorithms to help the school administrators maximize their scores. 

“As colleges position themselves to move up the U.S. News charts, they manage their student populations almost like an investment portfolio.”

Somewhat less directly, high school students are also driven by the U.S. News algorithm.

“Each college’s admissions model is derived, at least in part, from the U.S. News model, and each one is a mini-WMD.  These models lead students and their parents to run in frantic circles and spend obscene amounts of money.  And they’re opaque.  This leaves most of the participants (or victims) in the dark.  But it creates a big business for consultants….who manage to learn their secrets, either by cultivating sources at the universities or by reverse engineering their algorithms.”

The people who do not have a lot of money to spend are, as always, the victims.  They lose because the cost of education is being artificially driven up, and because the cost of getting admitted to a university is also being driven up.

“The victims, of course, are the vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spend on courses and consultants.  They miss out on precious insider knowledge.  The result is an education system that favors the privileged.  It tilts against needy students, locking out the great majority of them—and pushing them down a path toward poverty.  It deepens the social divide.”