How Statistics Turns Best into Worst
Jon Ronson, in the US edition of The Guardian, wondered what such a terrible school would look like:
[blockquote] So what’s it like, this worst college? What criteria put it there? The compiler, Ben Miller, a former senior policy advisor in the Department of Education, explained in the Washington Monthly that they were looking for colleges that “charge students large amounts of money to receive an education so terrible that most drop out before graduation.” Actually, Shimer topped a list that was adjusted for race and income. So a truer description is that it’s the worst college in America that doesn’t have many students of color or low-income students.[/blockquote]
Ronson points out later in the article that the adjustment for race may not be accurate:
[blockquote]It would be wrong to imagine Shimer as a wholly white community, though. I see lots of students of color during my day here.[/blockquote]
And the adjustment for low-income students may not be accurate either, since Shimer has quite a few of them. One of the students Ronson interviewed is currently living out of her car.
So it’s not a surprise that the “worst-college” ranking provoked outspoken defense from those who value genuine liberal education:
[blockquote] The thing is, in the hours after the rankings were published in October, something unexpected happened. Whilst almost nobody stepped forward in defense of the other colleges on the list, Shimer fans began vociferously attacking its inclusion. One graduate wrote that it’s “a totally unique snowflake, and comparing it to other schools is next to impossible.” He added that he wouldn’t take his time there back “for all the money in the world. Looking back, I’d go into even more debt to make sure it happened.” Another called it “an eccentric little school that appeals to few; but for those few, it evidently serves as a valuable and stimulating harbor.”[/blockquote]
Ronson implies that Ben Miller himself seems to think Shimer doesn’t really belong on the list:
[blockquote] Even Ben Miller, the list’s compiler, seems remorseful that Shimer topped his list. “I think their story is at least partly due to small sample sizes,” he emails. Then he reiterates this twice more in other emails to me.[/blockquote]
Partly due to small sample size—Yes. But also due to a fallacy of statistical reasoning called “overgeneralization.” That happens when what is being measured does not fit the whole class of things being examined. In this case, the measurements used for colleges are drawn from the average American college with 1500 to 50,000 students that aims at providing a mixture of general education, specialization, career training, sports, entertainment, and social life. Is it likely that a small college entirely devoted to liberal education is going to score well on that test?
This is just one of the problems with the college rankings that have become nearly ubiquitous in popular culture. They use the scientific “feel” of numerical precision to create the illusion of accuracy. Often the results are nonsensical lists that try to score individually unique schools as if they are all running in the same horse race. And education, by the way, has little if anything in common with a horse race.
The particular irony of this extreme ranking—and, quite frankly, of the whole enterprise of college ranking—is that Shimer, St. John’s, and schools like them, with curricula based on penetrating discussion of the great texts and artworks of humankind, are quite probably among the very best colleges anywhere in the world.