How Statistics Turns Best into Worst

On one of its several different college-ranking lists, Washington Monthly recently designated Shimer College, a small great books college in Chicago, the worst college in America.

Jon Ronson, in the US edition of The Guardian, wondered what such a terrible school would look like:

 So what’s it like, this worst college? What criteria put it there? The compiler, Ben Miller, a former senior policy advisor in the Department of Education, explained in the Washington Monthly that they were looking for colleges that “charge students large amounts of money to receive an education so terrible that most drop out before graduation.” Actually, Shimer topped a list that was adjusted for race and income. So a truer description is that it’s the worst college in America that doesn’t have many students of color or low-income students.

Ronson points out later in the article that the adjustment for race may not be accurate:

It would be wrong to imagine Shimer as a wholly white community, though. I see lots of students of color during my day here.

And the adjustment for low-income students may not be accurate either, since Shimer has quite a few of them. One of the students Ronson interviewed is currently living out of her car.

So it’s not a surprise that the “worst-college” ranking provoked outspoken defense from those who value genuine liberal education:

The thing is, in the hours after the rankings were published in October, something unexpected happened. Whilst almost nobody stepped forward in defense of the other colleges on the list, Shimer fans began vociferously attacking its inclusion. One graduate wrote that it’s “a totally unique snowflake, and comparing it to other schools is next to impossible.” He added that he wouldn’t take his time there back “for all the money in the world. Looking back, I’d go into even more debt to make sure it happened.” Another called it “an eccentric little school that appeals to few; but for those few, it evidently serves as a valuable and stimulating harbor.”

Ronson implies that Ben Miller himself seems to think Shimer doesn’t really belong on the list:

Even Ben Miller, the list’s compiler, seems remorseful that Shimer topped his list. “I think their story is at least partly due to small sample sizes,” he emails. Then he reiterates this twice more in other emails to me.

Partly due to small sample size—Yes. But also due to a fallacy of statistical reasoning called “overgeneralization.” That happens when what is being measured does not fit the whole class of things being examined. In this case, the measurements used for colleges are drawn from the average American college with 1500 to 50,000 students that aims at providing a mixture of general education, specialization, career training, sports, entertainment, and social life. Is it likely that a small college entirely devoted to liberal education is going to score well on that test?

This is just one of the problems with the college rankings that have become nearly ubiquitous in popular culture. They use the scientific “feel” of numerical precision to create the illusion of accuracy. Often the results are nonsensical lists that try to score individually unique schools as if they are all running in the same horse race. And education, by the way, has little if anything in common with a horse race.

The particular irony of this extreme ranking—and, quite frankly, of the whole enterprise of college ranking—is that Shimer, St. John’s, and schools like them, with curricula based on penetrating discussion of the great texts and artworks of humankind, are quite probably among the very best colleges anywhere in the world.

6 Comments

  1. Barbara Stone   •  

    Thanks! – We live in interesting times!

  2. k.C. Victor   •  

    Thank you for this post. Why anyone feels compelled to rank anything is a mystery to me. It has gotten worse in the last few decades.

  3. Daniel Shiner   •  

    Thank you, Chris. Every few years, I re-read my well-worn copy of How to Lie With Statistics, the 1954 classic by Darrell Huff. And as Mark Twain pointed out, “There is something fascinating about (statistics). One gets such wholesome returns of conjecture out of such a trifling investment of fact.”

  4. Warner W Johnston   •  

    Shimer 1973, I entered in 64, but my Uncle Sam made me an offer I couldn’t refuse.

    I’m an engineer, IEEE NY Section Chair 2009, I wouldn’t trade my Shimer experience for MIT or CalTech with whipped cream and a cherry on top.

  5. Douglas Peterson   •  

    Shimer ’84

    I am proud to be a Shimer grad and have never regretted a dime of what I spent or believe I could have received a better education anywhere.

    Teaching Chair and Professor of Computer Science

  6. Peter Piper   •  

    Thank you for the blog! I am able to keep up and more often than not surpass the big boys in my field…I wouldn’t trade my Shimer education for anything!!!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>