Paul L. Caron

Thursday, October 10, 2019

The U.S. News Citation Ranking Is A 'Rigged Metrics Game' That 'Imperils Legal Academia'

Following up on my previous posts (links below):  LawProfBlawg (Anonymous Professor, Top 100 Law School), The Taylorism Of Legal Academia: Another Rigged Metrics Game, and It Imperils Legal Academia:

U.S. News Law (2019)The legal academy is on a precipice.  As people seek to figure out exactly the mystery of what academics do, they want to come up with more metrics to determine which academics are good, and which academics are not.  It’s like if Santa Claus were a management consultant with a basic understanding of stats.

To some degree, academia has endured measurement in terms of student evaluations.  The good professors are the ones with good evaluations, and the bad ones are the ones who lack them.  It’s only recently that people have discovered that which many have known for decades: Student evaluations are rigged, and you can pretty much guess the direction of the biases.  Despite that, we still use them, apparently because measuring something poorly is way better than not measuring it at all.

Now, professors and university administrators are becoming more focused on measuring the impact of scholars.  The term “scholarly impact” describes the complicated system of measuring whose work makes a difference, at least according to whatever metrics are used.  In the old days, it was SSRN.  Now, with U.S. News teaming with Heinonline, a new king of the metric is in town.  And you’d be kidding yourself if you think it won’t be used to target some untenured professors and chide some tenured professors who think scholarly impact might be measured in a more meaningful way (or not at all).  My coauthor and I have said our peace about these measures of “quality” here.

But universities are starting to measure faculty productivity.  The alleged goal is quality, but I’m thinking the real goal is to produce “more stuff.” ...

Making the world a better place might mean spending more time working with students, or writing something not counted in the “stuff” measure that targets the general population.  In short, I fear that instead of focusing on making the world a better place, measuring “stuff” will lead to a more conformist academy (if that’s possible) and one whose direction has been handed over to university administrators and external data miners.

In other words, the notion that the same systems that were deployed to assure that I can buy a cheap TV that breaks more frequently too will somehow lead to improved quality in higher education is a pipe dream.  And I, for one, won’t be playing this rigged game.

Prior TaxProf Blog coverage:

Legal Ed News, Legal Ed Rankings, Legal Education | Permalink


"The same systems that were deployed to assure that [you] can buy a cheap TV" put much of humanity's knowledge - and an audience bigger than any Emperor's - at the fingertips of BILLIONS. And the entire POINT of a lawyer is to persuade and improve an imperfect world. So, how would *you* hold schools' feet to this fire?

Posted by: Anand Desai | Oct 10, 2019 5:30:32 AM