The U.S. News Citation Ranking Is A 'Rigged Metrics Game' That 'Imperils Legal Academia'

Following up on my previous posts (links below):  LawProfBlawg (Anonymous Professor, Top 100 Law School), The Taylorism Of Legal Academia: Another Rigged Metrics Game, and It Imperils Legal Academia:

The legal academy is on a precipice.  As people seek to figure out exactly the mystery of what academics do, they want to come up with more metrics to determine which academics are good, and which academics are not.  It’s like if Santa Claus were a management consultant with a basic understanding of stats.

To some degree, academia has endured measurement in terms of student evaluations.  The good professors are the ones with good evaluations, and the bad ones are the ones who lack them.  It’s only recently that people have discovered that which many have known for decades: Student evaluations are rigged, and you can pretty much guess the direction of the biases.  Despite that, we still use them, apparently because measuring something poorly is way better than not measuring it at all.

Now, professors and university administrators are becoming more focused on measuring the impact of scholars.  The term “scholarly impact” describes the complicated system of measuring whose work makes a difference, at least according to whatever metrics are used.  In the old days, it was SSRN.  Now, with U.S. News teaming with Heinonline, a new king of the metric is in town.  And you’d be kidding yourself if you think it won’t be used to target some untenured professors and chide some tenured professors who think scholarly impact might be measured in a more meaningful way (or not at all).  My coauthor and I have said our peace about these measures of “quality” here.

But universities are starting to measure faculty productivity.  The alleged goal is quality, but I’m thinking the real goal is to produce “more stuff.” ...

Making the world a better place might mean spending more time working with students, or writing something not counted in the “stuff” measure that targets the general population.  In short, I fear that instead of focusing on making the world a better place, measuring “stuff” will lead to a more conformist academy (if that’s possible) and one whose direction has been handed over to university administrators and external data miners.

In other words, the notion that the same systems that were deployed to assure that I can buy a cheap TV that breaks more frequently too will somehow lead to improved quality in higher education is a pipe dream.  And I, for one, won’t be playing this rigged game.

Prior TaxProf Blog coverage:

• U.S. News To Publish Law Faculty Scholarly Impact Ranking Based On 2014-2018 Citations (Feb. 13, 2019)
• U.S. News FAQ: Law School Scholarly Impact Rankings (Feb. 14, 2019)
• More Coverage Of The U.S. News Law Faculty Scholarly Impact Rankings (Feb. 15, 2019)
• Robert Anderson (Pepperdine), Some Contrarian Thoughts On The U.S. News Faculty Scholarly Impact Rankings (Feb. 18, 2019)
• Law Prof Commentary On The U.S. News Faculty Scholarly Impact Rankings (Feb. 19, 2019)
• U.S. News Updates FAQ On Law School Scholarly Impact Rankings To Address Inclusion Of Non-Doctrinal Faculty (Feb. 21, 2019)
• Joe Lawprofblawg (Anonymous Professor, Top 100 Law School) & Darren Bush (Houston), A Hilarious (In The Footnotes) Yet Serious (In The Text) Discussion Of Law Reviews And Law Professors (Feb. 26, 2019)
• U.S. News Updates FAQ On Law School Scholarly Impact Rankings To Address Inclusion Of Non-Doctrinal Faculty (Feb. 27, 2019)
• Derek Muller (Pepperdine), Gaming The New U.S. News Citation Rankings (Mar. 6, 2019)
• Jeff Sovern (St. John's), How The U.S. News Scholarly Impact Rankings Could Hurt Niche Subjects (March 11, 2019)
• Ted Sichelman (San Diego), A Defense And Explanation Of The U.S. News 'Citation' Ranking (March 20, 2019)
• U.S. News Offers More Guidance, Meeting With Law School Deans On New Scholarly Impact Rankings (May 2, 2019)