Paul L. Caron
Dean


Friday, February 14, 2020

Mandel: The Risks Of Incorporating Citations Data Into The U.S. News Law School Rankings

Gregory N. Mandel (Temple), Measure for Measure: The Risks of Incorporating Citations Data into U.S. News Rankings, 60 Jurimetrics J. 69 (2019):

This short essay responds to Paul Heald and Ted Sichelman’s article, Ranking the Academic Impact of 100 American Law Schools, 60 Jurimetrics J. 1 (2019). Heald and Sichelman's work provides a rigorous analysis of law school faculties’ citation and download statistics. Their recommendation to incorporate these statistics into U.S. News & World Report’s annual law school rankings, however, appears misguided.

Heald and Sichelman do not fully take into account the concerning gaming behavior and problematic incentives related to faculty hiring that such incorporation would likely produce over time.

https://taxprof.typepad.com/taxprof_blog/2020/02/mandel-the-risks-of-incorporating-citations-data-into-the-us-news-law-school-rankings.html

Legal Ed Scholarship, Legal Education | Permalink

Comments

"Heald and Sichelman do not fully take into account the concerning gaming behavior and problematic incentives related to faculty hiring that such incorporation would likely produce over time."

Uh, virtually every variable in USNWR's methodology falls under the auspices of Campbell's and Goodhart's Laws. In a nutshell, any time a metric becomes a target, it ceases to be a good metric because people will naturally seek to game/manipulate/lie about it. In that lensing the citations data isn't demonstrably different than the acceptance rate, LSAT scores, or starting salaries.

Posted by: Unemployed Northeastern | Feb 14, 2020 11:32:44 AM

Mandel, along with Heald, Sichelman, and all the distinguished academics who have commented on the Heald & Sichelman article, misses a glaring defect in the rankings: they include Oregon State, a law school that simply does not exist. The University of Oregon, which does have a law school, is listed separately so this is not simply an error in giving the school’s proper title.

I suspect that “Oregon State” is an alias for “Ohio State,” a school that seems oddly missing from the rankings. But what type of rigor do rankings like this have when they include a school that doesn’t exist? And when specialists in the field (some of whom have created their own rankings) don’t notice that fact?

I’ll go further: I suspect that no one noticed the sudden emergence of Oregon State’s faculty as the 35th most cited faculty (according to Hein) because people read these rankings primarily to see where their own school ranks. That’s what I did: I was curious where my own faculty ranked and, when Ohio State was absent, looked more closely. It was only then that I noticed a law school that doesn’t exist.

But if that is the purpose of these rankings, to give faculty comfort or angst about how their law school ranks, are these rankings worth producing? They will create an incentive for schools to invest still more of their resources in faculty scholarship—raising tuition, reducing discounts, and diverting money from efforts that could improve graduates’ service to clients. And for what greater purpose? To reassure faculty that their school ranks higher than one that does not exist?

Posted by: Deborah Merritt | Feb 15, 2020 9:55:59 AM

Post a comment