Paul L. Caron
Dean


Wednesday, February 19, 2020

Merritt: Ranking Academic Impact

Deborah J. Merritt (Ohio State), Ranking Academic Impact:

OOPaul Heald and Ted Sichelman have published a new ranking of the top U.S. law schools by academic impact. Five distinguished scholars comment on their ranking in the same issue of Jurimetrics Journal in which the ranking appears. But neither the authors of this ranking nor their distinguished commentators notice a singular result: The Heald/Sichelman rankings include a law school that does not exist.

According to Heald and Sichelman, Oregon State ranks 53d among U.S. law schools for its SSRN downloads; 35th for its citations in the Hein database; and 46th in a combined metric. Oregon State, however, does not have a law school. The University of Oregon has a law school, but it appears separately in the Heald/Sichelman rankings. So Heald and Sichelman have not simply fumbled the name of Oregon’s only public law school.

Instead, it appears that my own law school (Ohio State) has been renamed Oregon State. I can’t be sure without seeing Heald and Sichelman’s underlying data; even the “open” database posted in Dropbox refers to the nonexistent Oregon State. But Ohio State, currently tied for 34th in the US News survey, seems conspicuously absent from the Heald/Sichelman ranking.

I’m sure that my deans will contact Heald and Sichelman to request a correction–assuming that Oregon State actually is Ohio State. Oregon State Law School’s administrators probably will not complain. They can’t celebrate either, of course, because they don’t exist. But apart from that correction, let’s ruminate on this error. What does it have to say about rankings? ...

Heald and Sichelman’s error is troubling, but I am equally concerned about the failure of any of their readers to spot the mistake. How could five commentators, as well as numerous other readers and workshop participants, blithely skim over the nonexistent Oregon State? Even if they weren’t familiar with Oregon’s law schools, weren’t they surprised to see Oregon State ranked 35th for Hein citations? The three schools in that state currently appear as 83d, 104th, and in the unranked fourth tier of the US News survey. Shouldn’t someone have noticed the surprising strength of Oregon State’s faculty?

I suspect that no one noticed the presence of Oregon State because most faculty read rankings primarily to see where their own school ranks. That’s what I did: I was curious where my own faculty ranked and, when Ohio State was absent, I looked more closely. It was only then that I noticed a law school that doesn’t exist.

But if that is the use of these academic impact rankings, to give faculty comfort or angst about where their law school ranks, are these rankings worth producing? 

A mistake like this obviously raises doubts about the reliability of the Heald/Sichelman ranking. If an error of this magnitude exists, what other errors lurk in the data? Even if you like the Heald/Sichelman method, how do you know it was carried out faithfully?

 I think these rankings will harm the legal profession and its clients. Legal educators are key stewards of the legal profession. We are the profession’s primary gatekeepers: Few people become lawyers without first earning our diplomas. We are also responsible for giving students the foundation they need to serve clients competently and ethically.

Rankings of academic impact almost certainly will incentivize schools to invest still more of their resources in faculty scholarship—which, in turn, will raise tuition, reduce student discounts, and/or divert money from preparing students for their essential professional roles.

Scholarship is part of our commitment to the profession, clients, and society, but only one part. Over the last 20 years, I have seen law schools shift increasing resources to scholarship, while reducing teaching loads and raising tuition rapaciously. We produced excellent scholarship before 2000–scholarship that created fields like critical race theory, law and economics, feminist theory, and social science analyses of law-related issues. There is much still to explore, but why does today’s scholarship demand so many more resources? And will rankings further accelerate that trend?

https://taxprof.typepad.com/taxprof_blog/2020/02/merritt-ranking-academic-impact.html

Legal Ed Rankings, Legal Ed Scholarship, Legal Education | Permalink

Comments

Congrats to Oregon State. Maybe next year they'll jump past Princeton Law School.

Posted by: Scott Fruehwald | Feb 19, 2020 12:23:16 PM

Years ago, some professors conducted a "survey" that asked judges and law professors to rank various law schools, including "Penn State Law." The joke being that at that time, there was no such law school. Nonetheless, numerous highly learned law professors and judges confidently ranked Penn State Law as a mid-tier law school. I guess that must have worked, for several years later, the legislators in Pennsylvania realized that since this non-existent law school already had a good reputation, they may as well make it exist in reality! Hence the creation of PSU Law.

Posted by: Andy Patterson | Feb 20, 2020 7:49:23 AM