Anderson: Some Contrarian Thoughts On The U.S. News Faculty Scholarly Impact Rankings

Following up on my previous posts (links below):  Robert Anderson (Pepperdine), Some Preliminary Contrarian Thoughts on the US News Proposal to Rank Based on Scholarly Impact:

There are several lines of criticism. One group worriesthat releasing such a ranking will create incentives for law schools to focus too much on scholarship as opposed to teaching or cutting tuition. I don't have much to say to this group in this post, because that's a broader debate. Another group worries not about placing greater priority on scholarship, but about the idea of quantifying scholarly impact. Although there are real problems with quantifying scholarly impact, that too is a different debate. Scholarly impact or reputation is already quantified, it's just quantified through the use of a somewhat haphazard survey distributed by US News.

In this post, I want to focus on the line of criticism from a third group, those who value scholarship and accept at least in principle the idea of quantifying scholarly impact, but worry about the narrow focus of using law review citations. One of the most common critiques that surfaced on the day of the announcement is that the proposed methodology will undervalue interdisciplinary work. Interdisciplinary work often appears outside of law reviews and is cited by publications other than law reviews.  The concern is that if US News undervalues interdisciplinary work, then because of the sway that US News holds, it may even reduce the incentives for interdisciplinary work.

As an interdisciplinary scholar myself, this critique concerns me. I do almost no "traditional" or "doctrinal" work in my scholarship, although I have done some. However, I think there are several reasons why the concerns might be overblown even for someone like me. In addition, there are actually some advantages to the proposed methodology over one that included interdisciplinary materials more broadly. Let me explain. ...

[I]f I were creating a scholarly impact measure, would I do it the way US News proposes? No. I would use something that would take into account a broader set of disciplinary materials and would control for where citations come from, such as an eigenfactor type approach. But something like that is not likely to ever enter into the US News formula. Thus, we must ask whether the proposed measure of scholarly impact is better than the status quo. Does it improve the information currently contained in the US News rankings? There I believe the answer is yes, because much of what is currently contained in the rankings is immovable peer reputation based on the distant past. I do hope, however, that US News will listen to these critiques and refine the measure to better quantify scholarly impact without distorting the market.

Prior TaxProf Blog posts:

• U.S. News To Publish Law Faculty Scholarly Impact Ranking Based On 2014-2018 Citations (Feb. 13, 2019)
• U.S. News FAQ: Law School Scholarly Impact Rankings (Feb. 14, 2019)
• More Coverage Of The U.S. News Law Faculty Scholarly Impact Rankings (Feb. 15, 2019)