The Problems Of Measuring Scholarly Impact (‘Stuff’)

Following up on last week's post, The U.S. News Citation Ranking Is A 'Rigged Metrics Game' That 'Imperils Legal Academia':  LawProfBlawg (Anonymous Professor, Top 100 Law School), The Problems Of Measuring Scholarly Impact (‘Stuff’):

If we’re seeking to adopt some measure to assess scholarly impact, there are serious caveats that need to be addressed before we begin.

Professor Robert Anderson at Pepperdine Law School (place from which I wouldn’t mind a job offer — HINT) [How can we make you an offer (or measure your scholarly impact or teaching effectiveness) if we don't know who you are?] asked me a series of questions on Twitter, all of which are important.

If you don’t know Professor Anderson, you should.  His Twitter feed is a discussion of scholarly impact, and things related to problems of measurement and hierarchies in academia.  I’ve found his tweets cause me to think.  I blame him for this blog post.

His tweet that got me to thinking was this one:  “My pal @lawprofblawg has got some points here, but I think at some point s/he has got to get a little more concrete with an alternative. Is the status quo working? Why would citation-based metrics be worse? Should law schools evaluate scholarship at all? How should hiring work?”

All good questions.  There was some discussion in that thread about the fact that we ought to have some measure of stuff.  In fact, we already do when we hire people to join the faculty, when they go up for tenure, and even (to varying degrees of noneffectiveness) when we review them post-tenure.  They are imperfect, filled with biases, and are often deployed in an arbitrary fashion.  Sometimes they are hardly metrics at all.

Nonetheless, Professor Anderson is correct: I am in favor of having some metrics.  However, the current metrics aren’t working.   My coauthor and I have explained the biases and entry barriers facing certain potential entrants into legal academia.  Eric Segall and Adam Feldman have explained that there is severe concentration in the legal academy, focused on only a handful of schools that produce the bulk of law professors.  While in the academy, barriers block advancement.  And those barriers are reflected, in my opinion, in current citation and scholarly impact measurements seeking to measure stuff.

But if we’re seeking to adopt some measure, whether it is a global standard that could ultimately replace U.S. News or just a standard at one’s own law school, I think there are serious caveats that need to be addressed before we begin to measure stuff.  When I have seen these issues raised before, I have seen them too quickly dismissed.  So let’s try again.

1. Measure what and why? ...
2. Does the metric inherently favor the usual suspects? ...
3. Are we actually measuring things we aren’t seeking to measure? ...
4. Are we comparing apples to oranges? ...
5. If the purpose of the metric is to measure quality, will use of the metric affect quality? ...
6. Any scholarly impact metric must be free from gaming. ...
7. Are we measuring location location location?
8. Will we start writing to the beat of the metric? I feel the same way about this as I feel about schools that contend with standardized tests, only to ignore education in order to teach to the test. If the point of scholarship is to get cited, then perhaps we ought to rethink this whole scholarship endeavor. In my previous blog post, I contend that the purpose of scholarship is to make the world a better place. A friend pointed out that I didn’t define what that means. True. It is something that ought to be discussed before we try to measure it. Huh. I guess that’s my larger point.

TempDean, my frequent coauthor on Above the Law, pointed out that “the only metric that matters is: Will anyone care 20 years after we’re gone? And no one wins.” That’s a pretty harsh indictment. But it is also consistent with my concern about how law professors search for meaning.

• Paul J. Heald (Illinois) & Ted M. Sichelman (San Diego), Ranking the Academic Impact of 100 American Law Schools, 60 Jurimetrics J. 1 (2019)
• Gregory C. Sisk (St. Thomas), Measuring Law Faculty Scholarly Impact by Citations: Reliable and Valid for Collective Faculty Ranking, 60 Jurimetrics J. 41 (2019)
• Adam S. Chilton (Chicago) & Jonathan S. Masur (Chicago), What Should Law School Rankings Measure and How Should We Measure it: A Comment on Heald & Sichelman's Rankings, 60 Jurimetrics J. 61 (2019)
• Gregory N. Mandel (Temple), Measure for Measure: The Risks of Incorporating Citations Data into U.S. News Rankings, 60 Jurimetrics J. 69 (2019)
• Chris Guthrie (Vanderbilt), Toward A Mission-Based Ranking?, 60 Jurimetrics J. 75 (2019)