TaxProf Blog

Paul L. Caron
Dean





Monday, November 11, 2019

Sisk: Citations — 'A Valid, If Imperfect, Proxy For Faculty Scholarly Impact On A National Scale'

Gregory C. Sisk (St. Thomas-Minnesota), Measuring Law Faculty Scholarly Impact by Citations: Reliable and Valid for Collective Faculty Ranking, 60 Jurimetrics J. ___ (2019) (reviewing Paul J. Heald (Illinois) & Ted M. Sichelman (San Diego), Ranking the Academic Impact of 100 American Law Schools, 60 Jurimetrics J. ___ (2019)):

No single metric of faculty scholarly activity can fully capture every individual contribution. For that reason, evaluating a single professor’s scholarly work requires a nuanced, multifaceted, and individually focused assessment. However, for a contemporary sketch of the collective scholarly impact of a law school faculty, citation measurements in the legal literature are both reliable and valid.

The new Heald-Sichelman study of citations in the HeinOnline database confirms the reliability of the multiyear results of the Leiter-Sisk Scholarly Impact Ranking based on the Westlaw journals database. Despite using a different law journal database, counting citations differently, including pre-tenure faculty, and even adding download statistics into the mix, the Heald-Sichelman ranking correlates powerfully at 0.88 with the most recent Leiter-Sisk ranking. An objective citation measurement is time-sensitive and corresponds to informed awareness of law school faculty developments around the country. A citation-based ranking thus is a valid, if imperfect, proxy for faculty scholarly impact on a national scale.

With appropriate qualifications and necessary adjustments, a citation-based ranking should be considered in any evaluation of the overall quality of a law school faculty. For the U.S. News ranking of American law schools, an up-to-date, citation-based ranking would have considerable merit as an objective forward-directed control to the subjective past-looking academic reputation survey. ...

In an ideal world of infinitely elastic resources, the eternity of time, and omniscient observers, every individual law professor and every law school’s faculty would be fully known, sensitively understood, and thoroughly evaluated based on complete, detailed, and nuanced information. A dean or faculty committee conducting an annual evaluation of an individual faculty member may conduct a more focused individualized assessment. Similarly, a candidate for a faculty position at a particular law school may have the opportunity for a more targeted exploration of the scholarly culture and activity and arrive at a more specified assessment of that school’s progress as a scholarly community.

When comparing large numbers of law faculties across the country, however, a generalized assessment approach has considerable merit and the imperfections of a robust proxy for scholarly accomplishment will wash out at the macro level. That is no reason to be insensitive to flaws in a particular method or to resist adjustments that improve the accuracy and meaning of the results, even if at the margins. And honesty demands acknowledging the limitations of any single approach, allowing the reader to avoid ascribing perfect confidence.

With those qualifications in mind, a citation-based measurement of law faculty scholarly impact has proven to be a reliable method and should be recognized as a valid if imperfect proxy for faculty scholarly achievement. Citation ranking has established itself as a worthwhile factor in comparative assessment of law faculty scholarly impact.

https://taxprof.typepad.com/taxprof_blog/2019/11/sisk-citations-a-valid-if-imperfect-proxy-for-faculty-scholarly-impact-on-a-national-scale.html

Law School Rankings, Legal Ed Rankings, Legal Ed Scholarship, Legal Education, Scholarship | Permalink

Comments

Will we ever stop this silliness. It has been shown that citations are a function of the rank of the journal, the rank of the school where the author works, and the rank of the school from which he or she graduated. And placement is largely determined by the institutional authority of the author. More importantly, articles, when cited, rarely influence anything. If you take the time to actually look at 500 citation you would be lucky to find 50 in which the citing author actual pays any attention to the idea of the author whose article is cited. Instead most citations are "see generally" or citations to a factual assertions.

Posted by: JEFFREY HARRISON | Nov 17, 2019 7:41:11 PM

One of the more amusing aspects of such a ranking system is the idea that many people other than the closed system of law school academics actually read or cite the work of law professors. If you are going to measure anything approximating real impact, then try to measure what judges cite in their opinions, what legislators and regulators cite as supporting sources, and what those outside the world of law school rely upon. Then perhaps we could have a clearer picture of what matters. Incestuous citations within a scholarly political community don't really show much.

Posted by: David Barnhizer | Nov 12, 2019 8:01:07 AM

It's all fun & games until someone loses their teaching position over this. How soon before non-faculty "bean-counters" begin using the citation metric to judge an individual faculty member's productivity?

Posted by: Ben A Luddite | Nov 11, 2019 10:46:00 AM