Gregory C. Sisk (St. Thomas-Minnesota), Measuring Law Faculty Scholarly Impact by Citations: Reliable and Valid for Collective Faculty Ranking, 60 Jurimetrics J. ___ (2019) (reviewing Paul J. Heald (Illinois) & Ted M. Sichelman (San Diego), Ranking the Academic Impact of 100 American Law Schools, 60 Jurimetrics J. ___ (2019)):
No single metric of faculty scholarly activity can fully capture every individual contribution. For that reason, evaluating a single professor’s scholarly work requires a nuanced, multifaceted, and individually focused assessment. However, for a contemporary sketch of the collective scholarly impact of a law school faculty, citation measurements in the legal literature are both reliable and valid.
The new Heald-Sichelman study of citations in the HeinOnline database confirms the reliability of the multiyear results of the Leiter-Sisk Scholarly Impact Ranking based on the Westlaw journals database. Despite using a different law journal database, counting citations differently, including pre-tenure faculty, and even adding download statistics into the mix, the Heald-Sichelman ranking correlates powerfully at 0.88 with the most recent Leiter-Sisk ranking. An objective citation measurement is time-sensitive and corresponds to informed awareness of law school faculty developments around the country. A citation-based ranking thus is a valid, if imperfect, proxy for faculty scholarly impact on a national scale.
With appropriate qualifications and necessary adjustments, a citation-based ranking should be considered in any evaluation of the overall quality of a law school faculty. For the U.S. News ranking of American law schools, an up-to-date, citation-based ranking would have considerable merit as an objective forward-directed control to the subjective past-looking academic reputation survey. ...
In an ideal world of infinitely elastic resources, the eternity of time, and omniscient observers, every individual law professor and every law school’s faculty would be fully known, sensitively understood, and thoroughly evaluated based on complete, detailed, and nuanced information. A dean or faculty committee conducting an annual evaluation of an individual faculty member may conduct a more focused individualized assessment. Similarly, a candidate for a faculty position at a particular law school may have the opportunity for a more targeted exploration of the scholarly culture and activity and arrive at a more specified assessment of that school’s progress as a scholarly community.
When comparing large numbers of law faculties across the country, however, a generalized assessment approach has considerable merit and the imperfections of a robust proxy for scholarly accomplishment will wash out at the macro level. That is no reason to be insensitive to flaws in a particular method or to resist adjustments that improve the accuracy and meaning of the results, even if at the margins. And honesty demands acknowledging the limitations of any single approach, allowing the reader to avoid ascribing perfect confidence.
With those qualifications in mind, a citation-based measurement of law faculty scholarly impact has proven to be a reliable method and should be recognized as a valid if imperfect proxy for faculty scholarly achievement. Citation ranking has established itself as a worthwhile factor in comparative assessment of law faculty scholarly impact.
- Paul J. Heald (Illinois) & Ted M. Sichelman (San Diego), Ranking the Academic Impact of 100 American Law Schools, 60 Jurimetrics J. 1 (2019)
- Gregory C. Sisk (St. Thomas), Measuring Law Faculty Scholarly Impact by Citations: Reliable and Valid for Collective Faculty Ranking, 60 Jurimetrics J. 41 (2019)
- Adam S. Chilton (Chicago) & Jonathan S. Masur (Chicago), What Should Law School Rankings Measure and How Should We Measure it: A Comment on Heald & Sichelman's Rankings, 60 Jurimetrics J. 61 (2019)
- Gregory N. Mandel (Temple), Measure for Measure: The Risks of Incorporating Citations Data into U.S. News Rankings, 60 Jurimetrics J. 69 (2019)
- Chris Guthrie (Vanderbilt), Toward A Mission-Based Ranking?, 60 Jurimetrics J. 75 (2019)