In February 2019, U.S. News announced it would publish a law faculty scholarly impact ranking in 2019 based on 5-year citation data from HeinOnline (FAQ; Updated FAQ; Additional Guidance). In November 2020, U.S. News announced it would be publishing the ranking in 2021. In June 2021, U.S. News abandoned its effort to rank law faculty scholarly impact:
Bloomberg Law op-ed: Rankings Shift Could Force Big Changes at U.S. Law Schools, by Joshua Fischman (Virginia) & Michael A. Livermore (Virginia):
The U.S. News & World Report rankings are a powerful force in the world of law schools. Deans’ careers can rise or fall on their schools’ rankings, which affect everything from student recruitment to alumni giving to faculty retention. So when U.S. News announced in 2019 that it was considering creating a new ranking for law schools, heads turned across the legal academy.
Bloomberg Law received an email Aug. 19 from U.S. News stating that in June 2021 it decided that it would not proceed with its previously proposed law school scholarly impact ranking [more here]. However, we feel it is important to discuss concerns legal academics have about such proposals.
The crux of the U.S. News proposal was to develop a new measure of a law school’s prestige based on the “impact” of the scholarship produced by its faculty. This impact score would be calculated by counting the number of times a professor’s work was cited by other professors, and perhaps by courts.
Skeptics immediately raised objections. The most general challenge is that judgments of scholarly merit are inevitably subjective and cannot be quantified. Others expressed concern about biases against women, scholars of color, interdisciplinary scholars, and those in less-cited research areas.
But the citation-based rankings have supporters as well. They argue that the new approach will bring some needed objectivity to a system that is biased in favor of the old-school powerhouses and leaves little room for entrepreneurial upstarts to improve their standing.
As with many debates in academia, this might sound like a tempest in a teapot. But if U.S. News changes its ranking system, law schools will be pressured to alter how they recruit and promote faculty.
In particular, law schools will likely focus on professors with the most citations, instead of interdisciplinary credentials, peer-reviewed publications, or diversity. Ultimately, this affects who trains the next generation of lawyers and which ideas are circulated to courts and other legal decision-makers.
In a recent study of the law school lateral hiring market, we show that a focus on citations would result in dramatic changes in law school hiring. We find that the professors recruited into the top law schools are not necessarily the ones with the most citations. The citation counts of law professors who move to the most elite schools in the country—places like Harvard and Yale—are barely distinguishable from the rest of the field.
Joshua Fischman (Virginia) & Michael A. Livermore (Virginia), Empirically Validating Citation Metrics for Legal Scholars: A Market Approach:
Citation counts are a common quantitative metric used by researchers and analysts to assess scholarly output. When U.S. News & World Report announced in 2019 that it was developing a citation-based ranking for law schools, it brought new attention to debates about citations counts in legal scholarship. Supporters of citation metrics argue that they are superior to alternative measures of scholarly reputation, such as surveys. Critics are skeptical that citations serve as a meaningful proxy for scholarly quality and raise concerns that citation metrics could distort the incentives of law professors and faculties. We examine the validity of citation metrics by examining how well they correspond to a “market valuation” of legal scholars in lateral hiring.
We consider two outcomes: whether professors make lateral moves and the rank of the institution where they are hired. Using citation counts derived from the HeinOnline database, we find that citation metrics have a weak association with lateral outcomes. Metrics that mitigate the effect of the highly skewed distribution of citations, such as log citations and the h-index, perform slightly better. Article placements are stronger predictors of lateral outcomes. In particular, articles in top law reviews, top peer-reviewed journals, and online law reviews are all associated with moving to higher-ranked faculties, even after controlling for citations. This implies that citation rankings undervalue these kinds of publications. The divergence between citation counts and professors’ market valuation suggest that citation rankings could significantly distort publication and hiring in the legal academy.
Prior TaxProf Blog coverage:
- U.S. News To Publish Law Faculty Scholarly Impact Ranking Based On 2014-2018 Citations (Feb. 13, 2019)
- U.S. News FAQ: Law School Scholarly Impact Rankings (Feb. 14, 2019)
- More Coverage Of The U.S. News Law Faculty Scholarly Impact Rankings (Feb. 15, 2019)
- Robert Anderson (Pepperdine), Some Contrarian Thoughts On The U.S. News Faculty Scholarly Impact Rankings (Feb. 18, 2019)
- Law Prof Commentary On The U.S. News Faculty Scholarly Impact Rankings (Feb. 19, 2019)
- U.S. News Updates FAQ On Law School Scholarly Impact Rankings To Address Inclusion Of Non-Doctrinal Faculty (Feb. 21, 2019)
- Joe Lawprofblawg (Anonymous Professor, Top 100 Law School) & Darren Bush (Houston), A Hilarious (In The Footnotes) Yet Serious (In The Text) Discussion Of Law Reviews And Law Professors (Feb. 26, 2019)
- U.S. News Updates FAQ On Law School Scholarly Impact Rankings To Address Inclusion Of Non-Doctrinal Faculty (Feb. 27, 2019)
- Derek Muller (Pepperdine), Gaming The New U.S. News Citation Rankings (Mar. 6, 2019)
- Jeff Sovern (St. John's), How The U.S. News Scholarly Impact Rankings Could Hurt Niche Subjects (March 11, 2019)
- Ted Sichelman (San Diego), A Defense And Explanation Of The U.S. News 'Citation' Ranking (March 20, 2019)
- U.S. News Offers More Guidance, Meeting With Law School Deans On New Scholarly Impact Rankings (May 2, 2019)
- The U.S. News Citation Ranking Is A 'Rigged Metrics Game' That 'Imperils Legal Academia' (Oct. 10, 2019)
- Heald & Sichelman: The Top 100 Law School Faculties In Citations (Hein) And Impact (SSRN Downloads) (Nov. 11, 2019)
- Sisk: Citations — 'A Valid, If Imperfect, Proxy For Faculty Scholarly Impact On A National Scale' (Nov. 11, 2019)
- Bob Morse Discusses The U.S. News Law School Rankings Today At Texas A&M Virtual Conference (June 5, 2020)
- U.S. News To Publish Law Faculty Scholarly Impact Ranking In 2021 (Nov. 9, 2020)
- 2022 U.S. News Law School Rankings (Mar. 9, 2021)
- The New U.S. News Law School Rankings Methodology: Implications And Predictions (Mar. 22, 2021)
- U.S. News Denies Report That It Will Replace The 40% Reputation Component With HeinOnline Citations In Next Year's Law School Rankings (Mar. 22, 2021)