U.S. News FAQ: Law School Scholarly Impact Rankings

Following up on yesterday's post, U.S. News To Publish Law Faculty Scholarly Impact Ranking Based On 2014-2018 Citations:

U.S. News FAQ: Law School Scholarly Impact Rankings:

What are the Law School Scholarly Impact Rankings?
U.S. News is considering expanding its law school rankings to evaluate each law school’s scholarly impact. This will measure a school’s faculty’s impact in terms of scholarly productivity and impact by using citations, publications and other accepted bibliometric measures. U.S. News is working with William S. Hein & Co., Inc. to complete this effort.

What is U.S. News asking of law schools, and what role will U.S. News and Hein each play in the analysis?
U.S. News is asking each law school to provide U.S. News with the names of its fall 2018 full-time tenured and tenure-track faculty. Each law school’s participation in the law school faculty survey is voluntary.

U.S. News will then share each school’s faculty name and other faculty profile data with Hein, who will use this information to link each individual law school’s faculty names to citations and publications that were published in the previous 5 years and which are available in HeinOnline. Using this data, Hein will compile faculty scholarly impact indicators for each law school. This will include such measures as mean citations per faculty member, median citations per faculty member, and total number of publications. Those measures will then be provided to U.S. News for use in creating a comprehensive scholarly impact ranking.

Will scholarly impact indicators factor into the next law schools rankings from U.S. News?
Scholarly impact will not be a factor in the overall law schools rankings published by U.S. News in the late winter or early spring 2019. Rather, U.S. News is considering publishing a separate law school scholarly impact ranking during the 2019 calendar year.

Will U.S News publish a methodology on its new scholarly impact ranking?
In collaboration with Hein, U.S. News would produce a detailed methodology at the time of publication on how the rankings were calculated, how the faculty scholarly impact indicators were calculated and weighted, a description of the legal periodicals and publications used in the analysis, and other pertinent details about how the rankings were developed.

Who is William S. Hein & Co. Inc. and what is HeinOnline?
William S. Hein & Co., Inc. has been serving the library community for more than 95 years as a legal publisher, periodical subscription agent, and the world’s largest distributor of legal periodicals both in print and online. HeinOnline is a premier online database containing more than 165 million pages and 200,000 titles of historical and government documents in a fully searchable, image-based format. HeinOnline bridges an important research gap by providing comprehensive coverage from inception of more than 2,600 law-related periodicals. In addition to its vast collection of academic journals, HeinOnline contains the entire Congressional Record, Federal Register, and Code of Federal Regulations, complete coverage of the U.S. Reports back to 1754, and entire databases dedicated to treaties, constitutions, case law, world trials, classic treatises, international trade, foreign relations, U.S. Presidents, and much more.

If you have further questions for U.S. News about the rankings and faculty data collection:
Please email [email protected].
If you have further questions about Hein or the scholarly impact data:
Please email Shane Marmion at [email protected].

Inside Higher Ed, Do Law Schools Need a Second Ranking From 'U.S. News'?:

Kyle McEntee, executive director of Law School Transparency, a group that pushes for more data on law schools to be public and for more of a focus on the costs of attending law school, said that the new U.S. News ranking being planned is not what law schools need.

"Just because something can be ranked doesn't mean that it should be ranked," he said. Law schools need to focus on things other than faculty research, he added. "If schools respond to these incentives, it will severely limit their attention to what matters in legal education today," McEntee said. What should they focus on? "Access, affordability and curricular innovation," he said.

Brian Z. Tamanaha, a professor of law at Washington University in St. Louis and author of Failing Law Schools (University of Chicago Press), agreed. "We know that law schools will do whatever they can to improve their ranking," he said via email. "This new ranking will have the same consequence, prompting law schools to maximize this specific set of narrow metrics. Law schools already prioritize research, and allocate substantial resources to its production, too much as it is in my opinion, and this will likely make it worse."

Robert Morse, chief data strategist at U.S. News, defended the idea of starting a new ranking. He sent this statement: "U.S. News is going to emphasize scholarly impact by using different indicators that measure citations, not volume. U.S. News has been looking for ways to measure faculty quality at law schools, and analyzing scholarly impact in legal academia is a widely accepted measure. Prospective students are looking for schools with the highest quality law school faculty who are making an impact in legal academia and the law. This analysis would provide students with important information to make such comparisons."

Derek Muller (Pepperdine), Will Goodhart's Law Come to USNWR's Hein-Based Citation Metrics?:

Economist Charles Goodhart has an old saying attributed to him: “When a measure becomes a target, it ceases to be a good measure.” In the office setting, or in government, or anywhere, really, if you identify a measure as a target, people begin to behave in a way to maximize the value of that measure, and the measure loses its value—because it’s no longer accurately measuring what we hoped it would measure. ...

In the future, it may (this is an open question, so let’s not freak out too quickly yet!) be used in the overall rankings. On the whole, it may well be a good change (but I imagine many will disagree even on this hesitant claim!). Rather than the subjective, perhaps too sticky, assessments of faculty voting, this provides an objective (yes, even if imperfect!) metric of faculty productivity and influence. In the event that it used in the overall score in the future, it is one more small component of an overall ranking, it strikes me as appropriate. ...

I’m more concerned with this Goodhart principle: once we start measuring it, what impact might it have on faculty behavior? A few come to mind.

The temptation to recast senior would-be emeritus faculty with sufficient scholarly output as still full-time faculty members, among other ways of trying to recategorize faculty. Perhaps these are marginal changes that all schools will simultaneously engage in, and the results will wash out.

There is a risk of creating inflated or bogus citations. This is a much more realistic risk. Self-citations are ways that scholars might try to overstate their own impact. Men tend to self-cite at disproportionately higher rates than women. Journals that have excessive self-citations are sometimes punished in impact factor rankings. Pressure to place pieces in home institution journals may increase. ...

It will be much easier for schools to “game” the median citations—finding the single faculty member in the middle, and trying to climb the ladder that way. Median is probably a better metric, in my view (because mean can disproportionately be exaggerated by an outlying faculty member), but it also more likely to be susceptible to Goodhart’s Law. Mean citations would be a tougher thing to move as dramatically or with such precision.

Scott Fruehwald (Legal Skills Prof Blog), A Letter to Robert Morse at U.S. News:

Robert Morse. Your proposal to only include tenured and tenure-track professors in your scholarly rankings is an insult to the many full-time, non-tenure track professors who are vital parts of law school faculties. Legal writing professors and clinicians are an essential part of all law faculties. If you bothered to do a little research (ssrn and google), you would discover that these full-time faculty members are also scholars who have made important advancements in legal scholarship. In particular, they publish more articles on legal education than any other group. They also publish articles on constitutional law, rhetoric, procedure, contracts, and all other legal fields. I hope you will correct this mistake.

P.S.  Mr. Morse: This former legal writing prof's SSRN page lists over 17,000 downloads.   Many others in the clinical and legal writings fields also have thousands of downloads on SSRN.

Update:

• More Coverage Of The U.S. News Law Faculty Scholarly Impact Rankings (Feb. 15, 2019)
• Robert Anderson (Pepperdine), Some Contrarian Thoughts On The U.S. News Faculty Scholarly Impact Rankings (Feb. 18, 2019)
• Law Prof Commentary On The U.S. News Faculty Scholarly Impact Rankings (Feb. 19, 2019)
• U.S. News Updates FAQ On Law School Scholarly Impact Rankings To Address Inclusion Of Non-Doctrinal Faculty (Feb. 21, 2019)