U.S. News Ranking Metrics Stifle Law Libraries, Tie Hands of Law Schools

TaxProf Blog op-ed:  U.S. News Ranking Metrics Stifle Law Libraries, Tie Hands of Law Schools, by Amanda Runyon (Pennsylvania), Leslie A. Street (William & Mary), & Amanda Watson (Houston):

USNWR has released the 2022 Best of Law School rankings, including a new set of questions and methodology for law libraries. As library directors at schools that perform objectively well in these new calculations, we feel obligated to voice our concerns. We believe the unintended consequences and potential outcomes of these metrics are highly problematic for law libraries and the institutions that we serve. Specifically, we are concerned that the new metrics may erode the value of libraries and push libraries to focus resources on US News data points, rather than on the services and outcomes that are most beneficial to our institutions, as the ABA requires.

Law library questions are included in the Faculty Resources portion of the ranking methodology and differ significantly from prior years’ data collection. Recognizing that the library-related US News questions used outdated metrics, a group of volunteers solicited through four law library organizations developed these new metrics beginning in early 2020.[1]  In November 2020, US News announced the immediate implementation of these metrics.

An additional metric, “ratio of credit bearing hours of instruction provided by law librarians to FTE law students,” was removed two days before the rankings were released. Speculation is that it was removed because an individual school reported a huge number that was an outlier by many orders of magnitude. However, there are other obvious over-reports in databases and titles that were still included in the ranking.[2]

The new measures for libraries, which represent 1.75% of the total ranking score (up from .75%), as described by USNWR, are:

• Number of hours per day law students have access to library study space during regular semester and exam schedules
• Number of hours per week law students have access to real-time reference/research/library services during the regular semester schedule
• Ratio of FTE professional and paraprofessional library staff to FTE law students
• Ratio of number of seats with library spaces to FTE law students
• Ratio of total number of presentations by library staff to FTE law students
• Total number of licensed or owned digital/electronic databases available to law students as
• Total number of titles available to law students

While these seven metrics combine to account for only 1.75% of a school's total ranking score, even small differences can affect the ranking position because of how tightly packed schools are together in the overall rankings. When US News had to correct erroneous scoring of library hours before the rankings were publicly released, and after excluding the question regarding for-credit instruction, several schools were reordered in the rankings on both occasions. To demonstrate the impact of these questions, we include the top fifty law libraries based on the new metrics at the end of this post. The new metrics focus on internal resource allocations such as library hours, seats, services, and title counts with no corresponding measure of quality. We acknowledge that library metrics cannot easily measure quality; it is easier to count hours and the number of times instructional presentations are provided. However, in counting services as mere numbers, libraries can use lower quality substitutes, like student workers, to extend service hours for rankings purposes. Overall, the rankings demand librarians focus on the counted metrics, instead of allowing law schools to make meaningful allocations of librarian time and resources based on the institution’s curricular and programming needs and librarians’ expertise. 

Troublingly, in several of the new categories, the rankings can be easily gamed. Libraries can extend hours of library access and reference service, although this may be a poor way to treat library staff. Many libraries can add more seating or extend what is considered “library space.” Even staff ratios can be gamed depending on how other departments, (e.g., IT) fit within a law school’s reporting structure, or if the school uses titles like “graduate assistant” or “library fellow” to classify student employees as professional staff. This may be particularly true at institutions that also house Library or Information Science educational programs and may view these students as “professionals” under US News definitions. Presentations, which can range from library orientations and tours to credit-bearing instruction for LLM students, are counted the same regardless of length or attendance.

Further, the new library resource metrics advantage law schools that are part of large research universities since the measures include titles and databases available through the larger institution. Title counts and database offerings for non-law titles at large research institutions dwarf legal titles and databases, and yet, they are now counted in law school ranking metrics. The measure also overlooks that modern law libraries have resources such as purchasing on demand, rapid interlibrary loan, and consortial agreements, to provide quick access to titles not available in our own collections.  

A concerning long-term implication of these metrics is the pressure it places on law libraries and Deans to compete on measures that demonstrate no meaningful outcomes, and in many cases will be at odds with our law schools’ mission. The ABA Standards require law libraries to develop and implement collections and services that best support our law schools’ unique missions. Contrary to the Standards, the new US News library metrics incentivize uniformity and could force libraries to prioritize spending to maximize US News ranked metrics rather than directing resources to best meet our institutions’ research and educational missions. 

Potentially, the most devastating consequence of these new metrics is the slow stifling of innovation.  Rather than encouraging libraries to meaningfully contribute to their law schools and help students meet their goals, these metrics continue to reinforce outdated modes of operating -- collecting titles, adding seats, and opening library spaces “just in case” -- and limit us from aligning our operations with our community’s goals. For example, awarding top points to libraries with 24/7 access penalizes schools that have made an informed decision not to offer such access based on risk assessments of their own environments. Gaming this metric by offering 24/7 access could incentivize law schools to ignore their moral and legal responsibilities regarding students’ safety and well-being, and limitations of their own building’s structure.

Other existing US News measures capture the value that a strong law library provides to a law school and a legal education program. Expenditures per student include library expenditures. Peer assessment ranks also reflect how well libraries support outstanding faculty scholarship. Lawyer and judge assessment ranks include how well libraries prepare students for practice by teaching important skills in legal research and legal technology.  Professional librarians could easily be counted in ratios by altering the definitions of existing questions about faculty and clinicians. Law libraries should be seen through their contributions within these larger US News measures, rather than perpetuating outdated views that we are separate from our broader institutions.        

Unfortunately, many law schools and libraries may forgo discussion on these and other issues presented with the new library ranking metrics as they seek to maximize the 1.75% of value that libraries are assessed to have for US News purposes. As Directors, when we work with our Law School Deans, we would much prefer to have conversations about resource allocations based upon the needs of our students, faculty, staff, alumni, and other institutional stakeholders, not a conversation forced by the US News ranking metrics.

How Law Libraries Score Under the New Metrics[3]

Law Library	Law Library “Rank”	Law Library “Score”
University of Arizona (Rogers)	1	29
Brigham Young University (Clark)	2	32
Indiana University (Maurer)	3	34
University of Dayton	4	38
University of Houston	4	38
University of Iowa	6	40
Harvard University	7	44
Boston College	8	47
Stanford University	8	47
University of Pennsylvania (Carey)	8	47
Texas Tech University	11	49
University of Wisconsin--Madison	11	49
Penn State (Dickinson)	13	50
University of Kansas	13	50
University of Oregon	15	51
University of Michigan	15	51
William & Mary Law School	15	51
Cornell University	18	52
Northwestern University (Pritzker)	18	52
University of Richmond	20	54
University of Tennessee	20	54
Boston University	22	56
University of Georgia	22	56
University of Hawaii (Richardson)	22	56
University of Minnesota	25	57
University of Virginia	25	57
University of Wyoming	25	57
Case Western Reserve University	28	58
Columbia University	28	58
Georgetown	30	59
Indiana University (McKinney)	30	59
University at Buffalo (SUNY)	30	59
University of California (Davis)	30	59
Penn State (University Park)	34	60
Ohio State University (Moritz)	34	60
Tulane University	36	61
University of California (Berkeley)	36	61
Cleveland State University	38	62
University of Texas (Austin)	38	62
University of Chicago	40	63
University of California (Irvine)	41	64
University of Notre Dame	41	64
Drake University	43	65
Duke University	43	65
University of Florida (Levin)	43	65
University of Missouri	43	65
University of California (Los Angeles)	47	66
University of Pittsburgh	47	66
Washington University (St. Louis)	49	67

[1] The volunteers were led by Beth Adelman, University at Buffalo School of Law; Teresa Miguel-Stearns, Arizona Law; Adeen Postar, AU Washington College of Law; and Roger Skalbeck, Richmond School of Law.

[2] In databases, the top entry lists almost 155,000 more databases than any other entry, and in titles the top entry has almost 13,000,000 more than any other library.

[3] Note on table development/construction method: For each school’s law library in this table, the 7 library metrics were pulled directly from USNWR’s Academic Insight Product on March 29 (after the reported total hours open correction and the instruction deletion). Each entry was ranked with the highest value as 1, keeping schools with identical responses at the same tied rank. All seven ranks were then averaged together as “law library ‘score’” and ranked with the lowest value as 1 for “law library ‘rank’,” keeping schools with identical scores at the same rank. Only the “top 49” schools are included to demonstrate these measures’ “effectiveness.” Schools with N/A entries were removed for ease of calculation. If included, and averaged only on answered metrics, University of Memphis would have ranked in the top 49.