Paul L. Caron
Dean




Tuesday, March 30, 2021

U.S. News Ranking Metrics Stifle Law Libraries, Tie Hands of Law Schools

TaxProf Blog op-ed:  U.S. News Ranking Metrics Stifle Law Libraries, Tie Hands of Law Schools, by Amanda Runyon (Pennsylvania), Leslie A. Street (William & Mary), & Amanda Watson (Houston):

LibraryUSNWR has released the 2022 Best of Law School rankings, including a new set of questions and methodology for law libraries. As library directors at schools that perform objectively well in these new calculations, we feel obligated to voice our concerns. We believe the unintended consequences and potential outcomes of these metrics are highly problematic for law libraries and the institutions that we serve. Specifically, we are concerned that the new metrics may erode the value of libraries and push libraries to focus resources on US News data points, rather than on the services and outcomes that are most beneficial to our institutions, as the ABA requires.

Law library questions are included in the Faculty Resources portion of the ranking methodology and differ significantly from prior years’ data collection. Recognizing that the library-related US News questions used outdated metrics, a group of volunteers solicited through four law library organizations developed these new metrics beginning in early 2020.[1]  In November 2020, US News announced the immediate implementation of these metrics.

An additional metric, “ratio of credit bearing hours of instruction provided by law librarians to FTE law students,” was removed two days before the rankings were released. Speculation is that it was removed because an individual school reported a huge number that was an outlier by many orders of magnitude. However, there are other obvious over-reports in databases and titles that were still included in the ranking.[2]

The new measures for libraries, which represent 1.75% of the total ranking score (up from .75%), as described by USNWR, are:

  • Number of hours per day law students have access to library study space during regular semester and exam schedules
  • Number of hours per week law students have access to real-time reference/research/library services during the regular semester schedule
  • Ratio of FTE professional and paraprofessional library staff to FTE law students
  • Ratio of number of seats with library spaces to FTE law students
  • Ratio of total number of presentations by library staff to FTE law students
  • Total number of licensed or owned digital/electronic databases available to law students as
  • Total number of titles available to law students

While these seven metrics combine to account for only 1.75% of a school's total ranking score, even small differences can affect the ranking position because of how tightly packed schools are together in the overall rankings. When US News had to correct erroneous scoring of library hours before the rankings were publicly released, and after excluding the question regarding for-credit instruction, several schools were reordered in the rankings on both occasions. To demonstrate the impact of these questions, we include the top fifty law libraries based on the new metrics at the end of this post. The new metrics focus on internal resource allocations such as library hours, seats, services, and title counts with no corresponding measure of quality. We acknowledge that library metrics cannot easily measure quality; it is easier to count hours and the number of times instructional presentations are provided. However, in counting services as mere numbers, libraries can use lower quality substitutes, like student workers, to extend service hours for rankings purposes. Overall, the rankings demand librarians focus on the counted metrics, instead of allowing law schools to make meaningful allocations of librarian time and resources based on the institution’s curricular and programming needs and librarians’ expertise. 

Troublingly, in several of the new categories, the rankings can be easily gamed. Libraries can extend hours of library access and reference service, although this may be a poor way to treat library staff. Many libraries can add more seating or extend what is considered “library space.” Even staff ratios can be gamed depending on how other departments, (e.g., IT) fit within a law school’s reporting structure, or if the school uses titles like “graduate assistant” or “library fellow” to classify student employees as professional staff. This may be particularly true at institutions that also house Library or Information Science educational programs and may view these students as “professionals” under US News definitions. Presentations, which can range from library orientations and tours to credit-bearing instruction for LLM students, are counted the same regardless of length or attendance.

Further, the new library resource metrics advantage law schools that are part of large research universities since the measures include titles and databases available through the larger institution. Title counts and database offerings for non-law titles at large research institutions dwarf legal titles and databases, and yet, they are now counted in law school ranking metrics. The measure also overlooks that modern law libraries have resources such as purchasing on demand, rapid interlibrary loan, and consortial agreements, to provide quick access to titles not available in our own collections.  

A concerning long-term implication of these metrics is the pressure it places on law libraries and Deans to compete on measures that demonstrate no meaningful outcomes, and in many cases will be at odds with our law schools’ mission. The ABA Standards require law libraries to develop and implement collections and services that best support our law schools’ unique missions. Contrary to the Standards, the new US News library metrics incentivize uniformity and could force libraries to prioritize spending to maximize US News ranked metrics rather than directing resources to best meet our institutions’ research and educational missions. 

Potentially, the most devastating consequence of these new metrics is the slow stifling of innovation.  Rather than encouraging libraries to meaningfully contribute to their law schools and help students meet their goals, these metrics continue to reinforce outdated modes of operating -- collecting titles, adding seats, and opening library spaces “just in case” -- and limit us from aligning our operations with our community’s goals. For example, awarding top points to libraries with 24/7 access penalizes schools that have made an informed decision not to offer such access based on risk assessments of their own environments. Gaming this metric by offering 24/7 access could incentivize law schools to ignore their moral and legal responsibilities regarding students’ safety and well-being, and limitations of their own building’s structure.

Other existing US News measures capture the value that a strong law library provides to a law school and a legal education program. Expenditures per student include library expenditures. Peer assessment ranks also reflect how well libraries support outstanding faculty scholarship. Lawyer and judge assessment ranks include how well libraries prepare students for practice by teaching important skills in legal research and legal technology.  Professional librarians could easily be counted in ratios by altering the definitions of existing questions about faculty and clinicians. Law libraries should be seen through their contributions within these larger US News measures, rather than perpetuating outdated views that we are separate from our broader institutions.        

Unfortunately, many law schools and libraries may forgo discussion on these and other issues presented with the new library ranking metrics as they seek to maximize the 1.75% of value that libraries are assessed to have for US News purposes. As Directors, when we work with our Law School Deans, we would much prefer to have conversations about resource allocations based upon the needs of our students, faculty, staff, alumni, and other institutional stakeholders, not a conversation forced by the US News ranking metrics.

How Law Libraries Score Under the New Metrics[3]

  Law Library

Law Library “Rank”

Law Library “Score”

University of Arizona (Rogers)

1

29

Brigham Young University (Clark)

2

32

Indiana University (Maurer)

3

34

University of Dayton

4

38

University of Houston

4

38

University of Iowa

6

40

Harvard University

7

44

Boston College

8

47

Stanford University

8

47

University of Pennsylvania (Carey)

8

47

Texas Tech University

11

49

University of Wisconsin--Madison

11

49

Penn State (Dickinson)

13

50

University of Kansas

13

50

University of Oregon

15

51

University of Michigan

15

51

William & Mary Law School

15

51

Cornell University

18

52

Northwestern University (Pritzker)

18

52

University of Richmond

20

54

University of Tennessee

20

54

Boston University

22

56

University of Georgia

22

56

University of Hawaii (Richardson)

22

56

University of Minnesota

25

57

University of Virginia

25

57

University of Wyoming

25

57

Case Western Reserve University

28

58

Columbia University

28

58

Georgetown

30

59

Indiana University (McKinney)

30

59

University at Buffalo (SUNY)

30

59

University of California (Davis)

30

59

Penn State (University Park)

34

60

Ohio State University (Moritz)

34

60

Tulane University

36

61

University of California (Berkeley)

36

61

Cleveland State University 

38

62

University of Texas (Austin)

38

62

University of Chicago

40

63

University of California (Irvine)

41

64

University of Notre Dame

41

64

Drake University

43

65

Duke University

43

65

University of Florida (Levin)

43

65

University of Missouri

43

65

University of California (Los Angeles)

47

66

University of Pittsburgh

47

66

Washington University (St. Louis)

49

67

[1] The volunteers were led by Beth Adelman, University at Buffalo School of Law; Teresa Miguel-Stearns, Arizona Law; Adeen Postar, AU Washington College of Law; and Roger Skalbeck, Richmond School of Law.

[2] In databases, the top entry lists almost 155,000 more databases than any other entry, and in titles the top entry has almost 13,000,000 more than any other library.

[3] Note on table development/construction method: For each school’s law library in this table, the 7 library metrics were pulled directly from USNWR’s Academic Insight Product on March 29 (after the reported total hours open correction and the instruction deletion). Each entry was ranked with the highest value as 1, keeping schools with identical responses at the same tied rank. All seven ranks were then averaged together as “law library ‘score’” and ranked with the lowest value as 1 for “law library ‘rank’,” keeping schools with identical scores at the same rank. Only the “top 49” schools are included to demonstrate these measures’ “effectiveness.” Schools with N/A entries were removed for ease of calculation. If included, and averaged only on answered metrics, University of Memphis would have ranked in the top 49.

https://taxprof.typepad.com/taxprof_blog/2021/03/us-news-ranking-metrics-stifle-law-libraries-tie-hands-of-law-schools.html

Law School Rankings, Legal Ed Rankings, Legal Education | Permalink

Comments

We write to respond to the post about law library questions in the US News survey.

As members of the U.S. News Joint Task Force, we facilitated a year-long inclusive and rigorous effort to create a more modern picture of law libraries for prospective law students. The new questions achieve this. Prior questions had not been updated in many years. We recognize there is more work to be done.

Beginning in January 2020, our group sought ways to identify and explain the value and service our law libraries and employees provide. The Task Force explored a process to develop a fair and improved representation of law libraries for prospective students. Our working groups looked at collections, space, services and access.

We invited the academic law library community to contribute and comment throughout the process.

As Einstein once said, “Many of the things you can count, don't count. Many of the things you can't count, really count.” We know that quantifiable data cannot possibly capture all the value our libraries bring to legal education. However, the US News model restricts metrics to only those things that can be counted.

The Joint Task Force decided to work with US News to develop metrics that are accurate, consistent, transparent and verifiable. We believe that any errors made in the library survey last year were just that -- errors -- not attempts to game the 1.75% that comprises the law library portion of the overall law school rankings. Similarly, these metrics were not developed to rank law libraries against one another.

The Joint Task Force will continue to engage and work with the academic law library community and US News. We expect to revisit and improve metrics to better serve students, law schools and our libraries.


By Beth Adelman (University at Buffalo School of Law), Rich Leiter (Nebraska), Teresa Miguel-Stearns (Arizona Law), Adeen Postar (AU Washington College of Law), Roger V. Skalbeck (Richmond School of Law)

Posted by: Roger Skalbeck | Apr 1, 2021 2:12:09 PM