Karen Sloan (Law.com), US News Makes Last-Minute Changes to Law School Rankings, Fueling Criticism and Concern:
U.S. News & World Report said Thursday that it’s removing a controversial diversity ranking from its law school rankings package that is set to be released March 30.
That decision comes after deans from 162 law schools asked for a last-minute revision to the standalone “Most Diverse Law Schools Ranking,” citing the exclusion of students of more than one race from its calculation of “underrepresented minorities.” The deans requested that the ranking be recalculated to include multiethnic students—which would mark the second time the diversity ranking was changed in the past week. U.S. News this week recalculated that ranking after initially failing to include Asian students.
“After receiving feedback from law schools, we’ve removed the law diversity ranking and will publish it at a later, as yet undetermined date in order to include students of two or more races,” wrote U.S. News Chief Data Strategist Bob Morse in an email.
The deans' letter to Morse said that excluding multiethnic law students from the diversity rankings is “simply wrong and unacceptable.”
The controversy surrounding the diversity ranking comes amid mounting concern over the methodology U.S. News uses in its law school rankings and its handling of the trove of granular data that law schools supply to it each year. U.S. News has modified this year’s rankings twice since March 16—when it first provided law schools with an embargoed version of the upcoming rankings and underlying data. One change was to add Asian students into the diversity ranking after their initial exclusion, and the other was to fix incorrect data related to law libraries, which is used to calculate each school’s overall ranking.
Changing the rankings multiple times prior to their official publication is unprecedented, according to Erwin Chemerinsky, dean of the University of California, Berkeley School of Law. Those changes are bringing to the fore the longstanding uneasiness legal educators have over how U.S. News weighs law schools, as well as fresher concerns over its handling of data and the accuracy of its rankings. ...
University of Oregon Law Dean Marcilynn Burke, who organized the March 24 letter signed 162 fellow deans, said she was surprised that U.S. News opted to pull the entire diversity ranking instead of adding in students of more than one race, which is a data point it has on hand. But moving forward with a diversity rankings that excluded multiracial law students could have seriously undermined the credibility of U.S. News’ entire rankings operation, she added.
“Overall, it probably was the right decision to pull back now and go back to the drawing board,” Burke said. “But accurate information about the diversity of our law schools is important, as students try to figure out what their experience in law school will be. Sheer numbers don’t tell you the whole story, of course, but it is a significant part of the puzzle.”
Even before U.S. News changed, then pulled, that diversity ranking, it had already revised the much-watched overall law school ranking after schools pointed out an error in the data relating to law libraries, which accounts for 2% of each school’s overall ranking. One library metric—the number of hours a school’s law libraries are open to students—was computed incorrectly by U.S. News, according to its March 24 message to law schools. As a result, a large number of schools were inaccurately credited with fewer open hours than they should have been. Several law schools noticed the error when they combed through the early data—schools can pay U.S. News $15,000 annually for access to the granular data behind the rankings—and alerted Morse to the problem.
U.S. News pulled down all the rankings and recalculated with the correct figures for library open hours. The library open hours metric accounts for just .25% of a law school’s overall ranking, yet correcting that mistake changed the ranking of more than 30 schools, according to multiple law school faculty, some by as much as six spots. That illustrates just how compressed the law school rankings are and that small methodology tweaks can cause major shifts. ...
Law school officials have identified a second issue with the library data they say hasn’t been addressed—and which was not discussed in U.S. News’ March 24 message to schools—which is widespread confusion over a new question about the number of credit hours taught by law librarians. The answers schools provided ranged from a low of around 10 to as much as 300, which suggests that some schools reported the number of course credits offered in classes taught by law librarians while others calculated the actual number of credits individual students had earned in classes that were taught by law librarians. Those outlier responses in turn threw off the standard deviation calculation U.S. News uses to place schools in relation to each other according to individual rankings metrics, said one law professor who has closely studied this year’s figures. ...
To critics, the wide range of responses to the law librarian teaching question indicates that U.S. News is not auditing the data it receives from law schools to catch and fix clearly incorrect figures, and instead is using questionable data in calculations and rankings where miniscule differences separate schools.