U.S. News Rankings Potpourri: Waning Influence, Penalizing 53 Boycotting Law Schools, And Declining T14 Lawyer|Judge Reputation Scores

Reuters, After Setbacks, U.S. News Law School Rankings Show Signs of Waning Influence:

[T]he influence of U.S. News’ law school rankings is waning following a widespread boycott by schools that began in 2022 and a series of data errors that plagued the rollout of the 2023 list. According to U.S. News, 53 schools out of the nation's 197 accredited by the American Bar Association declined to provide data this year.

Those factors have hurt the credibility of the rankings, [Mike] Spivey said, and amplified longstanding criticism that a one-size-fits-all ranking can’t capture meaningful differences between schools.

Above the Law, U.S. News Law School Rankings Makes Its Smartest Methodological Change: Telling Holdout Schools To Suck It:

Law schools that refused to cooperate with U.S. News saw their peer surveys ignored. It's the right call. ...

Maryland Law’s Donald Tobin wrote an op-ed for TaxProf Blog laying out his pros and cons of the new USNWR methodology. Some of his points are pretty good (“using two-year averages for bar passage and employment data” to cut back on volatility), some less so (suggesting that outcome measurements should be downplayed), but all in all it’s a thoughtful critique that concludes by re-weighting the raw numbers to generate a ranking in a way that makes a lot more sense and is worth checking out.

But one snippet of the piece jumped out:

One last point, U.S. News appears to be throwing a little bit of a temper tantrum. ...

Frankly, the fact that USNWR refused to consider those surveys is less interesting than the fact that those schools submitted surveys at all. Marvel at the unmitigated chutzpah it takes to say, “well, we don’t like your ranking so we refuse to help you with it… also we demand that you consider this submission where we get to shit talk our competitors.”

But this decision has huge implications for the validity of the rankings. Has U.S. News done any analysis regarding who the people are who it is not excluding and whether U.S. News’ statistical pool of respondents continues to make up a diverse group of respondents? (Maybe they have, but they certainly haven’t addressed this publicly). Have they examined if there is a type of school that is more likely to avoid participation? This tantrum likely skews the results of their survey.

Excluding those results may negatively impact the results, but so might including those scores. If a school has signaled that it objects to the whole process and doesn’t care about cooperating to improve the validity of the rankings, how can U.S. News trust that the school is taking the survey seriously? The publication’s decision might seem petty, but it feels irresponsible to include submissions from schools that actively bad mouth the process. Given the landscape, the decision to disregard those surveys is a necessary evil brought about the law schools who decided to pout.

This revenge by U.S. News against those who don’t participate is also inappropriate for an organization that claims its standing to do these rankings through journalism. Do only people who do what a journalist wants count? Basically, U.S. News is saying that if you don’t give me the private information we want, you don’t get to vote. This doesn’t seem like sound journalism or like an institution interested in getting things right.

I get the analogy, but it doesn’t quite track. This is more like the subject refusing to comment for the story itself but demanding the publication grant them their own column space to vent. ...

Again, the professor’s post is an interesting take on the process and his version of the rankings rings far more true to this veteran industry observer. But while we enter another year of goofy U.S. News rankings, it’s important to keep the blame squarely focused on the law schools who brought all this on themselves.

Hopefully, they’ll end their temper tantrum sooner rather than later.

Derek Muller (Notre Dame; Google Scholar), The 2024-2025 USNWR Law School Rankings: Methodology Tweaks May Help Entrench Elite Schools, But Elite Schools See Reputation Decline Among Lawyers and Judges:

Changes to lawyer and judge peer reputation surveys.

One more methodological change of note:

Legal professionals – including hiring partners of law firms, practicing attorneys and judges – rated programs' overall quality on a scale from 1 (marginal) to 5 (outstanding), and were instructed to mark "don't know" for schools they did not know well enough to evaluate. A school's score is the average of 1-5 ratings it received across the three most recent survey years. U.S. News administered the legal professionals survey in fall 2023 and early 2024 to recipients that law schools provided to U.S. News in summer 2023. Of those recipients surveyed in fall 2022 and early 2023, 43% responded. For this edition, U.S. News complemented these ratings by surveying partners at big law firms, sampled based on their size – larger firms were more frequently surveyed – while establishing geographic dispersion. Leopard Solutions, which partnered with U.S. News on its Best Companies to Work For: Law Firms list, provided U.S. News with the contacts from which a sample was drawn.

USNWR recognized that as schools “boycotted” the survey, they would have a smaller universe of lawyers and judges to survey. In the past, schools submitted 10 names (up to ~2000 names). The response rate was quite low, so USNWR used a three-year average. As schools stop submitting names, USNWR looked elsewhere.

And it deliberately selected a category: “partners at big law firms, sampled based on their size—larger firms were more frequently surveyed.” ...

11 of the “top 14” schools are experiencing all-time lows, either new lows or lows that tie previous lows, since USNWR began this metric in 1998 in the lawyer and judge survey category. Here’s the score in this category (on a 1-5 scale), with the comparison of the all time high for each school.

• Stanford: 4.7 (all-time high: 4.9)
• Harvard: 4.6 (4.9)
• Chicago: 4.6 (4.8)
• Columbia: 4.5 (4.8)
• Yale: 4.5 (4.9)
• Michigan: 4.4 (4.7)
• Virginia: 4.4 (4.6)
• Duke: 4.3 (4.5)
• NYU: 4.3 (4.6)
• Berkeley: 4.3 (4.6)
• Georgetown: 4.2 (4.5)

Penn saw a decline from 4.4 to 4.3, and Cornell saw a decline from 4.3 to 4.2, but neither was an all-time low. Only Northwestern saw its score stable at 4.3 and not experience an all-time low. ...

Compare that to the next 86 schools in this category that have been ranked since 1998, and just 6 others experienced all-time low, again either new lows or lows that tie previous lows.

What could cause this disparity? Causation is tough to identify here, but let me posit two things.

First, we are seeing the slow phase-out of “boycotting” schools’ data inputs. Now two-thirds of the schools’ data is out of the mix. Schools were inclined to include their own supporters, and they are gone. Now, it’s hard to say that this is happening with such clinical precision at only the elite law schools and nowhere else. But perhaps a lot of elite law schools boycotted, and there’s some tag along effects as elite schools tend to rate elite schools comparably. That said, “complementing” with big law partner data should help shore up these figures, but perhaps there’s not enough here (indeed, we have no idea how they are mixed in with the data).

Second, it is possible that lawyers and judges—and perhaps in particular big law firm partners—are generally viewing elite law schools with less and less respect than at any time in recent history, and perhaps more so in the last year than at any time else. It might be law student or university protests about the Gaza conflict, fossil fuels, free speech—pick a cause. And perhaps the brunt of that publicity (and perhaps actual events) is falling on the most elite schools, which is creating fallout to their reputations in the legal community more generally. But that is very hard to assume and to pinpoint, and one might want to see what happens next year.

Law.com, A 'Bad Look'?: Legal Ed Professionals Weigh In on US News Rankings Methodology