Paul L. Caron

Tuesday, April 9, 2024

Muller: Did U.S. News Change Its Law School Rankings Methodology To Avoid A Steeper Fall For Cornell And NYU?

Derek Muller (Notre Dame; Google Scholar), The 2024-2025 USNWR Law School Rankings: Methodology Tweaks May Help Entrench Elite Schools, But Elite Schools See Reputation Decline Among Lawyers and Judges:

US News (2023)Hours after the release of last year’s dramatic change to the USNWR methodology, I noted the dramatic increase in “compression and volatility” in the coming rankings.

USNWR changed a couple of things in its methodology:

There were a couple differences in how the rankings were calculated, described below. In summary, U.S. News averaged its bar passage and employment indicators over two years. Also, the lawyers and judges assessment score had a second source of ratings besides names supplied by law schools. ...

[W]hy did USNWR decide to change this year? There are two possible explanations for this change, and, tellingly, either explanation looks bad for USNWR.

One explanation is that USNWR was simply unaware of the potential volatility in their ranking sand is responding now. That is a bad look for USNWR. It took me minutes to spot this likely problem. If it escaped their entire data team’s months-long vetting, it’s a telling concession.

The other explanation is that USNWR was aware of the potential volatility, but it took a step this year to react to reduce it. That’s a bad look, too—if it was aware of the problem, why didn’t it address the problem then? It did, after all, have all of the granular employment data in previous years. And if it was aware last year, what prompted the change this year?

The related answer to both, by the way, is that it saw something problematic in what the outputs would be, and it modified the weighing to avoid undesirable results. This is not something I have proof for, I admit. I can only infer from the actions take in response to some events of the last year.

But we saw a few schools—notably, as I pointed out, NYU and Cornell—that would disproportionately suffer under the new system. I projected NYU to slide to 11 and Cornell to 18. Instead, with a re-weighing, NYU slid only to 9, and Cornell to 14. Other schools—particularly Washington University in St. Louis, North Carolina, and Texas A&M—were projected to rise much faster. The rankings changes are designed to put a governor on moves down—or up—the rankings.

Now, it’s not possible to prove that USNWR saw that NYU and Cornell would slide much faster than they thought appropriate and changed the methodology.

But I can simply point out that these arguments were raised publicly for months, and this methodological change is designed to slow down the kinds of dramatic changes that we publicly expected this year. It’s not a good look whatever the motivation was, because it reflects a lack of competence about the changes instituted last year. Relatedly, USNWR is here conceding that too much volatility is a bad thing. That is, it would prefer to see less movement (and more entrenchment) in its final product.

(The lengthening window of data is creating increasingly strange results. For instance, today’s prospective law students are considering what their employment and outcome prospects look like in 2027. The current methodology has data stretching back to the Class of 2019 (two-year average of ultimate bar passage rates for the Classes of 2019 and 2020). That said, perhaps it’s better to think of schools over a longer period of time

Law School Rankings, Legal Ed Rankings, Legal Education | Permalink