Kuehn: Law School Specialty Program Ranking Credibility

Tax Prof Blog op-ed:  Law School Specialty Ranking Credibility, by Robert Kuehn (Washington University):

As part of its spring rite of ranking law schools, U.S. News ranks law school specialty programs. The specialty rankings are based solely on reputation. In particular, a faculty member teaching in a specialty area at each school is asked to assess their area at other ABA accredited schools. Yet the most recent rankings demonstrate that the current methodology does not yield credible results for many specialties.

Prior to 2020, the faculty member rating a specialty program was directed to choose the schools with the top 15 programs in the rater’s area. U.S. News would then rank the top up to 20 programs based on how often the school was listed. Beginning with its 2020 rankings, the system changed. The faculty member now was asked to rate the specialty program at every law school on a scale from Marginal (1) to Outstanding (5), mirroring the “peer assessment” method used to assess the academic reputation of schools. If a school’s program receives as few as ten ratings, it is ranked in that specialty. Whereas having an informed opinion on the 15 best programs in one’s field under the old system seemed manageable, the new system is overwhelming, if not impossible — provide an informed opinion, reduced to a number, on the programs at all schools so they can then be credibly ranked, based on their average score, in descending order from supposedly best to worst.

Since I highlighted problems with the new 2020 specialty ranking approach, U.S. News has added four more specialty programs: business/corporate law; constitutional law; contracts/commercial law; and criminal law. Unlike the prior nine categories, these four new programs cover core, not special, areas of a school’s educational program. An examination of the ranking of these core areas illustrates the continuing credibility problem with the specialty program ranking system.

Looking at the recently released 2023 specialty rankings, among the 187 ranked schools there are noticeable clusters of schools ranked highest by U.S. News at the top of the four new programs, and an overpopulation of lowest-ranked schools at the bottom. The top ten ranked constitutional and contracts/commercial law specialty programs are all at schools ranked 1-14 by U.S. News (“T-14” schools). There is a similar pattern for the new business/corporate and criminal law specialties — 9 of the top 10 ranked programs are at T-14 ranked schools (the tenth at the 15th ranked school). Conversely, the programs ranked in the bottom quarter of the constitutional and contracts/commercial law specialties are all at schools ranked over 100.

To analyze the statistical relationship between a school’s reputation and its specialty program ranking, scores for each school in every specialty were collected (accessible via U.S. News College Compass). Using this data, the estimated correlation coefficients between the school’s 2023 peer assessment and specialty program assessment scores are reported below. The correlations are similar when averaging scores over the 2021-2023 surveys and, though somewhat smaller, when using law school ranking instead of peer assessment.

A coefficient of 1.0 indicates a perfect relationship between two variables, with coefficients of .8 or greater characterized as showing a “very strong” positive relationship. In contrast to some of the original specialty areas, the ratings of a school’s four core programs stand out as very strongly associated with its peer reputation, ranging from .94 for criminal law to a near perfect .98 for constitutional law.

For the four new core specialties, peer assessment scores, therefore, are very highly predictive of a school’s specialty score. Regression models for the new specialties yielded R-squared values of between 0.89 for criminal law to 0.96 for constitutional law, indicating that peer assessment scores explain 89-96% of the variation in the specialty program ranking, leaving only 4-11% explained by anything other than the peer score.

The scatterplots (with fitted regression line) below illustrate the very strong relationship between a school’s peer assessment and specialty score for two of the new specialty areas.

For constitutional law, all but one school with a peer assessment score of at least 4.0 also had a constitutional law score above 4.0. Conversely, no school with a peer assessment score below 2.0 had a constitutional law score above 2.0. For contracts/commercial law, again, all but one school with a peer assessment score of 4.0 also had a specialty score above 4.0. Yet, the bottom 25 ranked contracts/commercial law programs were all at schools with peer assessment scores below 2.0.

Is the very strong association between school and specialty program reputations simply because the highest ranked schools also truly happen to have the highest quality specialty programs and vice-versa for the lowest ranked schools, with schools ranked in the middle likewise having middling specialty programs? Not likely. Given the enormous task of grading every school’s specialty program, no rater could know much about these core areas at all schools without extensive research. What is more likely with these extremely high correlations is that in many, if not most, instances, faculty members rating specialty programs are using their overall opinion of the school as a proxy for the quality of the specialty area. The raters, given the enormity of the task, are cueing off a school’s academic reputation and not, with reliable information, independently judging the quality of its specialty program.

Specialty program rankings may be here to stay, but the current methodology does not yield credible results for all schools. With strong evidence that the rankings in the newer specialty areas are not independent of the rater’s general view of the school, reliance on those rankings as a valid indicator of the quality of the program at many schools is not warranted.

2023 U.S. News Specialty Rankings:

• Omnibus Specialty Rankings (compilation of all 13 special rankings)
• Omnibus Specialty Rankings v. Overall Rankings
• Business/Corporate Law
• Clinical Law
• Constitutional Law
• Contracts/Commercial Law
• Criminal Law
• Dispute Resolution
• Environmental Law
• Health Law
• Intellectual Property Law
• International Law
• Legal Writing
• Tax Law
• Trial Advocacy