Seto on the Proposed Boycott of the U.S. News Rankings

Theodore P. Seto (Loyola-L.A.), author of the influential article, Understanding the U.S. News Law School Rankings, 60 SMU L. Rev. 493 (2007), shares his thoughts on Case Dean Gary Simson's call to boycott the U.S. News rankings (blogged here, here, and here): 

Writing in the on-line edition of the National Law Journal, Dean Gary Simson of Case Western says the following about U.S. News’ recent announcement of possible changes to its methodology:

This announcement, and the wrench that it threatens to throw into structural changes that have been made to avoid being disadvantaged by a deeply flawed methodology, should cause law school faculties and administrations everywhere to finally say ‘enough’ and that they are done participating in a ranking system that has done substantial harm and little, if any, good to legal education in the United States.

In response, Mr. Robert Morse of U.S. News states;

If a law school refuses to provide U.S. News directly with statistical data from their annual American Bar Association (ABA) accreditation data questionnaire, then U.S. News still can get almost all of that school’s official ABA data from the ABA website. U.S. News would still be able to rank a law school, even if it refused to participate

Mr. Morse’s response is correct, but only with significant caveats.

First, the data on the ABA website is already a year old – the data on the basis of which U.S. News computed and published its “2009” rankings this past spring. Next year’s data will not even be given to the ABA until this fall, when law schools fill out their ABA questionnaire and then, a few weeks later, their U.S. News questionnaires. As a practical matter, the new data is unlikely to be posted to the ABA website until much later.

Second, the ABA website does not disclose expenditure or library data. As I have noted elsewhere, see Understanding the U.S. News Law School Rankings, expenditures play a dominant role in U.S. News’ rankings. In the absence of expenditure data, for example, Yale would fall by roughly 8-9 overall score points and would be forced to compete head-to-head with Harvard for the number one slot.

Third and perhaps most importantly, U.S. News’ private monopoly on current school data would be broken. Currently, potential competitors face an enormous barrier to entry: law schools give U.S. News (but no other ranking organization) access to current ABA questionnaire information, including financial data they disclose to no one else. If U.S. News were forced to rely on the ABA website, it would be forced to play on a level field. Competing rankings would more likely arise. This would give students more information, not less.

Having said all of the foregoing, I note further that, in my view, U.S. News does publish two important pieces of information about each school that it obtains through its own efforts: the two reputation scores. Although many academics pooh-pooh these scores, in my view they provide students and other consumers of U.S. News’ product with important information not available elsewhere.

I will be elaborating this point on Thursday at SEALS. Statistical analysis of U.S. News’ reputation scores from 2002 to 2009 indicates that those scores are unbelievably reliable. The Cronbach’s Alpha (a measure of reliability) for the peer assessment scores is .999 – in effect, perfect. The Cronbach’s Alpha for the lawyer/judge assessment scores is .996 – closer to perfection than almost any other measure on which we base significant decisions. What this means is that different raters rate schools incredibly consistently over time. In addition, the two reputation scores are highly consistent with each other. Each predicts in excess of 95% of the variance in the other.

This does not mean that U.S. News’ reputation ratings are either accurate or unbiased. But it does mean that people important to law schools (deans, faculty, judges, lawyers) hold views about law schools that are extraordinarily consistent. That such consistent views exist is a fact – whether the views themselves are accurate or not. U.S. News is merely reporting those views. Those views’ relevance to any decision we might make – where to attend, where to hire, where to teach – is legitimately debatable. To contend that they are inconsequential, however, is unrealistic.

Thus, even if law schools were to withhold data from U.S. News, U.S. News would still be able to rank schools, and would still be able to provide what I view as its single most important contribution to its readers – the reputation scores. But its monopoly grip on the sources of information needed to create competing rankings would be broken.