Monday, July 28, 2008
Seto on the Proposed Boycott of the U.S. News Rankings
Theodore P. Seto (Loyola-L.A.), author of the influential article, Understanding the U.S. News Law School Rankings, 60 SMU L. Rev. 493 (2007), shares his thoughts on Case Dean Gary Simson's call to boycott the U.S. News rankings (blogged here, here, and here):
Writing in the on-line edition of the National Law Journal, Dean Gary Simson of Case Western says the following about U.S. News’ recent announcement of possible changes to its methodology:
This announcement, and the wrench that it threatens to throw into structural changes that have been made to avoid being disadvantaged by a deeply flawed methodology, should cause law school faculties and administrations everywhere to finally say ‘enough’ and that they are done participating in a ranking system that has done substantial harm and little, if any, good to legal education in the United States.
In response, Mr. Robert Morse of U.S. News states;
If a law school refuses to provide U.S. News directly with statistical data from their annual American Bar Association (ABA) accreditation data questionnaire, then U.S. News still can get almost all of that school’s official ABA data from the ABA website. U.S. News would still be able to rank a law school, even if it refused to participate
Mr. Morse’s response is correct, but only with significant caveats.
First, the data on the ABA website is already a year old – the data on the basis of which U.S. News computed and published its “2009” rankings this past spring. Next year’s data will not even be given to the ABA until this fall, when law schools fill out their ABA questionnaire and then, a few weeks later, their U.S. News questionnaires. As a practical matter, the new data is unlikely to be posted to the ABA website until much later.
Second, the ABA website does not disclose expenditure or library data. As I have noted elsewhere, see Understanding the U.S. News Law School Rankings, expenditures play a dominant role in U.S. News’ rankings. In the absence of expenditure data, for example, Yale would fall by roughly 8-9 overall score points and would be forced to compete head-to-head with Harvard for the number one slot.
Third and perhaps most importantly, U.S. News’ private monopoly on current school data would be broken. Currently, potential competitors face an enormous barrier to entry: law schools give U.S. News (but no other ranking organization) access to current ABA questionnaire information, including financial data they disclose to no one else. If U.S. News were forced to rely on the ABA website, it would be forced to play on a level field. Competing rankings would more likely arise. This would give students more information, not less.
Having said all of the foregoing, I note further that, in my view, U.S. News does publish two important pieces of information about each school that it obtains through its own efforts: the two reputation scores. Although many academics pooh-pooh these scores, in my view they provide students and other consumers of U.S. News’ product with important information not available elsewhere.
I will be elaborating this point on Thursday at SEALS. Statistical analysis of U.S. News’ reputation scores from 2002 to 2009 indicates that those scores are unbelievably reliable. The Cronbach’s Alpha (a measure of reliability) for the peer assessment scores is .999 – in effect, perfect. The Cronbach’s Alpha for the lawyer/judge assessment scores is .996 – closer to perfection than almost any other measure on which we base significant decisions. What this means is that different raters rate schools incredibly consistently over time. In addition, the two reputation scores are highly consistent with each other. Each predicts in excess of 95% of the variance in the other.
This does not mean that U.S. News’ reputation ratings are either accurate or unbiased. But it does mean that people important to law schools (deans, faculty, judges, lawyers) hold views about law schools that are extraordinarily consistent. That such consistent views exist is a fact – whether the views themselves are accurate or not. U.S. News is merely reporting those views. Those views’ relevance to any decision we might make – where to attend, where to hire, where to teach – is legitimately debatable. To contend that they are inconsequential, however, is unrealistic.
Thus, even if law schools were to withhold data from U.S. News, U.S. News would still be able to rank schools, and would still be able to provide what I view as its single most important contribution to its readers – the reputation scores. But its monopoly grip on the sources of information needed to create competing rankings would be broken.
The following conclusion, based on statistics (and statistics can be found to support any position and interpreted to mean what you want) is a self-fulfilling prophecy as the reputation scores are biased and based upon the US News Rankings themselves; accordingly, the logic is circular (the reputation scores correlate with the ranking, but that is because the reputation scores are based on those same rankings):
"The Cronbach’s Alpha (a measure of reliability) for the peer assessment scores is .999 – in effect, perfect. The Cronbach’s Alpha for the lawyer/judge assessment scores is .996 – closer to perfection than almost any other measure on which we base significant decisions. What this means is that different raters rate schools incredibly consistently over time. In addition, the two reputation scores are highly consistent with each other. Each predicts in excess of 95% of the variance in the other."
Again, I have have stated previously, I call for tiers, tiers, tiers (which may lead to tears, tears, tears). Yale, whether 1, 2,3, 8 or 9 is a "Top 10"/"First Tier" school. Fordham is a "Top 25" whether it is 24, 25, 26, or 27, AND as a "Top 25"/"Second Tier" does not compare to Yale. And Third and Fourth Tier schools (under the current system) are what they are and in no way compete or are comparable to with "Top 10"/"First Tier" or "Top 25"/"Second Tier" schools.
The actual number does not matter, and US News could sell just as many magazines with Tiers instead of numerical values (anyone remember when they only ranked the top 25 and the rest were labeled either First or Second Tier schools?). The numbers do immeasurable damage to the law schools, whereas tiers (although not a panacea) I assert would do less harm.
Posted by: Adjunct Law Prof | Jul 28, 2008 4:59:36 AM
I think we should stop calling them "reputation" surveys; it's misleading -- we're supposed to be actually assessing the quality of each school's JD program. U.S. News used to call them "reputation" surveys, I think, but does not anymore. This may seem like a small point, but I think it's quite important.
Here's what U.S. News asks law profs: "Identify the law schools you are familiar with, and then rate the academic quality of their J.D. program at each of these schools. Consider all factors that contribute to or give evidence of the excellence of the school's J.D. program, for example, curriculum, record of scholarship, quality of faculty and graduates."
They're asking law profs as experts on legal education and scholarship, not as experts on public opinion ("reputation"). And lawyers/judges are asked the same thing (assess "academic quality") except asked to particularly consider degree to which schools prepare students for practice.
By sticking with "reputation," we're not only not answering the question (as we often scold our students for doing), but we're saying it's OK to answer the survey year after year in the absence of any real information on relative educational quality. For a different approach that actually focuses on assessing quality, see http://prawfsblawg.blogs.com/prawfsblawg/2008/07/the-educational.html.
Posted by: Jason Solomon | Jul 28, 2008 7:16:41 AM