Did 16 Law Schools Commit Rankings Malpractice?

As U.S. News rankings aficionados know, the methodology used in the 2011 U.S. News Law School Rankings gives 18% weight to employment statistics: 14% to the percentage of the Class of 2008 employed nine months after graduation (which is reported to the ABA as well), and 4% to the percentage of the class employed at graduation (which is not reported to the ABA).

74 schools did not supply U.S. News with the percentage of the class employed at graduation. This continues a ten-year trend -- the number of nonreporting schools has more than doubled over this period:

U.S. News has publicly disclosed that it estimates the employed at graduation figure for nonreporting schools with this formula:

Employed at Nine Months - ~30 percentage points = Employed at Graduation

As Ted Seto notes, "This was apparently intended to capture the relationship, on average, between the two variables for schools reporting both numbers." Understanding the U.S. News Law School Rankings, 60 SMU L. Rev. 493, 500 (2007).

The 74 nonreporting schools presumably had an employed at graduation number more than 30 percentage points below their employed at nine months number and thus benefited in the rankings by not reporting their employed at graduation number to U.S. News.  Robert Morse, Director of Data Research at U.S. News, reports on his blog that Alabama "made errors reporting some of their data" -- they apparently "were too late" in reporting their employed at graduation number. Alabama's employed at nine months number was 96.9%, so U.S. News used 66.9% as Alabama's employed at graduation number in computing the rankings.  Partially as a result, Alabama slid from #30 to #38 in the overall rankings. Alabama now says that its correct employed at graduation number was 92.1% (25.2 percentage points higher than the figure used by U.S. News).

A more interesting question is why 16 law schools (compared to 23 law schools last year) reported employed at graduation numbers more than 30 percentage points lower than their employed at nine months number:

Employed at Graduation	Employed at 9 Months	School	Difference	Overall Rank
37.5%	95.8%	Arkansas-Little Rock	58.3%	Tier 3
41.1%	93.3%	Memphis	52.2%	Tier 3
44.8%	96.2%	John Marshall (Atl.)	51.4%	Tier 4
51.3%	96.4%	Whittier	45.1%	Tier 4
50.3%	92.4%	Creighton	42.1%	Tier 3
54.1%	94.3%	South Dakota	40.2%	Tier 3
50.7%	89.6%	Missouri-Columbia	38.9%	93
59.1%	96.6%	Seattle	37.5%	86
53.8%	89.6%	U. Mississippi	35.8%	Tier 3
56.1%	91.2%	South Carolina	35.1%	Tier 3
60.4%	95.2%	Vermont	34.8%	Tier 3
60.0%	94.6%	Loyola-N.O.	34.6%	Tier 3
57.9%	90.4%	Oregon	32.5%	80
61.0%	92.8%	Marquette	31.8%	Tier 3
58.6%	88.9%	Wyoming	30.2%	Tier 3
60.6%	90.7%	Richmond	30.1%	86

Many of these schools undoubtedly adversely affected their overall ranking by reporting their employed at graduation data to U.S. News.  As Ted Seto explains in his article, because U.S. News uses round numbers in determining a school's overall score, the 4% weighting of the employed at graduation data easily could impact a school's overall ranking -- i.e. a school whose overall score ended in .49 would move up to the next grouping with an increase in its overall score of merely .01.  Ted also explains that an increase of 22 percentage points in the employed at graduation figure would have improved a school's overall score by one full point (in the 2007 rankings) -- which Arkansas-Little Rock and Memphis could have achieved by declining to disclose their 58.3% and 52.2% figures, respectively, and instead allowing U.S. News to assign them 65.8% and 63.3% figures.

With the close clustering of schools ranked 78 (two schools), 80 (6 schools), 86 (7 schools), 93 (5 schools), and 98 (4 schools), Missouri-Columbia, Oregon, Richmond, and Seattle may have improved their rank within the Top 100 by declining to disclose their employed at graduation data.  Depending on their overall score, one or more of the schools in Tier 3 (Arkansas-Little Rock, Creighton, Loyola-New Orleans, Marquette, Memphis, Mississippi, South Carolina, South Dakota, Vermont, Wyoming) might have cracked the Top 100, and one or more of the schools in Tier 4 (John Marshall (Atlanta), Whittier) might have found themselves in Tier 3, by not disclosing their true employed at graduation data and instead allowing U.S. News to use its surrogate figure. Eight of these schools (Arkansas-Little Rock, Loyola-New Orleans, Missouri-Columbia, Oregon, Seattle, South Carolina, South Dakota, Whittier) also may have committed rankings malpractice last year by reporting their employed at graduation number to U.S. News.

Update: ABA Journal, Were 16 Law Schools Too Revealing in Disclosures to U.S. News?

Alabama responds on Bob Morse's blog:

Alabama Law completed and submitted the most recent U.S. News survey in October 2009. U.S. News sent back to us a verification questionnaire in December 2009. We updated and verified the data before submitting it back to U.S. News in January 2010. The fax confirmation shows January 14th, to be exact, which is three months before the 2011 rankings were completed and published.

Sure, we were surprised and disappointed that the law school’s 2011 ranking was calculated using an at-graduation employment figure nearly 30 points lower than the actual number. But we understand the error was unintentional and believe there is little value in pursuing this further.