TaxProf Blog

Editor: Paul L. Caron
Pepperdine University School of Law

A Member of the Law Professor Blogs Network

Monday, May 4, 2009

Did 23 Law Schools Commit Rankings Malpractice?

U.S. News Several readers noted the curious fact that 64 law schools did not supply U.S. News with the percentage of its Class of 2007 that was employed at graduation; this component counts 4% in the methodology used in the 2010 Law School Rankings.  One commenter explained:

According to U.S. News, when a school does not report this number, 30% is subtracted from the nine month employment number (i.e., for a school that reports a 95% employed at 9 months number, U.S. News will plug the graduation number with 65%).

Pepperdine [led] the way on this scheme in recent years, and it looks like a lot of schools have jumped on the band wagon, which presumably means that for schools who reported this figure last year, but are "N/A" this year, the real number was more than 30% lower than their 9 month number, so they chose not to report the at graduation number (by the way, US News has to use a plug because this is one data area that the ABA doesn't collect, so US News can't obtain it if the school doesn't report it).

See also Theodore Seto, Understanding the U.S. News Law School Rankings, 60 SMU L. Rev. 493, 500 (2007).

A more interesting question is why 23 law schools reported employed at graduation numbers more than 30% lower than their employed at nine months number:

Employed at

Graduation

Employed at

9 Months

 

School

 

Difference

Overall

Rank

44.2%

98.0%

Arkansas-Little Rock

53.8%

Tier 3

45.4%

96.5%

William Mitchell

51.1%

Tier 3

43.5%

94.1%

Northern Illinois

50.6%

Tier 4

47.2%

92.4%

Campbell

45.2%

Tier 4

52.9%

97.9%

Missouri-Columbia

45.0%

65

54.3%

97.9%

Louisville

43.6%

98

55.9%

96.7%

Mississippi College

40.8%

Tier 4

56.1%

96.0%

Mercer

39.9%

Tier 3

45.2%

85.0%

New England

39.8%

Tier 4

56.1%

95.6%

Samford

39.5%

Tier 3

56.7%

95.7%

Whittier

39.0%

Tier 4

50.9%

89.6%

Tulsa

38.7%

Tier 4

56.3%

93.4%

Valparaiso

37.1%

Tier 4

58.0%

94.9%

Oregon

36.9%

77

62.4%

99.2%

New Mexico

36.8%

77

60.5%

95.8%

Loyola-New Orleans

35.3%

Tier 3

63.9%

98.6%

South Dakota

34.7%

Tier 3

61.2%

95.2%

South Carolina

34.0%

87

60.9%

94.6%

Temple

33.7%

65

63.9%

96.9%

Oklahoma

33.0%

71

61.2%

93.3%

Baylor

32.1%

65

67.9%

99.7%

Seattle

31.8%

77

57.5%

88.9%

U. Mississippi

31.4%

Tier 3

Many of these schools undoubtedly adversely affected their overall ranking by reporting their employed at graduation data.  As Ted Seto explains in his article, because U.S. News uses round numbers in determining a school's overall score, the 4% weighting of the employed at graduation data easily could impact a school's overall ranking -- i.e. a school whose overall score ended in .49 would move up to the next grouping with an increase in its overall score of merely .01.  Ted also explains that an increase of 22 percentage points in the employed at graduation figure would have improved a school's overall score by one full point (in the 2007 rankings) -- which Arkansas-Little Rock could have achieved by declining to disclose its 44.2% figure and instead allowing U.S. News to assign it a 68.0% figure.

With the close clustering of schools ranked 65 (six schools), 71 (4 schools), 77 (8 schools), 87 (7 schools), and 98 (2 schools), Baylor, Louisville, Missouri-Columbia, New Mexico, Oklahoma, Oregon, Seattle, South Carolina, and Temple may have improved their rank within the Top 100 by declining to disclose their employed at graduation data.  Depending on their overall score, one or more of the schools in Tier 3 (Arkansas-Little Rock, Loyola-New Orleans, Mercer,U. Mississippi, Samford, South Dakota, William Mitchell) might have cracked the Top 100, and one or more of the schools in Tier 4 (Campbell, Mississippi College, New England, Northern Illinois, Tulsa, Valparaiso, Whittier) might have found themselves in Tier 3, by not disclosing their true employed at graduation data and instead allowing U.S. News to use its surrogate figure.

Update:  Thanks to an alert reader, I removed from the chart one school that was incorrectly listed as having an employed at graduation number more than 30% lower than its employed at nine months number.

http://taxprof.typepad.com/taxprof_blog/2009/05/rankings-malpractice.html

Law School Rankings, Legal Education | Permalink

TrackBack URL for this entry:

http://www.typepad.com/services/trackback/6a00d8341c4eab53ef011570684a43970b

Listed below are links to weblogs that reference Did 23 Law Schools Commit Rankings Malpractice?:

Comments

Why is the "employed at graduation" number important? A lot of employers wait and see who passed the bar before hiring, and those that don't often end up firing if someone doesn't pass. Should schools be able to claim someone is "employed at graduation" if 9 months later they are still unbarred?

Posted by: Ken Puckerson | May 4, 2009 7:10:03 AM

can i withhold info that is not entirely positive when applying to law school or the bar?

Posted by: redacted | May 4, 2009 9:54:27 AM

How exactly is this rankings malpractice if these schools were honest in reporting greater than a 30% difference?!?!?

Posted by: Josh | May 4, 2009 10:36:41 AM

With all due respect to your commentator, "Yeah Right," I don't think that USN&WR calculated substitute figures by simply subtracting 30% from a school's "employment at nine months" figures. I've not seen any official statement by USN&WR about what it does, though I believe Bob Morse has touched on the subject in presentations before law school professionals. From that, I gather that the formula changes from time to time, as it relies on regressions run on reported employment data.

Posted by: Tom W. Bell | May 4, 2009 11:30:49 AM

I know I emailed Prof. Caron about this already - but I feel this title is a misrepresentation of the situation.

Anyone conducting a cursory reading of this blog will see the title, scan to the 23 schools on the list, and assume they are the schools committing some sort of "malpractice". This is simply not the case. Being unaware that reporting *honest* numbers will hurt one's status in the USNWR rankings is not malpractice. Unsophisticated to the clear gaming the other 64 are doing, sure, but malpractice? No.

I maintain that this title is ill-conceived.

Posted by: Josh Newville | May 4, 2009 11:44:17 AM

Forget "rankings malpractice": How about consumer data fraud? All of these schools who claim the data is unavailable clearly are lying, and the definitive proof of this is simple: They all ask the question about the timinig of the job offer of the employed students as part of the NALP survey. Part III of the survey is where the question is asked.

http://www.washburnlaw.edu/career/nalpsurvey/

Posted by: JT | May 4, 2009 12:14:07 PM

As a student currently attending one of the schools that actually reported worse-than-30%-change, we could have benefitted because we always hang on the tipping point in our rankings. However, more questions would be asked if we went up this year, then back down the next because we failed to disclose information. The rankings are notoriously subjective, and sometimes necessarily so, but radical changes in one's ranking would highlight a school's disingenuousness, raising questions about what other corners they have have cut. Playing the numbers game may improve your rankings in the short term, but it does little to legitimize either you long-term standing in them, or the rankings as a whole.

Posted by: Tripp | May 4, 2009 12:45:29 PM

Am I getting this right. Honesty = malpractice?

Posted by: roscoe | May 4, 2009 4:54:58 PM

The lack of "professor" comments on this "professor" blog speaks volumes. Hope they can sleep well at night knowing they are leaving their graduates with 6 figure debt and knowledge of the Rule in Shelly's Case.

Posted by: david friedlander | May 4, 2009 7:41:49 PM

to Tom W. Bell

Robert Morse answered this question directly on his blog and explained what was done in last year's rankings. I assume the same methodology was followed this year, but who knows.

http://www.usnews.com/blogs/college-rankings-blog/2008/4/1/grad-rankings-share-your-feelings.html#73605

Posted by: Yeah Right | May 5, 2009 1:15:37 PM

The proper response by US News is to deter non-reports by simply setting the "assumption" to be the least common denominator - the "assumed value" will be the lowest reported difference of all other law schools in the country - in this case Arkansas-Little Rock at 53.8% - plus an additional percent. There's no reason that this information wouldn't be available to schools outside of gross incompetence, so the failure to report should never be excused. If you want transparency, you make obfuscation cost more than it gains.

While it may be "rankings malpractice" in a technical sense, if it were true it says far more negative about US News than it does these schools. But US News' foibles have been amply catalogued by folks more clever than me and exploited by law school administrators less clever.

Posted by: *Eyeroll* | May 8, 2009 4:12:54 AM

A more interesting question is why 23 law schools reported employed at graduation numbers more than 30% lower than their employed at nine months number:

A less interesting answer is: Honesty.

Posted by: Deyseeme T. Rollin | May 8, 2009 6:34:53 AM

I can't believe a law professor is trying to claim that telling the truth is malpractice. What a stupid post. The only thing it points to is how easily manipulable the rankings are. Those schools should be commended for telling the truth and not gaming the system.

Posted by: Darran | May 8, 2009 7:10:40 AM

The "employed at graduation" statistic is a meaningless number for most law schools. At all but the top 12-25 schools, depending on what coast you're on (East Coast is higher), the vast majority of students will not be employed at graduation because there aren't huge firms with summer programs where they are, or because the big firms who have those programs won't look at the school.

It says nothing about the success of those students or what kind of lawyers come from those schools. Although, as always, if you want to practice the dying art of Big Law, and you didn't go to an East Coast school or Stanford, you're fighting an uphill battle anyway.

So I ask again, why does "employed at graduation" matter for any reason other than to continue to perpetuate the inaccuracies of the ranking system as it is by grandfathering in the "name" schools?

Posted by: Furry | May 8, 2009 7:42:13 AM

I agree that the headline has serious issues.

Personally, I'd like to the see the list of the schools that did game the system. (And make sure mine isn't on it.)

Posted by: CC | May 8, 2009 7:46:21 AM

I also interpreted this as these are the schools that are being honest. They accurately reported their numbers, even knowing had they withheld, they could be in a higher tier.

Posted by: Katy | May 8, 2009 7:48:45 AM

It's interesting that the majority of the schools named are state universities. Why were these schools more likely to report honest figures when they would have been better off remaining silent and taking advantage of the higher default figures?

Perhaps the powers that be at state schools are more likely to feel like they have a duty to the public to be honest.

Or perhaps state schools just have less financial incentive to misrepresent their graduate employment statistics -- unlike private universities for which law (and business) schools are essentially for-profit enterprises that subsidize everything else.

Posted by: Holy Prepuce | May 8, 2009 9:33:23 AM

I think the intent of the title is that disclosing the damaging data without being required to is malpractice in the sense that the schools failed to "zealously advocate" for their positions

Posted by: hmm | May 8, 2009 9:38:16 AM

The questions about "Employed at graduation" or "employed nine months after graduation" assume the graduate is employed in the legal field. However, the question actually asked is whether the grad is "employed" at the relevent time. So if a given law school's grads are 100 percent employed in the copy center, Mcdonald's and parking cars, and the school accurately and honestly reports them as "employed at graduation" or "employed nine months after graduation" then the school gets a perfect rating in this category. Beware of statitics and conclusions based on unstated assumptions.

Posted by: David | May 8, 2009 10:00:51 AM

It is disappointing that a law professor would think that gaming the system and reporting accurate information is "malpractice". What does that say about the quality of professionalism training that law students receive when their professors display this type of attitude and value system?

Posted by: Antoinette Sedillo Lopez | May 8, 2009 10:02:03 AM

Let's be truly honest - 'employed at graduation' and 'employed at nine months' are nebulous terms. Flipping burgers at McDonald's? Congrats! You're 'employed'!

My law school hounded me at the nine-month point just because they wanted to report 90% employed. The true statistic would be how many grads are employed as attorneys, not how many people have jobs.

In the end, the USN&WR rankings are worth little more than the paper they're printed on, but people continue to rely on them as if they're gold.

Posted by: ChicagoLawyer | May 8, 2009 10:42:37 AM

I absolutely agree! The title of this blog is very misleading!

Posted by: Emma | May 8, 2009 11:12:39 AM