May 12, 2010
Did 16 Law Schools Commit Rankings Malpractice?
As U.S. News rankings aficionados know, the methodology used in the 2011 U.S. News Law School Rankings gives 18% weight to employment statistics: 14% to the percentage of the Class of 2008 employed nine months after graduation (which is reported to the ABA as well), and 4% to the percentage of the class employed at graduation (which is not reported to the ABA).
74 schools did not supply U.S. News with the percentage of the class employed at graduation. This continues a ten-year trend -- the number of nonreporting schools has more than doubled over this period:
U.S. News has publicly disclosed that it estimates the employed at graduation figure for nonreporting schools with this formula:
Employed at Nine Months - ~30 percentage points = Employed at Graduation
As Ted Seto notes, "This was apparently intended to capture the relationship, on average, between the two variables for schools reporting both numbers." Understanding the U.S. News Law School Rankings, 60 SMU L. Rev. 493, 500 (2007).
The 74 nonreporting schools presumably had an employed at graduation number more than 30 percentage points below their employed at nine months number and thus benefited in the rankings by not reporting their employed at graduation number to U.S. News. Robert Morse, Director of Data Research at U.S. News, reports on his blog that Alabama "made errors reporting some of their data" -- they apparently "were too late" in reporting their employed at graduation number. Alabama's employed at nine months number was 96.9%, so U.S. News used 66.9% as Alabama's employed at graduation number in computing the rankings. Partially as a result, Alabama slid from #30 to #38 in the overall rankings. Alabama now says that its correct employed at graduation number was 92.1% (25.2 percentage points higher than the figure used by U.S. News).
A more interesting question is why 16 law schools (compared to 23 law schools last year) reported employed at graduation numbers more than 30 percentage points lower than their employed at nine months number:
John Marshall (Atl.)
Many of these schools undoubtedly adversely affected their overall ranking by reporting their employed at graduation data to U.S. News. As Ted Seto explains in his article, because U.S. News uses round numbers in determining a school's overall score, the 4% weighting of the employed at graduation data easily could impact a school's overall ranking -- i.e. a school whose overall score ended in .49 would move up to the next grouping with an increase in its overall score of merely .01. Ted also explains that an increase of 22 percentage points in the employed at graduation figure would have improved a school's overall score by one full point (in the 2007 rankings) -- which Arkansas-Little Rock and Memphis could have achieved by declining to disclose their 58.3% and 52.2% figures, respectively, and instead allowing U.S. News to assign them 65.8% and 63.3% figures.
With the close clustering of schools ranked 78 (two schools), 80 (6 schools), 86 (7 schools), 93 (5 schools), and 98 (4 schools), Missouri-Columbia, Oregon, Richmond, and Seattle may have improved their rank within the Top 100 by declining to disclose their employed at graduation data. Depending on their overall score, one or more of the schools in Tier 3 (Arkansas-Little Rock, Creighton, Loyola-New Orleans, Marquette, Memphis, Mississippi, South Carolina, South Dakota, Vermont, Wyoming) might have cracked the Top 100, and one or more of the schools in Tier 4 (John Marshall (Atlanta), Whittier) might have found themselves in Tier 3, by not disclosing their true employed at graduation data and instead allowing U.S. News to use its surrogate figure. Eight of these schools (Arkansas-Little Rock, Loyola-New Orleans, Missouri-Columbia, Oregon, Seattle, South Carolina, South Dakota, Whittier) also may have committed rankings malpractice last year by reporting their employed at graduation number to U.S. News.
Update: ABA Journal, Were 16 Law Schools Too Revealing in Disclosures to U.S. News?
Alabama responds on Bob Morse's blog:
Alabama Law completed and submitted the most recent U.S. News survey in October 2009. U.S. News sent back to us a verification questionnaire in December 2009. We updated and verified the data before submitting it back to U.S. News in January 2010. The fax confirmation shows January 14th, to be exact, which is three months before the 2011 rankings were completed and published.
Sure, we were surprised and disappointed that the law school’s 2011 ranking was calculated using an at-graduation employment figure nearly 30 points lower than the actual number. But we understand the error was unintentional and believe there is little value in pursuing this further.
TrackBack URL for this entry:
Listed below are links to weblogs that reference Did 16 Law Schools Commit Rankings Malpractice?:
You are faulting the schools for telling the truth?
Posted by: Anon | May 11, 2010 10:40:50 AM
Anything to improve enrollment
Posted by: William | May 11, 2010 11:21:10 AM
It;s a rock and hard place problem, I'd say. Does anyone believe that Alabama had a 90%+ employed at graduation number? Just looking at the numbers from other schools, which I assume to also be inflated, it seems unlikely. On the other hand, should all Schools be required to subtract the number of their own graduates they employ within one year of graduation. Those inclusion are a misrepresentation. It's very hard to worry too much about the malpractice when it may do little more than offset a small percentage of the sharp practice.
Posted by: Jeff | May 11, 2010 12:41:25 PM
Seems to me that its US News and World Reports that's committing the "ranking malpractice", not the law schools.
Posted by: Looking closely | May 11, 2010 1:20:40 PM
Law schools being deceptive? Given the product they produced I'm not very suprised.
Posted by: TBlakely | May 11, 2010 1:27:30 PM
And the truth shall set you 3.
Posted by: The Comedian | May 11, 2010 2:02:27 PM
You do realize that some law schools deliberately hire their own unemployed third year students -- to work in the alumni office, as research assistants to professors, etc. -- to boost this statistic, don't you?
Posted by: Lee | May 11, 2010 3:13:20 PM
I came here to post the same comment as Lee - I know of several law schools that hired unemployed 2009 graduates as support staff, and then encouraged faculty to hire even more unemployed 2009 graduates BEFORE February 15, 2010 (not so coincidentally, 9 months after their graduation) as research assistants at $10/hour. The USNWR law school rankings have had the unfortunate effect of the tail wagging the dog, in that the law school administration is more concerned with their "rank" and reporting these percentages to a magazine than they are in assisting their graduates obtain meaningful post-graduation employment.
Posted by: anon | May 11, 2010 8:17:20 PM
Until there is an independent audit of all school-supplied employment figures, the future will be identical to the past: massive, widespread fraud - *thought* internally to skirt the legal definition of fraud.
I wonder just how closely the schools have reviewed the wire fraud statutes.
Posted by: cas127 | May 11, 2010 9:35:26 PM
There's no way that over half of Oregon law grads were employed at graduation. The only other statistic that may be more laughable is the supposedly 90% employed 9-months after graduation. I would bet that it includes Starbucks, servers, and retail jobs. What a joke.
Posted by: Not Sure | May 12, 2010 12:17:26 AM
Employed at..? Waiting tables?
Posted by: cbinflux | May 12, 2010 3:07:54 AM
SO we all laugh in the blogosphere at the incompetence of the U.S. News rankings for being inaccurate representations of what is actually occurring, particularly with the employment stats.
And then we laugh at 16 schools who report their employment stats in a way that accurately reflects a lower number then they otherwise would get from the US news "system".
Got disconnect? Which is it: the rankings are bad because they're inaccurate, or schools are dumb for providing accurate data?
I dont think you can have this both ways.
Posted by: anon | May 12, 2010 9:00:37 AM
Loyola's one of the schools who didn't report this number. I don't know why it's so hard to get a simple listing of graduates, their employer (this is not secret information unless the person is working undercover for the CIA or something!) and their position from each school.
Posted by: anon | May 12, 2010 10:31:14 AM
This isn't news. Law school professors have been making deals with each for the longest time now, each giving each other 5's on the school evaluations US NEWS sends out to professors. I've heard it directly from a Professor's mouth. Just Sayin'
Posted by: Alex | May 12, 2010 12:58:35 PM
I HAVE NO DOUBT THAT LAW SCHOOL'S FUDGE THEIR NUMBERS.
At Western State University College of Law, they state that 90% of graduates are employed within 6 months of graduation.
What they fail to state is whether they are talking about the graduates from 1985, 1990, 2008, or 2009. They could have cherry picked that crap. Also, they never even mention if they are employed in a "law-related" position. Employed could have meant bagger at the grocery store.
The more I see stuff like this, the more I hate Western State University College of Law. Did you guys know that they fail out students in their graduating semester by giving them 0's in their class, making such students prolong their stay in law school?
Bottom ranked or not, the school can't treat people that way.
Posted by: 4chanLawyer | May 12, 2010 9:55:20 PM
Some enterprising reporter (or unemployed law grad) should do an expose on this. I don't ever recall my school asking me if I had a job when I graduated. I went to law school late in life and had a very part-time consulting business in my prior field, so someone may have simply considered me "employed".
One has only to read comments to a story like this (or similar ones in the ABA on-line Journal) to realize that here are a lot of bitter, angry graduates out there. (Of course, the people ones who landed jobs are probably too busy to supply comments....)
I think it's an open question whether law schools (like, e.g., Ph.D. programs in English) are a kind of glorified Ponzi scheme -- needing to take a lot of new people into "the system" to support the structure. I have no regrets, come what may, but my heart goes out to people who may not have gone into it with their eyes wide open -- and deceptive statistics certainly don't help the matter. But the schools aren't going to say, "Don't come, because you won't get a job"; they need students to pay tutition to pay faculty salaries....
Posted by: Anon | May 13, 2010 5:16:55 PM
I am the Dean of one of the law schools who committed "malpractice" by truthfully disclosing our at graduation employment figure. If the USNEWS ranking has any value, it is because applicants can look at a standard data set. But 1) schools who fail to report this number undermine even this shallow benefit and 2) the at graduation figure makes little sense in a "slow" market" like ours where many employers tend to hire after the summer bar results are announced. If this matters, then USNEWS should remove the incentive to game the system: if a school fails to report at grad employment numbers, USNEWS should show a 0% employment rate.
Posted by: John DiPippa | May 14, 2010 11:43:15 AM
None of the reported numbers in U.S. News are accurate. The entire system is based on the schools lying to the magazine, which in turns prints the lies, which causes unsuspecting naive prospective law students to take out huge loans, which in turn allows Wall St. to pool the student loans and sell derivatives based on those loan pools. Everyone wins except the student, who can't find a job. I graduated from a school ranked in the mid-60's in 2006. Even in that boom time, the only people who had a chance of getting a decent job were in the top 20%. That left 80% of us with nothing. However, the school and the magazine reported 98% employment 9-months out, with an average salary of $75,000. I personally know more than 2% of the class who were unemployed for over a year after graduation. I also personally know more than 20 students who didn't ever respond to the survey. The WSJ had a great article a few years back in which the Brooklyn Law School basically admitted that they totally fabricate the numbers. If the schools and the magazine told the truth, far fewer people would attend and the schools and banks would lose billions of dollars.
Posted by: Guest | May 14, 2010 9:45:11 PM
This is what's wrong with law schools and the legal profession: a law professor saying it is "malpractice" for a law school to withhold truthful information when they know that "assumed" information will be more favorable, although entirely inaccurate and misleading. Where are the morals and ethics in this approach? Is a favorable deception to be preferred over an unfavorable truth? So instead this law professor would rather thousands of prospective law students make decisions to attend law schools based on lies and deception of the schools themselves? I say, post the truth and the let the chips fall where they may. The legal profession is supposed to be about exposing the truth not covering it up with lies and deceptions so that we can "win our case" (in this case, the Ratings case). Shame shame.
Posted by: Guest | May 26, 2010 2:46:09 PM