Paul L. Caron

Friday, May 21, 2010

U.S. News Responds to TaxProf Blog Post by Preventing Law Schools From Gaming Employed at Graduation Data

US News U.S. News & World Report has responded to my post last week, Did 16 Law Schools Commit Rankings Malpractice?, in U.S. News Takes Steps to Stop Law Schools From Manipulating the Rankings:

The problem with being totally transparent about key methodology details about U.S. News's America's Best Law Schools Rankings is that it's clear—based on our own analysis of historical trends in the our law school rankings and recently published blog posts—that certain law schools are taking advantage of that knowledge to game our rankings.

U.S. News is going to take steps to prevent these data manipulations by law schools in future rankings. The post serves as notice of those changes to be explained below. ...

U.S. News has said that the percent of J.D. graduates employed at graduation and those employed nine months after graduation count for 4.0% and 14.0% of the overall rankings, respectively. U.S. News has publicly disclosed the formula used in its ranking model to estimate the employed-at-graduation figure, should a law school not report the percentage of graduates who are employed at graduation.

The formula has been: that law school's employed-at-nine-months percentage minus approximately 30 percentage points equals employed at graduation. For example, for a law school with a 90% rate of employment at nine months, its ranking would be computed using an estimate of 60%. U.S. News publishes these nonreporters's data as N/A on the law school ranking table.

Why is U.S. News making this estimate? In the past, we had been told by many in law school career services offices that some law schools didn't or couldn't keep track of the proportion of their J.D. graduates with jobs at graduation, that what really mattered was nine months out, and that it would not be fair to penalize law schools in the rankings if they didn't have the employed-at-graduation data.

The problem created by this openness about our ranking model is that it's clear that more law schools have decided whether to report their at graduation employment based on how their actual percentage will compare to the estimate U.S. News will make for them. For example, a law school knows that its actual at-graduation employment rate is 40%, but knows because of our transparency that based on its nine-month rate, U.S. News will estimate 60% for its at-graduation rate, that school has chosen not to report its actual numbers and instead lets U.S. News make the estimate. In other words: ranking gamesmanship.

In the latest edition of the America's Best Law Schools rankings, 74 law schools (39% of those that were ranked) did not report their at-graduation employment rate. This is nearly double the number of schools (38) that did not report in the 2005 edition. U.S. News believes that this increase proves that far more law schools do track their students at graduation and believe that virtually all law schools could be reporting vital job placement data and have chosen not to do so in order to game the rankings.

Paul Caron, associate dean at University of Cincinnati's Law School and the publisher and editor of the widely followed Tax Prof blog, recently wrote a piece titled Did 16 Law Schools Commit Rankings Malpractice? that documented the growing number of schools who are choosing not to report their at-graduation employment rates to U.S. News. Caron wondered why 16 law schools purposely reported lower at-graduation employment rates than the higher estimated rate that U.S. News would calculate for them, calling this "ranking malpractice." The ABA Journal also wrote a story about this: "Were 16 Law Schools Too Revealing in Disclosures to U.S. News?" U.S. News strongly believes that schools should report their at-graduation data and finds the suggestion that schools that honestly report data are doing something wrong is misguided.

U.S. News is planning to significantly change its estimate for the at-graduation rate employment for nonresponding schools in order to create an incentive for more law schools to report their actual at-graduation employment rate data. This new estimating procedure will not be released publicly before we publish the rankings.

See also Tom Bell (Chapman), U.S. News: Less Transparency = More Fairness:

I understand that U.S. News generated the formula it formerly used to estimate the Emp9 figure for non-reporting schools by running a regression comparing the Emp0 and Emp9 data from reporting schools. It used to puzzle me that U.S. News did not evidently re-run the regression each year, but rather stuck with the original estimate. In retrospect, though, I see that sticking to the same formula might have partially helped U.S. News offset the gaming it so dislikes. After all, as more and more schools with low numbers refused to report Emp9 data, opting to rely instead on the publicized formula, the correlation between Emp0 and Emp9 scores would change so as to favor non-reporting schools. Better to stick with the old formula, dated though it might be, than to increase the incentive to opt out of reporting.

U.S. News thus avoided a vicious cycle, but only at the cost of signaling to schools exactly when hiding Emp9 data would help their rankings. Will its new reticence work? Schools can now only guess at how U.S. News will turn Emp0 numbers into Emp9 estimates, and will rightly worry that they might misjudge the new cutoff. Even if big-E ethics does not counsel reporting Emp9 numbers, therefore, small-c conservatism will. Granted, a school might reason, "U.S. News will still try to find a reasonably accurate way to turn Emp0 data into Emp9 estimates, and it has always helped us to not report in the past, so it remains a gamble worth taking." But such schools should also rightly worry that U.S. News might throw a punitive little kick into its new formula, to encourage schools to worry more about accuracy than about rankings.


Law School Rankings, Legal Education | Permalink

TrackBack URL for this entry:

Listed below are links to weblogs that reference U.S. News Responds to TaxProf Blog Post by Preventing Law Schools From Gaming Employed at Graduation Data:


wow good work! The employed @ graduation number is probably the single most important stat for non T-10 law schools (which all have virtually 99% emloyed at grad).

Posted by: anon | May 21, 2010 7:18:59 AM

While this is positive news, it still doesn't solve the overall worthlessness of the USN&WR rankings in general. Why do I say 'worthless'? Let's count the reasons:

1. Use of the subjective judgment of others. Allowing professors & deans (and only a small number of the total at that) to rank the competition invites gamesmanship. Same with allowing a (very) small number of attorneys to rank their perception of law schools. I know of one attorney who, with several classmates, formed an informal group called NOPE (Not One Penny Ever), whose purpose was to never donate a dime to their alma mater. How do you think they'd rank the school, if asked?
2. Use of self-provided employment data. Ugh. The article points out but one problem with self-provided data. Another problem? What does 'employed' mean? Work for BigLaw? Employed. Flipping burgers at McDonalds? Uh...employed. What really needs to be ranked is how many grads were working in the legal field as attorneys, and how did their salary compare to what they paid for their tuition? For example, if 45% of grads paid $33,000.00 a year for tuition, but managed to only earn $45,000 upon graduation, wouldn't you want to know that?
3. Failure to account for outside factors. The world changes rapidly; the rankings do not. By their nature, they discourage change (how else can you compare, year over year?). The problem is that fewer students use hardcover books for research, and more online access. So over the long run, a school's library will become less and less important, while electronic access to data will become more important. That's just one issue of many

I could go on, and others have. There have been whole papers written on the problems of the ratings, yet people continue to use them - probably because no one has been able to penetrate the market with something better. Too bad, because it's sorely needed.

Posted by: TaxHacker | May 21, 2010 7:39:01 AM

Great Job! Now get US News to mandate providing employment stats for jobs that require a JD. It would be nice for them to request clerkship data in a way that defines what qualifies as an Article III clerkship. So many schools lie about clerkships and average starting salarys. It's really sick.

Baby steps are good, but not enough.

Posted by: Michigan '13 | May 21, 2010 7:53:18 AM