Friday, May 21, 2010
U.S. News Responds to TaxProf Blog Post by Preventing Law Schools From Gaming Employed at Graduation Data
The problem with being totally transparent about key methodology details about U.S. News's America's Best Law Schools Rankings is that it's clear—based on our own analysis of historical trends in the our law school rankings and recently published blog posts—that certain law schools are taking advantage of that knowledge to game our rankings.
U.S. News is going to take steps to prevent these data manipulations by law schools in future rankings. The post serves as notice of those changes to be explained below. ...
U.S. News has said that the percent of J.D. graduates employed at graduation and those employed nine months after graduation count for 4.0% and 14.0% of the overall rankings, respectively. U.S. News has publicly disclosed the formula used in its ranking model to estimate the employed-at-graduation figure, should a law school not report the percentage of graduates who are employed at graduation.
The formula has been: that law school's employed-at-nine-months percentage minus approximately 30 percentage points equals employed at graduation. For example, for a law school with a 90% rate of employment at nine months, its ranking would be computed using an estimate of 60%. U.S. News publishes these nonreporters's data as N/A on the law school ranking table.
Why is U.S. News making this estimate? In the past, we had been told by many in law school career services offices that some law schools didn't or couldn't keep track of the proportion of their J.D. graduates with jobs at graduation, that what really mattered was nine months out, and that it would not be fair to penalize law schools in the rankings if they didn't have the employed-at-graduation data.
The problem created by this openness about our ranking model is that it's clear that more law schools have decided whether to report their at graduation employment based on how their actual percentage will compare to the estimate U.S. News will make for them. For example, a law school knows that its actual at-graduation employment rate is 40%, but knows because of our transparency that based on its nine-month rate, U.S. News will estimate 60% for its at-graduation rate, that school has chosen not to report its actual numbers and instead lets U.S. News make the estimate. In other words: ranking gamesmanship.
In the latest edition of the America's Best Law Schools rankings, 74 law schools (39% of those that were ranked) did not report their at-graduation employment rate. This is nearly double the number of schools (38) that did not report in the 2005 edition. U.S. News believes that this increase proves that far more law schools do track their students at graduation and believe that virtually all law schools could be reporting vital job placement data and have chosen not to do so in order to game the rankings.
Paul Caron, associate dean at University of Cincinnati's Law School and the publisher and editor of the widely followed Tax Prof blog, recently wrote a piece titled Did 16 Law Schools Commit Rankings Malpractice? that documented the growing number of schools who are choosing not to report their at-graduation employment rates to U.S. News. Caron wondered why 16 law schools purposely reported lower at-graduation employment rates than the higher estimated rate that U.S. News would calculate for them, calling this "ranking malpractice." The ABA Journal also wrote a story about this: "Were 16 Law Schools Too Revealing in Disclosures to U.S. News?" U.S. News strongly believes that schools should report their at-graduation data and finds the suggestion that schools that honestly report data are doing something wrong is misguided.
U.S. News is planning to significantly change its estimate for the at-graduation rate employment for nonresponding schools in order to create an incentive for more law schools to report their actual at-graduation employment rate data. This new estimating procedure will not be released publicly before we publish the rankings.
I understand that U.S. News generated the formula it formerly used to estimate the Emp9 figure for non-reporting schools by running a regression comparing the Emp0 and Emp9 data from reporting schools. It used to puzzle me that U.S. News did not evidently re-run the regression each year, but rather stuck with the original estimate. In retrospect, though, I see that sticking to the same formula might have partially helped U.S. News offset the gaming it so dislikes. After all, as more and more schools with low numbers refused to report Emp9 data, opting to rely instead on the publicized formula, the correlation between Emp0 and Emp9 scores would change so as to favor non-reporting schools. Better to stick with the old formula, dated though it might be, than to increase the incentive to opt out of reporting.
U.S. News thus avoided a vicious cycle, but only at the cost of signaling to schools exactly when hiding Emp9 data would help their rankings. Will its new reticence work? Schools can now only guess at how U.S. News will turn Emp0 numbers into Emp9 estimates, and will rightly worry that they might misjudge the new cutoff. Even if big-E ethics does not counsel reporting Emp9 numbers, therefore, small-c conservatism will. Granted, a school might reason, "U.S. News will still try to find a reasonably accurate way to turn Emp0 data into Emp9 estimates, and it has always helped us to not report in the past, so it remains a gamble worth taking." But such schools should also rightly worry that U.S. News might throw a punitive little kick into its new formula, to encourage schools to worry more about accuracy than about rankings.