Paul L. Caron
Dean




Wednesday, March 11, 2015

Dean Advises Other Deans to Punish Law Schools in U.S. News Peer Reputation Voting for Manipulating Rankings With School-Funded Jobs and Transfer Students

2016 U.S. News RankingsFollowing up on my previous posts (here and here) on the 2016 U.S. News Law School Rankings:  Law Deans on Legal Education Blog:  Using USNWR to Impose Reputational Costs, by Rick Bales (Dean, Ohio Northern):

Much as I despise the USNWR ranking system, I’m a bit surprised that we deans (and associate deans) don’t use our relatively outsized influence as voters in the peer-assessment component of the ranking to police our own ranks.

When a school subsidizes the employment of large numbers of graduates for nine months and a day after graduation, or a quarter of its second-year class is comprised of students who it rejected for admission as first-year students, it’s obvious that the school is playing games to artificially inflate its employment outcomes and student selectivity and ultimately its USNWR ranking. In my mind, this is unethical, because it actively seeks to mislead consumers (prospective students and employers of current students) who may not understand the numbers-manipulation that is occurring behind the curtain. It’s also, I believe, a sure sign of structural weakness – if a school has to play games to maintain its employment statistics or entering-student credentials, the school is masking significant underlying problems. ...

I am not, by any stretch, a fan of the USNWR rankings. Many law deans boycott the annual survey, or at least say they do, for good reason. But for those who do fill the survey out, perhaps it makes sense next year to consider using the survey to significantly penalize the law schools who are manipulating their numbers and misleading consumers. Forcing schools to pay a price for their misbehavior seems the best way to stop it.

https://taxprof.typepad.com/taxprof_blog/2015/03/dean-advises-other-deans-to-punish-law-schools-.html

Law School Rankings, Legal Education | Permalink

Comments

You write truth, Professor!

Posted by: Mike Petrik | Mar 17, 2015 10:33:07 AM

If anyone remains agnostic as to the objectivity and reliability of the U.S. News rankings, consider this. As usual, the latest list shows little movement among the “elite” law schools; these relative positions remained mostly stable from last year’s report. But beneath the top 50, there were some remarkable changes in overall score, in just one year. Why? Do these precipitous leaps and crashes accurate reflect what the rankings purport to represent, or something very different?

As the National Law Journal has reported, “Thirteen saw their rankings change by 20 or more spots—up from eight last year. Forty law schools moved 10 or more spots, up from 37 last year.” With only schools ranked 149 and above given an actual ranking, roughly 25% of currently accredited law schools are arbitrarily lumped together, unranked, in the U.S. News version of a Walmart clearance bin. This means that 13 schools of the supposedly top 150 moved above or below 20 of their competitors—in a single year. How is this possible in any legitimate, objectively-derived way?

As the recent controversy over “predatory poaching” illustrates, some law schools aggressively structure their initial admissions criteria and their transfer-student criteria to maximize both their U.S. news score and their surplus of revenues over expenses. It is human nature for self-interested people to manipulate that which can be manipulated to maintain or enhance their own advantage. Meanwhile, there is a perverse incentive to ignore or devote less attention to those aspects of the law school experience that are minimized or entirely excluded from the U.S. News numbers. Racial, ethnic, and gender diversity? Zero influence on the rankings. Community service, clinical opportunities, and outreach? Again, zero. Programs to benefit the students, including underserved and nontraditional students, such as academic support and bar exam preparation? Yes, zero.

The U.S. News chart has the appearance of mathematical precision without the actuality thereof. The numbers look so scientific, so definite. There are no margins of error or measures of probability mentioned. And with 12 factors considered, each with a weight calculated down to the hundredth of 1 percent, how can the rankings be less than reliable and rigorous? The truth is that the rankings convey the illusion of validity, camouflaging the immense subjectivity, bias, imprecision, and manipulability of the underlying input. I am tempted to dredge up the cliché “garbage in, garbage out,” but that would be a disservice to municipal household waste. At least some of that has actual value in composting and recycling.

Posted by: Prof. John C. Kunich | Mar 15, 2015 4:40:27 AM

To build on my previous comments, I think the “methodology” employed to concoct the annual U.S. News law school rankings is as flawed as any widely-used assessment in any field. The well-known innumeracy of many people, including bright, well-educated individuals, produces a large and unwarranted aura of reliability for a system like the law school rankings that purport to digest multifarious complex factors into a single number. The fact that U.S. uses a weighted mix of “12 measures of quality” to determine each school’s numerical score is all that many observers will care to know. Like Colonel Sanders’ famous “blend of 11 herbs and spices,” this concatenation of specially-chosen factors is assumed to be a guarantor of excellence.

The fact that the “assessment scores” obtained by surveying some unnamed number of legal academics and lawyers/judges combine for 40% of a school’s overall score should be cause for concern by anyone who considers the underlying validity of the rankings. Without precautions regarding sample size, ballot-box stuffing, collusion, insider bias, and subjective guessing, these “assessment scores” are as enormously imprecise, manipulable, and vulnerable to prejudice as any popularity contest. With only a reported 58% response rate among the legal academics surveyed, one must wonder how many people were invited to submit their opinions, how these invitees were distributed among all law schools, to what extent respondents were not evenly distributed, and what influenced that 58% to turn in their views.

An entire book could be written about the defects in the ranking methodology, but suffice it to say that the factors chosen, and the relative weight assigned to each factor, are extremely arbitrary and prone to game-playing. For example, why is “selectivity” given a weight of 25% of the total, while bar passage rate receives a weight of 2% and faculty-student ratio has a 3% weight? How was this comparative importance determined? Who decided, and on what basis, that a law school’s selectivity is more than 12 times as significant as its bar pass rate? And why should the possibly uninformed and unsubstantiated opinion of 58% of legal academics who were somehow invited to participate in the survey count for more than 8 times as much as a school’s actual results in the bar exam?

By what scientific or pseudo-scientific means was it determined that the median LSAT scores of entering 1Ls is worth 15% of a school’s total value, the median UGPA or entering 1Ls counts for 10%, and the LSAT/UGPA of any 2Ls, 3Ls, and actual graduates do not matter at all? What rigorous methodology produced the decision to count library resources as a minuscule ¾ of 1%, while diversity of students, diversity of faculty, experiential learning, academic support, community service, and innovative pedagogy do not even merit a fraction of 1%? The guesses, unsubstantiated opinions, rumor-fueled prejudices, and insider-trading mutual back-scratching of the fraction of invitees who submit opinions for the two “assessment scores” crush the scales at a combined 40% of the overall value, while these meaningful and impactful measures of a school’s merit are entirely ignored. Is it any wonder that the rankings largely replicate a certain form of hierarchy and privilege, year after predictable year?

Posted by: Prof. John C. Kunich | Mar 14, 2015 5:49:27 AM

The much-anticipated/dreaded annual ranking of law schools has been unleashed by U.S. News to the usual howls of outrage mingled with squeals of glee. There are few, if any, other lists that are simultaneously so heavily used and so widely criticized. The disproportionately immense influence these rankings exert on a law school’s volume of applications, applicants’ credentials, job placement results, and student retention, must be considered in light of the many assumptions, biases, and inaccuracies that taint the numbers.

Any metric that purports to assess the relative merit of institutions of higher education should be subject to the highest possible means of maximizing objectivity, transparency, proper focus, and freedom from manipulation. It is a poorly kept secret among law professors and deans that the U.S. News rankings are deeply flawed in every one of these criteria. This allows some law schools to game the system and exploit their inflated rank to their own advantage, while many others are thrown into a desperate whirlpool of negative publicity and unfair preconceptions.

This matters in several ways, but most notably in the distortion of alternative analysis for potential law students contemplating the law schools to which they may apply, and for rising 2Ls considering whether to transfer to a “better” school for the remainder of their law school career. Each year, thousands of such decisions are made. To the extent law school rankings are an important factor in shaping these decisions, it is imperative that these rankings be reformed to correct the multiple profound deficiencies that render the results both misleading and dangerous.

Posted by: Prof. John C. Kunich | Mar 13, 2015 1:26:12 PM

@Orin,

Jojo is the one who brought up the Sherman Act, not me. I'm just trying to foresee any possible troubles that might stem from law deans essentially colluding to punish law schools that inflate their employment metrics. Like most licensed attorneys with no to next-to-no actual legal experience, I would be as confident in applying the Sherman Act as I would be applying string theory. That's why you don't see me making legal pronouncements very often. Actually, that's not fair. I've read more books about string theory since the bar exam than I have books on law.

Posted by: Unemployed Northeastern | Mar 12, 2015 10:09:48 PM

Jojo, interesting theory, but do the cases go that far?

UN, I read the article when the draft came out, and I found it unpersuasive on the law (for reasons I shared with the authors at the time). Even if it's persuasive, I'm not sure what it has to do with the Sherman Act.

Posted by: Orin Kerr | Mar 12, 2015 9:22:44 PM

Wow! This is a horrible idea. Sure some schools are trying to game the rankings, but a lot of schools had bridge-to-practice/fellowship programs before USNWR and the ABA decided to credit these programs as providing FTLT employment. The fault lies with USNWR and the ABA for counting these programs as FTLT jobs. Not the schools who use these programs or the graduates that benefit from them.

Could the legal academy care a little more about student welfare and a little less about these useless rankings?

Posted by: Nathan A. | Mar 12, 2015 10:44:44 AM

Orin,

We’re not talking about law deans getting together to fill out the Cosmo survey of whether your man is a hunk or a lunk. We’re talking about using an influential ranking system (screwy as that might be) for a coordinated economic purpose to punish competitors who engage in economic competition by “stealing transfers” or providing students with jobs. I could see an AUSA taking that case from DOJ Antitrust and making it stick. To use the econ lingo, it’s a solicitation to conspire to stifle competition in the market for tuition dollars and post-law school jobs. It’s bad, and pretty bold faced. At this point it’s just huffing and puffing sour grapes by a school that’s getting hammered by competition, but that’s why we have the Sherman Act in the first place. Competition is like exercise; everyone agrees that it’s a good thing for other people to endure. It’s harder to endure it yourself.

Posted by: Jojo | Mar 12, 2015 9:42:11 AM

@Orin,

Emory Law professors George Cloud and Morgan Shepherd's paper "Law Deans in Jail" may (or may not; I'm not rereading it this morning) give some guidance into why it would be poor form for deans to conspire to *punish* this law school or that by giving them low prestige-o-meter scores. Then again, an ex-admissions official at Clemson admitted in a speech some years ago that one of that school's methods for climbing US News was to rate itself a 5.0 and everyone else a 1.0 on the prestige-o-meter, and I don't believe anything punitive came of it.

Posted by: Unemployed Northeastern | Mar 12, 2015 7:06:03 AM

The main reason deans don't like USNews is not because it isn't accurate, but rather because it is

Posted by: mike livingston | Mar 12, 2015 4:19:03 AM

Jojo, I don't know anything about antitrust law, but does the Sherman Act apply to filling out evaluations for a magazine?

Posted by: Orin Kerr | Mar 11, 2015 4:40:10 PM

A Game of Deans! Has anyone pitched HBO?

Posted by: Mike Spivey | Mar 11, 2015 1:38:36 PM

Section 1 of Sherman Act and the Law Schools: A New Heuristic for an Old Problem

"Every contract, combination in the form of trust or otherwise, or conspiracy, in restraint of trade or commerce among the several States, or with foreign nations, is declared to be illegal. Every person who shall make any contract or engage in any combination or conspiracy hereby declared to be illegal shall be deemed guilty of a felony, and, on conviction thereof, shall be punished by fine not exceeding $100,000,000 if a corporation, or, if any other person, $1,000,000, or by imprisonment not exceeding 10 years, or by both said punishments, in the discretion of the court."

Posted by: Jojo | Mar 11, 2015 10:08:46 AM