TaxProf Blog

Editor: Paul L. Caron, Dean
Pepperdine University School of Law

Monday, September 4, 2017

Taylor: A Little Bit Of Data Can Be A Lot Dangerous When It Comes To LSATs And Bar Passage

Following up on last month's post, The LSAT's 'Noteworthy' Correlation To Bar Passage:  The National Jurist op-ed:  A Little Bit of Data Can Be a Lot Dangerous, by Aaron Taylor (St. Louis; Executive Director, AccessLex Center for Legal Education Excellence):

I recently penned a commentary titled, For Diversity: Let’s Talk Less About Pipelines and More About Why Blacks Are Not Admitted. In it, I argue that the law school admission process disproportionately excludes Black people from legal education for reasons that are unsupported by relevant data. In making my point, I referenced research conducted by a few law schools showing the measurable, but limited, value of the LSAT in predicting bar exam performance and the acquisition of lawyering skills.

I received many responses to my inbox. Some favorable. Some not. I am writing though to address a public rebuttal by Robert Steinbuch, a law professor at Arkansas-Little Rock. Steinbuch challenged my thesis by presenting a data table showing that graduates of his law school who entered with lower LSAT scores passed the bar exam at lower rates.


For Steinbuch, the trends seem to be definitive proof that the LSAT does indeed predict bar exam performance in ways that justify its outsized role in the admissions process.

Steinbuch commits a common, but dangerous error of interpretation. He assumes that a linear association between LSAT scores and bar performance reflects a predictive or impactful relationship. This is a classic conflation of correlation and causation – an error that can lead to conclusions that are unsupported and often erroneous.

Correlations only explain the extent to which two variables flow together (or diverge). They explain nothing about the impact of one variable on the other. To a knowledgeable observer, Steinbuch’s data table prompts more questions than answers – most significantly, why are lower LSAT scores associated with higher bar exam failure? A simplistic analysis of trends is insufficient in answering this question (and others), and any suggestion to the contrary is simply wrong. ...

Better information about bar performance could help schools better assess failure risks and design more effective bar passage interventions. ... Filling the void is conventional wisdom that is often rooted in unsupported intuition and cringe-worthy misinterpretations of data. A little bit of data can be a lot dangerous. We need more and better.

| Permalink


"[W]hy are lower LSAT scores associated with higher bar exam failure? A simplistic analysis of trends is insufficient in answering this question (and others), and any suggestion to the contrary is simply wrong. ..."

Perhaps because they are both tests? Its not causation (LSAT low scores doesnt cause low bar exam scores and failure), they are both symptoms of poor test taking ability. But spare me the "correlation is not causation therefore LSAT is irrelevant to bar pass" nonsense.

Posted by: Anon | Sep 4, 2017 8:33:09 AM

So.... if we are to disregard LSAT score when analyzing if a 0L can pass a bar exam, and we already have no problem admitting students with C average uGPA scores into law school (check the data for any of the lowest-rung school), and law schools still by and large can't be bothered interviewing any prospective students, then gosh oh golly how can ABA Standard 501b be enforced? Oh, right - law schools would prefer not to acknowledge 501b's existence, knowing that the feckless ABA can't be bothered to enforce it.

Posted by: Unemployed Northeastern | Sep 4, 2017 9:05:21 AM

"Bar Passage" elides - or is wholly ignorant - of accreditation standards requiring law schools to only accept students they believe can pass a bar exam. The overseer of accreditors has not missed this, however, and recommended that the ABA have its accrediting powers yanked last summer over this very issue (among others).

And sorry, but local banks look for business majors and MBAs, and tend to shy away from hiring people with $150k in nondischargeable debt. There is nothing a fresh law school grad that a bank's HR department wants. And of course all the usual caveats about that "peer-reviewed study" still apply, including but not limited to the data is mostly at least a decade ago and covers no graduates since the recession, law is undergoing structural changes that render past financial performance of law school grads (who owed far less money to boot) irrelevant, the paper has also been criticized by multiple attorneys with graduate quantitative degrees, not to mention a Rhodes Scholar-winning labor economist at the New York Times, etc., et. al.

Posted by: Unemployed Northeastern | Sep 4, 2017 4:29:05 PM

What a goofy claim. It doesn't take a brain surgeon to understand that Low Intelligence = Low LSAT, Low Intelligence = Low Bar Score, therefore Low LSAT = Low Bar Score. It's not 1:1, but it's right!

Posted by: anonymous | Sep 4, 2017 9:17:24 PM

Who ever suggested that a low LSAT score causes anyone to fail the bar exam? This is a wonderfully easy straw man to knock down, because the unstated assumption is so ridiculous.

Correlation does not prove causation - great. The correlation does, however, by definition, show that a law school applicant with a low LSAT score is less likely to pass a bar exam. That, by itself, is enough to justify use of LSAT scores both in admission decisions and in enforcing the ABA's rule 501b.

Also, in response to the first posted comment - please share what research you are relying on in reaching your conclusion that "Students do not need to pass the bar in order to obtain the professional and financial benefits of a JD." The one study you did cite focused on practicing attorneys that passed the bar exam and does not support your conclusion in any way. It appears that the only purpose of your post is to mislead poor performing students into spending a fortune on law school even though they have little chance of a successful law career.

Posted by: r | Sep 5, 2017 6:46:21 AM

I agree with Anon. The "classic conflation of correlation and causation" is a red-herring here. There is no causal link between low LSAT score and bar exam failure, and no-one is trying to prove there is such a thing. There is a strong correlation between the two, however, and any honest assessment of the data that is (finally!) now public shows that low LSAT scores and bar failure go hand-in-hand.
Perhaps the student has poor test-taking skills, as Anon suggests is the cause of both low LSATs and bar failure. Or perhaps the student is just not where he should be academically, for whatever reason.

Posted by: Old Ruster | Sep 5, 2017 8:46:32 AM

You all are assuming that Bar Passage is expressing his opinion. Sure reads to me like he's engaging in satire. God help us if he's on the level.

Posted by: PaulB | Sep 5, 2017 5:52:00 PM

High powered employers love hiring employees with lots of student loan debt, whether it's for an MBA, JD or MD. People with debt also have high quality educations, and are motivated to work hard.

Posted by: debt | Sep 5, 2017 11:39:16 PM

The comment by "Bar Passage" is an obvious "false flag" written by a passive aggressive scamblogger. It's meant to discredit the view it pretends to espouse by exaggerating it and making it sound arrogant, prideful, and out of touch.

The comment about "Ivy League educated" professors is particularly obnoxious and dripping with resentment and sarcasm.

The people who have published peer reviewed studies are well aware that there's a rich peer reviewed literature by many serious and talented researchers, most of whom are not law professors. And all of them have tended to conclude that education is typically a good investment over the long term and the benefits are not tied to a particular occupation or profession.

Posted by: Reality check | Sep 6, 2017 1:02:15 AM

@ Bar Passage

If you more closely read the peer reviewed literature you claim to be citing (but are apparently mocking), you'll see that the benefits of a law degree are estimated to be closer to around $350K to $400K toward the low end of the distribution (25th percentile). The $1,000,000 lifetime premium is an average. People who fail the bar exam in states where most people pass might typically get an even lower premiums than the 25th percentile, although this is not necessarily specifically because they failed the bar exam.

Posted by: Reality check | Sep 6, 2017 1:08:55 AM