Monday, September 4, 2017
Following up on last month's post, The LSAT's 'Noteworthy' Correlation To Bar Passage: The National Jurist op-ed: A Little Bit of Data Can Be a Lot Dangerous, by Aaron Taylor (St. Louis; Executive Director, AccessLex Center for Legal Education Excellence):
I recently penned a commentary titled, For Diversity: Let’s Talk Less About Pipelines and More About Why Blacks Are Not Admitted. In it, I argue that the law school admission process disproportionately excludes Black people from legal education for reasons that are unsupported by relevant data. In making my point, I referenced research conducted by a few law schools showing the measurable, but limited, value of the LSAT in predicting bar exam performance and the acquisition of lawyering skills.
I received many responses to my inbox. Some favorable. Some not. I am writing though to address a public rebuttal by Robert Steinbuch, a law professor at Arkansas-Little Rock. Steinbuch challenged my thesis by presenting a data table showing that graduates of his law school who entered with lower LSAT scores passed the bar exam at lower rates.
For Steinbuch, the trends seem to be definitive proof that the LSAT does indeed predict bar exam performance in ways that justify its outsized role in the admissions process.
Steinbuch commits a common, but dangerous error of interpretation. He assumes that a linear association between LSAT scores and bar performance reflects a predictive or impactful relationship. This is a classic conflation of correlation and causation – an error that can lead to conclusions that are unsupported and often erroneous.
Correlations only explain the extent to which two variables flow together (or diverge). They explain nothing about the impact of one variable on the other. To a knowledgeable observer, Steinbuch’s data table prompts more questions than answers – most significantly, why are lower LSAT scores associated with higher bar exam failure? A simplistic analysis of trends is insufficient in answering this question (and others), and any suggestion to the contrary is simply wrong. ...
Better information about bar performance could help schools better assess failure risks and design more effective bar passage interventions. ... Filling the void is conventional wisdom that is often rooted in unsupported intuition and cringe-worthy misinterpretations of data. A little bit of data can be a lot dangerous. We need more and better.