Paul L. Caron

Thursday, May 19, 2016

LSAT Is Poor Predictor Of Law School Grades: 6 LSAT Points = 0.1 LGPA

LSAT (2015)Alexia Brunet Marks (Colorado) & Scott A. Moss (Colorado), What Predicts Law Student Success? A Longitudinal Study Correlating Law Student Applicant Data and Law School Outcomes, 13 J. Empirical Legal Stud. 205 (2016):

Despite the rise of "big data" empiricism, law school admission remains heavily impressionistic; admission decisions based on anecdotes about recent students, idiosyncratic preferences for certain majors or jobs, or mainly the Law School Admission Test (LSAT). Yet no predictors are well-validated; studies of the LSAT or other factors fail to control for college quality, major, work experience, etc. The lack of evidence of what actually predicts law school success is especially surprising after the 2010s downturn left schools competing for fewer applicants and left potential students less sure of law school as a path to future success. We aim to fill this gap with a two-school, 1400-student, 2005-2012 longitudinal study. After coding non-digitized applicant data, we used multivariate regression analysis to predict law school grades ("LGPA") from many variables: LSAT; college grades ("UGPA"), quality, and major; UGPA trajectory; employment duration and type (legal, scientific, military, teaching, etc.); college leadership; prior graduate degree; criminal or discipline record; and variable interactions (e.g., high-LSAT/low-UGPA or vice-versa).

Our results include not only new findings about how to balance LSAT and UGPA, but the first findings that college quality, major, work experience, and other traits are significant predictors: (1) controlling for other variables, LSAT predicts more weakly, and UGPA more powerfully, than commonly assumed – and a high-LSAT/low-UGPA profile may predict worse than the opposite; (2) a STEM (science, technology, engineering, math) or EAF (economics, accounting, finance) major is a significant plus, akin to 3½-4 extra LSAT points; (3) several years' work experience is a significant plus, with teaching especially positive and military the weakest; (4) a criminal or disciplinary record is a significant minus, akin to 7½ fewer LSAT points; and (5) long-noted gender disparities seem to have abated, but racial disparities persist. Some predictors were interestingly nonlinear: college quality has decreasing returns; UGPA has increasing returns; a rising UGPA is a plus only for law students right out of college; and 4-9 years of work is a "sweet spot," with neither 1-3 or 10 years' work experience significant. Some, such as those with military or science work, have high LGPA variance, indicating a mix of high and low performers requiring close scrutiny. Many traditionally valued traits had no predictive value: typical pre-law majors (political science, history, etc.); legal or public sector work; or college leadership.

These findings can help identify who can outperform overvalued predictors like the LSAT. A key caveat is that statistical models cannot capture certain difficult-to-code key traits: some who project to have weak grades retain appealing lawyering or leadership potential; and many will over- or under-perform any projection. Thus, admissions will always be both art and science – but perhaps with a bit more science.

Wall Street Journal Law Blog, New Study Tries to Predict Law School Grades:

Law school admission test scores are an “overvalued predictor” of law school grades,according to a new study. ...

[T]he magnitude of the predictive power of LSAT is modest compared to how heavily schools weight LSAT scores. A 6-point LSAT difference is enough to make a dispositive difference in where one attends law school and whether one receives a six-figure scholarship – but even that large an LSAT gap really predicts only a modest 0.1 difference in LGPA…

[C]hanges in LSAT do not appear to have increasing or decreasing returns; an X-point difference between a low and very low LSAT predicts the same as an X-point difference between a high and very high LSAT. Thus, contrary to some common assumptions, a “cutoff” driven by fear of an especially low LSAT is unsound: the difference between a 147 and a 152 is the same as the difference between a 157 and a 162.

Legal Education, Scholarship | Permalink


It's hard to know what to make of this study from the abstract (and because it's Wiley I can't access a full copy through my current subscriptions). What two schools we are looking at in terms of LSAT spread and grading practices seem to be pretty important to assessing the usefulness of the data.

For example, even if there aren't any other problems, the results get a lot less useful if the 1400 students all have LSAT ranges above the "in danger of failing the bar" cut off. If that's the case, the only take-aways from the study are that US News should rely less heavily on LSAT (which isn't news) and that LSAT is a weak predictor of how successful---in terms of law school grades---students who are already likely to be capable of graduating and passing the bar are likely to be (also not really news).

If, on the other hand, there is a reasonable sample size of students below the 150 or so LSAT mark and the results are the same for that cohort as the general cohort (neither of which is revealed by the abstract), then the study may be a really useful tool for admissions practices at lower-ranked schools.

Does anyone with access to the article know whether it is the former or latter case?

Posted by: Former Editor | May 20, 2016 4:28:52 AM

Coordinated attacks on the LSAT, and the fools at LSAC didn't even see it coming. Next attack is the bar exam. Law schools have no shame; they should have no money. Never have I seen an industry with so many dissatisfied customers.

Posted by: Jojo | May 20, 2016 3:45:14 AM

The LSAT is the one common experience all law applicants have. Their other experiences vary incredibly - major, rigor of particular schools and programs, personal qualities, etc. The LSAT score does to some extent predict academic success in law school and the bar exam. Or is it just a coincidence that the schools with the highest median LSATs also have the best bar passage stats? I think using LSAT cutoff scores is a good idea. Yes there are a few outliers who score badly, but their personal statements should be good enough to demonstrate they are among the few whose academic abilities are not well-measured by standardized testing, and whose other qualities, such as personal experiences, indicate high potential as a lawyer.

Posted by: Old Ruster from JDJunkyard | May 19, 2016 4:17:19 PM

Two caveats:

- Nothing correlates to USNWR rank so much as LSAT scores, so don't count on law schools abandoning their primacy anytime soon.

- As prior studies have shown, below a certain LSAT threshold, one is very unlikely to pass a bar exam.

Posted by: Unemployed Northeastern | May 19, 2016 1:24:16 PM