Paul L. Caron

Thursday, October 20, 2016

LSAT Is Poor Predictor Of Law School Grades: 6 LSAT Points = 0.1 LGPA

LSAT (2015)Alexia Brunet Marks (Colorado) & Scott A. Moss (Colorado), What Makes a Law Student Succeed or Fail? A Longitudinal Study Correlating Law Student Applicant Data and Law School Outcomes, 13 J. Empirical Legal Stud. 205 (2016):

Despite the rise of "big data" empiricism, law school admission remains heavily impressionistic; admission decisions based on anecdotes about recent students, idiosyncratic preferences for certain majors or jobs, or mainly the Law School Admission Test (LSAT). Yet no predictors are well-validated; studies of the LSAT or other factors fail to control for college quality, major, work experience, etc. The lack of evidence of what actually predicts law school success is especially surprising after the 2010s downturn left schools competing for fewer applicants and left potential students less sure of law school as a path to future success. We aim to fill this gap with a two-school, 1400-student, 2005-2012 longitudinal study. After coding non-digitized applicant data, we used multivariate regression analysis to predict law school grades ("LGPA") from many variables: LSAT; college grades ("UGPA"), quality, and major; UGPA trajectory; employment duration and type (legal, scientific, military, teaching, etc.); college leadership; prior graduate degree; criminal or discipline record; and variable interactions (e.g., high-LSAT/low-UGPA or vice-versa).

Our results include not only new findings about how to balance LSAT and UGPA, but the first findings that college quality, major, work experience, and other traits are significant predictors: (1) controlling for other variables, LSAT predicts more weakly, and UGPA more powerfully, than commonly assumed – and a high-LSAT/low-UGPA profile may predict worse than the opposite; (2) a STEM (science, technology, engineering, math) or EAF (economics, accounting, finance) major is a significant plus, akin to 3½-4 extra LSAT points; (3) several years' work experience is a significant plus, with teaching especially positive and military the weakest; (4) a criminal or disciplinary record is a significant minus, akin to 7½ fewer LSAT points; and (5) long-noted gender disparities seem to have abated, but racial disparities persist. Some predictors were interestingly nonlinear: college quality has decreasing returns; UGPA has increasing returns; a rising UGPA is a plus only for law students right out of college; and 4-9 years of work is a "sweet spot," with neither 1-3 or 10 years' work experience significant. Some, such as those with military or science work, have high LGPA variance, indicating a mix of high and low performers requiring close scrutiny. Many traditionally valued traits had no predictive value: typical pre-law majors (political science, history, etc.); legal or public sector work; or college leadership.

These findings can help identify who can outperform overvalued predictors like the LSAT. A key caveat is that statistical models cannot capture certain difficult-to-code key traits: some who project to have weak grades retain appealing lawyering or leadership potential; and many will over- or under-perform any projection. Thus, admissions will always be both art and science – but perhaps with a bit more science.

Wall Street Journal Law Blog, New Study Tries to Predict Law School Grades:

Law school admission test scores are an “overvalued predictor” of law school grades,according to a new study. ...

[T]he magnitude of the predictive power of LSAT is modest compared to how heavily schools weight LSAT scores. A 6-point LSAT difference is enough to make a dispositive difference in where one attends law school and whether one receives a six-figure scholarship – but even that large an LSAT gap really predicts only a modest 0.1 difference in LGPA…

[C]hanges in LSAT do not appear to have increasing or decreasing returns; an X-point difference between a low and very low LSAT predicts the same as an X-point difference between a high and very high LSAT. Thus, contrary to some common assumptions, a “cutoff” driven by fear of an especially low LSAT is unsound: the difference between a 147 and a 152 is the same as the difference between a 157 and a 162.

Legal Education, Scholarship | Permalink


Good pt on LSAT, but GPA similarly truncated to about 2 pts. Still close.

Posted by: Anonymous | Oct 24, 2016 9:05:17 PM

I wasn't disagreeing. Indeed, I think we are saying the same thing. My only point was that the lack of effect is not based on grade inflation, but instead just the nature of how grades work. Additionally, I can tell you from experience that law schools do not view it in their self-interest to have grade inflation, even if they find it hard to control due to a) individual teach discretion, and b) competition from other schools with inflation

Posted by: Michael Risch | Oct 22, 2016 6:33:52 PM

The LSAT is scored from 120 to 180, so there are actually only ~60 possible scores. A 6 point movement would be a 10% movement under the (likely incorrect) assumption that the scale is linear.

Posted by: LSAT | Oct 21, 2016 9:09:02 PM

I agree with curmudgeonly. Law school grades are one thing - bar passage is another.

Posted by: ibex | Oct 21, 2016 9:36:14 AM

Admittedly, I can be cynical, but I did not mean my comment above to be so. I don’t think law schools have covered themselves in glory of late, but I’m not committed to the law-school-is-a-scam narrative either. I just think incentives matter, and schools are very sensitive about factors that impact rank.

Disregarding incentives and instead focusing on facts about grade distribution, correlating LSAT scores with grades seems unlikely to yield meaningful data because grade distribution in law school is relatively compact. Consider the stated grading policy of my alma mater (UT School of Law at Austin):

a. Mandatory Distribution of Grades in First-Year Large Sections:
• 30 to 40% of grades must be A+, A, or A-; and,
• at least 5% of grades must be C+, C, D, or F.

b. Recommended Distribution of Grades in Other Courses:
• about 35% of grades should be A+, A, or A-;
• about 55% of grades should be B+, B, or B-; and,
• about 10% of grades should be C+, C, D, or F.

So I think it would be difficult to generate meaningful data because the vast majority of the student body will fit within a relatively narrow range of LSAT scores and the vast majority also get As and Bs as a matter of the grading policy. I suspect that, as a practical matter, Ds and Fs are rare and difficult to earn; so the distribution is likely even narrower than the grading policy suggests on its face.

Studies that focus on grades also have the additional obstacle that not all grades are comparable. Marks and Moss looked at cumulative law school GPA and first-year GPA. The latter should at least be an apples-to-apples comparison, assuming a uniform first-year curriculum. But I think cumulative GPAs are less so. Not all law school courses are alike; I see no reason to treat grades in courses like secured transactions or federal jurisdiction, for example, as being comparable with grades in less rigorous courses. I also think it likely that less capable students are the most likely to take less rigorous electives.

At bottom, these “success in law school” studies are meaningless because if success just means good grades—As or Bs—then most students are successful by definition. For students, law school is a means to an end. So if we are going to measure outcomes we might focus on outcomes that actually matter to them. For example, how does LSAT score correlate with prestigious clerkships, first-time bar passage, full-time employment within six months of graduation, landing a law school teaching job, salary?

Posted by: Curmudgeonly Ex-Clerk | Oct 21, 2016 8:08:30 AM

*bar failure. High negative correlation between LSAT and bar failure.

Posted by: Jojo | Oct 21, 2016 5:38:23 AM

There is, however, a strong negative correlation between LSAT score and bar passage.

Posted by: Jojo | Oct 21, 2016 5:36:35 AM

I agree with Curmudgeonly, though not as cynically. Every law school has a curve, and students fall within a range of grades, whether the LSAT median is 151 or 165. LSAC makes clear that it views each 5 LSAT band as providing very little difference in student ability. So, it's not surprising that at any one school a 5 point LSAT range won't predict much difference in grades. But if you put all of the students with all of the LSATs in one school all with the same curve, I suspect you would see much more of an effect.

Posted by: Michael Risch | Oct 20, 2016 7:32:20 PM

Interesting study. I wonder if law school grades are really the outcome of interest. If law school grades are highly random (as I suppose they are) then it wouldn't be surprising that LSAT scores don't correlate much with them. I also wonder how much variation the study has. Presumably the schools represented don't have very many high-LSAT or very low-LSAT students. Most students in the schools are clustered in the middle. That can impact the ability to see a statistically significant relationship between LSAT and law school gpa. Finally, if you take law school admissions officers seriously, the students they admit are admitted on the basis of a "holistic" admissions process. This means that the low-LSAT students presumably have other info in their files that demonstrates that despite the low LSAT they will succeed. I am not sure their study can control for this extra info. Maybe the low LSAT folks who would do badly in law school are excluded from the sample because admissions officers foresightedly refuse to admit them.

Posted by: Jason Yackee | Oct 20, 2016 4:57:59 PM

Whatever correlation exists or doesn't between LSAT and law school grades, the inquiry does not seem like a useful one. Given grade inflation and the schools' self-interest in giving students good marks (which make it easier to get a job, which in turn reflects on schools' job placement statistics), there's not much assurance that grades are a good measure of much of anything, excepting very high marks that truly distinguish students from the majority of their peers. What might be more useful know is the correlation between LSAT scores and bar passage on the first try.

Posted by: Curmudgeonly Ex-Clerk | Oct 20, 2016 1:06:33 PM

A 6 pt change on the LSAT is a 3.3% movement (6/180). A .1 change is GPA is a 2.5% movement (.1/4.0). Seems right on the mark.

Posted by: Anon | Oct 20, 2016 11:29:13 AM

Waiting for the "nuh-uhs" from the regular commentators (who won't, of course, actually bother to read the study)...

Posted by: anon | Oct 20, 2016 11:10:17 AM