Friday, September 13, 2013
Law School GPA Trumps LSAT, Undergraduate GPA, and Bar Review Courses as Bar Passage Predictor
Nicholas L. Georgakopoulos (Indiana-Indianapolis), Bar Passage: GPA and LSAT, Not Bar Reviews:
Probit regressions of bar passage on law GPA, undergraduate GPA (uGPA) and LSAT show GPA to have a very strong relation, LSAT a weaker one, and uGPA not to have any relation. 1L and upperclass GPA both have strong predictive power, favoring an interpretation of significant learning in small and elective courses compared to the mandated large ones of the first year. Linear regressions of GPA on uGPA and LSAT show a noisy relation to exist only for first-time bar exam takers, none for 2nd time takers. Analysis of bar review courses does not show the choice among them to have consequence. Possible interpretations favor legal education over innate skill and the training in legal analysis over memorization.
https://taxprof.typepad.com/taxprof_blog/2013/09/law-school-gpa-.html
Comments
To Catherine:
Great point, the data do not refute such a theory applied to law school courses (analysis?) but not bar review courses (memorization?). Perhaps if enough graduates delay taking the bar, we'd be able to tease out more detail--I will give it a shot, seeing if those who delay akin the bar experience rates more similar to second time takers than first.
Posted by: Nicholas Georgakopoulos | Sep 15, 2013 6:35:21 PM
Related to point 5, could your data be showing that measures of achievement closer in time to the bar exam have greater predictive power than measures further back in time? If we believe that GPA and the LSAT represent some mixture of effort and ability, and effort (aka motivation) varies over time, then we would expect law school GPA, which measures the student’s efforts in the 3 years immediately preceding the bar exam, to have the most predictive power, followed by the LSAT, followed by undergrad GPA (since while a student may take the LSAT in their last years of college, the undergrad UPA will be an average of their scores over the entire period).
Posted by: Catherine Fisher | Sep 13, 2013 7:09:22 PM
Many thanks to Paul for posting and to Eric for the, as always, very constructive comments.
Issues 1-4 I will try to adress in the next draft. The correlations are: GPA-LSAT: .345, GPA-uGPA:.346, GPA-1Lgpa: .904.
About point 5: Everything is possible. However, if what drives GPA is effort and not ability or analysis, then I would expect that to also be true in the undergraduate level, and then I would expect to see uGPA to have a stronger relation with GPA and with bar passage. Also, preparing for the bar is a demanding process, so a habit of effort should matter and uGPA should have some influence on the bar outcomes. Finally, something strange is happening with those who take the bar for a second time. For those all the relations are weaker. If industriousness matters, wouldn't we expect measures of industriousness, like GPA and uGPA, to retain relevance?
Point 6: The school does have a strict curve. I have done the analysis and in classes with 20+ students, the profs mostly adhere to the curve of mean 3.1 stdev .6 (with a bump at A's due to the truncation, the inability to award higher grades, from which my attempt to fit a normal curve suggests a "real" mean of 3.15 and stdev of .57). In smaller courses the curve is higher, mean 3.4 stdev .6 (best fit mean 3.5 stdev .7). The internal rule is that the curve becomes more flexible for classes under 20 students but in the data an important switch seems to happen at 40 students, above that the means are lower and stdev higher (.78). Also, whereas GPA treats plus and minus grades as .3 away from the whole grades, the distributions suggest that professors treat them as 1/3 (.333...) away from the whole grades, which makes sense intuitively but shows that GPA gets some noise from that (of two students with the same GPA, the one with more plus grades can justifiably claim to be disadvantaged and vice versa for minus grades). So all this informs the upperclass grade inflation versus selection/learning. Maybe a school with a more extreme discrepancy or less of a discrepancy would have a different story to tell.
Posted by: Nicholas Georgakopoulos | Sep 13, 2013 9:20:19 AM
Interesting paper! I'll post here the comments I just sent the author, since I'd like to see more discussion of working papers on blogs (the Million Dollar Law Degree paper was a good example).
1. Put more blank rows in the tables (i.e., white space) and fewer vertical and horizontal lines.
2. It's good to have a long discussion of descriptive statistics, but also give the key ones in one paragraph, medians and quartiles for LSAT, 1st-year GPA, overall GPA, LSAT, undergrad GPA, bar passage rate in your sample.
3. GIve a table of pairwise correlations between the key variables, so the reader can answer additional questions (e.g., if I know JUST first-year GPA, how well can I predict bar passage?) and puzzle out what's going on in the data.
4. Statistically, it's fine to have both 1st year GPA and overall GPA as right-hand-variables, even though overall GPA includes 1st-year GPA. If 1st-year GPA is all that matters, it will pick up ALL the explanatory power, and overall GPA will show up insignificant. It's only when a right-hand variable is partly made up of a left-hand variable that there's a statistical problem, or when two right-hand variable are perfectly correlated.
So DO have a regression with allt he RHS variables included. That will be the best one, the one you should focus on in making conclusions.
5. "From this perspective and in the case of bar
exam passage in this sample, nurture dominates nature as a
predictor of bar passage."
There's another possibility. LSAT measures innate ability. GPA measures innate ability, willingness to exert effort, and learning that is relevant to the bar exam. Suppose that nurture--- in the sense of learning that is relevant to the bar exam--- is completely unimportant, because what you learn in Torts class is useless for the bar exam. GPA still measures willingness to exert effort, and the hard-working person will work harder in the bar review class and do better on the bar exam.
This is one of the two big reasons GPA turns out so important, I think. LSAT measures innate (as of entering law school) ability, but GPA measures innate (as of entering law school) industriousness.
6. The other reason one would expect GPA to do so well in law school is that law schools often have strict curves. Does the school in the sample? Does it for both 1st year and upper-level courses? If a law school has a strict curve and required courses, then GPA becomes much much more informative about innate ability. What is surprising is that you find that upper-level GPA does better than 1st-year at predicting bar passage.
7. Here's something that ought to be done in your study and at all academic levels: find out if grades in PARTICULAR COURSES predict outcomes well. I hypothesize that doing well in Civil Procedure is a better predictor of bar passage success than anything else you have. Or, maybe a surprise will turn up , and we'll discover that the grade in Legal Ethics is the best predictor. It might even be that the student's grade in Freshman Calculus is the best predictor of bar passage, though you don't have that data.
Posted by: Eric Rasmusen | Sep 13, 2013 7:42:09 AM
To address Catherine's point, I ran the probitML regression of bar passage against the number of months since graduation and the coefficient was essentially zero (and not stat'ly significant).
Posted by: Nicholas Georgakopoulos | Sep 17, 2013 1:11:11 PM