Tuesday, January 24, 2023
Following up on this morning's post, ChatGPT Gets B|B- Grade On Wharton MBA Exam: Jonathan Choi (Minnesota; Google Scholar), Kristin Hickman (Minnesota; Google Scholar), Amy Monahan (Minnesota) & Daniel Schwarcz (Minnesota; Google Scholar), ChatGPT Goes to Law School:
How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.
Overall, ChatGPT passed all four classes based on its final exam, averaging a C+ across all exams, an outcome which would earn credit toward the JD but place the student on academic probation. Notably, if such performance were consistent throughout law school, the grades earned by ChatGPT would be sufficient for a student to graduate. Despite performing sufficiently well to theoretically earn a JD degree, ChatGPT generally scored at or near the bottom of each class. ChatGPT received a B in Constitutional Law (36th out of 40 students), a B- in Employee Benefits (18th out of 19 students), a C- in Taxation (66th out of 67 students), and a C- in Torts (75th out of 75 students). ...
The following figures depict ChatGPT’s performance on each question (or, in the case of multiple choice questions, each set of questions) relative to real students. The figures are density plots, where the x-axis reflects the score for each exam component, and the y-axis reflects the share of students who received the relevant score. The black dashed lines show mean scores for all students, and the red solid lines are ChatGPT’s scores. ChatGPT’s percentile performance for each question is also listed in red.