ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax)

Following up on this morning's post, ChatGPT Gets B|B- Grade On Wharton MBA Exam:  Jonathan Choi (Minnesota; Google Scholar), Kristin Hickman (Minnesota; Google Scholar), Amy Monahan (Minnesota) & Daniel Schwarcz (Minnesota; Google Scholar), ChatGPT Goes to Law School:

How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.

Overall, ChatGPT passed all four classes based on its final exam, averaging a C+ across all exams, an outcome which would earn credit toward the JD but place the student on academic probation. Notably, if such performance were consistent throughout law school, the grades earned by ChatGPT would be sufficient for a student to graduate. Despite performing sufficiently well to theoretically earn a JD degree, ChatGPT generally scored at or near the bottom of each class. ChatGPT received a B in Constitutional Law (36th out of 40 students), a B- in Employee Benefits (18th out of 19 students), a C- in Taxation (66th out of 67 students), and a C- in Torts (75th out of 75 students). ...

The following figures depict ChatGPT’s performance on each question (or, in the case of multiple choice questions, each set of questions) relative to real students. The figures are density plots, where the x-axis reflects the score for each exam component, and the y-axis reflects the share of students who received the relevant score. The black dashed lines show mean scores for all students, and the red solid lines are ChatGPT’s scores. ChatGPT’s percentile performance for each question is also listed in red.

Prior TaxProf Blog coverage:

• A Human Being Wrote This Law Review Article: GPT-3 And The Practice Of Law (May 11, 2022)
• The Implications Of OpenAI’s Assistant For Legal Services And Society (Dec. 7, 2022)
• ChatGPT And Law School Exams (Dec. 29, 2022)
• GPT Will Soon Be Able To Pass The Multistate Bar Exam (Jan. 5, 2023)
• Using ChatGPT To Write Law School Exams, Bar Exams, And Strategic Plans (Jan. 11, 2023)
• ChatGPT Gets B|B- Grade On Wharton MBA Exam (Jan. 24, 2023)
• ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax) (Jan. 24, 2023)
• The Rise Of The Robotic Tax Analyst (Jan. 27, 2023)
• Ryznar: Exams In The Time Of ChatGPT (Feb. 1, 2023)
• Bishop Posts Two Papers On ChatGPT (Feb. 8, 2023)
• ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More (Feb. 23, 2023)
• It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
• New AI Detector Is 97% Effective In Catching Students Cheating With ChatGPT (Feb. 28, 2023)
• It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
• ChatGPT's Tax Advice Was Wrong 100% Of The Time (Mar. 7, 2023)
• Does ChatGPT Produce Fishy Briefs? (Mar. 8, 2023)
• Colleges (And Law Schools) Are Rushing To Respond To ChatGPT (Mar. 9, 2023)
• Was The Sermon You Heard At Church Today Written By ChatGPT? (Mar. 12, 2023)
• GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam (Mar. 17, 2023)
• ChatGPT Thinks I Am Way More Interesting Than I Am (Mar. 22, 2023)
• Should ChatGPT Be In Law School? (Mar. 30, 2023)
• Merritt: GPT-4 On Legal Education And Lawyer Licensing (Apr. 4, 2023)
• Turnitin Plagiarism Detector Will Catch Students Who Cheat With ChatGPT With 98% Accuracy (Apr. 5, 2023)
• ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT (Apr. 6, 2023)
• AI Tools for Lawyers: A Practical Guide (Apr. 6, 2023)