Friday, March 17, 2023
GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam
Daniel Martin Katz (Chicago Kent), Michael James Bommarito (Michigan State), Shang Gao (Casetext) & Pablo Arredondo (Casetext), GPT-4 Passes the Bar Exam:
In this paper, we experimentally evaluate the zero-shot performance of a preliminary version of GPT-4 against prior generations of GPT on the entire Uniform Bar Examination (UBE), including not only the multiple-choice Multistate Bar Examination (MBE), but also the open-ended Multistate Essay Exam (MEE) and Multistate Performance Test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 as compared to much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human tast-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.
- ABA Journal, Latest Version of ChatGPT Aces Bar Exam With Score Nearing 90th Percentile
- Josh Blackman (South Texas), In January, ChatGPT Failed The Bar. In March, GPT-4 Exceeds The Nationwide Student Average
- Casetext Press Release, CoCounsel Builds on the Power of GPT-4, the AI That Outperformed Real Bar Candidates
- Chicago-Kent Press Release, GPT-4 Passes the Bar Exam
- Reuters, Bar Exam Score Shows AI Can Keep Up With 'Human Lawyers,' Researchers Say
- Eugene Volokh (UCLA), ChatGPT-4 Sharply Outperforms ChatGPT-3. But Humans-4 Don't Sharply Outperform Humans-3
Prior TaxProf Blog coverage:
- A Human Being Wrote This Law Review Article: GPT-3 And The Practice Of Law (May 11, 2022)
- The Implications Of OpenAI’s Assistant For Legal Services And Society (Dec. 7, 2022)
- ChatGPT And Law School Exams (Dec. 29, 2022)
- GPT Will Soon Be Able To Pass The Multistate Bar Exam (Jan. 5, 2023)
- Using ChatGPT To Write Law School Exams, Bar Exams, And Strategic Plans (Jan. 11, 2023)
- ChatGPT Gets B|B- Grade On Wharton MBA Exam (Jan. 24, 2023)
- ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax) (Jan. 24, 2023)
- The Rise Of The Robotic Tax Analyst (Jan. 27, 2023)
- Ryznar: Exams In The Time Of ChatGPT (Feb. 1, 2023)
- Bishop Posts Two Papers On ChatGPT (Feb. 8, 2023)
- ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More (Feb. 23, 2023)
- It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
- New AI Detector Is 97% Effective In Catching Students Cheating With ChatGPT (Feb. 28, 2023)
- It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
- ChatGPT's Tax Advice Was Wrong 100% Of The Time (Mar. 7, 2023)
- Does ChatGPT Produce Fishy Briefs? (Mar. 8, 2023)
- Colleges (And Law Schools) Are Rushing To Respond To ChatGPT (Mar. 9, 2023)
- Was The Sermon You Heard At Church Today Written By ChatGPT? (Mar. 12, 2023)
- GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam (Mar. 17, 2023)
- ChatGPT Thinks I Am Way More Interesting Than I Am (Mar. 22, 2023)
- Should ChatGPT Be In Law School? (Mar. 30, 2023)
- Merritt: GPT-4 On Legal Education And Lawyer Licensing (Apr. 4, 2023)
- Turnitin Plagiarism Detector Will Catch Students Who Cheat With ChatGPT With 98% Accuracy (Apr. 5, 2023)
- ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT (Apr. 6, 2023)
- AI Tools for Lawyers: A Practical Guide (Apr. 6, 2023)
https://taxprof.typepad.com/taxprof_blog/2023/03/gpt-4-beats-90-of-aspiring-lawyers-on-the-bar-exam.html