Bilek & Merritt: ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More

Bloomberg Law Op-Ed:  ChatGPT Almost Passed the Bar, But Competent Lawyers Do Much More, by Mary Lu Bilek (CUNY) & Deborah Merritt (Ohio State):

ChatGPT, OpenAI’s provocative artificial intelligence program, has come close to passing the multiple-choice portion of the bar exam. The bot has also earned passing grades on law school essays that resemble ones written for the exam. ...

Yet, even in a bill-by-the-hour world, clients who can afford it will still seek out human lawyers. Why? Because humans are far better than bots at eliciting facts and goals from clients, identifying new avenues of research, and solving multi-dimensional problems. Human experts will supplement those advantages by knowing when to consult AI, how to assess AI responses, and how to integrate AI knowledge with the human dimensions of a client problem. ...

The bar exam heavily tests candidates’ recall of detailed legal principles, like the arcane rule against perpetuities assessed on the most recent exam. A century ago, lawyers stored those principles in memory, pulling them out when necessary. But the explosive growth of legal rules since the New Deal has made that practice impossible: There are simply too many legal rules for any lawyer to accurately remember more than a tiny fraction of them. And good lawyers recognize that legal rules change, both over time and as lawyers move across state lines.

As ChatGPT demonstrates, well trained AI can perform these tasks in less than a minute—with flawless grammar, spelling, and organization. Astute clients will want lawyers who know how and when to use the bot, how to explore subtleties outside the bot’s comprehension, and how to devise creative solutions. Bots are better than humans at thinking inside the box, but humans excel at reaching outside the box. ...

ChatGPT provides yet more evidence that time-pressured, closed-book written exams reflect outdated lawyering practices. Those exams perpetuate exclusionary practices without adequately protecting clients.

Prior TaxProf Blog coverage:

• A Human Being Wrote This Law Review Article: GPT-3 And The Practice Of Law (May 11, 2022)
• The Implications Of OpenAI’s Assistant For Legal Services And Society (Dec. 7, 2022)
• ChatGPT And Law School Exams (Dec. 29, 2022)
• GPT Will Soon Be Able To Pass The Multistate Bar Exam (Jan. 5, 2023)
• Using ChatGPT To Write Law School Exams, Bar Exams, And Strategic Plans (Jan. 11, 2023)
• ChatGPT Gets B|B- Grade On Wharton MBA Exam (Jan. 24, 2023)
• ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax) (Jan. 24, 2023)
• The Rise Of The Robotic Tax Analyst (Jan. 27, 2023)
• Ryznar: Exams In The Time Of ChatGPT (Feb. 1, 2023)
• Bishop Posts Two Papers On ChatGPT (Feb. 8, 2023)
• ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More (Feb. 23, 2023)
• It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
• New AI Detector Is 97% Effective In Catching Students Cheating With ChatGPT (Feb. 28, 2023)
• It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
• ChatGPT's Tax Advice Was Wrong 100% Of The Time (Mar. 7, 2023)
• Does ChatGPT Produce Fishy Briefs? (Mar. 8, 2023)
• Colleges (And Law Schools) Are Rushing To Respond To ChatGPT (Mar. 9, 2023)
• Was The Sermon You Heard At Church Today Written By ChatGPT? (Mar. 12, 2023)
• GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam (Mar. 17, 2023)
• ChatGPT Thinks I Am Way More Interesting Than I Am (Mar. 22, 2023)
• Should ChatGPT Be In Law School? (Mar. 30, 2023)
• Merritt: GPT-4 On Legal Education And Lawyer Licensing (Apr. 4, 2023)
• Turnitin Plagiarism Detector Will Catch Students Who Cheat With ChatGPT With 98% Accuracy (Apr. 5, 2023)
• ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT (Apr. 6, 2023)
• AI Tools for Lawyers: A Practical Guide (Apr. 6, 2023)