TaxProf Blog

Paul L. Caron
Dean





Thursday, January 5, 2023

Blackman: GPT Will Soon Be Able To Pass The Multistate Bar Exam

Following up on my previous posts (links below):  Josh Blackman (South Texas; Google Scholar), Can GPT Pass the Multistate Bar Exam?:

GPTMy frequent co-authors, Mike Bommarito and Dan Katz utilized a different software tool from OpenAI, known as GPT-3.5, to answer the multiple choice questions on the Multistate Bar Examination (MBE). If there are four choices, the "baseline guessing rate" would be 25%. With no specific training, GPT scored an overall accuracy rate of 50.3%. That's better than what many law school graduates can achieve. And in particular, GPT reached the average passing rate for two topics: Evidence and Torts. (I'll let Evidence or Torts scholars speculate about why those topics may be easier for AI.) Here is a summary of the results from their paper:

The table and figure clearly show that GPT-3.5 is not yet passing the overall multiple choice exam. However, GPT-3.5 is significantly exceeding the baseline random chance rate of 25%. Furthermore, GPT-3.5 has reached the average passing rate for at least two categories, Evidence and Torts. 

On average across all categories, GPT-3.5 is trailing human test-takers by approximately 17%. In the case of Evidence, Torts, and Civil Procedure, this gap is negligible or in the single digits; at 1.5 times the standard error of the mean across our test runs, GPT-3.5 is already at parity with humans for Evidence questions. However, for the remaining categories of Constitutional Law, Real Property, Contracts, and Criminal Law, the gap is much more material, rising as high as 36% in the case of Criminal Law. ...

Overall, we find that GPT-3.5 significantly exceeds our expectations for performance on this task. Despite thousands of hours on related tasks over the last two decades between the authors, we did not expect GPT-3.5 to demonstrate such proficiency in a zero-shot settings with minimal modeling and optimization effort. While our ability to interpret how or why GPT-3.5 chooses between candidate answers is limited by understanding of LLMs and the proprietary nature of GPT, the history of similar problems strongly suggests that an LLM may soon pass the Bar. Based on anecdotal evidence related to GPT-4 and Bloom family of models, it is quite possible that this will occur within the next 0-18 months. ...

Worried yet?

Prior TaxProf Blog coverage:

https://taxprof.typepad.com/taxprof_blog/2023/01/gpt-will-soon-be-able-to-pass-the-multistate-bar-exam.html

Legal Ed Scholarship, Legal Ed Tech, Legal Education | Permalink