ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT

Press Release, ChatGPT LSAT Score Falls Short of Getting Into Top Law Schools:

Blueprint Prep, a premier test preparation and professional training company, was among the first test prep companies to analyze the LSAT results of #OpenAI's ChatGPT, the artificial intelligence chatbot capable of accurately responding to human language. The chatbot's LSAT results revealed that ChatGPT-3.5 series was unable to meet the typical scores required for admission to a top 14 law school.

A veteran Blueprint Prep LSAT instructor analyzed the chatbot's test results and discovered its chief errors lay in a failure to consistently apply logical and critical reasoning, as well as an inability to distinguish essential information from superfluous material. These are areas where input from an experienced instructor would help ChatGPT better understand the questions and how to interpret them. Despite its less-than-stellar scores, however, ChatGPT was convinced while taking the test that it was answering questions correctly.

"We know that the LSAT is designed to measure a student's analytical reasoning, critical thinking, and reading comprehension skills, which are essential for success in law school," said Gene Suhir, LSAT Academic Manager at Blueprint Prep. "These skills can be significantly strengthened, but to do so requires leveraging proven strategies and top-scoring instructors who can help students get into the mindsets of the test makers. This form of LSAT test prep not only enables the student to process information like a lawyer would, but it's been proven that strengthening reasoning skills via LSAT test prep can help wire students' brains to think more like a lawyer. This is not the specific kind of reasoning that ChatGPT is innately useful for, although it can learn these skills."

The LSAT is scored on a scale of 120-180, with 151 being roughly average. Students usually need a typical score in the 170s to be admitted to one of the nation's top 14 law schools. On the two tests it took, ChatGPT scored a 148 (37th percentile) and a 157 (70th percentile). [148 is the median LSAT score for law schools ranked 187th in this metric for the current 1L class (North Carolina Central, Ohio Northern, Oklahoma City, Western Michigan, Widener (PA)). 157 is the median LSAT score for law schools ranked 98th in this metric for the current 1L class (Kentucky, LSU, Michigan State, Mississippi, Regent, Seattle, Syracuse, Texas Tech).]

ChatGPT performed worst in logic games, was mediocre in logic reasoning, and scored best in reading comprehension, which is the LSAT's most straightforward test section. The chatbot had a hard time recognizing that a new rule only applied to a specific question and did not carry over into the next question. Additionally, the LSAT adds irrelevant information to questions to serve as distractions and ChatGPT was unable to distinguish between the peripheral and essential information.

Prior TaxProf Blog coverage:

• A Human Being Wrote This Law Review Article: GPT-3 And The Practice Of Law (May 11, 2022)
• The Implications Of OpenAI’s Assistant For Legal Services And Society (Dec. 7, 2022)
• ChatGPT And Law School Exams (Dec. 29, 2022)
• GPT Will Soon Be Able To Pass The Multistate Bar Exam (Jan. 5, 2023)
• Using ChatGPT To Write Law School Exams, Bar Exams, And Strategic Plans (Jan. 11, 2023)
• ChatGPT Gets B|B- Grade On Wharton MBA Exam (Jan. 24, 2023)
• ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax) (Jan. 24, 2023)
• The Rise Of The Robotic Tax Analyst (Jan. 27, 2023)
• Ryznar: Exams In The Time Of ChatGPT (Feb. 1, 2023)
• Bishop Posts Two Papers On ChatGPT (Feb. 8, 2023)
• ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More (Feb. 23, 2023)
• It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
• New AI Detector Is 97% Effective In Catching Students Cheating With ChatGPT (Feb. 28, 2023)
• It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship (Feb. 25, 2023)
• ChatGPT's Tax Advice Was Wrong 100% Of The Time (Mar. 7, 2023)
• Does ChatGPT Produce Fishy Briefs? (Mar. 8, 2023)
• Colleges (And Law Schools) Are Rushing To Respond To ChatGPT (Mar. 9, 2023)
• Was The Sermon You Heard At Church Today Written By ChatGPT? (Mar. 12, 2023)
• GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam (Mar. 17, 2023)
• ChatGPT Thinks I Am Way More Interesting Than I Am (Mar. 22, 2023)
• Should ChatGPT Be In Law School? (Mar. 30, 2023)
• Merritt: GPT-4 On Legal Education And Lawyer Licensing (Apr. 4, 2023)
• Turnitin Plagiarism Detector Will Catch Students Who Cheat With ChatGPT With 98% Accuracy (Apr. 5, 2023)
• ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT (Apr. 6, 2023) 
• AI Tools for Lawyers: A Practical Guide (Apr. 6)