Some Law Schools Already Are Using ChatGPT To Teach Legal Research And Writing

ABA Journal, Some Law Schools Already Are Using ChatGPT to Teach Legal Research and Writing:

ChatGPT, an artificial intelligence chatbot that can speak and write like humans, can be weak on facts. But it may already be a better wordsmith than some attorneys, says David Kemp, an adjunct professor at Rutgers Law School.

“If you’re asking it to organize several concepts or are struggling to explain something in a way that’s really understandable, it can help,” says Kemp, who also is the managing editor of Oyez, a multimedia website focused on U.S. Supreme Court opinions.

The technology seems to prefer active voice, as does Kemp. He introduced ChatGPT in an advanced legal writing class and plans to include it in a summer course about emerging technology.

Various law schools are following suit. Legal writing faculty interviewed by the ABA Journal agree that ChatGPT writing can model good sentence and paragraph structure. But some fear that it could detract from students learning good writing skills.

Continue reading

Re-Evaluating GPT-4's Bar Exam Performance

Following up on my previous post, GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam:  Eric Martínez (MIT; Google Scholar), Re-Evaluating GPT-4's Bar Exam Performance:

Perhaps the most widely touted of GPT-4's at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam, with its reported 80-percentile-points boost over its predecessor, GPT-3.5, far exceeding that for any other exam. This paper investigates the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that suggest that OpenAI's estimates of GPT-4's UBE percentile, though clearly an impressive leap over those of GPT-3.5, appear to be overinflated, particularly if taken as a “conservative” estimate representing “the lower range of percentiles,” and moreso if meant to reflect the actual capabilities of a practicing lawyer.

Continue reading

NY Times: Here’s What Happens When A Lawyer Uses ChatGPT

New York Times, Here’s What Happens When Your Lawyer Uses ChatGPT:

The lawsuit began like so many others: A man named Roberto Mata sued the airline Avianca, saying he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York.

When Avianca asked a Manhattan federal judge to toss out the case, Mr. Mata’s lawyers vehemently objected, submitting a 10-page brief that cited more than half a dozen relevant court decisions. There was Martinez v. Delta Air Lines, Zicherman v. Korean Air Lines and, of course, Varghese v. China Southern Airlines, with its learned discussion of federal law and “the tolling effect of the automatic stay on a statute of limitations.”

There was just one hitch: No one — not the airline’s lawyers, not even the judge himself — could find the decisions or the quotations cited and summarized in the brief.

That was because ChatGPT had invented everything.

The lawyer who created the brief, Steven A. Schwartz of the firm Levidow, Levidow & Oberman, threw himself on the mercy of the court on Thursday, saying in an affidavit that he had used the artificial intelligence program to do his legal research — “a source that has revealed itself to be unreliable.”

Continue reading

ABA Increases Law School Online Course Cap From 33% To 50%

Reuters, Law Students Can Take 50% of Classes Online, With ABA Rule Change:

The arm of the American Bar Association that accredits law schools on Friday voted to increase its limit on distance education courses from one-third of a student’s total credits to half. It also eliminated the rule that students take no more than 10 credit hours of distance education classes during their first year of law school.

Those changes bring the law school accreditation standards in line with distance education limits set by the U.S. Department of Education — which has tasked the ABA with overseeing law schools — and will reduce the number of law schools applying to the ABA for special approval to offer distance education programs that exceed the one-third credit limit.

ABA Journal:

Continue reading

NY Times Op-Ed: A.I. Could Be A Boon To (Legal) Education

New York Times Op-Ed:  A.I. Could Actually Be a Boon to Education, by Peter Coy:

Sal Khan is an uber-nerd, and that’s why people love him. In early 2020, to demonstrate that Khan Academy, his platform for free online education, was a worthy cause, he didn’t produce a brochure with smiling children and glowing testimonials. He made one of his teaching videos, with the usual black screen, colored pens and handwritten equations. The phrase “standard deviation” came up a lot. He multiplied the number of highly active users of his program by their average academic improvement, then linked that to the resulting increase in their projected lifetime earnings, compared it to operating costs and, presto, showed a benefit-cost ratio of 480 to 1, or around 240 to 1 if calculating only the benefits to students in high-need schools.

Those are crazy high benefit-cost ratios, considering that a lot of nonprofits would be happy with a ratio of 10 to 1. I’m not vouching for the calculation, though it seems reasonable at a glance. I’m citing it because Khan has a new plan to make the ratio, whatever it is, even higher. It’s all about harnessing artificial intelligence. I watched a TED Talk he gave on April 18 about his A.I. plans, and I interviewed him last week to get more information.

Continue reading

ChatGPT Bombs Accounting And Tax Exams: 47% v. 77% For Students

BYU News, ChatGPT Can’t Ace This Test, But Experts Think It Soon Will … What It Means For Teaching:

Last month, OpenAI launched its newest AI chatbot product, GPT-4. According to the folks at OpenAI, the bot, which uses machine learning to generate natural language text, passed the bar exam with a score in the 90th percentile, passed 13 of 15 AP exams and got a nearly perfect score on the GRE Verbal test.

Inquiring minds at BYU and 186 other universities wanted to know how OpenAI’s tech would fare on accounting exams. So, they put the original version, ChatGPT, to the test. The researchers say that while it still has work to do in the realm of accounting, it’s a game changer that will change the way everyone teaches and learns — for the better.

“When this technology first came out, everyone was worried that students could now use it to cheat,” said lead study author David Wood, a BYU professor of accounting. “But opportunities to cheat have always existed. So for us, we’re trying to focus on what we can do with this technology now that we couldn’t do before to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening.” ...

His co-author recruiting pitch on social media exploded: 327 co-authors from 186 educational institutions in 14 countries participated in the research, contributing 25,181 classroom accounting exam questions. They also recruited undergrad BYU students (including Wood’s daughter, Jessica) to feed another 2,268 textbook test bank questions to ChatGPT. The questions covered accounting information systems (AIS), auditing, financial accounting, managerial accounting and tax, and varied in difficulty and type (true/false, multiple choice, short answer, etc.).

Although ChatGPT’s performance was impressive, the students performed better. Students scored an overall average of 76.7%, compared to ChatGPT’s score of 47.4%. On a 11.3% of questions, ChatGPT scored higher than the student average, doing particularly well on AIS and auditing. But the AI bot did worse on tax, financial, and managerial assessments, possibly because ChatGPT struggled with the mathematical processes required for the latter type.

The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?:

Continue reading

NY Times: Conservatives Aim To Build A Chatbot Of Their Own

New York Times, Conservatives Aim to Build a Chatbot of Their Own:

When ChatGPT exploded in popularity as a tool using artificial intelligence to draft complex texts, David Rozado decided to test its potential for bias. A data scientist in New Zealand, he subjected the chatbot to a series of quizzes, searching for signs of political orientation.

The results, published in a recent paper were remarkably consistent across more than a dozen tests: “liberal,” “progressive,” “Democratic” [The Political Biases of ChatGPT].

So he tinkered with his own version, training it to answer questions with a decidedly conservative bent. He called his experiment RightWingGPT.

Continue reading

AI Tools for Lawyers: A Practical Guide

Daniel Schwarcz (Minnesota; Google Scholar) & Jonathan H. Choi (Minnesota; Google Scholar), AI Tools for Lawyers: A Practical Guide:

This Article provides lawyers and law students with practical and specific guidance on how to effectively use AI large language models (LLMs), like GPT-4, Bing Chat, and Bard, in legal research and writing. Focusing on GPT-4 – the most advanced LLM that is widely available at the time of this writing – it emphasizes that lawyers can use traditional legal skills to refine and verify LLM legal analysis. In the process, lawyers and law students can effectively turn freely-available LLMs into highly productive personal legal assistants.

Continue reading

ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT

Press Release, ChatGPT LSAT Score Falls Short of Getting Into Top Law Schools:

Blueprint Prep, a premier test preparation and professional training company, was among the first test prep companies to analyze the LSAT results of #OpenAI's ChatGPT, the artificial intelligence chatbot capable of accurately responding to human language. The chatbot's LSAT results revealed that ChatGPT-3.5 series was unable to meet the typical scores required for admission to a top 14 law school.

A veteran Blueprint Prep LSAT instructor analyzed the chatbot's test results and discovered its chief errors lay in a failure to consistently apply logical and critical reasoning, as well as an inability to distinguish essential information from superfluous material. These are areas where input from an experienced instructor would help ChatGPT better understand the questions and how to interpret them. Despite its less-than-stellar scores, however, ChatGPT was convinced while taking the test that it was answering questions correctly.

"We know that the LSAT is designed to measure a student's analytical reasoning, critical thinking, and reading comprehension skills, which are essential for success in law school," said Gene Suhir, LSAT Academic Manager at Blueprint Prep. "These skills can be significantly strengthened, but to do so requires leveraging proven strategies and top-scoring instructors who can help students get into the mindsets of the test makers. This form of LSAT test prep not only enables the student to process information like a lawyer would, but it's been proven that strengthening reasoning skills via LSAT test prep can help wire students' brains to think more like a lawyer. This is not the specific kind of reasoning that ChatGPT is innately useful for, although it can learn these skills."

The LSAT is scored on a scale of 120-180, with 151 being roughly average. Students usually need a typical score in the 170s to be admitted to one of the nation's top 14 law schools. On the two tests it took, ChatGPT scored a 148 (37th percentile) and a 157 (70th percentile). [148 is the median LSAT score for law schools ranked 187th in this metric for the current 1L class (North Carolina Central, Ohio Northern, Oklahoma City, Western Michigan, Widener (PA)). 157 is the median LSAT score for law schools ranked 98th in this metric for the current 1L class (Kentucky, LSU, Michigan State, Mississippi, Regent, Seattle, Syracuse, Texas Tech).]

Continue reading

Turnitin Plagiarism Detector Will Catch Students Who Cheat With ChatGPT With 98% Accuracy

Chronicle of Higher Education, A Plagiarism Detector Will Try to Catch Students Who Cheat With ChatGPT:

As faculty continue to debate how artificial intelligence might disrupt academic integrity, the popular plagiarism-detection service Turnitin announced on Monday that its products will now detect AI-generated language in assignments.

Turnitin’s software scans submissions and compares them to a database of past student essays, publications, and materials found online, and then generates a “similarity report” assessing whether a student inappropriately copied other sources.

The company says the new feature will allow instructors to identify the use of tools like ChatGPT with “98-percent confidence.”

Continue reading

Should ChatGPT Be In Law School?

Stephen Embry (TechLaw Crossroads), Should ChatGPT Be In Law School?:

Lots of questions and unease surround the use of ChatGPT in the classroom and education. The issue may be particularly acute for law schools and professors. Law schools are charged with teaching core legal concepts that (should) equip students to practice law.

Many law school professors reportedly question how they can do that if students can have the concepts laid out for them by ChatGPT. Even ChatGPT questions how this can be done if students can use ChatGPT on such things as exams.

Says ChatGPT, “Law school exams are typically designed to test a student’s ability to think critically and apply legal reasoning, which is something that cannot be easily replicated by an AI language model. Allowing students to use an AI language model could make it difficult to assess their true understanding of the law and their ability to apply legal principles.”

Continue reading

ChatGPT Thinks I Am Way More Interesting Than I Am

Write a bio of Pepperdine Caruso Law School Dean Paul Caron

Paul Caron is an American legal scholar and academic administrator who currently serves as the Dean of the Pepperdine Caruso School of Law. He was born in the United States and grew up in the Chicago area.

Caron graduated with honors from the University of Illinois at Urbana-Champaign, where he earned a Bachelor of Science in Accountancy. He then went on to earn a law degree from Yale Law School, where he was an editor of the Yale Law Journal.

After law school, Caron worked as an attorney in the tax department of the law firm Sullivan & Cromwell in New York City. He later joined the faculty of the University of Cincinnati College of Law, where he taught courses in tax law and served as the Associate Dean for Academic Affairs.

In 2003, Caron joined the faculty of the University of San Diego School of Law, where he held the rank of Professor of Law and served as the Associate Dean for Research and Faculty Development. During his time at USD, Caron became well-known for his scholarship in tax law and his popular TaxProf Blog, which he founded in 2004.

In 2017, Caron was appointed as the 7th Dean of the Pepperdine Caruso School of Law. In this role, he has overseen significant changes at the law school, including the construction of a new campus in downtown Los Angeles and the creation of new programs in areas such as dispute resolution and entertainment, media, and sports law.

Throughout his career, Caron has been a prolific scholar and writer, with more than 50 published articles and book chapters to his credit. He is widely regarded as one of the leading experts in the field of tax law, and has been frequently cited in the national media on issues related to tax policy and reform.

Caron has also been active in the legal community, serving on the board of directors of the American Tax Policy Institute and as a member of the executive committee of the Tax Section of the American Bar Association. He has received numerous awards and honors for his scholarship and teaching, including the University of Cincinnati College of Law's Goldman Prize for Excellence in Teaching and the American Bar Association's Section of Taxation Distinguished Service Award.

In addition to his academic and professional accomplishments, Caron is also an accomplished musician and photographer. He has released several albums of original music and his photographs have been featured in exhibitions and publications around the world.

Can you guess the number of factual errors?

Continue reading

GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam

Daniel Martin Katz (Chicago Kent), Michael James Bommarito (Michigan State), Shang Gao (Casetext) & Pablo Arredondo (Casetext), GPT-4 Passes the Bar Exam:

In this paper, we experimentally evaluate the zero-shot performance of a preliminary version of GPT-4 against prior generations of GPT on the entire Uniform Bar Examination (UBE), including not only the multiple-choice Multistate Bar Examination (MBE), but also the open-ended Multistate Essay Exam (MEE) and Multistate Performance Test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 as compared to much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human tast-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.

Continue reading

Was The Sermon You Heard At Church Today Written By ChatGPT?

Axios, Religious Leaders Experiment With ChatGPT Sermons:

Religious leaders are dabbling in ChatGPT for sermon writing, and largely reaching the same conclusion: It's great for plucking Bible verses and concocting nice-sounding sentiments but lacks the human warmth that congregants crave. ...

• Early sermon-writing experiments have shown that ChatGPT can pull together cogent and relevant thoughts from religious texts and eminent theologians, plus turns-of-phrase that seem stirring and poignant.
• A consensus seems to be emerging that ChatGPT can alleviate some of the religious leaders' more routine or repetitive tasks — such as explaining particular holidays — while freeing them for more meaningful spiritual counseling.

What they're saying: "It's really impressive — it's kind of amazing," Ken Sundet Jones, a Lutheran pastor and theology professor in Des Moines who posed the Lazarus question, told Axios.

Continue reading

Colleges (And Law Schools) Are Rushing To Respond To ChatGPT

Chronicle of Higher Education, ChatGPT Is Everywhere:

It’s hard to believe that ChatGPT appeared on the scene just three months ago, promising to transform how we write. The chatbot, easy to use and trained on vast amounts of digital text, is now pervasive. Higher education, rarely quick about anything, is still trying to comprehend the scope of its likely impact on teaching — and how it should respond.

ChatGPT, which can produce essays, poems, prompts, contracts, lecture notes, and computer code, among other things, has stunned people with its fluidity, although not always its accuracy or creativity. To do this work it runs on a “large language model,” a word predictor that has been trained on enormous amounts of data. Similar generative artificial-intelligence systems allow users to create music and make art.

Many academics see these tools as a danger to authentic learning, fearing that students will take shortcuts to avoid the difficulty of coming up with original ideas, organizing their thoughts, or demonstrating their knowledge. Ask ChatGPT to write a few paragraphs, for example, on how Jean Piaget’s theories on childhood development apply to our age of anxiety and it can do that.

Other professors are enthusiastic, or at least intrigued, by the possibility of incorporating generative AI into academic life. Those same tools can help students — and professors — brainstorm, kick-start an essay, explain a confusing idea, and smooth out awkward first drafts. Equally important, these faculty members argue, is their responsibility to prepare students for a world in which these technologies will be incorporated into everyday life, helping to produce everything from a professional email to a legal contract.

But skeptics and fans alike still have to wrestle with the same set of complicated questions. Should instructors be redesigning their assignments and tests to reduce the likelihood that students will present the work of AI as their own? What guidance should students receive about this technology, given that one professor might ban AI tools and another encourage their use? Do academic-integrity policies need to be rewritten? Is it OK to use AI detectors? Should new coursework on AI be added and, if so, what form should it take?

For many, this is a head-spinning moment.

Continue reading

Does ChatGPT Produce Fishy Briefs?

ABA Journal, Does ChatGPT Produce Fishy Briefs?:

Lawyers are abuzz about the possible uses of ChatGPT. Could the artificial intelligence-powered chatbot write a persuasive legal brief worthy of judicial consideration? Given its limitations, we believe that’s unlikely. ChatGPT, a large language model developed by the San Francisco company OpenAI that launched in November, can draw only on sources available on the web; it cannot crawl appellate records or access subscription-based services such as Westlaw. Still, the ABA Journal decided to put the technology to the test just for kicks. ...

Continue reading

ChatGPT's Tax Advice Was Wrong 100% Of The Time

Accountants' World visitors: if you would like to receive a daily email with links to that day's tax posts, please email me.

Bloomberg, ChatGPT Can Give Tax Advice, But You Really Get What You Pay For:

ChatGPT can write essays, poetry, letters, and more, so we posed the question—can it do our taxes? Our company, TaxBuzz, decided to take the technology for a test drive, asking, “What status I should use when filing my taxes?”

It offered typical definitions of the IRS’ various filing options. Then came the best advice of the day: “You can consult a tax professional to help determine the best option for you.” While ChatGPT might be able to give you a CliffsNotes-style answer to your query, expecting a how-to tip or complete strategy isn’t in the cards. ...

When it comes to analyzing the tax code, expertly trained humans have access to trusted and verified libraries of information that they can use to verify tax strategy claims. ChatGPT simply doesn’t have the same level of brainpower or education as a human tax professional. ...

Continue reading

It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship

Chronicle of Higher Ed Op-Ed:  It’s Not Just Our Students — ChatGPT Is Coming for Faculty Writing, by Ben Chrisinger (Oxford; Google Scholar):

Almost immediately after OpenAI released ChatGPT in late November, people began wondering what it would mean for teaching and learning. A widely read piece in The Atlantic that provided one of the first looks at the tool’s ability to put together high-quality writing concluded that it would kill the student essay. Since then, academics everywhere have done their own experimenting with the technology — and weighed in on what to do about it. Some have banned students from using it, while others have offered tips on how to create essay assignments that are AI-proof. Many have suggested that we embrace the technology and incorporate it into the classroom.

While we’ve been busy worrying about what ChatGPT could mean for students, we haven’t devoted nearly as much attention to what it could mean for academics themselves. And it could mean a lot. Critically, academics disagree on exactly how AI can and should be used. And with the rapidly improving technology at our doorstep, we have little time to deliberate.

Continue reading

Bilek & Merritt: ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More

Bloomberg Law Op-Ed:  ChatGPT Almost Passed the Bar, But Competent Lawyers Do Much More, by Mary Lu Bilek (CUNY) & Deborah Merritt (Ohio State):

ChatGPT, OpenAI’s provocative artificial intelligence program, has come close to passing the multiple-choice portion of the bar exam. The bot has also earned passing grades on law school essays that resemble ones written for the exam. ...

Yet, even in a bill-by-the-hour world, clients who can afford it will still seek out human lawyers. Why? Because humans are far better than bots at eliciting facts and goals from clients, identifying new avenues of research, and solving multi-dimensional problems. Human experts will supplement those advantages by knowing when to consult AI, how to assess AI responses, and how to integrate AI knowledge with the human dimensions of a client problem. ...

Continue reading

Bishop Posts Two Papers On ChatGPT

Lea Bishop (Indiana-McKinney): 

Can ChatGPT 'Think Like a Lawyer?' A Socratic Dialogue:

A witty socratic dialogue with a language-generation model, exploring the aims of legal education in the new era of machine writing.

If you are a lawyer, a law student, or a law professor, you already have a good understanding of what it means to “think like a lawyer.”

For everyone else, there is ChatGPT. Hey, ChatGPT, give me a hand here…Let’s have a Socratic Dialogue. I’ll type a question, and you do your best to answer it. Your answers don’t need to be perfect. Because I want my readers to see exactly how you do what you do, I am not going to edit your responses at all. Okay, let’s go…

A Computer Wrote this Paper: What ChatGPT Means for Education, Research, and Writing:

Continue reading

Ryznar: Exams In The Time Of ChatGPT

Margaret Ryznar (Indiana-McKinney), Exams in the Time of ChatGPT, 80 Wash. & Lee L. Rev. Online __ (2023):

This article offers various methods to administer assessments while maintaining their integrity—after asking artificial intelligence writing tool ChatGPT for its views on the matter. The sophisticated response of the chatbot, which students can use in their written work, only raises the stakes of figuring out how to administer exams fairly.

Continue reading

Alarie: The Rise Of The Robotic Tax Analyst

Benjamin Alarie (Toronto; Google Scholar; CEO, Blue J Legal), The Rise of the Robotic Tax Analyst, 178 Tax Notes Fed. 57 (Jan. 2, 2023):

As a bold taxwriting experiment, this installment of Blue J Predicts has been generated with the help of an AI assistant, OpenAI’s “Generative Pre-Trained Transformer 3” (GPT-3). GPT-3 is a large language model developed by OpenAI and backed by Microsoft. It is an inexhaustible generator of text and can write with accuracy in English about almost any topic. It has performed its duty, with my human companionship, indefatigably.

This isn’t the first time that a legal academic has invoked a robotic coauthor, and I expect that these kinds of tools will become increasingly commonplace. I expect that eventually they will be about as remarkable as using a spelling or grammar checker. At this moment, however, before the rise of the robotic tax analyst, using GPT-3 to help write this article is likely to raise some eyebrows.

In October 2021 I produced a peer-reviewed law review article with my academic colleague, the late (and great) tax law professor Arthur Cockfield of Queen’s University. The article was notable for being the first peer-reviewed law review article to extensively leverage GPT-3 in its production. In that article — after a short introduction penned by us, Benjamin Alarie and Cockfield — we gave GPT-3 control of the metaphorical keyboard and allowed it to produce its textual analysis uninterrupted and unedited.

The results were mixed and intriguing, and certainly pointed in the direction of future possibility. In our view, GPT-3 had potential. In the article, we speculated on the future of AI in legal scholarship and asked provocatively in the title, “Will Machines Replace Us?” Cockfield and I concluded that “although GPT-3 is not up to the task of replacing law review authors currently, we are far less confident that GPT-5 or GPT-100 might not be up to the task in future.”

Continue reading

ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax)

Following up on this morning's post, ChatGPT Gets B|B- Grade On Wharton MBA Exam:  Jonathan Choi (Minnesota; Google Scholar), Kristin Hickman (Minnesota; Google Scholar), Amy Monahan (Minnesota) & Daniel Schwarcz (Minnesota; Google Scholar), ChatGPT Goes to Law School:

How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.

Continue reading

ChatGPT Gets B|B- Grade On Wharton MBA Exam

Following up on this morning's post, ChatGPT Gets C+ Grade On Four Minnesota Law School Exams:  Christian Terwiesch (Wharton; Google Scholar), Would Chat GPT3 Get a Wharton MBA?:

OpenAI’s Chat GPT3 has shown a remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers, and consultants. Chat GPT3 has demonstrated the capability of performing professional tasks such as writing software code and preparing legal documents. The purpose of this paper is to document how Chat GPT3 performed on the final exam of a typical MBA core course, Operations Management. Exam questions were uploaded as used in a final exam setting and then graded.

Continue reading

Using ChatGPT To Write Law School Exams, Bar Exams, And Strategic Plans

Reuters, Some Law Professors Fear ChatGPT's Rise as Others See Opportunity:

The artificial intelligence program ChatGPT came up short last month on the multiple choice portion of the bar exam.

The free chatbot from OpenAI performed better than predicted, however, earning passing scores on evidence and torts. The academics behind the experiment expect it will pass the attorney licensing test someday.

Law professors are among those both alarmed and delighted by ChatGPT since its November release. The program generates sophisticated, human-like responses based on requests from users and mountains of data, including from legal texts.

Continue reading

Blackman: GPT Will Soon Be Able To Pass The Multistate Bar Exam

Following up on my previous posts (links below):  Josh Blackman (South Texas; Google Scholar), Can GPT Pass the Multistate Bar Exam?:

My frequent co-authors, Mike Bommarito and Dan Katz utilized a different software tool from OpenAI, known as GPT-3.5, to answer the multiple choice questions on the Multistate Bar Examination (MBE). If there are four choices, the "baseline guessing rate" would be 25%. With no specific training, GPT scored an overall accuracy rate of 50.3%. That's better than what many law school graduates can achieve. And in particular, GPT reached the average passing rate for two topics: Evidence and Torts. (I'll let Evidence or Torts scholars speculate about why those topics may be easier for AI.) Here is a summary of the results from their paper:

The table and figure clearly show that GPT-3.5 is not yet passing the overall multiple choice exam. However, GPT-3.5 is significantly exceeding the baseline random chance rate of 25%. Furthermore, GPT-3.5 has reached the average passing rate for at least two categories, Evidence and Torts. 

On average across all categories, GPT-3.5 is trailing human test-takers by approximately 17%. In the case of Evidence, Torts, and Civil Procedure, this gap is negligible or in the single digits; at 1.5 times the standard error of the mean across our test runs, GPT-3.5 is already at parity with humans for Evidence questions. However, for the remaining categories of Constitutional Law, Real Property, Contracts, and Criminal Law, the gap is much more material, rising as high as 36% in the case of Criminal Law. ...

Continue reading

Blackman: ChatGPT And Law School Exams

Following up on my previous post, The Implications Of OpenAI’s Assistant For Legal Services And Society:  Josh Blackman (South Texas; Google Scholar), Plagiarism and ChatGPT:

ChatGPT ... uses sophisticated technology to generate answers to questions. These responses are written in plain English, that are easy to understand, and incorporate information from a massive neural network. These responses are not perfect, but may pass muster with professors who are short on time. The temptation for cheating is real. And one professor in South Carolina caught plagiarism. He wrote about it on Facebook, and the New York Post followed up.

This technology should strike fear in all academics.

Continue reading

The Implications Of OpenAI’s Assistant For Legal Services And Society

Andrew M. Perlman (Dean, Suffolk; Google Scholar) & Open AI's Assistant, The Implications of OpenAI’s Assistant for Legal Services and Society:

On November 30, 2022, OpenAI released a chatbot called ChatGPT. To demonstrate the chatbot’s remarkable sophistication and its potential implications, both for legal services and society more generally, a human author generated this paper in about an hour through prompts within ChatGPT. Only this abstract, the outline headers, and the prompts were written by a person. ChatGPT generated the rest of the text with no human editing.

Continue reading

A Human Being Wrote This Law Review Article: GPT-3 And The Practice Of Law

Amy B. Cyphert (ASPIRE), A Human Being Wrote This Law Review Article: GPT-3 and the Practice of Law, 55 UC Davis L. Rev. 401 (2021):

Artificial intelligence tools can now “write” in such a sophisticated manner that they fool people into believing that a human wrote the text. None are better at writing than GPT-3, released in 2020 for beta testing and coming to commercial markets in 2021. GPT-3 was trained on a massive dataset that included scrapes of language from sources ranging from the NYTimes to Reddit boards. And so, it comes as no surprise that researchers have already documented incidences of bias where GPT-3 spews toxic language. But because GPT-3 is so good at “writing,” and can be easily trained to write in a specific voice — from classic Shakespeare to Taylor Swift — it is poised for wide adoption in the field of law. 

This Article explores the ethical considerations that will follow from GPT-3’s introduction into lawyers’ practices. GPT-3 is new, but the use of AI in the field of law is not. AI has already thoroughly suffused the practice of law. GPT-3 is likely to take hold as well, generating some early excitement that it and other AI tools could help close the access to justice gap. That excitement should nevertheless be tempered with a realistic assessment of GPT-3’s tendency to produce biased outputs. 

Continue reading