ChatGPT-4 Passes Multistate Professional Responsibility Exam (MPRE), Beats Law School Graduates And Other Generative AI Models

Gabor Melli (LegalOn Technologies), Daniel Lewis (LegalOn Technologies) & Dru Stevenson (South Texas; Google Scholar), Generative AI Passes the Legal Ethics Exam:

Can Generative AI Pass the Legal Ethics Exam?
Earlier this year, research found that GPT-4 could surpass law students in passing the Uniform Bar Exam. Our study builds on this discovery, testing whether generative AI models can also navigate the rules and fact patterns around legal ethics.

• We challenged OpenAI's GPT-4 and GPT-3.5, Anthropic's Claude 2, and Google's PaLM 2 Bison to 100 simulated exams, composed of questions crafted by Professor Stevenson to model the Multistate Professional Responsibility Exam (MPRE).
• GPT-4 performed best, answering 74% of questions correctly, an estimated 6% better than the average human test-taker.
• GPT-4 and Claude 2 both scored above the approximate passing threshold for the MPRE, estimated to range between 56-64% depending on the jurisdiction.

Continue reading

Alarie Presents The Impact Of AI On Legal Education And Law Practice At Pepperdine

Benjamin Alarie (Toronto; Google Scholar) presents The Legal Singularity: How Artificial Intelligence Can Make Law Radically Better (with Abdi Aidid (Toronto)) at Pepperdine today as part of its Dean's Speaker Series: 

Law today is incomplete, inaccessible, unclear, underdeveloped, and often perplexing to those whom it affects. In The Legal Singularity, Abdi Aidid and Benjamin Alarie argue that the proliferation of artificial intelligence–enabled technology — and specifically the advent of legal prediction — is on the verge of radically reconfiguring the law, our institutions, and our society for the better.

Revealing the ways in which our legal institutions underperform and are expensive to administer, the book highlights the negative social consequences associated with our legal status quo. Given the infirmities of the current state of the law and our legal institutions, the silver lining is that there is ample room for improvement. With concerted action, technology can help us to ameliorate the problems of the law and improve our legal institutions. Inspired in part by the concept of the "technological singularity," The Legal Singularity presents a future state in which technology facilitates the functional "completeness" of law, where the law is at once extraordinarily more complex in its specification than it is today, and yet operationally, the law is vastly more knowable, fairer, and clearer for its subjects. Aidid and Alarie describe the changes that will culminate in the legal singularity and explore the implications for the law and its institutions.

Continue reading

An Empirical Analysis Of Practice-Based Technology Training In Law Schools

Jessica De Perio Wittman (Connecticut; Google Scholar) & Katie Brown (Charleston; Google Scholar), Taking on the Ethical Obligation of Technology Competency in the Academy: An Empirical Analysis of Practice-Based Technology Training Today, 36 Geo. J. Legal Ethics 1 (2023):

Today’s lawyers must be technologically competent, per Model Rule of Professional Conduct 1.1. Law schools and law firms were keenly aware of this expectation and summarily responded. While law firms offered more professional development opportunities, law schools began offering various courses focusing on technology skills. These courses have increased and evolved over time as the curriculum has changed with the technology.

Continue reading

Generative AI In Law Schools And Colleges

ABA Journal, If Law Schools Prohibit ChatGPT in Writing, Can They Back It Up?:

Using artificial intelligence to write admissions essays now comes with significant risks at the University of Michigan Law School, which recently asked applicants to certify that they did not use the technology for drafting purposes.

False statements could result in the cancellation of an admissions offer or expulsion or rescinding a degree, according to the certification language.

Likewise, if admissions office readers go over a candidate’s essay and suspect that technology did the writing, it would give them serious pause, even for a strong candidate, says Sarah C. Zearfoss, the law school’s senior assistant dean. The certification language was introduced this year, and she has not heard of other law schools with similar prohibitions on AI technology.

Many lawyers interviewed by the ABA Journal found the University of Michigan Law School’s new policy surprising. The retributions would be difficult to carry out because there are no good tools to detect the technology in writing, they say. And even if there were, offerings such as ChatGPT will continue to evolve and likely outfox anything created to catch it. ...

Robert Brain, a professor at the Loyola Marymount University’s Loyola Law School, told the Journal that most, if not all, law schools currently have faculty committees considering the issue. ...

“I don’t think anybody is seriously suggesting going back to handwritten blue book essays, but they could cut off take-home tests,” says Brain, who thinks that law schools should teach students to use AI technology. “My personal view is we can’t stop them but also because lawyers are using it,” he says.

Bloomberg Law Op-Ed:  Legal Education in World of AI Should Follow Medical Model, by Marsha Cohen (UC Law-San Francisco):

What would “medical model” legal education look like?

Lawyers for America, invented at UC Hastings (now UC Law San Francisco), is a model that provides students with a different route to learning to be a lawyer while simultaneously helping to improve our country’s massive access to justice problem.

Ponder: Two years of classroom learning and a third year devoted entirely to a well-supervised externship with a legal nonprofit or government legal office. Given the current bar exam, summer is then devoted to study and the test, and then after the bar exam the fellows return for a full year of fellowship-paid work.

Continue reading

Abandoning Precedent: The Case For Bringing ChatGPT Into Law Schools

David S. Kemp (The Verdict), Abandoning Precedent: The Case for Bringing ChatGPT into Law Schools:

The legal writing classes required of first-year law students tend to polarize students. Some love legal writing, deriving pleasure from the highly formulaic yet eminently adaptable paradigms of legal analysis. Others loathe it, feeling that it stifles creativity, imposing a repetitive and cumbersome structure on what might otherwise be a compelling and concise argument. As a law student, I fell into the former category, but when I became a legal writing professor, I found myself somewhere between the two.

When ChatGPT was released to the public in November 2022, it was similarly polarizing—at least among academic administrators and instructors. Most responded to its arrival with dismay. Renowned linguist Noam Chomsky opined that ChatGPT doesn’t have “anything to do with education except undermining it.” Many school systems and at least one top-tier law school immediately adopted policies to ban its use on written assignments and exams, at least until it was better understood. Some law professors who historically gave open-book final exams expressed grave concerns that students could use ChatGPT to produce decent, if not good, exam answers. Some were more ambivalent, and a minority hailed ChatGPT as a robust tool that could reduce the tedium of law practice. Nearly all feared that students would grow to rely on ChatGPT rather than learn the critical skills of legal analysis and writing. Many educators whose institutions did not globally prohibit ChatGPT did so in their own classrooms and carried on as usual. Others ignored it altogether. I—and a few others—embraced the nascent technology and quickly began discussing it in the classroom.

As someone who fell in love with diagramming sentences in seventh grade and fostered that romance for over a quarter-century to become a writing instructor, I viewed ChatGPT with excitement and trepidation. In ChatGPT, I saw potential. I saw a tool that could almost effortlessly merge the science of technology and art of writing. But most importantly, I saw an opportunity to teach my students not just how to write, but how to adapt to the rapidly changing landscape of legal practice. ...

Continue reading

Can ChatGPT-4 Really Do Tax?

Andrew Blair-Stanek (Maryland; Google Scholar), Nils Holzenberger (Institut Polytechnique de Paris) & Benjamin Van Durme (Johns Hopkins; Google Scholar), OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?, 180 Tax Notes Fed. 1101 (Aug. 14, 2023):

In the livestream introducing GPT-4, OpenAI used one of our SARA [acronym for StAtutory Reasoning Assessment] tax cases verbatim, describing it as a real tax example, even though SARA is a simplified academic data set. In the demo, OpenAI also used our heavily edited SARA version of the IRC. OpenAI incorrectly thought GPT-4 had correctly calculated the tax liability because its answer matched the SARA answer, although our IRC edits change the result from the actual IRC. We tested GPT-4 on the entire SARA data set. It gets tax liabilities exactly right around one-third of the time and miscalculates tax liabilities by over 10 percent nearly a quarter of the time. GPT-4 often misreads even our simplified version of the IRC. In the livestream, the presenter warned, “You should always check with your tax adviser.” Wise advice. ...

Continue reading

AI Assistance In Legal Analysis: An Empirical Study

Jonathan H. Choi (USC; Google Scholar) & Daniel Schwarcz (Minnesota; Google Scholar), AI Assistance in Legal Analysis: An Empirical Study:

Can artificial intelligence (AI) augment human legal reasoning? To find out, we designed a novel experiment administering law school exams to students with and without access to GPT-4, the best-performing AI model currently available. We found that assistance from GPT-4 significantly enhanced performance on simple multiple-choice questions but not on complex essay questions. We also found that GPT-4’s impact depended heavily on the student’s starting skill level; students at the bottom of the class saw huge performance gains with AI assistance, while students at the top of the class saw performance declines. This suggests that AI may have an equalizing effect on the legal profession, mitigating inequalities between elite and nonelite lawyers. In addition, we graded exams written by GPT-4 alone to compare it with humans alone and AI-assisted humans.

Continue reading

The Legal Singularity: How Artificial Intelligence Can Make Law Radically Better

Abdi Aidid (Assistant Professor, University of Toronto Faculty of Law; Specialist, Legal Innovation, Blue J Legal) & Benjamin Alarie (Osler Chair in Business Law, University of Toronto Faculty of Law; CEO, Blue J Legal), The Legal Singularity: How Artificial Intelligence Can Make Law Radically Better (University of Toronto Press 2023):

Law today is incomplete, inaccessible, unclear, underdeveloped, and often perplexing to those whom it affects. In The Legal Singularity, Abdi Aidid and Benjamin Alarie argue that the proliferation of artificial intelligence–enabled technology — and specifically the advent of legal prediction — is on the verge of radically reconfiguring the law, our institutions, and our society for the better.

Revealing the ways in which our legal institutions underperform and are expensive to administer, the book highlights the negative social consequences associated with our legal status quo. Given the infirmities of the current state of the law and our legal institutions, the silver lining is that there is ample room for improvement. With concerted action, technology can help us to ameliorate the problems of the law and improve our legal institutions. Inspired in part by the concept of the "technological singularity," The Legal Singularity presents a future state in which technology facilitates the functional "completeness" of law, where the law is at once extraordinarily more complex in its specification than it is today, and yet operationally, the law is vastly more knowable, fairer, and clearer for its subjects. Aidid and Alarie describe the changes that will culminate in the legal singularity and explore the implications for the law and its institutions.

Continue reading

ABA Weighs New Free Speech Rule For Law Schools

Reuters, ABA Weighs New Free Speech Rule for Law Schools:

The American Bar Association may soon require law schools to adopt free speech policies, a change that follows several high-profile campus incidents in which students disrupted controversial speakers.

The ABA’s Council of the Section of Legal Education and Admissions to the Bar, which oversees law school accreditation, on Friday will consider a new rule mandating “written policies that encourage and support the free expression of ideas.”

Continue reading

Chat GPT Makes A Better Therapist Than A Lawyer

Diana Simon (Arizona), More True Confessions of a Legal Writing Professor: Chat GPT Makes a Better Therapist Than a Lawyer:

This, at times, irreverent, tongue-in-cheek article is about how Chat GPT (and generative AI in general) makes a better therapist than a lawyer. First, it references examples where generative AI has hallucinated cases, refers to the real-life example where an attorney was sanctioned for submitting fake cases, and refers to the court orders that followed where courts have limited the use of generative AI. Second, it discusses an example of how Google Bard missed a key issue when writing a case brief for first-semester law students and made improvements to an excellent office memo that a student prepared in a way that did not conform to best practices in legal writing.

Continue reading

Five Ways Generative AI Is Finding Its Way Into Law School

Legal Tech News, 5 Ways Generative AI Is Finding Its Way Into Law School:

Generative artificial intelligence quickly permeated through many areas of the legal industry, from its integration to many legal technology solutions, to the new legal challenges facing lawyers and their clients.

But in face of this rapid adoption, the legal education around the potential, and risks, that this new technology presents is struggling to keep up—leaving some attorneys leveraging the wrong tools in the courtroom.

However, some law schools quickly identified such gaps and are now providing additional courses, free resources and training programs to prepare the next generation of legal professionals to work alongside generative AI. ...

Legaltech News looked at five ways generative AI is finding its place in law schools in the U.S.

Continue reading

Unlike Michigan, Arizona State Allows Use Of ChatGPT On Law School Applications

Law.com, ASU Law School Welcomes Use of AI Tools on Admission Applications:

After the University of Michigan Law School announced last week that it’s banning the use of artificial intelligence tools for admission applications, the Sandra Day O’Connor College of Law at Arizona State University announced late Thursday that it invites prospective law students to use these tools.

Applicants to its degree programs are permitted to use generative artificial intelligence (AI) in the preparation of their application and certify that the information they submit is accurate, beginning in August, according to ASU Law’s announcement.

“Arizona State University is an institution that embraces change and innovation in all forms,” so the decision to permit the usage of AI tools “will allow for the responsible use of this rapidly evolving technology when future students prepare their application materials,” an ASU spokesperson said in an email to Law.com Thursday.

Continue reading

Is ChatGPT A Viable Research Alternative To LexisNexis|Westlaw For Some Lawyers?

Legal Tech News, Are Legal Research Costs Tempting Lawyers to Turn to ChatGPT?:

Earlier this year, the legal community balked at an error that has since become a clarion call for caution when using generative artificial intelligence tools. In May, an attorney used ChatGPT for legal research and filed a brief in federal court that was rife with fake case citations the chatbot had fabricated.

The news caused an uproar, leading to sanctions for the attorney in question, and pushing some judges to require the disclosure of use of generative AI in court filings. Still, it’s evident that when it comes to legal research, attorneys aren’t exactly spoiled for choice.

On one end exists ChatGPT—unreliable but free, or at very low cost—while on the other end are established legal research tools, such as Thomson Reuters’ Westlaw and the LexisNexis suite, which are more costly.

Continue reading

Crespi: How To Grade Law Students In 'Paper Classes' In The Age Of AI

Gregory S. Crespi (SMU; Google Scholar), How To Grade Law Students Taking 'Paper Classes' Given Student Access to Artificial Intelligence Programs: Some Preliminary Thoughts:

This short paper presents and discusses a new approach to teaching “paper classes” in law school, given current student access to artificial intelligence programs such as ChatGPT. These programs are now capable of producing research papers in response to carefully structured prompts that with only minor editing are generally sufficient to obtain at least minimal passing grades, and these programs are improving rapidly in their capabilities. This approach attempts to balance several different concerns that are raised by these programs. ...

Continue reading

Michigan Bans Use Of ChatGPT On Law School Applications

Reuters, Use of ChatGPT Prohibited on Michigan Law School Applications:

The admissions department at the University of Michigan Law School is looking for personal statements and application essays that reflect the traits and writing ability of aspiring attorneys.

But those essays better not have been written by ChatGPT.

Michigan Law has explicitly banned the use of the popular online chatbot and other forms of artificial intelligence tools on applications and now requires applicants to certify that they haven’t used those tools in the drafting process—an apparent first among law schools.

ABA Journal, T14 Law School Bans ChatGPT on Application Essays:

Continue reading

How ChatGPT Could Dramatically Reshape The Legal Industry

ABA, How ChatGPT and Other AI Platforms Could Dramatically Reshape the Legal Industry:

Hi, ChatGPT. What can ChatGPT do for lawyers (in 100 words or less)?

As an AI language model, ChatGPT can provide assistance to lawyers in various ways. It can help with legal research by quickly searching and analyzing large volumes of legal documents and cases to extract relevant information. It can also assist in drafting legal documents such as contracts, briefs and pleadings, by suggesting language and formatting options based on context and legal precedent. Additionally, ChatGPT can assist with legal analysis by providing insights and recommendations on complex legal issues based on its vast database of legal information.

Is ChatGPT trying to take lawyers’ jobs away (in 100 words or less)?

Continue reading

Dorf On ChatGPT: Is Resistance To AI In The Law School Classroom Futile?

The Verdict Op-Ed:  Is Resistance to AI in the Law School Classroom Futile?, by Michael C. Dorf (Cornell; Google Scholar):

As AI becomes integrated into more and more of the productivity tools that lawyers and other professionals routinely use, banning its use in the taking of exams will make less and less sense. Eventually, we will need to learn how to evaluate students’ ability to use AI and other tools effectively. More importantly, we will need to figure out how to teach students to use AI tools effectively.

Before any of that can happen, however, we will need to see how lawyers use AI in practice. It’s sensible for professional education to aim at preparing students for the profession they’re entering. It’s challenging to do so when the nature of that profession is changing rapidly. AI-based tools for managing document discovery have been available to litigators for years, but how practitioners come to use the new generation of large language models and whatever comes after that remains to be seen.

Continue reading

Law Schools Must Embrace AI

National Law Journal Op-Ed:  Law Schools Must Embrace AI, by Joseph Landau (Fordham; Google Scholar) & Ron Lazebnik (Fordham):

Artificial intelligence tools have been around for some time, but the emergence of ChatGPT was a game changer. In legal education, AI holds immense potential to revolutionize our administration, pedagogy, programming, and research. The rise of intelligent machines offers us the opportunity to explore new teaching methods, such as personalized learning experiences, data-driven assessments, and adaptive curricula. By leveraging AI, we can equip our students with the skills necessary to thrive in a technology-driven legal profession.

With summer break in full swing, law schools can reflect on the past year’s experiments with the release of bots like ChatGPT to chart a path for the coming academic year and beyond. While the initial concerns, skepticism and frustration with generative AI in the immediate aftermath of ChatGPT are understandable, it is only by focusing on the convergence of legal technology and legal education that we can best prepare students consistent with our core goals, values and principles. In short, this is the time to engage in strategic thinking and planning so that our students can learn to navigate this evolving landscape now and make important contributions to their fields once they graduate.

Continue reading

Does ChatGPT Make Law School Worth Less As An Investment?

Jonathan Wolf (Rink Noonan, St. Cloud, MN), Do AI Large Language Models Like ChatGPT Make Law School Less Worth It As An Investment?:

Younger attorneys, despite their tech savvy, could very well find themselves completely displaced.

Artificial intelligence has been pilloried as a career killer across several industries, especially since the latest version of ChatGPT came out. The legal profession has not been immune, and with ChatGPT now able to pass the bar exam, a certain amount of anxiety is warranted. ...

What AI will do, in particular large language models like ChatGPT, is replace some of us, and some of what we do.

Continue reading

The ChatGPT Commencement Address

Inside Higher Ed, The ChatGPT Commencement Address:

As Wells College graduates gathered last month, President Jonathan Gibralter delivered a commencement address that sounded like countless others delivered across the country. It hit the usual themes: be prepared for challenges and setbacks, cultivate perseverance, and embrace opportunities.

But his speech culminated with a twist.

“Before I end this address, I wanted to let you know that everything I just said to you was written by artificial intelligence. I asked ChatGPT to write a commencement address from the president of Wells College to the graduates,” Gibralter revealed to a surprised audience.

“Why did I do that, and why am I telling you this? Because the world you are graduating into is changing very rapidly, and you have in front of you a world where electric vehicles will replace those powered by fossil fuels, a world where chat bots can replace the need for intellectual curiosity, a world in which some people reflect on their lives by the number of people who are connected to them on social media, a world where people may soon live in the metaverse, and the list goes on and on,” Gibralter told students, challenging them to stay intellectually curious.

Continue reading

NY Times: ChatGPT Lawyers Are Ordered To Pay $5,000 Fine, Consider Seeking Forgiveness

Following up on my previous posts:

• New York Times, Here’s What Happens When A Lawyer Uses ChatGPT (May 30, 2023)
• New York Times, The ChatGPT Lawyer Explains Himself In Court (June 12, 2023)

New York Times, ChatGPT Lawyers Are Ordered to Consider Seeking Forgiveness:

A Manhattan judge on Thursday imposed a $5,000 fine on two lawyers who gave him a legal brief full of made-up cases and citations, all generated by the artificial intelligence program ChatGPT.

The judge, P. Kevin Castel of Federal District Court, criticized the lawyers harshly and ordered them to send a copy of his opinion to each of the real-life judges whose names appeared in the fictitious filing.

But Judge Castel wrote that he would not require the lawyers, Steven A. Schwartz and Peter LoDuca, whom he referred to as respondents, to apologize to those judges, “because a compelled apology is not a sincere apology.”

“Any decision to apologize is left to respondents,” the judge added.

The discovery that ChatGPT had helped create the brief in an otherwise unremarkable lawsuit reverberated throughout the legal profession. The revelation also riveted the tech community, which has been debating the dangers of overreliance on artificial intelligence — even as a existential threat to humanity. ...

Continue reading

Lawsky Website With Practice Problems And Quizzes For Federal Income Tax

Sarah Lawsky (Northwestern; Google Scholar) has created a wonderful website that generates practice problems and quizzes for federal income tax. For more details, see the FAQ page.

NY Times: The ChatGPT Lawyer Explains Himself In Court

Following up on my previous post, New York Times, Here’s What Happens When A Lawyer Uses ChatGPT:  New York Times, The ChatGPT Lawyer Explains Himself:

In a cringe-inducing court hearing, a lawyer who relied on A.I. to craft a motion full of made-up case law said he “did not comprehend” that the chat bot could lead him astray.

As the court hearing in Manhattan began, the lawyer, Steven A. Schwartz, appeared nervously upbeat, grinning while talking with his legal team. Nearly two hours later, Mr. Schwartz sat slumped, his shoulders drooping and his head rising barely above the back of his chair.

For nearly two hours Thursday, Mr. Schwartz was grilled by a judge in a hearing ordered after the disclosure that the lawyer had created a legal brief for a case in Federal District Court that was filled with fake judicial opinions and legal citations, all generated by ChatGPT. The judge, P. Kevin Castel, said he would now consider whether to impose sanctions on Mr. Schwartz and his partner, Peter LoDuca, whose name was on the brief.

Continue reading

Four Tax Questions For ChatGPT And Other Language Models

Libin Zhang (Fried Frank, New York), Four Tax Questions for ChatGPT and Other Language Models, 179 Tax Notes Fed. 969 (May 8, 2023):

Since ChatGPT was released to the public in late 2022, it and other language models have rapidly gained recognition for their ability to respond to queries with natural language text. This article tests the usefulness of three language models, ChatGPT, Bing Chat, and Google Bard, by asking each four tax-related questions:

1. Can I engage in a section 1031 like-kind exchange if I sell my Picasso painting?
2. Does a decedent recognize gain if she dies with assets that have debt in excess of tax basis?
3. Why should the federal government tax stock buybacks?
4. Can you tell me six original jokes about tax lawyers?

Continue reading

Chat GPT-4 Understands Academic Attrition's Impact On Bar Passage But Does Anyone Else?

Rory D. Bahadur (Washburn) & Kevin Ruth (PhD Mathematics, Miami), Chat Gpt-4 Understands Academic Attrition's Impact on Bar Passage But Does Anyone Else?, 63 Washburn L.J. Online __ (2023):

Artificial Intelligence allows us to understand the relationship between bar passage and academic attrition. This relationship is often obscured by the continued conflation of the concepts of causation and correlation. Disentangling these two concepts is essential to refute the empirically baseless narrative that the bar exam tests competency and to expose the systemic racism and monopolistic business practices the above narrative perpetuates. 

Continue reading

Texas Federal Judge Requires All Lawyers To File Certificate Verifying Any Use Of AI

Following up on my previous post, New York Times, Here’s What Happens When A Lawyer Uses ChatGPT: 

Eugene Volokh (UCLA; Google Scholar), Federal Judge Requires All Lawyers to File Certificates Related to Use of Generative AI:

The certificates must "attest[] either that no portion of the filing was drafted by generative artificial intelligence (such as ChatGPT, Harvey.AI, or Google Bard) or that any language drafted by generative artificial intelligence was checked for accuracy, using print reporters or traditional legal databases, by a human being."

David Lat (Original Jurisdiction), Judge of the Week: Judge Brantley Starr:

Remember the debacle I discussed last week involving lawyers who filed a brief replete with multiple citations to fake cases, courtesy of ChatGPT? One judge is already taking action to prevent that from happening in his court.

As reported by Professor Eugene Volokh, Judge Brantley Starr (N.D. Tex.) issued a standing order for a “Mandatory Certification Regarding Generative Artificial Intelligence” ... Judge Starr identified two problems with AI tools: “hallucinations,” when they just make stuff up, and bias, when they “act according to computer code rather than conviction, based on programming rather than principle.”

Judge Starr’s order generated ample buzz, positive and negative.

Continue reading

Some Law Schools Already Are Using ChatGPT To Teach Legal Research And Writing

ABA Journal, Some Law Schools Already Are Using ChatGPT to Teach Legal Research and Writing:

ChatGPT, an artificial intelligence chatbot that can speak and write like humans, can be weak on facts. But it may already be a better wordsmith than some attorneys, says David Kemp, an adjunct professor at Rutgers Law School.

“If you’re asking it to organize several concepts or are struggling to explain something in a way that’s really understandable, it can help,” says Kemp, who also is the managing editor of Oyez, a multimedia website focused on U.S. Supreme Court opinions.

The technology seems to prefer active voice, as does Kemp. He introduced ChatGPT in an advanced legal writing class and plans to include it in a summer course about emerging technology.

Various law schools are following suit. Legal writing faculty interviewed by the ABA Journal agree that ChatGPT writing can model good sentence and paragraph structure. But some fear that it could detract from students learning good writing skills.

Continue reading

Re-Evaluating GPT-4's Bar Exam Performance

Following up on my previous post, GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam:  Eric Martínez (MIT; Google Scholar), Re-Evaluating GPT-4's Bar Exam Performance:

Perhaps the most widely touted of GPT-4's at-launch, zero-shot capabilities has been its reported 90th-percentile performance on the Uniform Bar Exam, with its reported 80-percentile-points boost over its predecessor, GPT-3.5, far exceeding that for any other exam. This paper investigates the methodological challenges in documenting and verifying the 90th-percentile claim, presenting four sets of findings that suggest that OpenAI's estimates of GPT-4's UBE percentile, though clearly an impressive leap over those of GPT-3.5, appear to be overinflated, particularly if taken as a “conservative” estimate representing “the lower range of percentiles,” and moreso if meant to reflect the actual capabilities of a practicing lawyer.

Continue reading

NY Times: Here’s What Happens When A Lawyer Uses ChatGPT

New York Times, Here’s What Happens When Your Lawyer Uses ChatGPT:

The lawsuit began like so many others: A man named Roberto Mata sued the airline Avianca, saying he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York.

When Avianca asked a Manhattan federal judge to toss out the case, Mr. Mata’s lawyers vehemently objected, submitting a 10-page brief that cited more than half a dozen relevant court decisions. There was Martinez v. Delta Air Lines, Zicherman v. Korean Air Lines and, of course, Varghese v. China Southern Airlines, with its learned discussion of federal law and “the tolling effect of the automatic stay on a statute of limitations.”

There was just one hitch: No one — not the airline’s lawyers, not even the judge himself — could find the decisions or the quotations cited and summarized in the brief.

That was because ChatGPT had invented everything.

The lawyer who created the brief, Steven A. Schwartz of the firm Levidow, Levidow & Oberman, threw himself on the mercy of the court on Thursday, saying in an affidavit that he had used the artificial intelligence program to do his legal research — “a source that has revealed itself to be unreliable.”

Continue reading

ABA Increases Law School Online Course Cap From 33% To 50%

Reuters, Law Students Can Take 50% of Classes Online, With ABA Rule Change:

The arm of the American Bar Association that accredits law schools on Friday voted to increase its limit on distance education courses from one-third of a student’s total credits to half. It also eliminated the rule that students take no more than 10 credit hours of distance education classes during their first year of law school.

Those changes bring the law school accreditation standards in line with distance education limits set by the U.S. Department of Education — which has tasked the ABA with overseeing law schools — and will reduce the number of law schools applying to the ABA for special approval to offer distance education programs that exceed the one-third credit limit.

ABA Journal:

Continue reading

NY Times Op-Ed: A.I. Could Be A Boon To (Legal) Education

New York Times Op-Ed:  A.I. Could Actually Be a Boon to Education, by Peter Coy:

Sal Khan is an uber-nerd, and that’s why people love him. In early 2020, to demonstrate that Khan Academy, his platform for free online education, was a worthy cause, he didn’t produce a brochure with smiling children and glowing testimonials. He made one of his teaching videos, with the usual black screen, colored pens and handwritten equations. The phrase “standard deviation” came up a lot. He multiplied the number of highly active users of his program by their average academic improvement, then linked that to the resulting increase in their projected lifetime earnings, compared it to operating costs and, presto, showed a benefit-cost ratio of 480 to 1, or around 240 to 1 if calculating only the benefits to students in high-need schools.

Those are crazy high benefit-cost ratios, considering that a lot of nonprofits would be happy with a ratio of 10 to 1. I’m not vouching for the calculation, though it seems reasonable at a glance. I’m citing it because Khan has a new plan to make the ratio, whatever it is, even higher. It’s all about harnessing artificial intelligence. I watched a TED Talk he gave on April 18 about his A.I. plans, and I interviewed him last week to get more information.

Continue reading

ChatGPT Bombs Accounting And Tax Exams: 47% v. 77% For Students

BYU News, ChatGPT Can’t Ace This Test, But Experts Think It Soon Will … What It Means For Teaching:

Last month, OpenAI launched its newest AI chatbot product, GPT-4. According to the folks at OpenAI, the bot, which uses machine learning to generate natural language text, passed the bar exam with a score in the 90th percentile, passed 13 of 15 AP exams and got a nearly perfect score on the GRE Verbal test.

Inquiring minds at BYU and 186 other universities wanted to know how OpenAI’s tech would fare on accounting exams. So, they put the original version, ChatGPT, to the test. The researchers say that while it still has work to do in the realm of accounting, it’s a game changer that will change the way everyone teaches and learns — for the better.

“When this technology first came out, everyone was worried that students could now use it to cheat,” said lead study author David Wood, a BYU professor of accounting. “But opportunities to cheat have always existed. So for us, we’re trying to focus on what we can do with this technology now that we couldn’t do before to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening.” ...

His co-author recruiting pitch on social media exploded: 327 co-authors from 186 educational institutions in 14 countries participated in the research, contributing 25,181 classroom accounting exam questions. They also recruited undergrad BYU students (including Wood’s daughter, Jessica) to feed another 2,268 textbook test bank questions to ChatGPT. The questions covered accounting information systems (AIS), auditing, financial accounting, managerial accounting and tax, and varied in difficulty and type (true/false, multiple choice, short answer, etc.).

Although ChatGPT’s performance was impressive, the students performed better. Students scored an overall average of 76.7%, compared to ChatGPT’s score of 47.4%. On a 11.3% of questions, ChatGPT scored higher than the student average, doing particularly well on AIS and auditing. But the AI bot did worse on tax, financial, and managerial assessments, possibly because ChatGPT struggled with the mathematical processes required for the latter type.

The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?:

Continue reading

NY Times: Conservatives Aim To Build A Chatbot Of Their Own

New York Times, Conservatives Aim to Build a Chatbot of Their Own:

When ChatGPT exploded in popularity as a tool using artificial intelligence to draft complex texts, David Rozado decided to test its potential for bias. A data scientist in New Zealand, he subjected the chatbot to a series of quizzes, searching for signs of political orientation.

The results, published in a recent paper were remarkably consistent across more than a dozen tests: “liberal,” “progressive,” “Democratic” [The Political Biases of ChatGPT].

So he tinkered with his own version, training it to answer questions with a decidedly conservative bent. He called his experiment RightWingGPT.

Continue reading

AI Tools for Lawyers: A Practical Guide

Daniel Schwarcz (Minnesota; Google Scholar) & Jonathan H. Choi (Minnesota; Google Scholar), AI Tools for Lawyers: A Practical Guide:

This Article provides lawyers and law students with practical and specific guidance on how to effectively use AI large language models (LLMs), like GPT-4, Bing Chat, and Bard, in legal research and writing. Focusing on GPT-4 – the most advanced LLM that is widely available at the time of this writing – it emphasizes that lawyers can use traditional legal skills to refine and verify LLM legal analysis. In the process, lawyers and law students can effectively turn freely-available LLMs into highly productive personal legal assistants.

Continue reading

ChatGPT Gets 148 (37th Percentile) And 157 (70th Percentile) On The LSAT

Press Release, ChatGPT LSAT Score Falls Short of Getting Into Top Law Schools:

Blueprint Prep, a premier test preparation and professional training company, was among the first test prep companies to analyze the LSAT results of #OpenAI's ChatGPT, the artificial intelligence chatbot capable of accurately responding to human language. The chatbot's LSAT results revealed that ChatGPT-3.5 series was unable to meet the typical scores required for admission to a top 14 law school.

A veteran Blueprint Prep LSAT instructor analyzed the chatbot's test results and discovered its chief errors lay in a failure to consistently apply logical and critical reasoning, as well as an inability to distinguish essential information from superfluous material. These are areas where input from an experienced instructor would help ChatGPT better understand the questions and how to interpret them. Despite its less-than-stellar scores, however, ChatGPT was convinced while taking the test that it was answering questions correctly.

"We know that the LSAT is designed to measure a student's analytical reasoning, critical thinking, and reading comprehension skills, which are essential for success in law school," said Gene Suhir, LSAT Academic Manager at Blueprint Prep. "These skills can be significantly strengthened, but to do so requires leveraging proven strategies and top-scoring instructors who can help students get into the mindsets of the test makers. This form of LSAT test prep not only enables the student to process information like a lawyer would, but it's been proven that strengthening reasoning skills via LSAT test prep can help wire students' brains to think more like a lawyer. This is not the specific kind of reasoning that ChatGPT is innately useful for, although it can learn these skills."

The LSAT is scored on a scale of 120-180, with 151 being roughly average. Students usually need a typical score in the 170s to be admitted to one of the nation's top 14 law schools. On the two tests it took, ChatGPT scored a 148 (37th percentile) and a 157 (70th percentile). [148 is the median LSAT score for law schools ranked 187th in this metric for the current 1L class (North Carolina Central, Ohio Northern, Oklahoma City, Western Michigan, Widener (PA)). 157 is the median LSAT score for law schools ranked 98th in this metric for the current 1L class (Kentucky, LSU, Michigan State, Mississippi, Regent, Seattle, Syracuse, Texas Tech).]

Continue reading

Turnitin Plagiarism Detector Will Catch Students Who Cheat With ChatGPT With 98% Accuracy

Chronicle of Higher Education, A Plagiarism Detector Will Try to Catch Students Who Cheat With ChatGPT:

As faculty continue to debate how artificial intelligence might disrupt academic integrity, the popular plagiarism-detection service Turnitin announced on Monday that its products will now detect AI-generated language in assignments.

Turnitin’s software scans submissions and compares them to a database of past student essays, publications, and materials found online, and then generates a “similarity report” assessing whether a student inappropriately copied other sources.

The company says the new feature will allow instructors to identify the use of tools like ChatGPT with “98-percent confidence.”

Continue reading

Should ChatGPT Be In Law School?

Stephen Embry (TechLaw Crossroads), Should ChatGPT Be In Law School?:

Lots of questions and unease surround the use of ChatGPT in the classroom and education. The issue may be particularly acute for law schools and professors. Law schools are charged with teaching core legal concepts that (should) equip students to practice law.

Many law school professors reportedly question how they can do that if students can have the concepts laid out for them by ChatGPT. Even ChatGPT questions how this can be done if students can use ChatGPT on such things as exams.

Says ChatGPT, “Law school exams are typically designed to test a student’s ability to think critically and apply legal reasoning, which is something that cannot be easily replicated by an AI language model. Allowing students to use an AI language model could make it difficult to assess their true understanding of the law and their ability to apply legal principles.”

Continue reading

ChatGPT Thinks I Am Way More Interesting Than I Am

Write a bio of Pepperdine Caruso Law School Dean Paul Caron

Paul Caron is an American legal scholar and academic administrator who currently serves as the Dean of the Pepperdine Caruso School of Law. He was born in the United States and grew up in the Chicago area.

Caron graduated with honors from the University of Illinois at Urbana-Champaign, where he earned a Bachelor of Science in Accountancy. He then went on to earn a law degree from Yale Law School, where he was an editor of the Yale Law Journal.

After law school, Caron worked as an attorney in the tax department of the law firm Sullivan & Cromwell in New York City. He later joined the faculty of the University of Cincinnati College of Law, where he taught courses in tax law and served as the Associate Dean for Academic Affairs.

In 2003, Caron joined the faculty of the University of San Diego School of Law, where he held the rank of Professor of Law and served as the Associate Dean for Research and Faculty Development. During his time at USD, Caron became well-known for his scholarship in tax law and his popular TaxProf Blog, which he founded in 2004.

In 2017, Caron was appointed as the 7th Dean of the Pepperdine Caruso School of Law. In this role, he has overseen significant changes at the law school, including the construction of a new campus in downtown Los Angeles and the creation of new programs in areas such as dispute resolution and entertainment, media, and sports law.

Throughout his career, Caron has been a prolific scholar and writer, with more than 50 published articles and book chapters to his credit. He is widely regarded as one of the leading experts in the field of tax law, and has been frequently cited in the national media on issues related to tax policy and reform.

Caron has also been active in the legal community, serving on the board of directors of the American Tax Policy Institute and as a member of the executive committee of the Tax Section of the American Bar Association. He has received numerous awards and honors for his scholarship and teaching, including the University of Cincinnati College of Law's Goldman Prize for Excellence in Teaching and the American Bar Association's Section of Taxation Distinguished Service Award.

In addition to his academic and professional accomplishments, Caron is also an accomplished musician and photographer. He has released several albums of original music and his photographs have been featured in exhibitions and publications around the world.

Can you guess the number of factual errors?

Continue reading

GPT-4 Beats 90% Of Aspiring Lawyers On The Bar Exam

Daniel Martin Katz (Chicago Kent), Michael James Bommarito (Michigan State), Shang Gao (Casetext) & Pablo Arredondo (Casetext), GPT-4 Passes the Bar Exam:

In this paper, we experimentally evaluate the zero-shot performance of a preliminary version of GPT-4 against prior generations of GPT on the entire Uniform Bar Examination (UBE), including not only the multiple-choice Multistate Bar Examination (MBE), but also the open-ended Multistate Essay Exam (MEE) and Multistate Performance Test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 as compared to much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human tast-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.

Continue reading

Was The Sermon You Heard At Church Today Written By ChatGPT?

Axios, Religious Leaders Experiment With ChatGPT Sermons:

Religious leaders are dabbling in ChatGPT for sermon writing, and largely reaching the same conclusion: It's great for plucking Bible verses and concocting nice-sounding sentiments but lacks the human warmth that congregants crave. ...

• Early sermon-writing experiments have shown that ChatGPT can pull together cogent and relevant thoughts from religious texts and eminent theologians, plus turns-of-phrase that seem stirring and poignant.
• A consensus seems to be emerging that ChatGPT can alleviate some of the religious leaders' more routine or repetitive tasks — such as explaining particular holidays — while freeing them for more meaningful spiritual counseling.

What they're saying: "It's really impressive — it's kind of amazing," Ken Sundet Jones, a Lutheran pastor and theology professor in Des Moines who posed the Lazarus question, told Axios.

Continue reading

Colleges (And Law Schools) Are Rushing To Respond To ChatGPT

Chronicle of Higher Education, ChatGPT Is Everywhere:

It’s hard to believe that ChatGPT appeared on the scene just three months ago, promising to transform how we write. The chatbot, easy to use and trained on vast amounts of digital text, is now pervasive. Higher education, rarely quick about anything, is still trying to comprehend the scope of its likely impact on teaching — and how it should respond.

ChatGPT, which can produce essays, poems, prompts, contracts, lecture notes, and computer code, among other things, has stunned people with its fluidity, although not always its accuracy or creativity. To do this work it runs on a “large language model,” a word predictor that has been trained on enormous amounts of data. Similar generative artificial-intelligence systems allow users to create music and make art.

Many academics see these tools as a danger to authentic learning, fearing that students will take shortcuts to avoid the difficulty of coming up with original ideas, organizing their thoughts, or demonstrating their knowledge. Ask ChatGPT to write a few paragraphs, for example, on how Jean Piaget’s theories on childhood development apply to our age of anxiety and it can do that.

Other professors are enthusiastic, or at least intrigued, by the possibility of incorporating generative AI into academic life. Those same tools can help students — and professors — brainstorm, kick-start an essay, explain a confusing idea, and smooth out awkward first drafts. Equally important, these faculty members argue, is their responsibility to prepare students for a world in which these technologies will be incorporated into everyday life, helping to produce everything from a professional email to a legal contract.

But skeptics and fans alike still have to wrestle with the same set of complicated questions. Should instructors be redesigning their assignments and tests to reduce the likelihood that students will present the work of AI as their own? What guidance should students receive about this technology, given that one professor might ban AI tools and another encourage their use? Do academic-integrity policies need to be rewritten? Is it OK to use AI detectors? Should new coursework on AI be added and, if so, what form should it take?

For many, this is a head-spinning moment.

Continue reading

Does ChatGPT Produce Fishy Briefs?

ABA Journal, Does ChatGPT Produce Fishy Briefs?:

Lawyers are abuzz about the possible uses of ChatGPT. Could the artificial intelligence-powered chatbot write a persuasive legal brief worthy of judicial consideration? Given its limitations, we believe that’s unlikely. ChatGPT, a large language model developed by the San Francisco company OpenAI that launched in November, can draw only on sources available on the web; it cannot crawl appellate records or access subscription-based services such as Westlaw. Still, the ABA Journal decided to put the technology to the test just for kicks. ...

Continue reading

ChatGPT's Tax Advice Was Wrong 100% Of The Time

Accountants' World visitors: if you would like to receive a daily email with links to that day's tax posts, please email me.

Bloomberg, ChatGPT Can Give Tax Advice, But You Really Get What You Pay For:

ChatGPT can write essays, poetry, letters, and more, so we posed the question—can it do our taxes? Our company, TaxBuzz, decided to take the technology for a test drive, asking, “What status I should use when filing my taxes?”

It offered typical definitions of the IRS’ various filing options. Then came the best advice of the day: “You can consult a tax professional to help determine the best option for you.” While ChatGPT might be able to give you a CliffsNotes-style answer to your query, expecting a how-to tip or complete strategy isn’t in the cards. ...

When it comes to analyzing the tax code, expertly trained humans have access to trusted and verified libraries of information that they can use to verify tax strategy claims. ChatGPT simply doesn’t have the same level of brainpower or education as a human tax professional. ...

Continue reading

It’s Not Just Our Students: ChatGPT Is Coming For Faculty Scholarship

Chronicle of Higher Ed Op-Ed:  It’s Not Just Our Students — ChatGPT Is Coming for Faculty Writing, by Ben Chrisinger (Oxford; Google Scholar):

Almost immediately after OpenAI released ChatGPT in late November, people began wondering what it would mean for teaching and learning. A widely read piece in The Atlantic that provided one of the first looks at the tool’s ability to put together high-quality writing concluded that it would kill the student essay. Since then, academics everywhere have done their own experimenting with the technology — and weighed in on what to do about it. Some have banned students from using it, while others have offered tips on how to create essay assignments that are AI-proof. Many have suggested that we embrace the technology and incorporate it into the classroom.

While we’ve been busy worrying about what ChatGPT could mean for students, we haven’t devoted nearly as much attention to what it could mean for academics themselves. And it could mean a lot. Critically, academics disagree on exactly how AI can and should be used. And with the rapidly improving technology at our doorstep, we have little time to deliberate.

Continue reading

Bilek & Merritt: ChatGPT Almost Passed The Bar, But Competent Lawyers Do Much More

Bloomberg Law Op-Ed:  ChatGPT Almost Passed the Bar, But Competent Lawyers Do Much More, by Mary Lu Bilek (CUNY) & Deborah Merritt (Ohio State):

ChatGPT, OpenAI’s provocative artificial intelligence program, has come close to passing the multiple-choice portion of the bar exam. The bot has also earned passing grades on law school essays that resemble ones written for the exam. ...

Yet, even in a bill-by-the-hour world, clients who can afford it will still seek out human lawyers. Why? Because humans are far better than bots at eliciting facts and goals from clients, identifying new avenues of research, and solving multi-dimensional problems. Human experts will supplement those advantages by knowing when to consult AI, how to assess AI responses, and how to integrate AI knowledge with the human dimensions of a client problem. ...

Continue reading

Bishop Posts Two Papers On ChatGPT

Lea Bishop (Indiana-McKinney): 

Can ChatGPT 'Think Like a Lawyer?' A Socratic Dialogue:

A witty socratic dialogue with a language-generation model, exploring the aims of legal education in the new era of machine writing.

If you are a lawyer, a law student, or a law professor, you already have a good understanding of what it means to “think like a lawyer.”

For everyone else, there is ChatGPT. Hey, ChatGPT, give me a hand here…Let’s have a Socratic Dialogue. I’ll type a question, and you do your best to answer it. Your answers don’t need to be perfect. Because I want my readers to see exactly how you do what you do, I am not going to edit your responses at all. Okay, let’s go…

A Computer Wrote this Paper: What ChatGPT Means for Education, Research, and Writing:

Continue reading

Ryznar: Exams In The Time Of ChatGPT

Margaret Ryznar (Indiana-McKinney), Exams in the Time of ChatGPT, 80 Wash. & Lee L. Rev. Online __ (2023):

This article offers various methods to administer assessments while maintaining their integrity—after asking artificial intelligence writing tool ChatGPT for its views on the matter. The sophisticated response of the chatbot, which students can use in their written work, only raises the stakes of figuring out how to administer exams fairly.

Continue reading

Alarie: The Rise Of The Robotic Tax Analyst

Benjamin Alarie (Toronto; Google Scholar; CEO, Blue J Legal), The Rise of the Robotic Tax Analyst, 178 Tax Notes Fed. 57 (Jan. 2, 2023):

As a bold taxwriting experiment, this installment of Blue J Predicts has been generated with the help of an AI assistant, OpenAI’s “Generative Pre-Trained Transformer 3” (GPT-3). GPT-3 is a large language model developed by OpenAI and backed by Microsoft. It is an inexhaustible generator of text and can write with accuracy in English about almost any topic. It has performed its duty, with my human companionship, indefatigably.

This isn’t the first time that a legal academic has invoked a robotic coauthor, and I expect that these kinds of tools will become increasingly commonplace. I expect that eventually they will be about as remarkable as using a spelling or grammar checker. At this moment, however, before the rise of the robotic tax analyst, using GPT-3 to help write this article is likely to raise some eyebrows.

In October 2021 I produced a peer-reviewed law review article with my academic colleague, the late (and great) tax law professor Arthur Cockfield of Queen’s University. The article was notable for being the first peer-reviewed law review article to extensively leverage GPT-3 in its production. In that article — after a short introduction penned by us, Benjamin Alarie and Cockfield — we gave GPT-3 control of the metaphorical keyboard and allowed it to produce its textual analysis uninterrupted and unedited.

The results were mixed and intriguing, and certainly pointed in the direction of future possibility. In our view, GPT-3 had potential. In the article, we speculated on the future of AI in legal scholarship and asked provocatively in the title, “Will Machines Replace Us?” Cockfield and I concluded that “although GPT-3 is not up to the task of replacing law review authors currently, we are far less confident that GPT-5 or GPT-100 might not be up to the task in future.”

Continue reading

ChatGPT Gets C+ Grade On Four Minnesota Law School Exams (C- In Tax)

Following up on this morning's post, ChatGPT Gets B|B- Grade On Wharton MBA Exam:  Jonathan Choi (Minnesota; Google Scholar), Kristin Hickman (Minnesota; Google Scholar), Amy Monahan (Minnesota) & Daniel Schwarcz (Minnesota; Google Scholar), ChatGPT Goes to Law School:

How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.

Continue reading

ChatGPT Gets B|B- Grade On Wharton MBA Exam

Following up on this morning's post, ChatGPT Gets C+ Grade On Four Minnesota Law School Exams:  Christian Terwiesch (Wharton; Google Scholar), Would Chat GPT3 Get a Wharton MBA?:

OpenAI’s Chat GPT3 has shown a remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers, and consultants. Chat GPT3 has demonstrated the capability of performing professional tasks such as writing software code and preparing legal documents. The purpose of this paper is to document how Chat GPT3 performed on the final exam of a typical MBA core course, Operations Management. Exam questions were uploaded as used in a final exam setting and then graded.

Continue reading