Lawsky: The Intersection Of Tax Law And Computational Law

John Nay (Stanford; Google Scholar) & Sarah Lawsky (Northwestern; Google Scholar) et. al, Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence, 382 Phil. Transactions of the Royal Society A (2024):

Better understanding of Large Language Models’(LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence and leveraging LLMs to identify inconsistencies in law. This paper explores LLM capabilities in applying tax law. We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and math skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. We experiment with retrieving and using the relevant legal authority to assess the impact of providing additional legal context to LLMs. Few-shot prompting, presenting examples of question–answer pairs, is also found to significantly enhance the performance of the most advanced model, GPT-4. The findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels. As LLMs continue to advance, their ability to reason about law autonomously could have significant implications for the legal profession and AI governance. This article is part of the theme issue ‘A complexity science approach to law and governance’.

Sarah Lawsky (Northwestern; Google Scholar), Computational Law and Epistemic Trespassing, 2 J. Cross-Disciplinary Research in Computational L. 1 (2024):

This article uses the concept of ‘epistemic trespassing’ to argue that technologists who propose applications of computer science to the law should recognize and incorporate legal expertise, and that legal experts have a responsibility not to defer mindlessly to technologists’ claims. Computational tools or projects developed without an understanding of the substance and practice of law may harm rather than help, by diverting resources from actually useful tools and projects, focusing on unimportant questions, answering questions incorrectly, or providing purported solutions without sufficient attention to the larger context in which law is created and functions.

Editor's Note:  If you would like to receive a daily email with links to tax posts on TaxProf Blog, email me here.