Paul L. Caron

Thursday, March 11, 2021

A Guide To Quantitative Legal Research

Shai Dothan (Copenhagen; Google Scholar), A Guide to Quantitative Legal Research:

This is a simple, non-technical introduction to conducting quantitative legal research. It discusses the main tools for measuring statistical significance, some problems that occur in empirical research and how to solve them, specific tools for investigating courts, and the challenges that courts raise for quantitative research. The chapter is designed to serve as a guide for law students who wish to write a thesis or an essay that includes quantitative research.

This chapter is but a short introduction to the use of quantitative methods in legal analysis. To all of you who want to learn more, I can recommend an excellent book-length guide – Lee Epstein (Washington University) & Andrew D. Martin (Washington University), An Introduction to Empirical Legal Research (Oxford University Press 2014).

I would also recommend taking a few classes in empirical methods. Good classes are offered in most major universities and by studying in a group and solving exercises you can hone and improve your skills. It is a fitting way to conclude a chapter on the use of quantitative methods by acknowledging that this is a methodology that could potentially be used to mislead. Many of you must have heard the popular phrase "There are three kinds of lies: lies, damn lies, and statistics". Unfortunately, using quantitative methods to deceive is quite common. A simple way to misrepresent the results of empirical research has to do with presentation rather than with analysis. Sometimes graphs do not start at zero or the scale on one or both of the axes isn't proportional, all to make some changes look bigger or smaller than they actually are. Charts can be used to confuse the reader as well, sometimes without any bad intention. Some scholars simply don't realize that pie charts and three-dimensional charts are very difficult to read critically.

A much more severe form of cheating in statistics is not following the proper academic practice in the analysis of the data itself. Robustness checks are needed to ensure the accuracy of every quantitative analysis, but scholars are under enormous pressure to publish and nonsignificant results are usually unpublishable. The result is that scholars have a tendency to stop when they get a finding that appears significant even if they did not really prove any causal connection. And if the first result isn't satisfactory, scholars can use a lot of tricks to increase their chances of getting a semblance of significance. They can change the units that are being measured until a result seems significant, for example by measuring differences in timing in units of days, then weeks, then months etc. until one result looks presentable. Scholars can also increase the pool of observations that are tested just to the point when the result looks significant. Scholars can also find small differences that appear significant in a big pool of observations, even if such small differences do not have any practical meaning.

When publication-hungry scholars get their "significant" results and send them to a publisher, they are rarely subjected to serious scrutiny. Peer-review journals must rely on busy academics to go through the numbers, usually for free. No wonder few reviewers can stop a problematic paper from being published. When a paper is published, the chances that the study would be replicated to look for errors are usually quite small. Despite a wide understanding that repeating studies is crucial to get viable results and many initiatives to encourage replicating studies, career incentives usually push scholars to claim something new, instead of making other scholars angry by publicly disputing their results.

The truth is that there is no such thing as a perfect quantitative study. One can always do another robustness check and use other tools such as interviews to make a stronger claim. It is human weakness and a focus on short-term goals that usually prevent scholars from doing so. But these human weaknesses pale in comparison to the criminality of real cheating. There have been scholars who used fake data before. Some were caught and their lives were destroyed, some probably still walk among us. Some have tarnished the reputation of others: co-authors, supervisors, and colleagues who were not even aware of any wrong-doing but suffer the consequences of being too credulous or not careful enough in choosing their friends.

Before you engage in cheating, you should ask yourself if you want to spend the rest of your life looking over your shoulder, frightened that the career you have built will be taken away and your reputation will be forever ruined. If a utilitarian calculation isn't enough to convince you to choose the narrow path of virtue, a higher power may be necessary.

My grandfather always used to start his physics classes by telling his students: I don't care if you are thieves and criminals, but in my class you will NEVER falsify results. For him, proper research methods were simply a form of religion. It is the only religion that could get humanity to the moon, find a cure to Polio, and unlock the mysteries of human society. Its true followers should be very proud.

Legal Ed Scholarship, Legal Education | Permalink