Merritt: Small Sample Sizes Undermine Claims of 'The Economic Value of a Law Degree'

Deborah Jones Merritt (Ohio State), Small Samples:

I haven’t been surprised by the extensive discussion of the recent paper by Michael Simkovic and Frank McIntyre. The paper deserves attention from many readers. I have been surprised, however, by the number of scholars who endorse the paper–and even scorn skeptics–while acknowledging that they don’t understand the methods underlying Simkovic and McIntyre’s results. An empirical paper is only as good as its method; it’s essential for scholars to engage with that method.

I’ll discuss one methodological issue here: the small sample sizes underlying some of Simkovic and McIntyre’s results. Those sample sizes undercut the strength of some claims that Simkovic and McIntyre make in the current draft of the paper. ...

The sample of JD graduates ... is much smaller. Those totals range from 282 to 409 for the four panels, yielding a total of 1,342 lawyers. That’s still a substantial sample size, but Simkovic and McIntyre need to examine subsets of the sample to support their analyses. To chart changes in the financial premium generated by a law degree, for example, they need to examine reported incomes for each of the sixteen years in the sample. Those small groupings generate the uncertainty I discuss here. ...

As a professor who has taught Law and Social Science, I think the critics of the Simkovic/McIntyre paper raised many good questions. Empirical analyses need testing, and it is especially important to examine the assumptions that lie behind a quantitative study. ...

I hope to write a bit more on the Simkovic and McIntyre paper; there are more questions to raise about their conclusions. I may also try to offer some summaries of other research that has been done on the career paths of law school graduates and lawyers. We don’t have nearly enough research in the field, but there are some other studies worth knowing.

Prior TaxProf Blog coverage:

• What Is the Economic Value of a Law Degree -- $1 Million? (July 17, 2013)
• More on The $1 Million Value of a Law Degree (July 18, 2013)
• NY Times, WaPo, Others Debate The $1 Million Value of a Law Degree (July 18, 2013)
• What Is the Economic Value of a Law Degree -- $1 Million or $100,000? (July 19, 2013)
• More on The $1 Million Value of a Law Degree (July 20, 2013)
• Merritt on The $1 Million Value of a Law Degree (July 21, 2013)
• Diamond on The $1 Million Value of a Law Degree (July 22, 2013)
• Tamanaha:  How 'The Million Dollar Law Degree' Study Systematically Overstates Value (July 23, 2013)
• Tamanaha:  How the 'Million Dollar Law Degree' Study Understates Risk (Part I) (July 24, 2013)
• Tamanaha:  Why the “Million Dollar Law Degree” Study Fails (Final Post) (July 25, 2013)
• Rasmusen:  Critics of The Economic Value of a Law Degree Are Making the Paper Better (July 25, 2013)
• Pasquale and Simkovic Respond to Tamanaha (July 25, 2013)
• The American Lawyer:  Paper on Law Degree's Economic Value a Non-Sequitur (July 26, 2013)
• Simkovic Responds to Tamanaha (Part 3) (July 26, 2013)
• Tamanaha:  Short Term Versus Long Term Perspective (July 27, 2013)
• Harper, Diamond:  The $1 Million Value of a Law Degree: Distraction, Astronomy, or Astrology? (July 28, 2013)
• Simkovic Responds to American Lawyer Op-Eds (July 29, 2013)
• Diamond Responds to Tamanaha (July 29, 2013)
• Rasmusen:  The Economic Value of a Law Degree and the 'Typical' Law Student (July 29, 2013)
• Simkovic Responds to Tamanaha (Part 4) (July 30, 2013)
• Campos on The Economic Value of a Law Degree: The Problem With Back in the Day (July 30, 2013)
• Leiter's Reflections on The Economic Value of a Law Degree (July 30, 2013)
• Tamanaha Responds to Leiter (and Leiter Replies) (July 31, 2013)
• The $1 Million Value of a Law Degree and the Limits of Empirical Studies (July 31, 2013)
• Pasquale Responds to Tamanaha: A 'Torrent of Words' on the Value of Law School (July 31, 2013)
• Simkovic:  The Economic Value of a Law Degree (Final Post) (Aug. 1, 2013)
• Rosenzweig: 'The Economic Value of a Law Degree' and the Degraded State of Scholarly 'Debate' (Aug. 2, 2013)