Tamanaha: Why the 'Million Dollar Law Degree' Study Fails (Final Post)

Brian Tamanaha (Washington U.), Why the “Million Dollar Law Degree” Study Fails (Final Post):

My first post criticizing Simkovic and McIntyre’s (S&M) study showed how they overstated the earnings premium from a law degree. My second post showed that the only way to make a useful risk assessment is on a per school basis, and that it is indeed risky to attend a bunch of law schools today. In this final post I will expose what I believe is the most profound flaw in their analysis—and I will apologize for one thing I did wrong.

Let me begin by repeating their core claim: S&M have identified “historic earnings norms” that reflect what people who attend law school today are likely to earn over the course of their careers. They claim that even law grads at the bottom quartile stand to earn “hundreds of thousands of dollars” above what they would have earned without the degree.

We must separate two issues: 1) do their “earnings premiums” accurately represent historic law grad earnings?; and 2) are these earnings likely to hold for the coming generation? For their claims to have merit, they must establish both #1 and #2.

How Does One Establish "Historic Norms With Predictive Value?

Though it is fundamental to their analysis, this question is not addressed anywhere in their paper. S&M just count earnings data from 1996-2011, and call the results a “historic norm.” To constitute a “historic norm,” however, the earnings during these 16 years must be representative of law grad earnings over time. In other words, if we were to take a 16-year slice of law grad earnings at any random period in the past century, we would come up with roughly the same earnings (inflation adjusted). But S&M did not actually look at any other years of law grad earnings. So they cannot possibly know that this 16-year earnings slice is representative.

In response to this argument, S&M will likely say that they used the best data available. But that is non-responsive. If that’s the best data available, then the logically necessary conclusion is that they cannot determine whether this 16-year slice indeed represents a “historic norm.” The fact that they don’t have enough data to answer the question does not give them license to assume the answer.

The negative answer to #1 is enough to vitiate the predictive value of their study. In the absence of any empirical evidence (even rudimentary) that this 16-year slice is representative of lawyer earnings over time, then it has no bearing on what law grads will earn going forward because it has no predictive weight.

Will Earnings of the Coming Generation Resemble the Past?

S&M briefly address this question, noting the work of Bill Henderson, who argues that the legal profession is undergoing a structural shift that portends leaner times for lawyers going forward. S&M’s retort is that cycles come and go and there is no reason to think this time is different. Their response: “Predictions of structural change in the legal industry date back at least to the invention of the typewriter.”

Touche. But seriously, what are the chances that the earnings they found in their 1996-2011 slice are likely to continue? After all, S&M make a very strong claim: students who attend law school today are likely to obtain the lifetime earnings premiums they specify—and students who skip law school out of concern for the financial risks will make a foolish mistake.

To make a convincing case that the next generation of earnings will look like the past generation, S&M must delve into the abundant literature on the legal job market. But they don’t do this—at all. They close the issue with a conclusory assertion and a caveat:

The most sober interpretation of the recent decline in starting salaries and employment for recent law graduates is that it is part of a broad cyclical downturn following the shock of the financial crisis of 2007 to 2008 and the recession that followed. The historic data still offers the best, most objective indicator of value. That said, past performance does not guarantee future returns.

A quick word about each sentence: The first sentence dismisses the people (not sober) who believe a structural change is in the works without saying why they are wrong. The second sentence reveals, again, how heavily their argument depends on the claim that their earnings numbers represent “historic norms”—which, as I indicated above, they provide no evidence for. The third sentence misstates the issue: of course they cannot guarantee earnings results—what they must do is affirmatively show that the bounce back in the legal market (to match the circumstances in the 16 year slice) is likely to happen. And this they have not done.

Why the Past Generation and the Next Are Likely to be Different

What makes the “historic norm” claim of S&M so problematic is that the legal market underwent a series of unique developments in the past generation. This was the age of the rise of Big Law. Twenty-five years ago a large corporate firm had a few hundred lawyers—today the largest firms have several thousand. Over the course of this period, compensation for top partners rose to unprecedented heights, firms hired many more young lawyers and paid them very high salaries. The bi-modal distribution of starting salaries—about 10% of law grads earn far above the bulk of grads with a large earnings gap separating the two—began to develop around 2000.

S&M’s study measures law grad earnings during this unique period—and as I argued in the first post their study did not count the effects of the recession in the legal market from 1992-1995 and the dismal fate of law grads from 2009 to the present. Their 16-year employment data slice thus centers around the best years of the best of times in the corporate law market. The emergence of the unusual bi-modal pay distribution is strong evidence that earning patterns during this period were anything but typical.

Thus, we have good reason to doubt that the 16-year slice is representative of “historic” law grad earnings. It can’t be, since the rise of Big Law was a new historical development. To claim that earnings figures taken from this period constitute representative “historic norms” is ahistorical.

As for the coming generation, there is a flood of recent studies, books, and articles saying that things are indeed different now. S&M dismisses Bill Henderson, but what about Richard Susskind (The End of Lawyers; Tomorrow’s Lawyers), Bruce MacEwen (Growth is Dead: Now What?), Stephen Harper (The Lawyer Bubble: A Profession in Crisis)? These are knowledgeable people.

The New Republic recently ran a lengthy feature article, “The Last Days of Big Law.” The chairman of a major law firm, Mayer Brown, said “I don’t think anybody who follows the profession would suggest that this is only a temporary situation.” A couple of months ago, another prominent law firm, Weil Gotschal, which was thought to have weathered the legal recession better than most firms, stunned the corporate law world by announcing a large layoff of associates. The Executive Partner explained the move: “We believe that this is not just a cycle, but that the supply-demand balance is out of whack across the industry.” I can multiply quotes like this.

S&M cannot dismiss managing partners at leading law firms who say the boom days are history, and are firing lawyers and cutting partner pay to adjust to the “new normal.” These are sober people. The suggestion that we are seeing “the last days of Big Law” is journalistic hype (none of the people mentioned above say this). And things will firm up when the current shake out has been completed. But at the end of this process Big Law will be leaner and more efficient. It is already happening: many corporate law firms now hire about half the number of new lawyers they once hired in 2008 and before, preferring instead to use cheaper and more flexible contract workers to handle excess work.

S&M are free to argue that this growing chorus of voices is wrong, but they need to make a case with more substance than “cycles happen.” In the end, here is their response: “It remains easy to tell stories about how various changes will eventually have this or that effect, and currently impossible to falsify, since we cannot measure the future.” Sure—but lots of people who make their living following the legal market are telling the same story—and it’s opposite the story S&M are telling. And S&M forget that the validity of their claims presupposes that their version of the future—that things will be the same as the past—is correct.

S&M are so convinced by their (unsupported) “historic norm” claim that they misunderstand what needs to happen in the job market going forward. They say:

The results suggest that—absent catastrophic and unprecedented changes exceeding changes already seen from 2008 to 2011 and uniquely affecting law graduates rather than the broader labor market—many college graduates who follow the critics’ advice and skip law school will forgo a lucrative career and face higher long-term risks of financial hardship.

That’s wrong. It is not necessary for things to get worse to falsify the predictive validity of their projected lifetime earnings figures. To match the 16-year slice from which they derived their “historic norms,” the legal market must substantially improve going forward. Their earnings numbers were taken during the peak of Big Law—and if we do not get back to that, future law grad earnings will be lower. No one who follows the legal market, not even the optimists, says we are going back to the peak years of Big Law.

It is a mistake, moreover, to think that these developments are limited to the corporate law sector. What happens in this sector reverberates throughout the legal employment market. When grads from top law schools don’t land big firm jobs in NYC, DC, Chicago, and LA, they fan out to less desirable secondary markets like Indianapolis, St. Louis, Atlanta, etc., pushing out law grads from regional and local schools, who in turn are forced to seek less desirable employment (small firms or solo), or end up unemployed.

This brings me to another problematic assertion by S&M. About 40% of the law grads in their database do not practice law; they mention that a significant proportion of law students say they want to do other things with the degree, and point out that 10% of CEOs and 50% of Senators have law degrees. S&M suggest that we therefore need not worry about the many law grads who have not obtained full time jobs as lawyers in recent years (only 55% and 56%, respectively, had obtained long-term full time jobs as lawyers nine months after graduation in 2011 and 2012). But how many of those CEOs and Senators began their careers unemployed, or working as paralegals, or as compliance officers, or went back to their previous jobs as school teachers? Leaving law for something better is a world of difference from getting no employment boost from the JD to begin with. The latter is what many law grads struggle with today. The percentage of recent law grads who do not obtain initial employment as lawyers is unprecedented—and not by choice. And there is every reason to expect that their earnings will be lower than the earnings premiums S&M found in their study.

Final Words

For S&M’s study to have predictive value, they must establish two points: 1) that their 16-year slice is sufficiently representative of law grad earnings over time to be considered a “historic norm”; and 2) that the next generation of law grad earnings will resemble the earnings period captured in their study. They did not make either argument—indeed they assumed away the first—and the weight of evidence is against them on both.

Now let me conclude with an apology. My first post in this series was less charitable toward Simkovic and MacIntyre than I should have been. I should have made the same criticisms with a less harsh edge. Given the tone of my comments, it may come as a surprise to many that a couple of months ago I was asked by a law journal to evaluate S&M’s draft for publication, and, despite my reservations, I gave it a positive recommendation because I thought it raised a useful new perspective on the issue of economic return on a law degree.

I have been debating these issues for over two years now and sometimes lose my equanimity. I believe getting it right is of utmost importance to young people thinking about a career in law. What knocked me off balance was this line in their article: “many college graduates who follow the critics’ advice and skip law school will forego a lucrative career and face higher long-term risks of financial hardship.” To say something that strong requires that they make a rock solid case for the correctness of their findings. I hope I have convincingly demonstrated in these three posts that their claims are anything but rock solid.

[Cross-posted at Balkinization.] 

Update:  Pasquale and Simkovic Respond to Tamanaha

Prior TaxProf Blog coverage:

• What Is the Economic Value of a Law Degree -- $1 Million? (July 17, 2013)
• More on The $1 Million Value of a Law Degree (July 18, 2013)
• NY Times, WaPo, Others Debate The $1 Million Value of a Law Degree (July 18, 2013)
• What Is the Economic Value of a Law Degree -- $1 Million or $100,000? (July 19, 2013)
• More on The $1 Million Value of a Law Degree (July 20, 2013)
• Merritt on The $1 Million Value of a Law Degree (July 21, 2013)
• Diamond on The $1 Million Value of a Law Degree (July 22, 2013)
• Tamanaha:  How 'The Million Dollar Law Degree' Study Systematically Overstates Value (July 23, 2013)
• Tamanaha:  How the 'Million Dollar Law Degree' Study Understates Risk (Part I) (July 24, 2013)

Update:

• Rasmusen:  Critics of The Economic Value of a Law Degree Are Making the Paper Better (July 25, 2013)
• Pasquale and Simkovic Respond to Tamanaha (July 25, 2013)
• The American Lawyer:  Paper on Law Degree's Economic Value a Non-Sequitur (July 26, 2013)
• Simkovic Responds to Tamanaha (Part 3) (July 26, 2013)
• Tamanaha:  Short Term Versus Long Term Perspective (July 27, 2013)
• Harper, Diamond:  The $1 Million Value of a Law Degree: Distraction, Astronomy, or Astrology? (July 28, 2013)
• Simkovic Responds to American Lawyer Op-Eds (July 29, 2013)
• Diamond Responds to Tamanaha (July 29, 2013)
• Rasmusen:  The Economic Value of a Law Degree and the 'Typical' Law Student (July 29, 2013)
• Simkovic Responds to Tamanaha (Part 4) (July 30, 2013)