TaxProf Blog

Tuesday, July 23, 2013

Tamanaha: How 'The Million Dollar Law Degree' Study Systematically Overstates Value

Brian Tamanaha (Washington U.), How “The Million Dollar Law Degree” Study Systematically Overstates Value: Three Design Choices that Improperly Skewed the Results:

Michael Simkovic and Frank McIntyre (S&M) have created a sensation with the release of their new study purporting to show that a law degree has a value of $1,000,000. Legal educators cheered and skeptics jeered.

S&M pitch the article as a crushing empirical refutation of the argument I press in Failing Law Schools that getting a law degree today can be financially risky, especially for students who attend expensive low ranked law schools. Simkovic writes, “we disagree with [Tamanaha’s] conclusions about the riskiness of a law degree because data on law degree holders does not support his conclusions.” So confident are they in their findings that they warn prospective students: “many college graduates who follow the critics’ advice and skip law school will forego a lucrative career and face higher long-term risks of financial hardship.”

All sides in this debate carry the same responsibility, of course, because if S&M are wrong, people who follow their advice risk serious financial hardship. Thus it is incumbent on all of us to consider these issues carefully and honestly.

Everyone agrees that many lawyers make a good living, so the significance of their study is at the bottom, where the greatest risk lies. S&M assert that even law graduates at the lower end of the earnings spectrum, including grads who don’t land jobs as lawyers, stand to earn “hundreds of thousands” of dollars over what they would have obtained with a bachelor’s degree alone. Great news for all!

Unfortunately, it’s not true.

Let me state up front that their study contributes to the debate over the value of a law degree. Let me also confirm that their study is far more sophisticated than my admittedly crude efforts. But as the old saying goes, it is far better to be approximately right than precisely wrong. Despite having the external trappings of precision and rigor, their study is faulty and misleading.

S&M Overstate the Economic Value of a Degree Over the Past Two Decades

Their analysis is based upon data provided by United States Census Bureau’s Survey of Income and Program Participation (SIPP), which tracks earnings over time by educational degree. The authors examine the earnings of people with law degrees from 1996 to 2011, and compare this to the earnings of people with bachelor’s degrees (in selected majors) over the same period. The “JD earnings premium” they identify is the difference between the two.

The great strength of their study, S&M say, is that it is a “long term” examination which establishes “historic norms”—thereby providing a much more reliable picture of lawyer earnings than my own immediate short term focus. The credibility of their study rests entirely upon this claim.

All their talk of “historic norms,” however, is puffed up exaggeration: 16 years is neither “long” nor remotely enough time to establish “historic norms.”

Had S&M extended their study by just four years—to 1992, covering twenty years in all—they would have found reduced “earnings premium” for law grads. This is because the legal job market suffered a severe recession from 1992 to 1998, while the general economic recession had ended in early 1992. (The downturn in the legal market is visible in Figure 2, p.9.) Since the “earnings premium” is the difference between the earnings of bachelor’s degree holders and law degree holders, it constricts when the general economy does relatively well while the legal market does relatively poorly. That was precisely the situation between 1992 and 1998.

There is no doubt that including 1992 to 1995 in their study would measurably reduce the “earnings premium.” In Figure 6 (p. 35), S&M set forth the earnings premium in four blocks: 1996-99, 2000-03, 2004-07, and 2008-2011. 1996-99 has the lowest earnings premium among the four (substantially below average)—and it is likely that the 1992-95 premium would be lower still because the legal market was down for all four years, whereas it was beginning to recover in 1998. Had 1992-95 been included, it would have constituted 1/5 of the total, undoubtedly bringing down the overall “earnings premium” by no small amount.

As other critics have pointed out, furthermore, their study does not include the earnings of the graduating classes of 2009, 2010, and 2011 (not to forget 2012 and 2013)—which have suffered devastatingly poor job results. (The earnings included in the study through 2011 are only for 2008 grads and earlier.)

To be clear, I’m not asserting that S&M began to count income in 1996, rather than 1992, deliberately to boost the “earnings premium” by shaving off four of the worst law degree earning years in the past 20 years, although it had this effect. No doubt they can offer reasons to explain their decision. (Although SIPP was redesigned in 1996, there are surveys for 1993 and 1992, which allow continuity; SIPP reports go back to the mid-1980s, first appearing in the middle of yet another recession in the legal market.) And they cannot be blamed for leaving out post-2008 law grads because the latest SIPP was from 2008—though this information will be available when the next SIPP is released.

Why these crucial gaps in information are present in their study is irrelevant to the more pressing issue of the reliability of their findings. As a consequence of these gaps, their study overstates the actual JD “earnings premium” over these 20 years. Had the very same study been done, but with full earnings data from 1992 through 2011 on all people with law degrees, lawyer earnings would be lower. This is certain.

This conclusion completely undermines the validity of their figures. Why should anyone put stock in “earnings premium” numbers derived from a study limited to 16 years, when extending the study just 4 more years, to 20, would produce notably lower figures? Why should we take these numbers as reliable indicators when forthcoming SIPPs will begin to show the devastating economic results suffered by recent law grads (2009, 2010, 2011, 2012, 2013 and counting), none of which are reflected in S&M’s figures?

Genuine long term historical studies span multiple cycles, several generations, because only then can the distorting impact of contingencies—like starting near the end of a recession (1996) versus at the beginning (1992)—even out over multiple up-and-down cycles. This may require data over several centuries. Their study does not even cover one full recession in the legal market. In fact, their 16-year slice is unrepresentative of long-term trends because, while it leaves out significant negative effects from two recessions, it fully captures the positive effects of the boom years in the legal market of the mid 2000s. (A fortuitous slice indeed for a study that aims to prove the high value of a law degree.)

It is exceedingly rare to find reliably predictive “historic norms” in the social sciences because social life is too complex and circumstances are constantly changing. (That’s why economists are so often wrong in their predictions.) Industries rise, transform, and expire; trades and professions wax and wane; formerly dominant national economies fall into decline while others take over prominence. S&M’s bold assertion that their 16-year study establishes valid “historic norms” on law degree earnings would be scoffed at by social scientists who take the notion of “historic norms” seriously. That is more than enough time to confirm norms governing the mating behavior of fruit flies, but 16 years is laughably inadequate for predicting something as complex and subject to change as the lifetime earnings of future law grads.

S&M have produced a narrow, partial, time-bound study that has zero predictive relevance for anyone thinking about attending law school today. The most we can glean from it is that a bunch of law grads will probably end up doing well financially—but we knew that already. This criticism on its own is sufficient to debunk the entire validity of their study, but it suffers from additional flaws that merit mention.

Their Earnings Premiums Are Artificially Inflated in Two Ways That Fail to Account for Ability

Picture two groups of recent college graduates with bachelor’s degrees. One group achieved above average GPA’s and standardized test scores, they attended better colleges, attached greater value to education, had higher expectations about future salaries, and came from wealthier families. The second group was average on all these characteristics.

Most people would assume that over time the first group on average would earn more money than the second group. And they would be right. In addition to the fact that people in the first group have greater ability and motivation, the offspring of people with more money tend to do better because they have more social and economic advantages to leverage (that’s American meritocracy in action).

As S&M confirm, law grads as a group are above average in all these characteristics. They also find (using National Education Longitudinal Survey data) that these characteristics, individually and collectively, are associated with higher earnings—entirely apart from the law degree itself. This is relevant to their study because, as they observe, “If ability sorting is positive, then some portion of the earnings premium associated with a law degree is due to differences in ability levels, and the causal effect of the law degree on earnings will be smaller than the observed differences in earnings.”

After running their analysis, S&M find that “a typical law degree holder would earn 10.3 percent more than a typical bachelor’s degree holder, even if the law degree holder had chosen to terminate his education with a bachelor’s degree. Column 2 suggests a slightly lower 8 percent, as it ignores differences in expectations.”

Thus, typical law grads can expect roughly10% higher earnings than average bachelor’s degree holders, entirely apart from the JD, because as a group they are smarter, more motivated, and come from wealthier families, etc. According to their study design, S&M should therefore subtract this 10% because that amount cannot be counted as part of the “JD earnings premium.”

Surprisingly, without explanation, S&M do not subtract this amount. They downplay the size of earnings attributable to ability, saying, “we find evidence of modest ability sorting that explains only a small fraction of the observed law degree earnings premium.” [Comment: 10% of average bachelor’s earnings is a sizable chunk.] They add that if they stripped away their controls the difference falls to 5% (as if it is fine to strip away controls). Then S&M include the entire amount within the JD earnings premium, forgetting their earlier statement that earning attributable to ability should not be counted. This move, of course, gives the “earnings premium” an under-the-covers boost. (Elsewhere in the paper they speculate that SIPP understates earnings at the high end, so perhaps they ignore the “ability earnings” 10% as a rough compensating trade off; if so, this would introduce yet another distortion, because the earnings we care about are at the low end, which they say are over-reported in SIPP.)

For what purports to be a rigorous empirical study, S&M’s treatment of this factor is ad hoc, and, most problematically, it slants the findings in favor of the thesis they set out to prove: that a law degree has positive value even at the low end. This move on their part was not a design flaw but a departure from their stated design. Empiricists who aspire to rigor don’t get to ignore inconvenient findings and make fudges in their favor.

Even if earnings attributable to ability are “small” (which does not appear to be the case), they cannot be included in the “JD earnings premium” not only because it is false to do so, but also because the values they come up with are used to identify the point at which law grads begin to reap a negative return from their law degree (i.e. when they would have been better off skipping law school altogether). So it matters.

There was a simple way for S&M to account for the 10% ability earnings: increase the average bachelor’s earnings figures by 10%; whereupon, the JD premium they calculate would be the difference between how much law grads earn and this higher figure. Obviously, had they done this, the JD premium would fall. That they failed to do this is not right.

S&M did something else related to ability that produced an even greater distortion. They claim the “JD earnings premium” is the amount law grads earn over what they would have earned had they only gotten a bachelor’s degree. In order to figure out this difference, we must know what they would have earned without the JD.

Recall that law grads as a group have above average GPA’s, test scores, and motivation, they have more family wealth and went to better colleges, etc. And keep in mind that many bachelor’s degree holders would not get into law school because they lack the requisite motivation and credentials—most SIPP interviewees obtained their law degree in the 1970’s, 1980’s, and 1990’s, when there were far fewer law schools and a large percentage of applicants were rejected (compared to the loose 80% admittance rate at bottom schools today).

Based upon these considerations, it is reasonable to surmise that the people in the “smarter-more motivated-pool of potential law students” who choose to forego law school would earn at least as much as the median earnings of a bachelor’s degree holder. After all, S&M found that their characteristics correlate with 10% earnings above average bachelor’s earnings (and average is higher than median).

When S&M selected the baseline to calculate the earnings premium, however, they did not use the median bachelor’s earnings, but instead they compared the earnings between law grads and bachelor’s holders at each quartile: law grad earnings at 25th percentile were compared to bachelor’s at 25th percentile; law grads at 50th percentile were compared to bachelor’s at 50th percentile; law grads at 75th percentile were compared to bachelor’s at 75th percentile.

But this raises a large objection: Why should we expect that bottom quartile law grads would earn at the bottom quartile of bachelor’s degree holders had they not gone to law school? S&M already told us that law grads in general have above average characteristics that correlate with earnings 10% above average bachelor’s holders!

If S&M had instead compared law grad earnings at all three quartiles to bachelor’s earnings at 50th percentile, the earnings premium of law grads at the 75th percentile would go up, the premium at the 50th percentile would remain unchanged, and the premium at the 25th percentile would plummet.

This conclusion casts huge doubt on S&M’s results and bears repeating. If S&M had assumed that potential law students who forego law school would have earned what median bachelor’s holders earn—which seems the most logical assumption given the superiority of the law student pool—then the “earnings premium” for law grads at the 25th percentile falls significantly.

S&M might respond that just because the pool of law grads is superior on average doesn’t mean bottom law grads are more capable than median bachelor’s holders. Perhaps. But then they should compare the 25th percentile law grad earnings to bachelor’s earnings at, say, the 40th percentile.

S&M might then say that they only have quartile numbers, so comparisons against intermediate percentiles are not possible. Fine. Then run two comparisons (at 25th and at 50th of bachelor’s earnings), display the results, and let readers decide which is the more apt comparison. When faced with these two choices, most prospective law students (a smart bunch) would correctly surmise that it makes more sense to compare the 25th percentile of law students against bachelor’s holders at the median.

In light of the momentous implications of this choice for their findings, it is remarkable that S&M do not even discuss whether they have made the correct comparison. They simply assert that it is appropriate to examine “the 25th percentile of earners with a given level of education [bachelor’s] compared to the 25th percentile of earners with a higher level of education [law degree]. Such a distributional analysis would test claims that advanced degrees may not benefit less capable students as much as they benefit average or above average students.” What S&M apparently failed to appreciate (or decided to ignore) is that, given the higher-level talent pool of law grads, “less capable” law grads can still be as capable as median bachelor’s degree holders. This seems elementary.

A Compromised Study

A striking feature of S&M’s study is their chest-pounding confidence. Social scientists usually are very cautious about making causal claims owing to the complexity of social factors and how they interact, unobserved influences, and limited data points. No false modesty for S&M. They claim they have proven a direct causal relation between a law degree and specified lifetime earning values. They claim they have established reliable “historic norms” for law degree earnings (based upon a “long term” study covering only 16 years). They warn prospective students who do not accept their findings—instead foolishly believing benighted critics of legal education—that they will forego a lucrative career and risk future financial hardship. And their original title blared, “The Million Dollar Law Degree.”

S&M gush about their robust showing that law grads, including those on the low end, reap oodles of surplus value from a law degree:

These results suggest that even at the 25th percentile, the value of a law degree exceeds typical net-tuition costs by hundreds of thousands of dollars.” At the mean and 75th percentiles, the difference is close to one million dollars. We therefore reject the claim that law degrees are priced above their value. Indeed, the value compared to net-tuition prices suggests that legal education is a competitive market in which surplus redounds to the benefit of student-consumers. (41)

Not really. Their study is full of holes and they made a series of dubious choices that were mostly hidden. Rather than conduct an open-minded inquiry into the economic value of a law degree, it appears that S&M were hell bent on proving that a law degree pays off handsomely for nearly all law grads. This orientation compromised the soundness of their study.

The paragraph quoted above oozes exaggeration. According to S&M, the present lifetime value of the JD earnings premium for law grads at the 25th percentile is $350,000. Minus federal taxes (following their formula), this comes to $262,500. Subtracting three years of tuition (they calculate scholarship discounted average tuition at $30,000 x 3) leaves us with $172,500. For male law grads at the 25th percentile, the same calculation comes to $147,000. Needless to say, that’s not hundreds of thousands of dollars. Overstatement runs through their article.

And these numbers are further inflated by other questionable choices they made. Providing no support whatsoever, S&M assert that law students earn $24,000 while in law school (1st year $5,000; 2nd year $7,000; 3rd year $12,000)—this is an implausibly high figure at a time when many law students work in unpaid externships during the summer and wait tables on the side. (S&M reduce opportunity cost by $24,000.) They do not include living expenses in the cost of a law degree because, they reason, students must pay this even if do not attend law school (ignoring the higher cost urban areas many law schools are located in); we can let that pass, but that does not explain why they take no account of the interest on loans taken out to pay for living expenses (borrowing in the $50,000-plus range for living alone), which working people cover with current income. Making adjustments for these fudges would bring the net earnings premium down even more.

Still, one might think, a net lifetime premium of $125,000 is pretty good. Keep in mind, however, this is earned over 42 years. Three years of law school at this earnings premium gets you an additional $250 a month (over a bachelor’s degree), and hopefully a rewarding career. Nothing to sniff at, to be sure, but this is hardly a generous “surplus value” redounding to the benefit of law grads that S&M so proudly mention. (Tellingly, Figure 1. shows that lawyer earnings have remained flat in real terms since 1998, while law school tuition has risen far above the rate of inflation, so we have been absorbing an ever greater share of this surplus value over time.)

Now, this discussion is completely artificial because it takes S&M’s stated JD earnings premiums at face value, when I have already shown that the $350,000 figure is substantially overstated owing to three major flaws in their study: 1) they do not count the full effects of 2 significant bouts of reduced lawyer earnings in the past twenty years (4 years of legal recession from 1992-95, and the dismal fate of law grads from 2009-2011); 2) they should have subtracted the 10% increase in earnings correlated with higher ability, motivation, etc., from their “JD earnings premium;” and 3) when calculating the JD earnings premium at the 25th percentile, they should have used bachelor’s degree earnings at the 50th percentile as the baseline.

Fixing any one of these flaws would bring the $350,000 down by amounts that matter (moving the total after taxes and tuition ever closer to the negative line); fixing all of them would flip their results from reassuring to positively worrisome. And we must keep in mind that the issue at stake does not stop at the 25th percentile. Let’s say that—after fixing these flaws and netting costs and taxes—law grads at the 25th percentile still get a positive gain of $25,000. S&M don’t get to declare victory and walk away, saying they have proven critics of legal education wrong. What about 20th percentile law grad earnings? If those folks are in the negative, it is hardly a vindication of the economic value of a law degree to say that “only” 1 in 5 law grads would make a bad financial decision by going to law school.

Although S&M present their findings as empirically robust, applying sophisticated econometrics and the latest theories of labor economics, it is plain that these three flaws were choices they made, mostly beneath the surface, which had the effect of artificially inflating their reported JD earnings premiums. This study well illustrates why many people are properly skeptical of proclaimed statistical findings (“lies, damn lies, and statistics”).

If S&M want to fairly determine “The Economic Value of a Law Degree” (their new more sober title), they should correct all three flaws and let us see how the numbers turn out. (A legitimate empirical journal, I assume, would require them to address these questions.) Their results would still have zero predictive value (these findings are not historic norms), but at least they would be less distorted and the information might be interesting. Remember, it was S&M who ominously warned that the financial futures of thousands of potential law students are at stake. If S&M correct these flaws, there is no doubt that the JD premium they tout will come down substantially, and a sizable percentage at the bottom will show negative value.

In this post, I have exposed how S&M substantially overstated the value of a law degree at the bottom quartile. In a follow up post I will show how they significantly understate the risk law students face. To briefly preview the argument, I will make two quick points.

S&M cannot properly assert (as they do now), “See, historical norms prove lifetime results are great, even at the bottom, so go to law school now.” This ignores a critical consideration. Economists agree that people who enter job markets during a recession experience lower lifetime earnings as a result. Given that the legal job market is mired in a severe multi-year recession that shows no real signs of easing (and multiple informed observers of the legal profession believe this is the new normal), S&M must offer a risk adjusted assessment of the chances that someone who enters the job market three years from now (after starting law school this fall) will obtain the “historic earnings norms” they claim exist. Students who entered law school 3-6 years ago, graduating in the heart of the legal recession, have a reduced chance of obtaining the lifetime earnings values S&M identify. S&M say nothing about the necessity to adjust their earnings figures to account for this risk.

Second, a useful risk assessment must be done on a school-by-school basis. Law grads from Harvard, Chicago, Penn, etc., have a great chance at high lifetime earnings, while grads of Thomas Jefferson are likely to end up with poor lifetime earnings. That is how my book examines the risk of attending law school, while S&M say absolutely nothing about particular schools. For this reason, my analysis is far more useful and reliable than their broad reassurances that law grads at the 25th percentile can expect “hundreds of thousands of dollars” in net lifetime earnings.