Paul L. Caron

Tuesday, July 23, 2013

Tamanaha: How 'The Million Dollar Law Degree' Study Systematically Overstates Value

TamanahaBrian Tamanaha (Washington U.), How “The Million Dollar Law Degree” Study Systematically Overstates Value: Three Design Choices that Improperly Skewed the Results:

Michael Simkovic and Frank McIntyre (S&M) have created a sensation with the release of their new study purporting to show that a law degree has a value of $1,000,000. Legal educators cheered and skeptics jeered.

S&M pitch the article as a crushing empirical refutation of the argument I press in Failing Law Schools that getting a law degree today can be financially risky, especially for students who attend expensive low ranked law schools. Simkovic writes, “we disagree with [Tamanaha’s] conclusions about the riskiness of a law degree because data on law degree holders does not support his conclusions.” So confident are they in their findings that they warn prospective students: “many college graduates who follow the critics’ advice and skip law school will forego a lucrative career and face higher long-term risks of financial hardship.”

All sides in this debate carry the same responsibility, of course, because if S&M are wrong, people who follow their advice risk serious financial hardship. Thus it is incumbent on all of us to consider these issues carefully and honestly.

Everyone agrees that many lawyers make a good living, so the significance of their study is at the bottom, where the greatest risk lies. S&M assert that even law graduates at the lower end of the earnings spectrum, including grads who don’t land jobs as lawyers, stand to earn “hundreds of thousands” of dollars over what they would have obtained with a bachelor’s degree alone. Great news for all!

Unfortunately, it’s not true.

Let me state up front that their study contributes to the debate over the value of a law degree. Let me also confirm that their study is far more sophisticated than my admittedly crude efforts. But as the old saying goes, it is far better to be approximately right than precisely wrong. Despite having the external trappings of precision and rigor, their study is faulty and misleading.

S&M Overstate the Economic Value of a Degree Over the Past Two Decades

Their analysis is based upon data provided by United States Census Bureau’s Survey of Income and Program Participation (SIPP), which tracks earnings over time by educational degree. The authors examine the earnings of people with law degrees from 1996 to 2011, and compare this to the earnings of people with bachelor’s degrees (in selected majors) over the same period. The “JD earnings premium” they identify is the difference between the two.

The great strength of their study, S&M say, is that it is a “long term” examination which establishes “historic norms”—thereby providing a much more reliable picture of lawyer earnings than my own immediate short term focus. The credibility of their study rests entirely upon this claim.

All their talk of “historic norms,” however, is puffed up exaggeration: 16 years is neither “long” nor remotely enough time to establish “historic norms.”

Had S&M extended their study by just four years—to 1992, covering twenty years in all—they would have found reduced “earnings premium” for law grads. This is because the legal job market suffered a severe recession from 1992 to 1998, while the general economic recession had ended in early 1992. (The downturn in the legal market is visible in Figure 2, p.9.) Since the “earnings premium” is the difference between the earnings of bachelor’s degree holders and law degree holders, it constricts when the general economy does relatively well while the legal market does relatively poorly. That was precisely the situation between 1992 and 1998.

There is no doubt that including 1992 to 1995 in their study would measurably reduce the “earnings premium.” In Figure 6 (p. 35), S&M set forth the earnings premium in four blocks: 1996-99, 2000-03, 2004-07, and 2008-2011. 1996-99 has the lowest earnings premium among the four (substantially below average)—and it is likely that the 1992-95 premium would be lower still because the legal market was down for all four years, whereas it was beginning to recover in 1998. Had 1992-95 been included, it would have constituted 1/5 of the total, undoubtedly bringing down the overall “earnings premium” by no small amount.

As other critics have pointed out, furthermore, their study does not include the earnings of the graduating classes of 2009, 2010, and 2011 (not to forget 2012 and 2013)—which have suffered devastatingly poor job results. (The earnings included in the study through 2011 are only for 2008 grads and earlier.)

To be clear, I’m not asserting that S&M began to count income in 1996, rather than 1992, deliberately to boost the “earnings premium” by shaving off four of the worst law degree earning years in the past 20 years, although it had this effect. No doubt they can offer reasons to explain their decision. (Although SIPP was redesigned in 1996, there are surveys for 1993 and 1992, which allow continuity; SIPP reports go back to the mid-1980s, first appearing in the middle of yet another recession in the legal market.) And they cannot be blamed for leaving out post-2008 law grads because the latest SIPP was from 2008—though this information will be available when the next SIPP is released.

Why these crucial gaps in information are present in their study is irrelevant to the more pressing issue of the reliability of their findings. As a consequence of these gaps, their study overstates the actual JD “earnings premium” over these 20 years. Had the very same study been done, but with full earnings data from 1992 through 2011 on all people with law degrees, lawyer earnings would be lower. This is certain.

This conclusion completely undermines the validity of their figures. Why should anyone put stock in “earnings premium” numbers derived from a study limited to 16 years, when extending the study just 4 more years, to 20, would produce notably lower figures? Why should we take these numbers as reliable indicators when forthcoming SIPPs will begin to show the devastating economic results suffered by recent law grads (2009, 2010, 2011, 2012, 2013 and counting), none of which are reflected in S&M’s figures?

Genuine long term historical studies span multiple cycles, several generations, because only then can the distorting impact of contingencies—like starting near the end of a recession (1996) versus at the beginning (1992)—even out over multiple up-and-down cycles. This may require data over several centuries. Their study does not even cover one full recession in the legal market. In fact, their 16-year slice is unrepresentative of long-term trends because, while it leaves out significant negative effects from two recessions, it fully captures the positive effects of the boom years in the legal market of the mid 2000s. (A fortuitous slice indeed for a study that aims to prove the high value of a law degree.)

It is exceedingly rare to find reliably predictive “historic norms” in the social sciences because social life is too complex and circumstances are constantly changing. (That’s why economists are so often wrong in their predictions.) Industries rise, transform, and expire; trades and professions wax and wane; formerly dominant national economies fall into decline while others take over prominence. S&M’s bold assertion that their 16-year study establishes valid “historic norms” on law degree earnings would be scoffed at by social scientists who take the notion of “historic norms” seriously. That is more than enough time to confirm norms governing the mating behavior of fruit flies, but 16 years is laughably inadequate for predicting something as complex and subject to change as the lifetime earnings of future law grads.

S&M have produced a narrow, partial, time-bound study that has zero predictive relevance for anyone thinking about attending law school today. The most we can glean from it is that a bunch of law grads will probably end up doing well financially—but we knew that already. This criticism on its own is sufficient to debunk the entire validity of their study, but it suffers from additional flaws that merit mention.

Their Earnings Premiums Are Artificially Inflated in Two Ways That Fail to Account for Ability

Picture two groups of recent college graduates with bachelor’s degrees. One group achieved above average GPA’s and standardized test scores, they attended better colleges, attached greater value to education, had higher expectations about future salaries, and came from wealthier families. The second group was average on all these characteristics.

Most people would assume that over time the first group on average would earn more money than the second group. And they would be right. In addition to the fact that people in the first group have greater ability and motivation, the offspring of people with more money tend to do better because they have more social and economic advantages to leverage (that’s American meritocracy in action).

As S&M confirm, law grads as a group are above average in all these characteristics. They also find (using National Education Longitudinal Survey data) that these characteristics, individually and collectively, are associated with higher earnings—entirely apart from the law degree itself. This is relevant to their study because, as they observe, “If ability sorting is positive, then some portion of the earnings premium associated with a law degree is due to differences in ability levels, and the causal effect of the law degree on earnings will be smaller than the observed differences in earnings.”

After running their analysis, S&M find that “a typical law degree holder would earn 10.3 percent more than a typical bachelor’s degree holder, even if the law degree holder had chosen to terminate his education with a bachelor’s degree. Column 2 suggests a slightly lower 8 percent, as it ignores differences in expectations.”

Thus, typical law grads can expect roughly10% higher earnings than average bachelor’s degree holders, entirely apart from the JD, because as a group they are smarter, more motivated, and come from wealthier families, etc. According to their study design, S&M should therefore subtract this 10% because that amount cannot be counted as part of the “JD earnings premium.”

Surprisingly, without explanation, S&M do not subtract this amount. They downplay the size of earnings attributable to ability, saying, “we find evidence of modest ability sorting that explains only a small fraction of the observed law degree earnings premium.” [Comment: 10% of average bachelor’s earnings is a sizable chunk.] They add that if they stripped away their controls the difference falls to 5% (as if it is fine to strip away controls). Then S&M include the entire amount within the JD earnings premium, forgetting their earlier statement that earning attributable to ability should not be counted. This move, of course, gives the “earnings premium” an under-the-covers boost. (Elsewhere in the paper they speculate that SIPP understates earnings at the high end, so perhaps they ignore the “ability earnings” 10% as a rough compensating trade off; if so, this would introduce yet another distortion, because the earnings we care about are at the low end, which they say are over-reported in SIPP.)

For what purports to be a rigorous empirical study, S&M’s treatment of this factor is ad hoc, and, most problematically, it slants the findings in favor of the thesis they set out to prove: that a law degree has positive value even at the low end. This move on their part was not a design flaw but a departure from their stated design. Empiricists who aspire to rigor don’t get to ignore inconvenient findings and make fudges in their favor.

Even if earnings attributable to ability are “small” (which does not appear to be the case), they cannot be included in the “JD earnings premium” not only because it is false to do so, but also because the values they come up with are used to identify the point at which law grads begin to reap a negative return from their law degree (i.e. when they would have been better off skipping law school altogether). So it matters.

There was a simple way for S&M to account for the 10% ability earnings: increase the average bachelor’s earnings figures by 10%; whereupon, the JD premium they calculate would be the difference between how much law grads earn and this higher figure. Obviously, had they done this, the JD premium would fall. That they failed to do this is not right.

S&M did something else related to ability that produced an even greater distortion. They claim the “JD earnings premium” is the amount law grads earn over what they would have earned had they only gotten a bachelor’s degree. In order to figure out this difference, we must know what they would have earned without the JD.

Recall that law grads as a group have above average GPA’s, test scores, and motivation, they have more family wealth and went to better colleges, etc. And keep in mind that many bachelor’s degree holders would not get into law school because they lack the requisite motivation and credentials—most SIPP interviewees obtained their law degree in the 1970’s, 1980’s, and 1990’s, when there were far fewer law schools and a large percentage of applicants were rejected (compared to the loose 80% admittance rate at bottom schools today).

Based upon these considerations, it is reasonable to surmise that the people in the “smarter-more motivated-pool of potential law students” who choose to forego law school would earn at least as much as the median earnings of a bachelor’s degree holder. After all, S&M found that their characteristics correlate with 10% earnings above average bachelor’s earnings (and average is higher than median).

When S&M selected the baseline to calculate the earnings premium, however, they did not use the median bachelor’s earnings, but instead they compared the earnings between law grads and bachelor’s holders at each quartile: law grad earnings at 25th percentile were compared to bachelor’s at 25th percentile; law grads at 50th percentile were compared to bachelor’s at 50th percentile; law grads at 75th percentile were compared to bachelor’s at 75th percentile.

But this raises a large objection: Why should we expect that bottom quartile law grads would earn at the bottom quartile of bachelor’s degree holders had they not gone to law school? S&M already told us that law grads in general have above average characteristics that correlate with earnings 10% above average bachelor’s holders!

If S&M had instead compared law grad earnings at all three quartiles to bachelor’s earnings at 50th percentile, the earnings premium of law grads at the 75th percentile would go up, the premium at the 50th percentile would remain unchanged, and the premium at the 25th percentile would plummet.

This conclusion casts huge doubt on S&M’s results and bears repeating. If S&M had assumed that potential law students who forego law school would have earned what median bachelor’s holders earn—which seems the most logical assumption given the superiority of the law student pool—then the “earnings premium” for law grads at the 25th percentile falls significantly.

S&M might respond that just because the pool of law grads is superior on average doesn’t mean bottom law grads are more capable than median bachelor’s holders. Perhaps. But then they should compare the 25th percentile law grad earnings to bachelor’s earnings at, say, the 40th percentile.

S&M might then say that they only have quartile numbers, so comparisons against intermediate percentiles are not possible. Fine. Then run two comparisons (at 25th and at 50th of bachelor’s earnings), display the results, and let readers decide which is the more apt comparison. When faced with these two choices, most prospective law students (a smart bunch) would correctly surmise that it makes more sense to compare the 25th percentile of law students against bachelor’s holders at the median.

In light of the momentous implications of this choice for their findings, it is remarkable that S&M do not even discuss whether they have made the correct comparison. They simply assert that it is appropriate to examine “the 25th percentile of earners with a given level of education [bachelor’s] compared to the 25th percentile of earners with a higher level of education [law degree]. Such a distributional analysis would test claims that advanced degrees may not benefit less capable students as much as they benefit average or above average students.” What S&M apparently failed to appreciate (or decided to ignore) is that, given the higher-level talent pool of law grads, “less capable” law grads can still be as capable as median bachelor’s degree holders. This seems elementary.

A Compromised Study

A striking feature of S&M’s study is their chest-pounding confidence. Social scientists usually are very cautious about making causal claims owing to the complexity of social factors and how they interact, unobserved influences, and limited data points. No false modesty for S&M. They claim they have proven a direct causal relation between a law degree and specified lifetime earning values. They claim they have established reliable “historic norms” for law degree earnings (based upon a “long term” study covering only 16 years). They warn prospective students who do not accept their findings—instead foolishly believing benighted critics of legal education—that they will forego a lucrative career and risk future financial hardship. And their original title blared, “The Million Dollar Law Degree.”

S&M gush about their robust showing that law grads, including those on the low end, reap oodles of surplus value from a law degree:

These results suggest that even at the 25th percentile, the value of a law degree exceeds typical net-tuition costs by hundreds of thousands of dollars.” At the mean and 75th percentiles, the difference is close to one million dollars. We therefore reject the claim that law degrees are priced above their value. Indeed, the value compared to net-tuition prices suggests that legal education is a competitive market in which surplus redounds to the benefit of student-consumers. (41)

Not really. Their study is full of holes and they made a series of dubious choices that were mostly hidden. Rather than conduct an open-minded inquiry into the economic value of a law degree, it appears that S&M were hell bent on proving that a law degree pays off handsomely for nearly all law grads. This orientation compromised the soundness of their study.

The paragraph quoted above oozes exaggeration. According to S&M, the present lifetime value of the JD earnings premium for law grads at the 25th percentile is $350,000. Minus federal taxes (following their formula), this comes to $262,500. Subtracting three years of tuition (they calculate scholarship discounted average tuition at $30,000 x 3) leaves us with $172,500. For male law grads at the 25th percentile, the same calculation comes to $147,000. Needless to say, that’s not hundreds of thousands of dollars. Overstatement runs through their article.

And these numbers are further inflated by other questionable choices they made. Providing no support whatsoever, S&M assert that law students earn $24,000 while in law school (1st year $5,000; 2nd year $7,000; 3rd year $12,000)—this is an implausibly high figure at a time when many law students work in unpaid externships during the summer and wait tables on the side. (S&M reduce opportunity cost by $24,000.) They do not include living expenses in the cost of a law degree because, they reason, students must pay this even if do not attend law school (ignoring the higher cost urban areas many law schools are located in); we can let that pass, but that does not explain why they take no account of the interest on loans taken out to pay for living expenses (borrowing in the $50,000-plus range for living alone), which working people cover with current income. Making adjustments for these fudges would bring the net earnings premium down even more.

Still, one might think, a net lifetime premium of $125,000 is pretty good. Keep in mind, however, this is earned over 42 years. Three years of law school at this earnings premium gets you an additional $250 a month (over a bachelor’s degree), and hopefully a rewarding career. Nothing to sniff at, to be sure, but this is hardly a generous “surplus value” redounding to the benefit of law grads that S&M so proudly mention. (Tellingly, Figure 1. shows that lawyer earnings have remained flat in real terms since 1998, while law school tuition has risen far above the rate of inflation, so we have been absorbing an ever greater share of this surplus value over time.)

Now, this discussion is completely artificial because it takes S&M’s stated JD earnings premiums at face value, when I have already shown that the $350,000 figure is substantially overstated owing to three major flaws in their study: 1) they do not count the full effects of 2 significant bouts of reduced lawyer earnings in the past twenty years (4 years of legal recession from 1992-95, and the dismal fate of law grads from 2009-2011); 2) they should have subtracted the 10% increase in earnings correlated with higher ability, motivation, etc., from their “JD earnings premium;” and 3) when calculating the JD earnings premium at the 25th percentile, they should have used bachelor’s degree earnings at the 50th percentile as the baseline.

Fixing any one of these flaws would bring the $350,000 down by amounts that matter (moving the total after taxes and tuition ever closer to the negative line); fixing all of them would flip their results from reassuring to positively worrisome. And we must keep in mind that the issue at stake does not stop at the 25th percentile. Let’s say that—after fixing these flaws and netting costs and taxes—law grads at the 25th percentile still get a positive gain of $25,000. S&M don’t get to declare victory and walk away, saying they have proven critics of legal education wrong. What about 20th percentile law grad earnings? If those folks are in the negative, it is hardly a vindication of the economic value of a law degree to say that “only” 1 in 5 law grads would make a bad financial decision by going to law school.

Although S&M present their findings as empirically robust, applying sophisticated econometrics and the latest theories of labor economics, it is plain that these three flaws were choices they made, mostly beneath the surface, which had the effect of artificially inflating their reported JD earnings premiums. This study well illustrates why many people are properly skeptical of proclaimed statistical findings (“lies, damn lies, and statistics”).

If S&M want to fairly determine “The Economic Value of a Law Degree” (their new more sober title), they should correct all three flaws and let us see how the numbers turn out. (A legitimate empirical journal, I assume, would require them to address these questions.) Their results would still have zero predictive value (these findings are not historic norms), but at least they would be less distorted and the information might be interesting. Remember, it was S&M who ominously warned that the financial futures of thousands of potential law students are at stake. If S&M correct these flaws, there is no doubt that the JD premium they tout will come down substantially, and a sizable percentage at the bottom will show negative value.

In this post, I have exposed how S&M substantially overstated the value of a law degree at the bottom quartile. In a follow up post I will show how they significantly understate the risk law students face. To briefly preview the argument, I will make two quick points.

S&M cannot properly assert (as they do now), “See, historical norms prove lifetime results are great, even at the bottom, so go to law school now.” This ignores a critical consideration. Economists agree that people who enter job markets during a recession experience lower lifetime earnings as a result. Given that the legal job market is mired in a severe multi-year recession that shows no real signs of easing (and multiple informed observers of the legal profession believe this is the new normal), S&M must offer a risk adjusted assessment of the chances that someone who enters the job market three years from now (after starting law school this fall) will obtain the “historic earnings norms” they claim exist. Students who entered law school 3-6 years ago, graduating in the heart of the legal recession, have a reduced chance of obtaining the lifetime earnings values S&M identify. S&M say nothing about the necessity to adjust their earnings figures to account for this risk.

Second, a useful risk assessment must be done on a school-by-school basis. Law grads from Harvard, Chicago, Penn, etc., have a great chance at high lifetime earnings, while grads of Thomas Jefferson are likely to end up with poor lifetime earnings. That is how my book examines the risk of attending law school, while S&M say absolutely nothing about particular schools. For this reason, my analysis is far more useful and reliable than their broad reassurances that law grads at the 25th percentile can expect “hundreds of thousands of dollars” in net lifetime earnings.

[Cross-posted at Balkinization.]  Prior TaxProf Blog coverage:

Update #1:

Update #2:

Legal Education | Permalink

TrackBack URL for this entry:

Listed below are links to weblogs that reference Tamanaha: How 'The Million Dollar Law Degree' Study Systematically Overstates Value:


" job market suffered a severe recession from 1992 to 1998..."

I *was* right! I'd been accepted to BU Law in the late '80s but decided not to go in part because I saw a shortage of legal jobs coming -- and thought that I had just predicted the current one early, but apparently not. And I'd been entering the job market -- with big debt -- during this job market recession.

But what I'm waiting for -- and I do believe it will happen by the end of the decade -- is *anyone* being permitted to take the bar exam, an end to the prerequisite of having graduated from an ABA-approved law school. The business community is pushing for this, and states would dearly love to pay less for public defenders. In the era of paper books, law school was a necessary instruction on how to look things up -- laws, cases, deeds and the rest. Now that is all electronically available to anyone and specialized library skills are no longer needed -- even LEXIS is looking at some serious competition from free databases that are far in advance of what LEXIS itself was, circa 1990.

Combined with the abysmal first-try bar exam failure rates of some law schools, the argument could be made that law school should not be mandated -- through much of history, law schools didn't even exist.

Posted by: Ed Cutting, EdD | Jul 25, 2013 12:34:45 PM

MacK, the most persistent of the Campos trolls, should not be permitted to get away with his bald-faced lie that Steve Diamond engaged in "online attacks against [his] deceased father." He did nothing of the kind. Diamond rather expressed wonder that the pathetic blowhard MacK could have been sired by a father, for whom Diamond had nothing but praise. I agree it was off-topic (but it was not an attack on anyone's father), but the Faculty Lounge folks have been deleting quite a lot lately, including some of MacK's mischief. MacK, who has hidden behind his pseudonym--while insulting Matt Bodie, Eric Miller, Stephen Diamond, and lots of others--is in no position to complain.

Posted by: Brian | Jul 24, 2013 4:33:01 PM

He says on another post on this blog that law professors are among the low-paid law school graduates. The man has absolutely no shame. This is all too clear.

Posted by: other things steve diamond says | Jul 24, 2013 2:22:47 PM

The Poverty of Historicism, by Karl Popper.

Posted by: teapartydoc | Jul 24, 2013 4:47:04 AM

It is interesting that Steve Diamond (also known as btraven and Anon) has a posting trumpeting victory over the "good cop" Tamahama and demanding that Tamahama apologise to various of Diamond's designees (this from a man who has yet to apologise for trying to support his argument by making off topic online attacks against my deceased father, deleted by shocked admins at the Faculty Lounge Diamond went too far for even them! Ever wonder why you might not want Steve Diamond to know your name?)

A careful read of his posting though shows that nowhere in it does he deal with the enormous problems which Tamahama raised - he does not address the selection bias for the comparison BA holders, he sniffs off the issue of ignoring the class of '92 outcomes by quoting someone else. Mostly he raised arguments about tone, which well... see the paragraph above.

There is a basic problem with the Simkovic/McIntyre study which a number of the people even Diamond cites as having given is a positive peer review have identified in various - most of which are very similar or identical to those cited by Tamahama. The most acute is that Simkovic/McIntyre seek to compare law students with a control group of BA only holders - cohort for cohort. However, this is hardly fair.

Even today, Seton Hall law school, relatively lowly ranked and regarded, claims an overall median undergraduate GPA of 3.50 (B+) and median LSAT score of 158 - which would put most of its students in the upper 25% of their graduating BA class and upper quarter of LSAT test takers. Santa Clara, a lower ranked school, still claims a median LSAT of 160 and a median GPA of 3.25. Even the bottom 25% of those admitted to Santa Clara have an LSAT of 158 and a GPA of 3.07. Nonetheless, Steve Diamond (and Simkovic) thinks his low ranked law school is selecting from the bottom 25% of BA graduates - is he willing to say that in writing?

Posted by: MacK | Jul 24, 2013 3:08:00 AM

"If you don't understand it, you're not smart enough to dismiss it."

If you're not smart enough to understand it, then are you smart enough to benefit from going massively into debt to attempt acquisition of a questionable credential from a lower ranked law school, which is worth little afterwards (even if you attempt succeeds) if you can't pass your state's bar exam?

Probably not.

Posted by: Desi Erasmus | Jul 24, 2013 2:38:32 AM

If you are a prospective law student that is "not smart enough" to understand the financial NPV-IRR MBA gibberish presented in the working paper, but wants to know the "value" of a law degree, please do the following:

1) Go to
2) Pay $25 and place an ad for “Entry-level Attorney” in a metropolitan area that you wish to practice.
3) Write a description for your ideal job out of law school. For example, “mid-size litigation firm seeking entry level attorney”
4) Put in a salary that you would like or deem reasonable for an entry level attorney, for example, $70k or even $85k as that is what most law school claim is the average salary of law graduates.
5) Watch in horror as your email inbox blows up with hundreds of resumes within just two to three few days
6) Decided whether you want to compete in this market for a job after you graduate and what "value" the law degrees of these people is worth.

Posted by: Dave Camel | Jul 23, 2013 3:00:57 PM

In addition to Tamanaha's cogent criticisms, accepting the results of this study as anything relevant to today's market requires accepting the very controversial premise that historical returns on a JD will be realized in future. There's little reason to accept that premise, however.

Posted by: Jobs | Jul 23, 2013 2:01:16 PM


We'll see if they incorporate the comments of Tamahama's and numerous others. Some would take day at most ... I'd be stunned if they did. Because most would undermine their argument massively

Posted by: Anon | Jul 23, 2013 1:36:43 PM

Anyone who is dismissive of Simkovic and McIntyre should read their working paper and rethink. If you don't understand it, you're not smart enough to dismiss it. If you do, and you think you've found a flaw, at that point you can start to make fun of them if you think it appropriate. Even that can be embarassing for you, since maybe it's you who missed something big. Prof. Tamanaha thinks he's reached the point of being able to say the paper is wrong, so he has a right to, but he does still risk embarassing himself. Note that he signs his name, though, which is also important to do. Note, too, that the S-M paper is a *working paper*, not a finished paper. They're quite open to suggestions for improvement. I've seen it presented at a conference and done that myself. Peer-reviewed papers aren't more trustworthy just because of their name--- it's because they get months of comments from other scholars before they're accepted for publicatoin.

A simple reason to pause before you judge S-M is this. Most critics say, "They must be wrong, because the demand for lawyers is weak." But that doesn't refute their results at all. They find that over a J.D.'s lifetime, he earns more than someone like him who didn't get a J.D. This can be true even if no J.D.'s get law degrees, just as a history BA increases earnings even though the history job market is weak. Indeed,the history BA's who got history jobs probably earn *less* than the "failures" who took business jobs. That's certainly true of econ PhD's who fail on the academic job market and have to take consulting firm jobs.

Thus, review your personal knowledge and decide if you know of people who got J.D.'s and didn't take law jobs who are worse off or no better off 20 years later than if they'd skipped law school.

Posted by: Eric Rasmusen | Jul 23, 2013 12:26:51 PM

"There are three kinds of lies: lies, damned lies, and statistics." This study is nothing but the kind of lie under #3.

Posted by: Dave Camel | Jul 23, 2013 12:06:24 PM

Prof. Seto, your criticism of Prof. Tamanaha's tone would have more resonance if law schools had not spent the last twenty years doing exactly what Simkovic and McIntyre appear to have done in their paper, i.e., making arbitrary choices in the selection and presentation of employment data on the basis of what made it easiest for them to sell seats to prospective law students.

I wonder what the 58% of Loyola-L.A. JDs from last year's class who weren't in full-time, bar-required jobs would have to say about the importance of Prof. Tamanaha's tone relative to his questions about Simkovic and McIntyre's methodology.

Posted by: Morse Code for J | Jul 23, 2013 11:29:28 AM

I don't know, I feel blatant attempts to muddy the waters through statistical manipulation warrant ad hominems (well, as much as those can be considered "ad hominems"). I could think of worse than those examples to call people who attempt to continue to mislead potential law students.

Posted by: NLL | Jul 23, 2013 10:31:00 AM

"Simkovic and McIntyre's analysis is the most sophisticated and careful we have seen to date, by orders of magnitude." And yet the study is still woefully incomplete and inaccurate.

Posted by: ... | Jul 23, 2013 10:29:45 AM


Out of respect for your professionalism, I removed from the Balkinization post "brazen bluff" and "sloppy." The other terms were entirely appropriate and descriptively accurate and well supported. (And what they did was sloppy, by the way, but I can see that it is an insensitive way to put it.)

Sorry about the S&Bs (also fixed on Balk). It was late last night when I posted it.

Posted by: Brian Tamanaha | Jul 23, 2013 10:29:21 AM

I love the fact that Interesting Post and Ted Seto's comments are right next to each other. With one you have the interests of students, practitioners, and society at large. With the other you the professoriate at a lower ranked school.

Posted by: Stan | Jul 23, 2013 8:56:05 AM

Prof. Tamanaha needs to tone things down: "puffed up exaggeration, a brazen bluff", "sloppy, ad hoc", "chest-pounding confidence". He explicitly attacks Simkovic's and McIntyre's motivations: "Rather than conduct an open-minded inquiry into the economic value of a law degree, it appears that S&B were hell bent on proving that a law degree pays off handsomely for nearly all law grads." I would have expected ad hominem attacks from Above the Law. They seem out of place in scholarly debate.

Simkovic and McIntyre's analysis is the most sophisticated and careful we have seen to date, by orders of magnitude. For the first time, they apply the methods of labor economics to a problem previously addressed largely through amateurish, back-of-the-envelope approximations. Whether they have correctly applied those methods, time will tell. But calling them names and questioning their motivations is unprofessional.

The standard scholarly response to an empirical study with which one does not agree is a more sophisticated empirical study that takes into account one's critiques. I look forward to reading Prof. Tamanaha's study.

Posted by: Theodore Seto | Jul 23, 2013 8:32:11 AM

This discussion is fascinating.

Since the year 2000, does anyone in legal academia really believe that a law graduate in the bottom half of the class at a good school (or in the bottom 90 percent at a bad one) is and will be economically better off than had he or she not gone to law school in the first place?

It just strains credibility. If you have had any contact at all with LS graduates 0 to 10 years out, the predictions of this article just are observably not true. The fact that there are angry scamblogs out there attests to this. You don't see angry pharma grads or philosphy majors, or quant jocks out there blogging their rage at their educational institutions.

Why do you see this in law?

Answer: The official position of the schools is so different from the experience of most graduates that it fuels their contempt.

Posted by: Interesting Post | Jul 23, 2013 8:21:04 AM

Hey, why stop in 1992 with the downturn, why not include the Reagan era boom?

Posted by: Anon | Jul 23, 2013 8:15:47 AM

There are so many flaws in this type of research that it is effectively worthless. The best information for prospective law students is 9-month post graduation employment stats from the school they are considering attending. That is the best glimpse into their future.

It is also amazing to me how much law school proponents disregard the importance of having some wealth in the near term. It is indisputable that a large number of recent grads will be astonishingly poor throughout their twenties, thirties, and probably a good part of their forties. That is the time when most people want to start a family, or at least enjoy their youth and their health by traveling and engaging in fun activities that require some money. Even if they manage to recover financially in their 50's, it might be too late to mean anything.

Posted by: JM | Jul 23, 2013 6:48:09 AM