TaxProf Blog

Paul L. Caron
Dean





Tuesday, May 30, 2017

Anderson, Merritt & Muller Debate: Is The Bar Exam Broken?

California Bar ExamFollowing up on my previous post, Deborah Jones Merritt (Ohio State), The Bar Exam Is Broken:

Robert Anderson (Pepperdine), Some Skepticism About Bar Exam Skepticism:

The pass rate on the California bar exam last July was the lowest since 1984. This has sent many people scrambling for an explanation for the dive in the pass rate. In reality, however, the reason for the drop is not mysterious. The main reason the bar pass rate dropped is that as law school applications declined beginning in 2010, schools have admitted weaker and weaker incoming classes that were less able to pass the bar. Although other factors likely played a role, especially the increasingly predatory law school transfer market, the decline in applicant quality is the main cause of the current "crisis" in the California bar exam.

The stress of declining bar pass rates has prompted many critics of the bar exam to emerge, especially among California law school deans. The California deans predictably contend that the passing score for the bar exam is too high in California. Other critics (including some law school deans) go farther, however, arguing that the exam lacks "validity" even if the passing score were changed. That is, the critics believe the exam doesn't test the skills that it is intended to test, regardless of how difficult it is to pass. According to these critics, although lowering the passing score in California might help, really the solution is to revamp the test altogether.

I refer to these critics as "bar exam skeptics," in the sense that they express profound doubt about the conventional wisdom that the bar exam tests "minimal competence" to practice law. One of the more thoughtful skeptics is Professor Deborah Merritt. ...

[T]here is a big problem for those who contend that the bar exam does not test the skills needed for law practice. The score on the bar exam tends to be very highly correlated with law school GPA. The correlation is approximately .68, according to this New York Bar study (see Table 5.15), meaning that law school GPA explains about half the variation in bar exam scores. This figure is consistent with other analysis by the National Conference of Bar Examiners, which reports that law school GPA correlates with the Multistate Bar Exam score at .55-.70. These very high correlations strongly suggest the bar exam and law school grades are measuring the same type of abilities. Indeed, first year law school GPAs typically correlate with upper-division GPAs at about .75, suggesting that law school grades are almost as good at predicting the bar exam score as they are in predicting other law school grades. ...

[T]o argue that the bar exam does not test the skills needed for law practice is to argue that law school grades to not test the skills needed for law practice. And to argue that law school grades do not test the skills needed for law practice is to argue that law firms are profoundly mistaken about the skills needed for law practice. That is a debate I am willing to entertain, but it's an uphill argument for law school professors and deans to argue that they know more about the skills needed for practicing law than do law firms themselves.

Deborah Jones Merritt (Ohio State), Reflections of a Bar Exam Skeptic:

Robert Anderson has posted a thoughtful comment on the bar exam in which he dubs me a “bar exam skeptic.” I accept the label with pride. ...

[T]he bar exam pipes are leaking and people are paying attention. The leak doesn’t mean we should patch things up just to revive pass rates; the bar exam should measure competence, not admit a predetermined number of lawyers. But now that people are paying attention, this is a good time to consider whether we’re using the right type of filter and piping in our rather antiquated system.

As Professor Anderson suggests, the bar exam mimics law school exams. ... [But if] the bar exam simply tests what is learned in law school, why aren’t graduates ready to take the exam a week after graduation — saving themselves thousands of dollars and putting their hard-earned knowledge to work more quickly? And why are law schools creating special courses or workshops to prepare students for the bar exam, if that preparation is already happening throughout the law school curriculum?

The answer lies in the M word, memorization. The bar exam requires applicants to store an astounding number of detailed legal principles in memory. The adult human brain is not good at memorization; most faculty members can’t remember the names of sixty students without several weeks of practice or the use of a seating chart.

The bar exam no longer tests just basic principles in each subject; it requires test-takers to recall complex details. The need to memorize all of that material is why the bar exam is so arduous and expensive. ...

Is this exercise, which relies so heavily on memorization, a valid test of an aspiring lawyer’s competence? Can’t we do better for a billion dollars a year? Or do better at a much lower cost? As I suggested in a recent essay, there are many ways to reduce the bar exam’s overreliance on memorization. ... We could also modify the exam to measure cognitive skills (like interviewing, fact gathering, counseling, and negotiating) that the current exam ignores. Law schools now teach those skills, students fill those classes, and employers increasingly seek them in hiring. There are creative ways to make sure that new lawyers possess those competencies – for far less than a billion dollars a year.

Derek Muller (Pepperdine), Does the Bar Exam Adequately Test Prospective Lawyers' Minimum Competence?:

The critiques of the bar exam have grown louder over the last few years on the heels of declining bar pass rates. But the most popular critiques have changed somewhat. It used to be that external factors--such as the ExamSoft debacle — were a target. Then came charges that the bar exam was harder than usual. But the most recent charges are actually quite a longstanding critique of the bar exam — it simply isn't a good measure of prospective lawyers' "minimum competence." ...

[W]hat ought the bar test? And, perhaps a more subtle question: what if it turns out that the answer to what the bar ought to test actually aligns quite closely with the results from the existing bar exam?

A study in 1980 in California is one of the most impressive I've seen on this subject. And while it's a little old, it's the kind of thing that ought to be replicated before state bars go about making dramatic changes to their exams or scoring methods. ...

It might be that there are different things we ought to be testing, or that this experiment has its own limitations (again, I encourage you to read it if you're interested in the details). But before anything is done about the bar exam, it might be worth spending some time thinking about how we can evaluate what we think ought to be evaluated--and recognize that there are decades of studies addressing very similar things that we may ignore to our peril.

Prior TaxProf Blog coverage:

https://taxprof.typepad.com/taxprof_blog/2017/05/anderson-and-merritt-debate-is-the-bar-exam-broken-.html

Legal Education | Permalink

Comments

I agree with UNE here, no one is entirely right or wrong.

But we are focusing too much right now on the third and fourth order effects, and missing how we got to this point.

Sure the bar exam is a flawed standardized test. But as my old psych professor used to say before an exam, “Yes I taught you that standardized tests are flawed. But we don’t have a better way to test whether you learned the material.”

Bar exam rates plummeted at the lower ranked schools because they admitted less qualified students to stay in business. Those schools admitted less qualified students, because of the substantial drop in law school applicants, particularly higher qualified students. Law school applications fell, because a significant number of grads from lower ranking schools struggle to obtain employment. Nobody wants to be unemployed 10 months after graduating from law school with $150k in debt.

There is no denying the struggles of the legal profession. The BLS reports under the employment outlook for lawyers that “competition for jobs should continue to be strong because more students graduate from law school each year than there are jobs available.” The BEA reports that the real GDP of the legal industry has declined 1.2% since 1998. The fundamental problem in the legal industry is not new technology, globalization, or some general trend impacting all industries. Many other industries in the United States are growing, employing more workers, and doing great. The fundamental problem with the legal industry is declining demand. Tort reform has crippled the legal industry and hurt consumers.

Republicans trashed trial lawyers and shifted the blame for rising healthcare costs from the insurance companies onto medical malpractice lawsuits. If a doctor makes a mistake resulting in millions of dollars in damages, the patient should be compensated. The Republicans convinced the people to go against their own self-interest and cap medical malpractice damages, just to stick it to the lawyers.

Consumers have been hurt in other contexts as well. Try opening a new bank account, obtaining a credit card, mortgage, cell phone, or taking a new job without agreeing to settle any disputes in mandatory arbitration. Those clauses protect big business and hurt consumers. Arbitrators are hardly impartial, when they rely on corporations for continuing business. Just look at Wells Fargo. They cheated customers. But customers signed an agreement when they opened Wells Fargo accounts that they would resolve all disputes in arbitration. One judge ruled that defrauded customers must settle their disputes in private arbitration. Wells Fargo is fighting to keep other lawsuits out of court as well. Wells Fargo knows they will get a more favorable result from an arbitrator compared to a jury.

We wouldn’t be debating the bar exam if tort reform hadn’t crippled the legal industry.

Posted by: anon JD/MD | May 30, 2017 11:45:49 AM

No one is entirely right or wrong here.

1) Bar exam rates are falling because law school admissions have fallen. Res ipsa, folks.

2) The calls for lowering the minimum pass scores on the CA bar exam are just CYA by law school admins to mask the problems caused by 1).

3) Yes, the bar exam is just a huge memorization test that has only passing resemblance to minimum lawyering competence; obviously many, many people pass bar exams who have never, inter alia, read a contract, taken a deposition, handled an IOLTA account, stepped foot in a courthouse, or have the foggiest idea how to practice an area of law beyond the basics of Wexis research and a smattering of basic legal writing.

4) Testing for things in the list in 3) would be difficult, to say the least. And the reality is that precious few law professors themselves have such skills, given the longstanding assembly lines of "HYS --> 1 year in the basement of a corporate law firm proofreading documents --> legal academia" and the even more ridiculous "PhD --> JD --> legal academia with 0.0 days' experience practicing law." You want to create "practice-ready" law school grads? Be prepared for a full-body transplant of the professoriate. And no, "experiential education" (i.e. let's assume that an internship or two will teach them everything they need) is not a solution.

Posted by: Unemployed Northeastern | May 30, 2017 9:32:56 AM