Following up on my previous post, Deborah Jones Merritt (Ohio State), The Bar Exam Is Broken:
Robert Anderson (Pepperdine), Some Skepticism About Bar Exam Skepticism:
The pass rate on the California bar exam last July was the lowest since 1984. This has sent many people scrambling for an explanation for the dive in the pass rate. In reality, however, the reason for the drop is not mysterious. The main reason the bar pass rate dropped is that as law school applications declined beginning in 2010, schools have admitted weaker and weaker incoming classes that were less able to pass the bar. Although other factors likely played a role, especially the increasingly predatory law school transfer market, the decline in applicant quality is the main cause of the current "crisis" in the California bar exam.
The stress of declining bar pass rates has prompted many critics of the bar exam to emerge, especially among California law school deans. The California deans predictably contend that the passing score for the bar exam is too high in California. Other critics (including some law school deans) go farther, however, arguing that the exam lacks "validity" even if the passing score were changed. That is, the critics believe the exam doesn't test the skills that it is intended to test, regardless of how difficult it is to pass. According to these critics, although lowering the passing score in California might help, really the solution is to revamp the test altogether.
I refer to these critics as "bar exam skeptics," in the sense that they express profound doubt about the conventional wisdom that the bar exam tests "minimal competence" to practice law. One of the more thoughtful skeptics is Professor Deborah Merritt. ...
[T]here is a big problem for those who contend that the bar exam does not test the skills needed for law practice. The score on the bar exam tends to be very highly correlated with law school GPA. The correlation is approximately .68, according to this New York Bar study (see Table 5.15), meaning that law school GPA explains about half the variation in bar exam scores. This figure is consistent with other analysis by the National Conference of Bar Examiners, which reports that law school GPA correlates with the Multistate Bar Exam score at .55-.70. These very high correlations strongly suggest the bar exam and law school grades are measuring the same type of abilities. Indeed, first year law school GPAs typically correlate with upper-division GPAs at about .75, suggesting that law school grades are almost as good at predicting the bar exam score as they are in predicting other law school grades. ...
[T]o argue that the bar exam does not test the skills needed for law practice is to argue that law school grades to not test the skills needed for law practice. And to argue that law school grades do not test the skills needed for law practice is to argue that law firms are profoundly mistaken about the skills needed for law practice. That is a debate I am willing to entertain, but it's an uphill argument for law school professors and deans to argue that they know more about the skills needed for practicing law than do law firms themselves.
Deborah Jones Merritt (Ohio State), Reflections of a Bar Exam Skeptic:
Robert Anderson has posted a thoughtful comment on the bar exam in which he dubs me a “bar exam skeptic.” I accept the label with pride. ...
[T]he bar exam pipes are leaking and people are paying attention. The leak doesn’t mean we should patch things up just to revive pass rates; the bar exam should measure competence, not admit a predetermined number of lawyers. But now that people are paying attention, this is a good time to consider whether we’re using the right type of filter and piping in our rather antiquated system.
As Professor Anderson suggests, the bar exam mimics law school exams. ... [But if] the bar exam simply tests what is learned in law school, why aren’t graduates ready to take the exam a week after graduation — saving themselves thousands of dollars and putting their hard-earned knowledge to work more quickly? And why are law schools creating special courses or workshops to prepare students for the bar exam, if that preparation is already happening throughout the law school curriculum?
The answer lies in the M word, memorization. The bar exam requires applicants to store an astounding number of detailed legal principles in memory. The adult human brain is not good at memorization; most faculty members can’t remember the names of sixty students without several weeks of practice or the use of a seating chart.
The bar exam no longer tests just basic principles in each subject; it requires test-takers to recall complex details. The need to memorize all of that material is why the bar exam is so arduous and expensive. ...
Is this exercise, which relies so heavily on memorization, a valid test of an aspiring lawyer’s competence? Can’t we do better for a billion dollars a year? Or do better at a much lower cost? As I suggested in a recent essay, there are many ways to reduce the bar exam’s overreliance on memorization. ... We could also modify the exam to measure cognitive skills (like interviewing, fact gathering, counseling, and negotiating) that the current exam ignores. Law schools now teach those skills, students fill those classes, and employers increasingly seek them in hiring. There are creative ways to make sure that new lawyers possess those competencies – for far less than a billion dollars a year.
Derek Muller (Pepperdine), Does the Bar Exam Adequately Test Prospective Lawyers' Minimum Competence?:
The critiques of the bar exam have grown louder over the last few years on the heels of declining bar pass rates. But the most popular critiques have changed somewhat. It used to be that external factors--such as the ExamSoft debacle — were a target. Then came charges that the bar exam was harder than usual. But the most recent charges are actually quite a longstanding critique of the bar exam — it simply isn't a good measure of prospective lawyers' "minimum competence." ...
[W]hat ought the bar test? And, perhaps a more subtle question: what if it turns out that the answer to what the bar ought to test actually aligns quite closely with the results from the existing bar exam?
A study in 1980 in California is one of the most impressive I've seen on this subject. And while it's a little old, it's the kind of thing that ought to be replicated before state bars go about making dramatic changes to their exams or scoring methods. ...
It might be that there are different things we ought to be testing, or that this experiment has its own limitations (again, I encourage you to read it if you're interested in the details). But before anything is done about the bar exam, it might be worth spending some time thinking about how we can evaluate what we think ought to be evaluated--and recognize that there are decades of studies addressing very similar things that we may ignore to our peril.
Prior TaxProf Blog coverage:
- July 2016 California Bar Exam (Nov. 22, 2016)
- More On The California Bar Exam (Dec. 5, 2016)
- Who Is To Blame For UC-Hastings 'Horrific' 51% Bar Pass Rate? (Dec. 7, 2016)
- July 2016 California Bar Exam Results: Nine Law Schools (Including UC-Hastings) Are At Risk Of Failing ABA's Proposed New Bar Passage Accreditation Standard (Dec. 13, 2016)
- Anderson: Deans, Denial, And The California Bar Exam (Dec. 20, 2016)
- Despite 'Horrific' 51% Bar Passage Rate, New UC-Hastings Dean Says School Is Poised To 'Catapult Into National Preeminence' Using NYU As A Model (Dec. 21, 2016)
- California Law School Bar Pass Rates Recalculated For New York: Stanford, UCB, USC > NYU; UCI, UCLA > Columbia; Chapman, Loyola, McGeorge, Pepperdine, Santa Clara, UCD, USD > Fordham (Dec. 22, 2016)
- More On The July 2016 California Bar Exam (Dec. 26, 2016)
- California’s New Bar Exam Format And ABA’s Proposed 75% Bar Passage Requirement Will Adversely Impact Diversity, Women, And Access To The Legal Profession (Jan. 30, 2017)
- Deans Of 20 Of California's 21 ABA-Accredited Law Schools Ask State Supreme Court To Lower Bar Exam Pass Score (Feb. 2, 2017)
- ABA 'Overwhelmingly' Rejects 75% Bar Passage Requirement (Feb. 7, 2017)
- California Law Deans Take Bar Exam Complaints To Lawmakers; State Bar Director Admits There Is 'No Good Answer' For High MBE Pass Score (Feb. 15, 2017)
- Beyond The Cut Score: Piercing The Veil Of The California Bar Exam's Validity (Mar. 2, 2017)
- Lawmakers Ask Calif. Chief Justice To Cut State Bar Exam Score (Mar. 9, 2017)
- UC-Hastings Dean: The California Bar Exam Flunks Too Many Law School Graduates (Mar. 27, 2017)