Paul L. Caron

Monday, January 30, 2017

California’s New Bar Exam Format And ABA’s Proposed 75% Bar Passage Requirement Will Adversely Impact Diversity, Women, And Access To The Legal Profession

TaxProf Blog op-ed:  California’s New Bar Exam Format in Conjunction with ABA’s Proposed Bar Pass Standard Will Adversely Impact Diversity, Women and Access to the Profession, by Dennis P. Saccuzzo & Nancy E. Johnson:

Considerable concern is being expressed concerning the effects on diversity and access to the profession due to proposed changes in ABA accreditation standards and changes in the format and scoring of the bar, such as those in California. According to Lawrence P. Nolan, President of the State Bar of Michigan, for example, ABA’s proposed amendments to the current accreditation standards will “adversely impact efforts to diversify the profession.”

Indeed, 90 law school Deans have asked the Council of the ABA Section on Legal Education and Admissions to the Bar to slow down and think about its proposal to tighten accreditation requirements on bar pass rates. Again, the effects on diversity and access to the profession are among the main concerns. Access not only includes women and minorities, but also non-traditional students such as those who have no family members who ever graduated from college. An important justification for lower tier law schools is that they increase access to the profession.

Now that the California State Bar has decided to change the format of the bar exam, it would seem prudent to ask if the changes will have any effect on access to the profession.

The changes, which will shorten the length of the exam from three days to two, increase the weight of the MBE from its current 35% to 50%, reduce the weight of the written part from its current 65% to 50%, and reduce the weight of the performance test, were made after years of deliberation and study.

At the heart of the decision by the Committee of Bar Examiners (“Committee”) to make the aforementioned changes is a retrospective study by Stephen P. Klein and Roger Bolus of all General Bar Exam (“GBX”) applicants from 2001 through 2010, a study that was commissioned by the California State Bar and published in 2011. According to the Committee, and based on the findings of Klein and Bolus, “administering a two-day examination would yield essentially the same results as those for a three-day examination…” and that it would be “more efficient.”

This optimism from the California State Bar is based on Klein and Bolus’ conclusion that reliability, a measure of internal consistency, will actually be improved by shortening the test from three days to two. All of this seems to fly in the face of one of the most fundamental principles of test theory. It is widely known, and provable with formulas, that the longer the test the higher its reliability.5 It is just as well known that the best way to increase a test’s reliability is to increase its length.6 So how could reducing the test increase its reliability?

In fact, Klein and Bolus’ conclusion was correct. Shortening the test did improve its reliability because of the increased weight given to the MBE. The Committee could even save another million dollars and increase the California bar exam’s reliability still further by going exclusively to a one-day MBE, with no written component. The reliability would be even higher because the 200-item MBE is always more reliable than the 8-item written portion of the exam (under the current format). Why not do this?

The problem with an MBE-only test is that it would disadvantage women and minorities because these groups tend to score more poorly on multiple choice tests than white men and tend to have lower LSAT scores. As it is, the new format is guaranteed to reduce the percentage of women who will pass and increase that of men.

Women were passing at a higher rate than men under the 65/35 written/MBE format because the extra weight given to the written slightly overcompensated for women’s disadvantage on the MBE. Klein and Bolus were aware that the pass rate for women will actually decrease under the new format. They stated, “Assigning equal weights [to the MBE and written portions of the exam] eliminates the differences in passing rates between men and women.” Based on the entire sample between 2001 and 2010, women passed the July exam at an overall rate of 55% and men at 52%. The new formula will eliminate this difference, according to Klein and Bolus. Thus, the first predictable consequence of the new format will be a reduction in the percentage of women who pass the exam and gain entrance into the profession.

We will explain why we believe a perfect storm is brewing that will result in some unanticipated and undesired consequences from the July 2017 California bar exam forward. This is not only because of changing conditions that were not anticipated, but also because of limitations in the Klein and Bolus study. The validity of this study may hinge on the characteristics of the population that was actually tested (2001-2010), whereas the characteristics of those of who will be taking the changed bar exam in July 2017 are different. These highly relevant differences may lead to distortions in the pattern of results that were not anticipated when the decision was made to change the format.

Because the key Klein and Bolus study was reported in 2011 and retrospectively examined performances from 2001 through 2010, these findings apply to that time frame and likely will not generalize so well to 2017 and later. This is because, as Erica Moeser from NCBE has so forcefully stated, there have been many changes in legal education the past several years. As Ms. Moeser noted, LSAT scores have declined, legal education has seen a rise in experiential learning, bar prep courses in law schools are being outsourced to bar prep companies, and the academic support services offered to students in the lower quartile may not be adequate.

We suggest that two factors may be more relevant in explaining the drop in bar pass rates, in conjunction with decreasing LSATs. First, a higher proportion of women study law than was the case in prior years, and second, the exam itself is standardized by equating the written portion to the MBE. That equating becomes less reliable when the population changes from one test to another in terms of the underlying ability measured by the test. The evidence of declining LSAT scores that Ms. Moeser cites is also evidence that the test equating is less reliable because the population has changed in terms of the underlying ability being measured, and that decrease in reliability will have consequences for those who score within the confidence interval around the pass line set by each jurisdiction.

Looking more carefully at the actual data from the most recent California bar exam administration in July 2016, the picture may be even more grim. Table 1 illustrates a variety of relevant features of the latest administration in July 2016. In this table, we give the rank order first-time bar pass results of all of California’s ABA-accredited law schools along with the various mean scores obtained by these schools on the exam. The table also shows the median and 25th percentile LSAT scores for the matriculants in the schools for 2013 and 2016, as well as the percentages of men versus women for these same time periods.

Table 1 (2017)

Inspection of Table 1 comparing the male-to-female ratio in matriculants in 2013 to that in 2016 reveals a clear trend of a higher proportion of women entering law school for most of California’s 18 ABA approved law schools. Most schools saw the percentage of women increase by two to four percentage points. Only USD saw a decline in the percentage of women (by one percentage point). LaVerne saw a whopping increase of 13 percentage points. Indeed, in 2016, women represented the majority of entering students for all schools except Stanford, USC, UCLA, and USD. The bottom line is that the percentage of women is increasing. As a corollary, given that more women will fail, if all else remains equal, relative pass rates at most law schools will decrease.

Inspection of Table 1 indicates a second clear trend: LSAT scores are falling to some degree for every school except UCLA and UC Hastings. Falling LSAT scores mean that bar pass rates will fall still further.

According to Susan Case, past Director of Testing for NCBE, the correlation between the MBE and LSAT is 0.57. This means that individuals with high LSAT scores tend to get higher MBE scores and individuals with low LSAT scores tend to get lower MBE scores. The increased weighting of the MBE will mean that the effect of decreasing LSAT scores will be amplified in terms of bar pass rates. Again, even without any change in format of the bar exam in California, falling LSAT scores mean bar pass rates will fall, with the greatest effect being for schools with the greatest decrease in LSAT scores. Increased weighting of the MBE beginning with the July 2017 exam will amplify that effect.

LSAT scores for schools in the top half of the table (the schools with the highest pass rates) have declined less than those for schools in the lower half. Therefore, a second clear consequence of the new format will be a widening gap between top tier schools and the so-called access schools. Bar pass rates will go down at a much faster pace for the schools in the lower third.

As a consequence of falling bar pass rates, many of the so-called access schools, already in trouble with ABA standards of accreditation as it relates to bar pass, will fall hopelessly below ABA criteria. Several ABA schools will simply be unable to meet ABA’s bar pass standards for accreditation. As the 90 Deans wrote in their letter to the ABA, “The California bar results, if they become the ‘new normal’…could potentially imperil the accreditation of a very large number of law schools.” Indeed, based on data from ABA Standard 509 Information Reports, the top ten ABA schools in Table 1 awarded 38% of their JD degrees in 2016 to minorities, whereas the bottom seven schools in Table 1 awarded 47.4% of their degrees to minorities.

In sum, in contrast to the Committee’s and Klein and Bolus’ optimistic predictions, changing factors not anticipated could result in disastrous consequences for individuals who would have passed under the old system but not under the new, and women will bear the brunt. In addition, if ABA enforces new standards of accreditation for bar pass, we can anticipate that quite a few schools at the bottom will risk loss of their ABA accreditation, with an adverse effect on diversity.

An interesting feature of the Klein and Bolus study was that despite all the models tested, in no case did they report how pass/fail decisions from any shortened test would compare to pass/fail decisions from the current three-day test format. They reported that changing the weighting of the MBE without changing the three-day format would result in 95-96% of the pass/fail decisions being the same. They also told us that shortening the written test to half of its current length would result in only 91-93% of decisions agreeing with those of a three-day test, but that test was their model weighting the current test 50/50. There is no evidence presented to tell us how many of the pass/fail decisions would remain the same if the current test were shortened and weighted 50/50. However, based on the actual data presented to the schools for the July 2016 bar exam, we modeled in Table 2 how the new format will affect each of the 18 ABA schools.

Table 2 goes beyond these basic numbers and provides predictions as to how the new standards will affect bar passage and rank order of the schools, based on the detailed information derived by the Committee and provided to The Recorder. That information includes comparative index scores for each individual essay and PT, as well as a comparative index score for the written portion, the MBE, and the total score. Comparative index scores indicate for each school (in standard deviation units) the extent to which their mean score was above or below the mean for all California first-time takers. As Table 2 illustrates, because the MBE will be more heavily weighted, schools with high MBE scores relative to their written scores will benefit; schools with relatively low MBE scores will suffer. In addition, due to the decreased weight of the PT within the written score, from the current equivalent of four essays to the new equivalent of two or less, schools with a strong PT performance relative to essays and MBE will be hurt by the new format and vice versa. The predictions in Table 2 were derived by applying the new scoring format to the results achieved by each school’s July 2016 candidates in each domain.

Table 2 (2017)

Inspection of Table 2 indicates that UC Davis will drop in rank, while USD will go up a notch. Also losing ground will be Santa Clara and CWSL. These changes in rank are not dramatic because of ceiling effects at the top, in which the school performs so far above the minimum that changes in format have little effect. Similarly, there are floor effects, in which schools at the bottom are so far below the requirements that again the changes in the new format are negligible.

It is the schools in the middle, especially those with a lot of students near the cut, where the differences will have their greatest effect. UC Hastings, for example, is in the middle, is accepting more women and, in addition, has a relatively good PT score but MBE and essay scores below expectations, which again will negatively impact the school’s relative bar pass ranking. Therefore, unless UC Hastings makes some kind of significant change, more of its students will fail than will students from comparable schools.

We are making predictions, but the real proof will be in the results of the July 2017 California bar examination and beyond. In terms of the California bar, while some of the changed circumstances could have been anticipated, there was much that could not have been foreseen all those years ago when the decision was made to change the scoring and format of the California bar exam. Unfortunately, all of those affected, from schools and bar candidates to the profession, will have to deal with the aftermath.

Previous TaxProf Blog coverage:

Legal Education | Permalink


First impression is a minimum LSAT of say 160 for admission would lead for a 75% pass rate. Whether implicit or explicit, setting the bar (sic) at 75%, will require higher LSATs. The only practical alternative would be to teach the bar as part of the curriculum, similar to what accounting programs do for the CPA.

Posted by: Dale Spradling | Jan 31, 2017 8:37:03 AM

Odd to say the least. Prejudice is said to keep talented women and minorities out of professions such as law. And yet, regularly as clockwork, every time a reform is announced, warnings pour forth that it will adversely impact women and minorities.

That makes no sense. More demanding standards should adversely impact white men the most. After all, they're the less talented, less willing to work ones who are playing their "white card" and their "male card" to get into and through law school.
One additional comment about this posting from Scott:

Quote: "What good does it do society to have incompetent lawyers?"

Oh, it does a lot of good if you're on the other side. It means you're more likely to win.

I saw that when I was in a 2003 copyright dispute in Seattle federal court with a Manhattan lawyer representing the Tolkien estate. He (white and male) was an honors graduate of a top law school, but he'd yet to learn lawyering. Disagreeing in our negotiations, I refused to budge. Foolishly, he allowed me to leave him as the one to bring the issue before the judge, making him the one burdening her time.

Not only that, at our conference, I attended in person with him on a speaker phone. The judge, quite reasonablly, wanted to split the differences between us and I agreed. He didn't. I still recall thinking "I can't believe he is arguing with this judge as if she were his lacky." Yet there he was, to her growing frustration.

In the end, I won in the best way possible. Having had enough of his ways, she tossed out his lawsuit "with prejudice."

And yes, there are skills in law and every profession that can't be measures on tests but are often critical for success. That's no grounds for complaint, however. Simply endure the tests, jumping through their hoops, and then go out and practice the profession as it ought to be practiced.

And stop playing the victim. Historical periods often acquire names such as the Victoria Era or the Gilded Age. I suspect our will be called the Whinning Age.

Posted by: Michael W. Perry | Jan 31, 2017 7:57:40 AM

So, why not just lower the pass bar to 50%? That way, CA/ABA can guarantee almost 100% pass (Some one will manage to fail, even at that point.), thereby assuring "diversity, women, and access" to the legal profession for all. After all, it's no different that being a plumber, right? Right?

Posted by: Strelnikov | Jan 31, 2017 7:05:51 AM

Jim W's comment about "toxic clown nose" is spot on.

Posted by: UNT Grad | Jan 31, 2017 5:51:53 AM

I'm a severely disabled person who managed to graduate from a top law school and secure a good job. Lowering the standards would put an "asterisk" next to the name of graduates fro lower-tiered law schools. If they managed to pass the bar with lower standards, they wouldn't be able to repeat the feat under more stringent requirements. If you have the mental acuity, you can pass the bar. Lowering the standards helps no one.

Posted by: UNT Grad | Jan 31, 2017 5:50:24 AM

It will adversely affect their ability to fleece those groups for a quarter million in non-dischargeable student loan debt. People with 140-something LSATs shouldn't be taking classes in law school, shouldn't be practicing law and shouldn't be taking out loans in support thereof.

It has nothing to do with race or gender, it's just a painful fact of life- if you lack the english literacy or mental horsepower necessarily to clear the low hurdle of a 150 LSAT, you need to pick a different career. I would sing a different tune if it were zero cost (financial or opportunity) but law school and practice is extremely expensive in terms of money and missed opportunities. Honestly, law school these days is a gamble if you have 170 LSATs and T14 admission status. It's suicide if you have 145 on the LSAT and a TTT or lower on the resume. Going to Georgetown and ending up as a public defender in some shithole is infuriating and sad, but understandable under the current market conditions. Thomas Jefferson School of Law costs the exact same and is a toxic clown nose on your resume. You are basically banned from JD required employment from that point onwards.

So the question really should be "why would we inflict this cruel fate on already-disadvantaged minority groups?"

Posted by: Jim W | Jan 30, 2017 4:32:12 PM

California's shift to fewer written performance test (PT) questions and more scoring weight for the multiple-choice MBE increases testing reliability, of course, but it correspondingly decreases the level of testing validity. The PT is a more valid measure of lawyering skills, and I was and am disappointed to see California downgrade its weight on the bar exam. This article reinforces my disappointment and concern.

Posted by: Ben Bratman | Jan 30, 2017 3:11:11 PM

Woohoo! We are predicted to beat Hastings.

Posted by: Cheyanna Jaffke | Jan 30, 2017 2:03:26 PM

Ah, the "access schools," the "opportunity schools," the "mission-driven schools." At this point, in this market, why don't you just refer to them as what they have become: criminal fraud scams using students who will never be lawyers as mechanisms by which to obtain federal loan dollars to keep their doors open and their jobs intact. Don't just adversely impact them - CLOSE THEM DOWN!

Posted by: Head Honcho | Jan 30, 2017 1:04:21 PM

Umm...perhaps law schools' declining admissions standards are also "adversely impacting diversity" in the profession: The California pass rates for both Black and Hispanic first-time takes went down significantly (double digits) from the July 2013 to the July 2016 administration, which corresponds precisely to the tanking of admissions standards from 2010 to 2013 (the numbers of takers for both groups remained pretty even, pass rates dropped)! Could the authors kindly speak to that?

Posted by: Anon | Jan 30, 2017 12:23:14 PM

Couple of educational consultants scared of losing their only clients.

Posted by: jm | Jan 30, 2017 11:50:53 AM

First, I'd like to thank Paul for posting editorials on both sides of this very important issue.

Two questions for the authors.
1. How does it help diversity to admit students who can't pass the bar? Also, when students can't pass the bar, they are usually stuck with massive debt, which takes a lifetime to pay off.
2. What good does it do society to have incompetent lawyers? Yes, we could keep lowering the bar passage bar until anyone could become an attorney. But, does this help society? I do believe that most of the purpose behind California's bar passing criteria is to protect the public.

As I have written many times before, the key to creating more diversity in the legal profession is to use better teaching methods. "How to Help Students from Disadvantaged Groups Succeed in Law School," 1 Texas A & M Law Review 83 (2013). FIU has shown that a law school can raise its bar pass rate using the kinds of techniques set forth in my article. (FIU had the highest passage rate for first time takers on the last three Florida bars.)

In sum, if we want diversity in the legal field, we need to better educate our students.

Posted by: Scott Fruehwald | Jan 30, 2017 10:36:04 AM