TaxProf Blog op-ed: California’s New Bar Exam Format in Conjunction with ABA’s Proposed Bar Pass Standard Will Adversely Impact Diversity, Women and Access to the Profession, by Dennis P. Saccuzzo & Nancy E. Johnson:
Considerable concern is being expressed concerning the effects on diversity and access to the profession due to proposed changes in ABA accreditation standards and changes in the format and scoring of the bar, such as those in California. According to Lawrence P. Nolan, President of the State Bar of Michigan, for example, ABA’s proposed amendments to the current accreditation standards will “adversely impact efforts to diversify the profession.”
Indeed, 90 law school Deans have asked the Council of the ABA Section on Legal Education and Admissions to the Bar to slow down and think about its proposal to tighten accreditation requirements on bar pass rates. Again, the effects on diversity and access to the profession are among the main concerns. Access not only includes women and minorities, but also non-traditional students such as those who have no family members who ever graduated from college. An important justification for lower tier law schools is that they increase access to the profession.
Now that the California State Bar has decided to change the format of the bar exam, it would seem prudent to ask if the changes will have any effect on access to the profession.
The changes, which will shorten the length of the exam from three days to two, increase the weight of the MBE from its current 35% to 50%, reduce the weight of the written part from its current 65% to 50%, and reduce the weight of the performance test, were made after years of deliberation and study.
At the heart of the decision by the Committee of Bar Examiners (“Committee”) to make the aforementioned changes is a retrospective study by Stephen P. Klein and Roger Bolus of all General Bar Exam (“GBX”) applicants from 2001 through 2010, a study that was commissioned by the California State Bar and published in 2011. According to the Committee, and based on the findings of Klein and Bolus, “administering a two-day examination would yield essentially the same results as those for a three-day examination…” and that it would be “more efficient.”
This optimism from the California State Bar is based on Klein and Bolus’ conclusion that reliability, a measure of internal consistency, will actually be improved by shortening the test from three days to two. All of this seems to fly in the face of one of the most fundamental principles of test theory. It is widely known, and provable with formulas, that the longer the test the higher its reliability.5 It is just as well known that the best way to increase a test’s reliability is to increase its length.6 So how could reducing the test increase its reliability?
In fact, Klein and Bolus’ conclusion was correct. Shortening the test did improve its reliability because of the increased weight given to the MBE. The Committee could even save another million dollars and increase the California bar exam’s reliability still further by going exclusively to a one-day MBE, with no written component. The reliability would be even higher because the 200-item MBE is always more reliable than the 8-item written portion of the exam (under the current format). Why not do this?
The problem with an MBE-only test is that it would disadvantage women and minorities because these groups tend to score more poorly on multiple choice tests than white men and tend to have lower LSAT scores. As it is, the new format is guaranteed to reduce the percentage of women who will pass and increase that of men.
Women were passing at a higher rate than men under the 65/35 written/MBE format because the extra weight given to the written slightly overcompensated for women’s disadvantage on the MBE. Klein and Bolus were aware that the pass rate for women will actually decrease under the new format. They stated, “Assigning equal weights [to the MBE and written portions of the exam] eliminates the differences in passing rates between men and women.” Based on the entire sample between 2001 and 2010, women passed the July exam at an overall rate of 55% and men at 52%. The new formula will eliminate this difference, according to Klein and Bolus. Thus, the first predictable consequence of the new format will be a reduction in the percentage of women who pass the exam and gain entrance into the profession.
We will explain why we believe a perfect storm is brewing that will result in some unanticipated and undesired consequences from the July 2017 California bar exam forward. This is not only because of changing conditions that were not anticipated, but also because of limitations in the Klein and Bolus study. The validity of this study may hinge on the characteristics of the population that was actually tested (2001-2010), whereas the characteristics of those of who will be taking the changed bar exam in July 2017 are different. These highly relevant differences may lead to distortions in the pattern of results that were not anticipated when the decision was made to change the format.
Because the key Klein and Bolus study was reported in 2011 and retrospectively examined performances from 2001 through 2010, these findings apply to that time frame and likely will not generalize so well to 2017 and later. This is because, as Erica Moeser from NCBE has so forcefully stated, there have been many changes in legal education the past several years. As Ms. Moeser noted, LSAT scores have declined, legal education has seen a rise in experiential learning, bar prep courses in law schools are being outsourced to bar prep companies, and the academic support services offered to students in the lower quartile may not be adequate.
We suggest that two factors may be more relevant in explaining the drop in bar pass rates, in conjunction with decreasing LSATs. First, a higher proportion of women study law than was the case in prior years, and second, the exam itself is standardized by equating the written portion to the MBE. That equating becomes less reliable when the population changes from one test to another in terms of the underlying ability measured by the test. The evidence of declining LSAT scores that Ms. Moeser cites is also evidence that the test equating is less reliable because the population has changed in terms of the underlying ability being measured, and that decrease in reliability will have consequences for those who score within the confidence interval around the pass line set by each jurisdiction.
Looking more carefully at the actual data from the most recent California bar exam administration in July 2016, the picture may be even more grim. Table 1 illustrates a variety of relevant features of the latest administration in July 2016. In this table, we give the rank order first-time bar pass results of all of California’s ABA-accredited law schools along with the various mean scores obtained by these schools on the exam. The table also shows the median and 25th percentile LSAT scores for the matriculants in the schools for 2013 and 2016, as well as the percentages of men versus women for these same time periods.
Inspection of Table 1 comparing the male-to-female ratio in matriculants in 2013 to that in 2016 reveals a clear trend of a higher proportion of women entering law school for most of California’s 18 ABA approved law schools. Most schools saw the percentage of women increase by two to four percentage points. Only USD saw a decline in the percentage of women (by one percentage point). LaVerne saw a whopping increase of 13 percentage points. Indeed, in 2016, women represented the majority of entering students for all schools except Stanford, USC, UCLA, and USD. The bottom line is that the percentage of women is increasing. As a corollary, given that more women will fail, if all else remains equal, relative pass rates at most law schools will decrease.
Inspection of Table 1 indicates a second clear trend: LSAT scores are falling to some degree for every school except UCLA and UC Hastings. Falling LSAT scores mean that bar pass rates will fall still further.
According to Susan Case, past Director of Testing for NCBE, the correlation between the MBE and LSAT is 0.57. This means that individuals with high LSAT scores tend to get higher MBE scores and individuals with low LSAT scores tend to get lower MBE scores. The increased weighting of the MBE will mean that the effect of decreasing LSAT scores will be amplified in terms of bar pass rates. Again, even without any change in format of the bar exam in California, falling LSAT scores mean bar pass rates will fall, with the greatest effect being for schools with the greatest decrease in LSAT scores. Increased weighting of the MBE beginning with the July 2017 exam will amplify that effect.
LSAT scores for schools in the top half of the table (the schools with the highest pass rates) have declined less than those for schools in the lower half. Therefore, a second clear consequence of the new format will be a widening gap between top tier schools and the so-called access schools. Bar pass rates will go down at a much faster pace for the schools in the lower third.
As a consequence of falling bar pass rates, many of the so-called access schools, already in trouble with ABA standards of accreditation as it relates to bar pass, will fall hopelessly below ABA criteria. Several ABA schools will simply be unable to meet ABA’s bar pass standards for accreditation. As the 90 Deans wrote in their letter to the ABA, “The California bar results, if they become the ‘new normal’…could potentially imperil the accreditation of a very large number of law schools.” Indeed, based on data from ABA Standard 509 Information Reports, the top ten ABA schools in Table 1 awarded 38% of their JD degrees in 2016 to minorities, whereas the bottom seven schools in Table 1 awarded 47.4% of their degrees to minorities.
In sum, in contrast to the Committee’s and Klein and Bolus’ optimistic predictions, changing factors not anticipated could result in disastrous consequences for individuals who would have passed under the old system but not under the new, and women will bear the brunt. In addition, if ABA enforces new standards of accreditation for bar pass, we can anticipate that quite a few schools at the bottom will risk loss of their ABA accreditation, with an adverse effect on diversity.
An interesting feature of the Klein and Bolus study was that despite all the models tested, in no case did they report how pass/fail decisions from any shortened test would compare to pass/fail decisions from the current three-day test format. They reported that changing the weighting of the MBE without changing the three-day format would result in 95-96% of the pass/fail decisions being the same. They also told us that shortening the written test to half of its current length would result in only 91-93% of decisions agreeing with those of a three-day test, but that test was their model weighting the current test 50/50. There is no evidence presented to tell us how many of the pass/fail decisions would remain the same if the current test were shortened and weighted 50/50. However, based on the actual data presented to the schools for the July 2016 bar exam, we modeled in Table 2 how the new format will affect each of the 18 ABA schools.
Table 2 goes beyond these basic numbers and provides predictions as to how the new standards will affect bar passage and rank order of the schools, based on the detailed information derived by the Committee and provided to The Recorder. That information includes comparative index scores for each individual essay and PT, as well as a comparative index score for the written portion, the MBE, and the total score. Comparative index scores indicate for each school (in standard deviation units) the extent to which their mean score was above or below the mean for all California first-time takers. As Table 2 illustrates, because the MBE will be more heavily weighted, schools with high MBE scores relative to their written scores will benefit; schools with relatively low MBE scores will suffer. In addition, due to the decreased weight of the PT within the written score, from the current equivalent of four essays to the new equivalent of two or less, schools with a strong PT performance relative to essays and MBE will be hurt by the new format and vice versa. The predictions in Table 2 were derived by applying the new scoring format to the results achieved by each school’s July 2016 candidates in each domain.
Inspection of Table 2 indicates that UC Davis will drop in rank, while USD will go up a notch. Also losing ground will be Santa Clara and CWSL. These changes in rank are not dramatic because of ceiling effects at the top, in which the school performs so far above the minimum that changes in format have little effect. Similarly, there are floor effects, in which schools at the bottom are so far below the requirements that again the changes in the new format are negligible.
It is the schools in the middle, especially those with a lot of students near the cut, where the differences will have their greatest effect. UC Hastings, for example, is in the middle, is accepting more women and, in addition, has a relatively good PT score but MBE and essay scores below expectations, which again will negatively impact the school’s relative bar pass ranking. Therefore, unless UC Hastings makes some kind of significant change, more of its students will fail than will students from comparable schools.
We are making predictions, but the real proof will be in the results of the July 2017 California bar examination and beyond. In terms of the California bar, while some of the changed circumstances could have been anticipated, there was much that could not have been foreseen all those years ago when the decision was made to change the scoring and format of the California bar exam. Unfortunately, all of those affected, from schools and bar candidates to the profession, will have to deal with the aftermath.
Previous TaxProf Blog coverage:
- July 2016 California Bar Exam (Nov. 22, 2016)
- More On The California Bar Exam (Dec. 5, 2016)
- Who Is To Blame For UC-Hastings 'Horrific' 51% Bar Pass Rate? (Dec. 7, 2016)
- July 2016 California Bar Exam Results: Nine Law Schools (Including UC-Hastings) Are At Risk Of Failing ABA's Proposed New Bar Passage Accreditation Standard (Dec. 13, 2016)
- Anderson: Deans, Denial, And The California Bar Exam (Dec. 20, 2016)
- Despite 'Horrific' 51% Bar Passage Rate, New UC-Hastings Dean Says School Is Poised To 'Catapult Into National Preeminence' Using NYU As A Model (Dec. 21, 2016)
- California Law School Bar Pass Rates Recalculated For New York: Stanford, UCB, USC > NYU; UCI, UCLA > Columbia; Chapman, Loyola, McGeorge, Pepperdine, Santa Clara, UCD, USD > Fordham (Dec. 22, 2016)
- More On The July 2016 California Bar Exam (Dec. 26, 2016)