Beyond The Cut Score: Piercing The Veil Of The California Bar Exam's Validity

Following up on my previous post, California’s New Bar Exam Format And ABA’s Proposed 75% Bar Passage Requirement Will Adversely Impact Diversity, Women, And Access To The Legal Profession:

The Recorder op-ed: Beyond the Cut Score: Piercing the Veil of the California Bar Exam's Validity, by Dennis Saccuzzo & Nancy Johnson:

In July 2016, California's bar pass rate fell to 43.1 percent, a 32-year low for July bar exams. In December, the chief justice of the California Supreme Court, Tani Cantil-Sakauye, expressed her concern about California's falling bar pass rate, but said that she wanted to wait and see what happens to scores when the new, shortened format is administered in July 2017, with its scoring and format changes.

More recently, Elizabeth Rindskopf Parker, the executive director of the California State Bar, stated she had no good answer for California's unusually high cut score for the MBE. She had been testifying before the California Assembly Judiciary Committee following a Feb. 1, 2017, letter from nearly all of California's ABA law school deans to the California Supreme Court requesting that the cut score be lowered. As disconcerting as is this inability to justify the cut score, here we identify even deeper concerns about the validity of the test itself and its ability to make reliable pass/fail decisions, revealed in the bar's own statistical studies to justify the changes being implemented in July.

As we previously noted on the TaxProf Blog, the justification for the changes to the California bar is heavily based on conclusions by Klein and Bolus, statistical consultants for the California Bar.

The message presented by Klein and Bolus, echoed by the California State Bar, is that the change in format and scoring, to be implemented beginning with the July 2017 exam, will lead to essentially the same results as under the older format. This conclusion, however, may be misleading, if not incorrect. What this study actually does is show us, for the first time, what would happen to 10 years of bar candidates if the cut score remained the same but the format and/or scoring of the exams they took were changed; for how many would that pass/fail decision be different?

The study by Klein and Bolus actually demonstrates that there will be large differences in the pass/fail decisions for individuals under the new format. ...

A closer look at the analyses conducted by the bar's professional statisticians, in conjunction with the California State Bar's inability to explain the rationale for its cut score combined with the likely effects of the new format raise some serious questions about the validity, integrity, and fairness of the bar exam. As we have seen, many will pass in July 2017 who would not have passed under the old system, and vice versa. In exchange, the bar purports to save just over $1.1 million.

One might wonder how this is protecting the public and being fair to the hardworking bar candidates who would have passed under the old format. The validity problems of the California Bar Exam run much deeper than a cut score that could not be justified as having some validity in itself. As to the cut score, the Standards and Guides for Educational and Psychological Testing mandate that credentialing tests "may be strict but not so stringent as to unduly restrain the right of qualified individuals to offer their services to the public" (AERA, APA, & NCME, 2014, page 175).

Previous TaxProf Blog coverage:

• July 2016 California Bar Exam (Nov. 22, 2016)
• More On The California Bar Exam (Dec. 5, 2016)
• Who Is To Blame For UC-Hastings 'Horrific' 51% Bar Pass Rate? (Dec. 7, 2016)
• July 2016 California Bar Exam Results: Nine Law Schools (Including UC-Hastings) Are At Risk Of Failing ABA's Proposed New Bar Passage Accreditation Standard (Dec. 13, 2016)
• Anderson: Deans, Denial, And The California Bar Exam (Dec. 20, 2016)
• Despite 'Horrific' 51% Bar Passage Rate, New UC-Hastings Dean Says School Is Poised To 'Catapult Into National Preeminence' Using NYU As A Model (Dec. 21, 2016)
• California Law School Bar Pass Rates Recalculated For New York: Stanford, UCB, USC > NYU; UCI, UCLA > Columbia; Chapman, Loyola, McGeorge, Pepperdine, Santa Clara, UCD, USD > Fordham (Dec. 22, 2016)
• More On The July 2016 California Bar Exam (Dec. 26, 2016)
• California’s New Bar Exam Format And ABA’s Proposed 75% Bar Passage Requirement Will Adversely Impact Diversity, Women, And Access To The Legal Profession (Jan. 30, 2017)
• Deans Of 20 Of California's 21 ABA-Accredited Law Schools Ask State Supreme Court To Lower Bar Exam Pass Score (Feb. 2, 2017)
• ABA 'Overwhelmingly' Rejects 75% Bar Passage Requirement (Feb. 7, 2017)
• California Law Deans Take Bar Exam Complaints To Lawmakers; State Bar Director Admits There Is 'No Good Answer' For High MBE Pass Score (Feb. 15, 2017)