Paul L. Caron

Tuesday, January 7, 2020

Why Does It Take So Damn Long To Grade Law School Finals?

Karen Sloan (, Why Does It Take So Damn Long to Grade Law School Finals?:

Lucky law students will find out their grades in a few weeks, but most will be on pins and needles for a month or more as they await their results. What’s more, they’ll return to campus for the start of the spring semester without knowing how they fared during the previous one.

So what gives? Why does it take so darn long for law professors to grade their final exams? We reached out to some professors to find out the truth. It turns out that—despite what some students may think—most law professors don’t jet off for a tropical vacation the minute finals wrap, leaving their exams behind. There’s a confluence of reasons, they say, for why it may take professors weeks to turn in final grades, ranging from the sheer length of law school finals and the meticulousness with which professors approach grading to the timing of the holidays at the end of the fall semester. They also say that the professors themselves—not teaching assistants—do all the grading. ...

[T]here’s only so much a school can do to motivate tenured professors, said Pepperdine University Caruso School of Law professor Derek Muller. Wrangling grades from adjuncts can also be tough given that they have jobs and other responsibilities, he said, recalling a story of an administrator who threatened to sit in an adjunct’s office until the that person submitted class grades.

Procrastination is a factor for plenty of law professors, Muller said. Some travel for the holidays, and the timing of the Association of American Law School’s annual meeting, which takes place right after New Years, can also cut into grading time. It doesn’t help that grading ranks as one of the least enjoyable aspects of the job. (Muller said pre-finals office hours are actually his least favorite part of his gig, but grading is a close second.)

“Grading is tough,” he said. “You’re reading versions of the same essay—for a first-year class it could be 60, 70, 80 times in the row. You’re trying to your best to differentiate between the Bs and the B+s, because a lot rides on this, especially during the first semester.”

Legal Ed News, Legal Education | Permalink


Perhaps one reason why “It Take(s) So Damn Long to Grade Law School Finals” is that so many law professors are reluctant to use anything other than long and often complex essay questions, often of the IRAC [answer format = Issue, Rule of law, Analysis, Conclusion] type.

This type of exam takes a very long time to grade reliably as compared with an exam which is composed - in whole or in part - of carefully written multiple-choice questions. The latter can be computer scored in minutes.

There is also a substantial body of literature showing that law school exams of the multiple choice type - where they are appropriate given the subject matter - may be as good or better than the long-essay type in that they are more reliable, completely eliminate human bias and inconsistency in grading (a major problem in large classes), and may more accurately measure student ability and mastery of the course.

Since I don’t have time to write about it here in detail or to provide citations, I have taken the liberty of simply setting forth below materials from slides I use - based upon the literature - when explaining to my law students why I use multiple-choice questions, in whole or as a major part of an exam, rather than only IRAC questions.

VALIDITY AND RELIABILITY: The two most important characteristics of an achievement test are its content validity and reliability.

Test reliability depends upon grading consistency and discrimination between students of differing performance levels. Well-designed multiple choice tests - even in Law - are generally more valid and more reliable than essay tests in part because:
(1) they sample course material more broadly;
(2) discrimination between performance levels is easier to determine; and
(3) scoring consistency is virtually guaranteed.

Even if law school exams test "the right stuff" and are valid, the grading process provides daunting reliability hurdles.
(1) Instructors must decide whether to evaluate students completely objectively, if possible, or somewhat holistically, based on the independent judgment of the grader.
(2) Whichever method is chosen, reliability is an essential lynchpin of exams, and without it, a fair and accurate testing of valid objectives will not result.
(3) Thus, reliability is crucial to a fair test.

The national standardized test-givers, such as the bodies who administer the Scholastic Aptitude Test (SAT) and Law School Admissions Test (LSAT) implicitly recognized the problems of reliability inherent in grading essay exams and switched to entirely objective multiple-choice type items.
These bodies recognized that grading essays is invariably a holistic enterprise that would not meet the requirements of standardized testing.
There is little consistency or predictability in grading, particularly with how to weigh writing ability, understanding of the rules, ability to apply the rules, and other important considerations.
In essence, it is no wonder that overwhelming evidence shows that the essay test item is highly unreliable.

A California study's finding that when the same examiner graded an exam answer twice he or she had only a 75% chance of being consistent in deciding whether an answer passed or failed.
This evidence of lack of reliability occurred under a system [BAR EXAMS] that has sophisticated techniques for promoting reliability; techniques which are usually absent in law school grading.

Most law professors have probably experienced anxiety about reliability when grading for a long period, grading under fatigue, grading after reading a particularly good or particularly bad answer, or grading after experiencing anything that changes the assessor's "frame of reference."
The grading of law school examinations differs from grading the bar in ways that make law school exams even less likely to be accurate than bar exams.

Lack of reliability can result from difficulties in grading objectively, guessing on the exam, a short exam (which makes successful guesses on a large portion of the questions more likely), and unclear directions or questions.
Essay exams, by requiring knowledge to be organized and written down, are a comparatively time-consuming way of testing knowledge.

Because the time for taking exams is limited, an essay examination can test only a relatively small sampling of the course.
This means that essay examinations are especially prone to sampling error in the items tested, thereby misrepresenting students' knowledge.
This reduces the reliability of the exam. Unfortunately, there is no fully satisfactory way to eliminate sampling error from essay exams.

Posted by: LawProf John Banzhaf | Jan 8, 2020 10:37:58 AM