November 11, 2008
Building a Better LSAT: Predicting Lawyer, Not Law Student, Success
Following up on last week's post, UC-Berkeley: Testing for Empathy as an LSAT Alternative?: Today's Inside Higher Ed: Building a Better Admissions Test, by Scott Jaschik:
Most standardized admissions tests — from the SAT and ACT to those used for admission to graduate and professional schools, such as the Law School Admission Test — promise one thing: to predict academic success in the first year enrolled. Most standardized tests also face growing skepticism because white and Asian students tend to outperform, on average, black and Latino students.
What if a standardized test managed to predict much more than first-year success? And what if there existed the possibility of having standardized tests that didn’t have ethnic or racial gaps, but better predicted long-term success?
Researchers at the University of California at Berkeley have been engaged in a long-term research project to produce such tests for use in its law school — and they think they have a model that does those things exactly: predicts success as a lawyer (not just as a first-year student) and finds success across demographic groups. Given that law schools exist to produce lawyers, not first-year law students, Berkeley officials think their findings are significant and they are now releasing them for public view and — they hope — for testing on a national scale.
While Berkeley is at this time calling for more research and not an abandonment of traditional admissions processes, the report it has released suggests that the time is ripe for a major reconsideration of how law students are admitted.
“Rising numbers of law school applicants, concern over litigation and preoccupation with school rankings have pushed overemphasis on the LSAT to the breaking point,” says the Berkeley report. “Definitions of ‘merit’ and ‘qualification’ have become too narrow and static; they hamper legal education’s goal of producing diverse, talented and balanced generations of law graduates who will serve the many mandates and constituencies of the legal profession. New predictors combined with existing LSAT measures could extend from prediction of law school success to prediction of professional effectiveness in law school admissions.”
The Berkeley experiment involved a large-scale effort to identify the qualities that make good lawyers, and then to compare the correlation between scores on the Law School Admission Test as well as alternative measures that could be used for admission to later success as a lawyer. The alternative measures are a range of biographical, personality and “situational judgment” tests. What the researchers found was that while the LSAT correlates with first-year grades, as promised, it doesn’t correlate with later success as a lawyer. Combinations of the other tests do correlate with success as a lawyer, as defined by having various qualities of success measured in the study, and without racial and ethnic gaps. ...
“To admit primarily on the basis of LSAT test scores and grades to a professional field that has great importance to our society, seemed short-sighted,” the report on the new research says. “Lawyering requires a variety of talents and skills beyond those represented in these important, but limited, measures. Over subsequent years, the emphasis on the LSAT plus grades has actually grown with the advent of such highly publicized rankings as the U.S. News & World Report for whom entering class median LSAT scores are a key factor. These trends were playing out against a desire on the part of law schools to train a diverse population of legal practitioners, a goal that overemphasis on purely cognitive measures suppressed.”
TrackBack URL for this entry:
Listed below are links to weblogs that reference Building a Better LSAT: Predicting Lawyer, Not Law Student, Success:
I reviewed the paper, and I have a suspicion that using alternative tests (emotional tests, situational judgment tests, etc.) won't end up working any better than the LSAT. In order to be useful for admissions, the tests need to be standardized (i.e., multiple choice). The alternative tests probably would work to predict success if one could require test takers to answer the questions honestly. But it won't take long for test-prep companies to figure out which "profile" law schools are looking for and to coach applicants accordingly. The people with the highest cognitive abilities (those who score highest on the LSAT now) will probably be the ones who are most sucessful at ferreting out the "right" answer on personality and situational-reasoning tests. I am certain I could flip each component of my score on the Meyers-Briggs inventory without any preparation at all. In short, this is interesting in theory, but probably won't work in practice to achieve either goal of the authors (increased diversity and better lawyers).
Posted by: jm | Nov 11, 2008 6:15:38 PM
Why is it that the research tries to "[find] success across demographic groups"? That is no way to begin the scientific method, and not a serious way to identify which students will ultimately become the best lawyers. Yet it looks like this is the true ultimate goal, after all, as the author noted, the skepticism with the current system arises out of the fact that particular races do better than other races. Are there complaints that the LSAT does not predict lawyering skills, based on something other than the fact of the racial scores gap?
Wouldn't it be uncanny if every race exhibited the exact same score distribution? One way to do that: a test that doesn't identify anything at all. I imagine any test that achieves the goal of racial equality of results will be utterly arbitrary.
Posted by: John Galt | Nov 11, 2008 8:19:12 PM
Maybe instead of trying to fix a perfectly fine standardized test that does a very good job of what it is supposed to do (which is measure aptitude for logical reasoning, by the way), we should worry about fixing our broken law schools that do less than nothing to prepare students for the practice of law. The focus should not be on making the LSAT useful, it should be on making law school useful.
Posted by: Anon | Nov 11, 2008 8:42:18 PM
Heaven forbid that we evaluate law school applicants with strong "cognitive measures."
Posted by: YF | Nov 11, 2008 9:20:10 PM
Simple Fix: Keep the LSAT as-is, but find a way to schedule it only on Thanksgiving Day, Christmas Day, Yom Kippur, or, preferably, during the birth of an examinee's first child. The addition of a proofreading component is also recommended.
Posted by: wingsandvodka | Nov 12, 2008 11:40:20 AM
So if the test measures what makes a good lawyer--why aren't the law schools teaching what the test measures?
Posted by: John | Nov 12, 2008 1:28:17 PM
So if the test measures what makes a good lawyer--why aren't the law schools teaching what the test measures?
Posted by: John | Nov 12, 2008 1:29:17 PM
Mindless ivory tower claptrap like this is nothing more or less than the subterfuge du jour to effectively reinstate race-based admissions policies.
My clients (middle class, blue collar workers, for the most part) don't give a rat's ass whether lawyers as a group are diverse or balanced and they sure as hell are not paying me to serve "many mandates and constituencies." They have work that needs to be done and they pay me to do it. My clients are the only "constituency" that matters. Simple. Except that every public activity is now an excuse to indulge in leftist social engineering and screw you to the clients, we have loftier goals you groundlings wouldn't understand...
Snobbish elitist self-absorbed solipsistic... augh! It makes me sick.
Posted by: jagcap | Nov 12, 2008 1:43:24 PM
This doesn't seem like a call to change the LSAT but the law school curricula. It seems like a good idea to use a test that screens out those who would not be able to compete in the first year of law school. If there is a mismatch between the first year and other years or between law graduates and those who pass the bar or between those who pass the bar and lawyers, those should be the gaps addressed. The article admits that the LSAT does a good job of predicting first year success. The fact that it doesn't predict first year success shows that the first year is not preparing students for real lawyering.
Posted by: Zach | Nov 12, 2008 2:25:55 PM
Regarding the comment on broken law schools, law schools are a perfect example of the problems with a monopoly (a cartel in this case).
The cartel sets the standards and, with trivial exceptions, the cartel members all line up and salute. The profile for law school professors is people who scored well under the current system and spent a trivial amount of time outside of the law school world (does clerking for a federal judge really teach anything about being a real lawyer? I think not.).
The only thing that the standard law school model does a decent job of producing is associates for very large law firms, a tiny fraction of the universe of lawyers.
I remember what a breathtaking change my first CLE program was from the law school instructional experience. The speakers, practicing lawyers all, were clear, concise, organized and focused on conveying useful information for their audience. (I was an honors graduate of law school, law review, etc., so this isn't sour grapes from someone that did poorly under the Ancien Régime.)
Rather than focus on the LSAT, I think the solution to the real problem with law schools lies in making a fundamental change within the culture that permeates law schools, but that would be hard, so Berkeley wants to command that the Educational Testing Service or somebody, concoct some new test to salve white guilt.
Posted by: Publius | Nov 12, 2008 2:34:14 PM
You know how many people I went to law school with that got high LSAT scores and high grades that I wouldn't trust to defend me against a traffic ticket? All of these people have approximately the same problem: they can't respond to failure, they can't deal with it when things do not go exactly according to the plan, or and wilt under any type of pressure.
No standardized test can evaluate that.
Posted by: Brian | Nov 12, 2008 3:22:28 PM
I'm all for broadening the measures used for admission so long as objective measures are used. What's truly awful are admission criteria that are subjective and hence subject to personal whim, prejudice, and politics.
Zach - I don't know where you live or what CLE programs you've attended, but in my experience (and I've been practicing law since 1981) they are all awful, mind-numbing compendiums of war stories and useless detail, devoid of intellectual substance. That's true even for the ones I've taught. Give me a good Socratic dialogue with a law professor any day.
Posted by: Connecticut Lawyer | Nov 12, 2008 7:36:00 PM
If you take a Kaplan course, and you'll score 10 points better on the LSAT. Did the Kaplan course teach you how to do 10 points better in law school?
Posted by: daytripper | Nov 12, 2008 11:29:30 PM