Paul L. Caron

Wednesday, September 20, 2017

Anderson: Are LSAT/GPA A Leading Indicator, And Peer Reputation A Lagging Indicator, Of The U.S. News Overall Rankings?

US NewsFollowing up on my recent post, Law School Rankings By Student Quality (LSAT And UGPA):  Rob Anderson (Pepperdine):  Predicting the Future of US News Law School Rankings With Revealed Preference Rankings?:

The approach used by US News includes a variety of factors with varying weights but among the most important are factors based on surveys of academics and of lawyers and judges. Specifically, the US News ranking methodology is based 25% on "peer assessment score" (academics) and 15% on "assessment score by lawyers and judges." The both categories are weighted heavier than LSAT scores (.125) and GPA (.10), which are the raw material for the Ryan and Frye ranking.

As might be expected, the Ryan and Frye rankings correlate strongly with US News rankings, but there are some significant outliers. ... [I] thought it might be interesting to examine the potential causes of divergence between the Ryan-Frye approach and US News by comparing the US News survey-based rankings between 1993 (the year of the first full ranking of law schools) and 2018 (the most recent ranking).

The peer ranking is the largest single component of US News and is measured somewhat comparably across the years so I will focus on that component of US News. The chart below shows a plot of the 1993 peer rankings (then called "academic" rankings) and those for 2018. Because higher ranked schools have lower ranking numbers, the highest ranked schools are in the lower left and the lowest ranked schools in the upper right. Schools above the line have improved in their rankings between 1993 and 2018. Schools below the line have lower rankings in 2018 than in 1993. 

Anderson 3

The correlation between the 1993 peer ranks and the 2018 peer ranks is .93, which is evidence of incredible stability over time. As a result, the 1993 rank can predict with a high degree of accuracy the 2018 rank, especially for the higher-ranked schools (the lower left). However, there are some notable outliers, which I've noted with text in the figure. It is interesting to note that among the largest gainers are three that changed names by affiliating with an existing university (Michigan State, New Hampshire, and Quinnipiac). The remainder of the schools with large jumps in peer rankings (Alabama, CUNY, Georgia State, and Pepperdine) have other explanations. My institution (Pepperdine) and Alabama have made major pushes toward emphasis on research productivity, which may explain the changes in their scores. ...

[I]t may be that law students' choices are a "leading" indicator and the peer rankings are a "lagging" indicator of law school overall rankings. Still, the peer rankings appear to lead the lawyer/judge rankings, or at least not lag them. Although there are other potential interpretations of these relationships, identification of leading and lagging predictors could be significant. If this is true, then the revealed preference approach to law school rankings may predict the future US News overall rankings. Whether this suggests that increasing LSATs will cause increases in the peer and lawyers/judges rankings (with lags) is another question, but one whose answer could pay off for rankings-focused law schools.

Law School Rankings, Legal Education | Permalink


One wonders if any (many) law schools may be trying to *help* their peer reputation metrics in the vein that an employee at Clemson copped to some years ago:

"A presentation by Catherine Watt, the former institutional researcher and now a staff member at Clemson University, laid bare in a way that is usually left to the imagination the steps that Clemson has (rather brazenly) taken since 2001 to move from 38th to 22nd in U.S. News's ranking of public research universities.... [long list of the usual USNWR manipulations - capping upper-level class enrollment at 20, counting faculty benefit packages as part of their salaries so as to make it appear higher than is the case, etc) ... And to actual gasps from some members of the audience, Watt said that Clemson officials, in filling out the reputational survey form for presidents, rate "all programs other than Clemson below average," to make the university look better. "And I'm confident my president is not the only one who does that," Watt said."

Posted by: Unemployed Northeastern | Sep 20, 2017 12:15:00 PM