Paul L. Caron

Wednesday, July 27, 2016

Are The Google Law Review Rankings 'Worthless'?

Google Scholar (2015)Following up on yesterday's post, 2016 Google Law Review Rankings:

Brian Leiter (Chicago), Annals of "Bullshit" Rankings:

The problem (we've encountered it in philosophy in the past, but now everyone there knows Google  Scholar is worthless for measuring journal impact) is that there is no control for the volume of publishing by each journal, so any journal that publishes more pages and articles per year will do better than a peer journal with the same actual impact that publishes fewer articles and pages.

Rob Anderson (Pepperdine), Google Scholar Releases 2016 Journal Rankings, Controversy Ensues:

Leiter's arguments are (mostly) incorrect. And as my previous posts about Google Scholar were used as part of the ranking, I felt the need to respond. ...

Leiter's comment about "more pages" having an effect on the Google Scholar ranking is (mostly) unfounded. Although at least one study has suggested that more pages in an article lead to more citations, the effect is modest compared to other factors....

One could argue that the journals should be ranked by the "h5-median" (which is easy to do from the provided information) or that the h5-index should be weighted less than the h5-median, but the ranking itself is only marginally sensitive to journal volume output and the number of pages doesn't directly affect the ranking at all.

So although Google Scholar may or may not be "the best" way of ranking law journals, it's certainly not worthless. It provides valuable information alongside other measures of journal quality, and is clearly superior to some other well-known methods of citation ranking. Among other advantages, Google Scholar is not limited to the somewhat provincial Westlaw JLR database.

Law Review Rankings, Legal Education | Permalink


Oh boy. Life is short, so I'm going to keep this short:

1. If a journal publishes more articles, it has more chances of publishing a highly cited article (this is especially true with law reviews, where almost anyting published, e.g., in the Harvard Law Review will be read and cited subsequently). That drives up the h5-index.
2. That latter fact will affect the h5-median.

This ain't rocket science! Everyone in academic philosophy now knows this, everyone in academic law should figure it out too!

Posted by: Brian | Jul 27, 2016 5:31:08 AM

Yes, life is short, but if you take the time to work out the math I think you'll find that I'm right.

A low-cited journal will never get there through volume. The h-index is always less than the maximum-cited article. Assuming that citations per article follow a poisson distribution or perhaps negative binomial with a reasonable variance, that maximum does not grow very fast with the number of articles. Even as the number of articles gets very large, as long as the poisson parameter is reasonably low, the h-index will never get very big. Indeed, citation rankings that merely count the number of citations are far, far more susceptible to the effect you describe, as they can grow to infinity with no articles ever receiving many citations. They are just the sum of a bunch of poisson variables.

Now, if the argument is that everyone will always cite the Harvard Law Review even if many of the articles published are garbage (which may well be true), that's just an argument against the use of citations in general, not Google Scholar in particular.

So it's not rocket science, but it may be closer to rocket science than to "academic philosophy."

Posted by: Robert Anderson | Jul 27, 2016 8:56:11 AM

I never claimed "a low-cited journal will []ever get [highly ranked for impact] through volume." I claimed (correctly) that that "any journal that publishes more pages and articles per year will do better than a peer journal with the same actual impact that publishes fewer articles and pages." That follows quite naturally from the way the metric works for the reasons given already.

Posted by: Brian | Jul 27, 2016 11:36:31 AM

I think what the h-index supporters are trying to say is that it's hard to grow an h-index through volume. Once you get up to an h-index of say 30, you need an article that is cited at least 31 times to move the h-index up to 31. There aren't that many articles that are going to end up being highly cited, and you pick up the low hanging fruit first.

Posted by: h-index | Jul 27, 2016 12:22:57 PM

Professor Leiter, your claim was that the Google Scholar rankings are "worthless." The much narrower claim above was given as support for the claim of "worthlessness." I get the concept of hyperbole, and if what you meant by saying the rankings are "worthless" was that they are "imperfect," then I agree with you. But the rankings you yourself disseminate (even today!) based on raw citation counts are afflicted with those limitations to a much greater degree than is the h-index. That doesn't make them worthless, as I believe the volume and page effects do not significantly affect the overall results. But there really is no disputing that raw citation counts are distored far more by volume effects more than the h-index is, which is part of the reason the h-index was developed-- by someone who could more credibly claim than either of us to actually be a rocket scientist.

Posted by: Robert Anderson | Jul 27, 2016 12:45:24 PM

Prof. Anderson: I take it you now implicitly acknowledge that you misstated my objection, They are worthless precisely because they will mislead about the relative impact of publishing in particular reviews because they are not adjusted for the volume of articles published in each review. Anyone who makes publication decisions based on this nonsense will just be making a mistake.

In the case of faculty citation counts, no one is interested in the average impact of a faculty member's article: if someone were, they would do a different study than the one that Sisk did and from which I've been pulling individualized data. So this is just yet another non-sequitur. As to the false assertion that "there really is no disputing that raw citation counts are distored far more by volume effects," think of it this way: if Cass Sunstein's citation count goes up because he publishes a lot, is that a distortion or a reflection of his enormous influence due (in part only) to his productivity?

Posted by: Brian | Jul 27, 2016 3:03:10 PM

Prof Anderson: Looks like you've just signed up for coal in your stocking for the rest of your life. I reckon you're on the Naughty List now.

Posted by: no soup for you! | Jul 27, 2016 8:00:21 PM

Professor Leiter, my point is exactly that the h-index numbers effectively *are* adjusted for the article volume output. When I say that the volume does not significantly affect the h-index, that is equivalent to saying that the h-index is effectively adjusted for the volume of article output. It's just that it's not adjusted by simply dividing one number by the other. The ranking produced by the h5-index is very highly correlated with the ranking produced by cites per article (impact factor), and the ranking produced by the h5-median is virtually identical to the ranking produced by cites per article. So if you think cites per article (impact factor) is not worthless, then you must think the h-index is not worthless, and you certainly must think the h5-median is not worthless. For these purposes, they are effectively the same thing. There is a vast literature on this that readers can easily find (on Google Scholar), if they are interested.

Posted by: Robert Anderson | Jul 27, 2016 9:01:45 PM

h-index doesn’t make any sense. Although there are so many examples for that, few examples are given here. There is one scientist with h-index of 82 and 37,900 citations even at the beginning of her career. But she hasn’t written a single research paper in her life. Because she got the membership of many CERN groups in the world, she got her name among the thousand names given at the end of many research papers. She is not even a coauthor of any of those papers. Just because CERN members publish papers in so called US or European journals, google scholar search robot finds all those papers. Most of the CERN members got higher h-index and higher number of citations like that.
On the other hand, there are some researchers who have written 70 to 100 papers. But they have a lower h-index below 10 and less number of citations, just because google search robot can’t find even many good journals. Google search robot easily finds US journals, because it thinks that US journals are reputed journals. When I was doing my Ph. D at early nineties, I read several research papers. I found one good paper with original data of ferrite thin films published by some Japanese scientists on a Japanese journal. Few years after that, I found that some US researchers have deposited the same material on the same substrate using same techniques. But the data of US researchers are worse than the data published by Japanese researchers. But US researchers have published their worse data even after one year in US journal of applied Physics. So how can someone say that US journals are the best?

Posted by: P. Sam | Aug 15, 2016 12:21:50 AM