Chronicle of Higher Education, Is This College Ranking Faulty?:
A dean found something fishy in a magazine’s list of business schools. The editors say he’s off base.
There was a lot that Anjani Jain liked about Bloomberg Businessweek’s ranking of business schools. It was only when the deputy dean at the Yale School of Management dug deeper that it stopped making sense to him.
Like most publications that rate institutions of higher education, this one chose certain categories to evaluate, such as how much money graduates make, and weighted each category based on its importance. But unlike with many other rankings, Businessweek asked students, recent alumni, and recruiters what was important to them, and used their responses to determine how much weight to give to each of the five categories it used to evaluate schools: compensation, learning, networking, entrepreneurship, and diversity. To Jain, this seemed like a good idea.
The dean, who has a background in mathematics, physics, and operations research, began poking around at the numbers, trying to replicate the ranking using the information that was public. He has tinkered with other rankings too; digging into the calculations can be a way for him to procrastinate when he has other work. But when following Businessweek’s published methodology, Jain’s ranking of the business schools did not come out in the same order.
“I noticed that somehow there were numerical anomalies,” he said. The crowdsourced weights that the magazine said it gave to each of its five categories didn’t seem to yield the ranking order the magazine had published.
Perplexed, Jain wrote to Businessweek. An editor wrote back, explaining that the weights had been applied to raw scores, rather than the scores published.
Jain was skeptical. He believed that he had deduced what the weights would have to be in order to recreate the list that Businessweek published. Those weights, he said, were not the same as what the magazine had gotten from its surveys.
The way Jain sees it, one of two things happened. Either the weights were applied to the data before it was normalized — a mathematical process that can be used to ensure that data sets are being compared on the same numerical scale — which would be a statistical error, he said. Or the ranking was calculated accurately, but after the fact some sort of “mysterious manipulation,” as Jain put it, took place.
A Businessweek editor did not respond to The Chronicle’s request for comment. The magazine stands by its ranking. ... The spokesperson said that the magazine’s methodology was vetted by multiple data scientists and that disclosing raw data “would create the possibility that the rankings could be reverse-engineered or gamed by a school for an unfair advantage.” ...
November 24, 2021 in Law School Rankings, Legal Ed Rankings, Legal Education | Permalink