Yale Dean Finds 'Something Fishy' In B-School Rankings, Highlights Need For Greater Transparency

Chronicle of Higher Education, Is This College Ranking Faulty?:

A dean found something fishy in a magazine’s list of business schools. The editors say he’s off base.

There was a lot that Anjani Jain liked about Bloomberg Businessweek’s ranking of business schools. It was only when the deputy dean at the Yale School of Management dug deeper that it stopped making sense to him.

Like most publications that rate institutions of higher education, this one chose certain categories to evaluate, such as how much money graduates make, and weighted each category based on its importance. But unlike with many other rankings, Businessweek asked students, recent alumni, and recruiters what was important to them, and used their responses to determine how much weight to give to each of the five categories it used to evaluate schools: compensation, learning, networking, entrepreneurship, and diversity. To Jain, this seemed like a good idea.

The dean, who has a background in mathematics, physics, and operations research, began poking around at the numbers, trying to replicate the ranking using the information that was public. He has tinkered with other rankings too; digging into the calculations can be a way for him to procrastinate when he has other work. But when following Businessweek’s published methodology, Jain’s ranking of the business schools did not come out in the same order.

“I noticed that somehow there were numerical anomalies,” he said. The crowdsourced weights that the magazine said it gave to each of its five categories didn’t seem to yield the ranking order the magazine had published.

Perplexed, Jain wrote to Businessweek. An editor wrote back, explaining that the weights had been applied to raw scores, rather than the scores published.

Jain was skeptical. He believed that he had deduced what the weights would have to be in order to recreate the list that Businessweek published. Those weights, he said, were not the same as what the magazine had gotten from its surveys.

The way Jain sees it, one of two things happened. Either the weights were applied to the data before it was normalized — a mathematical process that can be used to ensure that data sets are being compared on the same numerical scale — which would be a statistical error, he said. Or the ranking was calculated accurately, but after the fact some sort of “mysterious manipulation,” as Jain put it, took place.

A Businessweek editor did not respond to The Chronicle’s request for comment. The magazine stands by its ranking. ... The spokesperson said that the magazine’s methodology was vetted by multiple data scientists and that disclosing raw data “would create the possibility that the rankings could be reverse-engineered or gamed by a school for an unfair advantage.” ... 

Jain did not back down. Instead, he urged the magazine to correct its ranking. “I understand why this departure from your dug-in position is difficult,” he wrote to an editor. “But I hope that ethical considerations will ultimately prevail.” ...

[Jain] heard from multiple deans, he said. One of them was Hasan Pirkul, dean of the school of management at the University of Texas at Dallas. His institution was ranked 32 on Businessweek’s list, but Jain said that by his calculation, it should have been No. 9. “He’s done a service to all of us,” Pirkul said. His said appreciation of Jain’s calculations had nothing to do with the fact that his school would bounce up the list. “You want transparency. You want people to be able to believe in the results.” ...

Tatiana Melguizo, a professor at the University of Southern California’s Rossier School of Education who has a background in economics and researches quantitative methods of analysis, took a look at Jain’s work at The Chronicle’s request.

“My feeling is that he’s right,” she said. “Depending on when you normalize and when you apply the weights, the final ranking is subject to change.”

Melguizo said she was glad someone was pushing back on the ranking by asking for more transparency. “The issue of replicability is huge,” she said, arguing that people should be able to see how these lists are being created. “We really need to push these rankings agencies to show the code.”

Anjani Jain (Deputy Dean, Yale School of Management), Bloomberg Businessweek’s MBA Ranking Cannot Be Replicated By The Published Data & Methodology (Oct. 8, 2021):

The only way to replicate the published ranking is to apply index weights that are vastly different from the stakeholder-generated weights that BBW claims to have used. I computed these true weights by using a constrained optimization model that minimizes variances from the published ranking.  When applied to BBW’s ‘normalized scores’, the true weights replicate very closely the overall scores published by BBW for the 84 US schools, and thus its published ranking:

Poets & Quants has extensively covered the controversy over the Bloomberg Businessweek MBA rankings:

• Anjani Jain (Deputy Dean, Yale School of Management), Bloomberg Businessweek To Dean Questioning Its Ranking: Fuggedaboutit! (Oct. 22, 2021)
• John A. Byrne (Editor-In-Chief, Poets & Quants), Why The Businessweek MBA Ranking Challenge Matters (Oct. 20, 2021)
• Anjani Jain (Deputy Dean, Yale School of Management), Concerns Now Raised Over Businessweek’s 2018 & 2019 MBA Rankings (Oct. 15, 2021)
• John A. Byrne (Editor-In-Chief, Poets & Quants), Bloomberg Businessweek Stands By Its 2021 MBA Ranking (Oct. 14, 2021)
• Anjani Jain (Deputy Dean, Yale School of Management), Bloomberg Businessweek’s MBA Ranking Cannot Be Replicated By The Published Data & Methodology (Oct. 8, 2021)
• John A. Byrne (Editor-In-Chief, Poets & Quants), Did Businessweek Botch Its Latest MBA Ranking? (Oct. 8, 2021)
• John A. Byrne (Editor-In-Chief, Poets & Quants), Ten Biggest Surprises In Businessweek’s 2021 MBA Ranking (Oct. 8, 2021)
• John A. Byrne (Editor-In-Chief, Poets & Quants), Wharton Plunges To Ninth In New Bloomberg BusinessWeek MBA Ranking (Sept. 15, 2021)