Do The ‘U.S. News’ Rankings Rely On Dubious Data?

Chronicle of Higher Education, Do the ‘U.S. News’ Rankings Rely on Dubious Data?:

Researchers who submit to the publication say survey answers are subject to errors, ambiguity, and pressure to look good. ...

[There is a] complicated relationship between colleges’ data submitters and U.S. News, the best-known college-ranking system in the United States. Many resent the time and oxygen the ranking takes up. After The Chronicle asked to interview her, Christine M. Keller, executive director of the Association for Institutional Research, conducted an informal poll of the group’s members about their views of U.S. News. One major theme: Answering the magazine’s survey requires too many resources, a situation they see as taking away from internal data projects that contribute more to student success than rankings do. Yet they know that responding to the survey is an important part of their jobs, and often campus leaders are paying close attention.

Recently that relationship has undergone renewed scrutiny, as rankings-data controversies have piled up. First, a former dean of Temple University’s business school was sentenced to 14 months in federal prison for leading an effort to inflate statistics his school had sent to U.S. News. Then a Columbia University mathematics professor publicized his belief that his institution is sending inaccurate data to Morse and his colleagues, a contention Columbia has denied. Finally, the University of Southern California pulled out of the rankings for its graduate program in education because it discovered it had submitted wrong data for at least five years.

The headline-grabbers are the latest in a decades-long history of scandals about colleges gaming U.S. News and unintentionally sending inaccurate data to it. Every few years, it seems, another incident comes to light. Criticisms of the rankings have also been longstanding, but there’s been some fresh attention since the popular journalist Malcolm Gladwell covered them on his podcast last year.

Willis Jones, an associate professor at the University of Miami who studies higher-education leadership, has noticed more of a social-justice bent to rankings criticisms lately. ...

As the college staffers typically responsible for gathering and submitting the high-stakes data, institutional researchers are on the front lines of this much-scrutinized process. And while outright lying may be relatively rare, there’s always human error, plus ample room for interpretation in the U.S. News questions. That ambiguity can create incentives to finesse the data in a way that makes one’s institution score better in the magazine’s rubric. ...

The Chronicle spoke with institutional researchers at two large public universities and two small, private liberal-arts colleges, to get a sense of what it’s like to make those high-stakes decisions. Most said that while they personally had not experienced pressure to report positive data, they had heard of others who had. That kind of within-the-rules-but-not-the-spirit massaging is more common than outright fraud, they believe.