Monday, September 26, 2011
U.S. News & World Report is not going to change the way it composes its annual rankings of colleges and universities in a significant way any time soon, regardless of how admissions counselors and higher education admissions officers feel about them.
That's what the head of the rankings said here Friday at a session of the annual meeting of the National Association for College Admissions Counseling after a special committee from the association presented its final report examining the rankings. The report, which took about two years to compile, is largely based on a survey of NACAC members the committee conducted in 2010 and repeats much of what was established in a draft of the report released earlier this year. Officials from U.S. News worked closely with the committee throughout the process, a relationship that has bothered some rank-and-file members of the association. (The concern was that involving the rankings officials in the discussion might co-opt the process, although the final results suggest that the association did not hold back on its criticisms of the rankings.)
The committee’s report notes that while U.S. News’ rankings are undoubtedly influential, college admissions officials and high school counselors question their value to students applying to college. The report takes issue with the relevance of several components of the rankings -- particularly the reputation survey (in which presidents evaluate similar institutions) and emphasis on incoming students’ class rank and standardized test scores -- and the “distorting effects” they can have on both students and higher education institutions. It calls for U.S. News to remove test scores and class ranks as a component of the rankings, and to reduce the weight of the peer-assessment survey that currently counts for 22.5 percent of an institution’s score.
The main thrust of the committee's argument against the use of standardized test scores is that they are not a complete measure of student quality, and that the continued emphasis on using them as a key metric of comparison discounts other aspects of admissions. ...
Robert Morse, who oversees the rankings for U.S. News, said at the presentation that he and others at U.S. News appreciated the dialogue with the committee and NACAC and the responses presented in the survey. He said officials from U.S. News would work with the association to educate members on the rankings -- another of the report’s recommendations -- but it would not make substantial deviations from its formula because of the report.
As long as colleges and universities continue to weight test scores and class ranking as a crucial component of admissions criteria, Morse said, it is hypocritical for institutions to ask U.S. News not to do the same. For the 2011 rankings, test scores accounted for 7.5 percent of the overall ranking for all institution types, which Morse asserts is probably less than the weight most colleges give the scores in admissions. Morse said U.S. News would consider lessening the importance of students’ test scores and class rankings if institutions discounted such measures. “Schools are using it to build their class,” he said at Friday’s session. “We believe that makes it a credible metric.” ...
The other area of methodological concern addressed in the report was the reputation surveys that U.S. News sends to college and university presidents. The survey found that most admissions officials felt the peer surveys could not be an accurate reflection of institutions’ quality. “The peer assessments are highly subjective and may be disproportionately influenced by social factors that do not measure institutional quality,” the report states. Questions about the validity of the peer survey came to the forefront of the discussion at a meeting of institutional researchers in 2009 when a former institutional researcher from Clemson University spelled out how her institution had rated peer institutions poorly while rating itself highly.
There was some concern among admissions officials in the audience at Friday’s presentation that the rankings have become a self-fulfilling prophecy. Members of the audience said college and university presidents could rarely understand the quality of undergraduate experience at other institutions, and therefore would have to rely on third-party information, such as the U.S. News rankings, to determine how they evaluate programs.
Morse said he has not seen social science evidence that proves that such concerns are valid. If such evidence turned up, he said, U.S. News might reevaluate how it approaches and weighs the surveys.