By Dan Berrett
Washington
[Updated (5/20/2013, 6:14 a.m.) with comment from Professor Arum.]
Students show substantial gains in learning during college, as measured by a standardized test of critical thinking, according to two studies conducted by the creator of the test.
While perhaps not a direct rebuke to Academically Adrift: Limited Learning on College Campuses, the blockbuster 2011 book that documented what its authors argued was meager learning on campuses, the studies, by the Council for Aid to Education, do offer a sunnier counternarrative.
"It's probably a more nuanced story," said Roger Benjamin, the council's president, in an interview on Friday. The results described in reports on the studies, "Does College Matter? Measuring Critical-Thinking Outcomes Using the CLA" and "Three Principle Questions About Critical-Thinking Tests," were presented in an off-the-record session here at the American Enterprise Institute.
In "Does College Matter?," the council found that, at a typical college, students' scores on the Collegiate Learning Assessment, or CLA, rose 108 points, on a scale that ranges from about 400 to 1600, between freshman and senior years.
The difference was quantified as a 0.78 "effect size," a metric that describes the difference between the average scores of freshmen and seniors divided by the span in scores one would find within each of those groups. An effect size of 0.78 is conventionally thought to represent a "medium" to "large" impact of a program—in this case, an undergraduate education.
"That's a solid effect in social science," said Mr. Benjamin. "College does have significant effects from freshman to graduating-senior levels."

Lost in Translation

In Academically Adrift, Richard Arum, a professor of sociology and education at New York University, and Josipa Roksa, an associate professor of sociology and education at the University of Virginia, documented much smaller growth in learning over a typical undergraduate career, as measured by the test. They found average score increases on the CLA were 86 points. Their effect size was 0.47 over four years, far smaller than the 0.78 the council found.
There were, however, differences between the council's studies and Mr. Arum and Ms. Roksa's that the council said may explain the differences they measured in growth. Mr. Arum and Ms. Roksa analyzed the CLA scores of about 2,300 freshmen at 24 institutions who entered college in 2005 and were retested at the end of their sophomore year, in 2007. Their longitudinal analysis spanned two years of growth among the same students.
In one of the council's studies, by contrast, freshmen's CLA scores in 2011-12 were measured against those of a group of seniors in the same year, at 158 colleges. In the other, the performance of about 10,000 students was analyzed from 2005 to 2012, yielding an effect size of 0.73, very similar to the council's first study.
The council also found distinctions in the performance of students at different types of institutions. Students at baccalaureate colleges demonstrated the highest average growth on the CLA, followed by those at master's-level colleges and universities. Students at doctoral and research universities showed the lowest average growth.
Mr. Arum said this weekend that other studies, using different data sets and tests, had produced similar results. He agreed that much of the discrepancy between their study and the council's could be attributable to the differing methodologies.
Comparing the scores of a sample of freshmen with a different sample of seniors, for instance, could make gains appear larger than the ones that individual students actually experience, he said, because many students drop out before their senior year.
"Roksa and I strongly prefer identifying growth from observing individual gains using longitudinal data," he wrote in an e-mail from a conference in Italy.
Mr. Benjamin was careful to praise the work behind Academically Adrift as "very important."
But, he added, the authors' results "got translated by some people in politics to say, 'College doesn't matter.'"
"I think," he said, "that's an incorrect interpretation."