Measuring success
THIS PAST AUGUST, when the American Federation of Teachers (AFT) announced the results of the first national comparison of test scores among children in charter schools and in regular public schools, it set off a firestorm of debate. Some complained that the study -- which showed that fourth-graders attending charter schools lagged slightly behind their public school counterparts on the National Assessment of Educational Progress -- was biased. Though the data was collected by federal officials, the analysis came from the AFT, a union which has produced research in recent years raising doubts about the expansion of charters. Others questioned why it had taken so long to produce such a national comparison. Still others felt the assessment came too soon, testing schools too young to be properly evaluated. And while charter opponents felt the results confirmed their worst fears about the schools' uneven quality, proponents argued that charters were too diverse to be lumped together for such an assessment.
Perhaps most importantly, the study shone a spotlight on the controversy surrounding the way the research itself is conducted. Caroline Hoxby, a Harvard professor of economics who has written extensively on educational choice, dismissed the AFT's findings as unscientific. "If an undergraduate produced this kind of study in one of my courses, they'd fail," said Hoxby in a telephone interview.
According to Hoxby, the data the AFT presented measured only 3 percent of all charter school students, and assessed them at one point in time, without regard to improvement. In addition, she claims the study compares apples to oranges: As Hoxby and many other researchers point out, most charter schools are in low-income communities, and many parents choose charters because their children are already performing poorly. The AFT's published results pit charter students against all regular public school students; when the AFT disaggregated the data, Hoxby says, they found virtually no differences between black and Hispanic students in charter and public schools.
After the AFT report was released, Hoxby sped up the release of her own study, which covered 99 percent of charter school students, and, she explained, controlled for differences between charter and regular public school populations using a method called "random assignment," which compares charter students with students in the nearest regular public schools they would have otherwise attended. In contrast to the data revealed by the AFT, Hoxby found that charter students were 4 percent more likely to be proficient in reading, and 2 percent more likely to be proficient in math.
Though Hoxby's work itself is not immune to criticism -- The New York Times reported that she has been criticized for excluding some Washington, D.C. charter schools from her study set, and for using a lower measure to determine success in charter schools -- the random assignment technique has gained favor among most current researchers.
The Education Department's own most recent study, released late last month, which found that charter schools were less likely to meet state performance standards than regular public schools, was not based on random assignment. Yet according to Mark Schneider, deputy commissioner of the National Center for Education Research at the Institute for Education Sciences (a research arm of the Department of Education), the DOE now considers this methodology to be the purest way to control for self-selection bias. And in September, the department announced it will undertake 14 studies over the next five years, about half directly related to charter schools. The largest study will use random assignment, and many of the other smaller projects will rely on that data.
Meanwhile, some researchers say the controversies have scared them off work on charter schools entirely. "I've stopped doing this work because research has become so ideologically biased. . .," says Richard Elmore of Harvard's Graduate School of Education. "The issue has become so polarized on both sides, there's not much room for scrutiny."
