Charter schools grade highest

Hub's traditional, pilot systems lag; Findings surprise state's specialists

Aliyah Marsh and Derek Welcome prepared for their eighth-grade reading class at the Roxbury Preparatory Charter School. (Globe Staff Photo / Michele McDonald)

A new study indicates that Boston charter schools significantly outperform the city's traditional schools, but raises new questions about the city's experimental pilot schools, which in many cases posted "ambiguous" or "disconcerting" results.

The study, being released today at a Boston Foundation forum, examined state standardized test scores for students of similar backgrounds at the three kinds of schools over a four-year period. In the most stark example, charters - independent public schools dedicated to innovative teaching - excelled significantly in middle school math. However, pilots, which have similar goals but are run by the School Department, performed at slightly lower rates than traditional schools, according to the study.

The findings could present a setback for Governor Deval Patrick's education overhaul, which seeks to emulate pilot schools around the state while resisting calls for more charter schools.

The divergent performance among charter and pilot school students surprised state and local education leaders because the schools share a similar history and philosophy.

Charter schools were created as part of the 1993 Education Reform Act, as a way to develop new teaching strategies that could eventually be transferred to public schools. The approximately 60 schools operate under looser state regulations than traditional schools, have mostly nonunion teachers, and are run by independent boards that report directly to the state. They have been particularly popular in urban school districts among parents and students frustrated with traditional schools.

Boston created the pilot school concept a year later, with the idea of embracing the innovative teaching methods while avoiding some of the controversies. Unlike charters, the pilot schools are still run by the School Department, which means that the district still receives state funding for each student who attends them. Many school districts across the state have complained that charters drain much-needed resources from traditional schools because the state funding follows the students to the new school.

The Boston's 18 pilot schools also have teachers' unions, although the provisions are scaled back to allow for experimentation with longer school days and other changes that unions have traditionally resisted.

The Boston Foundation - which has been a champion of pilot schools and provided money to help some get started - believes the schools are still a worthwhile investment.

"I would've loved it if pilots matched the results charters are producing, but I still believe very strongly pilots are superior to traditional schools," said Paul Grogan, the foundation's president. He noted that another study a year ago found that pilot schools had higher graduation and college attendance rates than traditional schools, among other positive indicators.

Researchers at Harvard University and MIT, who conducted the study, said they do not know why a performance gap emerged, noting that the study was designed to merely uncover whether charter and pilot schools increased student achievement. They have recommended that the state appoint a task force to identify and analyze what's working best at charter schools and how to transfer those strategies to traditional schools.

Paul Reville, the state's education secretary, yesterday called the results for pilot schools "disappointing," but reaffirmed the governor's commitment to developing the so-called readiness schools, which would in part draw on the pilot school model.

"We have to examine the results closely and ultimately be part of an effort to take the study to the next level," Reville said.

The study stands apart from volumes of other research produced over the more than decadelong debate over charter schools by including a section that compared the performance of students at the charter and pilot schools to students who entered the lottery to attend those schools but did not get in.

This was an attempt to dispel the perception that charter schools perform well in comparative studies because they generally attract more academically-motivated students and parents - not necessarily because they have better teaching methods.

Only charter and pilot schools that had more students apply than available seats - and therefore had to conduct a lottery - were included in that part of the study.

Charter students in middle and high schools showed consistent gains on the math and English exams. The results of pilot schools were less clear. Middle school pilots performed slightly below students in regular middle schools in math and about the same in English. High school pilot performance was a little better, but researchers still deemed those results ambiguous.

"The thing that was most surprising given other studies that have been done was the large magnitude of the charter effect in middle school math," said Thomas Kane, a Harvard education and economics professor who lead the research along with Joshua Angrist, an MIT economics professor.

While Reville said the study offered some insight into the role of student motivation, he said the study did not address the effect of having such students grouped together in a pilot or charter school. The students who did not gain entry, by contrast, were dispersed to schools throughout Boston where students may not have similar aspirations.

Also, other education observers questioned whether focusing on schools that held lotteries slanted findings toward the more academically successful charter and pilot schools that families would be clamoring for, while the faltering ones would not likely have had to conduct a lottery.

The lackluster performance of Boston middle school pilots highlights what can happen when the pilot schools are imposed by district edict, rather than growing out of grass-roots proposals from the teachers and administrators, said Dan French, executive director for the Center for Collaborative Education, which works with Boston on the pilots. He noted that the school system's previous administration created two large middle school pilots - Orchard Gardens and Lilla Frederick - when two new school buildings came on line. And these schools have struggled.

"It's a promising model but as we have grown to scale, there are pilots that are of some concern," French said. "You need to pay attention to accountability."

Carol R. Johnson, superintendent of Boston schools, intends to continue creating more pilot schools, which she said have been popular among parents. She is also interested in the next step of the study, which will examine best practices from all the schools that can be shared across the district.

But the study will certainly add ammunition to a movement, supported by the Foundation and others, to increase the number of charter schools beyond the state-imposed cap.

James Vaznis can be reached at jvaznis@globe.com.