Better effort promised on checking SAT scores

After sending incorrect scores to 4,000 students, those who give and process the Scholastic Assessment Tests are pledging to do more to monitor their automated scoring, even while insisting that they already do almost everything possible to catch errors.

State education officials and the company that administers the annual MCAS exams, meanwhile, say they, too, have many checks in place to prevent errors. They even send scores to school principals to look for anything amiss before results are released.

Some testing industry observers, however, say the testmakers are not doing enough, and they view the SAT scoring problem as a sign that standardized testing is more error-prone than the public realizes. Testing, they say, requires more self-monitoring, higher spending, and even government oversight, particularly since federal requirements have led to a vast expansion in standardized testing.

''Errors are proliferating, but unless it's in the press, no one knows," said Kathleen Rhoades, a researcher at Boston College's Center for the Study of Testing, Evaluation, and Educational Policy.

The tests ''are a product, like a piece of bread or a toaster, but you can't go to Consumer Reports," she said. ''These tests make much more of a difference in a person's life than a toaster, and no one is checking them."

The College Board, the nonprofit association that administers the SATs, acknowledged another mistake this week: After having said they had rescored every October test, College Board officials said they discovered 1,600 exams that had not yet been checked for scanning errors. The exams had been previously set aside for other reasons and were forgotten in the midst of the scoring controversy. A spokeswoman said the exams would be rescored by the end of next week.

Pearson Educational Measurement, the company contracted by the College Board to scan SAT answer sheets into electronic files, said that the problem with the October 2005 tests was partly caused by high moisture content in the test sheets. Oct. 8, the test date, was very rainy in the Northeast, and the students with scores lower than they should have been were concentrated in this area. Massachusetts, behind New York and New Jersey, had the third highest number of bogus scores, with 540 students affected.

The moisture caused the paper to expand and literally moved the bubbles where students marked their answers in Number 2 pencil. In addition, some students had filled in the answer bubbles lightly or not completely. Those two problems led the scanning machines to miss some correct answers, the company said. Pearson said the problem had never before been identified.

In response, Pearson is designing software to examine paper for signs of expansion and will also allow more time for answer sheets to acclimate -- in other words, to dry out -- in its facility in Austin, Texas, said spokesman David Hakensen.

Hakensen said the company already rigorously monitors the scoring. After configuring its roughly 30 optical scanners to catch the correct answers on a particular exam, the company performs diagnostic tests on the scanners, running through dummy answer sheets, he said. In one experiment, the scanner is fed two sheets stuck together to make sure the system shuts down, as it should, to signal an error.

The real answer sheets are fed into the machines in batches. The batches vary in size, but each includes three diagnostic sheets, which are examined to make sure the scanner is picking up the right answers. In a final visual check, an employee doublechecks two sheets in each batch.

A spokesman for the College Board, Brian O'Reilly, said he didn't know yet if the nonprofit, which administers the test, would impose more quality controls on Pearson.

''If you look at the number of tests we administer, there are very, very few where errors are made," O'Reilly said.

Some specialists said the SAT has in the past been less error-prone than many state tests for primary and secondary school students. But with only a few companies handling more and more volume, any given test is likely to see more errors, they said. Because of the federal No Child Left Behind law, states are giving 11.4 million more standardized tests this school year than last year, according to a recent analysis by Education Sector, a think tank in Washington, D.C.

The inspector general's office of the US Department of Education is planning to study test-scoring mistakes this year, although a spokeswoman said it was too early to say more about the study.

Rhoades believes that a far more radical solution is needed: a consumer protection law mandating closer government oversight of the testing agencies.

O'Reilly said he couldn't comment on proposals for more government oversight without seeing them, but added, ''The College Board does its business in a very upfront and open way and does not require an external body overseeing it."

Several testing specialists said the MCAS is one of the better-run state tests, in part because it is better funded than many state exams. The state Department of Education received about $30 million this year from state and federal government to test a half million students, said Kit Viator, director of the MCAS program.

The federal government is currently spending about $408 million on state testing under the No Child Left Behind law, said Thomas Toch, codirector of Education Sector. If all the tests were raised to the standard of MCAS, he said, the government would be spending $860 million.

Yet the MCAS test has still experienced some high-profile gaffes; last year there were errors in 1,500 10th-grade English test booklets. In 2002, hundreds of students who were initially given failing scores on the MCAS were able to pass when alternative answers were found to an eighth-grade history question and a 10th grade math question.

There have been a number of lawsuits in recent years over standardized testing errors. Pearson, the company now scanning the SAT results, settled with Minnesota students after 8,000 were given lower scores than they deserved. Some had been denied high school diplomas. Would-be teachers have also sued another company after 4,100 were erroneously flunked on a licensing exam.

Richard Walcek, head of guidance at Framingham High School, said he sees no need for serious concern about the scoring of standardized tests, even though nine Framingham students out of roughly 400 who took the SAT in October received erroneous scores.

As a guidance counselor, he works with students who take numerous tests given by The College Board, including the PSAT and Advanced Placement exams.

''It is a massive industry," Walcek said. ''Errors are rare."

Bombardieri can be reached at bombardieri@globe.com.