NASA releases data on pilot survey
Format makes analysis difficult
NASA grudgingly released some results yesterday from an $11.3 million federal air safety study it previously withheld from the public over concerns it would upset travelers and hurt airline profits.
It published the findings in a format that made it cumbersome for analysis by outsiders.
The unprecedented research conducted over nearly four years relates to safety problems identified by some 29,000 pilots interviewed by telephone.
Earlier characterizations from people who have seen the results said they would show that events like near-collisions and runway interference occur far more frequently than previously recognized. The data was based on interviews with about 8,000 pilots per year from 2001 until the end of 2004.
The NASA website shows formatted, printed reports that the space agency scrubbed to ensure none of the pilots who were promised anonymity could be identified. The 16,208 pages were posted as NASA officials began a telephone news conference, allowing no time to look at the material and ask them questions about it. NASA did not provide documentation on how to use its data, nor did it provide keys to unlock the cryptic codes used in the dataset.
NASA administrator Michael Griffin told reporters the agency typically releases information in Adobe System's portable document format, known as pdf, which presents the information on formatted, printed pages. But there are dozens of reports available from NASA's website about other subjects in
Griffin said NASA wanted to ensure that no one modified the survey results and circulated false data as NASA's research product. He said even inexpensive optical character recognition software could convert the formatted reports. Such software can risk introducing errors in the data as it performs these conversions.
"We've gone the extra mile with this data, and well beyond our original intentions," Griffin said.
He dismissed suggestions that NASA chose to release the data late on New Year's Eve, when the public is distracted by holidays and news organizations are thinly staffed.
"We didn't deliberately choose to release on the slowest news day of the year," Griffin said.
NASA drew harsh criticism from Congress and news organizations for keeping the information secret. Rejecting an Associated Press request under the Freedom of Information Act, NASA explained that it did not want to undermine public confidence in the airlines or hurt airline fortunes.
Griffin later overruled his staff and promised Congress that he would release at least some data by the end of the year.
NASA's survey, the National Aviation Operations Monitoring System, was launched to see if a massive pilot survey would help pinpoint problems and prevent accidents. Survey planners said it was unique because it was a random survey, with an 80 percent response rate, that did not rely on pilots to take the initiative to report problems but rather reached out and interviewed them.
Griffin said NASA never intended to analyze the data it collected, but rather they planned to pass on its methodology to the aviation community.
He said he had only looked at a few results, but that, "It's hard for me . . . to see any data here that the traveling public would care about or ought to care about."
Pilots were asked how many times they encountered safety incidents such as near-collisions, equipment failure, runway interference, trouble communicating with the tower and unruly passengers.
Griffin outraged some NASA employees by saying the project had been poorly managed and its methodology not properly vetted. Survey experts who worked on it, however, said they used state-of-the-art industry techniques and carefully validated the results.
NASA's handling of the matter prompted a congressional investigation and investigations by its inspector general and by a union representing NASA workers.
Jon Krosnick, a Stanford University professor who helped design the project for NASA, said the released data were inadequate and "intentionally designed to prevent people from analyzing the rates properly and are designed to entrap analysts into computing rates that are much higher than the survey really shows."
Federal Aviation Administration officials in a recent telephone interview questioned the project's results showing more safety incidents than the FAA's own data.
"It's just something that were going to have to try and understand," Peggy Gilligan, FAA deputy associate administrator, said. "We are always interested in any kind of safety data, but we always want to look at it in terms of its quality, its quantity and how we're going to use it and what assumptions underlie it."
She noted NASA's interview questions didn't track specifically with FAA report language and said pilot responses were their subjective views over 30- to 90-day time frames.
FAA officials also said multiple pilots could have seen the same events, which Krosnick said should easily be taken into account with corrective tools that were designed to make sure the survey did not skew totals.
The Air Transport Association of America, an organization representing major airlines, responded to the NASA release with a statement praising US skies as the safest in the world and describing NASA's study "as not designed to capture real-time, verifiable data."
Representative Bart Gordon, Democrat of Tennessee, chairman of the House Science and Technology Committee, urged NASA to finish reviewing the data for further release as soon as possible.