On Wednesday I wrote about growing skepticism among psychologists about the validity of behavioral priming. The controversy reflects more general critiques that have been made about research methods in psychology and other fields in recent years. Together, these critiques suggest that false-positives—when an experiment detects a causal relationship where none in fact exists—are published more often in academic journals than has been supposed.
One of the main problems that research watchdogs point to is something called the “file-drawer effect”—the tendency among researchers to publish their positive results while tucking their negative results in a drawer and forgetting about them.
This is problematic for a number of reasons. For one, negative results still contribute useful knowledge, even if they won’t make a researcher’s career. And two, it’s important to publish negative results as a check against false-positives (even a completely spurious experiment might turn up a false-positive one time in 100; but unless the negative results are also published, you wouldn't know to discount the positive result).
There are a number of proposals circulating that would address this problem. One, AllTrials, is a petition calling for all clinical trials to be registered in a central database before they begin. This would ensure that methods don’t change midstream and that negative results are made public. It would also prevent scenarios where an experiment is run a number of times but only the single positive result is submitted for publication. The AllTrials proposal is similar to decisions that two leading medical journals—The Lancet and the Journal of the American Medical Association—have made on their own, to only publish the results of experiments that are registered ahead of time.
The federal government has also recognized the dangers of the file-drawer effect. In 1997 the Food and Drug Administration Modernization Act established Clinicaltrials.gov, a database to which researchers are expected to post results of all clinical trials within a year of conducting them. The requirement is minimally enforced, however. A 2009 review of the database published in PLoS Medicine found major holes in the data posted to Clinicaltrials.gov and concluded that, as currently operated, the site is of “limited” value.
Image courtesy of American Libraries
The author is solely responsible for the content.
Leon Neyfakh is the staff writer for Ideas. Amanda Katz is the deputy Ideas editor. Stephen Heuser is the Ideas editor.
Guest blogger Simon Waxman is Managing Editor of Boston Review and has written for WBUR, Alternet, McSweeney's, Jacobin, and others.
Guest blogger Elizabeth Manus is a writer living in New York City. She has been a book review editor at the Boston Phoenix, and a columnist for The New York Observer and Metro.
Guest blogger Sarah Laskow is a freelance writer and editor in New York City. She edits Smithsonian's SmartNews blog and has contributed to Salon, Good, The American Prospect, Bloomberg News, and other publications.
Guest blogger Joshua Glenn is a Boston-based writer, publisher, and freelance semiotician. He was the original Brainiac blogger, and is currently editor of the blog HiLobrow, publisher of a series of Radium Age science fiction novels, and co-author/co-editor of several books, including the story collection "Significant Objects" and the kids' field guide to life "Unbored."
Guest blogger Ruth Graham is a freelance journalist in New Hampshire, and a frequent Ideas contributor. She is a former features editor for the New York Sun, and has written for publications including Slate and the Wall Street Journal.
Joshua Rothman is a graduate student and Teaching Fellow in the Harvard English department, and an Instructor in Public Policy at the Harvard Kennedy School of Government. He teaches novels and political writing.