Pathologists in medical laboratories creating laboratory-developed tests (LDTs) should be aware that some in the scientific community want more transparency about technology and methods
Developers of clinical laboratory tests and medical diagnostic technologies might soon be feeling the pressure to increase their push for transparency and standards that ultimately would make replication easier.
That’s thanks to a review project’s inability to reproduce results from three of five high-profile cancer studies.
The review project is called the Reproducibility Project: Cancer Biology and is a collaboration between network provider Science Exchange of Palo Alto, Calif., and the Center for Open Science in Charlottesville, Va. They attempted to independently replicate selected results from high-profile cancer biology papers in an open fashion.
Reproducibility Failures Spark Controversy
A summary of results from the investigators’ first set of attempts to replicate original research findings was published the January 19, 2017, issue of eLife, the open-access journal of eLife Sciences. The first five Replication Studies included two successful replications of original papers, one failed attempt, and two results that could not be interpreted.
“Reproducibility is a central feature of how science is supposed to be,” said Brian Nosek, PhD, Co-Founder and Executive Director, Center for Open Science, in a National Public Radio article. Nosek, a Professor of Psychology at the University of Virginia, also spearheaded a 2015 review study of findings from 100 psychology papers that found two-thirds could not be reproduced.
University of Virginia Psychology Professor Brian Nosek, PhD, is Co-founder and Executive Director of the Center for Open Science, which is working with network provider Science Exchange to replicate high-impact cancer studies in an open and transparent fashion. Though the “Reproducibility Project,” as it’s dubbed, replicated only two of the first five cancer studies it reviewed, Nosek says the process of attempting to replicate study findings is as important as the results. (Photo copyright: UVAToday.)
John Ioannidis, MD, DSc, a Stanford University professor who serves on the Center’s advisory board, said in a Washington Post article, “Scientists are very smart people, but smart people can still fool themselves. You have a high risk of seeing faces in the clouds. They just appear once, and then they’re gone.”
But Nosek argues the process of attempting to replicate cancer findings is as important as the project’s results.
“The goal of this project is to say, ‘We are going to investigate reproducibility, see if we run into problems, [locate] where those problems are, and try to identify what are the potential causes of irreproducibility? What gets in the way of being able to confirm a result?’” Nosek said in a UVAToday interview.
What Did the Original Laboratories Do?
Though the Reproducibility Project does not involve studies about specific diagnostic tests, its results may increase public pressure on all researchers to follow standards aimed at improved research reproducibility. Because most scientific papers do not provide a recipe for replicating an experiment, one of the project’s challenges has been to determine what the original laboratories did.
“I’ve done it myself: you reference a previous paper and that one references a paper and you’ve gone years and the methodology doesn’t exist,” Timothy Errington, PhD, Metascience Manager at the Center for Open Science, said in an article published in The Atlantic. He added, “We spent a boatload of time trying to get back to ground zero.”
The reasons why a study’s findings cannot be replicated are not always clear-cut. Differences in techniques, as well as the mice used in an experiment, could cause a replication effort to fail, the Washington Post article noted.
Because biological systems are so variable, Charles Sawyers, MD, cautions against assuming studies that fail to be replicated are flawed. Sawyers is an eLife Senior Editor and a Cancer Biologist at Memorial Sloan Kettering Cancer Center in New York.
“People make these flippant comments that science is not reproducible,” Sawyers stated in an article published in Science. “These first five papers show there are layers of complexity here that make it hard to say that.”
In fact, in a January 19, 2017, editorial, the editors of eLife cautioned “not to over-interpret” initial results. They noted that the “approach taken by the Reproducibility Project: Cancer Biology is itself an experiment and, again, it is too early to say whether it is working.”
Who Watches the Watchers?
Reviewers anticipate completing nearly 30 replication studies. However, the Reproducibility Project’s initial results have generated backlash from scientists whose work is under the microscope.
Atul Butte, MD, PhD, Director of the Institute of Computational Health Sciences at the University of California at San Francisco, said in the Washington Post article that he was “annoyed” about the way the project handled a study he co-authored. He said the replication team essentially found the same results but took an extra analytical step that rendered them statistically insignificant.
“They altered the methodology. If you replicate the study, why change the statistical methods at the end?” Butte asked. He added, “Who watches the watchers?”
Erkki Ruoslahti, MD, PhD of the Sanford Burnham Prebys Medical Discovery Institute in La Jolla, Calif., also criticized the Reproducibility Project for failing to contact him to discuss why reviewers could not replicate his results. He fears the resulting negative publicity could harm cancer research.
“I am really worried about what this will do to our ability to raise funding for our clinical development,” he wrote in an e-mail to Shots, the online health stories channel of the NPR Science Desk. “If we, and the many laboratories who have reproduced our results, are right and the reproducibility study is wrong, which I think is the case, they will not be doing a favor to cancer patients.”
Nevertheless, Daniel Engber writing for Slate, argued the Reproducibility Project goes to the heart of challenges facing cancer research.
“If there’s no way to reproduce experiments, there’s no way to know if we can reproduce results,” wrote Engber, noting the Reproducibility Project has shown that attempting to replicate cancer research is a “hopeless slog.”
“We’ll never understand the problems with cancer studies, let alone figure out a way to make them ‘hackable,’ until we’ve figured out a way to make them reproducible,” he noted.
Pathologists and clinical lab managers should take this developments as one more sign that the bar is being raised on quality and transparency. Consumers of all kinds—be they other researchers, physicians who may want to use these tests with their patients, or payers seeking to validate the accuracy and utility of the diagnostic technology—are progressively demanding more access to data and the methods used during scientific studies.
—Andrea Downing Peck
Related Information:
Reproducibility in Cancer Biology: Making Sense of Replications
Reproducibility in Cancer Biology: The Challenges of Replication
What Does It Mean When Cancer Findings Can’t Be Reproduced?
Massive Study Reports Challenges in Reproducing Published Psychology Findings
Can Cancer Research Findings Be Replicated? It’s Complicated
Drug Development: Raise Standards for Preclinical Cancer Research
Rigorous Replication Efforts Succeeds for Just Two of Five Cancer Papers
