Is science really facing a reproducibility crisis?

NAS calls for US lawmakers to bring change also brings warning that crisis talk may ultimately ‘stifle frontier discoveries’

April 23, 2018
twins sunflowers
Source: Getty

US researchers have called for Congress to pass a new science reform act to prevent government agencies from making decisions based on “abused statistics” amid claims of a reproducibility crisis in science.

Publishing an extensive new report, the National Association of Scholars – a network of academics and campaigners set up in support of “intellectual freedom” – has warned of “shoddy ‘science’ flooding journals, conferences and news releases” in the US.

Too much of this science is irreproducible, and statistics are being misinterpreted by government bodies and by members of the judiciary system, according to the NAS, which is often considered to have politically conservative leanings, although it states that it has no political affiliation.

The report, The Irreproducibility Crisis of Modern Science, was presented to policymakers including Lamar Smith, chairman of the House Committee on Science, Space and Technology, in Washington on 17 April. In it, the NAS offers a number of recommendations for improving scientific standards and the treatment of data in academia and in the law.

“Federal and state judiciaries should review their treatment of scientific and social-scientific evidence in light of the crisis of reproducibility,” the report states. “While judges generally have maintained a degree of scepticism toward scientists’ and social scientists’ claims to provide authoritative knowledge, such claims have influenced judicial decision-making, and have helped to weave the nation’s tapestry of controlling precedent.”

The group argues that legislation should be enacted to oblige researchers to employ a reduced margin of error with regard to statistical standards. It also says that researchers “should make their data available for public inspection after publication of their results”.

“Government agencies should fund scientists’ efforts to replicate earlier research,” the report continues, and should prioritise grant funding for those who “pre-register their research protocols”, meet “new best practice standards” and make their data publicly available through open-access channels.

Federal agencies have already begun to implement such policies, the authors note, but the government, “which both funds and relies upon statistically driven research, should also work to reform science” by implementing a “standardised” approach.

Speaking to Times Higher Education after the presentation of the report, David Randall, director of research at the NAS and co-author of the report, said that the reproducibility crisis narrative had been an “ongoing, long-term and serious problem for the conduct of scientific research”. 

Responding to the report, Daniele Fanelli, a fellow in methodology at the London School of Economics, said that he agreed with some of the recommendations made but remained “sceptical” of others, while adding that, fundamentally, the report gave him a “weary” feeling. “The report premises embody perfectly the kind of ‘crisis narrative’ that I think is factually incorrect and unnecessarily damaging for the public’s support for science,” he told THE.

A paper published by Dr Fanelli last month, “Is science facing a reproducibility crisis?”, rejects the crisis narrative, arguing that the debate distracts from more important issues in science.

But he agreed with the NAS recommendation that the quality of science could be improved by introducing better teaching of statistics from a young age. “A great number of ill-advised debates and flawed evidence would disappear not just from science, but from society at large,” Dr Fanelli said.

“I am generally more sceptical, and sometimes concerned, about suggestions to impose universal standards of research or publication practices on all fields,” he continued. “We do not have standards even to define what reproducibility exactly consists in, let alone how to measure it and enforce it. 

“Moreover, in making such recommendations we are oblivious to the potential costs that these initiatives entail in terms of resources, time, and possible stifling of frontier discoveries.”

Please login or register to read this article

Register to continue

Get a month's unlimited access to THE content online. Just register and complete your career summary.

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments

Reader's comments (2)

A thorough, albeit too gentle, appraisal of the NAS report has been provided by Daniele Fanelli, April 19, 2018, on his blog at . Fanelli, in essence, disagrees with virtually all the major novel claims of the NAS report, while he supports all the unoriginal claims that most statisticians have long supported. The report is a betrayal of scholarship and those of us who belong to the organization. It was prepared by two historians with no credentials whatsoever in subject, shows little understanding of many of the topics on which it makes recommendations, and underwent no peer review by competent statisticians. It never would have been accepted in its present form by any scientific journal or other outlet with high, objective standards.
Perhaps both Fanelli and the NAS authors play a disingenuous political game; it would be a disgrace if the crisis narrative were to be appropriate by conservatives with liberals trenching under 'no-crisis'. Hope this does not happen and that we can discuss the crisis openly and pro-actively. See my comments under Fanelli's piece:

Have your say

Log in or register to post comments