Journals’ statistics rules ‘help tackle reproducibility crisis’

Study outlines how publishers’ guidelines led to adoption of positive research practices

May 29, 2017
Statistics on digital screen

Academic journals may be able to help tackle science’s reproducibility crisis by issuing guidelines on the use of statistics in papers submitted for publication, researchers say.

A study found that introducing recommendations on the responsible use of statistics in psychology journals was followed by an increase in authors reporting their findings using certain good research practices.

Such improvements could be key to tackling the crisis of confidence in several fields of science where researchers are struggling to replicate the findings of other academics. Many believe that the use of questionable research practices is to blame for this problem, which is seen most prominently in psychology and the life sciences.

Scientists can bias their evidence towards a certain conclusion by omitting unfavourable data from an analysis or using a summary of data instead of the original measures, for example. But in recent years there has been a growing movement to counter these bad habits.

In 2014 the journal Psychological Science introduced specific guidelines for researchers, asking them not to describe their results as “statistically significant” or not and instead to report a statistical measure that describes their confidence in the findings. It also asks them to outline why certain data are included or excluded from analysis and awards badges for making the full datasets and materials used in the research openly available, among other things.

The study authors, led by David Giofre, senior lecturer in the School of Natural Sciences and Psychology at Liverpool John Moores University, compared what effect these guidelines had on the reporting of statistics in articles in this journal to that of an equivalent periodical, the Journal of Experimental Psychology: General. This journal refers authors to recommendations on statistics published by the American Psychological Association rather than having its own.

“Our findings suggest that journal specific submission guidelines may encourage desirable changes in authors’ practices,” write Dr Giofre and his colleagues in their paper, published in Plos One.

Over the three years studied, the researchers found that the use of confidence intervals increased by 41 percentage points in papers published in Psychological Science. The number of papers describing their criteria for excluding data from analysis increased by 53 percentage points, and the number making full datasets available increased by 36 percentage points.

The use of these practices also increased in the Journal of Experimental Psychology: General, but not to the same extent. “[T]here were larger and broader changes in [Psychological Science],” the authors say.

Although the authors admit that they cannot say for sure whether the guidelines actually changed the way researchers reported their results, they say that they may be useful in changing behaviour.

Brian Nosek, professor of psychology at the University of Virginia, said that most researchers want their work to be transparent and reproducible.

“The problem is that the culture does not provide incentives for it,” he said. “If transparency and reproducibility are not part of the journals’ requirements for publication, then there is little reason for researchers to do it.”

However, Professor Nosek added that funders and institutions also have a role to play in fostering good research practices.

Daniele Fanelli, a senior research scientist at Stanford University, agreed that journals have an “important role” to play in guaranteeing the quality of publications but that the effect of regulations should not be “overestimated”.

holly.else@timeshighereducation.com

You've reached your article limit.

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 3 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Most Commented

Monster behind man at desk

Despite all that’s been done to improve doctoral study, horror stories keep coming. Here three students relate PhD nightmares while two academics advise on how to ensure a successful supervision

Female professor

New data show proportion of professors who are women has declined at some institutions

opinion illustration

Eliminating cheating services, even if it were possible, would do nothing to address students’ and universities’ lack of interest in learning, says Stuart Macdonald

Reflection of man in cracked mirror

To defend the values of reason from political attack we need to be more discriminating about the claims made in its name, says John Hendry

But the highest value UK spin-off companies mainly come from research-intensive universities, latest figures show