‘Publish or perish’ blamed for extreme results

Methodological controversy and a “publish or perish” culture explain why papers from US behavioural studies are more likely to exaggerate findings.

August 28, 2013

That is the conclusion of research by Daniele Fanelli, a Leverhulme early-career fellow in science, technology and innovation studies and John Ioannidis, C. F. Rehnborg professor in disease prevention at Stanford University.

The researchers looked at nearly 1,200 studies in health-related fields which were combined into 82 meta-analyses published between 2009 and 2012.

Their findings confirm earlier suspicions that studies based on behavioural, as opposed to physiological, data are more inclined to report “extreme effects”.

This is assessed according to the extent to which each paper’s reported results deviated significantly from the size of effect reported in the meta-analysis, which the researchers assume “should approximate the true effect that primary studies were trying to measure”.

The paper, US studies may overestimate effect sizes in softer research, published in Proceedings of the National Academy of Sciences of the United States of America, does not report the same effect in non-behavioural research.

The authors suggest the exaggeration of results is particularly rife in behavioural studies due to the field’s complexity and its lack of consensus over the correct methodological approaches.

“When choices are less rigidly determined by theory, they are more likely to be influenced, consciously or unconsciously, by scientists’ own beliefs, expectations, and wishes, and the most basic scientific desire is that of producing an important research finding,” the paper says.

The authors find that the tendency to exaggeration was particularly prevalent in US-based studies, which were also more likely to deviate in the direction predicted by the paper’s hypothesis. 

They suggest this is explained by “the fact that researchers in the US have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards.

“This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesising, and other more or less questionable practices.”

paul.jump@tsleducation.com

You've reached your article limit

Register to continue

Registration is free and only takes a moment. Once registered you can read a total of 6 articles each month, plus:

  • Sign up for the editor's highlights
  • Receive World University Rankings news first
  • Get job alerts, shortlist jobs and save job searches
  • Participate in reader discussions and post comments
Register

Have your say

Log in or register to post comments

Featured Jobs

Professor-Keith Cameron Chair of Australian History UNIVERSITY COLLEGE DUBLIN (UCD)
Senior Procurement Officer UNIVERSITY OF THE WEST OF SCOTLAND
Clinician, Small Animal Emergency Services UNIVERSITY COLLEGE DUBLIN (UCD)
Director COVENTRY UNIVERSITY COLLEGE

Most Commented

question marks PhD study

Selecting the right doctorate is crucial for success. Robert MacIntosh and Kevin O'Gorman share top 10 tips on how to pick a PhD

India, UK, flag

Sir Keith Burnett reflects on what he learned about international students while in India with the UK prime minister

Pencil lying on open diary

Requesting a log of daily activity means that trust between the institution and the scholar has broken down, says Toby Miller

Construction workers erecting barriers

Directly linking non-EU recruitment to award levels in teaching assessment has also been under consideration, sources suggest

Microlight pilot flies with flock of cranes

Reports of UK-based researchers already thinking of moving overseas after Brexit vote